Crawler and Web mining project for LinkedIn

You will (or already have) 1) write a distributed crawler system running multiple crawlers with a command an control 2) eventually crawl over 120M pages of a social network (Linkedin) and defeat potential crawling roadblocks 3) deliver results in a text comma delineated file(s) based on our data format You will get: -Massive cloud computing accounts. There will be almost no limits. -LI Accounts -help from our Engineering team. You will get paid along the way... so don't worry about that. And this will be fun.