Crawler Development

I require an automated application (Crawler) that can access a car auction website, plug in certain search criteria, and then record the car data along with pictures, prices, the resulting prices returned after the auction completes in a mysql table. I’ll provide you with MySql script of the table. The application will run on our local server and update the specified tables on our live server through crone job. ## Deliverables **Site Scrapper ??" Crawler Development Program Specification** **? ** **Overview** I require an automated application (Crawler) that can access a car auction website, plug in certain search criteria, and then record the car data along with pictures, prices, the resulting prices returned after the auction completes in a mysql table. I’ll provide you with MySql script of the table. The application will run on our local server and update the specified tables on our live server through crone job. This application should get data from the “Running Auctions?? as well as from other links available on site like? “Special Auction??, “Post Auction Negotiation??, “Aucnet Shared Inventory??, “Apple??, “One Price??, “Orix One Price?? and “Aucnet AA??. **Detailed Requirements** The application will need to access the website at [[login to view URL]][1].? Credentials to log in will be provided to the selected bidder only. The following criteria (or select all data) will be specified in an .ini or other configuration file to be located in the programs application directory: ·? ? ? ? ? ? ? ? Auction Day / Date ·? ? ? ? ? ? ? ? Auction Name ·? ? ? ? ? ? ? ? Vehicle Maker ·? ? ? ? ? ? ? ? Vehicle Brand / Name These criteria would be inserted into the appropriate fields when an inquiry was made.? All vehicles are categorized in “Maker??, “Brand?? and “Model??. All makers have specific Brands while all Brands have specific Models. There will be separate tables for each New Makers/Brands/Models having foreign keys for relation. I’ll provide you with MySql script for tables. ? As the new Make/Brand/Models could be added by the site owner at any time so our application should be capable to grab the new listings and update existing tables. Make/Brand/Models are not added frequently so we can manage it by running a script manually which will be invoked by some buttons. The script will see if there is any new listing of Make/Brand/Model in the original data which is not added in our respective tables before, grab the new listings and update our existing tables. This approach will speed up our crawler. New vehicle listings along with pictures are added by the site owners regularly for the upcoming auctions. Some records are removed due to any reason those cars are not for sale anymore. Our application should be capable to grab the new records added and remove some existing records if removed from the original site at any point of time. ? All the data and images grabbed should be placed in a separate raw table called auct_lots_raw. Records grabbed will be validated automatically and then moved to a new table called auct_lots_full. ? This auct_lots_full table on our local server will be synchronized with our live server auct_lots_full table at every hour. Complete records for auctions of any particular day should be removed from the auct_lots_full table by 23:59:59 Japan Standard Time and dumped into a separate table called auct_lots_full_3. I’ll provide you the script for the table. I want to keep the 3 months data backup in this table so the ID’s of records should be managed properly. Vehicles Images downloaded should be named properly and placed in proper folders for easy access and easy to be pointed for linking with the data record of that vehicle. The data will be moved on daily basis from auct_lots_full table to auct_lots_full_3 table, but the images should be pointed from the same location. As our objective is to grab vehicles data from auction site and display at our site so the time frame does matters a lot. My experience with the car images says that we might not get all the images downloaded from the auction site and upload to our live server within time. So if this is the case then you need to grab the images urls and dump into the table for today’s auctions while a separate script should be running parallel who will be downloading images and uploading to our live server. Later when we run a query to display images on our front end site, we should be capable to call the image on a condition that if it is available on our live server then display from there and if it is not available at our live server then it could be called direct from auction site. Image files could be .jpg .gif and in some auctions it could be a pdf format. As a matter of fact, all the vehicles displayed in one auction do not sale out. There are lots of residual vehicles which might come again in the same auction next week or might come in some other auction someday in future. In this case all the vehicle data remains the same except its Lot Number. Images already grabbed could be used in this case to reduce the processing time. Also note that there can be some text in Japanese Language (Unicode). **User Interface** The application needs to have a basic UI to show its progress.? The UI should show the current criteria being processed, the number of codes already processed and the elapsed time since the application started.? It should have buttons to run a script which will update our Make/Brand/Models tables. The UI should also contain a Start and a Cancel button to start and cancel the current operation. **Resiliency** Since the application will run unattended for long periods of time, it should be resilient and be able to handle basic errors.? This means that if it gets an unexpected response from the server or some sort of error, it should be able to log the error and retry that criterion without crashing or getting hung up.? Basic tests should be performed like pulling out the network cable for a minute, and replacing it to make sure that the application can deal with that sort of situation. The application should be coded in a way that it should be resilient to minor changes which a site owner might cause down the road. Of course for the major changes in site would require us to redo the project. **Logging** The application should log all errors to a text file to be located in the application directory.? The log should show the date/time of the error, the criteria being processed at the time and a basic cause of the error or the server’s response. The crone job should return its result through email. **Platform** The application should be designed to run on a wamp/apache server. The development environment should be php, but I am open to other platforms as well. The deployment of the project and make it working should be your responsibility. **Source Code** All source code should be provided at the completion of the project and it will be our property. The code should be properly commented so that if later at any stage we need to modify the application we should have enough information available with us. **Other Notes** **I am very much concerned about the speed of the data and images we get. I’ll make the machine ready and up to date as per your requirements which should be ideally dedicated to run this application. But I want the application to run smoothly and as fast as it could be.** **I expect that once the application is built and running you should be available for support in case if it stuck at some stage. Of course if it is a kind of redo work due to site owner, I’ll pay you for your time.** **Any other point which is an integral part of the application to function properly and I might omit to express here should be considered as a part of this document. I am very open for suggestions and always welcome your inquiries regarding project clarification.**