Introductory Information:
There are 2 popular websites for online auctions of used cars in the USA:
https://www.iaai.com/
https://copart.com/
Every day, except weekends, hundreds of cars are auctioned there.
There are websites that show the history of these auctions.
One of the most popular:
https://bidfax.info/
This website has a catalog by make/model and a search by VIN number of the car.
For example, you can enter the search 2G1FB1E34F9307496 and find information about the sold lot ( https://bidfax.info/chevrolet/camaro/11295493-chevrolet-camaro-ls-2015-red-36l-6-vin-2g1fb1e34f9307496.html )
It shows information about the car and also photographs.
Examples of lot pages on the auction sites can be seen by clicking on the lots on these pages:
1) https://www.iaai.com/Search?url=wpS89AfdHeLqu3OH0Uq89l93d2uHNn%2fpJ3I9uZfX1MA%3d
2) https://www.copart.com/lotSearchResults?free=true&query=camaro
Task:
It is necessary to develop an information parser and collect a database similar to BidFax.
1. Yes, I understand the volume of data. It is very large. A resource with a large disk storage size will be provided for storing information.
2. You need to collect the same information for each lot that BidFax collects, but in addition also:
- HD quality photographs
- 360-degree panoramas
- videos.
For example, you can see HD quality photos here: https://autohelperbot.com/ if you search for the same VIN code 2G1FB1E34F9307496
3. To ensure that the volume of data is not excessive, we will ignore some lots (actually more than half) based on certain criteria. For example, we will not save lots:
a) older than 2015
b) we will exclude a whole range of brands
This is only about developing the parser. That is, this is a software product that will daily update the database and file storage with information about new lots. We are independently developing a website that will display this data.
Complex Questions:
1. The main difficulty is that we need to parse not just the catalog of these sites, but also the auction results, including to see the amount for which the car was sold. How will the parser obtain information about the auctions? After all, the auction itself takes place in a separate interface. You need to log in, wait for the auction start time, and launch these interfaces to see. Everything there is extremely complicated…
2. One of the challenging tasks is to ensure that the data structure is the same. This is a question that currently causes me confusion. In particular, the tree of car makes/models is different on both sites. This needs to be standardized somehow.
3. By the way, I do not understand how different Chrome extensions know the VIN number of the car. Because on the lot page this information is hidden under asterisks, it looks like this: 2G1FA1E31D9****** but if you use, for example, this extension https://autohelperbot.com/ you can immediately see the VIN code of the car.
In addition to developing the parser, it will need to be maintained in the future. Because changes on the auction sites may lead to issues. So it is important to understand not only the cost of development but also the cost of further maintenance (rate per hour of work).
We understand the complexity of the project and know that it will not cost 100-200 dollars; this is a complex task. But we expect thoughtful and detailed responses, not copy-pasting "I will gladly complete your project!". Please respond with more or less specific proposals.