Data Engineer - automotive analytics (data collection and storage)
We need a Data Engineer with experience in web scraping and working with large volumes of data for a used car market analysis project.
We are developing a car price analysis service - a reliable data collection system from platforms for selling used cars for further analytics and visualization.
What needs to be done?
- Web scraper - collecting listings from the largest platforms with bypassing blocks
- Database - architecture for storing millions of records with optimization for analytical queries
- API/interface - access to data for Backend developer
- Automation - daily collection, monitoring, error handling
- Historical data - storing price dynamics for trend analysis
Project features
- Large volumes - expecting 3K+ new listings daily, millions of records in the database
- Cost optimization - solutions needed for infrastructure savings (partitioning, archiving, compression)
- High performance - analytical queries should run quickly on large datasets
- Long-term support - this is not a one-time collection, the system must operate continuously
Business requirements, interaction scheme with Backend developer, and technical details - https://docs.google.com/document/d/1utRzA8TntpXpO-RSgR2lYKoIFrggE6PPFKi71P9qKxc/edit?usp=sharing
When responding, please specify:
- Experience with similar projects (data volumes, scraping, database optimization)
- Proposed technology stack
- Estimated budget and timeline
- Examples of previous work (if available)
-
1 day601 USD
1 day601 USD
Good day
I can take on the role of Data Engineer and Backend Developer
I have extensive experience parsing websites with cars such as Copart, IAAI, AUTO.RIA. I also successfully parse automotive parts websites, each with over 100k products.
I combine my parsers with an admin panel built on Django, but I can also write backend on FastAPI if you have a frontend developer.
To provide an estimate, we need to discuss the technical specifications in more detail.
On average, such projects are estimated at $1000-3000 depending on the details of the technical specifications.
-
1 day601 USD
564 2 0 1 day601 USD
Good day!
Ready to participate in both roles simultaneously. I have four years of experience in developing high-load backend and automation on Golang, of which over 2.5 years are specifically in the automotive sector. Worked with large volumes of data: collected, processed, and aggregated them at various stages.
I would be happy to discuss the details — you can message me privately. If needed, we can arrange a call in a format convenient for you.
-
7 days601 USD
1621 35 1 6 7 days601 USD
Good day. I have a ready and working solution that collects and analyzes data from Auto.ria and Olx.
If interested, I can show a demo (screen recording). The cost is $5000, with possible modifications and improvements (for an additional fee).
-
7 days334 USD
720 6 1 7 days334 USD
Hello,
I described the details in a private message.
Waiting for your response.
Current freelance projects in the category Data Parsing
A specialist in Telegram promotion is required.
29 USD
Tasks: invite real users from the username database to new chats and send messages to the target database. Only quality traffic and work with a live audience are of interest — performers using bots, fake engagement, or low-quality methods are requested NOT TO DISTURB. Work… Data Parsing, Social Media Marketing (SMM) ∙ 1 day 5 hours back ∙ 6 proposals |
Collection of B2B database of companies in Germany
40 USD
Goal: To obtain a list of potential employers (clients) for B2B mailing. Region: Munich (München) + radius of 50 km. Required niches: Construction companies (Bauunternehmen) Food enterprises (Lebensmittelhersteller, meat processing plants, bakeries) Logistics and… Data Parsing, Lead Generation & Sales ∙ 1 day 7 hours back ∙ 26 proposals |
Carrier databaseInterested in compiling a database of carriers in Ukraine for the year 2026, including tankers, tarpaulins, grain carriers, and others. It is preferable to develop a table. Information Gathering, Data Parsing ∙ 1 day 8 hours back ∙ 29 proposals |
Consultation on parsing Instagram account subscribersHello. It is necessary to conduct a preliminary assessment of the feasibility of the following task. I have a list of Instagram accounts. The goal is to obtain contact information (primarily email addresses) of users who follow these accounts. Previously, I encountered companies… Data Parsing ∙ 5 days back ∙ 12 proposals |
A specialist is needed to find contacts of decision-makers in Ukraine.It is necessary to gather a database (or ready database) of contacts of decision-makers (DMs) in companies in Ukraine. Information Gathering, Data Parsing ∙ 5 days 4 hours back ∙ 18 proposals |
Kyiv