Parsing million records
156 USD There is a CSV file with an assortment of 200k records with 10+ columns
One of the columns is a link - when the link is opened, there is JSON with 20+ photos
That is, we have an assortment of 200k items with 20 photos for each, totaling 4,000,000 photos
Task:
1. Upload the original CSV as it is to a MySQL database table + column "record update date"
2. Upload to a separate table with links to photos (2 million records) + key from table p.1
3. Update the tables 5 times a day with a new file (access to the file will be organized via a link)
The main task is outlined in the technical specification, and the functionality for logging and checking the correct operation of the parser is not specified - this part is to be defined by the specialist independently.
We are looking for a person with experience working on similar projects where it is necessary to quickly process millions of records.
P.S. request not to write without experience and those who rely on "AI power". Thank you
Client's review of cooperation with Illia Vasilenko
Parsing million recordsgreat job, responsible person!
Freelancer's review of cooperation with Dmitro P.
Parsing million recordsInteresting project, a lot of data) The client is clear and structured. Always in contact. Clearly formulates the technical specifications and understands their needs. I recommend!
-
3 days36 USD
3 days36 USD
Good day. It's not a problem to do this, but I would still like to first take a look at the table and the JSON. You can review my work experience in the feedback history.
-
2 days67 USD
2 days67 USD
Hello! I have been engaged in data parsing for over 3 years. I implemented a similar project, but instead of images, we worked with archives.
I will be able to implement everything quickly and efficiently! As soon as we discuss the details - I will start working.
-
3 days67 USD
414 3 days67 USD
Hello! 👋
I have experience in developing systems for processing large volumes of data (millions of records), including CSV import, working with JSON, optimizing MySQL, and automatically updating data on a schedule. I will be able to implement a stable process for processing your file with the assortment and photos.
Here’s how I will implement it:
importing the original CSV (200k records) into a MySQL table while preserving the structure + update date field;
processing JSON from the link and extracting URLs of all photos;
…
writing photos into a separate table (up to 4 million records) linked to the product;
optimizing the database structure (indexes, batch inserts, chunk processing);
automatically updating data 5 times a day via cron;
a logging system, import correctness checks, and error control;
optimizing the speed of processing large volumes of data.
I have worked on tasks that require processing millions of records and large data arrays, so I pay special attention to the performance and stability of the process.
I am ready to discuss the details of the CSV and JSON structure, after which I will be able to estimate the timeline and budget.
-
3 days67 USD
511 3 0 3 days67 USD
Hello. I have experience with parsing csv and json. I am ready to create a structured database, but first I would like to see the structure of the csv. I can also provide a script if information will be added to the csv with this structure.
-
2 days112 USD
2127 31 0 2 days112 USD
Good day, Dmytro
I have advanced knowledge of MySQL and working with similar parsing.
Feel free to contact me, I will help create a solution that works quickly and reliably.
I am waiting for feedback.
-
3 days112 USD
10038 117 0 3 days112 USD
Hello.
I develop bots and parsers in NodeJS. I am ready to take on the project. Write to me, we will discuss.
-
3 days112 USD
727 6 0 3 days112 USD
I will implement without problems, the price on the condition that there will be no blocks, please contact me, I will implement in a stack convenient for you.
-
2 days16 USD
3392 84 0 2 days16 USD
Ready to take it on.
But we need to clarify the details of the order, write!
I will implement it with a script in Python.
-
3 days112 USD
3318 70 1 3 days112 USD
Hello, Dmytro.
I have the relevant experience to implement this project.
-
3 days67 USD
3700 17 0 3 days67 USD
Hello! 🚀
I have experience working with large volumes of data: importing CSV files with hundreds of thousands of records, processing JSON via links, loading millions of rows into MySQL, building stable ETL/parsing pipelines with scheduled updates, logging, retry mechanisms, and data integrity checks. For your task, I can implement a fast loading system for the assortment, extracting photos into a separate table, updating 5 times a day.
The stack I will be using:
Backend: Python, MySQL, asyncio / aiohttp, multiprocessing
Integrations: CSV import pipeline, JSON parsing, cron / scheduler, logging, retry / monitoring
Infrastructure: Docker, VPS / server, Redis (if needed), CI/CD
…
I am ready to discuss the project details in person and am ready to start working.
Best regards, Andriy!
-
2 days33 USD
6920 109 4 1 2 days33 USD
Good day. I will be happy to collaborate. I have good experience in data parsing from files.
-
2 days67 USD
344 2 days67 USD
Good day, Dmytro!
We have experience working with large datasets — parsing Google Maps with over 200K entries, processing JSON responses, loading into PostgreSQL with deduplication and incremental updates.
Regarding your task:
1. Import CSV (200K rows, 10+ columns) into MySQL with indexes for fast searching + an update date column
2. Parse JSON from links → a separate photo table (2-4 million entries) with a foreign key to the main table
3. Automatic updates 5 times a day on schedule (cron) with logging and integrity checks
… For fast processing of millions of entries, we use batch-insert (LOAD DATA / bulk insert), parallel processing of JSON, and process monitoring.
We are ready to show an example of our approach after reviewing the CSV and JSON. We propose working through Safe.
-
1 day89 USD
172 1 1 1 day89 USD
Good day. I am ready to complete this project as I have extensive experience in application development.
-
3 days33 USD
6262 74 1 3 days33 USD
Good day. I have experience with similar tasks in processing csv and json. It would be good to see a sample of the data. I will be happy to help.
-
3 days67 USD
1328 35 1 3 days67 USD
Hello. I have extensive experience working with MySQL databases. I will complete the entire project in PHP, feel free to contact me.
-
3 days22 USD
5011 41 4 1 3 days22 USD
Good day!
I have significant experience working on projects that require the rapid processing of millions of records, including efficient data parsing from APIs and working with MySQL. I am ready to implement your task of updating the assortment and photos.
Write to me in private messages, and we will clarify the details.
-
2 days67 USD
9003 367 0 2 days67 USD
Good day
I am interested in your project.
I would like to discuss everything in more detail.
-
Я правильно зрозуів:
Беремо рядок CSV.
Витягуємо URL до JSON.
Робимо HTTP-запит до цього JSON (200,000 запитів!).
Парсимо масив (20+ лінків).
Записуємо в БД.
-
Current freelance projects in the category Data Parsing
Consultation on parsing Instagram account subscribersHello. It is necessary to conduct a preliminary assessment of the feasibility of the following task. I have a list of Instagram accounts. The goal is to obtain contact information (primarily email addresses) of users who follow these accounts. Previously, I encountered companies… Data Parsing ∙ 1 day 5 hours back ∙ 12 proposals |
A specialist is needed to find contacts of decision-makers in Ukraine.It is necessary to gather a database (or ready database) of contacts of decision-makers (DMs) in companies in Ukraine. Information Gathering, Data Parsing ∙ 1 day 10 hours back ∙ 14 proposals |
Need to scrape data from LinkedIn
We need to scrape data from LinkedIn based on our list. For each entry, we need to find and collect available data if it exists on the LinkedIn profile, including the profile picture on the LinkedIn social network, email address, links to social media, company website, and… Data Parsing ∙ 1 day 16 hours back ∙ 24 proposals |
Parsing and classification of data
We are looking for a developer to implement a system for collecting and structuring data from open sources. We have a database of small business owners in the USA, which contains the person's name, company name, address, and state. It is necessary to build a process for… Web Programming, Data Parsing ∙ 1 day 17 hours back ∙ 38 proposals |
Svitlahata
17 USD
It is necessary to import 1819 products from the XML/YML feed of Prom.ua to OpenCart 3. A ready XML file is available, which contains product names, descriptions, prices, photos, specifications, manufacturers, and categories. Requirements: import all products to OpenCart… Content Management Systems, Data Parsing ∙ 2 days 20 hours back ∙ 33 proposals |
Kyiv