Development of a database update system for the website using supplier data
Active online store, approximately 100 thousand products. Custom CMS, PHP, but there should be no dependencies on it. Server based on Ubuntu, all server services in Docker containers, SQL - Percona DB
Suppliers provide daily data dumps (stocks and prices) in the form of CSV files, which need to be updated directly in the SQL tables with products, bypassing the site's CMS.
Requirements:
Create a separate service (independent of our CMS) that daily takes CSV files from suppliers, updates the table with prices/stocks, and does not interfere with customers browsing the site.
Currently, there are 3 suppliers, each providing CSV files (40-80 thousand records in one file). It should be easy to add new suppliers and new input data formats (API, XML, Excel) in the future.
There should be a simple screen or status page: showing when it was uploaded, how many products were updated, if there were errors, and the ability to restart the update.
Important:
During the update, the main product table should not be locked.
Updates must be reliable: if something goes wrong, it should be possible to repeat without corrupting the data.
Logging of updates
-
1 day11 USD
1 day11 USD
Hello, I have extensive experience in data engineering: I work with various databases, ETL/ELT pipelines, etc. Where do your suppliers upload the data files? Is the upload done once a day or more frequently? Can I also take a look at the file structure (are they the same or not?)? For any additional questions, please send a private message. I look forward to collaborating.
-
14 days200 USD
487 4 0 14 days200 USD
Good day! 👋
I can create a separate service on FastAPI + Python that will daily upload CSV from suppliers and update the product table in PerconaDB directly, without tying it to a CMS.
I will foresee logging, a simple status page, and safe updates without blocking the site.
I am ready to discuss the details.
-
14 days245 USD
4272 25 0 14 days245 USD
Hello! I will write such a service in Go. But I have a question regarding the matching of products, how will this happen, is there a common attribute? There are characteristics, filters... that also need to be updated, or are we only interested in price and stock? We are not creating new cards? The estimate is approximate.
-
5 days223 USD
3184 115 2 5 days223 USD
I have extensive experience with PHP, Laravel/Symfony, and working with large CSV/SQL (100k+ records), implementing similar services for data updates through staging tables and atomic updates without locking the main database. I can create a separate service in Docker, anticipate support for various formats (CSV, XML, API), add a status page with logging, errors, and the ability to restart. I guarantee reliability, scalability, and transparent integration without being tied to a CMS.
-
5 days178 USD
4188 46 0 5 days178 USD
I will develop a separate service that will automatically upload CSV files from your suppliers daily, update the price and stock table without any ties to the CMS and without blocking the website's operation. The system will be easily scalable: you will be able to add new suppliers or other data formats (XML, API, Excel). Logging will be implemented, a reliable recovery mechanism in case of an error, and a convenient status page with a history of updates and the ability to manually restart the process. Thus, you will receive a stable and flexible solution to maintain the relevance of the catalog in a large online store.
-
3 days223 USD
307 3 days223 USD
Hello.
I have extensive experience in creating high-performance data synchronization services using PHP, Node.js, and Python, as well as working with large SQL databases (Percona/MySQL) in Docker systems. I can develop a standalone service that will reliably import supplier CSV dumps into your product tables without locking them, using batch updates and transactional safety to ensure data integrity. The system will be designed with scalability and flexibility in mind, allowing for easy integration of new suppliers and formats (CSV, XML, API, Excel) using customizable adapters. I will also create a status monitoring dashboard with logs, update history, error messages, and the ability to restart for complete transparency and control. This will ensure the full operability of your store during updates, maintaining reliable automated synchronization of stock and prices.
-
5 days111 USD
4028 11 0 2 5 days111 USD
What I will do (briefly)
Separate service in Docker (Python 3.11 + FastAPI) without CMS binding.
Daily CSV import from 3 suppliers (40–80k rows/file) with the ability to easily add new sources (CSV/XML/Excel/API).
Non-blocking price/stock updates in PerconaDB: incremental UPSERTs with batches, staging tables, atomic swaps/merges, indexes for updates.
Status page/admin panel: last run, how many updated/skipped, errors, manual restart.
-
3 days223 USD
1328 35 1 3 days223 USD
Good day. A lot of experience with such developments. What I would like to say right away. It is impossible to write a solution for any donor that is easy to manage. Different donors, different formats, different structures, etc. You will waste your time and money for nothing. Contact me if you need a quality and understandable product.
-
1 day223 USD
913 29 0 1 day223 USD
Good day.
I can write a service in C# and put it in Docker.
It will work when you upload files to it.
I will create a REST API access for convenience, or I can make a small web page.
Additional formats need to be looked at, there may be additional work.
Current freelance projects in the category Databases & SQL
Refinement of 1C UT 11 for Zebra TSD (RDP): different sound signals when scanning
22 USD
Configuration: 1C UT 11 Address warehouse Zebra TC26 TSD Work via RDP Product scanning is performed in receiving, placement, picking documents, and other warehouse operations. Current problem: Warehouse workers operate through the Zebra TSD. When scanning, they do not always… C#, Databases & SQL ∙ 53 minutes back ∙ 1 proposal |
Heal the 1C configuration
111 USD
Configuration of CRM & ERP SmartCeiling (2.8.26.0) Protection via Registration Code. Registered until the end of the year. Databases & SQL ∙ 15 hours 59 minutes back ∙ 7 proposals |
Need a 1C specialist for refinements and development.I am looking for a 1C specialist for freelance collaboration. I am currently working with a contractor who provides support and maintenance for the 1C system. However, due to the contractor's workload, there is a need for prompt execution of additional tasks, improvements, and… Databases & SQL ∙ 5 days 15 hours back ∙ 11 proposals |
Restoring Instagram after a banOn March 3, 2026, my Instagram was blocked due to a violation of rules. I believe there has been some mistake, as the page was personal with photos and videos of my personal trips. I did not post any comments anywhere. I have sent several appeals. I cannot download my files… Databases & SQL, Web Programming ∙ 6 days 2 hours back ∙ 6 proposals |
Eliminate the issue of incomplete data import from Excel files of Nova Poshta specifications into 1C:
45 USD
Goal: There is a processing in 1C for uploading the specifications of Nova Poshta. For unclear reasons, it has stopped loading some tabular data. We need to find the reason and eliminate the problem of incomplete data import from Excel files of Nova Poshta specifications into… System & Network Administration, Databases & SQL ∙ 9 days 9 hours back ∙ 16 proposals |
Lvov