Resolve the issue with parsing lots from Copart and their display in the catalog after indexing in El..
1. Task Description:
• It is necessary to fix the issue where lots from the Copart auction are parsed via CSV but do not appear in the catalog until they are indexed in Elasticsearch.
2. Goals:
• Ensure correct parsing of lots from Copart via CSV.
• Automatically index lots in Elasticsearch and display them in the catalog without delays.
• Ensure the preservation of all data and images in the database and their correct indexing.
3. Work Stages:
3.1. Analysis of the Current Parsing Process:
• Study the current process of parsing lots from Copart (the tool or script being used).
• Evaluate the data format (CSV) used for parsing and the process of converting this data to Parquet.
• Determine how data is transferred from Parquet to the database (Postgres/MSSQL).
3.2. Analysis of Data Processing and Indexing:
• Review the data indexing process in Elasticsearch.
• Check the Elasticsearch configuration (indexes, data types, sharding).
• Analyze indexing logs to identify the causes of delays or timeouts.
3.3. Fixing the Parsing and Indexing Issue:
• Fix the CSV processing to ensure data is correctly indexed in Elasticsearch.
• Ensure proper transmission and processing of images (especially HD versions) in the database and during indexing.
• Set up monitoring for indexing and check if lots correctly appear in the catalog after processing.
3.4. Fixing Issues with Copart File Hanging:
• Investigate the timeout issue when downloading files from Copart.
• Determine why the download process hangs and resolve it (possibly due to changes on the Copart side or connection issues).
• Check and configure logging of the process to avoid such issues in the future.
4. Technical Requirements:
• Possession of basic knowledge of working with databases (Postgres, MSSQL) and indexing in Elasticsearch.
• Understanding the principles of data parsing and working with CSV and Parquet formats.
• Access to logs of parsing, indexing, and database processes.
5. Performance Criteria:
• Lots from the Copart auction must be parsed without delays and immediately displayed in the catalog after indexing in Elasticsearch.
• All images of the lots must be correctly stored in the database and available in high quality (HD).
• Timeouts during file downloads from Copart must be eliminated.
-
1094 10 0 Good evening!
I have an API from one team, it helps to quickly parse Copart/Iaai.
There is an example in the portfolio.
Write to me, we will discuss the details and get started!
Respectfully, Andriy!
-
8753 60 0 1 Hello!
We have experience in parsing and integration with Elasticsearch. We will quickly fix indexing issues, optimize processes, and ensure stable operation.
Our rate is $20 per hour.
I write in Python. I hold 3rd place on the platform for this language.
Portfolio:Freelancehunt
-
Valeriu Y. company
парсинг через cvs думаю не очень хорошая идея, там много данных отсуствуют, рекомендую лучше использовать готовые решения, что-то типо carstat.dev
насчет postgresql, думаю данные можно писать сразу в БД и elasticsearch, без использования parquet
объем данных не такой велик чтоб использовать parquet -
Current freelance projects in the category Databases & SQL
Refinement of 1C UT 11 for Zebra TSD (RDP): different sound signals when scanning
22 USD
Configuration: 1C UT 11 Address warehouse Zebra TC26 TSD Work via RDP Product scanning is performed in receiving, placement, picking documents, and other warehouse operations. Current problem: Warehouse workers operate through the Zebra TSD. When scanning, they do not always… C#, Databases & SQL ∙ 1 day 8 hours back ∙ 5 proposals |
Heal the 1C configuration
111 USD
Configuration of CRM & ERP SmartCeiling (2.8.26.0) Protection via Registration Code. Registered until the end of the year. Databases & SQL ∙ 1 day 23 hours back ∙ 8 proposals |
Need a 1C specialist for refinements and development.I am looking for a 1C specialist for freelance collaboration. I am currently working with a contractor who provides support and maintenance for the 1C system. However, due to the contractor's workload, there is a need for prompt execution of additional tasks, improvements, and… Databases & SQL ∙ 6 days 22 hours back ∙ 12 proposals |
Restoring Instagram after a banOn March 3, 2026, my Instagram was blocked due to a violation of rules. I believe there has been some mistake, as the page was personal with photos and videos of my personal trips. I did not post any comments anywhere. I have sent several appeals. I cannot download my files… Databases & SQL, Web Programming ∙ 7 days 9 hours back ∙ 6 proposals |
Eliminate the issue of incomplete data import from Excel files of Nova Poshta specifications into 1C:
45 USD
Goal: There is a processing in 1C for uploading the specifications of Nova Poshta. For unclear reasons, it has stopped loading some tabular data. We need to find the reason and eliminate the problem of incomplete data import from Excel files of Nova Poshta specifications into… System & Network Administration, Databases & SQL ∙ 10 days 16 hours back ∙ 16 proposals |