1. Task Description:
• It is necessary to fix the issue where lots from the Copart auction are parsed via CSV but do not appear in the catalog until they are indexed in Elasticsearch.
2. Goals:
• Ensure correct parsing of lots from Copart via CSV.
• Automatically index lots in Elasticsearch and display them in the catalog without delays.
• Ensure the preservation of all data and images in the database and their correct indexing.
3. Work Stages:
3.1. Analysis of the Current Parsing Process:
• Study the current process of parsing lots from Copart (the tool or script being used).
• Evaluate the data format (CSV) used for parsing and the process of converting this data to Parquet.
• Determine how data is transferred from Parquet to the database (Postgres/MSSQL).
3.2. Analysis of Data Processing and Indexing:
• Review the data indexing process in Elasticsearch.
• Check the Elasticsearch configuration (indexes, data types, sharding).
• Analyze indexing logs to identify the causes of delays or timeouts.
3.3. Fixing the Parsing and Indexing Issue:
• Fix the CSV processing to ensure data is correctly indexed in Elasticsearch.
• Ensure proper transmission and processing of images (especially HD versions) in the database and during indexing.
• Set up monitoring for indexing and check if lots correctly appear in the catalog after processing.
3.4. Fixing Issues with Copart File Hanging:
• Investigate the timeout issue when downloading files from Copart.
• Determine why the download process hangs and resolve it (possibly due to changes on the Copart side or connection issues).
• Check and configure logging of the process to avoid such issues in the future.
4. Technical Requirements:
• Possession of basic knowledge of working with databases (Postgres, MSSQL) and indexing in Elasticsearch.
• Understanding the principles of data parsing and working with CSV and Parquet formats.
• Access to logs of parsing, indexing, and database processes.
5. Performance Criteria:
• Lots from the Copart auction must be parsed without delays and immediately displayed in the catalog after indexing in Elasticsearch.
• All images of the lots must be correctly stored in the database and available in high quality (HD).
• Timeouts during file downloads from Copart must be eliminated.