Resolve the issue with parsing lots from Copart and their display in the catalog after indexing in El..

1.    Task Description:

    •    It is necessary to fix the issue where lots from the Copart auction are parsed via CSV but do not appear in the catalog until they are indexed in Elasticsearch.

    2.    Goals:

    •    Ensure correct parsing of lots from Copart via CSV.

    •    Automatically index lots in Elasticsearch and display them in the catalog without delays.

    •    Ensure the preservation of all data and images in the database and their correct indexing.

    3.    Work Stages:

3.1. Analysis of the Current Parsing Process:

    •    Study the current process of parsing lots from Copart (the tool or script being used).

    •    Evaluate the data format (CSV) used for parsing and the process of converting this data to Parquet.

    •    Determine how data is transferred from Parquet to the database (Postgres/MSSQL).

3.2. Analysis of Data Processing and Indexing:

    •    Review the data indexing process in Elasticsearch.

    •    Check the Elasticsearch configuration (indexes, data types, sharding).

    •    Analyze indexing logs to identify the causes of delays or timeouts.

3.3. Fixing the Parsing and Indexing Issue:

    •    Fix the CSV processing to ensure data is correctly indexed in Elasticsearch.

    •    Ensure proper transmission and processing of images (especially HD versions) in the database and during indexing.

    •    Set up monitoring for indexing and check if lots correctly appear in the catalog after processing.

3.4. Fixing Issues with Copart File Hanging:

    •    Investigate the timeout issue when downloading files from Copart.

    •    Determine why the download process hangs and resolve it (possibly due to changes on the Copart side or connection issues).

    •    Check and configure logging of the process to avoid such issues in the future.

    4.    Technical Requirements:

    •    Possession of basic knowledge of working with databases (Postgres, MSSQL) and indexing in Elasticsearch.

    •    Understanding the principles of data parsing and working with CSV and Parquet formats.

    •    Access to logs of parsing, indexing, and database processes.

    5.    Performance Criteria:

    •    Lots from the Copart auction must be parsed without delays and immediately displayed in the catalog after indexing in Elasticsearch.

    •    All images of the lots must be correctly stored in the database and available in high quality (HD).

    •    Timeouts during file downloads from Copart must be eliminated.