Budget: 27000 UAH Deadline: 7 days
Estimate - 35,000 UAH, deadline - 7 days after access to the technical specifications and code example.
For such a task, I would not create four separate scripts, but rather a general processing chain - loading, proxy, parsing, normalization, deduplication, hashing by 3 fields, exporting, and logging errors. Note that proxies and directory protection often take more time than the actual page parsing, so I will check this on the first 1-2 sites.
Questions
> which specific directories and what output is needed - CSV, database, API, or file for your system
> should deduplication be done only within one site or across all 4 sources
Similar examples from Ingello
> https://business.ingello.com/prime-eva - similar in working with product data and operational automation
> https://business.ingello.com/vorfahr - close in integrations, processing chains, and data
> https://systems-fl.ingello.com - main page for system development
To start, we need technical specifications, a code example, test proxies or requirements for them, the result format, and criteria for considering a record a duplicate. Overall, it's fine, we can keep it simple - first, we create a stable core, then we connect 4 sources =)