Creation of an automated data collection system from dynamic websites

Description:

A professional programmer is needed to develop a data collection automation system. The project includes working with dynamic elements, bypassing restrictions, the ability to manually edit parameters, and ensuring reliable data protection.

Main tasks:

1. Data collection:

Automation of information search (name, address, phone numbers, business phone numbers) on the platform.

Processing lists of relatives and collecting their contact details.

Saving data to Google Sheets or Excel.

2. Flexibility of search parameters:

The ability to edit search parameters:

Pause time between actions (for example, between searching for cases, switching between profiles, loading pages).

The number of cases to process per hour (configuration parameters should be accessible through the program interface).

The ability to quickly change time settings manually depending on needs.

3. Working with dynamic elements:

Using Puppeteer, Selenium, or Playwright to process pages with JavaScript rendering.

Ensuring stable operation during dynamic data loading.

4. Simulating user actions:

Randomizing pauses between actions.

Emulating mouse movements, clicks, and other user interactions.

Random transitions and pauses to reduce the risk of blocking.

Simulating user actions (mouse movement, clicks, scrolling the page, delay on the page, pauses between actions)

adding "human" delays, random clicks, errors, mouse movements, randomly hovering the cursor incorrectly, lingering on the page.

Add a site search function

Add mouse and keyboard simulation

Cursor movement, random clicks, scrolling before pressing the search button

5. Bypassing restrictions:

Using proxies and account rotation.

Solving CAPTCHA (automatic + manual).

6. Error handling:

Automatic restart during failures.

Error logging and notifications.

Manual fallback for critical failures.

7. Security:

Using JWT tokens to protect data.

Encrypting logins and passwords.

8. Monitoring changes:

Automatic tracking of changes in the site structure (DOM).

Quick adaptation of the program to new conditions.

9. Documentation and source code:

Complete source code with comments.

Installation, configuration, and search parameter modification instructions.

Bug fixing guarantees within 15 days.

Technical support for 30 days.

Candidate requirements:

Experience with Puppeteer, Selenium, or Playwright, Python, JavaScript (Node.js)

Knowledge of proxies and bypassing anti-bot protection.

Skills in automating complex processes and working with dynamic elements.

Ability to test performance and troubleshoot.

Willingness to provide documentation and consultation on working with the program.

Additionally:

The program should be flexible, with the ability to:

Edit pause times.

Configure the number of cases per hour.

Manage account rotation and proxy parameters.

User-friendly interface.

Timeline:

21-31 days.

Important, signing a contract between both parties is mandatory!!!