Development of a solution for parsing/collecting data from API/websites and gathering contacts for outbound/sales.

Develop a solution (script/service/Make.com scenario) that:

1. regularly retrieves data from specified sources (API and/or websites),

2. performs website parsing

3. finds new entities that are not in our internal list,

4. applies configured filters,

5. finds the official website of the entity,

6. collects contacts of people in relevant roles for launching outbound/sales campaigns,

7. stores the result in a database/storage + provides export.

Expected Result

A ready solution that creates and maintains a contact database in the structure:

• Project website

• Email

• LinkedIn profile link

• Job title

• Twitter/X profile link

• Phone number

• Telegram handle or Telegram link

Additionally (preferably): date of finding, source (source), validation/completeness status of the contact.

Data Sources and Processing Streams (3 streams)

Case 1 — Source of the “new entries catalog” type (API/web)

Functions:

• Retrieve entries for a period and/or incrementally (new since the last run).

• Identify “new” entities relative to our internal list (CSV/Google Sheet/JSON).

• Apply filtering (configured).

• For each entity:

  • find the official website,

  • collect contacts for the required roles,

  • record in the database/storage.

Case 2 — Aggregators/directories (multiple independent sources)

Functions:

• Retrieve data from multiple aggregators/directories (via API or scraping).

• Merge/normalize entries, perform deduplication.

• Identify “new” entities (not present in our internal list).

• Apply filtering (configured).

• Find the official website and contacts for the required roles.

• Store the result.
  
  

Stream 3 — Comparing presence between sources

Functions:

• Parse/retrieve data from 3–5 platforms/directories (API/web).

• Compare the “presence” of the entity between sources (matching by name/domain/identifier).

• Identify entities that:

  • are not in our internal list,

  • meet the filters,

• For relevant entities — extract contacts + save.

Contact Collection Logic

For each selected entity:

• Determine the official website (from the source data or through additional transitions/checks).

• Collect contacts for roles (configured list of roles), for example:

  • Founder / C-level

  • Head/Lead of Partnerships / BD / Sales / Growth

  • Head/Lead of Operations / Product (and others, to be agreed)

• Contact sources:

  • “Team”, “About”, “Contact”, “Careers”, “Press” pages

  • LinkedIn (links to profiles/company, if available)

  • Twitter/X, Telegram, email/forms, phones

• Minimum requirement: store contacts only in the specified field structure.

Data, Storage, Export Requirements

• Storage: PostgreSQL or SQLite (the executor proposes the optimal option) or storage with export to Google Sheets.

• Deduplication:

  • by domain/website

  • by email

  • by LinkedIn URL

  • by Twitter/X URL

  • by phone/Telegram

• Export: at least 1 option from (CSV / Google Sheets / Airtable), preferably 2.

• Technical requirements:

  • logs,

  • error handling,

  • retry/backoff,

  • rate limiting.

Preferably:

• status “processed / already contacted” for entries.

Configuration (mandatory)

Filters and rules must be changeable without rewriting code:

• via config file (YAML/JSON) or

• via launch parameters or

• via simple admin configuration (if service).

Technologies (acceptable options)

• Python (Requests/Scrapy + Playwright) or Node.js (Playwright/Puppeteer) or Make.com.

• Priority: official APIs (if available), otherwise — careful scraping.

• Docker (preferably).

• Scheduler: cron/Celery/other scheduler.

• Run on VPS.

We will provide

• Internal list for exclusion (CSV/Google Sheet/JSON).

• Initial list of filters and roles.

• API keys (if needed).

The executor must provide

• Repository with code or Make.com scenarios (with access/export).

• README: launch, configuration, dependencies.

• Architecture description (brief) + data flow diagram.

• Demonstration run: log/screen/example of export results.

Executor Requirements

• Experience in web scraping / API integrations / lead generation automation.

• Understanding of rate limits, anti-bot protection, stability, deduplication.

• Examples of similar works (links/descriptions).