Brief Description:
The task is to gather information about transactions on the Uniswap V3 platform (thegraph.com). The project consists of two parts. At this stage, a cost estimate is needed for the first part – collecting historical data. Two scrapers/parsers need to be created:
1. for collecting historical data (triggered on demand, for collecting data within a specified time interval) — please specify only the cost of this part.
2. Real-time parser, which continuously collects data arriving at thegraph.com with minimal delay (this part will be developed later, after the successful completion of the first part)
Currently, we are looking for a person who can create a scraper/parser for historical data. Please provide your cost estimate and timeframe only for this part. Data is collected via API, openly accessible, i.e., not protected by captcha or other methods. If this part is successful, we will proceed with developing the real-time data parser.
For the first part – collecting historical data – strong knowledge of Python, MySQL, experience in creating parsers, and working with APIs are required.
For the second part of the project, skills in Redis, Docker, Google Pub/Sub, and real-time data parsing are necessary.
The system stores all transaction data in a MySQL database, runs background tasks using cron jobs, and is maintained and monitored with PM2. The solution is entirely backend-based without a frontend, deployed on a VPS server.
We are seeking an experienced and reliable data engineer/backend developer to create a robust and fast data collection script that gathers, analyzes, and stores data from Uniswap V3. The main requirement is collecting transactions, storing structured data in Redis cache, and streaming it into Google Pub/Sub messaging system to route each incoming transaction into our machine learning model. Additionally, data will also be stored in a database.
Project Scope
• Transaction-level data collection via Graph API from Uniswap V3 Subgraph on Arbitrum
https://thegraph.com/explorer/subgraphs/5zvR82QoaXYFyDEKLZ9t6v9adgnptxYpKpSbxtgVENFV?view=Query&chain=arbitrum-one
• Pagination. Implementation of pagination if needed for efficient collection of large data sets
• Saving data in Redis. Deduplication (for the second part of the project)
• Integration with Google Pub/Sub. Sending (push) each incoming transaction to two different pubsub topics for further processing. Also receiving output data and saving it back into Redis and the database. (for the second part of the project)
• Deploying the application/script using Docker. Ensuring Dockerfiles are error-free, uploading, deploying, and monitoring container operation in real-time.
Required Skills
• Python expert: confident in asyncio, fast data parsing, optimized network calls. Experience with batch processing is a plus.
• GraphQL: experience with API queries or any Web3 subgraph
• Redis: real-time data storage with keys and sorted sets, streams, pipelines (needed only for the second phase of the project)
• Google Pub/Sub: publishing events from received data (needed only for the second phase of the project)
• Docker: containerizing the solution with clear configuration (needed only for the second phase of the project)
• Database development: query optimization, schema planning (Redis and underlying databases) (needed only for the second phase of the project)
Technology Stack You Will Use
• Python 3.x (aiohttp/requests + graphql libs)
• Graph API / GraphQL (Uniswap V3 Arbitrum)
• MySQL Database
• Redis (Sorted Sets, Streams, Pipelines) (only for the second phase of the project)
• Google Cloud Pub/Sub (only for the second phase of the project)
• Docker (only for the second phase of the project)
Attached is the file in English of the above text.
If you are interested in the project - send your price, when you can start, and how many days the work will take (for the implementation of the first part of the project). If you need clarification for the calculation, send specific questions.
