Technical Requirements - OLX Real Estate Parser Rewrite

Project Overview

The current project is a Python-based parser that extracts real estate listings from OLX.ua and sends them to an EasyBase API. The system monitors multiple OLX categories (apartments, houses, land, parking) and processes offers by extracting property details, phone numbers, and location data. When writing parser, consider that it’ll be parsing a few regions in Ukraine. So maybe there should be some split inside of codebase.

Core Functionality

• Real Estate Data Parsing: Scrapes OLX.ua listings for apartments, houses, land plots, and parking spaces

• Phone Number Extraction: Retrieves seller contact information using OLX accounts

• Location Processing: Maps coordinates to districts/microdistricts using JSON lookup files

• Data Transformation: Converts OLX format to EasyBase API format

• Account Management: Rotates through multiple OLX user accounts to avoid rate limiting

• Duplicate Prevention: Tracks parsed offers in Mongo database to avoid re-processing

Technical Requirements for Rewrite

1. Asynchronous Architecture

Mandatory: All operations must be asynchronous using async/await pattern.

• HTTP Requests: Replace `requests` with `httpx` for all API calls

• Database Operations: Use async database drivers 

• File I/O: Use `aiofiles` for JSON file operations

• Concurrency: Implement concurrent processing of multiple offers

• Rate Limiting: Async-friendly rate limiting for API calls

• Session Management: Async session handling for OLX accounts

2. Database Migration to MongoDB

Mandatory: Replace SQLite with MongoDB for all data storage. Store accounts data and urls to parse inside of mongoDB. 

3. User Credentials Rotation System

Mandatory: Implement robust credential management system.

Options:

1. **OLX User Account Rotation** (Current approach)
2. **API Developer Credentials** (Alternative approach - preferable, but should be investigated if it will be working)

Requirements:

• Automatic Token Refresh: Async background task for token renewal

• Health Monitoring: Check account status and availability

• Cooldown Management: Temporarily disable accounts that hit rate limits

• Load Balancing: Distribute requests across available accounts

• Fallback Strategy: Handle account failures gracefully

• Rotation Strategies: Round-robin, least-recently-used, random selection

5.2 Configuration Files

• `district_mapping.json`: Location mapping data
• `category_mapping.json`: OLX category to property type mapping

6. Error Handling and Monitoring

6.1 Async Error Handling

• Retry Logic: Exponential backoff for failed requests

• Circuit Breaker: Prevent cascading failures

• Graceful Degradation: Continue with available accounts

• Error Aggregation: Collect and report parsing statistics

7. Scalability Considerations

• Horizontal Scaling: Design for multiple parser instances

• Queue System: Optional message queue for offer processing

• Database Sharding: Prepare for MongoDB scaling

• Microservices Ready: Modular design for future service separation

8. Development Guidelines

8.1 Technology Stack

• aiohttp: Async HTTP client

• motor: Async MongoDB driver

• pydantic: Data validation and serialization

• asyncio: Concurrency management

• aiofiles: Async file operations

* Alternative libraries are welcomed too.

8.2 Code Organization (preferable, not obligatory)

src/
 ├── models/
 │   ├── offer.py           # Offer data models
 │   ├── account.py         # Account models
 │   └── enums.py          # Constants and enums
 ├── parsers/
 │   ├── olx_parser.py      # OLX parsing logic
 │   └── phone_extractor.py # Phone number extraction
 ├── database/
 │   ├── mongodb.py         # MongoDB connection and operations
 │   └── repositories.py    # Data access methods
 ├── accounts/
 │   ├── manager.py         # Account management
 │   └── rotation.py        # Account rotation logic
 ├── api/
 │   └── easybase_client.py # EasyBase API client
 ├── utils/
 │   ├── config.py          # Configuration management
 │   ├── async_helpers.py   # Async utility functions
 │   └── exceptions.py      # Custom exceptions
 └── main.py                # Application entry point

Simple and Practical:

• Models: Pydantic models for data validation

• Parsers: OLX-specific parsing logic

• Database: MongoDB operations and data access

• Accounts: User account management and rotation

• API: External API clients

• Utils: Shared utilities and configuration

• Flat structure: Easy to navigate and understand