Kafka, FastAPI, Telegram Bot & Celery Worker
Project Description
This project was developed for a client in the cybersecurity industry, aiming to automate the collection of OSINT (Open Source Intelligence) data. The system utilizes Kafka for message handling, FastAPI for building the backend API, Celery for background task management, and a Telegram bot for user interaction.
Client's Goal:
The client wanted an automated system that could fetch OSINT data using Shodan and external bots, process this information asynchronously, and provide results to users via a Telegram bot. The challenge was to integrate multiple services and ensure smooth communication across the system.
Challenges:
Ensuring reliable message exchange between services using Kafka.
Setting up Celery workers for asynchronous data processing.
Solution:
I designed a scalable solution where Kafka facilitates communication between the FastAPI server, Celery workers, and the Telegram bot. The system efficiently handles requests, fetches data from Shodan, processes it in the background, and sends the results back to users in real-time. By leveraging Celery, I was able to ensure that all background tasks (like fetching and processing data) run smoothly without blocking the main API, which enhanced the user experience.
GitHub Repository: https://github.com/MLokatsiun/qos_bot.git
https://github.com/MLokatsiun/qOs_API.git
This project was developed for a client in the cybersecurity industry, aiming to automate the collection of OSINT (Open Source Intelligence) data. The system utilizes Kafka for message handling, FastAPI for building the backend API, Celery for background task management, and a Telegram bot for user interaction.
Client's Goal:
The client wanted an automated system that could fetch OSINT data using Shodan and external bots, process this information asynchronously, and provide results to users via a Telegram bot. The challenge was to integrate multiple services and ensure smooth communication across the system.
Challenges:
Ensuring reliable message exchange between services using Kafka.
Setting up Celery workers for asynchronous data processing.
Solution:
I designed a scalable solution where Kafka facilitates communication between the FastAPI server, Celery workers, and the Telegram bot. The system efficiently handles requests, fetches data from Shodan, processes it in the background, and sends the results back to users in real-time. By leveraging Celery, I was able to ensure that all background tasks (like fetching and processing data) run smoothly without blocking the main API, which enhanced the user experience.
GitHub Repository: https://github.com/MLokatsiun/qos_bot.git
https://github.com/MLokatsiun/qOs_API.git
Kyiv 
Available for hire