Data parsing

Data Parsing, Python — incorrectly specified categories?

110 USD

Project translated automatically. Log in or register, to view the original Technical task 1. General description It is necessary to develop a software solution for extracting data about suppliers from the state procurement portal www.goszakup.gov.kz. The data should be obtained from contracts won by suppliers for the year 2023 and the first 3 months of 2024. After extraction, the data of each supplier must be checked for presence in the registry of unscrupulous procurement participants. 2. Functional requirements Parsing supplier data from open contracts: Supplier name BIN/TIN Bank details Phone number Email address Full name of the manager Checking the supplier's BIN/TIN in the registry of unscrupulous procurement participants. Data export in CSV format or other specified format. Logging of the parsing process and verification results. 3. Non-functional requirements Performance: Parsing execution time should not exceed reasonable limits (to be determined at the design stage). Reliability: Handling errors and exceptional situations. Ability to resume work from the stopping point. Security: Parsing must be done within legal and ethical norms (without DDoS and server overload). 4. Technological stack Programming language: Python 3.x Libraries: requests, beautifulsoup4, pandas, and others as needed. Version control system: Git 5. Development stages Architecture design and selection of suitable libraries. Development of data parsing module from the portal. Development of the module for checking in the registry of unscrupulous participants. Integration of modules and implementation of data export. Testing and debugging. Deployment. 6. Team and resources One developer with experience in Python and web scraping is required. Access to the portal www.goszakup.gov.kz and the registry of unscrupulous procurement participants. The development period will be determined after assessing the volume of work at the design stage. Ready to discuss any additional details for the successful implementation of the project.

Proposals 9

Yevhen Syzintsev

10 0

Projects 11
Rating -
Rating 1 030

Budget: 550 USD Deadline: 14 days

Good day, I have extensive experience in web scraping, completed projects of various complexity of data collection, ETL automation, proficient in handling all functionalities (working with requests, cookies, proxies, user-agents, authorization, bypassing CAPTCHA including Cloudflare), with subsequent data processing and transformation.

Oleksandr Bodyul

8 2

Projects 8
Rating -
Rating 1 080

Budget: 250 USD Deadline: 5 days

Hello, I have reviewed your project brief and I am ready to start. In your code, I would replace `request` with `aiohttp`, `beautifulsoup4` can be replaced with `lxml`, `html5lib`, or `scrapy`. I use the version control system GitHub for each project, with over 4 years of experience working with Python. I have recently completed a similar project involving pauses and parsing of large volumes of data. I implement parsing within legal and ethical boundaries.
About me: I design a solid architecture for easy deployment using Docker and can package everything needed there. I have a high level of knowledge in `bin`, `bash`, `zsh`, and am a proficient Linux user.
I confidently implement both web interfaces and GUI interfaces. You can view my portfolio at https://github.com/sashabodiul and I can personally demonstrate real completed projects. (PyQt5, kivy, tkinter, Flask, FastAPI, Django)
Regarding technology stack, I have a strong background.
In parsing, I use proxies, user-agents, and specify various headers when necessary. I can work with inspectors, JavaScript, DOM, and extract data even from AJAX and GraphQL.
Main parsing libraries: `aiohttp`, `lxml`, `bs4`, `scrapy`
Web automation: Selenium
Creating APIs in Golang and Python, GinGo, Flask, FastAPI, RestfulAPI, Swagger
I have experience working with Google Sheets and Google Cloud Console.
Ability to work with various SQL and NoSQL databases.
Solving captchas with anticaptcha.
Knowledge of Semaphores, threading, concurrent.future. Ability to work with numpy and more_itertools, specifically `chunked` for dividing parallel scraping into uniform threads, which greatly speeds up data collection.
Basic knowledge for building web interfaces.
Additionally, I can work with APIs, extract hidden APIs, and work with XML.
PyQuery for working with HTML documents.
Pyppeteer for automations.

Talia L.

0 0

Projects -
Rating -
Rating 257

Budget: 500 USD Deadline: 7 days

Good time,

Interesting project.
Possibly missing a database or a description of how data handling works (where and how to store them while in use and whether it is necessary).

In principle, the logic is clear, there is a rough solution from commercial experience.
And it will be necessary to clarify the legislation regarding the resource, 1 - perhaps they have an option to receive the same information in xls, csv format or just a download link.
2 - how legal is it to check a site without permission))

My advantages - I have diverse commercial experience in development for over 6 years) and a broader project view due to my specialized higher education.

My stack
Python/Django/FastAPI
PostgreSQL + SqlAlchemy + Alembic

All the best,
Sincerely,
Tali.