Programming task: Development of an algorithm for website parsing
Objective of the task:
To develop an algorithm for automatic parsing of web pages that will collect and process data from specified websites.
Requirements:
1. Choice of programming language:
The specialist must use one of the following languages: Python, JavaScript, Ruby, or another by agreement.
2. Functionality:
- The algorithm must be able to extract data from the specified website(s).
- It is necessary to configure the ability to extract data from different HTML structures.
- The parser must be able to process both static and dynamic web pages (if applicable).
- Data processing: the collected data must be cleaned and structured (for example, in CSV or JSON format).
3. Parsing parameters:
- Specification of target elements for extraction (for example, titles, prices, descriptions, etc.).
- Ability to limit the depth of parsing (for example, only the main page or all pages of the site).
4. Error handling:
- Implement exception handling mechanisms for cases when the site is unavailable or the page structure has changed.
- Logging: maintain a log of errors and successful parsing operations.
5. Documentation:
All code must be well documented. A README file with installation and usage instructions for the parser must be provided.
6. Deadline:
To be discussed
8. Additional wishes:
The ability to scale the algorithm to work with a large number of websites. Optimization of parser performance.
Evaluation criteria:
- Quality and readability of the code.
- Completeness of requirements fulfillment.
- Efficiency of the algorithm.
- Level of error handling and resilience to failures.
Conclusion:
The specialist must provide examples of completed parsing tasks and recommendations from previous clients.