Scraper for Obtaining Titles from Web Pages

This project is a Python script for web scraping that automatically retrieves webpage headers (tags) from a URL using the requests and BeautifulSoup libraries. The main goal of the project is to collect information from the target website and save the results in a CSV file for further analysis.

The script has been created with important aspects of security and efficiency in mind, including User-Agent header rotation, the use of a proxy server, and retrying connections in case of failed requests.

Key features:
Retrieving headers:

Automatic parsing of the tag from the specified URL.
Checking the success of the request and handling exceptions.
User-Agent header rotation:

Using different User-Agents to simulate requests from various devices and browsers.
Reducing the risk of blocking when sending requests.
Proxy server support:

Adding proxies to bypass geographical restrictions and ensure the anonymity of requests.
Exception handling:

Using a retry mechanism in case of temporary errors (500, 502, 503, 504).
Handling connection errors and other unforeseen situations.
Saving results:

All retrieved headers are saved in a CSV file in a structured format (request number and corresponding header).
Dynamic delay between requests:

Implementing a random delay to simulate user behavior.

Technologies:
Python — the main programming language.
requests — for HTTP requests.
BeautifulSoup — for parsing HTML documents.
csv — for working with CSV format data.
random — for generating random delays and rotating headers.
aiohttp — for working with asynchronous HTTP requests (can be integrated in the future to improve performance).
Practical applications:
Data collection for SEO: Retrieving headers from webpages for metadata analysis.
Website monitoring: Tracking changes in page headers.
Competitor analysis: Collecting information from target resources for marketing research.