Freelance projects

Freelance projects

Automation of data collection from the real estate page

Bot Development, Python — incorrectly specified categories?

684 USD

Project translated automatically. Log in or register, to view the original

### Detailed technical assignment for automating real estate data collection

**Goal:** Automate the collection of real estate data from the website myhome.ge, update the database in Excel, and send email alerts every hour with new listings that meet the specified criteria.

#### Main tasks:

1. **Data collection from the website myhome.ge:**

- **Scraping technology:** Use `Scrapy` for efficient web scraping.

- **Proxy and captcha:** Implement proxy rotation using `Scrapy-Proxy-Pool` and captcha solving through the `2Captcha` service if necessary.

- **Data extraction:** Collect the following data:

- Listing title

- Price

- Location

- Number of rooms

- Area (square meters)

- Posting date

- Description

- Contact information

2. **Filtering data by specified criteria:**

- **Criteria:**

- Minimum and maximum price

- District

- Type of property (e.g., apartment, house)

- Number of rooms

- Minimum and maximum area

- Additional features (e.g., balcony, floor, heating type, property condition)

- **Real-time updates:** Provide an interface for users to dynamically change filtering criteria.

3. **Managing the database in Excel:**

- **File format:** Use `pandas` to create and manage the Excel file.

- **Data updates:**

- Update the database hourly by adding new listings.

- Remove outdated or duplicate listings.

- **Backup and versioning:** Maintain versioned backups of the Excel file to prevent data loss.

4. **Email alert system:**

- **Frequency:** Send alerts every hour.

- **Email content:** Include new listings that meet the specified criteria.

- **Email sending service:** Use `smtplib` to send emails.

- **Customizable schedule:** Allow users to change the frequency of alerts if needed.

#### Technical requirements:

1. **Programming language:**

- Python is recommended

2. **Libraries and tools:**

- **Web scraping:** `Scrapy`, `BeautifulSoup`

- **HTTP requests:** `requests`

- **Data processing:** `pandas`, `openpyxl`

- **Email sending:** `smtplib`

- **Proxy rotation:** `Scrapy-Proxy-Pool`

- **Captcha solving:** `2Captcha`

3. **Monitoring and support:**

- **Real-time monitoring:** Implement monitoring using `Prometheus` and `Grafana`.

- **Metrics:**

- Response time

- Number of successful/unsuccessful requests

- Errors

- **Alert system:** Set up alerts for critical issues (e.g., high error rates, downtime).

4. **Testing and documentation:**

- **Testing framework:** Use `pytest` for thorough testing.

- **Documentation:** Provide complete documentation, including installation instructions, user guide, and troubleshooting guide.

5. **Security and compliance:**

- **Data privacy:** Ensure compliance with GDPR and other relevant regulations for data processing.

- **Secure access:** Use secure methods for accessing and storing sensitive information (e.g., contact details).

6. **Deadlines and delivery:**

- **Project plan:** Provide a detailed project plan with key milestones and deadlines.

- **Prototype:** Deliver a working prototype within the first two weeks.

- **Final delivery:** Complete the project within the agreed timeframe.

#### Additional conditions:

- **User interface:** Develop a simple web interface or command line for users to change filtering criteria and alert settings.

- **Scalability:** Ensure the ability to handle an increasing number of listings and user interactions without significant performance degradation.

- **Support plan:** Offer a support plan post-deployment to ensure long-term functionality and address any issues that arise.

---

Please confirm your readiness to take on the project, provide information on the cost of completing the assignment, and the timelines for execution.

Proposals 17

Serhii Yanovsky

22 0

Projects 22
Rating -
Rating 848

Budget: 1000 EUR Deadline: 7 days

Ready to complete the project. Contact me for clarification of details.

Vitaliy Fedytnyk

1 0

Projects -
Rating -
Rating 481

Budget: 600 EUR Deadline: 20 days

Hello!
I am interested in this project and ready to take it on. I have over 3 years of experience in backend + RDBMS development, particularly using Python, Scrapy, pandas, and other necessary tools. I know how to effectively set up automated scraping, data filtering, database updates, and notification systems. I am ready to discuss the details and the cost of completing the task.

Feel free to write for a detailed discussion)

Andrew Proskurin

28 1

Projects 29
Rating -
Rating 431

Budget: 650 EUR Deadline: 7 days

Hello, I am ready to do it, feel free to reach out!
The work will be done with high quality!
I will be happy to collaborate!

Oleksandr Yarotsky

3 0

Projects -
Rating -
Rating 798

Budget: 600 EUR Deadline: 20 days

Good day

I am the founder of the IT company Zentriflex. We have a UX/UI designer on our team with extensive experience who creates selling websites, and Full Stack developers who write quality code using the latest technologies and SEO support. Our DevOps specialist will set up a reliable and cost-effective infrastructure.

We work with technologies such as Node.js and Nest.js for Back-end and React.js and Next.js for Front-end. We are ready to implement your idea.

Mykhailo Pavlov

60 0

Budget: 1200 EUR Deadline: 15 days

Hello!

I am ready to implement the automation of data collection about real estate according to your technical specifications.

I have experience in developing similar projects, particularly in web scraping, data processing, and setting up alert systems.

I write in Python. I am ranked 4th on the platform for this language.

I complete projects quickly and efficiently, ensuring compliance with all requirements.

Portfolio: Freelancehunt

Oleksandr D.

0 0

Projects -
Rating -
Rating 127

Budget: 800 EUR Deadline: 10 days

Good day. I can write in Go language. With quality. Feel free to contact me.

Artur Lavrinovich

63 1

Projects 63
Rating -
Rating 8 150

Budget: 600 EUR Deadline: 1 day

Good afternoon!

I am ready to help you with automating the data collection on real estate from the website myhome.ge, updating it in Excel, and sending email notifications with new listings every hour.

I look forward to discussing the details and getting started!

Maksym Sokolskyi

24 0

Projects 24
Rating -
Rating 3 853

Budget: 600 EUR Deadline: 1 day

Hello,

I am ready to take on your project. I have experience in automating data collection from websites, updating databases, and setting up alerts.

My skills include using `Scrapy` for web scraping, creating and managing a database in Excel with `pandas`, sending email notifications via `smtplib`, as well as working with proxies and solving captchas through the `2Captcha` service.

My rate is $16 per hour. To start working, I need to discuss the project details and terms of cooperation further.

Thank you for your attention.

Sincerely,
Maxim

Yevhenii Hudymenko

23 0

Budget: 600 EUR Deadline: 7 days

Good afternoon, I can implement it in Node.js. I have experience working with scrapers and integrating databases in the form of Google Sheets. Please message me privately to discuss the details.