Add to the parser in Python: Proxy, sound, conditions in config.yaml.
Hello. This task is for those who have connected a list of proxy ipv4 with a password to a parser in Python. Also, for those who know how to add logic so that the parser, in case of an error, takes another proxy from the list I provided. I know where the code is located, and I will show you.
To evaluate the code and the cost of work. And you will write a conditional figure. I understand you. Therefore, a convenient option. Connect to my PC via Anydesk or Teamviewer. Look at the code. Understand that you can complete the task. Update your bid under the project, and I will accept you into the project. I will reserve funds. You will connect again and write the necessary code. I will test it, and if it works, I will accept the project. I will compile it myself, create the build. And only this way! Because! If you do not look at the code and write any bid, what will be the outcome? My and your time wasted and money? I think neither you nor I need that. I think we clarified this.
I have the PyCharm compiler. I made it possible to run the parser, the main one, in the compiler. Convenient for testing the code.
About the code itself: the code is written in Python. Requests to the server are executed via aiohttp, as the project is asynchronous, meaning requests are sent to the server in parallel, not sequentially.
The build is done using the PyInstaller library. I run the .exe program, and the command line opens. The parser itself opens in the browser, locally at address 127: and so on.
1. Connect proxy Ipv4 with a password. Write code to connect the proxy. Implement it so that the proxy is taken from my config.yaml and automatically written in the config.yaml itself. There should be a proxy field. Then there should be a list of my proxies: in this order.
All proxies will be listed in a column. Each proxy has its own ip:port:password.
When the program starts, it should take any proxy from the list. This can be specified in the main.py file. So that at the start of the program, a proxy is taken. We must write code that if there is an error when connecting to the proxy, which can always happen, it should select another one from the list.
Now the most important thing! When collecting results from Google, an error occurs. After a while, a captcha appears. The limits are randomly specified. Changing the ip helps me; I did this manually. There is code written to reconnect after 15-20 seconds. And here it needs to be added. During the pause, take another proxy from the list in the config.yaml file. If there is an error again, take another one again. But not just any, not randomly, but in order. It took 2 proxies, now take 3 proxies. I tested manually; while there was a pause, I changed the ip, and then Google took its captcha, and the parser continued to work. ON THE SCREEN CODE. Pause and delay. Also, a standard sound is specified. The sounds will need to be replaced with mine. About this in point 2.
2. Write code for sound. Now there are standard sounds specified. Which are generated automatically. I want you to replace the code. With code that specifies to take sounds from config.yaml. In other words, for different actions of the program, we specify the path to config.yaml.
And I will specify the address of the sound location in the Config myself. Just show me how to write the address correctly as an example.
In which moments and where to insert these sounds, I will show in the code. But we write our values for each action of the program; the sound will be different. Well, in the config there will be.
A Sound field. And in it will be. A list of 5 sounds. Start of search parsing. End of search parsing. Started parsing websites. Error 429. There will be a sound, we will insert it. Completion of website parsing. Conditions for example, I wrote down.
google_search_start "disk:C/folder/1-sound.mp3"
google_search_end "disk:C/folder/2-sound.mp3"
And so on. 5 sounds.
3. By default. Make 2 active buttons. So that I do not have to turn them on every time. But they are already turned on. This is clear and simple.
Make it so that after the results collection is completed, the parsing button is automatically pressed. Because now I press it every time. There is nothing difficult here either. Just output the activation and deactivation of this action in config.yaml
Let's call the field. Google_Search_And_Parsing: "1" #1 - activates the automatic action. 0 - disables this action.
Well, if I go in to press this button as before, then I should not go into the code every time. But only go into CONFIG. And there turn it on and off.
Applications 1
-
3893 24 0 Hello,
I am ready to take on your project. I have experience in connecting a password-protected IPv4 proxy to a parser in Python and skills in adding logic for automatically selecting another proxy in case of an error. I know where the code is and I am ready to show it to you.
My rate is $16 per hour. To start working, I need to familiarize myself in more detail with the requirements and scope of the project.
Stay in touch,
Maxim
-
357 8 0 1 The work is not difficult, I have been working with parsing for more than two years and I can complete it without problems. There are a few questions I would like to discuss in private.
-
Чому Ви для кожного нового завдання над Вашим парсером шукаєте нового фрілансера щоб виконав а не знайдете одного який би Вам робив би все по цьому парсері ?
-
оо який грандіозний авторитаризм. Самий зайнятий замовник, що готовий купу часу витратити на підключення кожного хто тут підпишеться, дочекається поки той знайде помилку ... і так по черзі, але відповісти на просте, логічне запитання - та ну...
Не знаю, як ви Віталій, а я вже здогадуюся чому роботодавець не звертається до попередніх виконавців 😉
-
Current freelance projects in the category Data Parsing
Parsing products from supermarketsNeed a specialist (parsing + Google Sheets + automation). Goal: Create a system for automatic retrieval and updating of food prices from Silpo and NOVUS supermarkets with data output to Google Sheets. What needs to be implemented: Create a main Google Sheets table with a product… Data Parsing, Information Gathering ∙ 1 minute back |
LinkedIn mailing
120 USD
LinkedIn Mailing It is necessary to launch a LinkedIn mailing to the specified database. 30,000+ contacts We are waiting for proposals Email Marketing, Data Parsing ∙ 1 day 7 hours back ∙ 9 proposals |
Parsing company emails from a specific industry
113 USD
Find contacts (e-mail, phone number) of companies engaged in the specified business mentioned in this Google document https://docs.google.com/document/d/1_d-iLxLSL3gjRvo1GmUpWNm0Pm5YoKGcEvaiDc_2I0g/edit?usp=sharing The result should be in the form of a table with the following… Data Parsing ∙ 2 days 7 hours back ∙ 31 proposals |
Need a parser for the online store https://www.lcsc.com/It is necessary to regularly (once a month, or upon script launch) obtain up-to-date information about the products available in the store. https://www.lcsc.com/ from the catalog of all sections.… Data Parsing ∙ 3 days 13 hours back ∙ 43 proposals |
OpenCart — rental catalog of special equipment
135 USD
OpenCart — Equipment Rental Catalog Need to launch an equipment rental catalog on OpenCart. Theme: excavators cherry pickers forklifts generators cranes scaffolding other construction equipment. It is preferable that you already have a ready-made template or developments… Web Programming, Data Parsing ∙ 4 days 5 hours back ∙ 59 proposals |