Hello!
This task is for those who have parsed information from Google search. Who has experience with Selenium. And who has dealt with captchas.
Various specialists helped me develop a parser from this exchange. Reviews are in the profile. It has been constantly improved. I am looking for another Python specialist for long-term collaboration. I would gladly give you the code. Because it would be convenient for me too! But this is a personalized program. And I cannot! Because it contains confidential data. And every time I have to delete them before giving someone the code, and then rewrite them back, well, that’s not much fun. No GitHub. Or send the code in an archive. Everyone who worked with me connected through any program, for example, Anydesk, if my editor and compiler PyCharm are not convenient for you, then you connected and copied the necessary fragment of code. You wrote it in your editor, returned back, and replaced the old code. I can compile a new version of the program myself, so that’s not a problem. This approach worked for everyone who worked with me. I can immediately make a backup in the project. I can add funds for the inconvenience. I understand everything! But please understand me! You won’t need to write a lot of code. The foundation is already written!
TASK
You don’t need to write anything from scratch! My bot easily took the search results from Google without Selenium, directly. If a captcha appeared, the bot didn’t even solve it. It simply changed the IP and the captcha disappeared. And the bot continued parsing. A couple of months passed. And Google became smarter. And we added that parsing occurs with the browser window loading in Selenium. At the first request, a captcha comes immediately. I solve it. The bot copies some key cookies from the tab that is open in Selenium. And remembers them. Google allows 5 searches, and then it gives a 403 error instead of a captcha. Apparently, there are just not enough cookies. Or they were not written correctly or not all of them were written for the bot to remember. There is just a lot of different info and what exactly is needed to remember the session, or something else. This is important to know. And it would work even without pauses. Because I note that when I pass the captcha in Selenium myself, and manually open at least 100 tabs and search, Google lets everything through. And does not issue captchas. And I did this without pauses and so on. And the bot has random pauses, IP changes as well. The problem is not in this. There is also one more option to do this but that is if you understand Selenium. I will write to you in detail personally then.
About the parser itself.
The code is written in Python. Requests to the server are executed via aiohttp, because the project is asynchronous, meaning requests are sent to the server in parallel, not in order. The build is done through the PyInstaller library. I run the .exe program, and the command line starts. And the parser itself opens in the browser, locally at address 127: and so on. Thank you for your attention!
