Detect and fix the error. Program in python + selenium
Hello.
If you read what is below and are confident that you can fix this error, then I will send you a 2-minute video, without sound, in which there is code for detailed familiarization.
This task is for someone who has experience with Selenium, possibly even without it.
Because requests are made to Google through Selenium. Selenium opens Google search pages. 7 pages. On them, I even counted 62 results manually.
A function that processes and retrieves the necessary data. And this is simply the URL of the site.
And nothing more. It does not retrieve all the data. URL 62. But for some reason, it takes 52.
The words of the programmer who tried to fix this error:
In your case, there is a file (gsearch... something, I don't remember), in which there is a function that retrieves links from the search engine. I added logging output there, it works fine, it displays all available links - in our case - all 62 pieces. In utils.py, there is a function get_google_search_results_alt(), in which we do not understand why we get 56-57 pieces instead of those 62.
The problem, as I saw, was after moving to the second page (using pagination to navigate), on which we lose 4 links, and therefore in the final version we get 56-57 pieces, although Selenium retrieves them all.
Perhaps there is some limitation in the function get_google_search_results_alt, or maybe it checks something and in the final version removes these links, possibly due to the link format, possibly due to other issues - I have not understood yet.
The developer can add logs if necessary, it is not that difficult to do.
If you are not sure that you can solve this problem. Then I allow you to first connect to my code and take a look. If you are confident that you can solve this problem. Then just let me know. Then I will accept you into the project and allocate funds for the project. I can also accept you into the project right away, it is not a principle for me.
-
3631 17 0 Hello✋,
I have experience working with Selenium and web scraping.
Having reviewed your description, I will be able to identify and resolve the issue with incomplete retrieval of URLs when navigating through Google search pages.
I am ready to review your code or start working on the solution right away.
Thank you for the opportunity, I will be happy to help!
… Best regards,
Andriy!)
-
759 3 1 Good evening. I am ready to take on your order, the problem is clear. I have extensive experience with Selenium and parsing. I would be happy to collaborate!
-
289 Hello
I can try to take a look and try to find a solution to the problem.
-
покажите код.
-
Current freelance projects in the category Data Parsing
I am looking for a programmer for OpenCart.Good day 1) It is necessary to implement on the website dneprkomfort.dp.ua A module for Ukrainian banks has been purchased, and we have already integrated Mono Bank Here is an example from our competitor It is necessary to implement installment payments, purchase in parts… Web Programming, Data Parsing ∙ 14 hours 35 minutes back ∙ 39 proposals |
A specialist in Telegram promotion is required.
28 USD
Tasks: invite real users from the username database to new chats and send messages to the target database. Only quality traffic and work with a live audience are of interest — performers using bots, fake engagement, or low-quality methods are requested NOT TO DISTURB. Work… Data Parsing, Social Media Marketing (SMM) ∙ 4 days 16 hours back ∙ 9 proposals |
Consultation on parsing Instagram account subscribersHello. It is necessary to conduct a preliminary assessment of the feasibility of the following task. I have a list of Instagram accounts. The goal is to obtain contact information (primarily email addresses) of users who follow these accounts. Previously, I encountered companies… Data Parsing ∙ 8 days 11 hours back ∙ 13 proposals |
A specialist is needed to find contacts of decision-makers in Ukraine.It is necessary to gather a database (or ready database) of contacts of decision-makers (DMs) in companies in Ukraine. Information Gathering, Data Parsing ∙ 8 days 16 hours back ∙ 21 proposals |
Need to scrape data from LinkedInWe need to scrape data from LinkedIn based on our list. For each entry, we need to find and collect available data if it exists on the LinkedIn profile, including the profile picture on the LinkedIn social network, email address, links to social media, company website, and… Data Parsing ∙ 8 days 22 hours back ∙ 28 proposals |