1

Dears, i am trying to get the links of google search results , i was using this profiles = driver.find_elements_by_xpath('//*[@class="r"]/a[1]') this was working correctly , i am using it now , but not giving me results

            import csv
            from parsel import Selector
            from selenium import webdriver
            from selenium.webdriver.common.keys import Keys
            import time
            import itertools
            import random
            listlong = []
            
            driver = webdriver.Chrome('C:/chromedriver.exe')
            driver.get('https://www.google.com/search?q=&num=200&start=0&sourceid=chrome')
            time.sleep(random.randint(10,11))
            search_input = driver.find_element_by_name('q')
            search_input.send_keys('site:linkedin.com/in/ AND USA')
            time.sleep(3)
            search_input.send_keys(Keys.RETURN)
            time.sleep(3)
            # grab all linkedin profiles from first page at Google
            profiles = driver.find_elements_by_xpath('//*[@class="r"]/a[1]')
            time.sleep(8)
            profiles = [profile.get_attribute('href') for profile in profiles]
            listprof = profiles.copy()
            listlong.extend(listprof)
            print(profiles)
            print("f1")
            print(listprof)
            print("f2")
            print(listlong)
            print("f3")
            driver.quit()

what is the xpath can be used instead of this driver.find_elements_by_xpath('//*[@class="r"]/a[1]')

abbasgol
  • 21
  • 3
  • Does this answer your question? [How to extract a Google link's href from search results with Selenium?](https://stackoverflow.com/questions/35241230/how-to-extract-a-google-links-href-from-search-results-with-selenium) – MBaas Jun 05 '21 at 12:43
  • Can you add a screenshot of what exactly you want to get? – patrickgerard Jun 05 '21 at 12:59

1 Answers1

0

based on MBaas answer i tried this. I added it so that will first find the search result, then find based on css selector ("div > div > div.yuRUbf > a").

Hope this helps, enter image description here

# grab all linkedin profiles from first page at Google
search_result = driver.find_elements_by_css_selector('div.g')
profiles = driver.find_elements_by_css_selector("div > div > div.yuRUbf > a")
time.sleep(8)
profiles = [profile.get_attribute('href') for profile in profiles]
Kitten
  • 56
  • 1
  • 3
  • Thank you , that works perfectly , if i would use the xpath , do you have idea? – abbasgol Jun 05 '21 at 13:26
  • As far that i can see, it would be hard use xpath as each on the "href" has a different xpath. Possible to mark my answer as accepted? thanks – Kitten Jun 05 '21 at 14:04
  • 1
    hey @abbasgol , was doing some research on Xpath on my own project, figured it out. Can give this try : `profiles = driver.find_elements_by_xpath("//a[contains(@href,'https://www.linkedin.com/in')]")` Found a good reference here : `https://www.guru99.com/xpath-selenium.html#4` – Kitten Jun 08 '21 at 08:02