1

My question is related to this post: Enter query in search bar and scrape results

I am able to execute the answer given to this former question, but am not able to scrape data from the website that Chrome navigates to by the looping over book. I only found answers that would show how to scrape data from d in my code, but not from the search result after having used send_keys.

I tried to access the element but can't do so and I would like to scrape the data from the resulting website after searching for book and then going to the next round of the loop.

I tried:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
d = webdriver.Chrome('mypath/chromedriver.exe')
books = ['9780062457738']
for book in books:
  d.get('https://www.bol.com/nl/')
  e = d.find_element_by_id('searchfor')
  f = print(e.send_keys(book, Keys.ENTER))

I also tried without the print() function but it returns no real element if I type f? I get:

Type:        NoneType
String form: None
Docstring:   <no docstring>

Any help on how to parse the data of for example the author of the book, title or other information after having submitted a search query is very welcome!

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
tiny
  • 129
  • 6
  • Did you wait for the results? It might take a while to load the query, reading those elements before they are filled produces empty results. – nagyl Jan 05 '21 at 14:50
  • I don't think so @nagyl. I tried with `time.sleep(2)` but still don't get a proper result. – tiny Jan 05 '21 at 15:07

1 Answers1

1

To extract and book title i.e. The Subtle Art of Not Giving a F*ck you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR:

    driver.get("https://www.bol.com/nl/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.js-confirm-button>span"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#searchfor"))).send_keys("9780062457738")
driver.find_element_by_css_selector("button[type='submit']").click()
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "a.product-title"))).get_attribute("innerHTML"))

  • Using XPATH:

    driver.get("https://www.bol.com/nl/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Accepteren']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='searchfor']"))).send_keys("9780062457738")
driver.find_element_by_xpath("//button[@type='submit']").click()
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//a[contains(@class, 'product-title')]"))).get_attribute("innerHTML"))

  • Console Output:

    The Subtle Art of Not Giving a F*ck

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352