How to parse data with looping over search field and appending output in a data set in python?

Question

My question is related to the previous question: How to parse several attributes of website with same class name in python?

I want to include the parsing within a loop over cap and append the resulting parsed text in a vector or data set at the end of the loop and then continue at the top.

My loop now looks like this:

driver = webdriver.Chrome('pathtoChrome/chromedriver.exe')
caps = ['11100']
for cap in caps:
   driver.get("https://www.conad.it/")
   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@href='javascript:void(0)']"))).click() # accept the cookies
   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='location-input']"))).send_keys(caps)
   WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//input[@class = 'btn btn-default btn-lg btn-block']"))).find_element_by_xpath("//input[@class = 'btn btn-default btn-lg btn-block']").click()
   #WebDriverWait(driver, 20).until(EC.element_to_be_clickable(driver.find_element_by_xpath("//input[@class = 'btn btn-default btn-lg btn-block']").click()
   print([item.text for item in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@class,'col-md-8')]//p")))])

The out-commented line is just another try. In both of my attempts, when I include the .click() command line within the loop, the CAP doesn't get answered.

It works however, if I do not loop, i.e.:

driver = webdriver.Chrome('pathtoChrome/chromedriver.exe')
driver.get("https://www.conad.it/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@href='javascript:void(0)']"))).click() # accept the cookies
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='location-input']"))).send_keys('11100')
driver.find_element_by_xpath("//input[@class = 'btn btn-default btn-lg btn-block']").click()
print([item.text for item in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@class,'col-md-8')]//p")))])

I want to write the results in a data set or vector and then append the next round of the loop to it, something like this, which should append text to data found by typing 11100 or11020, but should not print anything when typing 11000, since there is no entry for this cap:

driver = webdriver.Chrome('pathtoChrome/chromedriver.exe')
caps = ['11000', '11100', '11020']
data = []

for cap in caps:
   driver.get("https://www.conad.it/")
   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@href='javascript:void(0)']"))).click() # accept the cookies
   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='location-input']"))).send_keys(caps)
   WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//input[@class = 'btn btn-default btn-lg btn-block']"))).find_element_by_xpath("//input[@class = 'btn btn-default btn-lg btn-block']").click()
   #WebDriverWait(driver, 20).until(EC.element_to_be_clickable(driver.find_element_by_xpath("//input[@class = 'btn btn-default btn-lg btn-block']").click() 
   print(data.append([item.text for item in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@class,'col-md-8')]//p")))]))

Any help is appreciated!

KunduK · Accepted Answer · 2021-01-06T17:22:40.150

1

Use try..except block if item not found then continue the loop.

caps = ['11000','11100', '11020','13022']
data = []
driver.get("https://www.conad.it/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@href='javascript:void(0)']"))).click() # accept the cookies
for cap in caps:
   driver.get("https://www.conad.it/")
   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='location-input']"))).clear()
   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='location-input']"))).send_keys(cap)
   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"input.btn.btn-default.btn-lg.btn-block"))).click()
   try:
      data.append([item.text for item in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@class,'col-md-8')]//p")))])

   except:
       print("no data found")
       continue

print(data)

Output:

[['Frazione Condemine 84, 11010 Sarre', 'Grand Chemin C/c Centreville 3, 11020 Saint-christophe', "Localita' Arensod 27, 11010 Sarre"], ['Grand Chemin C/c Centreville 3, 11020 Saint-christophe', "Localita' Perolle 21, 11024 Chatillon", 'Frazione Condemine 84, 11010 Sarre', "Localita' Arensod 27, 11010 Sarre"], ['Via Durio 26, 13019 Varallo', 'Via Brigate Garibaldi 24/a, 13019 Varallo']]

edited Jan 06 '21 at 17:22

answered Jan 06 '21 at 16:53

KunduK

32,888
5
17
41

Dear @KunduK: It doesn't work entirely when I try with `caps = ['11000','11100', '11020', '13022']` (adding one) and checked that there should be an entry for `'13022'` but the loop doesn't return it in the `data`. It just returns: ```[['Frazione Condemine 84, 11010 Sarre', 'Grand Chemin C/c Centreville 3, 11020 Saint-christophe', "Localita' Arensod 27, 11010 Sarre"], ['Frazione Condemine 84, 11010 Sarre', 'Grand Chemin C/c Centreville 3, 11020 Saint-christophe', "Localita' Arensod 27, 11010 Sarre"]]```. Sorry for the bother.. – tiny Jan 06 '21 at 17:12
@tiny : Can you try the updated answer and know how this goes. – KunduK Jan 06 '21 at 17:23
it doesn't find any data, returns four times `no data` (e.g. for every element of the loop) and then returns an empty data set. – tiny Jan 06 '21 at 17:46
I think the problem is still with the 9th line above: `WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"input.btn.btn-default.btn-lg.btn-block"))).click()`. The caps get written into the search field, but as this line is within the loop, they don't get searched for (the code doesn't really click the "CERCA" button). I don't know why this doesn't work within the loop, without the loop this line works. – tiny Jan 06 '21 at 17:54
As an alternative Can you add some sleep in between send_keys() and button click() I mean between line 8 & 9 time.sleep(3) and check if this works. – KunduK Jan 06 '21 at 17:58
Yes @KunduK this solves the problem! Thank you so much, so is it really the case that the program was too fast for the website/browser to react on time? – tiny Jan 07 '21 at 07:35

How to parse data with looping over search field and appending output in a data set in python?

1 Answers1

Linked