Webscraping: ElementNotInteractableException while trying to fill the form on this website

Question

I have been trying to scrape data from this site. I need to fill Get the precise price of your car form ie. the year, make, model etc.. I have written the following code till now:

import requests
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait, Select
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import StaleElementReferenceException
from selenium.webdriver.support import expected_conditions
from bs4 import BeautifulSoup
import re

chrome_options = webdriver.ChromeOptions()
driver = webdriver.Chrome('chromedriver_win32/chromedriver.exe', options=chrome_options)
url = "http://www.indianbluebook.com/"
driver.get(url)
save_city = driver.find_element_by_xpath('//*[@id="cityPopup"]/div[2]/div/div[2]/form/div[2]/div/a[1]').click()  #Bangalore

#fill year
year_dropdown = Select(driver.find_element_by_xpath('//*[@id="car_value"]/div[2]/div[1]/div[1]/div/select'))
driver.implicitly_wait(50)
year_dropdown.select_by_value('2020')

time.sleep(5)

But, its giving this error:

ElementNotInteractableException           Traceback (most recent call last)
<ipython-input-25-a4eb8001e649> in <module>
      8 year_dropdown = Select(driver.find_element_by_xpath('//*[@id="car_value"]/div[2]/div[1]/div[1]/div/select'))
      9 driver.implicitly_wait(50)
---> 10 year_dropdown.select_by_value('2020')
     11 
     12 time.sleep(5)

~\anaconda3\lib\site-packages\selenium\webdriver\support\select.py in select_by_value(self, value)
     80         matched = False
     81         for opt in opts:
---> 82             self._setSelected(opt)
     83             if not self.is_multiple:
     84                 return

~\anaconda3\lib\site-packages\selenium\webdriver\support\select.py in _setSelected(self, option)
    210     def _setSelected(self, option):
    211         if not option.is_selected():
--> 212             option.click()
    213 
    214     def _unsetSelected(self, option):

~\anaconda3\lib\site-packages\selenium\webdriver\remote\webelement.py in click(self)
     78     def click(self):
     79         """Clicks the element."""
---> 80         self._execute(Command.CLICK_ELEMENT)
     81 
     82     def submit(self):

~\anaconda3\lib\site-packages\selenium\webdriver\remote\webelement.py in _execute(self, command, params)
    631             params = {}
    632         params['id'] = self._id
--> 633         return self._parent.execute(command, params)
    634 
    635     def find_element(self, by=By.ID, value=None):

~\anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py in execute(self, driver_command, params)
    319         response = self.command_executor.execute(driver_command, params)
    320         if response:
--> 321             self.error_handler.check_response(response)
    322             response['value'] = self._unwrap_value(
    323                 response.get('value', None))

~\anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py in check_response(self, response)
    240                 alert_text = value['alert'].get('text')
    241             raise exception_class(message, screen, stacktrace, alert_text)
--> 242         raise exception_class(message, screen, stacktrace)
    243 
    244     def _value_or_default(self, obj, key, default):

ElementNotInteractableException: Message: element not interactable: Element is not currently visible and may not be manipulated
  (Session info: chrome=85.0.4183.102)

Note: I have tried many available solutions on internet like using Expected conditions with WebDriverWait. Sometimes I get the error, StaleElementException. I don't know what to do now. Please help. I'm new to this.

Please check this https://www.selenium.dev/exceptions/#stale_element_reference — Karthik, Sep 11 '20 at 06:42
@DilipMeghwal sure. any method would work as long as i get the scrapping going on. — Harsh Vardhan, Sep 11 '20 at 07:27

Dilip Meghwal · Accepted Answer · 2020-09-11T09:40:21.647

You can use the below approach to achieve the same.

#Set link according to data need
driver.get('http://www.indianbluebook.com/')

#Wait webpage to fully load necessary tables
def ajaxwait():
    for i in range(1, 30):
        x = driver.execute_script("return (window.jQuery != null) && jQuery.active")
        time.sleep(1)
        if x == 0:
            print("All Ajax element loaded successfully")
            break


ajaxwait()
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable((By.XPATH, "//*[@id='cityPopup']/div[2]/div/div[2]/form/div[2]/div/a[1]")))

save_city = driver.find_element_by_xpath('//*[@id="cityPopup"]/div[2]/div/div[2]/form/div[2]/div/a[1]').click()  #Bangalore

ajaxwait()
#fill year
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[@class='form-group']//select[@class='form-control' and @name='manufacture_year']/following-sibling::div/a")))
#//div[@class='form-group']//select[@class='form-control' and @name='manufacture_year'] this is the only unique elemnt with reference to this we can find other element.
#click on select year field then a dropdown will be open we will enter the year in the input box. Then select the item from the ul list.
driver.find_element_by_xpath("//div[@class='form-group']//select[@class='form-control' and @name='manufacture_year']/following-sibling::div/a").click()
driver.find_element_by_xpath("//div[@class='form-group']//select[@class='form-control' and @name='manufacture_year']/following-sibling::div//input").send_keys("2017")
driver.find_element_by_xpath("//div[@class='form-group']//select[@class='form-control' and @name='manufacture_year']/following-sibling::div//em").click()

Similarly you can select other drop down by changing the @name='manufacture_year' attribute value.

Note: Updated the code with Ajax wait.

Hi. Its producing the same error. StaleElementReferenceException: Message: stale element reference: element is not attached to the page document (Session info: chrome=85.0.4183.102) — Harsh Vardhan, Sep 11 '20 at 07:41
Sometimes its working while sometimes its throwing that exception. why and how could we avoid that? — Harsh Vardhan, Sep 11 '20 at 08:21
These are the Ajax elements and require proper wait applied. You can refer this [Link](https://www.blazemeter.com/blog/five-ways-to-test-ajax-calls-with-selenium-webdriver). Or we have to use JavaScriptExecutor. — Dilip Meghwal, Sep 11 '20 at 08:32
I have tried increasing the wait. but its quite frequently throwing the exception. — Harsh Vardhan, Sep 11 '20 at 08:41
Added the code to wait till the jQuery loaded properly. I have applied max 30 second wait, update accordingly. — Dilip Meghwal, Sep 11 '20 at 09:41

score 0 · Answer 2 · answered Sep 11 '20 at 09:44

To click on BANGALORE and then select 2020 from the dropdown, you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategies:

Using XPATH:

driver.get('http://www.indianbluebook.com/')
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.LINK_TEXT, "BANGALORE"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Select Year']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//ul[@class='chosen-results']//li[@class='active-result' and text()='2020']"))).click()

Note: You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Browser Snapshot:

Webscraping: ElementNotInteractableException while trying to fill the form on this website

2 Answers2