0

I am fairly new to Python and Selenium, but I have been able to navigate through many webpages via Selenium just fine. But a site recently changed their html code and now Selenium is not able to find any element on the page. Specifically I would like to click the Export-button, which is a span inside an a inside many divs.

This is the page: https://www.st.com/en/diodes-and-rectifiers/power-schottky/products.html#

I am using the Selenium Chromedriver on Windows 10 with Python.

I have tried every selector I know (Rel XPATH, Abs XPATH, CSS-Selector, name, ID, link text, ...) but I always get selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element:.

The website does use iframes, but as I see it, the Export-button is not inside an iframe. Also, while trying to test, I was not able to find the iframes with Selenium. Really any element besides the body can not be found anymore. I am making to make sure the element is loaded, but the error message appears even while waiting for the element, because Selenium can't find it.

Here is an example using the relative XPath, but I have not had any success with Css-Selector or anything else, or any other element:

# setup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC

# setup
options = webdriver.ChromeOptions ( ) ;
prefs = { "download.default_directory" : mypath };
options.add_experimental_option ( "prefs", prefs );
options.add_experimental_option ( "excludeSwitches", ["enable-logging"] )

options.add_argument ( '--headless' )
driver = webdriver.Chrome ( executable_path = 'chromedriver.exe', options = options );

driver.get ( https://www.st.com/en/diodes-and-rectifiers/power-schottky/products.html# ) 

# wait
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='st-site']/div[3]/div[1]/main/div[2]/div[5]/div[3]/div/div[3]/div[1]/div[3]/div/a/span")))


# download
driver.execute_script('arguments[0].click()', driver.find_element(By.XPATH, "//*[@id='st-site']/div[3]/div[1]/main/div[2]/div[5]/div[3]/div/div[3]/div[1]/div[3]/div/a/span"))

Which gives me the timeout error:

Traceback (most recent call last):
    line 228, in the_magic
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='st-site']/div[3]/div[1]/main/div[2]/div[5]/div[3]/div/div[3]/div[1]/div[3]/div/a/span")))
    line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

If I make sure to wait for the webpage to be loaded with time.sleep(30) and skipping the WebDriverWait, the error message is:

Traceback (most recent call last):
    line 231, in the_magic
    driver.execute_script('arguments[0].click()', driver.find_element(By.XPATH, "//*[@id='st-site']/div[3]/div[1]/main/div[2]/div[5]/div[3]/div/div[3]/div[1]/div[3]/div/a/span"))
    line 976, in find_element
    return self.execute(Command.FIND_ELEMENT, {
    line 321, in execute
    self.error_handler.check_response(response)
    line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id='st-site']/div[3]/div[1]/main/div[2]/div[5]/div[3]/div/div[3]/div[1]/div[3]/div/a/span"}
  (Session info: headless chrome=110.0.5481.178)

I don't really know what to try anymore. Any help would be appreciated.

alexschrada
  • 1
  • 1
  • Well, there is no Export button at all on the page, or you need to provide steps-to-reproduce how to get one. – Danny Briskin Mar 03 '23 at 15:38
  • 1
    @DannyBriskin There clearly is an Export-button, directly visible when visiting the page, and I obtained the XPath via copying its XPath from the elements console. – alexschrada Mar 03 '23 at 15:44
  • @alexschrada - Hope you are clicking the 'Accept Cookies' button before searching for the desired element(Export button)? – Shawn Mar 03 '23 at 16:11
  • @Shawn This may be it. Maybe the Cookie-alert shows up for Selenium, but not in my browser. However, I am also not able to click (or wait for) the "Accept Cookies" button neither via XPath nor CSS-Selector. – alexschrada Mar 03 '23 at 16:23

2 Answers2

0

Well, the Export button disappears upon zoom/unzoom, but okay, I have found two of them. I guess you need the one that is not hidden. The issue is in hidden spaces/EOLs in the Export span text. Here you go, the XPath

//div[@class='stsel-selector-ctn']/descendant::a/span[normalize-space(text())='Export']

In order to force webdriver completely mock user browser usage, please add following options (use your paths)

# setup
options.add_argument("--disable-web-security")
options.add_argument("--disable-site-isolation-trials")
options.add_argument("--user-data-dir=E:\\!!!!\\prof")
options.add_argument("--start-maximized")
options.add_argument("--disable-features=NetworkService,NetworkServiceInProcess")
options.add_argument("--test-type")
options.add_argument('--no-sandbox')
options.add_argument("--disable-infobars")
options.add_argument("--start-maximized")
options.add_argument("--disable-extensions")
options.add_argument('--window-size=1920,1080')
options.add_argument('--disable-gpu')

options.add_argument('--headless=new')
driver = webdriver.Chrome(options=options)
Danny Briskin
  • 144
  • 7
  • Thanks for your work, but I still get the same NoSuchElementException with this XPath. So far, exactly 0 XPaths I have tried have worked for any of the elements on the page, I am really confused. Where did you find the hidden Export-button? – alexschrada Mar 03 '23 at 16:03
  • as I told you, there is something with your page, so the button is deleted from the DOM. or you need to wait until the page(and the button) is loaded completely, That's may be the reason. Second button can be found using //a/span[normalize-space(text())='Export'] XPath (without leading div) – Danny Briskin Mar 03 '23 at 16:07
  • I definitely waited for the button to be loaded completely. The problem is not exclusive to the Export-button, but really any element on the website. – alexschrada Mar 03 '23 at 16:11
  • oh my, there is a bunch of security headers at the website that is not used by webdriver, let me edit the response with some additional options – Danny Briskin Mar 03 '23 at 17:04
  • Not sure I can follow you here. I get the error message that Google Chrome is not allowed to write or read into mypath, and if I comment out that line, I still get the access denied screen. – alexschrada Mar 03 '23 at 17:39
  • make sure that --user-data-dir= is a writable folder – Danny Briskin Mar 03 '23 at 19:00
0

Check the below code, this is the closest I could get. Below code manages to go past Accept Cookies. And it manages to click on Export button as well. However, I think the application is detecting the automated software as it is not allowing to download.

# options.add_argument('--headless')
driver = webdriver.Chrome(options=options)
driver.maximize_window()
driver.get("https://www.st.com/en/diodes-and-rectifiers/power-schottky/")
driver.execute_script('arguments[0].click();', driver.find_element(By.XPATH, "//button[text()='Accept Cookies']"))
driver.get("https://www.st.com/en/diodes-and-rectifiers/power-schottky/products.html#")
driver.execute_script("arguments[0].click();", driver.find_element(By.XPATH, "(//span[contains(text(),'Export')])[1]"))
time.sleep(15)

Result: enter image description here

Shawn
  • 4,064
  • 2
  • 11
  • 23