Selenium sometimes throws TimeoutException with presence_of_element_located()

Question

I'm using Selenium in Python 3.6 and it works fine, but not always. I have this code:

 try:

    table = wait.until(
        EC.presence_of_element_located((
            By.XPATH,
            "/html/body/div/div[2]/div[1]/table/tbody/"
            "tr[2]/td/table/tbody/tr/td/div/table[2]"
        ))
    )
except TimeoutException:
    driver.quit()
    return {
        "statusCode": 500,
        "body": json.dumps({"error": "WebPage dont load"}),
        "headers": {
            "Content-Type": "application/json"
        }
    }

And this works, but sometimes there is a timeout exception, and I need this working 100% of the time. I already tried using time.sleep(5) , presence_of_all_elements_located , visibility_of_element_located, function to wait the webpage to load, but the problem is not fixed.

Also I have:

driver.implicitly_wait(20)

and

wait = WebDriverWait(driver, 30)

to make waits implicit and explicit.

Don't mix explicit and implicit waits. That could be your problem right there. Also, you're path-based XPATH is brittle... try to find a better way to get the element you are looking for. (The path may change during a DOM update...) — pcalkins, Dec 14 '20 at 21:24
Thanks you for your comment, I will try using only explicit waits. — fukurowl, Dec 14 '20 at 21:42

score 2 · Accepted Answer · answered Dec 14 '20 at 22:08

2

Ensure that the issue is not a bug, maybe the element is not always present on the page. If it is a bug, discuss it with developers.

If it is not a bug, use relative XPath instead of absolute, as absolute XPaths are very fragile. Or even better, use a unique ID or class. You can find more info about XPath here.

answered Dec 14 '20 at 22:08

K. B.

3,342
3
19
32

I think the problem is in the page actually, I'm already using only explicit waits and change the xpath for relative paths and the problem stil there. – fukurowl Dec 14 '20 at 22:14

undetected Selenium · Answer 2 · 2020-12-14T22:22:03.987

presence_of_element_located()

presence_of_element_located() is the expectation for checking that an element is present on the DOM of a page. This does not necessarily mean that the element is visible. Hence it may not suffice to your needs.

visibility_of_element_located()

visibility_of_element_located() is the expectation for checking that an element is present on the DOM of a page and visible. Visibility means that the element is not only displayed but also has a height and width that is greater than 0.

This usecase

So ideally you need to replace presence_of_element_located() with visibility_of_element_located() and your effective code block will be:

try:

   table = wait.until(
       EC.visibility_of_element_located((
           By.XPATH,
           "/html/body/div/div[2]/div[1]/table/tbody/"
           "tr[2]/td/table/tbody/tr/td/div/table[2]"
       ))
   )

Additional Consideration

Instead of using an absolute xpath you would be better with a relative xpath.

References

You can find a couple of relevant detailed discussions in: