Python, WebDriver, how to get number of elements without triggering implicit wait if there are none?

Question

What's the best way to get the number of elements on a page without triggering an implicit wait if there are no such elements on the page? At the moment, I'm using this:

    def get_number_of_elements(self, by, locator):
        self.driver.implicitly_wait(0)
        elements = self.driver.find_elements(by, locator)
        self.driver.implicitly_wait(self.config.IMPLICIT_TIMEOUT)
        return count(elements)

This works, but is there a better way of doing this?

UPDATE:

The issue was cause because of the way I was instantiating webdriver.Chrome(options=options. In short, do not set driver.implicitly_wait():

import time

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

if __name__ == '__main__':
    options = Options()
    options.add_argument('--disable-dev-shm-usage')
    options.add_argument('--crash-dumps-dir=tmp')
    options.add_argument('--remote-allow-origins=*')
    options.add_argument("--no-sandbox")
    options.add_argument("--disable-gpu")
    options.page_load_strategy = "eager"
    options.add_experimental_option("excludeSwitches", ["enable-automation"])

    driver = webdriver.Chrome(options=options)

    driver.set_page_load_timeout(30)
    # driver.implicitly_wait(30) # <-- This triggers an implicit wait of 30 seconds for find_elements if no elements exist.
    driver.set_script_timeout(30)

    driver.get("https://www.google.com")
    driver.find_element(By.XPATH, "//div[text()='Accept all']").click()
    driver.find_element(By.NAME, "q").send_keys("Stack Overflow")
    driver.find_element(By.XPATH, "//input[@type='submit'][@value='Google Search']").submit()

    elements = driver.find_elements(By.XPATH, "//h1[text()='No such thing...']")
    print(len(elements))
    elements = driver.find_elements(By.XPATH, "//a")
    print(len(elements))

    time.sleep(5)
    driver.close()

Yaroslavm · Accepted Answer · 2023-08-12T07:58:07.377

1

The answer is "not to call redundant" implicit waits. If you are expecting that no elements can be present on the page, you just call

def get_number_of_elements(self, by, locator):
   return len(self.driver.find_elements(by, locator))

And that's all.

If you expect collection not to be empty, you should wait for first element and then get collection length.

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

...

def get_elements_length(self, by, locator, should_not_be_empty=None):
    if should_not_be_empty is not None:
        self.wait.until(EC.visibility_of_any_elements_located((by, locator)))
    return len(self.driver.find_elements(by, locator))

Try to avoid implicit waits in any case. (And in this case they are doing nothing, just lines of code that shouldn't be present)

If you want to speed up find_elements, you can use native js script to get collection length (but if you need it, probably, you do smth wrong in your test scenario)

def get_number_of_elements(self, locator):
   return self.driver.execute_script("return document.querySelectorAll('{}').length".format(locator))

I tested it's time execution and in general it's 3 times faster.

edited Aug 12 '23 at 07:58

answered Aug 11 '23 at 19:20

Yaroslavm

1,762
2
7
15

I've tried your suggestion for `get_number_of_elements`, but it always results in a wait being triggered if there are zero elements. – Scott Deagan Aug 12 '23 at 03:41
After I instantiate the webdriver, I'm calling `context.driver.set_script_timeout(60)`. Should this be there to avoid implicit waits? Or should I set the value to '0'? – Scott Deagan Aug 12 '23 at 03:43
@ScottDeagan `find_elements` is executed immediately and doesn't wait for anything. For sure, there is some time for request / response between Selenium and driver. It just executes js script by general logic. – Yaroslavm Aug 12 '23 at 07:47
1

@ScottDeagan If you want to speed up this method, just use simple js execution `self.driver.execute_script("return document.querySelectorAll('your_locator').length")` . For example, for collection with 25 elements find_elements is executed 0.015s, querySelectorAll is executed in 0.005s. – Yaroslavm Aug 12 '23 at 08:04
I might be doing something wrong. For me, `find_elements` is triggering an implicit wait if there are no elements at the time it is called. I'll write a very simple test harness to test this more thoroughly. Perhaps it's the version of webdriver/selenium or something... – Scott Deagan Aug 13 '23 at 05:00
1

Thanks @Yaroslavm, it was something I was doing wrong. I should have provided more information about the options I was using to instantiat `webdriver.Chrome()`. When I remove `driver.implicitly_wait(30)`, it works as expected. I'll update my question with more information, and will accept your answer. – Scott Deagan Aug 13 '23 at 21:56

score 0 · Answer 2 · answered Aug 11 '23 at 20:41

Implicitly Wait isn't an effective way to wait.

To to get the number of the matched elements ideally you need to induce WebDriverWait for visibility_of_all_elements_located() and you can use the following solution:

def get_number_of_elements(self, by, locator):
    elements = WebDriverWait(self.driver, 20).until(EC.visibility_of_all_elements_located((by, locator")))
    return count(elements)

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

PS: You need to remove implicitly_wait completely as:

Warning:Do not mix implicit and explicit waits. Doing so can cause unpredictable wait times. For example, setting an implicit wait of 10 seconds and an explicit wait of 15 seconds could cause a timeout to occur after 20 seconds.

Python, WebDriver, how to get number of elements without triggering implicit wait if there are none?

2 Answers2