Python/Selenium, xpath suddenly stops working

Question

I have a script that extracts names from a crypto list on coinmarketcap. To get the names I use the function:

num = 0
def print_name():
    global num
    num = num + 1
    if num == 100:
         exit()
    sleep(0.1)
    names = driver.find_element(By.XPATH, '/html/body/div[1]/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr['+str(num)+']/td[3]/div/a/div/div/div/p')
    print(names.text)

To go through the list, there is a number I have to change which is +str(num)+ in the xpath so the xpath moves to the next name in the list every time, but at number 18 the script just randomlly stops working and gives an error:

Message: no such element: Unable to locate element:

even though the xpath stays the same, I checked multiple times, made the driver refresh, even scroll down to maybe load the data but the program just doesn't recognize the path.

It is not because the website blocks the automation, I tried beginning with the number 5, so I make less requests before reaching number 18 but it still didn't work.

Here is the Error Message:

Traceback (most recent call last):
  File "C:\Users\fkahd\PycharmProjects\coingecko api\sandbox.py", line 27, in <module>
    print_name()
  File "C:\Users\fkahd\PycharmProjects\coingecko api\sandbox.py", line 23, in print_name
    names = driver.find_element(By.XPATH, '/html/body/div[1]/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr['+str(num)+']/td[3]/div/a/div/div/div/p')
  File "C:\Users\fkahd\PycharmProjects\coingecko api\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 1244, in find_element
    return self.execute(Command.FIND_ELEMENT, {
  File "C:\Users\fkahd\PycharmProjects\coingecko api\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 424, in execute
    self.error_handler.check_response(response)
  File "C:\Users\fkahd\PycharmProjects\coingecko api\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 247, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div[1]/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr[18]/td[3]/div/a/div/div/p"}
  (Session info: chrome=98.0.4758.82)
Stacktrace:
Backtrace:
    Ordinal0 [0x00557AC3+2587331]
    Ordinal0 [0x004EADD1+2141649]
    Ordinal0 [0x003E3BB8+1063864]
    Ordinal0 [0x004101CE+1245646]
    Ordinal0 [0x004103CB+1246155]
    Ordinal0 [0x0043A612+1418770]
    Ordinal0 [0x004286D4+1345236]
    Ordinal0 [0x00438A0A+1411594]
    Ordinal0 [0x004284A6+1344678]
    Ordinal0 [0x004053F6+1201142]
    Ordinal0 [0x004062E6+1204966]
    GetHandleVerifier [0x006FDF22+1680738]
    GetHandleVerifier [0x007B0DBC+2413564]
    GetHandleVerifier [0x005ED151+563089]
    GetHandleVerifier [0x005EBF13+558419]
    Ordinal0 [0x004F081E+2164766]
    Ordinal0 [0x004F5508+2184456]
    Ordinal0 [0x004F5650+2184784]
    Ordinal0 [0x004FF5BC+2225596]
    BaseThreadInitThunk [0x764AFA29+25]
    RtlGetAppContainerNamedObjectPath [0x77BB7A9E+286]
    RtlGetAppContainerNamedObjectPath [0x77BB7A6E+238]

It says: Unable to locate element: {"method":"xpath","selector":"/html/body/div[1]/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr[18]/td[3]/div/a/div/div/div/p"}

But, when I go to inspect element, the path is

/html/body/div/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr[18]/td[3]/div/a/div/div/div/p

Which is exactly the same as the one in error message.

Html of text: <p color="text3" class="sc-1eb5slv-0 gGIpIK coin-item-symbol" font-size="1">SCRT</p>

Full code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep

driver = webdriver.Chrome("C:\Program Files (x86)\chromedriver.exe")
driver.get("https://coinmarketcap.com/view/defi/")
sleep(2)
driver.refresh()

num = 0
num2 = 17

def loop2():
    global num2
    global names2
    num2 = num2 + 1
    if num2 == 100:
        exit()
    sleep(0.1)
    names2 = driver.find_element(By.XPATH, '/html/body/div/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr['+str(num2)+']/td[3]/div/a/div/div/div/p')

def print_name():
    global num
    num = num + 1
    if num == 18:
         loop2()
    sleep(0.1)
    names = driver.find_element(By.XPATH, '/html/body/div[1]/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr['+str(num)+']/td[3]/div/a/div/div/div/p')
    print(names.text)

while True:
    print_name()

Quick update here are an example of xpath 17 and 18, but 18 doesn't work:

/html/body/div/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr[18]/td[3]/div/a/div/div/div/p

/html/body/div/div/div[1]/div[2]/div/div[1]/div[2]/table/tbody/tr[17]/td[3]/div/a/div/div/div/p

Thanks for your help, have a nice day.

score 0 · Accepted Answer · answered Feb 16 '22 at 21:17

To extract the names from the Top DeFi Tokens by Market Capitalization within Coinmarketcap you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following locator strategies:

Using CSS_SELECTOR:

driver.get("https://coinmarketcap.com/view/defi/")
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "p[class$='coin-item-symbol']")))])

Using XPATH:

driver.get("https://coinmarketcap.com/view/defi/")
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//p[contains(@class, 'coin-item-symbol')]")))])

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Python/Selenium, xpath suddenly stops working

1 Answers1