Python Selenium finding element after driver.get() unreliable—sometimes works others doesn't

Question

I am having an issue where after calling get(), the code sometimes works, other times it doesn't. No search results have this issue. Why would it work sometimes, other times it won't. Simple code to reproduce:

def get_url(search):
    options = Options()
    options.add_argument('-headless')
    driver = webdriver.Firefox()
    driver.get('https://genius.com/')
    print('go')
    elem = driver.find_element(By.ID, 'application')
    print(elem)
    driver.quit()

I've been troubleshooting for hours, so it's an excerpt at it's simplest, with some debugging. It could work twice in a row, then not work twice or three times. Rinse and repeat. Any ideas?

EDIT: Also, I have tried WebDriverWait as well. Same thing happens. Sometimes it succeeds, other times it doesn't. When it doesn't succeed, it seems to get stuck at get(). After 1 or 2 mins, it errors out and the script closes. Odd.

Answer 1

As per your code trials:

elem = driver.find_element(By.ID, 'application')

The locator strategy identifies the element:

<div id="application">...</div>

which is more or less the entire webpage

In realtime scenarios there would be seldom any usecase where where you would like to identify the enire application as a WebElement. Generally, WebElement are much smaller elements for e.g. an <input>, a <span>, a <div> etc.

---

This usecase

To print the DOM Tree entire application a better approach would be to print the page_source as follows:

print(driver.page_source)