0

I have a script that creates multiple selenium.webdriver-instances, executes a js-script and reads the resulting logs of them. Most of the time the script runs without problems, but in a few cases the logs suddenly stop after running for a while. I am not sure how to mimic the error.

My headless webdriver is based on this answer and defined as follows:

import threading
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

threadLocal = threading.local()
DRIVER_PATH = '/chromedriver'

def create_driver_headless() -> webdriver.Chrome:
    driver = getattr(threadLocal, 'driver', None)
    if driver is None:
        dc = DesiredCapabilities.CHROME
        dc['goog:loggingPrefs'] = {'browser': 'ALL'}
        chrome_options = Options()
        chrome_options.add_argument('--disable-infobars')
        chrome_options.add_argument('--disable-gpu')
        chrome_options.add_argument('--disable-logging')
        chrome_options.add_argument('--disable-extensions')
        chrome_options.add_argument('--disable-notifications')
        chrome_options.add_argument('--disable-default-apps')
        chrome_options.add_argument('--window-size=1920,1080')
        chrome_options.add_argument('--headless')
        chrome_options.add_argument('--no-sandbox')
        chrome_options.add_argument('--disable-dev-shm-usage')
        chrome_options.add_experimental_option('excludeSwitches', ['enable-automation'])
        chrome_options.add_experimental_option('useAutomationExtension', False)

        driver = webdriver.Chrome(executable_path=CHROME_DRIVER,
                                  desired_capabilities=dc,
                                  chrome_options=chrome_options)

        setattr(threadLocal, 'driver', driver)

    return driver

I read the logs using

    while True:
       logs = driver.get_log('browser')
       for log in logs:
          # do something

My initial guess was that the headless driver crashed when this occured. However, later in my script I have the following code which is satisfied (returned None):

if len(logs) == 0:
    try:
        if 'END' in driver.find_element(By.XPATH, f"(.//*[@class='sr-match-set-sports__status-str srm-is-uppercase']").text:
            return None

    except NoSuchElementException:
        continue

I assume that if the driver crashed this line should return a NoSuchElementException, so I can conclude that it did not crash?

I am also certain that additional logs should have been received by checking simultaneously the url in Google Chrome.

Any idea what's causing this behaviour?

HJA24
  • 410
  • 2
  • 11
  • 33
  • Can you please share the error message that is raised during crash? Or there's no errors at all, but logs are simply stop appearing? – An Se Jan 08 '23 at 20:42

2 Answers2

1

In your if len(logs) == 0: check you should try to write to tmp file the driver.page_source, this way you'll be a bit more certain on what exactly happens during the crash.

Yet, I'd suggest trying to run the multiple selenium instances via remote driver which is hosted as a standalone-chrome docker container(it's designed to do exactly what you're trying to achieve).

Starting remote webdriver is as easy as:

from selenium import webdriver
opts = webdriver.ChromeOptions()
opts.add_argument("--no-sandbox")
opts.add_argument("--disable-dev-shm-usage")
driver = webdriver.Remote(command_executor="http://localhost:4444/wd/hub", desired_capabilities=opts.to_capabilities())
An Se
  • 870
  • 11
  • 24
0

well I'm not sure but this might be because of traffic you are getting due to headless. try driver.implicitly_wait(5)

sr freak
  • 61
  • 7