Using Python Selenium and ChromeDriver in a Shiny app

Question

I have a shiny application that looks like this:

When the "Scrape" button is clicked, a Python function runs. This function uses selenium to navigate to the site of interest and proceeds to scrape its contents. To initialise the driver, I need to pass the path to the chrome driver to webdriver.Chrome(executable_path=path). Here is what this part of my Python function looks like:

from selenium import webdriver
import urllib3
import re
import time
import pandas as pd


def scrape(chromedriver="C:/Users/Robpr/OneDrive/Documents/chromedriver.exe"):
        
    # Create driver object. Opens browser
    driver = webdriver.Chrome(executable_path=chromedriver)

    # Rest of the function ...

I call this in my shiny server function like this:

server <- function(input, output, session) {
  
  # Scrape and store returned data frame from .py module in df()
  df = reactive({
    
    if (input$scrape) {
      dfii = sii$scrape(chromedriver="chromedriver.exe")
      dfii$`Filing Date` = as.Date(x=dfii$`Filing Date`, format="%B %d, %Y")
      write_sheet(dfii, ss=sheetId, sheet='filings')
      dfii
    } else if (input$load) {
      read_sheet(ss=sheetId, sheet='filings')
    }
  })

    # Rest of the server function ...

This works beautifully locally. When I publish my app and try to click "Scrape", I get an error: "Error: An error has occurred. Check your logs or contact the app author for clarification.". So, I check my logs:

2021-07-02T18:51:07.355310+00:00 shinyapps[4346040]: Detailed traceback:
2021-07-02T18:51:07.355307+00:00 shinyapps[4346040]: Warning: Error in py_call_impl: WebDriverException: Message: 'chromedriver.exe' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
2021-07-02T18:51:07.355310+00:00 shinyapps[4346040]: 
2021-07-02T18:51:07.355320+00:00 shinyapps[4346040]:   File "/home/shiny/.virtualenvs/statcanEnv/lib/python3.8/site-packages/selenium/webdriver/chrome/webdriver.py", line 73, in __init__
2021-07-02T18:51:07.355311+00:00 shinyapps[4346040]:     driver = webdriver.Chrome(executable_path=chromedriver)
2021-07-02T18:51:07.355311+00:00 shinyapps[4346040]:   File "/srv/connect/apps/StatCanWebScrapingApp/scrapeinsolvencyinsider.py", line 11, in scrape
2021-07-02T18:51:07.355321+00:00 shinyapps[4346040]:     self.service.start()
2021-07-02T18:51:07.355321+00:00 shinyapps[4346040]:     raise WebDriverException(
2021-07-02T18:51:07.355321+00:00 shinyapps[4346040]:   File "/home/shiny/.virtualenvs/statcanEnv/lib/python3.8/site-packages/selenium/webdriver/common/service.py", line 81, in start
2021-07-02T18:51:07.355322+00:00 shinyapps[4346040]: 
2021-07-02T18:51:07.362398+00:00 shinyapps[4346040]:   135: <Anonymous>

I've published the chromedriver.exe file along with my app and module:

I don't see how this isn't working. How can I get this to work on the server?

UPDATE:

I have tried the following solutions:

1. Installing chromedriver-py into my virtualenv and importing binary_path from chromedriver_py and then passing binary_path as the executable_path like this:

from selenium import webdriver
from chromedriver_py import binary_path

driver = webdriver.Chrome(executable_path=binary_path)

Result: Warning: Error in py_call_impl: WebDriverException: Message: unknown error: cannot find Chrome binary

2. Publishing chromedriver.exe along with the app as shown above and passing "/srv/connect/apps/StatCanWebScrapingApp/chromedriver.exe" as the executable path.

Result: Warning: Error in py_call_impl: WebDriverException: Message: 'chromedriver.exe' executable may have wrong permissions. Please see https://sites.google.com/a/chromium.org/chromedriver/home

3. Downloading chromedriver to server working directory and using that path:

chromeDriverUrl <- "https://chromedriver.storage.googleapis.com/index.html?path=91.0.4472.101/"
destFile <- paste(getwd(), "chromedriver.exe", sep="/")
download.file(url=chromeDriverUrl, destfile=destFile)

server <- function(input, output, session) {
  
  # Scrape and store returned data frame from .py module in df()
  df = reactive({
    
    if (input$scrape) {
      dfii = sii$scrape(chromedriver=destFile)
      dfii$`Filing Date` = as.Date(x=dfii$`Filing Date`, format="%B %d, %Y")
      write_sheet(dfii, ss=sheetId, sheet='filings')
      dfii
    } else if (input$load) {
      read_sheet(ss=sheetId, sheet='filings')
    }
  })

    # ...

Result: Warning: Error in py_call_impl: WebDriverException: Message: 'chromedriver.exe' executable may have wrong permissions. Please see https://sites.google.com/a/chromium.org/chromedriver/home

4. Installing webdriver-manager on the virtualenv. In the python module, importing the ChromeDriverManager class from webdriver_manager and passing ChromeDriverManager.install() into webdriver.Chrome() as suggested here.

Result:

2021-07-06T14:32:15.715695+00:00 shinyapps[4346040]: ====== WebDriver manager ======
2021-07-06T14:32:15.727647+00:00 shinyapps[4346040]: /bin/sh: 1: google-chrome: not found
2021-07-06T14:32:15.731757+00:00 shinyapps[4346040]: Detailed traceback:
2021-07-06T14:32:15.731756+00:00 shinyapps[4346040]: 
2021-07-06T14:32:15.731758+00:00 shinyapps[4346040]:   File "/srv/connect/apps/StatCanWebScrapingApp/scrapeinsolvencyinsider.py", line 12, in scrape
2021-07-06T14:32:15.731758+00:00 shinyapps[4346040]:     driver = webdriver.Chrome(ChromeDriverManager().install())
2021-07-06T14:32:15.727881+00:00 shinyapps[4346040]: /bin/sh: 1: google-chrome-stable: not found
2021-07-06T14:32:15.731759+00:00 shinyapps[4346040]:   File "/home/shiny/.virtualenvs/statcanEnv/lib/python3.8/site-packages/webdriver_manager/chrome.py", line 25, in __init__
2021-07-06T14:32:15.731755+00:00 shinyapps[4346040]: Warning: Error in py_call_impl: ValueError: Could not get version for Chrome with this command: google-chrome --version || google-chrome-stable --version
2021-07-06T14:32:15.731759+00:00 shinyapps[4346040]:     self.driver = ChromeDriver(name=name,
2021-07-06T14:32:15.731759+00:00 shinyapps[4346040]:   File "/home/shiny/.virtualenvs/statcanEnv/lib/python3.8/site-packages/webdriver_manager/driver.py", line 57, in __init__
2021-07-06T14:32:15.731760+00:00 shinyapps[4346040]:     self.browser_version = chrome_version(chrome_type)
2021-07-06T14:32:15.731760+00:00 shinyapps[4346040]:   File "/home/shiny/.virtualenvs/statcanEnv/lib/python3.8/site-packages/webdriver_manager/utils.py", line 155, in chrome_version
2021-07-06T14:32:15.731761+00:00 shinyapps[4346040]:     raise ValueError(f'Could not get version for Chrome with this command: {cmd}')
2021-07-06T14:32:15.731761+00:00 shinyapps[4346040]: 
2021-07-06T14:32:15.738260+00:00 shinyapps[4346040]:   135: <Anonymous>

Answer 1

The solution is to set a direct path to the driver, see here for details: Error message: "'chromedriver' executable needs to be available in the path"

An alternative solution would be to add the executable to the PATH of the server.