Selenium Web Scraping

Question

I would like my web scraper to go to a web site and press the log in button, then put random credentials and submit, here is the html code, please help

<div class="container">
        <div class="row header-box">
            <div class="col-md-8">
                <h1>
                    <a href="/" style="text-decoration: none">Quotes to Scrape</a>
                </h1>
            </div>
            <div class="col-md-4">
                <p>
                
                    <a href="/login">Login</a>
                
                </p>
            </div>
        </div>

Here is the code I have by far

# Start driver
driver_path = "chromedriver"
driver = webdriver.Chrome(driver_path)
# Navigate to the website
driver.get('http://quotes.toscrape.com/')
driver.maximize_window()

driver.find_element('//bento/orange[contains(@Class,"small")]').click()

To press login button and put random credentials in

score 0 · Answer 1 · answered Feb 02 '23 at 12:44

Use WebDriverWait() and wait for element to be clickable. User following xpath options

driver.get("http://quotes.toscrape.com/")
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//a[text()='Login']"))).click()
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//input[@id='username']"))).send_keys("testuser")
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//input[@id='password']"))).send_keys("testuser")
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//input[@value='Login']"))).click()

Import below libraries

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By

score 0 · Answer 2 · answered Feb 02 '23 at 12:45

To click on the Login link, fill in Username and Password and click on Login you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following locator strategies:

driver.get("https://quotes.toscrape.com/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[href='/login']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#username"))).send_keys("Orkhan")
driver.find_element(By.CSS_SELECTOR, "input#password").send_keys("Karimov")
driver.find_element(By.CSS_SELECTOR, "input[value='Login']").click()

Selenium Web Scraping

2 Answers2