Scraping Notion

Question

After the bot clicks on the first book,it should wait until all quotes are visible; however, it doesn't. It only gets 4 quotes out of 25. It can get the 25 quotes if I time.sleep(10) but this is not efficient. How can I solve this?

class Scraping:
    def __init__(self):
        pass

    def openWebDriver(self):
        chromeOptions = Options()
        self.driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chromeOptions) 

    
    def fetchNotion(self):
        self.driver.get('https://sumptuous-salesman-ca6.notion.site/bf68366a212e45e1ae9bee853867c225?v=85ee9cedeb5a44c994e49033053f593b')

    def getBooks(self):
        books = WebDriverWait(self.driver, 10).until(
            EC.presence_of_all_elements_located((By.XPATH, '//div[@class="notion-selectable notion-page-block notion-collection-item"]')))
        books[0].click()
    def getQuotes(self):
        quotesElements = WebDriverWait(self.driver, 10).until(
            EC.presence_of_all_elements_located((By.XPATH, '//div[@placeholder="Empty quote"]')))
        quotes = []
        for quoteElement in quotesElements:
            quotes.append(quoteElement.text)
        print(len(quotesElements))
        return quotes

scraping = Scraping()
scraping.openWebDriver()
scraping.fetchNotion()
scraping.getBooks()
scraping.getQuotes()

score 1 · Answer 1 · answered Apr 10 '22 at 20:15

You were close enough. Instead of presence_of_all_elements_located() you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following ocator strategies:

Using CSS_SELECTOR:

def getQuotes(self):
    print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div[placeholder='Empty quote']")))])

Using CSS_SELECTOR in a single line:

def getQuotes(self):
    print([my_elem.get_attribute("innerText") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@placeholder='Empty quote']")))])

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

What is the difference between presence and visibility of an element? I don't quite understand. Also, your code has the same problem: it doesn't detect all quotes because selenium doesn't wait until all of them load. — asfa afs, Apr 11 '22 at 14:50
_What is the difference between presence and visibility of an element_: Did you get some time to check the embedded links? — undetected Selenium, Apr 11 '22 at 14:52

Scraping Notion

1 Answers1