Page won't automatically load when scrolled down with selenium Python

Question

I am using headless Chrome with the selenium package. When I manually visit the website and scroll it down, it loads more itens and the list "nomes" updates in the while loop shown below. When I do it with selenium and a headed browser it also works. Why the page isn't loading with headless? Maybe it is irrelevant, but I also changed the userAgent from ua.random to ua['Chrome'].

import fake_useragent
import selenium

chrome_options = Options()
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument('--incognito')
chrome_options.add_argument('--headless')
chrome_options.add_argument("--window-size=1920,1080")
userAgent = ua['Chrome']
chrome_options.add_argument(f'user-agent={userAgent}')
driver = webdriver.Chrome(executable_path=chromedriver_path, options=chrome_options)
driver.get('https://www.website.com/')
nomes = driver.find_elements_by_css_selector('my.css')
iteracao = 0
        if nomes:
            while iteracao < 3:
                iteracao += 1
                nomes = driver.find_elements_by_css_selector('my.css')
                driver.execute_script("arguments[0].scrollIntoView();", nomes[-1])
                time.sleep(1)
                wait(driver, 10).until(
                    wait_for_more_than_n_elements((By.CSS_SELECTOR, 'my.css'), len(nomes)))

where, which I got from here,

class wait_for_more_than_n_elements(object):
    def __init__(self, locator, count):
        self.locator = locator
        self.count = count

    def __call__(self, driver):
        try:
            count = len(EC._find_elements(driver, self.locator))
            return count >= self.count
        except selexcept.StaleElementReferenceException:
            return False

Answer 1

I had the same issue a while back! Try adding these to your chrome options, that's what worked for me.

chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("disable-infobars")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-dev-shm-usage")

Worth noting, I guess, that I've heard users say having the --no-sandbox arg defined first is important, but it never seemed to matter for me.

Answer 2

This worked for my website.

loc = nomes[-1].location['y']
driver.execute_script(f"window.scrollTo(0, {loc} + 200);")
driver.execute_script(f"window.scrollTo(0, {loc} + 210);")