How to select dropdown menu items on webpage with Puppeteer

Question

I am trying to learn JavaScript and trying to select the sort by drop down menu and click on one of its four items on this website: https://www.centris.ca/en/properties~for-sale?view=Thumbnail

But I keep getting no node found for selector.

This is my code:

const puppeteer = require('puppeteer')
function run () {
    return new Promise(async (resolve, reject) => {
        try {
            const browser = await puppeteer.launch({headless: false})
            const page = await browser.newPage()
            await page.goto("https://www.centris.ca/")

           
            await page.$eval('i.far.fa-search', el => el.click())
            
            await page.select("select#selectSortById", "3")
 
            //browser.close()
            return resolve(page)
        } catch (e) {
            return reject(e)
        }
    })
}
run().then(console.log).catch(console.error)

ggorlen · Accepted Answer · 2022-11-27T23:09:55.120

The code you're using is for manipulating HTML <select> and <option> elements. But the structure you're looking at are just some <a>s in a <div> that are styled and have JS that makes them behave like a dropdown, but aren't really dropdowns as far as Puppeteer is concerned, likely explaining why your code doesn't work.

I'd just select the <a> (acting like an "option") you want, then click it (using the native click to avoid visibility weirdness). I tossed in a waitForFunction to detect when the filter has actually been applied, but that might not be what you want to do next. Even so, it helps verify that this works before dumping the screenshot.

I also set a user agent so that headless mode works, if desired.

const puppeteer = require("puppeteer"); // ^19.1.0

let browser;
(async () => {
  browser = await puppeteer.launch({headless: true});
  const [page] = await browser.pages();
  await page.setUserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36");
  await page.goto("https://www.centris.ca/en/properties~for-sale");
  const recent = await page.waitForSelector('#selectSortById [data-option-value="3"]');
  await recent.evaluate(el => el.click());

  // assume the whole first page are new listings and assume a few classes exist
  await page.waitForFunction(`
    document.querySelectorAll(".banner.new-property").length === 
      document.querySelectorAll(".property-thumbnail-item").length
  `);
  await page.screenshot({path: "result.png"});
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close())
;

Unrelated to the scraping task, but worth a peek to improve your JS code: What is the explicit promise construction antipattern and how do I avoid it?

How to select dropdown menu items on webpage with Puppeteer

1 Answers1