Scraping inside IFrame with Puppeteer

Question

I'm trying to capture information within a website but I'm having problems with the iframes.

I need to access the site https://www.saisp.br/online/produtos-publicos/, click on the left menu Paraíba do Sul Telemetric Network and capture the value of the line Rio Paraitinga - Passarela / São Luiz do Paraitinga.

My code below doesn't capture the information from the second iframe. How can I do this?

const puppeteer = require('puppeteer');

(async () => {
  // Launch a new browser instance
  const browser = await puppeteer.launch();

  // Create a new page instance
  const page = await browser.newPage();

  // Navigate to the specified webpage
  await page.goto('https://www.saisp.br/online/produtos-publicos/');

  await page.waitForTimeout(4000)

  const link = await page.$x('/html/body/app-home/div/nav/ul/li[12]/div/div[1]/a');
  await link[0].click();
  await page.waitForTimeout(2000)

  // Espera o iframe carregar e troca o contexto para ele
  // Selecione o iframe usando XPath
  const iframe = (await page.$x('/html/body/app-home/div/main/iframe'))[0];
  const frameContent = await iframe.contentFrame(); 
  await page.waitForTimeout(1000)
  await frameContent.waitForSelector('#tbTelemBody');

  const elemento = (await frameContent.$x('/html/body/div/table/tbody/tr/td[2]'))[0];
  const value = await frameContent.evaluate(el => el.textContent, elemento);

  console.log(value);

  await browser.close();
})();

Answer 1

This is actually a pretty complex page and your attempt is a good one.

As a general point, avoid timeouts. Stick to event-driven code for speed and accuracy. Prefer waitForSelector, waitForXPath and waitForFunction.

Avoid browser-generated paths, and prefer CSS selectors to XPaths when possible.

The website doesn't have much in the way of identifying attributes on its elements, so we should prefer text for selecting elements here. See this thread for text selection techniques in Puppeteer.

Here's one approach:

const puppeteer = require("puppeteer"); // ^19.7.2

const url = "<Your URL>";

let browser;
(async () => {
  browser = await puppeteer.launch();
  const [page] = await browser.pages();
  await page.goto(url, {waitUntil: "domcontentloaded"});
  const el = await page.waitForSelector("text/ba do Sul");
  await el.evaluate(el => el.click());
  const row = await page.waitForFunction(() =>
    [
      ...document
        .querySelector("iframe")
        .contentDocument.querySelectorAll("#tbDadosTelem tr"),
    ].find(e => e.textContent.includes("Luiz do Para"))
  );
  const rowData = await row.$$eval("td", els =>
    els
      .filter(e => !e.querySelector("td") && e.textContent.trim())
      .map(e => e.textContent.trim().replace(/\s+/g, " "))
  );
  console.log(rowData);
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close());

Output:

[
  'Rio Paraitinga - Passarela / São Luiz do Paraitinga',
  '0.6',
  '3.29'
]