puppeteer: How to wait for pages in SPA's?

Question

I am trying to navigate through an SPA with puppeteer, the problem I am facing here is that I am unable to wait for the page to load then proceed with my program.

I fill a form and then click submit, depending on the contents of the form, different pages can be loaded so I can't use page.waitFor(Selector) as there can be many different pages depending on the input.

I tried using waitUntil: load, networkidle2, networkidle0, domcontentloaded but all of them trigger before the elements are loaded.

The page I am trying to automate is Link. (If you want to check for yourself, then choose booking reference and fill out random details and press continue.)

After choosing "booking-reference" in the link I fill in the details with puppeteer and then press the continue button, What I cannot figure out is how to wait for the page to be completely loaded without relying on selectors.

Does this answer your question? [How to listen to history.pushstate with Puppeteer?](https://stackoverflow.com/questions/52900248/how-to-listen-to-history-pushstate-with-puppeteer) — ggorlen, Oct 31 '20 at 23:50

score 5 · Accepted Answer · answered Mar 26 '18 at 13:09

I think you should know what those pages are and use Promise.race with page.waitFor for each page, like this:

const puppeteer = require('puppeteer');

const html = `
<html>
  <body>
    <div id="element"></div>
    <button id="button">load</button>

    <script>
      document.getElementById('button').addEventListener("click", () => {
        document.getElementById('element').innerHTML =
          '<div id="element' + (Math.floor(Math.random() * 3) + 1)  + '"></div>';
      });
    </script>
  </body>
</html>`;

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto(`data:text/html,${html}`);

  await page.click('#button');

  const element = await Promise.race([
    page.waitFor('#element1'),
    page.waitFor('#element2'),
    page.waitFor('#element3')
  ]);

  console.log(await (await element.getProperty('id')).jsonValue());
  await browser.close();
})();

Thanks! This is exactly what I was looking for, also is there any way to know which won the race? I mean which element came up? — Nagarjun Prasad, Mar 26 '18 at 13:15
`Promise.race` return only one element: the fastes one - in your case element which distinguishes what page is loaded. I'm detecting in line `await (await element.getProperty('id')).jsonValue()` what ID of this element is, next it's up to you what you do with this information. — Everettss, Mar 26 '18 at 13:20

ggorlen · Answer 2 · 2021-05-18T20:22:29.267

For those looking for a quick answer, here's the main code:

await Promise.all([page.waitForNavigation(), el.click()]);

...where el is a link that points to another page in the SPA and click can be any event that causes navigation. See below for details.

I agree that waitFor isn't too helpful if you can't rely on page content. Even if you can, in most cases it seems like a less desirable approach than naturally reacting to the navigation. Luckily, page.waitForNavigation does work on SPAs. Here's a minimal, complete example of navigating between pages using a click event on a link (the same should work for a form submission) on a tiny vanilla SPA mockup which uses the history API (index.html below). I used Node 10 and Puppeteer 5.4.1.

`index.html`:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
  </head>
  <body>
    <script>
      const nav = `<a href="/">Home</a> | <a href="/about">About</a> | 
                   <a href="/contact">Contact</a>`;
      const routes = {
        "/": `<h1>Home</h1>${nav}<p>Welcome home!</p>`,
        "/about": `<h1>About</h1>${nav}<p>This is a tiny SPA</p>`,
      };
      const render = path => {
        document.body.innerHTML = routes[path] || `<h1>404</h1>${nav}`;
        document.querySelectorAll('[href^="/"]').forEach(el => 
          el.addEventListener("click", evt => {
            evt.preventDefault();
            const {pathname: path} = new URL(evt.target.href);
            window.history.pushState({path}, path, path);
            render(path);
          })
        );
      };
      window.addEventListener("popstate", e =>
        render(new URL(window.location.href).pathname)
      );
      render("/");
    </script>
  </body>
</html>

`index.js`:

const puppeteer = require("puppeteer");

let browser;
(async () => {
  browser = await puppeteer.launch();
  const page = await browser.newPage();

  // navigate to the home page for the SPA and print the contents
  await page.goto("http://localhost:8000");
  console.log(page.url());
  console.log(await page.$eval("p", el => el.innerHTML));

  // navigate to the about page via the link
  const [el] = await page.$x('//a[text()="About"]');
  await Promise.all([page.waitForNavigation(), el.click()]);

  // show proof that we're on the about page
  console.log(page.url());
  console.log(await page.$eval("p", el => el.innerHTML));
})()
  .catch(err => console.error(err))
  .finally(async () => await browser.close())
;

Sample run:

$ python3 -m http.server &
$ node index.js
http://localhost:8000/
Welcome home!
http://localhost:8000/about
This is a tiny SPA

If the await Promise.all([page.waitForNavigation(), el.click()]); pattern seems strange, see this issue thread which explains the gotcha that the intuitive

await page.waitForNavigation(); 
await el.click();

causes a race condition.

The same thing as the Promise.all shown above can be done with:

const navPromise = page.waitForNavigation({timeout: 1000});
await el.click();
await navPromise;

See this related answer for more on navigating SPAs with Puppeteer including hash routers.

Mouneer · Answer 3 · 2021-05-20T00:30:28.260

A workaround for single page application to wait for navigation and get the response status and data. Whether fetch or XHR is being used to do the Ajax request, the main idea should be the same. The following example demonstrates it with fetch

  async spaClick (selector) {
    const res = await this.eval(selector, el => {
      window.originalFetch = window.originalFetch || window.fetch
      return new Promise(resolve => {
        window.fetch = function (...args) {
          return window.originalFetch.apply(this, args)
            .then(async response => {
              resolve({
                status: response.status,
                data: await response.clone().text()
              })

              return response
            })
        }

        el.click()
      })
    })

    if (!res) throw new Error('spaClick() Navigation triggered before eval resolves!')
    return res
  }

const puppeteer = require('puppeteer');
const url = 'http://www.faalkaart.nl';

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    // await Promise.all([
    //   page.waitForNavigation({ waitUntil: 'networkidle0' }),
    //   page.click('selector-that-triggers-navigation'),
    // ]);
    const response = await spaClick('selector-that-triggers-navigation')
    console.log(response) // {status, data}
    await browser.close();
})();

puppeteer: How to wait for pages in SPA's?

3 Answers3

`index.html`:

`index.js`:

Sample run:

Linked

puppeteer: How to wait for pages in SPA's?

3 Answers3

index.html:

index.js:

Sample run:

Linked

`index.html`:

`index.js`: