Puppeteer wait page load after form submit

Question

I submit a form using the following code and i want Puppeteer to wait page load after form submit.

await page.click("button[type=submit]");

//how to wait until the new page loads before taking screenshot?
// i don't want this:
// await page.waitFor(1*1000);  //← unwanted workaround
await page.screenshot({path: 'example.png'});

How to wait for page load with puppeteer?

Answer 1

You can wait for navigation asynchronously to avoid getting null on redirection,

await Promise.all([
    page.click('button[type=submit]'),
    page.waitForNavigation({waitUntil: 'networkidle2'})
]);

This will help you if the page.click already triggers a navigation.

Answer 2

await page.waitForNavigation();

Answer 3

According to the Official Documentation, you should use:

page.waitForNavigation(options)

• options <Object> Navigation parameters which might have the following properties:
  • timeout <number> Maximum navigation time in milliseconds, defaults to 30 seconds, pass 0 to disable timeout. The default value can be changed by using the page.setDefaultNavigationTimeout(timeout) method.
  • waitUntil <string|Array<string>> When to consider navigation succeeded, defaults to load. Given an array of event strings, navigation is considered to be successful after all events have been fired. Events can be either:
    • load - consider navigation to be finished when the load event is fired.
    • domcontentloaded - consider navigation to be finished when the DOMContentLoaded event is fired.
    • networkidle0 - consider navigation to be finished when there are no more than 0 network connections for at least 500 ms.
    • networkidle2 - consider navigation to be finished when there are no more than 2 network connections for at least 500 ms.
• returns: <Promise<[?Response]>> Promise which resolves to the main resource response. In case of multiple redirects, the navigation will resolve with the response of the last redirect. In case of navigation to a different anchor or navigation due to History API usage, the navigation will resolve with null.

Readability:

You can use page.waitForNavigation() to wait for a page to navigate:

await page.waitForNavigation();

Performance:

But since page.waitForNavigation() is a shortcut for page.mainFrame().waitForNavigation(), we can use the following for a minor performance enhancement:

await page._frameManager._mainFrame.waitForNavigation();

Answer 4

Sometimes even using await page.waitForNavigation() will still result in a Error: Execution context was destroyed, most likely because of a navigation.

In my case, it was because the page was redirecting multiple times. The API says the default waitUntil option is Load—this required me to wait for navigation each redirect (3 times).

Using only a single instance of page.waitForNavigation with the waitUntil option networkidle2 worked well in my case:

await button.click();

await page.waitForNavigation({waitUntil: 'networkidle2'});

Finally, the API suggests using a Promise.All to prevent a race condition. I haven't needed this but provide it for completeness:

await Promise.all([button.click(), page.waitForNavigation({waitUntil:'networkidle2'})])

If all else fails, you can use page.waitForSelector as recommended on a Puppeteer github issue—or in my case, page.waitForXPath()

Answer 5

i suggest to wrap page.to in a wrapper and wait for everything loaded

this is my wrapper

loadUrl: async function (page, url) {
    try {
        await page.goto(url, {
            timeout: 20000,
            waitUntil: ['load', 'domcontentloaded', 'networkidle0', 'networkidle2']
        })
    } catch (error) {
        throw new Error("url " + url + " url not loaded -> " + error)
    }
}

now you can use this with

await loadUrl(page, "https://www.google.com")

Answer 6

I know it is bit late to answer this. It may be helpful for those who are getting below exception while doing waitForNavigation.

(node:14531) UnhandledPromiseRejectionWarning: TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded at Promise.then (/home/user/nodejs/node_modules/puppeteer/lib/LifecycleWatcher.js:142:21) at -- ASYNC -- at Frame. (/home/user/nodejs/node_modules/puppeteer/lib/helper.js:111:15) at Page.waitForNavigation (/home/user/nodejs/node_modules/puppeteer/lib/Page.js:649:49) at Page. (/home/user/nodejs/node_modules/puppeteer/lib/helper.js:112:23) at /home/user/nodejs/user/puppeteer/example7.js:14:12 at

The correct code that worked for me is as below.

await page.click('button[id=start]', {waitUntil: 'domcontentloaded'});

Similarly if you are going to a new page, code should be like

await page.goto('here goes url', {waitUntil: 'domcontentloaded'});

Answer 7

None of the above answers solved my issue. Sometimes waitForNavigation just timeout. I came up with other solution using the waitForFunction, checking if document is in ready state.

await page.waitForFunction(() => document.readyState === "complete");

Answer 8

await Promise.all([
      page.click(selectors.submit),
      page.waitForNavigation({ waitUntil: 'networkidle0' }),
]);

This would be the first priority to use as it waits for all network to complete and assumes it is done when you don't have more than 0 network call for 500ms.

you can also use

await page.waitForNavigation({ waitUntil: 'load' })

or else, you can use

await page.waitForResponse(response => response.ok())

this function can also be used in various places as it only allows to proceed further when all the calls are a success that is when all the response status is ok i.e (200-299)

Answer 9

If submitting the form opens some other page, then you may just want to wait for a selector in that page. I have often had issues using page.waitForNavigation() since it's options don't really ensure we have effectively navigated to another page.

// login page
page.click("#login");
// homepage, after login
page.waitForSelector("#home", {visible: true}); // page.waitForXpath()

Of you course you can increase the wait time for the selector.

Answer 10

This works for me

Puppeteer version: 19.2.2

page.click(".clickable-selector");
await page.waitForNavigation({ waitUntil: "load" });

Note: If you do this inside a loop. ( scrapping page-1, click to page-2, scrapping page-2 and so on... )

await page.waitForSelector(".clickable-selector", { visible: true });

Wait for this clickable selector before doing any other scrapping on the page.

Answer 11

This worked for me:

await Promise.all([
    page.goto(URL),
    page.waitForNavigation({ waitUntil: 'networkidle0' }),
]);
console.log('page loaded')

For some reason I was not able to click button (Handled an event, not in form)

<button onclick="someFunction();" class="button button2">Submit</button>

The problem was that page was rendered on server side. Thus the button didn't existed whenever I waited for input field await page.waitForSelector('button.button2')

The solution was to bind page.goto(URL) and page.waitForNavigation({ waitUntil: 'networkidle0' }) in Promise

await Promise.all([
    page.goto(URL),
    page.waitForNavigation({ waitUntil: 'networkidle0' }),
]);
console.log('page loaded')

await page.waitForSelector('button.button2')
console.log('button is here');

Answer 12

I ran into a scenario, where there was the classic POST-303-GET and an input[type=submit] was involved. It seems that in this case, the click of the button won't resolve until after the associated form's submission and redirection, so the solution was to remove the waitForNavigation, because it was executed after the redirection and thus was timing out.

Answer 13

Please try

await page.waitForNavigation()

or

await page.waitForSelector("#indecator_of_any_element_of_you_are_waiting_for")

Answer 14

I achieved the best results for a similar use case with the following:

await page.waitForSelector("button[type=submit]");

await page.click("button[type=submit]", {
      waitUntil: "networkidle2",
});

await page.screenshot({path: 'example.png'});

Basically, the use case differs for every website configuration. I highly recommend to create a command based function like i did for my project:

const puppeteer = require('puppeteer');

// Sample command set
const commands = [
{ id: 1, type: 'goto', target: '/dummy-product/' },
{ id: 2, type: 'goto', target: '/about-page' },
{ id: 4, type: 'goto', target: '/test-page' }, //size text label
];

const bot = async (url, commands) => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(url);

const statuses = [];

for (let command of commands) {
    let status = { id: command.id, status: 'failed' };

    try {
        switch (command.type) {
            case 'click':
                await page.waitForSelector(command.target);
                await page.click(command.target, {
                    waitUntil: "networkidle2",
                  });
                if(command.form){
                    await page.waitForNavigation({ waitUntil: 'networkidle0' })
                }
                status.status = 'success';
                status.message = `${command.type}: ${command.target}`;
                break;
            case 'goto':
                await page.goto(url + command.target, {
                    waitUntil: "networkidle2",
                  });
                await page.screenshot({path: 'example.png'});
                status.status = 'success';
                status.message = `${command.type}: ${command.target}`;
                break;
            case 'select':
                await page.waitForSelector(command.target);
                await page.focus(command.target, {delay: 1000});
                await page.select(command.target, command.value);
                status.status = 'success';
                status.message = `${command.type}: ${command.target}`;
                break;
            case 'input':
                await page.waitForSelector(command.target);
                await page.focus(command.target);
                await page.keyboard.type(command.value);
                status.status = 'success';
                status.message = `${command.type}: ${command.target}`;
                break;
            // You can add more command types here.
            default:
                console.log(`Unknown command type: ${command.type}`);
                status.status = 'failed';
                status.message = 'Unknown command type';
                break;
        }
    } catch (error) {
        status.status = `failed`;
        status.message = error.message;
    }

    statuses.push(status);
}

await browser.close();

return statuses;

}

bot('https://example.com/', commands)
.then(statuses => console.log(statuses))
.catch(err => console.error(err));