Puppeteer how to retry url fetch with delay if it failed

Question

I try to write simple web-scraper using puppeteer library.

When I get page by url via page.goto, I need to retry if it failed, i.e response code is >= 400.

My snippet:

'use strict';
const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch({headless: false});
    const page = await browser.newPage();
    await page.setViewport({width: 1024, height: 768});
    await page.setDefaultNavigationTimeout(0);

    await page.goto('https://google.com');

    await browser.close();
    process.exit();
})();

I need to implement fail strategy to retry url if response.code is >= 400.
I need delay beetween attempts equal to retryNumber * 1000ms:

1000 ms for first attempt;
2000 ms for second attempt;
3000 ms for third attempt and so on.

Promise should be rejected if retryNumber exceeds maxRetryNumber.

Who knows how to implement this via code? Are there any ready to use packets or snippets to achieve the goal?

https://github.com/zeit/async-retry – laggingreflex Aug 18 '19 at 18:31 — laggingreflex, Aug 18 '19 at 18:31

score 3 · Accepted Answer · edited Aug 19 '19 at 04:12

You can then use a simple for loop to execute your retries (exit the for loop when your request was successful):

'use strict';
const puppeteer = require('puppeteer');
const delay = (ms) => {
    return new Promise(resolve => setTimeout(resolve, ms));
};

(async () => {
    const browser = await puppeteer.launch({headless: false});
    const page = await browser.newPage();
    await page.setViewport({width: 1024, height: 768});
    await page.setDefaultNavigationTimeout(0);

    const maxRetryNumber = 10;
    let success = false;
    for (let retryNumber = 1; retryNumber <= maxRetryNumber; retryNumber++) {
        const response = await page.goto('https://google.com');
        if (response.status() < 400) {
            success = true;
            break;
        }
        await delay(1000 * retryNumber);
    }

    if (!success) {
        // do something
    }

    await browser.close();
    process.exit();
})();

Source of delay function.

Thanks for pseudocode, it helped me to build real code. – userlond Aug 19 '19 at 03:50 — userlond, Aug 19 '19 at 03:50

score 1 · Answer 2 · edited Jun 01 '21 at 13:05

var maxRetryNumber = 10 ;
var retryNumber  = 0 ;

scrape();

async function scrape(){

    retryNumber++;
    if(retryNumber >= maxRetryNumber )
    {
        console.log(' retryNumber exceeded maxRetryNumber ! ');
        return ;
    }

    try {
        const browser = await puppeteer.launch({headless: false});
        const page = await browser.newPage();
        await page.setViewport({width: 1024, height: 768});
        await page.setDefaultNavigationTimeout(0);
        await page.waitFor(retryNumber*1000);
        let response = await page.goto('https://google.com');
        await browser.close();

        if(response.status()   >= 400)
            scrape();
        else
        {
            console.log('ALL OK');
        }
    }
    catch(e){

        scrape();

    }
}

Seems like recursive solutuion isn't the best approach in this case — userlond, Aug 19 '19 at 03:49

Puppeteer how to retry url fetch with delay if it failed

2 Answers2