Command line headless browser

Question

I am looking for a command line option to get a webpage, and execute the associated JavaScript code. In other words, call a headless browser via command line.

I can't use wget, it does not load and execute the associated JavaScript:

wget --load-cookies cookies.txt -O /dev/null https://example.com/update?run=1

Use case: we have web pages that read elastisearch indexes, do some data manipulation, and update elastisearch indexes. We'd like to do the update on an hourly basis via a cron job. We don't need to capture anything, e.g. no png capture, no HTML capture. We simply need to load the webpage and execute its JavaScript via a cron job, ideally something like run-headless https://example.com/update. OS is CentOS 7.

I searched stackoverflow and did not find any answer satisfying my needs. selenium etc seem like an overkill:

• Play Framework: Run single selenium test on the command line (headless browser)
• Python Headless Browser for GAE
• How to run headless browser inside docker? - this looks intersting, but docker seems like overkill

Answer 1

After some research I found a solution using puppeteer headless browser. Ideally I wanted a single command like run-headless https://example.com/update, but login was required, hence driving the headless browser with puppeteer.

Installation steps for CentOS 7.6:

1. Install chrome

# cd
# mkdir install
# cd install/
# wget http://mirror.centos.org/centos/7/os/x86_64/Packages/vulkan-filesystem-1.1.97.0-1.el7.noarch.rpm
# yum localinstall vulkan-filesystem-1.1.97.0-1.el7.noarch.rpm
# wget http://mirror.centos.org/centos/7/os/x86_64/Packages/vulkan-1.1.97.0-1.el7.x86_64.rpm
# yum localinstall vulkan-1.1.97.0-1.el7.x86_64.rpm
# wget http://mirror.centos.org/centos/7/os/x86_64/Packages/liberation-fonts-1.07.2-16.el7.noarch.rpm
# yum localinstall liberation-fonts-1.07.2-16.el7.noarch.rpm
# vi /etc/yum.repos.d/google-chrome.repo
# cat /etc/yum.repos.d/google-chrome.repo
[google-chrome]
name=google-chrome
baseurl=http://dl.google.com/linux/chrome/rpm/stable/x86_64
enabled=1
gpgcheck=1
gpgkey=https://dl.google.com/linux/linux_signing_key.pub
# yum install google-chrome-stable

2. Install node.js

# curl -sL https://rpm.nodesource.com/setup_14.x | sudo bash -
# yum install nodejs

3. Patch /etc/sysctl.conf

This was needed to run puppeteer without disabling the sandbox:

# echo "user.max_user_namespaces=15000" >> /etc/sysctl.conf
# reboot

4. Create run-hourly.js puppeteer script

This node script has to run as a regular user, not root:

$ cd /path/to/script
$ npm install --save puppeteer
$ npm install --save pending-xhr-puppeteer
$ mkdir userDataDir
$ vi run-hourly.js # (content below)
$ node run-hourly.js

File content of run-hourly.js script:

const config = {
    userDataDir: __dirname + '/userDataDir',
    login: {
        url:        'https://www.example.com/login/',
        username:   'foobar',
        password:   'secret',
    },
    pages: [{
        url:        'https://www.example.com/update/hourly',
        pdfFile:    __dirname + '/page.pdf'
    }]
};

const puppeteer = require('puppeteer');
const { PendingXHR } = require('pending-xhr-puppeteer');

(async() => {
    // initialize headless browser
    const browser = await puppeteer.launch({
        headless:       true,   // run headless
        dumpio:         true,   // capture console log to stdout
        userDataDir:    config.userDataDir // custom user data
    });
    const page = await browser.newPage();
    const pendingXHR = new PendingXHR(page);

    // login
    await page.goto(config.login.url, {waitUntil: 'load'});
    await page.type('#loginusername', config.login.username);
    await page.type('#password', config.login.password);
    await page.click('#signin');
    await page.waitForNavigation();

    // load pages of interest
    await Promise.all(config.pages.map(async (pageCfg) => {
        await page.goto(pageCfg.url, {waitUntil: 'networkidle0'}); // wait for page load
        await page.setRequestInterception(true);  // intercept requests for next line
        await pendingXHR.waitForAllXhrFinished(); // wait for all requests to finish
        await page.pdf({path: pageCfg.pdfFile});  // generate PDF from rendered page
    }));

    await browser.close();
})();

5. Add hourly job to cron

Install the cron job as same user as the script owner

$ crontab -l
$ crontab -e
25 * * * * cd /path/to/script && node run-hourly.js > hourly.log 2>&1