Scraping dynamic data of a web page in nodejs

Question

By using node.js I am trying to scrape a web page. For this, I am using cheerio and tinyreq modules. My source code is as follows:

// scrape function
function scrape(url, data, cb) {
    req(url, (err, body) => {
        if (err) { return cb(err); }
        let $ = cheerio.load(body)
          , pageData = {};
        Object.keys(data).forEach(k => {
            pageData[k] = $(data[k]).text();
        });
        cb(null, pageData);
    });
}
scrape("https://www.activecubs.com/activity-wheel/", {
     title: ".row h1"
   , description: ".row h2"
}, (err, data) => {
    console.log(err || data);
});

In my code, the text in the h1 tag is static and in the h2 tag, it is dynamic. While I run the code, I am only getting the static data i.e., the description field data is empty.By following previous StackOverflow questions, I tried using phantom js to overcome this issue but it doesn't work for me. The dynamic data here is the data which is obtained by rotating a wheel. For any doubts on the website I am using, you can check https://www.activecubs.com/activity-wheel/.

Does this answer your question? [How can I scrape pages with dynamic content using node.js?](https://stackoverflow.com/questions/28739098/how-can-i-scrape-pages-with-dynamic-content-using-node-js) — ggorlen, Jul 02 '21 at 17:13

score 0 · Answer 1 · answered Sep 04 '17 at 11:28

0

Cheerio documentation is pretty clear https://github.com/cheeriojs/cheerio#cheerio-is-not-a-web-browser

see also https://github.com/segmentio/nightmare

answered Sep 04 '17 at 11:28

score 0 · Answer 2 · answered Sep 04 '17 at 12:27

User action can be performed using SpookyJS

SpookyJS makes it possible to drive CasperJS suites from Node.js. At a high level, Spooky accomplishes this by spawning Casper as a child process and controlling it via RPC. Specifically, each Spooky instance spawns a child Casper process that runs a bootstrap script. The bootstrap script sets up a JSON-RPC server that listens for commands from the parent Spooky instance over a transport (either HTTP or stdio). The script also sets up a JSON-RPC client that sends events to the parent Spooky instance via stdout. Check the documentation

Example

Scraping dynamic data of a web page in nodejs

2 Answers2