scraping using cheerio returns empty array

Question

I am trying to scrape a website https://buff.163.com/market/csgo#tab=buying&page_num=1 using request-promise and cheerio. my entire code is below :

const request = require('request-promise');
const cheerio = require('cheerio');

const url = "https://buff.163.com/market/csgo#tab=buying&page_num=1";

const scrapeArr = [];

async function scrape() {
    try {
        const htmlResult = await request.get(url);
        const $ = await cheerio.load(htmlResult);
        
        $(".card_csgo li")
            .each((i, e) => { 
                const title = $(e).children("h3").text() 
                const link = $(e).children("a").attr("href")
                const scrapeObj = { title , link }
                scrapeArr.push(scrapeObj)
            })
            console.log(scrapeArr);
    } catch(e) {
        console.log(e)
    }
}

scrape()

this results in a empty scrapeArr, The jQuery commands work perfectly in the chrome dev tools console but when I copy paste the same command in cheerio it results in empty array. can someone tell me what's the problem here?

Canonical: [How can I scrape pages with dynamic content using node.js?](https://stackoverflow.com/questions/28739098/how-can-i-scrape-pages-with-dynamic-content-using-node-js) — ggorlen, Nov 23 '22 at 06:14

score 0 · Accepted Answer · answered Nov 23 '22 at 06:07

If you open the source of the page, you will see that there are no such DOM elements in HTML. They are generated by browser in runtime, so you need full-fledged browser (and not raw network request + cheerio) to render them.

The good news is that this website has hidden API which you should better use to extract the data you need, just use Chrome Dev Tools to discover it: https://www.youtube.com/watch?v=kPe3wtA9aPM

scraping using cheerio returns empty array

1 Answers1