Javascript fetch multiple pages

Question

there is url: "http:/myurl/mydata/pages/" the task is call url+"1",url+"2",...,until there is no correspondent page, number of pages is unknown I am not sure about design of the decision. I know I can you "async await" approach, can I avoid it? Is the next approach ok?

let url= "http:/myurl/mydata/pages/";
let finished=true;
let page=1;
let datas={};
while(!finished) {
   fetch(url+page)
  .then(function(data) {
      collectData(page,data);
      page+=1;
    })
  .catch(function(error) {
      finished=true;  
  });
}  
function collectData(page,data){
  datas[page]=data;
}

Can I use Promises for this task?

Does this answer your question? [Resolve promises one after another (i.e. in sequence)?](https://stackoverflow.com/questions/24586110/resolve-promises-one-after-another-i-e-in-sequence) — jknotek, Nov 14 '19 at 21:02
Seems that you want to scrap some webs, if that is the case, you may use Cheerio.js (https://cheerio.js.org/) — lerichard_v, Nov 14 '19 at 21:04
If it is for creating offline version of data, then you might want to use some sort of search algorithm to find how many pages there are first. Like try binary searching for the last page, if this is in the hundreds. I wouldn't want to wait for each request if I was running this on many urls. Then once you find out the last page, you can request all of them at once, if there are no limitations, using Promise.all. — Sam Ulloa, Nov 14 '19 at 22:44

LCPunch · Answer 1 · 2019-11-14T22:14:58.020

1

If you want a async-await approach:

async function fetchPages(url, page) {
    try{
        const data = await fetch(url+page);
        collectData(page+url, data);
        await fetchPages(url, page + 1);
    } catch(e) {
        return 'Its over';
    }
  }
};

edited Nov 14 '19 at 22:14

answered Nov 14 '19 at 21:42

LCPunch

29
4

score 0 · Answer 2 · answered Nov 14 '19 at 21:38

This looks like a job for recursion!

let url= "http:/myurl/mydata/pages/";
let page=1;
let datas={};
recurse().then(/*...*/)


function recurse() {
    return new Promise(function(resolve, reject) {
        fetch(url+page)
        .then(function(data) {
            collectData(page,data);
            page+=1;
            return recurse()
        })
        .catch(function(error) {
            resolve()
        });
    })
}

function collectData(page,data){
    datas[page]=data;
}

NOTE: I didn't test this, so might not work exactly as written. But it should set you on the right track.

Javascript fetch multiple pages

2 Answers2