0

I am building a web scraper to get some specific information from some web pages. I have the links of the pages on a table and I want to create a loop so that in every iteration my function can load the html of a particular website and check if a phrase is located in that site. In the case where the phrase doesn't exist that table row will be deleted.

I realized that for some reason when I am running the loop, the JQuery function that should scrape an individual link waits for the loop to be over and then scrapes the same page 50 times, instead of scraping a different web page each time.

    //snippet has been simplified to make the problem clear
    //assume urlList is a list of 50 different url's 

    function dataScraper(urlList){  
      for(i in urlList){
        $.get('https://allorigins.me/get?method=raw&url=' + encodeURIComponent(urlList[i]) + '&callback=?', function(data){
          localStorage.setItem("urlHTML"+String(i),data.contents)
          console.log(i) //for example this just prints 49, 50 times instead of printing all the separate values of i
          });
        }
       }
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>

I get that JQuery is waiting for the whole thing to load, but how do I make sure that the JQuery function accepts each individual (i) instead of the max (i).

Disclaimer: I am a self-taught newbie, and have recently figured out JavaScript, HTML and JQuery. Apologies if I am missing something completely obvious.

Cefi Menda
  • 1
  • If this is javascript, maybe you want to try reworking it to use "forEach()" and a counter instead? Something like var i = 0; urlList.forEach(function(url) { // all your get stuff ; i++; }); ? – Avenar May 10 '18 at 22:43
  • `i` is not within the scope of your callback functions, so it just outputs the global value from the last iteration of the loop. – showdev May 10 '18 at 23:25
  • Possible duplicate of [jQuery pass more parameters into callback](https://stackoverflow.com/questions/939032/jquery-pass-more-parameters-into-callback) – showdev May 10 '18 at 23:31
  • Here's a [working example](https://jsfiddle.net/x0g5m11x/) based on the referenced accepted answer. – showdev May 10 '18 at 23:35

1 Answers1

0

Here is the way I reworked it based on comments provided and some more research.

function dataScraper(urlList){
  var i = 0

  $.each(urlList,function(){
    $.get('https://allorigins.me/get?method=raw&url=' + encodeURIComponent(urlList[i]) + '&callback=?', function(data){
    localStorage.setItem("urlHTML"+String(i),data.contents)
    console.log(i)
    i++
        });
  });
}
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
Cefi Menda
  • 1