1

I need to scrape a website with Scrapy that uses AJAX Pagination. How can I be sure that the page is fully loaded ?

I used: 

Download_Delay = 2

I need something like document.ready in js.

RattleyCooper
  • 4,997
  • 5
  • 27
  • 43
parik
  • 2,313
  • 12
  • 39
  • 67
  • scrapy doesn't have that, try with selenium or [Splash](https://github.com/scrapinghub/splash). – eLRuLL May 04 '16 at 14:49
  • you could still check which specific request was done to get the information you need and replicate that with scrapy. To debug requests I would recommend [Firebug](http://getfirebug.com/) for firefox. – eLRuLL May 04 '16 at 14:51
  • On the Splash note I recommend this library [ScrapyJS](https://github.com/scrapy-plugins/scrapy-splash) – Rafael Almeida May 04 '16 at 16:04
  • @RafaelAlmeida on Scrapy we can use ScrapyJs too, but i send my requests by FormRequest and i don't use scrapyJS for clicking – parik May 04 '16 at 16:11
  • 1
    FormRequest doesn't use any javascript, it only makes the post or get request on a form with the data provided. ScrapyJS loads the javascript before returning the page source which is precisely what you want – Rafael Almeida May 04 '16 at 16:14
  • 1
    @parik the great mighty Alecxe explains best. Check out his answers http://stackoverflow.com/questions/17975471/selenium-with-scrapy-for-dynamic-page/17979285#17979285 http://stackoverflow.com/questions/30345623/scraping-dynamic-content-using-python-scrapy – Rafael Almeida May 04 '16 at 16:18

0 Answers0