1

I'm trying to create a program that uses a scrapy sipder multiple times for different uses. My structure is like the following:

def init_crawl(OUTPUT_FILE, ticker):
    OUTPUT_FILE = "file:///" + OUTPUT_FILE
    process = CrawlerProcess({
        'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)',
        'FEED_URI': OUTPUT_FILE,
        'LOG_ENABLED': False,
    })
    process.crawl(SeccrawlerSpider, ticker=ticker)
    process.start() 


class GUIapp():
    def __init__(self, master):
        # stuff

    def execute(self, ticker):
        # tiker is user input
        init_crawl(file_loc, ticker)

def main():
    master = tk.Tk()
    app = GUIapp(master)
    master.mainloop()

if __name__ == "__main__":
    main()

This works fine the first time, but I always get a twisted.internet.error.ReactorNotRestartable when I input a second ticker after the getting the results for the first ticker. I've tried several possible solutions on SO relating to this issue but none of them seem to be directly applicable to this. What am I missing?

Alex McLean
  • 2,524
  • 5
  • 30
  • 53
  • Possible duplicate of [ReactorNotRestartable error in while loop with scrapy](https://stackoverflow.com/questions/39946632/reactornotrestartable-error-in-while-loop-with-scrapy) – aufziehvogel Jul 16 '17 at 21:26
  • @Aufziehvogel I've tried this example, it doesn't work because it just blocks the process. I need the process to execute and then re-run the spider later – Alex McLean Jul 17 '17 at 05:01
  • Did you also see the comment `The alternative scrapy.crawler.CrawlerRunner's .crawl() "Returns a deferred that is fired when the crawling is finished."`? This sounds promising. You'd have to use [`CrawlerRunner`](https://doc.scrapy.org/en/latest/topics/api.html#scrapy.crawler.CrawlerRunner) instead of `CrawlerProcess` then. However, you must take care of setting up the twisted reactor yourself with `CrawlerRunner`. Here is more information: https://doc.scrapy.org/en/latest/topics/practices.html?highlight=CrawlerRunner – aufziehvogel Aug 31 '17 at 19:15

0 Answers0