Call a Spider's method from Pipeline - Python Scrapy

Question

This is Spider

class TicketsSpider(scrapy.Spider):

    def __set_last_start_date(self, dateString):
        #code here

This is Pipeline

class TicketsPipeline(object):

    def spider_closed(self, spider):
        spider.__set_last_start_date(spider.lastAdScrapedDate)

    @classmethod
    def from_crawler(cls, crawler):
        pipeline = cls()
        crawler.signals.connect(pipeline.spider_closed, signals.spider_closed)
        return pipeline

I want to call __set_last_start_date() function from spider_closed() method. But I am getting this error

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 149, in maybeDeferred
    result = f(*args, **kw)
  File "build/bdist.linux-x86_64/egg/pydispatch/robustapply.py", line 55, in robustApply
  File "tickets/pipelines.py", line 236, in spider_closed
    spider.__set_last_start_date(spider.lastAdScrapedDate)
AttributeError: 'TicketsSpider' object has no attribute '_TicketsPipeline__set_last_start_date'
2016-12-13 02:49:53 [scrapy] INFO: Dumping Scrapy stats:

I can assure you that I can get spider.lastAdScrapedDate but I am unable to call spider.__set_last_start_date

Are you able to call any other spider method? Is `__set_last_start_date` correctly defined? Maybe you need to investigate with a debugger — paul trmbrth, Dec 13 '16 at 11:21
I can get `spider.lastAdScrapedDate` but I am unable to call `spider.__set_last_start_date` — Umair Ayub, Dec 13 '16 at 11:22
I'm talking about methods, not attributes. Why would the error mention `_TicketsPipeline__set_last_start_date` and not `'TicketsSpider' object has no attribute '__set_last_start_date'`? Try with pdb — paul trmbrth, Dec 13 '16 at 11:24
@paultrmbrth `spider.__get_last_start_date()` is also giving same error ... I am not expert in Python and Debugging Python ... what could be reason for this error as first thought? — Umair Ayub, Dec 13 '16 at 12:06

score 3 · Accepted Answer · edited May 23 '17 at 12:24

3

Names, in a class, with double underscore leading are intended to be private.

Any identifier of the form __spam (at least two leading underscores, at most one trailing underscore) is textually replaced with _classname__spam, where classname is the current class name with leading underscore(s) stripped.

Please see this for detail.

option 1. You can rename your method name, not use double underscore leading.

option 2. If you want to keep your method name, then call it in this way, but I don't think it's a good idea:

    def close_spider(self, spider):
        spider._TicketsSpider__set_last_start_date(spider.lastAdScrapedDate)

edited May 23 '17 at 12:24

Community

1
1

answered Dec 14 '16 at 01:40

pennyliangliping

93
5

Thanks @pennyliangliping, I had no idea that double-underscore did that. – paul trmbrth Dec 14 '16 at 09:42
@pennyliangliping never thought of that :P – Umair Ayub Dec 14 '16 at 11:03

Call a Spider's method from Pipeline - Python Scrapy

1 Answers1