1

I subclassed a CrawlSpider and want to extract data from website. However, I always get redirected to the site's mobile version. I tried to change the USER_AGENT variable in scrapy's settings to Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1, but still get redirected.

Is there another way to signal another client and avoid redirection?

Jon
  • 11,356
  • 5
  • 40
  • 74
  • Could you please show the whole code of your spider (with url you are parsing)? – alecxe Sep 27 '13 at 09:16
  • 1
    In general, when scraping, you should actively announce you are a robot, like the GoogleBot does. I can't think of a situation in which it might be ethical to misrepresent this - can you shed more light on your purpose and the nature of the target site? – halfer Sep 28 '13 at 22:47
  • I try to crawl parts of http://derstandard.at/ - however, I always get redirected to it's (express/mobile) version: http://express.derstandard.at/ – Jon Sep 30 '13 at 09:22

1 Answers1

2

There are two types of redirection supported in Scrapy:

  • RedirectMiddleware - Handle redirection of requests based on response status
  • MetaRefreshMiddleware - Handle redirection of requests based on meta-refresh html tag

So, maybe your html page uses second type of redirection?

See also:

warvariuc
  • 57,116
  • 41
  • 173
  • 227