Scrapy get request url in parse

Question

How can I get the request url in Scrapy's parse() function? I have a lot of urls in start_urls and some of them redirect my spider to homepage and as result I have an empty item. So I need something like item['start_url'] = request.url to store these urls. I'm using the BaseSpider.

instead of storing them aside, during scraping you can access `requested_url`, check below my answer — Rohan Khude, Dec 13 '17 at 12:19

score 108 · Accepted Answer · edited Jan 17 '18 at 21:59

108

The 'response' variable that's passed to parse() has the info you want. You shouldn't need to override anything.

eg. (EDITED)

def parse(self, response):
    print "URL: " + response.request.url

edited Jan 17 '18 at 21:59

Per Quested Aronsson

11,380
8
54
76

answered Jan 25 '15 at 07:50

Jagu

2,471
2
22
26

11

But that is not the request url, but the response url. Scrapy's middleware handles redirections, therefore you can obtain a different url. – gusridd Jan 20 '16 at 13:09
6

response.request.url – Mayank Jaiswal Jul 27 '16 at 15:30
2

If the url has redirection, then it gives redirected url not the provided url – Rohan Khude Dec 13 '17 at 09:42

score 18 · Answer 2 · answered Dec 29 '15 at 03:57

18

The request object is accessible from the response object, therefore you can do the following:

def parse(self, response):
    item['start_url'] = response.request.url

answered Dec 29 '15 at 03:57

gusridd

864
10
16

score 11 · Answer 3 · answered Dec 13 '17 at 12:17

Instead of storing requested URL's somewhere and also scrapy processed URL's are not in same sequence as provided in start_urls.

By using below,

response.request.meta['redirect_urls']

will give you the list of redirect happened like ['http://requested_url','https://redirected_url','https://final_redirected_url']

To access first URL from above list, you can use

response.request.meta['redirect_urls'][0]

For more, see doc.scrapy.org mentioned as :

RedirectMiddleware

This middleware handles redirection of requests based on response status.

The urls which the request goes through (while being redirected) can be found in the redirect_urls Request.meta key.

Hope this helps you

I believe all you need is: `redirect_urls = response.meta.get("redirect_urls")` — Jack, Oct 29 '20 at 10:44

score 7 · Answer 4 · answered Nov 19 '13 at 22:06

You need to override BaseSpider's make_requests_from_url(url) function to assign the start_url to the item and then use the Request.meta special keys to pass that item to the parse function

from scrapy.http import Request

    # override method
    def make_requests_from_url(self, url):
        item = MyItem()

        # assign url
        item['start_url'] = url
        request = Request(url, dont_filter=True)

        # set the meta['item'] to use the item in the next call back
        request.meta['item'] = item
        return request


    def parse(self, response):

        # access and do something with the item in parse
        item = response.meta['item']
        item['other_url'] = response.url
        return item

Hope that helps.

score 3 · Answer 5 · answered Apr 17 '18 at 08:07

Python 3.5

Scrapy 1.5.0

from scrapy.http import Request

# override method
def start_requests(self):
    for url in self.start_urls:
        item = {'start_url': url}
        request = Request(url, dont_filter=True)
        # set the meta['item'] to use the item in the next call back
        request.meta['item'] = item
        yield request

# use meta variable
def parse(self, response):
    url = response.meta['item']['start_url']

Scrapy get request url in parse

5 Answers5

Linked

Related