7

There are some URLs with [] in it like

http://www.website.com/CN.html?value_ids[]=33&value_ids[]=5007

But when I try scraping this URL with Scrapy, it makes Request to this URL

http://www.website.com/CN.html?value_ids%5B%5D=33&value_ids%5B%5D=5007

How can I force scrapy to not to urlenccode my URLs?

Umair Ayub
  • 19,358
  • 14
  • 72
  • 146

1 Answers1

3

When creating a Request object scrapy applies some url encoding methods. To revert these you can utilize a custom middleware and change the url to your needs.

You could use a Downloader Middleware like this:

class MyCustomDownloaderMiddleware(object):

    def process_request(self, request, spider):
        request._url = request.url.replace("%5B", "[", 2)
        request._url = request.url.replace("%5D", "]", 2)

Don't forget to "activate" the middleware in settings.py like so:

DOWNLOADER_MIDDLEWARES = {
    'so.middlewares.MyCustomDownloaderMiddleware': 900,
}

My project is named so and in the folder there is a file middlewares.py. You need to adjust those to your environment.

Frank Martin
  • 2,584
  • 2
  • 22
  • 25