5

why is the content-lenght different in case of using requests and urlopen(url).info()

>>> url = 'http://pymotw.com/2/urllib/index.html'

>>> requests.head(url).headers.get('content-length', None)
'8176'
>>> urllib.urlopen(url).info()['content-length']
'38227'
>>> len(requests.get(url).content)
38274

I was going to make a check for size of file in bytes to split the buffer to multiple threads based on Range in urllib2 but if I do not have the actual size of file in bytes it won't work..

only len(requests.get(url).content) gives 38274 which is closest but still not correct and moreover it is downloading the content which i didn't wanted.

Ciasto piekarz
  • 7,853
  • 18
  • 101
  • 197
  • It might be that the server does not correctly support `HEAD`. Or it might be that the server returns different things based on other headers sent (or not) by the respective methods (user-agent, cookies...). Try using `curl -v url` and `curl -I`, or any other method that sends the exact same request save for the `HEAD` instead of post, and check the results. – jcaron Jul 05 '14 at 09:31
  • Maybe first size is the compressed size? – BlackBear Jul 05 '14 at 09:35
  • @BlackBear : then how to get uncompressed size ? – Ciasto piekarz Jul 05 '14 at 09:36
  • You can have a (very rough) estimate by looking at the compression method. For example, gzip has a compression ratio of 3:1 - 5:1 (source: http://superuser.com/questions/139253/what-is-the-maximum-compression-ratio-of-gzip#answer-139273) – BlackBear Jul 05 '14 at 09:49

1 Answers1

15

By default, requests will send 'Accept-Encoding': 'gzip' as part of the request headers, and the server will respond with the compressed content:

>>> r = requests.head('http://pymotw.com/2/urllib/index.html')
r>>> r.headers['content-encoding'], r.headers['content-length']
('gzip', '8201')

But, if you manually set the request headers, then you'll get the uncompressed content:

>>> r = requests.head('http://pymotw.com/2/urllib/index.html',headers={'Accept-Encoding': 'identity'})
>>> r.headers['content-length']
'38227'
Burhan Khalid
  • 169,990
  • 18
  • 245
  • 284