How to catch 404 error in urllib.urlretrieve

Question

Background: I am using urllib.urlretrieve, as opposed to any other function in the urllib* modules, because of the hook function support (see reporthook below) .. which is used to display a textual progress bar. This is Python >=2.6.

>>> urllib.urlretrieve(url[, filename[, reporthook[, data]]])

However, urlretrieve is so dumb that it leaves no way to detect the status of the HTTP request (eg: was it 404 or 200?).

>>> fn, h = urllib.urlretrieve('http://google.com/foo/bar')
>>> h.items() 
[('date', 'Thu, 20 Aug 2009 20:07:40 GMT'),
 ('expires', '-1'),
 ('content-type', 'text/html; charset=ISO-8859-1'),
 ('server', 'gws'),
 ('cache-control', 'private, max-age=0')]
>>> h.status
''
>>>

What is the best known way to download a remote HTTP file with hook-like support (to show progress bar) and a decent HTTP error handling?

Not providing an HTTP Status on your request should probably be considered a bug in the stdlib (but check out the far better library, requests, below) — fatal_error, Mar 17 '16 at 20:37
it's so stupid that urlretrieve can't handle this with a return status — Sibbs Gambling, Aug 09 '18 at 00:37

score 28 · Answer 1 · answered Aug 20 '09 at 21:11

28

Check out urllib.urlretrieve's complete code:

def urlretrieve(url, filename=None, reporthook=None, data=None):
  global _urlopener
  if not _urlopener:
    _urlopener = FancyURLopener()
  return _urlopener.retrieve(url, filename, reporthook, data)

In other words, you can use urllib.FancyURLopener (it's part of the public urllib API). You can override http_error_default to detect 404s:

class MyURLopener(urllib.FancyURLopener):
  def http_error_default(self, url, fp, errcode, errmsg, headers):
    # handle errors the way you'd like to

fn, h = MyURLopener().retrieve(url, reporthook=my_report_hook)

answered Aug 20 '09 at 21:11

orip

73,323
21
116
148

I don't want to specify handlers; does it throw exceptions like urllib2.urlopen? – Sridhar Ratnakumar Aug 20 '09 at 21:14
4

It's very easy to make it throw. FancyURLopener subclasses URLopener which does throw, so you can try calling the base class's implementation: def http_error_default(...): URLopener.http_error_default(...) – orip Aug 20 '09 at 21:35
2

You should rather do opener = MyURLopener() and then opener.retrieve() to keep the opener object alive. Otherwise (if you do all on a single line) the newly created opener will be immediately deallocated right after the retrieve operation. This will erase the temporary file the data was downloaded to before you have a chance to use it. – KT. Feb 21 '13 at 22:25

score 15 · Answer 2 · answered Feb 04 '10 at 20:17

You should use:

import urllib2

try:
    resp = urllib2.urlopen("http://www.google.com/this-gives-a-404/")
except urllib2.URLError, e:
    if not hasattr(e, "code"):
        raise
    resp = e

print "Gave", resp.code, resp.msg
print "=" * 80
print resp.read(80)

Edit: The rationale here is that unless you expect the exceptional state, it is an exception for it to happen, and you probably didn't even think about it -- so instead of letting your code continue to run while it was unsuccessful, the default behavior is--quite sensibly--to inhibit its execution.

score 2 · Answer 3 · answered Aug 20 '09 at 21:13

2

The URL Opener object's "retreive" method supports the reporthook and throws an exception on 404.

http://docs.python.org/library/urllib.html#url-opener-objects

answered Aug 20 '09 at 21:13

Mark

106,305
20
172
230

How to catch 404 error in urllib.urlretrieve

3 Answers3

Linked

Related