1

EDIT: I've majorly edited the content of this post since the original to specify my problem:

I am writing a program to download webcomics, and I'm getting this weird error when downloading a page of the comic. The code I am running essentially boils down to the following line followed by the error. I do not know what is causing this error, and it is confusing me greatly. 

>>> urllib.request.urlopen("http://abominable.cc/post/47699281401")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 470, in open
    response = meth(req, response)
  File "/usr/lib/python3.4/urllib/request.py", line 580, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.4/urllib/request.py", line 502, in error
    result = self._call_chain(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 442, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 685, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 464, in open
    response = self._open(req, data)
  File "/usr/lib/python3.4/urllib/request.py", line 482, in _open
    '_open', req)
  File "/usr/lib/python3.4/urllib/request.py", line 442, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 1211, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib/python3.4/urllib/request.py", line 1183, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib/python3.4/http/client.py", line 1137, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.4/http/client.py", line 1172, in _send_request
    self.putrequest(method, url, **skips)
  File "/usr/lib/python3.4/http/client.py", line 1014, in putrequest
    self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 37-38: ordinal not in range(128)

The entirety of my program can be found here: https://github.com/nstephenh/pycomic

NStephenH
  • 53
  • 1
  • 6
  • It would've been a lot of help if you had posted the code where it failed. Anywho, as mentioned in [this post](http://stackoverflow.com/a/9942822/2399772), the reason might be that you need to use `.encode()` instead of `str` to convert unicode to encoded text. – Demitrian Oct 27 '15 at 15:10
  • The code fails at: return str(urllib.request.urlopen(req).read()) Using decode() instead of str gives me what I assume is a similar error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x97 in position 5218: invalid start byte – NStephenH Oct 27 '15 at 15:27
  • 1
    Python version? Also don't use `urllib`. There's `urllib2` and `urllib3`, but do yourself a favor and use [`requests`](http://docs.python-requests.org/en/latest/). – jpmc26 Oct 28 '15 at 03:06
  • I'm using python 3. I just switched to requests, but unfortunatly, this page still produces an error, albeit a different one: requests.exceptions.TooManyRedirects: Exceeded 30 redirects. – NStephenH Oct 28 '15 at 13:39
  • Related: [UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' - -when using urlib.request python3](https://stackoverflow.com/q/22734464), except here the URL came from a location header. – Martijn Pieters Jul 14 '19 at 19:25

1 Answers1

0

I was having the same problem. The root cause is that the remote server isn't playing by the rules. HTTP Headers are supposed to be US-ASCII only but apparently the leading http webservers (apache2, nginx) doesn't care and send direct UTF-8 encoded string.

However in http.client the parse_header function fetch the headers as iso-8859, and the default HTTPRedirectHandler in urllib doesn't care to quote the location or URI header, resulting in the aformentioned error.

I was able to 'work around' both thing by overriding the default HTTPRedirectHandler and adding three line to counter the latin1 decoding and add a path quote:

import urllib.request
from urllib.error import HTTPError
from urllib.parse import (
  urlparse, quote, urljoin, urlunparse)

class UniRedirectHandler(urllib.request.HTTPRedirectHandler):
    # Implementation note: To avoid the server sending us into an
    # infinite loop, the request object needs to track what URLs we
    # have already seen.  Do this by adding a handler-specific
    # attribute to the Request object.
    def http_error_302(self, req, fp, code, msg, headers):
        # Some servers (incorrectly) return multiple Location headers
        # (so probably same goes for URI).  Use first header.
        if "location" in headers:
            newurl = headers["location"]
        elif "uri" in headers:
            newurl = headers["uri"]
        else:
            return

        # fix a possible malformed URL
        urlparts = urlparse(newurl)

        # For security reasons we don't allow redirection to anything other
        # than http, https or ftp.

        if urlparts.scheme not in ('http', 'https', 'ftp', ''):
            raise HTTPError(
                newurl, code,
                "%s - Redirection to url '%s' is not allowed" % (msg, newurl),
                headers, fp)

        if not urlparts.path:
            urlparts = list(urlparts)
            urlparts[2] = "/"
        else:
            urlparts = list(urlparts)
            # Header should only contain US-ASCII chars, but some servers do send unicode data
            # that should be quoted back before reused
            # Need to re-encode the string as iso-8859-1 before use of ""quote"" to cancel the effet of parse_header() in http/client.py
            urlparts[2] = quote(urlparts[2].encode('iso-8859-1'))

        newurl = urlunparse(urlparts)

        newurl = urljoin(req.full_url, newurl)

        # XXX Probably want to forget about the state of the current
        # request, although that might interact poorly with other
        # handlers that also use handler-specific request attributes
        new = self.redirect_request(req, fp, code, msg, headers, newurl)
        if new is None:
            return

        # loop detection
        # .redirect_dict has a key url if url was previously visited.
        if hasattr(req, 'redirect_dict'):
            visited = new.redirect_dict = req.redirect_dict
            if (visited.get(newurl, 0) >= self.max_repeats or
                len(visited) >= self.max_redirections):
                raise HTTPError(req.full_url, code,
                                self.inf_msg + msg, headers, fp)
        else:
            visited = new.redirect_dict = req.redirect_dict = {}
        visited[newurl] = visited.get(newurl, 0) + 1

        # Don't close the fp until we are sure that we won't use it
        # with HTTPError.
        fp.read()
        fp.close()

        return self.parent.open(new, timeout=req.timeout)

    http_error_301 = http_error_303 = http_error_307 = http_error_302

[...]
# Change default Redirect Handler in urllib, should be done once at the beginning of the program
opener = urllib.request.build_opener(UniRedirectHandler())
urllib.request.install_opener(opener)

This is python3 code but should be easily adapted for python2 if need be.

silmaril
  • 423
  • 3
  • 10