getting http debug info

Question

I'm going through Dive into Python3. When I get to the chapter on http web services section 14.4, I can't seem to duplicate the following output in the python3 shell. Here's what the sample code looks like:

from http.client import HTTPConnection
HTTPConnection.debuglevel = 1
from urllib.request import urlopen
response = urlopen('http://diveintopython3.org/examples/feed.xml')

send: b'GET /examples/feed.xml HTTP/1.1
Host: diveintopython3.org
Accept-Encoding: identity
User-Agent: Python-urllib/3.1'
Connection: close
reply: 'HTTP/1.1 200 OK'
…further debugging information omitted…

When I enter this in ipython3, the final command gives no output. So why am I not getting the debug info in the example? After entering the above code, response.debuglevel == 0. I'm using python3.5.2.

you should be setting your debug level on HTTPConnection, not on response — danyamachine, Aug 31 '16 at 22:59
And you should use `urllib2.urlopen()` since `urllib` got deprecated in Python3: https://docs.python.org/2/library/urllib.html — Nir Alfasi, Aug 31 '16 at 23:02
@alfasin That's wrong per your own link: "Also note that the urllib.request.urlopen() function in Python 3 is equivalent to urllib2.urlopen() and that urllib.urlopen() has been removed." — Jakub, Jun 06 '17 at 19:55
@alfasin That's exactly what I'm saying, you're not reading that correctly. It's saying "the urllib.request.urlopen() function in Python 3 is equivalent to urllib2.urlopen() [in Python 2] and that urllib.urlopen() has been removed [from Python 3]." `urllib` is absolutely still in python (see: https://docs.python.org/3.6/library/urllib.html) and `urllib2` isn't a module in Python 3. — Jakub, Jun 06 '17 at 20:03

Nir Alfasi · Answer 1 · 2017-06-06T20:35:22.313

2

The final command should not give any output, what you probably want is:

print(response.read())

edited Jun 06 '17 at 20:35

answered Aug 31 '16 at 23:09

Nir Alfasi

53,191
11
86
129

It gives the debug output shown in the question when I run the python code on my laptop. I checked the docs you link. urllib got split into urllib.request, urllib.parse, and urllib.error. It seems that urllib2 is actually deprecated. – William234234 Aug 31 '16 at 23:27

score 0 · Answer 2 · answered Nov 14 '22 at 21:04

I know this is an old question, bit I thought I would answer to help those who might still be seeing this question.

Ever since Python version 3.5.2 (release ~June 2016) the http.client.HTTPConnection.debuglevel is entirely ignored in favor of the debuglevel constructor argument for urllib.request.HTTPHandler.

This is due to this change that sets the value of http.client.HTTPConnection.debuglevel to whatever is set in urllib.request.HTTPHandler's constructor argument debuglevel, on this line.

A PR has been opened to fix this, but in the mean time you can either do one of two things:

You can use the constructor argument for HTTPHandler and HTTPSHandler (as demonstrated in this SO answer):

import urllib.request

handler = urllib.request.HTTPHandler(debuglevel=10)
opener = urllib.request.build_opener(handler)
content = opener.open('http://stackoverflow.com').read()

print(content[0:120])

You can also monkey patch the __init__ methods of HTTPHandler and HTTPSHandler to respect the global values like so:

https_old_init = urllib.request.HTTPSHandler.__init__

def https_new_init(self, debuglevel=None, context=None, check_hostname=None):
    debuglevel = debuglevel if debuglevel is not None else http.client.HTTPSConnection.debuglevel
    https_old_init(self, debuglevel, context, check_hostname)

urllib.request.HTTPSHandler.__init__ = https_new_init

http_old_init = urllib.request.HTTPHandler.__init__

def http_new_init(self, debuglevel=None):
    debuglevel = debuglevel if debuglevel is not None else http.client.HTTPSConnection.debuglevel
    http_old_init(self, debuglevel)

urllib.request.HTTPHandler.__init__ = http_new_init

Note: I don't recommend setting the debuglevel in HTTPHandler's as a method argument default value because the default values for method arguments get evaluated at function definition evaluation time, which, for HTTPHandler's constructor, is when the module urllib.request is imported.

getting http debug info

2 Answers2