Python urllib urlopen not working

Question

I am just trying to fetch data from a live web by using the urllib module, so I wrote a simple example

Here is my code:

import urllib

sock = urllib.request.urlopen("http://diveintopython.org/") 
htmlSource = sock.read()                            
sock.close()                                        
print (htmlSource)

But I got error like:

Traceback (most recent call last):
  File "D:\test.py", line 3, in <module>
    sock = urllib.request.urlopen("http://diveintopython.org/") 
AttributeError: 'module' object has no attribute 'request'

Martijn Pieters · Accepted Answer · 2014-09-16T07:50:46.613

26

You are reading the wrong documentation or the wrong Python interpreter version. You tried to use the Python 3 library in Python 2.

Use:

import urllib2

sock = urllib2.urlopen("http://diveintopython.org/") 
htmlSource = sock.read()                            
sock.close()                                        
print htmlSource

The Python 2 urllib2 library was replaced by urllib.request in Python 3.

edited Sep 16 '14 at 07:50

answered Sep 16 '14 at 07:38

Martijn Pieters

1,048,767
296
4,058
3,343

score 7 · Answer 2 · edited Apr 24 '17 at 00:37

7

import requests
import urllib

link = "http://www.somesite.com/details.pl?urn=2344"

f = urllib.request.urlopen(link)
myfile = f.read()

writeFileObj = open('output.xml', 'wb')
writeFileObj.write(myfile)
writeFileObj.close()

edited Apr 24 '17 at 00:37

Enrique Saez

2,504
16
21

answered Apr 23 '17 at 19:44

Mostafa Ezz

83
1
2

4

Usually a good answer doesn't contain just code but a little information on what exactly got fixed. – Loïc Faure-Lacroix Apr 23 '17 at 20:34
1

isn't requests an different module that you're not actually using in this answer? – Rick Jul 01 '22 at 21:10

score 6 · Answer 3 · answered Mar 06 '18 at 15:59

In Python3 you can use urllib or urllib3

urllib:

import urllib.request
with urllib.request.urlopen('http://docs.python.org') as response:
    htmlSource = response.read()

urllib3:

import urllib3
http = urllib3.PoolManager()
r = http.request('GET', 'http://docs.python.org')
htmlSource = r.data

More details could be found in the urllib or python documentation.

yamm · Answer 4 · 2014-09-16T08:49:45.550

3

This is what i use to get data from urls, its nice because you could save the file at the same time if you need it:

import urllib

result = urllib.urlretrieve("http://diveintopython.org/")

print open(result[0]).read()

output:

'<!DOCTYPE html><body style="padding:0; margin:0;"><iframe src="http://mcc.godaddy.com/park/pKMcpaMuM2WwoTq1LzRhLzI0" style="visibility: visible;height: 2000px;" allowtransparency="true" marginheight="0" marginwidth="0" frameborder="0" scrolling="no" width="100%"></iframe></body></html>'

Edit: urlretrieve works in python 2 and 3

edited Sep 16 '14 at 08:49

answered Sep 16 '14 at 07:55

yamm

1,523
1
15
25

`urlretrieve` is available on Python 3 as well as part of the [legacy interface](https://docs.python.org/3/library/urllib.request.html#legacy-interface). If all you wanted was the data *in memory*, then `urlretrieve` is the wrong tool; you downloaded the data to disk then opened it again from the file. – Martijn Pieters Sep 16 '14 at 08:34
1

Moreover, this doesn't even attempt to answer the question stated. – Martijn Pieters Sep 16 '14 at 08:35
well i stated that urlretrieve is nice because you can save the file ... thats the whole point. if someone tells me he wants to fetch data from the internet why not suggesting something like that. – yamm Sep 16 '14 at 08:48
and i didnt know that it works in python 3 ;-) though it is outdated or something – yamm Sep 16 '14 at 08:49

score 0 · Answer 5 · edited Apr 18 '19 at 08:31

0

Make sure you import requests from urllib, then try this format, it worked for me:

from urllib import request
urllib.request.urlopen( )

edited Apr 18 '19 at 08:31

DaveyDaveDave

9,821
11
64
77

answered Jul 18 '18 at 09:27

Seth Okeyo

59
2

score 0 · Answer 6 · answered Feb 09 '20 at 09:09

I just queried the same question which is now over 5 years old.

Please note that the URL given is also old, so i substituted the python welcome page.

We can use the requests module in python 3.

I use python 3 and the solution is below:

import requests

r = requests.get('https://www.python.org/')
t = r.text

print(t)

This works and is clean.

score 0 · Answer 7 · answered Mar 31 '20 at 17:02

For python 3 the correct way should be:

import cv2
import numpy as np
import urllib.request

req = urllib.request.urlopen('http://answers.opencv.org/upfiles/logo_2.png')
arr = np.asarray(bytearray(req.read()), dtype=np.uint8)
img = cv2.imdecode(arr, -1) # 'Load it as it is'

cv2.imshow('image_name', img)
if cv2.waitKey() & 0xff == 27: quit()

Here you can find the documentation related to urllib.request

score -1 · Answer 8 · edited Mar 17 '21 at 16:28

-1

Use this

    import cv2
    import  numpy as np
    import urllib //import urllib using pip
    import requests // import requests using pip`enter code here`
    url = "write your url"
    while True:
    imgresp = urllib.request.urlopen(url)
    imgnp = np.array(bytearray(imgresp.read()),dtype=np.uint8)
    img = cv2.imdecode(imgnp,-1)
    cv2.imshow("test",img)
    cv2.waitKey('q')

edited Mar 17 '21 at 16:28

Matt Sephton

3,711
4
35
46

answered Apr 18 '19 at 08:06

Rajpurohit Dhanpal Singh

121
2
3

2

Please try to correctly format your code if you're posting an answer - this isn't readable. Also, it doesn't really seem like you're addressing the question; you're using numpy, which isn't mentioned in the question, then seem to have some commented-out imports, and finally it seems to be an image you're downloading, which again isn't what the question asks for. – DaveyDaveDave Apr 18 '19 at 08:31

Python urllib urlopen not working

8 Answers8

Linked