How to read in url in python and then print each URL on the website?

Question

I am trying to figure out how to only read in each line that is a url from a website, every time I run the code I get the error:

AttributeError: module 'urllib' has no attribute 'urlopen'

My code is below

import os
import subprocess
import urllib

datasource = urllib.urlopen("www.google.com")

while 1:
        line = datasource.readline()
        if line == "": break
        if (line.find("www") > -1) :
                print (line)


li = ['www.apple.com', 'www.google.com']
os.chdir('..')
os.chdir('..')
os.chdir('..')
os.chdir('Program Files (x86)\\LinkChecker')

for s in li:
    os.system('Start .\linkchecker ' + s)

afaik urllib.urlopen is python2 ... in python3 try `urllib.request.urlopen` — Joran Beasley, Jun 07 '17 at 20:15

score 1 · Answer 1 · answered Jun 07 '17 at 20:18

1

This is very simple example.

This works in Python 3.2 and greater.

import urllib.request
with urllib.request.urlopen("http://www.apple.com") as url:
    r = url.read()
print(r)

For reference, go through this question. Urlopen attribute error.

answered Jun 07 '17 at 20:18

danglingpointer

4,708
3
24
42

itzMEonTV · Accepted Answer · 2017-06-07T20:19:12.937

0

Seems python3X, so you should use

urllib.request.urlopen

edited Jun 07 '17 at 20:19

answered Jun 07 '17 at 20:16

itzMEonTV

19,851
4
39
49

Must be `datasource = urllib.request.urlopen("http://www.google.com")` (`urllib.request.urlopen` does not add `"http://"`) – DYZ Jun 07 '17 at 20:17
removed that part. OP will understand that once it works :) – itzMEonTV Jun 07 '17 at 20:20

score 0 · Answer 3 · answered Feb 12 '18 at 18:19

The AttributeError was because it should be urllib.request.urlopen instead of urllib.urlopen.

Apart from the AttributeError mentioned in the question, I faced 2 more errors.

ValueError: unknown url type: 'www.google.com'

Solution: Rewrite the line defining datasource as follows where the https part is included:

datasource = urllib.request.urlopen("https://www.google.com")
TypeError: a bytes-like object is required, not 'str' in the line ' if (line.find("www") > -1) :`.

The overall solution code is:

import os
import urllib

datasource = urllib.request.urlopen("https://www.google.com")

while 1:
        line = str(datasource.read())
        if line == "": break
        if (line.find("www") > -1) :
                print (line)

li = ['www.apple.com', 'www.google.com']
os.chdir('..')
os.chdir('..')
os.chdir('..')
os.chdir('Program Files (x86)\\LinkChecker')

for s in li:
    os.system('Start .\linkchecker ' + s)

How to read in url in python and then print each URL on the website?

3 Answers3