Error when assigning variable to urlopen(my_url)

Question

I'm using BeautifulSoup and urlopen to webscrape. When I try to open the url using urlopen a large error is prompted. What am I doing wrong?

Im using python 3.7.2, Ive installed Bs4, and urlopen.

import bs4
from urllib.request import urlopen
from bs4 import BeautifulSoup as soup
my_url = 'https://ebgames.com.au/search?q=Skyrim'
var = urlopen(my_url)

ERROR:

Traceback (most recent call last):
  File "<pyshell#8>", line 1, in <module>
    urlopen (my_url)
  File "C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 531, in open
    response = meth(req, response)
  File "C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 569, in error
    return self._call_chain(*args)
  File "C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

The expected result is no error.

In case of 3.4 or more use: `import requests` and `requests.get(url)` — Anup Yadav, May 04 '19 at 09:05

score 0 · Answer 1 · answered May 04 '19 at 09:20

Try setting a browser user agent with your request as a request header:

import bs4
from urllib.request import Request, urlopen
from bs4 import BeautifulSoup as soup
my_url = req = Request('https://ebgames.com.au/search?q=Skyrim', headers={'User-Agent': 'Mozilla/5.0'})
var = urlopen(my_url).read()

for more info, refer this answer

Error when assigning variable to urlopen(my_url)

1 Answers1