How to scrape whole website using beautifulsoup

Question

I am trying to get all the unique urls of the website by calling the all_pages function recursively but this function is not giving all the urls of the website.

All I want to do is get all the unique urls of the website using BeautifulSoup. My code looks like this:

base_url = "http://www.readings.com.pk/"
unique_urls=[]

def all_pages(base_url,unique_urls=[]):

    response = requests.get(base_url)
    soup = BeautifulSoup(response.content, "html.parser")

    for link in soup.find_all("a"):
        url = link["href"]
        absolute_url = urljoin(base_url, url)
        if absolute_url not in unique_urls:

            if base_url in absolute_url:

                unique_urls.append(absolute_url)
                print (absolute_url)

                all_pages(absolute_url,unique_urls,book_urls)





all_pages(base_url,unique_urls)

Welcome Syed. I see you have your code as an image. I strongly recommend you place it as text and mark it as a code snippet using {} in editor. People will be more able to help you. Please read [how to ask](http://stackoverflow.com/help/how-to-ask). — micstr, Feb 14 '17 at 07:02

score 3 · Accepted Answer · edited Jun 19 '19 at 19:26

Use response.text instead of response.content

Also, you need to return at some point. Additionally, instead of making unique_urls a list, make it a set and they will always be unique.

Additionally, your method is recursive and python has a max recursion depth, so maybe you should instead do this:

base_url = "http://www.readings.com.pk/"

def all_pages(base_url):

    response = requests.get(base_url)
    unique_urls = {base_url}
    visited_urls = set()
    while len(unique_urls) > len(visited_urls)
        soup = BeautifulSoup(response.text, "html.parser")

        for link in soup.find_all("a"):
            try:
                url = link["href"]
            except:
                continue
            absolute_url = base_url + url
            unique_urls.add(absolute_url)

        unvisited_url = (unique_urls - visited_urls).pop()
        visited_urls.add(unvisited_url)
        response = requests.get(unvisited_url)

    return unique_urls

all_pages(base_url)

How to scrape whole website using beautifulsoup

1 Answers1