Python Link Scraper

Question

    focus_Search = raw_input("Focus Search ") 
    url = "https://www.google.com/search?q=" 
    res = requests.get(url + focus_Search) 
    print("You Just Searched") 
    res_String = res.text 
    #Now I must get ALL the sections of code that start with "<a href" and end with "/a>"

Im trying to scrape all the links from a google search webpage. I could extract each link one at a time but I'm sure theres a better way to do it.

Use a html parser, there are countless examples on SO – Padraic Cunningham May 12 '16 at 21:37 — Padraic Cunningham, May 12 '16 at 21:37

score 0 · Answer 1 · answered May 12 '16 at 22:16

This creates a list of all links in the search page with some of your code, without getting into BeautifulSoup

import requests
import lxml.html

focus_Search = input("Focus Search ") 
url = "https://www.google.com/search?q=" 
#focus_Search
res = requests.get(url + focus_Search).content 
# res

dom = lxml.html.fromstring(res)
links = [x for x in dom.xpath('//a/@href')] # Borrows from cheekybastard in link below
# http://stackoverflow.com/questions/1080411/retrieve-links-from-web-page-using-python-and-beautifulsoup
links

Python Link Scraper

1 Answers1