Ruby parsing HTTPresponse with Nokogiri

Question

Parsing HTTPresponse with Nokogiri

Hi, I am having trouble parsing HTTPresponse objects with Nokogiri.

I use this function to fetch a website here:

fetch a link

def fetch(uri_str, limit = 10)
   
  
  # You should choose better exception.
  raise ArgumentError, 'HTTP redirect too deep' if limit == 0
  
  url = URI.parse(URI.encode(uri_str.strip))
  puts url
  
  #get path
  req = Net::HTTP::Get.new(url.path,headers)
  #start TCP/IP
  response = Net::HTTP.start(url.host,url.port) { |http|
        http.request(req)
  }
  case response
  when Net::HTTPSuccess
    then #print final redirect to a file
    puts "this is location" + uri_str
    puts "this is the host #{url.host}"
    puts "this is the path #{url.path}"
    
    return response
    # if you get a 302 response
  when Net::HTTPRedirection 
    then 
    puts "this is redirect" + response['location']
    return fetch(response['location'],aFile, limit - 1)
  else
    response.error!
  end
end




            html = fetch("http://www.somewebsite.com/hahaha/")
            puts html
            noko = Nokogiri::HTML(html)

When I do this html prints a whole bunch of gibberish and Nokogiri complains that "node_set must be a Nokogiri::XML::NOdeset

If anyone could offer help it would be quite appreciated

You should use mechanize instead of this hot mess. It takes care of redirects and deals with encodings for you. — pguardiario, Jul 05 '12 at 23:15

score 5 · Accepted Answer · edited May 23 '17 at 11:46

5

First thing. Your fetch method returns a Net::HTTPResponse object and not just the body. You should provide the body to Nokogiri.

response = fetch("http://www.somewebsite.com/hahaha/")
puts response.body
noko = Nokogiri::HTML(response.body)

I've updated your script so it's runnable (bellow). A couple of things were undefined.

require 'nokogiri'
require 'net/http'

def fetch(uri_str, limit = 10)
  # You should choose better exception.
  raise ArgumentError, 'HTTP redirect too deep' if limit == 0

  url = URI.parse(URI.encode(uri_str.strip))
  puts url

  #get path
  headers = {}
  req = Net::HTTP::Get.new(url.path,headers)
  #start TCP/IP
  response = Net::HTTP.start(url.host,url.port) { |http|
        http.request(req)
  }

  case response
  when Net::HTTPSuccess
    then #print final redirect to a file
    puts "this is location" + uri_str
    puts "this is the host #{url.host}"
    puts "this is the path #{url.path}"

    return response
    # if you get a 302 response
  when Net::HTTPRedirection
    then
    puts "this is redirect" + response['location']
    return fetch(response['location'], limit-1)
  else
    response.error!
  end
end

response = fetch("http://www.google.com/")
puts response
noko = Nokogiri::HTML(response.body)
puts noko

The script gives no error and prints the content. You may be getting Nokogiri error due to the content you're receiving. One common problem I've encountered with Nokogiri is character encoding. Without the exact error it's impossible to tell what's going on.

I'd recommnend looking at the following StackOverflow Questions

ruby 1.9: invalid byte sequence in UTF-8 (specifically this answer)

How to convert a Net::HTTP response to a certain encoding in Ruby 1.9.1?

edited May 23 '17 at 11:46

Community

1
1

answered Jul 05 '12 at 13:02

Pierre-Luc Simard

2,723
19
27

Thanks a lot Mr.Simard, I'll look up the character encoding. – Max Pie Jul 05 '12 at 16:02
How can I see a more verbose debug messages? The only error Nokogiri is giving me is this node_set has to be a Nokogiri::XML::Nodeset – Max Pie Jul 05 '12 at 17:54
Ok, I did script.rb -d and this is the output:YPscraper.rb:103: warning: (...) interpreted as grouped expression script.rb:118: warning: mismatched indentations at 'end' with 'begin' at 87 YPscraper.rb:119: warning: mismatched indentations at 'end' with 'for' at 83 Exception `LoadError' at C:/Ruby193/lib/ruby/site_ruby/1.9.1/rubygems/custom_req uire.rb:36 - cannot load such file -- nokogiriException `ArgumentError' at script.rb:102 - node_set must be a Nokogiri::XML ::NodeSet node_set must be a Nokogiri::XML::NodeSet – Max Pie Jul 05 '12 at 20:02
It really seem like Nokogiri does not like the content you receive. I'd suggest saving it to a file and inspecting it. – Pierre-Luc Simard Jul 05 '12 at 21:12
That's what I did. On the surface it seems like a perfectly normal html file, it opens in the Chrome normally , the file declares that its encoding is UTF-8... – Max Pie Jul 05 '12 at 21:30
Which version of Nokogiri are you using? which version of ruby? Can you disclose the URL that's causing the problem? – Pierre-Luc Simard Jul 06 '12 at 11:04
1

Hi, Pierre-Luc, I've resolved the problem yesterday basically I did something like this: page = Nokogiri::HTML(response.body) page.xpath("//div[(position()=1 and @class='listing_content')]/div/div/h3/a/text()") puts "this is name" puts name #Get Business phone phone = page.xpath("//div[@class='listing_content']//span[@class='business-phone phone']/text()") puts "this is phone" puts phone aFile.syswrite(noko + " " + phone) and apparently syswrite didn't like the attempt at concatenation Thanks for your help.If you have any more comments, that always welcome. – Max Pie Jul 06 '12 at 13:36

Ruby parsing HTTPresponse with Nokogiri

fetch a link

1 Answers1