Juixe TechKnow » Using Hpricot

Using Hpricot

Posted May 19, 2008.

Hpricot is a HTML parser for the Ruby programming language. With Hpricot you can scan and scape a HTML document. To illustrate how to use Hpricot i’ll write a list the code of a short script I recently wrote. The script grabs all the links for the past week from A Rubyist Railstastic Adventure, a tumblelog.

The general structure of the HTML used by the web page that I will be scraping is something like the following.

One thing to note about the HTML produced by the site we will scape is that the date is optional in the post. The date is only displayed once for a day, so some posts don’t have a given date. Also, there are several other types of posts such as quotes, images, etc. We are only interested in posts with links. Again, the Ruby/Hpricot script will only gather the links for the past week.

require 'rubygems'
require 'hpricot'
require 'open-uri'
require 'parsedate'

# Convert days to number to seconds
def days_to_sec(days) 
  secs = days.to_i
  secs *= 24
  secs *= 60
  secs *= 60
  secs
end

# pretty print the link in a list
def print_link(link)
  print "   <li>", 
    "<a href='#{link.attributes['href']}'>",
    "#{link.inner_html.strip}",
    "</a>",
    "</li>\n"
end

def get_links(doc)
  curr_date = Time.new
  (doc/"div.post").each do |post|
    post_date_elem = (post/"div.date/em")
    date = post_date_elem.inner_html.strip
    
    # Parse the date of the post
    if date != ""
      date_day = (post_date_elem/"big").text
      date_mon = nil
      date.each_line do |line| 
        date_mon = line.strip if date_mon.nil?
        break if date_mon.nil?
  	  end
      date_str = "#{date_mon} #{date_day}, #{$week_ago.year}"
  	  data = ParseDate.parsedate date_str
  	  curr_date = Time.local data[0], data[1], data[2]
    end
  
    # Stop if already looking past one week
    break if curr_date < $week_ago
  
    # Handle all links in post
    (post/"a.link").each do |link| 
      print_link link
    end
  end

if curr_date > $week_ago
    next_page = Hpricot(open($rubyist_next))
    get_links next_page
  end
end

# The Rubyist home page to be scraped
$rubyist_home = "http://rubyist.tumblr.com/"
$rubyist_next = "http://rubyist.tumblr.com/page/2"
# Scrape one weeks worth of links 
$week_ago = Time.new - days_to_sec(7)

# Run the script
doc = Hpricot(open($rubyist_home))
get_links doc

Technorati Tags: ruby, rubyist, hpricot, html, scape

The $100,000 Customer »

« This Week in Ruby

2 Comments so far

David Madden on May 23rd, 2008

I really love Hpricot. It’s a really useful library.

Good post; as a working example can only help in get more people using it.
Hakeem on May 26th, 2008

How do I submit button or click links with hpricot if I was setting a pluggable parser for mechanize?

Using Hpricot

2 Comments so far

Leave a reply

Categories

Monthly Archives

Del.icio.us