Using Hpricot « Juixe TechKnow

Using Hpricot

19
May3

Hpricot is a HTML parser for the Ruby programming language. With Hpricot you can scan and scape a HTML document. To illustrate how to use Hpricot i’ll write a list the code of a short script I recently wrote. The script grabs all the links for the past week from A Rubyist Railstastic Adventure, a tumblelog.

The general structure of the HTML used by the web page that I will be scraping is something like the following.

One thing to note about the HTML produced by the site we will scape is that the date is optional in the post. The date is only displayed once for a day, so some posts don’t have a given date. Also, there are several other types of posts such as quotes, images, etc. We are only interested in posts with links. Again, the Ruby/Hpricot script will only gather the links for the past week.

require 'rubygems'
require 'hpricot'
require 'open-uri'
require 'parsedate'

# Convert days to number to seconds
def days_to_sec(days) 
  secs = days.to_i
  secs *= 24
  secs *= 60
  secs *= 60
  secs
end

# pretty print the link in a list
def print_link(link)
  print "   <li>", 
    "<a href='#{link.attributes['href']}'>",
    "#{link.inner_html.strip}",
    "</a>",
    "</li>\n"
end

def get_links(doc)
  curr_date = Time.new
  (doc/"div.post").each do |post|
    post_date_elem = (post/"div.date/em")
    date = post_date_elem.inner_html.strip
    
    # Parse the date of the post
    if date != ""
      date_day = (post_date_elem/"big").text
      date_mon = nil
      date.each_line do |line| 
        date_mon = line.strip if date_mon.nil?
        break if date_mon.nil?
  	  end
      date_str = "#{date_mon} #{date_day}, #{$week_ago.year}"
  	  data = ParseDate.parsedate date_str
  	  curr_date = Time.local data[0], data[1], data[2]
    end
  
    # Stop if already looking past one week
    break if curr_date < $week_ago
  
    # Handle all links in post
    (post/"a.link").each do |link| 
      print_link link
    end
  end

if curr_date > $week_ago
    next_page = Hpricot(open($rubyist_next))
    get_links next_page
  end
end

# The Rubyist home page to be scraped
$rubyist_home = "http://rubyist.tumblr.com/"
$rubyist_next = "http://rubyist.tumblr.com/page/2"
# Scrape one weeks worth of links 
$week_ago = Time.new - days_to_sec(7)

# Run the script
doc = Hpricot(open($rubyist_home))
get_links doc

Technorati Tags: ruby, rubyist, hpricot, html, scape

Enjoy. Share. Be Happy.

Filed under: HTML/XML, Ruby, TechKnow

3 Comments

David Madden
3:44 am on May 23rd, 2008
I really love Hpricot. It’s a really useful library.

Good post; as a working example can only help in get more people using it.
Hakeem
6:04 am on May 26th, 2008
How do I submit button or click links with hpricot if I was setting a pluggable parser for mechanize?
cw
8:49 pm on October 19th, 2008
The line:

date_day = (post_date_elem/”big”).text

fails saying undefined ‘text’

Any suggestions?

Thanks

Juixe TechKnow knowing is half the battle

Using Hpricot

3 Comments

Leave a comment

Categories

Archive