A Simple Feed Aggregator

Credit: Rod Gaither

XML is the basis for many specialized langages. One of the most popular is RSS, an XML format often used to store lists of articles from web pages. With a tool called an aggregator, you can collect weblog entries and articles from several web sites RSS feeds, and read all those web sites at once without having to skip from one to the other. Here, well create a simple aggregator in Ruby.

Before aggregating RSS feeds, lets start by reading a single one. Fortunately we have several options for parsing RSS feeds into Ruby data structures. The Ruby standard library has built-in support for the three major versions of the RSS format (0.9, 1.0, and 2.0). This example uses the standard rss library to parse an RSS 2.0 feed and print out the titles of the items in the feed:

require ss/2.0 require open-uri url = http://www.oreillynet.com/pub/feed/1?format=rss2 feed = RSS::Parser.parse(open(url).read, false) puts "=== Channel: #{feed.channel.title} ===" feed.items.each do |item| puts item.title puts " (#{item.link})" puts puts item.description end # === Channel: OReilly Network Articles === # How to Make Your Sound Sing with Vocoders # (http://digitalmedia.oreilly.com/2006/03/29/vocoder-tutorial-and-tips.html) # …

Unfortunately, the standard rss library is a little out of date. Theres a newer syndication format called Atom, which serves the same purpose as RSS, and the rss library doesn support it. Any serious aggregator must support all the major syndication formats.

So instead, our aggregator will use Lucas Carlsons Simple RSS library, available as the simple-rss gem. This library supports the three main versions of RSS, plus Atom, and it does so in a relaxed way so that ill-formed feeds have a better chance of being read.

Heres the example above, rewritten to use Simple RSS. As you can see, only the name of the class is different:

require ubygems require simple-rss url = http://www.oreillynet.com/pub/feed/1?format=rss2 feed = RSS::Parser.parse(open(url), false) puts "=== Channel: #{feed.channel.title} ===" feed.items.each do |item| puts item.title puts " (#{item.link})" puts puts item.description end

Now we have a general method of reading a single RSS or Atom feed. Time to work on some aggregation!

Although the aggregator will be a simple Ruby script, theres no reason not to use Rubys object-oriented features. Our approach will be to create a class to encapsulate the aggregators data and behavior, and then write a sample program to use the class.

The RSSAggregator class that follows is a bare-bones aggregator that reads from multiple syndication feeds when instantiated. It uses a few simple methods to expose the data it has read.

#!/usr/bin/ruby # rss-aggregator.rb - Simple RSS and Atom Feed Aggregator require ubygems require simple-rss require open-uri class RSSAggregator def initialize(feed_urls) @feed_urls = feed_urls @feeds = [] read_feeds end protected def read_feeds @feed_urls.each { |url| @feeds.push(SimpleRSS.new(open(url).read)) } end public def refresh @feeds.clear read_feeds end def channel_counts @feeds.each_with_index do |feed, index| channel = "Channel(#{index.to_s}): #{feed.channel.title}" articles = "Articles: #{feed.items.size.to_s}" puts channel + , + articles end end def list_articles(id) puts "=== Channel(#{id.to_s}): #{@feeds[id].channel.title} ===" @feeds[id].items.each { |item| puts + item.title } end def list_all @feeds.each_with_index { |f, i| list_articles(i) } end end

Now we just need a few more lines of code to instantiate and use an RSSAggregator object:

test = RSSAggregator.new(ARGV) test.channel_counts puts " " test.list_all

Heres the output from a run of the test program against a few feed URLs:

$ ruby rss-aggregator.rb http://www.rubyriver.org/rss.xml http://rss.slashdot.org/Slashdot/slashdot http://www.oreillynet.com/pub/feed/1 http://safari.oreilly.com/rss/ Channel(0): RubyRiver, Articles: 20 Channel(1): Slashdot, Articles: 10 Channel(2): OReilly Network Articles, Articles: 15 Channel(3): OReilly Network Safari Bookshelf, Articles: 10 === Channel(0): RubyRiver === Mantis style isn eas… Its wonderful when tw… Red tailed hawk 37signals …

While a long way from a fully functional RSS aggregator, this program illustrates the basic requirements of any real aggregator. From this starting point, you can expand and refine the features of RSSAggregator.

One very important feature missing from the aggregator is support for the If-Modified-Since HTTP request header. When you call RSSAggregator#refresh, your aggregator downloads the specified feeds, even if it just grabbed the same feeds and none of them have changed since then. This wastes bandwidth.

Polite aggregators keep track of when they last grabbed a certain feed, and when they request it again they do a conditional request by supplying an HTTP request header called If-Modified Since. The details are a little beyond our scope, but basically the web server serves the reuqested feed only if it has changed since the last time the RSSAggregator downloaded it.

Another important feature our RSSAggregator is missing is the ability to store the articles it fetches. A real aggregator would store articles on disk or in a database to keep track of which stories are new since the last fetch, and to keep articles available even after they become old news and drop out of the feed.

Our simple aggregator counts the articles and lists their titles for review, but it doesn actually provide access to the article detail. As seen in the first example, the SimpleRSS.item has a link attribute containing the URL for the article, and a description attribute containing the (possibly HTML) body of the article. A real aggregator might generate a list of articles in HTML format for use in a browser, or convert the body of each article to text for output to a terminal.

See Also

Related

Категории