RailsTips - Home

MongoMapper, The Rad Mongo Wrapper

2009-06-27T16:59:00Z

In which I formally release MongoMapper, a high level wrapper similar to ActiveRecord, but for MongoDB.

A few weeks ago, I wrote about Mongo and how awesome it is. Towards the end of the article (and in the slideshow) I mentioned MongoMapper, a project I’ve been working on.

Over the past few weeks my buddies at Squeejee and Collective Idea have started using MongoMapper and they’ve helped me squash a few bugs and add a few features.

Despite the fact that I would call it far from finished, I’ve decided to release it in hopes that people can start playing with it, finding bugs, adding features and submitting pull requests. The documentation is sparse to none, but there are plenty of tests and the code is pretty readable, I believe.

Installation

# from rubyforge
sudo gem install mongomapper
# or from github
sudo gem install jnunemaker-mongomapper

Usage

So how do you use this thing? It’s pretty simple. MongoMapper uses a default connection from the Ruby driver. This means if you are using Mongo on the standard port and localhost, you don’t have to give it connection information. If you aren’t, you can do it like this:

MongoMapper.connection = XGen::Mongo::Driver::Mongo.new('hostname')

Connection accepts any valid Mongo ruby driver connection. The only other setup you need to do is to tell MongoMapper what the default database is. This is pretty much the same as setting up the connection:

MongoMapper.database = 'mydatabasename'

These two operations only define the default connection and database information. Both of these can be overridden on a per model basis so that you can hook up to multiple databases on different servers.

Include Instead of Inherit

To create a new model, I went with the include pattern, instead of inheritance. In ActiveRecord, you would define a new model like this:

class Person < ActiveRecord::Base
end

In MongoMapper, you would do the following:

class Person
  include MongoMapper::Document
end

Just like ActiveRecord, this makes assumptions. It assumes you have a collection named people. Oh, and the good news is you don’t need a migration for it. The first time you try to create a person document, the collection will be created automatically. Heck yeah! I mentioned that you can override the default connection and database on a per document level. If you need to do that, it would look like this:

class Person
  include MongoMapper::Document

  connection XGen::Mongo::Driver::Mongo.new('hostname')
  database 'otherdatabase'
end

Defining Keys

Each document is made up of keys. Keys are named and type-casted so you know your data is stored in the correct format. Lets fill out our Person document a bit.

class Person
  include MongoMapper::Document

  key :first_name, String
  key :last_name, String
  key :age, Integer
  key :born_at, Time
  key :active, Boolean
  key :fav_colors, Array
end

Now that we have defined our schema, we can create, update and delete documents.

person = Person.create({
  :first_name => 'John',
  :last_name => 'Nunemaker',
  :age => 27,
  :born_at => Time.mktime(1981, 11, 25, 2, 30),
  :active => true,
  :fav_colors => %w(red green blue)
})

person.first_name = 'Johnny'
person.save

person.destroy
# or you could do this to destroy
Person.destroy(person.id)

Looks pretty familiar, eh? Where it made sense, I tried to stay close to ActiveRecord in API.

Validations

But wait you say, how do I validate my data? Well, you can do it pretty much the same way as ActiveRecord.

class Person
  include MongoMapper::Document

  key :first_name, String
  key :last_name, String
  key :age, Integer
  key :born_at, Time
  key :active, Boolean
  key :fav_colors, Array

  validates_presence_of :first_name
  validates_presence_of :last_name
  validates_numericality_of :age
  # etc, etc
end

But, if you find that a bit tedious as I do, you can use some shortcuts that I’ve added in.

class Person
  include MongoMapper::Document

  key :first_name, String, :required => true
  key :last_name, String, :required => true
  key :age, Integer, :numeric => true
  key :born_at, Time
  key :active, Boolean
  key :fav_colors, Array
end

Most of the validations from Rails are supported. I still need to build in support for validates_uniqueness of and some of the options that rails supports might not be right now, but it is a good first pass.

Callbacks

Did you hear that? I swear I just heard someone whisper about callbacks. Umm, yeah, we got that too. The good news? I just used ActiveSupport’s callbacks so they are identical to Rails and most of Rails defined callbacks are supported such as before_save and the like.

Embedded Documents

So the cool thing about Mongo is that you can embed documents in other documents. Let’s say our person has multiple addresses. To handle that, we would create an embedded address document to go along with our person document.

class Address
  include MongoMapper::EmbeddedDocument

  key :address, String
  key :city,    String
  key :state,   String
  key :zip,     Integer
end

class Person
  include MongoMapper::Document

  many :addresses
end

Now we can add addresses to the person like so:

person = Person.new
person.addresses << Address.new(:city => 'South Bend', :state => 'IN')
person.addresses << Address.new(:city => 'Chicago', :state => 'IL')
person.save

Doing this actually saves the address right inside the person document. Yep, no joins. Yay! Cheers resound from the heavens! You can even query for documents based on these embedded documents. For example, if you wanted to find all people that are in the city Chicago, you could do this:

Person.all(:conditions => {'addresses.city' => 'Chicago'})

Finding Documents

The find API is very similar to AR as well. Below are a bunch of other examples:

Person.find(1)
Person.find(1,2,3,4)
Person.find(:first)
Person.first
Person.find(:last)
Person.last
Person.find(:all)
Person.all
Person.all(:conditions => {:last_name => 'Nunemaker'}, :order => 'first_name')

For more information about how to provide criteria to find, you can see the stuff covered in the finder options. If you need to, you can even throw custom mongo stuff into the mix and it just gets passed through to the mongo ruby driver (ie: $gt, $gte, $lt, $lte, etc.).

Conclusion

We take ActiveRecord for granted. It really has a lot of handy features and does a pretty good job at modeling our applications. I never realized how much it does, until I decided to create MongoMapper. That said, the experience has been fun thus far and I’m excited to see what people use it for.

There is a ton more I could talk about, but frankly, this article is long enough. Rest assured that I think Mongo is cool and that MongoMapper is headed in the right direction, but far from complete. I haven’t actually built anything with MongoMapper yet, but I will be soon. I’m sure that will lead to a lot of handy new features.

Any general discussion can happen in the comments below while they are open or over at the google group. If you find a bug or have a feature idea, create an issue at github.

JSONQuerying Your Rails Responses

2009-06-22T22:44:00Z

In which I show how to use a Ruby implementation of JSONQuery to test JSON in Rails apps.

I’m writing an application right now that is really JSON heavy. Some of the functional tests are cucumber and some of them are just rails functional tests using shoulda.

I hit a point today where I wanted to verify that the JSON getting output was generally what I want. I could have just JSON parsed the response body and compared that with what I was looking for, but a little part of me thought this might be a cool application of JSONQuery.

JSONQuery provides a comprehensive set of data querying tools including filtering, recursive search, sorting, mapping, range selection, and flexible expressions with wildcard string comparisons and various operators.

The quote above is fancy and can be boiled down to “a query language for JSON”. If you want to read more about JSONPath and JSONQuery here are some posts:

Finding a Ruby JSONQuery Implementation

I knew there was a JavaScript implementation of JSONQuery and that Jon Crosby has been doing some cool stuff with it in CloudKit, but I couldn’t find a Ruby implemenation that didn’t require johnson.

After some googling and Github searching, I came across Siren. Siren was pretty much what I wanted, so I started playing around with it. I forked it, gem’d it and wrapped it with some shoulda goodness.

What I ended up with was pretty specific to my needs at the moment, but I post it here in hopes that it sparks some ideas.

Bringing It All Together

First, I added the following to my environments/test.rb file.

config.gem 'jnunemaker-siren',
            :lib     => 'siren',
            :version => '0.1.1',
            :source  => 'http://gems.github.com'

Then I added the following in my test helper (actually put it in separate module and file and included it but I’m going for simplicity in this post).

class ActiveSupport::TestCase
  def self.should_query_json(expression, string_or_regex)
    should "have json response matching #{expression}" do
      assert_jsonquery expression, string_or_regex
    end
  end

  def assert_jsonquery(expression, string_or_regex)
    json = ActiveSupport::JSON.decode(@response.body)
    query = Siren.query(expression, json)

    if string_or_regex.kind_of?(Regexp)
      assert_match string_or_regex, query, "JSONQuery expression #{expression} value did not match regex" 
    else
      assert_equal string_or_regex, query, "Expression #{expression} value #{query} did not equal #{string_or_regex}" 
    end
  end
end

The code is quick and dirty. The first thing you’ll notice is that assert_jsonquery actually uses @response.body, which means it can only be used in a controller test. I could easily expand it, but, like I said above, I just got it working for what I needed right now. The cool part is that now in my functional tests, I can do stuff like this:

context "on POST to :create" do
  setup { post :create, :status => {'action' => 'In', 'body' => 'Working on PB' }

  # ... code removed for brevity ...
  should_query_json "$['replace']['#current_status']", /Working on PB/
end

That is a really basic query. Trust me you can do a heck of a lot more. Check out the Siren tests if you don’t believe me.

Conclusion

Overall, working with siren was a little rough because I wasn’t familiar with JSONQuery syntax. Also, siren tends to return nil instead of a more helpful error message about my expression compilation failing, but I’m kind of excited to see how this works out in the long run.

What are you doing to test JSON in your apps? Does something like this seem cool or overkill? Just kind of curious.

What Is The Simplest Thing That Could Possibly Work?

2009-06-08T18:31:00Z

In which I summarize my favorite points from an old, but awesome article.

I am always amazed when I read an article from 2004 and find interesting goodies. I’m probably late to the game on a lot of these articles, as I didn’t really dive into programming as a career until 2005, but I just read The Simplest Thing that Could Possibly Work, a conversation with Ward Cunningham by Bill Venners. The article was published on January 19, 2004, but it is truly timeless.

The Shortest Path

Simplicity is the shortest path to a solution.

“Shortest” doesn’t necessarily refer to lines of code or number of characters, but I see it more as the path that requires the least amount of complexity. As he mentions in the article, if someone releases a 20 page proof to a math problem and then later on, someone releases a 10 page proof for the same problem, the 10 page proof is not necessarily more simple.

The 10 page proof could use some form of mathematics that is not widely used in the community and takes some time to comprehend. This means the 10 page version could be less simple as it requires learning to understand, whereas the 20 page uses generally understood concepts.

I think this is a balance that we always fight with as programmers. What is simple? I can usually say simple or not simple when I look at code, but it is hard to define the rules for simplicity.

Work Today Makes You Better Tomorrow

The effort you expend today to understand the code will make you a more powerful programmer tomorrow.

This is one of the concepts that has made the biggest different in my programming knowledge over the past few years. The first time that I really did this was when I wrote about class and instance variables a few years back. Ever since then, when I come across something that I don’t understand, that I feel I should, I spend the time to understand it. I have grown immensely because of this and would recommend that you do the same if you aren’t already.

Narrow What You Think About

We had been thinking about too much at once, trying to achieve too complicated a goal, trying to code it too well.

This is something that I have been practicing a lot lately. You know how sometimes you just feel overwhelmed and don’t want to start a feature or project? What I’ve found is that when I feel this way it is because I’m trying to think about too much at once.

Ward encourages over and over in the article, think about what is the most simple possible thing that could work. Notice he did not say what is the simplest thing that would work, but rather what could work.

This is something that I’ve noticed recently while pairing with Brandon Keepers. Both of us almost apologize for some of the code we first implement, as we are afraid the other will think that is all we are capable of. What is funny, is that we both realize that you have to start with something and thus never judge. It is far easier to incrementally work towards a brilliant solution than to think it in your head and instantly code it.

Start with a test. Make the test pass. Rinse and repeat. Small, tested changes that solve only the immediate problem at hand always end up with a more simple solution than trying to do it all in one fell swoop. I’ve also found I’m more productive this way as I have less moments of wondering what to do next. The failing test tells me.

Anyway, I thought the article was interesting enough that I would post some of the highlights here and encourage you all to read it. If you know of some oldie, but goodie articles, link them up in the comments below.

What If A Key/Value Store Mated With A Relational Database System?

2009-06-03T16:22:00Z

In which I provide an intro to MongoDB and it’s awesomeness.

Last night, the folks from the Grand Rapids ruby group were kind enough to allow me to present on MongoDB. The talk went great. I’ve been excited about Mongo for a couple weeks now, so it was cool to see that it wasn’t just me.

The funny thing is, at nearly the same time, Wynn Netherland presented on MongoDB to the Dallas ruby group. We discovered that he wrote part 1 and I wrote part 2 of the presentation despite not working together on it so we ended up showing each other’s slides as well.

I figured since I spent the time to throw some slides together, I might as well put an intro up here too. First, the slides (they probably won’t mean a lot as they were mostly outlines for me to speak from).

Intro to MongoDB

Ok, so what the crap is Mongo? I find the best way to describe Mongo is the best features of key/values stores, document databases and RDBMS in one. No way, you say. That sounds perfect. Well, Mongo is not perfect, but I think it brings something kind of new to the database table.

Mongo is built for speed. Anything that would slow it down (aka transactions) have been left on the chopping block. Instead of REST, they chose sockets and have written drivers for several languages (of course one for Ruby).

Collections

It is collection/document oriented. Collections are like tables in MySQL (they are even grouped in databases) and serve the purpose of breaking up the top level entities in your application (User, Article, Account, etc.) by type and thus into smaller query sets, to make queries faster.

Documents

Inside of each collection, you store documents. Documents are basically objects that have no schema. The lack of schema may be scary to some, but I look at it this way. You have to know your application schema at the app level, so why put the schema in the database and in your app. Why not just put the schema in your app and have the database store whatever you put in it? This way, you database schema is kind of versioned with your application code. I think that is pretty cool.

Documents are stored in BSON (blog post), which is binary encoded JSON that is built to be more efficient and also to include a few more data types than JSON. This means that if you send Mongo a document that has values of different types, such as String, Integer, Date, Array, Hash, etc., Mongo knows exactly how to deal with those types and actually stores them in the database as that type. This differs from traditional key/value stores, which just give you a key and a string value and leave you to handle serialization yourself.

Object Relationships

There are two ways to relate documents in Mongo. The first, is to simply embed a document into another document. An example of this would be tags embedded in article. Let’s take a look.

{
  title: 'Mongolicious', 
  body: 'I could teach you, but I would have to charge...', 
  tags: ['mongo', 'databases', 'awesome']
}

As you can see, tags are just a key in the article document. The benefits of this are that you never have to do any joins when you show the article and it’s tags as they are all stored in the same place. The other cool thing is that Mongo can index the tags and understand indexing keys that have multiple values (such as arrays and hashes). This means if you index tags, you can find all documents tagged with ‘foo’ and it will be performant. Embedded documents work great for some things, but other things wouldn’t make sense embedded.

Let’s imagine that you have an client document and you want the client to have multiple contacts. If you embedded the contacts for the client with it in a document, it would be inefficient to have a page that listed all the contacts. To have a contact list, you would have to pull out every client and collect all the contacts and then sort them. Also, if a contact should be associated with multiple clients, you would have to duplicate their information for each client.

In SQL, you would have a clients table and a contacts table and then a join model between them so that any contact would be in the system once and could be associated with one or more clients without duplicate. So how would you do this in Mongo? The same way…kind of.

In Mongo, you’d have a client collection and a contact collection. To associate a contact to a client, you just create a db reference to to the contact from the client.

Dynamic Queries

Yep, Mongo has dynamic queries. It actually has a kind of quirky, yet lovable syntax for defining criteria. Below are a few examples from my presentation which are mostly self-explanatory. These are examples of what you would run in Mongo’s JavaScript shell.

# finds all Johns
db.collection.find({‘first_name’: ‘John’})

# finds all documents with first_name 
# starting with J using a regex
db.collection.find({‘first_name’: /^J/}) 

# finds first with _id of 1
db.collection.find_first({‘_id’:1})

# finds possible drinkers (age > 21)
db.collection.find({‘age’: {‘$gt’: 21}})

# searches in embedded document author for 
# author that has first name of John
db.collection.find({‘author.first_name’:‘John’})

# worse case scenario, or if you need "or" 
# queries you can drop down to JavaScript
db.collection.find({$where:‘this.age >= 6 && this.age <= 18’})

You can also sort by one or more keys, limit the number of results, offset a number of results (for pagination), and define which keys you want to select. The other thing that is slick is Mongo supports count and group. Count is the same idea as MySQL’s count. It returns the number of documents that match provided criteria. Group is the same concept, but is accomplished with map/reduce.

To really get a feel for all that you can do with queries, check out Mongo’s advanced query documentation.

Random Awesomeness

Capped collections (blog post): Think memcache. You can set a limit for a collection to a certain number of documents or size of space. When the number or size goes over limit the old document gets pushed out. For more info, see MongoDB and Caching
Upserts: Think find or create in one call. You provide criteria and the document details and Mongo determines if the document exists or not and either inserts or updates it. You can also do special things like incrementers with $inc. For more, read Using mongo for real time analytics
Multikeys: for indexing arrays of keys. Think tagging.
GridFS and auto-sharding: Storing files in the database in a way that doesn’t suck. They have mentioned in IRC that they might even make Apache/Nginx modules that server files straight from GridFS so requests can go straight from web server to Mongo instead of traveling through your app server. For more, read You don’t need a file system

How do I use it with Ruby?

If you have made it this far, you are probably intrigued and are wondering how you can use Mongo with Ruby. There is an official mongo-ruby-driver on GitHub for starters. It supports most of Mongo’s features, if not all, and gets the job done, but it is really low level. It would be like writing an application using the MySQL gem. You can do it, but it won’t be fun. I’ve even started giving back to the driver.

There are two “ORM’s” for Mongo and both are on GitHub. The first is an ActiveRecord adapter and the second is MongoRecord. I took a look at both of these, and decided to write my own. Why?

Mongo is not a RDBMS (like MySQL) so why use RDBMS wrappers (like the AR adapter)?
I think the DSL for modeling your application should teach you Mongo.
Mongo is perfect for the website management system I’m building and I just didn’t like the other wrappers. Why would I want to build something with something that I didn’t like?
It sounded fun!

MongoMapper

I started the Friday of Memorial weekend and was able to crank out most of the functionality. Since then, I’ve been working on it whenever I get time and it is really close to being ready for a first release. That said, it is not public yet. Don’t worry, as soon as it is ready for prime time, I’ll be posting more here. So what features does MongoMapper have built in?

Typecasting
Callbacks (uses ActiveSupport callbacks)
Validations (uses my fork of validatable)
Connection and database can differ per document
Create, update, delete, delete_all, destroy, destroy_all that work just like ActiveRecord
Find with id, multiple ids, :all, :first, :last, etc. Also supports Mongo specific find critieria like $gt, $lt, $in, $nin, etc.
Associations
Drop in Rails compatibility

So out of the features listed above, all are complete but the last two at the time of this post. I’m currently working through associations and then I’m going to start making a Rails app with MongoMapper to figure out what I need for “drop in and forget” Rails compatibility. I have a few other smart people helping me so my guess is that it will be out in the next two weeks.

Let me know with a comment below what you like and don’t like about Mongo. I’m very curious what other Rails developers think after reading this intro and the articles I’ve linked to. I’m stoked, but I’m sure it is not for everyone.

Swine Flu and the Twitter Gem

2009-05-18T22:11:00Z

In which I wax poetic about the trendy new addition to the Twitter gem.

I had some extra time today and I’ve been spotty on open source work over the past few weeks, so I decided to add support for the Twitter trends API to my Twitter gem.

Using HTTParty, the code for this turned out to be insanely simple, so short, in fact, that I’ll just put it inline here so you don’t even have to go over to Github. Aww, I’m so nice.

module Twitter
  class Trends
    include HTTParty
    base_uri 'search.twitter.com/trends'
    format :json

    # :exclude => 'hashtags' to exclude hashtags
    def self.current(options={})
      mashup(get('/current.json', :query => options))
    end

    # :exclude => 'hashtags' to exclude hashtags
    # :date => yyyy-mm-dd for specific date
    def self.daily(options={})
      mashup(get('/daily.json', :query => options))
    end

    # :exclude => 'hashtags' to exclude hashtags
    # :date => yyyy-mm-dd for specific date
    def self.weekly(options={})
      mashup(get('/weekly.json', :query => options))
    end

    private
      def self.mashup(response)
        response['trends'].values.flatten.map { |t| Mash.new(t) }
      end
  end
end

Pure TDD

I am most definitely a tester, but I’ll admit I usually write code and then write the test. Of late, I’ve been reversing this trend and actually practicing TDD in full force by writing a small test, then only enough code to make it pass, followed by another test or more code for the existing test, finished with just enough code to make the new addition pass.

It is a different mindset to code in this way, compared to my code first and then make sure my butt is covered method and I’ve loving it. I thought I would find pure TDD tedious, but on the contrary, I think I’m coding faster and cleaner.

The Tests

So how did I test the code above? Again, inline for your viewing pleasure, are the tests I added to make sure I don’t break something in the future and get yelled at. Feel free to take a gander and I’ll meet back up with you at the bottom of it.

require File.dirname(__FILE__) + '/../test_helper'

class TrendsTest < Test::Unit::TestCase
  include Twitter

  context "Getting current trends" do
    should "work" do
      stub_get('http://search.twitter.com:80/trends/current.json', 'trends_current.json')
      trends = Trends.current
      trends.size.should == 10
      trends[0].name.should == '#musicmonday'
      trends[0].query.should == '#musicmonday'
      trends[1].name.should == '#newdivide'
      trends[1].query.should == '#newdivide'
    end

    should "be able to exclude hashtags" do
      stub_get('http://search.twitter.com:80/trends/current.json?exclude=hashtags', 'trends_current_exclude.json')
      trends = Trends.current(:exclude => 'hashtags')
      trends.size.should == 10
      trends[0].name.should == 'New Divide'
      trends[0].query.should == %Q(\"New Divide\")
      trends[1].name.should == 'Star Trek'
      trends[1].query.should == %Q(\"Star Trek\")
    end
  end

  context "Getting daily trends" do
    should "work" do
      stub_get('http://search.twitter.com:80/trends/daily.json?', 'trends_daily.json')
      trends = Trends.daily
      trends.size.should == 480
      trends[0].name.should == '#3turnoffwords'
      trends[0].query.should == '#3turnoffwords'
    end

    should "be able to exclude hastags" do
      stub_get('http://search.twitter.com:80/trends/daily.json?exclude=hashtags', 'trends_daily_exclude.json')
      trends = Trends.daily(:exclude => 'hashtags')
      trends.size.should == 480
      trends[0].name.should == 'Star Trek'
      trends[0].query.should == %Q(\"Star Trek\")
    end

    should "be able to get for specific date (with date string)" do
      stub_get 'http://search.twitter.com:80/trends/daily.json?date=2009-05-01', 'trends_daily_date.json'
      trends = Trends.daily(:date => '2009-05-01')
      trends.size.should == 440
      trends[0].name.should == 'Swine Flu'
      trends[0].query.should == %Q(\"Swine Flu\")
    end

    should "be able to get for specific date (with date object)" do
      stub_get 'http://search.twitter.com:80/trends/daily.json?date=2009-05-01', 'trends_daily_date.json'
      trends = Trends.daily(:date => Date.new(2009, 5, 1))
      trends.size.should == 440
      trends[0].name.should == 'Swine Flu'
      trends[0].query.should == %Q(\"Swine Flu\")
    end
  end

  context "Getting weekly trends" do
    should "work" do
      stub_get('http://search.twitter.com:80/trends/weekly.json?', 'trends_weekly.json')
      trends = Trends.weekly
      trends.size.should == 210
      trends[0].name.should == 'Happy Mothers Day'
      trends[0].query.should == %Q(\"Happy Mothers Day\" OR \"Mothers Day\")
    end

    should "be able to exclude hastags" do
      stub_get('http://search.twitter.com:80/trends/weekly.json?exclude=hashtags', 'trends_weekly_exclude.json')
      trends = Trends.weekly(:exclude => 'hashtags')
      trends.size.should == 210
      trends[0].name.should == 'Happy Mothers Day'
      trends[0].query.should == %Q(\"Happy Mothers Day\" OR \"Mothers Day\")
    end

    should "be able to get for specific date (with date string)" do
      stub_get 'http://search.twitter.com:80/trends/weekly.json?date=2009-05-01', 'trends_weekly_date.json'
      trends = Trends.weekly(:date => '2009-05-01')
      trends.size.should == 210
      trends[0].name.should == 'TGIF'
      trends[0].query.should == 'TGIF'
    end

    should "be able to get for specific date (with date object)" do
      stub_get 'http://search.twitter.com:80/trends/weekly.json?date=2009-05-01', 'trends_weekly_date.json'
      trends = Trends.weekly(:date => Date.new(2009, 5, 1))
      trends.size.should == 210
      trends[0].name.should == 'TGIF'
      trends[0].query.should == 'TGIF'
    end
  end
end

So, yeah, nothing earth shattering. It feels a bit repetitive, but I don’t mind some amount of repetition in my tests. The fixture files were created quite simply using curl.

cd test/fixtures
curl http://search.twitter.com:80/trends/weekly.json?date=2009-05-01 > trends_weekly_date.json
# rinse and repeat for each file

The stub_get method is a simple wrapper around FakeWeb and looks something like this:

def stub_get(url, filename, status=nil)
  options = {:string => fixture_file(filename)}
  options.merge!({:status => status}) unless status.nil?
  FakeWeb.register_uri(:get, url, options)
end

def fixture_file(filename)
  file_path = File.expand_path(File.dirname(__FILE__) + '/fixtures/' + filename)
  File.read(file_path)
end

I’m lazy and find that stub_get is much shorter than FakeWeb.register_uri blah, blah, blah. The tests use FakeWeb, shoulda and my fork of matchy, in case you are curious.

Example Uses

So what can you do with the new trends addition? Below are some examples of how you can obtain trend information.

Twitter::Trends.current
Twitter::Trends.current(:exclude => 'hashtags')

Twitter::Trends.daily # current day
Twitter::Trends.daily(:exclude => 'hashtags')
Twitter::Trends.daily(:date => Date.new(2009, 5, 1))

Twitter::Trends.weekly # current day
Twitter::Trends.weekly(:exclude => 'hashtags')
Twitter::Trends.weekly(:date => Date.new(2009, 5, 1))

That’s all for now. Enjoy the new trends and build something cool. Oh, and if you want to play with trends, but don’t have an idea, I have one and most likely won’t have time to build it. I’d be happy to collaborate.

Include vs Extend in Ruby

2009-05-15T05:13:00Z

In which I show the difference between include and extend when working with modules in Ruby.

Now that we know the difference between an instance method and a class method, let’s cover the difference between include and extend in regards to modules. Include is for adding methods to an instance of a class and extend is for adding class methods. Let’s take a look at a small example.

module Foo
  def foo
    puts 'heyyyyoooo!'
  end
end

class Bar
  include Foo
end

Bar.new.foo # heyyyyoooo!
Bar.foo # NoMethodError: undefined method ‘foo’ for Bar:Class

class Baz
  extend Foo
end

Baz.foo # heyyyyoooo!
Baz.new.foo # NoMethodError: undefined method ‘foo’ for #<Baz:0x1e708>

As you can see, include makes the foo method available to an instance of a class and extend makes the foo method available to the class itself.

Include Example

If you want to see more examples of using include to share methods among models, you can read my article on how I added simple permissions to an app. The permissions module in that article is then included in a few models thus sharing the methods in it. That is all I’ll say here, so if you want to see more check out that article.

Extend Example

I’ve also got a simple example of using extend that I’ve plucked from the Twitter gem. Basically, Twitter supports two methods for authentication—httpauth and oauth. In order to share the maximum amount of code when using these two different authentication methods, I use a lot of delegation. Basically, the Twitter::Base class takes an instance of a “client”. A client is an instance of either Twitter::HTTPAuth or Twitter::OAuth.

Anytime a request is made from the Twitter::Base object, either get or post is called on the client. The Twitter::HTTPAuth client defines the get and post methods, but the Twitter::OAuth client does not. Twitter::OAuth is just a thin wrapper around the OAuth gem and the OAuth gem actually provides get and post methods on the access token, which automatically handles passing the OAuth information around with each request.

The implementation looks something like this (full file on github):

module Twitter
  class OAuth
    extend Forwardable
    def_delegators :access_token, :get, :post

    # a bunch of code removed for clarity
  end
end

Rather than define get and post, I simply delegate the get and post instance methods to the access token, which already has them defined. I do that by extending the Forwardable module onto the Twitter::OAuth class and then using the def_delegators class method that it provides. This may not be the most clear example, but it was the first that came to mind so I hope it is understandable.

A Common Idiom

Even though include is for adding instance methods, a common idiom you’ll see in Ruby is to use include to append both class and instance methods. The reason for this is that include has a self.included hook you can use to modify the class that is including a module and, to my knowledge, extend does not have a hook. It’s highly debatable, but often used so I figured I would mention it. Let’s look at an example.

module Foo
  def self.included(base)
    base.extend(ClassMethods)
  end

  module ClassMethods
    def bar
      puts 'class method'
    end
  end

  def foo
    puts 'instance method'
  end
end

class Baz
  include Foo
end

Baz.bar # class method
Baz.new.foo # instance method
Baz.foo # NoMethodError: undefined method ‘foo’ for Baz:Class
Baz.new.bar # NoMethodError: undefined method ‘bar’ for #<Baz:0x1e3d4>

There are a ton of projects that use this idiom, including Rails, DataMapper, HTTParty, and HappyMapper. For example, when you use HTTParty, you do something like this.

class FetchyMcfetcherson
  include HTTParty
end

FetchyMcfetcherson.get('http://foobar.com')

When you add the include to your class, HTTParty appends class methods, such as get, post, put, delete, base_uri, default_options and format. I think this idiom is what causes a lot of confusion in the include verse extend understanding. Because you are using include it seems like the HTTParty methods would be added to an instance of the FetchyMcfetcherson class, but they are actually added to the class itself.

Conclusion

Hope this helps those struggling with include verse extend. Use include for instance methods and extend for class methods. Also, it is sometimes ok to use include to add both instance and class methods. Both are really handy and allow for a great amount of code reuse. They also allow you to avoid deep inheritance, and instead just modularize code and include it where needed, which is much more the ruby way.

Class and Instance Methods in Ruby

2009-05-11T19:03:00Z

In which I show the difference between a class method and an instance method.

The other day I was explaining the difference between class and instance methods to a friend and I realized that I should probably write up a post. I figured since I’m on a plane headed back home, now was as good of time as ever. If you want a little history, you can read about the difference between class and instance variables.

One Line Summary

Class methods are methods that are called on a class and instance methods are methods that are called on an instance of a class. Here is a quick example and then we’ll go into a bit more detail.

class Foo
  def self.bar
    puts 'class method'
  end

  def baz
    puts 'instance method'
  end
end

Foo.bar # => "class method" 
Foo.baz # => NoMethodError: undefined method ‘baz’ for Foo:Class

Foo.new.baz # => instance method
Foo.new.bar # => NoMethodError: undefined method ‘bar’ for #<Foo:0x1e820>

See the difference? bar is a class method, so calling bar on the Foo class works fine. baz is an instance method, so calling baz on the Foo class raises a NoMethodError. Then, on the next couple lines, we call both methods on an instance of Foo (Foo.new).

Now that we have a base, let’s show a few of the ways you can create class and instance methods and examples of what they would be used for.

Class Methods

Ruby is very flexible and as such it allows several ways to define a class method. The following is a sample of the most commonly used ways.

# Way 1
class Foo
  def self.bar
    puts 'class method'
  end
end

Foo.bar # "class method" 

# Way 2
class Foo
  class << self
    def bar
      puts 'class method'
    end
  end
end

Foo.bar # "class method" 

# Way 3
class Foo; end
def Foo.bar
  puts 'class method'
end

Foo.bar # "class method"

The first way is my preference. When I see self.method_name it is immediately apparent to me that this is a class method. A lot of people use way #2 and it is pretty heavily used in Rails.

There is nothing wrong with it, but when you have a class with a lot of class methods in a class << self block, it can be hard to tell if the method is a class or instance method because it is defined the same (def bar). If this doesn’t make sense, feel free to use it for a while and you’ll probably run into what I’m talking about.

Way 3 is not that common as far as I have seen and is more often a way to quickly add methods on the fly to a class. These are not the only three ways to define class methods, but they seem to be the ones that I see the most.

So when would you use a class method? Class methods are for anything that does not deal with an individual instance of a class. ActiveRecord::Base#find is one example. If you look in ActiveRecord::Base, you’ll see something like this:

module ActiveRecord
  class Base
    # some stuff
    class << self
      def find(...)
        # blah
      end
    end
  end
end

Looks familiar, eh? Some other uses of class methods in Rails are validations and associations in ActiveRecord and before/after/around filters in ActionPack. The way this works is something like this (simplified for clarity):

module ActiveRecord
  class Base
    def self.validates_presence_of(...)
      # make sure present
    end
  end
end

class Foo < ActiveRecord::Base
  validates_presence_of :bar
end

When you say validates_presence_of, the class method in AR::Base is what gets called.

Instance Methods

Enough about class methods, lets move on. Instance methods are a bit more simple. Here are a few common ways that instance methods are defined.

# Way 1
class Foo
  def baz
    puts 'instance method'
  end
end

Foo.new.baz # "instance method" 

# Way 2
class Foo
  attr_accessor :baz
end

foo = Foo.new
foo.baz = 'instance method'
puts foo.baz

# Way 3
class Foo; end

foo = Foo.new
def foo.bar
  puts 'instance method'
end

Foo.new.baz # "instance method"

The key difference is instance methods only work with an instance and thus you have to create a new instance to use them (Foo.new). Again, there are more ways to define instance methods than this, especially if you look into meta programming.

So what are some examples uses of instance methods in Rails, to give you a better idea? Ever do a find in a destroy action and then call destroy on the found instance? destroy is an instance method.

class FoosController < ActionController
  def destroy
    foo = Foo.find(params[:id])
    foo.destroy
    redirect_to foos_url
  end
end

So are save and update_attributes, which you have definitely used before if you’ve done any Rails.

foo = Foo.new(:title => 'Bar')
foo.save # is an instance method

Conclusion

Class methods can only be called on classes and instance methods can only be called on an instance of a class. It’s simple when you understand it, but I remember being confused when I was learning Ruby. Hope this helps. If I was unclear or incorrect at any point above, let me know.

The next article that I also wrote on the plane ride home from Railsconf is Include verse Extend, which builds on this article.

RailsConf 2009

2009-05-03T20:33:00Z

In which I note that I’ll be at RailsConf 2009.

This week is RailsConf and I decided kind of last minute to attend. Steve, the other half of Ordered List, will also be there, though mostly just to hang out.

My first RailsConf was in 2007 and I blogged my little heart out (summary). Last year, I took a different approach and just posted a summary of the whole thing.

This year I’m going to take a different approach than the previous two. I’m planning on mostly just walking around with a moleskin and my new camera. I found the moleskin approach worked pretty well for the sessions I used it with at RubyConf last year, so I’m going to give it a shot for a whole conference. I took better notes and was less distracted than if I used my computer.

Say Hi!

If you see me and I don’t see you, be sure to say hi. Each conference I meet more and more people and I want this one to be no different. Meeting people really is the best part of any conference. If you don’t know what I look like, here is a picture of me enjoying a blueberry dessert a la mode. I’m guessing from this you can figure out what I look like without the blueberry dessert. :)

Don’t worry, though I look angry in the picture, I am actually blissfully happy. Also, I spend about 95% of the time with a smile on my face.

Now With Navigation and Charted Archives

2009-04-24T02:57:00Z

In which I discuss some recent updates to the site.

Every couple weeks, I sit down and try to think of things that will make this site better. Rather than post each time I come up with something, I typically wait until I have a few to share and then post about it. This is another one of those posts.

Charted Archives

A few weeks ago, I thought it would be cool to show my archives by month in a bar chart. My thought process was that someone new to the site could see the graph and very easily get a feel for the frequency of my posting. Below is a little screenshot for archival purposes and those in a feed reader. You can also just scroll towards the bottom of any page to see it live.

Navigation and Search

I’ve always put a lot of emphasis on my footer. From there you can get to pretty much anywhere on the site, but I thought it was high time that I finally put some navigation at the top. Then I thought, while I was at it, I might as well add some new content, so there are now two new pages to peruse.

The first page is dedicated to all my projects. It lists those that I care about, along with a little history and such. I think sometimes our Github profiles and a websites get too disconnected, so I think this will help connect the two a bit.

The second page I am simply referring to as “dude”: http://railstips.org/dude/. Whenever I find something new and fall in love with it, I’m always telling my friends, “Dude, you have to check this out.” That theory along with the success of my setup and software post, led me to believe I should have a page here at RailsTips dedicated solely to the things that make me say “dude!”.

The other tweak I did is I moved the search box to the top of the page from the sidebar. The reason for the move is that I think the search box is easily overlooked in the sidebar, but should do fine up at the top. It also helps balance the content above the navigation a bit.

Again, for archival purposes, I’m including a picture of the nav and search tweaks.

The only other addition worth noting is that I’ve switched from sections/categories to tags for articles. More a semantic change than anything, but I thought I would mention it. One of the next tweaks I’ll do is add a tag page where you can view all the tags listed alphabetically to get an idea for the topics covered here.

That is all for now. Just thought I would mention the updates for those who read the posts from their feed reader and for those that might be new to the site. I hope they make it even easier to find and discover the content I’ve been posting.

How to Add Simple Permissions into Your Simple App. Also, Thoughtbot Rules!

2009-04-20T14:53:00Z

In which I discuss how I added simple permissions into flightcontrolled.com an app I created and how cool clearance, shoulda, factory girl and paperclip are.

Last week, in a few hours, I whipped together flightcontrolled.com for Flight Control, a super fun iPhone game. The site allows users to upload screenshots of their high scores. I thought I would provide a few details here as some may find it interesting.

It is a pretty straightforward and simple site, but it did need a few permissions. I wanted users to be able to update their own profile, scores and photos, but not anyone else’s. On top of that, I, as an admin, should be able to update anything on the site. I’m sure there is a better way, but this is what I did and it is working just fine.

Add admin to users

I added an admin boolean to the users table. You may or may not know this, but Active Record adds handy boolean methods for all your columns. For example, if the user model has an email column and an admin column, you can do the following.

user = User.new
user.email? # => false
user.email = 'foobar@foobar.com'
user.email? # => true

user.admin? # => false
user.admin = true
user.admin? # => true

Simple permissions module

Next up, I created a module called permissions, that looks something like this:

module Permissions
  def changeable_by?(other_user)
    return false if other_user.nil?
    user == other_user || other_user.admin?
  end
end

I put this in app/concerns/ and added that directory to the load path, but it will work just fine in lib/.

Mixin the permission module

Then in the user, score and photo models, I just include that permission module.

class Score < ActiveRecord::Base
  include Permissions
end

class Photo < ActiveRecord::Base
  include Permissions
end

class User < ActiveRecord::Base
  include Permissions
end

Add checks in controllers/views

Now, in the view I can check if a user has permission before showing the edit and delete links.

<%- if score.changeable_by?(current_user) -%>
  <li class="actions">
    <%= link_to 'Edit', edit_score_url(score) %>
    <%= link_to 'Delete', score, :method => :delete %>
  </li>
<%- end -%>

And in the controller, I can do the same.

class ScoresController < ApplicationController
  before_filter :authorize, :only => [:edit, :update, :destroy]

  private
    def authorize
      unless @score.changeable_by?(current_user)
        render :text => 'Unauthorized', :status => :unauthorized
      end
    end
end

Macro for model tests

I didn’t forget about testing either. I created a quick macro for shoulda like this (also uses factory girl and matchy):

class ActiveSupport::TestCase
  def self.should_have_permissions(factory)
    should "know who has permission to change it" do
      object     = Factory(factory)
      admin      = Factory(:admin)
      other_user = Factory(:user)
      object.changeable_by?(other_user).should be(false)
      object.changeable_by?(object.user).should be(true)
      object.changeable_by?(admin).should be(true)
      object.changeable_by?(nil).should be(false)
    end
  end
end

Which I can then call from my various model tests:

class ScoreTest < ActiveSupport::TestCase
  should_have_permissions :score
end

Looking at it now, I probably could just infer the score factory as I’m in the ScoreTest, but for whatever reason, I didn’t go that far.

A sprinkle of controller tests

I also did something like the following to test the controllers:

class ScoresControllerTest < ActionController::TestCase  
  context "A regular user" do
    setup do
      @user = Factory(:email_confirmed_user)
      sign_in_as @user
    end

    context "on GET to :edit" do
      context "for own score" do
        setup do
          @score = Factory(:score, :user => @user)
          get :edit, :id => @score.id
        end

        should_respond_with :success
      end

      context "for another user's score" do
        setup do
          @score = Factory(:score)
          get :edit, :id => @score.id
        end

        should_respond_with :unauthorized
      end
    end
  end

  context "An admin user" do
    setup do
      @admin = Factory(:admin)
      sign_in_as @admin
    end

    context "on GET to :edit" do
      context "for own score" do
        setup do
          @score = Factory(:score, :user => @admin)
          get :edit, :id => @score.id
        end

        should_respond_with :success
      end

      context "for another user's score" do
        setup do
          @score = Factory(:score)
          get :edit, :id => @score.id
        end

        should_respond_with :success
      end
    end
  end
end

Summary of Tools

I should call flightcontrolled, the thoughtbot project as I used several of their awesome tools. I used clearance for authentication, shoulda and factory girl for testing, and paperclip for file uploads. This was the first project that I used factory girl on and I really like it. Again, I didn’t get the fuss until I used it, and then I was like “Oooooh! Sweet!”.

One of the cool things about paperclip is you can pass straight up convert options to imagemagick. Flight Control is a game that is played horizontally, so I knew all screenshots would need to be rotated 270 degress. I just added the following convert options (along with strip) to the paperclip call:

has_attached_file :image, 
  :styles => {:thumb => '100>', :full => '480>'},
  :default_style => :full,
  :convert_options => {:all => '-rotate 270 -strip'}

Conclusion

You don’t need some fancy plugin or a lot of code to add some basic permissions into your application. A simple module can go a long way. Also, start using Thoughtbot’s projects. I’m really impressed with the developer tools they have created thus far.

Twitter Gem Reborn with Fewer Features and 100% More OAuth

2009-04-05T18:14:00Z

In which I show off the rewritten Twitter gem and an example app that takes advantage of it.

The Twitter gem is kind of my baby. It was pretty much the first gem I created and as such I care for it a little more. When I originally created it, I knew nothing. I sucked. Now over time, I suck less and Twitter’s addition of OAuth seemed like the perfect time to completely rewrite it. Yay for rewrites!

The gem is now leaner, meaner and works swell with OAuth. I would imagine as I use OAuth a bit more, I’ll figure out some ways to make it even easier in the twitter gem but it works for now.

The Cuts

One of the biggest headaches and coolest things about the Twitter gem was the command line interface. It allowed for multiple accounts and was SQLite database backed. That was cool and all but it is ridiculous to have main and active record as dependencies for someone who just wanted the API wrapping aspects of the gem.

Knowing this, I have finally killed the CLI. Fear not, CLI fans, it will make a triumphant return, but that return will be in a separate gem that merely relies on the Twitter gem. The Twitter gem itself will be only a wrapper for the REST and Search APIs that Twitter provides. No more extra cruft.

OAuth

As I said, the main reason for the rewrite was that Twitter now supports OAuth. I thought about it long and hard and decided to not continue support for HTTP Authentication. Twitter has not discontinued support for HTTP Authentication, but they will in the future and I didn’t want to add a bunch of hours to releasing the updated gem simply to support something that is going to be deprecated.

So how does one use the OAuth stuff in the Twitter gem? First off, familiarize yourself with OAuth. Then, you can check out the example below pulled straight from the examples directory included in the gem (or over here on github).

require File.join(File.dirname(__FILE__), '..', 'lib', 'twitter')
require File.join(File.dirname(__FILE__), 'helpers', 'config_store')
require 'pp'

config = ConfigStore.new("#{ENV['HOME']}/.twitter")
oauth = Twitter::OAuth.new(config['token'], config['secret'])

if config['atoken'] && config['asecret']
  oauth.authorize_from_access(config['atoken'], config['asecret'])
  twitter = Twitter::Base.new(oauth)
  pp twitter.friends_timeline

elsif config['rtoken'] && config['rsecret']  
  oauth.authorize_from_request(config['rtoken'], config['rsecret'])
  twitter = Twitter::Base.new(oauth)
  pp twitter.friends_timeline

  config.update({
    'atoken'  => oauth.access_token.token,
    'asecret' => oauth.access_token.secret,
  }).delete('rtoken', 'rsecret')
else
  config.update({
    'rtoken'  => oauth.request_token.token,
    'rsecret' => oauth.request_token.secret,
  })

  # authorize in browser
  %x(open #{oauth.request_token.authorize_url})
end

The ConfigStore is a simple class that allows me to save the request and access token and secrets as I’m completing the handshakes between the script and Twitter.

As you can see, it is pretty simple. Get a request token, then use that to create an access token. From there, you can reuse the access token to authorize future requests.

An Example App

Now one might think, based on that previous example, “Thanks so much John! I really appreciate you getting the ball rolling for me with a simple example.” Friends do I have news for you. I didn’t stop there. I whipped together a little sample app, that I have been using for a few days now. Eventually, it will become a full Twitter client, but for now it is pretty basic and dirty.

It uses Clearance for authentication for authentication and automatically sends you to Twitter to authorize after firing it up (if you haven’t previously). I used Rails rescue_from to automatically send you to Twitter if any of the authorization stuff gets messed up at a later point to.

rescue_from Twitter::Unauthorized, :with => :twitter_unauthorized  
private
  def twitter_unauthorized(exception)
    redirect_to new_authorization_url
  end

Anytime the Twitter gem throws the Twitter::Unauthorized exception, you are redirected to new_authorization_url, which is powered by the AuthorizationsController, which looks like this.

class AuthorizationsController < ApplicationController
  def new
    oauth = current_user.oauth
    session['rtoken'] = oauth.request_token.token
    session['rsecret'] = oauth.request_token.secret
    redirect_to oauth.request_token.authorize_url
  end

  def show
    oauth = current_user.oauth
    oauth.authorize_from_request(session['rtoken'], session['rsecret'])

    session['rtoken'] = nil
    session['rsecret'] = nil

    current_user.update_attributes({
      :atoken => oauth.access_token.token, 
      :asecret => oauth.access_token.secret,
    })

    redirect_to root_path
  end
end

The new action stores the request token and secret and then redirects you to Twitter to authorize the app. Over at Twitter, I have registered an application and set the redirect url to go to /authorization in the app, which as you can see above, obtains the access token and secret and stores them with the current user.

You’ll probably notice that I am calling current_user.oauth and current_user.client throughout the app. These are helper methods I added to the user model to shorten up what was happening in the controller.

class User < ActiveRecord::Base
  include Clearance::App::Models::User

  attr_accessible :atoken, :asecret

  def authorized?
    !atoken.blank? && !asecret.blank?
  end

  def oauth
    @oauth ||= Twitter::OAuth.new(ConsumerConfig['token'], ConsumerConfig['secret'])
  end

  def client
    @client ||= begin
      oauth.authorize_from_access(atoken, asecret)
      Twitter::Base.new(oauth)
    end
  end
end

I could shorten up the controllers even more (and will at some point) by delegating several methods on user to user.client. I’ll post more on that later when I get around to it. Also, at some point, I’m going to remove the atoken and asecret from the user and allow support for multiple Twitter accounts for one user in the app. This should be pretty easy, but I wanted to get something working first before I complicated things.

Conclusion

From a user standpoint, OAuth is really nice. I love not having to put in my username and password. From a developer standpoint, OAuth is also pretty nice. Basically, you register your application with Twitter, provide the consumer token and key in a config file and this example app is ready to go.

As with any rewrite there are bound to be bugs. I have fixed one already that someone caught. If you catch any weird bugs using the rewritten Twitter gem, there is group and ticket information on the Twitter gem website.

Crack, The Easiest Way to Parse XML and JSON

2009-04-01T04:20:00Z

In which I release a new gem that allows parsing XML or JSON with a simple, consistent API.

An astute reader will remember that a while back, HTTParty divorced both ActiveSupport and the JSON gem in order to make it easier to use all around. With the JSON gem went the last gem dependency, which was kind of cool.

A few days back, it occurred to me that the parsing of XML and JSON that HTTParty used might be handy outside of HTTParty. In the spirit of sharing, I whipped together a new gem, named crack, that contains the XML and JSON parsers that formerly were bundled in HTTParty.

Why Crack?

I figured the name was easy and memorable, which is a requirement for anything I’m going to release. When I thought about parsing XML and JSON, for some reason, cracking the code came to mind and thus crack had a name.

Credits

First, I’d like to make it abundantly obvious that I did not author any of this code. I tweaked it a bit and made sure it had tests, but the XML parsing was extracted from Merb (extlib) and the JSON parsing from Rails (ActiveSupport). I merely packaged them together for all to enjoy. Ok, now that we have that out of the way, let’s move onward.

So I ripped the two parsers out of HTTParty and put them in their own gem and then just set that as a dependency for HTTParty. HTTParty will still work the exact same, but if all you need is a really simple way to parse JSON or XML, crack is his name and parsing is his game.

Details

As always these days, I used shoulda and matchy for testing and jeweler to make the gem maintenance easy. That is pretty much it for details on this project. It is focused and simple so there isn’t much behind the scenes.

Installation

I registered a rubyforge project, but I’m waiting for approval. For now, you can get the gem from Github.

sudo gem install jnunemaker-crack -s http://gems.github.com

Usage

It has always slightly annoyed me that all the different XML and JSON (JSON.parse, ActiveSupport::JSON.decode) parsing mechanisms available in Ruby have different APIs. I think parse is the easiest to remember and it is consistent with HappyMapper, another project of mine, so whether you are working with XML or JSON, all you have to remember is parse.

xml = '<posts><post><title>Foobar</title></post><post><title>Another</title></post></posts>'
Crack::XML.parse(xml)
# => {"posts"=>{"post"=>[{"title"=>"Foobar"}, {"title"=>"Another"}]}}

json = '{"posts":[{"title":"Foobar"}, {"title":"Another"}]}'
Crack::JSON.parse(json)
# => {"posts"=>[{"title"=>"Foobar"}, {"title"=>"Another"}]}

That is pretty much all there is to it. Given XML or JSON, you get back a hash. The repositoryhas been up for a couple days, but I thought I would mention it here as well. The keys here are simple and consistent. If you just want to get dirty and you aren’t worried about performance, crack is a perfect fit.

OAuth Explained and What It Is Good For

2009-03-29T02:33:00Z

In which I attempt to explain OAuth in simple terms and what it is good for.

Twitter recently announced OAuth support and that eventually they will be deprecating HTTP Basic Authentication in favor of OAuth. Knowing this, I figured it was about time to get familiar with OAuth and update the Twitter gem to use it.

Let me start by explaining my history with OAuth. I have none. There that was fast. I didn’t read the specifications or any articles on OAuth. I simply dove in code first and tried to figure out what was going on and how to make things work. I promise that what I’m about to explain will be simple and that I will not spend any time on hmac-sha1 signatures.

For the code examples below, I’ll be using the OAuth gem. Install is typical and I’ll provide it below for copy and paste.

sudo gem install oauth

Tokens and Secrets

At first, the hardest thing to figure out was all the tokens and secrets. Basically, there are three sets of made up of a token and a secret. Each set builds upon the last. The three sets are consumer, request and access.

The consumer token and secret are provided for you by the OAuth provider, when you register an application with them. These basically define what application is attempting to do the deed. You can create a new consumer like this:

consumer = OAuth::Consumer.new(
  'consumer token', 
  'consumer secret', 
  {:site => 'http://twitter.com'}
)

Before I go on, let me explain the end goal. The end goal with OAuth is to get an access token and secret. Once you have these, requesting a users information is much like it would be with HTTP Basic Authentication from a Ruby API point of view.

Request Token

In order to get the access token, you have to create a request token, keep track of it, and then redirect the user to the provider to authorize your application. You can create a new request token, using the consumer you just created, like so:

request_token = consumer.get_request_token
puts request_token.token, request_token.secret

The other thing that the request token provides is an authorization URL. The authorization URL is where your application redirects to, to allow the user to grant or deny you access to the their data. You can get the authorization URL in quite a predictable manner:

request_token.authorize_url

If you were using Rails, you would simple redirect to this URL. Before you redirect, be sure to store the request token and secret as you’ll need those to create the access token when the user returns to your application.

Whether your app is on the web or something else, the aforementioned steps are the same. The difference is if you have a web app, you provide a redirect URL that the provider sends the user back to upon granting access, whereas if you have a desktop app, or the like, the page will just inform the user to head back to the app.

Access Token

Either way, once the user is back at your app, you use the request token and secret to generate the access token. In the code below, you would use the request token and secret that you stored in the session before redirecting to the authorization URL.

request_token = OAuth::RequestToken.new(consumer, 'request token', 'request secret')
access_token = request_token.get_access_token

Once you have the access token, you can make authenticated requests like so:

puts access_token.get('/statuses/friends_timeline.json')

I haven’t had to go this far yet, but I’m assuming you would store the access token and secret in the database or something for later use. Someone correct me if I’m wrong about this. Next time you need to access the user’s data, you can simply create a new access token from the token and secret, and once again make authenticated requests.

access_token = OAuth::AccessToken.new(consumer, 'access token', 'access secret')
puts access_token.get('/statuses/friends_timeline.json')

The Twitter Gem

So how does all this affect the Twitter gem? Currently, setting up the authentication for the Twitter gem is as easy as this:

Twitter::Base.new('email', 'password')

In the OAuth version of the Twitter gem, setting up the authentication will be something like this:

Twitter::Base.new('access token', 'access secret')

So, pretty much the same. The only difference is that you will have to register your application with Twitter and then go through the process of getting the access token and secret. The good news is I already have a Twitter::OAuth class that will make this process as easy as possible.

What It Is Good For

I see three common methods for authenticating web services—HTTP Authentication, API keys and OAuth. HTTP Auth is simple, but then you end up with usernames and passwords all over the interwebs. OAuth is better in that users don’t give out passwords and can cancel at anytime with any app they have granted permission to.

The next question might be, “Wouldn’t an API key system be more simple than OAuth?” The answer is yes—for developers. For developers, it is easier to just tack an API key on every request than to go through the process of all the aforementioned tokens and secrets. So if an API key system is easier for developers, why use OAuth?

I see OAuth as the easiest solution for users. When determining how something will work for users, I often user the mom test. I guarantee you if my mom wanted to print a picture from flickr, she would not be able to figure out how to create an API key on Flickr and then use it with the printing site. Think about the opposite though. I definitely think that when redirected to flickr from the printing site, she could click yes to grant the printing site access.

OAuth is easier for users and API keys are easier for developers. When you go to build your API, take your primary audience into consideration and pick the solution that suites them best.

Hopefully this primer helps those (like me a day ago) who haven’t taken the time to get to know OAuth, and also those who have enjoyed the Twitter gem and are curious about what implications OAuth brings to the table. Formerly, I thought OAuth was over complicated, but now that I’ve spent a few hours with it, I’m pretty comfortable. Oh, and the other cool thing about this is now I have a base to work from for adding OAuth support to HTTParty.

Any OAuth junkies out there, feel free to correct me or elaborate in the comments.

Update: For more on OAuth, you can read how I updated the Twitter gem to use it.

Some Further Reading

OAuth website – Docs and history.
OAuth Gem
Official Twitter OAuth Example – Includes pictures. Ooooh ahhh. Pictures. :)
TwitterAuth – OAuth Twitter client (competition for the Twitter gem when I finish OAuth support for it)
Sinitter – Sinatra app using TwitterAuth.
Developing OAuth clients in Ruby
How to turn your Rails app into an OAuth provider

In 1492, Columbus Discovered...A Feed

2009-03-26T14:00:00Z

In which I release a new gem named Columbus that auto-discovers feed urls.

Sorry for the rush of posts the past few days. I’ve been feeling a bit inspired of late, and with the discovery of Jeweler, it is now easier to make a gem than it is to just let the code sit in ~/dev/ruby/ on my computer.

A few weeks ago, I showed how to follow redirects using net/http, but I provided no context really as to why I wanted to follow redirects. Basically, that code was one piece of some code I wrote to auto-discover feed urls for a given url.

I was a little more creative with the name this time than last time, calling it Columbus. Get it? Auto-discovery and Columbus was a discoverer. Yeah, you get it. I’m sure I don’t need to explain it. Right? Yeah.

Usage

There isn’t much code in the gem and using it is even easier.

# get the primary feed
primary = Columbus.new('http://railstips.org').primary
puts primary.url, primary.title, primary.body

# get all the feeds
Columbus.new('http://railstips.org').all

The first returns a single feed if one is found else nil. The second returns all the feeds found in an array. That probably doesn’t feel like much, but there is a lot more going on behind the scenes.

Behind the Scenes

Gets the response for the passed in url.
If the URL is a redirect, it follows the redirect up to 5 times to find the endpoint.
Once it has the endpoint, it uses Hpricot to get all the link tags in the response body that appear to be RSS or Atom feeds.
For each link tag found, it gets the response for the URL and once again follows redirects up to 5 times until it finds an endpoint.
Once the endpoint for each feed is found, it returns the URL, the title and the response body for you to fart around with.

Some Details

Once again, I used shoulda, matchy and fakeweb to do the testing. I didn’t need HTTParty, but I did break out an old friend Hpricot, which I haven’t used since XML parsing in the Twitter gem. Kind of funny that this is the first time I used Hpricot for its original intent, parsing HTML.

Installation

For now the gem is just up on Github so the usual routine will get your going.

sudo gem install jnunemaker-columbus --source http://gems.github.com

Hopefully someone finds it useful someday. :) I’ve already got my mileage out of it.

Building API Wrapping Gems Could Not Get Much Easier

2009-03-25T11:00:00Z

In which I show how easy it is now to create ruby gems that wrap APIs, using Google Weather as an example.

Google has a weather api that is dead simple to use. Just discovered that tonight so I whipped together a wrapper using HTTParty. I decided to try out Jeweler, a project by Josh Nichols, that makes creating gems a snap and it delivered. I used shoulda and fakeweb for the tests. Holy crap has making a gem that wraps a web service become really easy.

The New Way

jeweler google-weather —shoulda —create-repo
%w(matchy fakeweb).each { |x| require x } (in your test_helper)
require ‘httparty’
Add some code and tests
rake version:bump:minor
rake gemspec
git push origin master
blog

I did all of these in about an hour or two tonight.

The Old Way

Create a bunch of files and directories and make a bunch of decisions
mock and stub all net/http stuff
net/http and rexml (or hpricot once that came along)
Add some code and maybe some tests
Add a version
Figure out how to build a gemspec
svn commit your files
Request project to be created on rubyforge
Wait a few days
Project approved, release files, blog

And it would take a few days from first code scratched to gem released. My how times are a changing.

Stuff You Can Learn From This Gem

At any rate, the GoogleWeather gem I just created is a really simple example of how to use:

jeweler to create and manage a gem
httparty to pwn an API
shoulda to test the gem
fakeweb to make sure your tests aren’t making real web requests
matchy for some syntactical sugar

If you want to learn any of those things, poke around in the code a bit and you should be good to go. Also, if you want a really easy way to get weather information, this gem makes that possible.

Sorry I didn’t give it some fancy name like HTTParty or HappyMapper. Maybe I need to make another gem that spits out fancy names. After all, naming the project is the only thing left that is hard. ;)

RailsTips - Home

MongoMapper, The Rad Mongo Wrapper

Installation

Usage

Include Instead of Inherit

Defining Keys

Validations

Callbacks

Embedded Documents

Finding Documents

Conclusion

JSONQuerying Your Rails Responses

Finding a Ruby JSONQuery Implementation

Bringing It All Together

Conclusion

What Is The Simplest Thing That Could Possibly Work?

The Shortest Path

Work Today Makes You Better Tomorrow

Narrow What You Think About

What If A Key/Value Store Mated With A Relational Database System?

Intro to MongoDB

Collections

Documents

Object Relationships

Dynamic Queries

Random Awesomeness

How do I use it with Ruby?

MongoMapper

Links

Swine Flu and the Twitter Gem

Pure TDD

The Tests

Example Uses

Include vs Extend in Ruby

Include Example

Extend Example

A Common Idiom

Conclusion

Class and Instance Methods in Ruby

One Line Summary

Class Methods

Instance Methods

Conclusion

RailsConf 2009

Say Hi!

Now With Navigation and Charted Archives

Charted Archives

Navigation and Search

How to Add Simple Permissions into Your Simple App. Also, Thoughtbot Rules!

Add admin to users

Simple permissions module

Mixin the permission module

Add checks in controllers/views

Macro for model tests

A sprinkle of controller tests

Summary of Tools

Conclusion

Twitter Gem Reborn with Fewer Features and 100% More OAuth

The Cuts

OAuth

An Example App

Conclusion

Crack, The Easiest Way to Parse XML and JSON

Why Crack?

Credits

Details

Installation

Usage

OAuth Explained and What It Is Good For

Tokens and Secrets

Request Token

Access Token

The Twitter Gem

What It Is Good For

Some Further Reading

In 1492, Columbus Discovered...A Feed

Usage

Behind the Scenes

Some Details

Installation