RailsTips by John Nunemaker

Canable: The Flesh Eating Permission System

2010-02-28T13:12:31-0500

A while back I wrote about how to add simple permissions to your apps. Since then, I have worked on a few applications (Harmony among them) where I have taken that concept and expanded it. Yesterday, I decided that I had repeated myself enough times (3) and that I should abstract the shared functionality of those apps into a gem. Thus, Canable, the flesh eating permission system, was born.

Can

Canable does not actually implement any permissions for you (or actually eat flesh). Instead, it provides you with all the helpers and then (gasp) you have to do the work. The idea centers around running all permissions through current_user. Anytime you check if a user can do something you use a can method:

user.can_create?(article)
user.can_update?(article)

Able

Instead of having a big case statement in those can methods for each different type of object, I use the strategy pattern to just ask the object if the user has permission to do the action. This is done by having a matching “able” method to the “can” method, thus canable.

class Item
  def updatable_by?(user)
    creator == user
  end
end

The above code, for example, makes it so that only the creator of an item can update it. Obviously, you can get more in depth from there. By default, I add the following can and able methods:

:view => :viewable
:create => :creatable
:update => :updatable
:destroy => :destroyable

Custom Actions

If you need permissions for actions other than the defaults, you can add your own quite easily:

Canable.add(:publish, :publishable)

The readme over on Github has far more details, but I figured I would at least cover it here a bit. It might seem a bit weird at first, but once you start rolling with it, it makes for a pretty easy to implement and understand permission system.

The really funny part is that it is only like 80 lines of code, as most of the methods are dynamically generated. I am perfectly fine if I am the only one who uses this and finds it helpful, but you never know, so feel free to install it as a gem or fork it on github.

Note: No permissions were harmed in the making of this gem.

MongoMapper 0.7: Identity Map

2010-02-21T17:29:44-0500

While it isn’t quite ready to shove down everyone’s throats, IdentityMap is a hot new plugin I have been working on, directly due to a need that arose in Harmony.

One of the things I started to notice in Harmony right away was an epic amount of queries, 90% of which were simple id lookups repeated over and over. I remembered hearing about the identity map pattern before and figured I would research it a bit.

An old proverb says that a man with two watches never knows what time it is. If two watches are confusing, you can get in an even bigger mess with loading objects from a database.

So true. Now that I had the plugin system in place, I knew exactly how I would go about integrating IM (identity map) with MM. The great thing about the process was that I had a need from a real application. Over half of the IM plugin was created while vendored in Harmony. Several test cases that are currently in MM are directly due to weird bugs that we experienced in Harmony. I think it is really important to note that MM features grow from real applications, not theoretical fun.

Foo is Foo is Foo

The first thing I had to get working was to make sure that each document in the database equated to one object in Ruby. This means whether I call Item.find(id) or item.parent, if they are both loading the same document, they need to be the same object id in Ruby.

The easiest way I could think to do this was to have one method I always call whenever loading a document from the database. The obvious method name was load. Now, whenever an document is found in the database, I call load instead of new. This seems really insignificant, but down the rabbit trail it has significant implications.

Taking Advantage of Method Lookups and Super

Nothing fancy happens in load, but where it gets cool is in the IM plugin. Because every document coming from the database runs through load, all the IM plugin has to do is override load and ensure that the same document in the database is the same object in Ruby. It does this by storing each document in the map when loaded and then checking each document loaded against that store. If the document is already there, it returns the object for that document instead of creating a new one.

Each class gets an IM (just a plain old hash for now). Because each id is unique, the id is the key in the IM hash and the value is the Ruby object. Because of how method lookups work in Ruby, all that I do is include the IM plugin after the plugin that defines load and I can access the original load method using super. Below is the load method in the IM plugin:

def load(attrs)
  document = identity_map[attrs['_id']]

  if document.nil? || identity_map_off?
    document = super
    identity_map[document._id] = document if identity_map_on?
  end

  document
end

Without even knowing what the other methods do, you can get a feel for what is going on. First, I set document to where I assume it will be in the identity map for the class. If it is nil, that means it doesn’t exist, so I call super which calls the normal load method, and returns a new instance of the class based on the document from the database. Then, if the identity map is turned on, I add it to the map by setting the id as the key and the object as the value.

At the end, I return the document no matter what has happened before. With this tiny bit of code (and a few more lines for setting up hash store), I ensure that no matter how or when a document is queried from the database, it is always the same object in memory. Soooo sweet. Like I said, huge ramifications.

Identity Map Lookups Zap Simple Queries

Once I had this part done, all that was left was to zap the actual queries to the database if the document has already been loaded. I refactored the MM internal methods for finding documents to find_one and find_many. This means whenever you do any kind of find query in MM, eventually it hits one of those two methods.

Taking the same approach as load, if you are doing everything through one or two methods, all you have to do to change those methods is redefine them in the plugin to behave differently. The best part is you still have access to the originals using super. find_one in the IM plugin looks like this:

def find_one(options={})
  criteria, query_options = to_query(options)

  if simple_find?(criteria) && identity_map.key?(criteria[:_id])
    identity_map[criteria[:_id]]
  else
    super.tap do |document|
      remove_documents_from_map(document) if selecting_fields?(query_options)
    end
  end
end

simple_find? returns true if doing a query only by _id or _id and _type (which happens when using single collection inheritance). This method addition means we can return the document from the identity map without doing a query, if it has already been loaded.

A Simple, Yet Awesome Example

I learn best with code examples, so here is one for you. A simple item class for creating an even more simple tree.

MongoMapper.connection = Mongo::Connection.new('127.0.0.1', 27017, :logger => Logger.new(STDOUT))
MongoMapper.database = 'testing'

class Item
  include MongoMapper::Document
  
  key :title, String
  key :parent_id, ObjectId
  
  belongs_to :parent, :class_name => 'Item'
end

root = Item.create(:title => 'Root')
child = Item.create(:title => 'Child', :parent => root)
grand_child = Item.create(:title => 'Grand Child', :parent => child)

puts root.equal?(child.parent) # false
puts child.equal?(grand_child.parent) # false

If you run this code, you get false and false as the output, along with a few queries to find the parent of child and grand_child. Below is some sample output:

MONGODB admin.$cmd.find({:ismaster=>1}, {}).limit(-1)
MONGODB db.items.update({:_id=>4b81a114d072c40c3f000001}, {"title"=>"Root", "_id"=>4b81a114d072c40c3f000001, "parent_id"=>nil})
MONGODB db.items.update({:_id=>4b81a114d072c40c3f000002}, {"title"=>"Child", "_id"=>4b81a114d072c40c3f000002, "parent_id"=>4b81a114d072c40c3f000001})
MONGODB db.items.update({:_id=>4b81a114d072c40c3f000003}, {"title"=>"Grand Child", "_id"=>4b81a114d072c40c3f000003, "parent_id"=>4b81a114d072c40c3f000002})
MONGODB testing.items.find({:_id=>4b81a114d072c40c3f000001}, {}).limit(-1)
false
MONGODB testing.items.find({:_id=>4b81a114d072c40c3f000002}, {}).limit(-1)
false

Check out the same script, with the addition of the identity map plugin:

MongoMapper.connection = Mongo::Connection.new('127.0.0.1', 27017, :logger => Logger.new(STDOUT))
MongoMapper.database = 'testing'

class Item
  include MongoMapper::Document
  plugin MongoMapper::Plugins::IdentityMap
  
  key :title, String
  key :parent_id, ObjectId
  
  belongs_to :parent, :class_name => 'Item'
end

root = Item.create(:title => 'Root')
child = Item.create(:title => 'Child', :parent => root)
grand_child = Item.create(:title => 'Grand Child', :parent => child)

puts root.equal?(child.parent) # true
puts child.equal?(grand_child.parent) # true

Note that we get true for both. Also, we get no queries for the parents of child and grand_child, as they are already in the identity map. The output looks something like this:

MONGODB admin.$cmd.find({:ismaster=>1}, {}).limit(-1)
MONGODB db.items.update({:_id=>4b81a0c9d072c40c2c000001}, {"title"=>"Root", "_id"=>4b81a0c9d072c40c2c000001, "parent_id"=>nil})
MONGODB db.items.update({:_id=>4b81a0c9d072c40c2c000002}, {"title"=>"Child", "_id"=>4b81a0c9d072c40c2c000002, "parent_id"=>4b81a0c9d072c40c2c000001})
MONGODB db.items.update({:_id=>4b81a0c9d072c40c2c000003}, {"title"=>"Grand Child", "_id"=>4b81a0c9d072c40c2c000003, "parent_id"=>4b81a0c9d072c40c2c000002})
true
true

Obviously, this is a really small and simple example, but with one small addition, we saved a few queries. Imagine how much of a difference it makes in a big application making lots of requests. The thing I am most amazed at is how much punch this plugin adds compared to the amount of code in the implementation. As of 0.7, the identity map plugin is only 122 lines of code.

Using the IM Plugin with 0.7

The IM plugin is by no means feature complete, so I am not automatically including it in every Document yet. I will say that what I have added is production ready and we have been using it in Harmony for over a month now. You can use it on a model by model basis like this:

class Foo
  include MongoMapper::Document
  plugin MongoMapper::Plugins::IdentityMap
end

Or you can turn it on for all documents by dropping this in an initializer (stolen directly from Harmony):

module IdentityMapAddition
  def self.included(model)
    model.plugin MongoMapper::Plugins::IdentityMap
  end
end

MongoMapper::Document.append_inclusions(IdentityMapAddition)

Correct, Beautiful, Fast

The IM plugin, for me, is a great example of Correct, Beautiful, Fast. First, we built Harmony in a way that worked and was easy to read (correct and beautiful). Then, when we needed to make it fast, all we had to do was override the implementation in a few spots and we cut our queries in half (or more) over night. It is far easier to find where you need to optimize when your code is correct and beautiful.

MongoMapper 0.7: Plugins

2010-02-22T08:31:32-0500

Whew, been a bit of a whirlwind around here. Lots to do lately, so I have been slow talking about MongoMapper’s 0.7 release from a week ago. I’ll take a crack now. Let’s start with how to extend MongoMapper with plugins and some other fun functionality.

Separation Is Good

Originally, because Document and EmbeddedDocument shared so much, in my head it made sense to just include EmbeddedDocument in Document. What this led to was confusion in the code. Lots of things sharing, some that should not be and some that should. Yes, I wanted to share the functionality between the two modules, but including one in the other is not the best way.

So I sat down and thought about it. I figured the best way would be to split out each bit of functionality into separate modules and then include each of those modules that were shared into both Document and EmbeddedDocument. With this in mind, I went to down. I broke all the refactoring rules and just started ripping stuff out and going to down. What I ended up with is a few tests that, despite all odds, I could not get to pass. My problem was that I took on too much at once.

Thankfully, several hours in, I realized what I had done and stopped. I blew away all my changes and started over. This time, I focused on small changes and made sure to run all the tests after each change. Attacking the code base with this mindset allowed me to slowly and gently remove bits and pieces into modules and begin the process of separating the conjoined twins known as Document and EmbeddedDocument.

Plugins Are Born

As I was doing this, I realized that I was doing the same things over and over. Including a module here, extending one there and occasionally dropping down to class_eval when brute force was needed. That is when it hit me that some sort of system that abstracted this process out would be nice. Thus was born MongoMapper::Plugins. MM::Plugins is the tiniest bit of code, but it led to sweeping changes that make MM far more readable and separated. It is so tiny, in fact, that I will paste it below.

module MongoMapper
  module Plugins
    def plugins
      @plugins ||= []
    end

    def plugin(mod)
      extend mod::ClassMethods     if mod.const_defined?(:ClassMethods)
      include mod::InstanceMethods if mod.const_defined?(:InstanceMethods)
      mod.configure(self)          if mod.respond_to?(:configure)
      plugins << mod
    end
  end
end

This small addition made separating out all the shared functionality really easy to complete and really easy to read. For example, when you create a MongoMapper::Document, you include that module. This calls the Ruby included hook, which now looks like this in Document:

def self.included(model)
  model.class_eval do
    include InstanceMethods
    extend  Support::Find
    extend  ClassMethods
    extend  Plugins

    plugin Plugins::Associations
    plugin Plugins::Clone
    plugin Plugins::Descendants
    plugin Plugins::Equality
    plugin Plugins::Inspect
    plugin Plugins::Keys
    plugin Plugins::Dirty # for now dirty needs to be after keys
    plugin Plugins::Logger
    plugin Plugins::Pagination
    plugin Plugins::Protected
    plugin Plugins::Rails
    plugin Plugins::Serialization
    plugin Plugins::Validations
    plugin Plugins::Callbacks # for now callbacks needs to be after validations

    extend Plugins::Validations::DocumentMacros
  end

  super
end

As you can see, all I do is declare a bunch of plugins. It is really easy to see what is going on without even looking at those plugins. This little change was huge mentally for me. It has made the project a joy to maintain once again, as I do not have to think about how to implement each new bit of functionality. The new feature just goes into a new or existing plugin.

An Example Plugin

So what does a plugin look like? Below is the logger plugin:

module MongoMapper
  module Plugins
    module Logger
      module ClassMethods
        def logger
          MongoMapper.logger
        end
      end

      module InstanceMethods
        def logger
          self.class.logger
        end
      end
    end
  end
end

This plugin adds simple methods for accessing MongoMapper’s logging functionality on any Document or EmbeddedDocument. It means if you have a Foo model, you get Foo.logger and Foo.new.logger. Very simple and very readable. Most of the plugins are tiny like this.

Extending All Models

The other cool thing, an idea taken from DataMapper, is append_inclusions and append_extensions. These two methods are added to both Document and EmbeddedDocument and allow you to add any modules to be included or extended on every Document or EmbeddedDocument. Examples work best for me, so check this out:

class Item
  include MongoMapper::Document
end

module Fooer
  def foo
    puts 'Foo'
  end
end

MongoMapper::Document.append_extensions(Fooer)

class Sandle
  include MongoMapper::Document
end

Item.foo # puts 'Foo'
Sandle.foo # puts 'Foo'

Note that it does not matter whether you append_extensions before or after the model is defined. Either way, anything that includes MongoMapper::Document gets the extension. Behind the scenes, this calls extend Fooer on each model. append_inclusions behaves the same except that instead of calling extend with the module, it calls include. See below:

class Item
  include MongoMapper::Document
end

module Fooer
  def foo
    puts 'Foo'
  end
end

MongoMapper::Document.append_inclusions(Fooer)

class Sandle
  include MongoMapper::Document
end

Item.new.foo # puts 'Foo'
Sandle.new.foo # puts 'Foo'

Note on this example that we are calling new and that foo is now an instance method instead of a class method (include vs extend). Also, append inclusions and extensions are added to EmbeddedDocument so you can apply functionality specfically to it if needed.

Adding A Plugin To All Documents

The cool thing about this is if you want to add a plugin to every Document, you can just do something like this, instead of manually calling plugin in each model:

class Item
  include MongoMapper::Document
end

module FooPlugin
  module ClassMethods
    def foo
      puts 'Foo'
    end
  end

  module InstanceMethods
    def foo
      self.class.foo
    end
  end
end

module FooPluginAddition
  def self.included(model)
    model.plugin FooPlugin
  end
end
MongoMapper::Document.append_inclusions(FooPluginAddition)

class Sandle
  include MongoMapper::Document
end

Item.foo        # puts 'Foo'
Sandle.foo      # puts 'Foo'
Item.new.foo    # puts 'Foo'
Sandle.new.foo  # puts 'Foo'

We create the FooPlugin, then we create a module that uses Ruby’s included hook to call plugin on the model. Because append_inclusions is called on each Document, all our models get the new plugin. Fun stuff.

Thanks to plugins and the append extensions/inclusions stuff there is a standardized way of extending MongoMapper models. In fact, people have already hopped on the bandwagon and started adding features as plugins instead of directly into Document or EmbeddedDocument. The first such plugin to be merged in is the protected attributes plugin.

These additions are huge for MongoMapper in general, but I have found them handy for organizing code on Harmony as well.

There is still more work to do in regards to organizing the MM code base, as some plugins are almost catch alls, but this has really cleaned things up in the code, especially for people new to MM.

MongoTips: All Mongo, All The Time

2010-02-17T15:17:52-0500

So last night I realized that I really love MongoDB. I love it so much that I want everyone to be using it whenever it makes sense (which is quite often). I thought about what I could do to help get the word out more and starting a new site made the most sense (instead of flooding RailsTips with Mongo stuff).

MongoTips is a new blog dedicated to MongoDB news and information. I’ll be posting things there that we have learned building Harmony and anything I can find from the community.

If you are interested in Mongo, be sure to subscribe to the feed or follow on twitter.

The Process

What I love about the process of making this site is how smooth it was with Harmony. I registered the domain, setup the DNS, added the site in Harmony, duplicated the RailsTips theme and a few minutes later the site was ready to go. All that was needed was a few posts and it was ready to launch.

That is really what I want Harmony to bring to the table. I want it to take away all the pain so all you are left with is authoring your site. Lovely.

Great Lakes Ruby Bash

2010-02-17T14:28:38-0500

April 17th of this year, a lot of good people I know are throwing a conference for the Great Lakes region at Michigan State University (Go Irish!). They currently have a call for proposals open until February 28th.

I’m already on the docket and I have something awesome planned, so whether you want to talk or just attend, I would highly recommend keeping this conference on your radar.

Correct, Beautiful, Fast (In That Order)

2010-03-04T11:41:42-0500

One of the books I am occasionally grazing on is Beautiful Code. The title of this post is the title of Chapter 6. I’ll be honest, I only skimmed the chapter as one can only digest so much Java, but the title is spot on.

Step 1: Correct (With Tests)

I have been thinking exactly this as of late, though I will admit in no way as succinctly. First, you get it working. It does not matter if it is dirty. Rather, what matters is that it functions and is well tested.

If it works and it is not tested, you can never make it beautiful with confidence, so testing is very important to this step.

Step 2: Beautiful

Once your program is working and testing, make it beautiful. Leave the tests alone and head back into the code. Branch out and try new idioms that you have not tried before. Ensure that someone with no prior knowledge could jump in and know what is going on (without ridiculous comments all over).

Note that this step is not optional. Once your code is functional and well tested, you still have an obligation to make it beautiful. I think a lot of people skip this step. They think once it works and is tested, that is the end. Wrong.

Not only is this step the most fun, it also forces you to think through the code more. I often find edge cases during this step that I would never find if I stopped and correct and tested.

Step 3: Fast (optional)

If you pass these first two steps and you run into slowness at an unacceptable level, make it fast. The author puts it best in the summary at the end of the chapter, so I will quote them:

If there’s a moral to this story, it is this: do not let performance considerations stop you from doing what is right. You can always make the code faster with a little cleverness. You can rarely recover so easily from a bad design.

So true. I have never focused on speed in MongoMapper. No benchmarks to wow the noobs. No dramatic statements about performance. I have focused on cleaning up the code and making it easier for myself to maintain and others to contribute.

Not long ago, one of those contributors sent me a tiny patch (~30 lines) that improved overall performance by at least two, maybe three times what it was in 0.6.10. I do not think that it would have been that simple to to find and fix the performance issues if the code had been a mess.

Conclusion

Correct: make it work and test it.
Beautiful: refactor original code in a manner that others (and yourself down the road) can easily understand.
Fast: often the first two steps leave code fast enough, but if they do not, make it faster.

Just In Time, Not Just In Case

2010-01-24T23:32:55-0500

Something that has finally become a habit for me is adding code when it is needed, not in case it is needed. Often times we “think” we are going to need something, so we add the code to support it. What happens most often with that code is it sits and rots. It adds bloat and weight to your program that is not needed.

Example

I like to see examples first, so I will start with one. In Harmony, sites have users. I had a method in the site model named add_user.

class Site
  def add_user(user)
    user_id = user.is_a?(User) ? user.id : user
    # add user code here, removed for clarity
  end
end

Some might look at that and think it is a perfectly fine method, but I would disagree. A quick search through my code revealed that I was never passing a user id. Every time I called the method, I passed in a user as an argument. That was the first smell.

The second, which may be for another article is that I used is_a?. We are programming in Ruby folks. It is always better to duck type and ask respond_to? than is_a?. The difference is asking an object what it is, instead of asking an object what it can do.

Having add_user take either a user or a user id meant twice as many tests and an extra line of code. I killed the is_a? ternary line and the accompanying tests and went on my merry way.

How to Avoid Just in Case

The best way that I have found to avoid “just in case” is to test first. When I write my tests first and start with the most basic test I can think of and go from there, my code tends to only be as complex as the test requires. Whenever I begin with program code instead of test code, I end up with “just in case” features like the one above.

This might seem small as it was only one line and it was well tested, but this was only one small example. Cases like this multiply as the program grows and the “real” code begins to get obfuscated behind the cases that might happen someday.

The moral of the story is a line removed is a line that will never have bugs down the road.

*To give credit where I believe credit is due, I am 99% positive that Jamis Buck said “Just in time, not just in case.” at a conference. Could not find a reference though.

Multiple Domain Page Caching

2010-02-02T11:48:37-0500

The other day Brandon Wright emailed me about the following tweet:

Just deployed full page caching on Harmony. Our log file stopped spinning by which made me happy and sad.

Routing

It might seem like black magic, but it isn’t all that hard. The front side for Harmony is not the same as a typical Rails app as we have multiple domains pointed at Harmony and the paths are not known up front so they don’t go in the routes file. In order to get everything headed to a controller, the last route in our file is this:

map.dispatch '*path', :controller => 'the', :action => 'dispatch'

This uses Rails route globbing to send every path to an action named dispatch in a controller dubiously named “the” (because it made us laugh). From there, we determine if it we can find the site and if the site has an item (page, link, blog, post, etc.) that matches the path.

Caching

Somewhere down the rabbit hole we render that item based on it’s liquid template, immediately after which we call something like this:

cache_item(@item, contents)

# which looks kind of like this
def cache_item(item, contents)
  # gone for brevity
  
  FileUtils.mkdir_p(File.dirname(item.page_cache_path))
  File.open(item.page_cache_path, 'w+') { |f| f.puts(contents) }
end

*We could have used caches_page in Rails, but we are already using that without including the http host for asset and theme file caching, so it was easier to just roll our own.

All cache_item does is ensure that the directory exists and then write the contents of what we are about to send back to the browser into a file. Really nothing fancy. So what does item.page_cache_path look like? For a site like railstips.org and a path of /dude/, we end up with the following cache path:

#{RAILS_ROOT}/public/cache/railstips.org/dude/index.html

Note the use of the domain in the cache path. Since we have that, we can use apache rewrites along with conditions to tell apache to check if a cached file exists based on the host. If it does, we server that file and if it doesn’t, we just hit rails, cache the file, and return the response. We use Moonshine for our deployments so all we need to do is set the Passenger page cache directory like this:

:passenger:
  :page_cache_directory: '/cache/%{HTTP_HOST}'

When we deploy, this sets up the following Apache rewrite rules:

# Rewrite to check for Rails non-html cached pages (i.e. xml, json, atom, etc)
RewriteCond  %{THE_REQUEST} ^(GET|HEAD)
RewriteCond  %{DOCUMENT_ROOT}/cache/%{HTTP_HOST}%{REQUEST_URI} -f
RewriteRule  ^(.*)$ /cache/%{HTTP_HOST}$1 [QSA,L]

# Rewrite to check for Rails cached html page
RewriteCond  %{THE_REQUEST} ^(GET|HEAD)
RewriteCond  %{DOCUMENT_ROOT}/cache/%{HTTP_HOST}%{REQUEST_URI}index.html -f
RewriteRule  ^(.*)$ /cache/%{HTTP_HOST}$1index.html [QSA,L]

Note that in the RewriteRule, we include the HTTP_HOST, which when visiting railstips.org, would be railstips.org.

One URL to Rule Them All

The key to this being effective is only having one true url for each page. We do this right now by redirecting www to no-www and ensuring that each page has a trailing slash. First, no-www.

# no www
RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
RewriteRule ^(.*)$ http://%1$1 [R=301,L]

Next, we ensure that there is always a trailing slash when needed. This means that /foo redirects to /foo/ and foo.json just stays as foo.json.

RewriteCond  %{THE_REQUEST} ^(GET|HEAD)
RewriteCond %{REQUEST_URI} !^/admin/
RewriteRule ^(.*/[^/\.]+)$ $1/ [R]

Ensuring that each page has one URL is better for search engines and analytics. You don’t end up with split page rank for the same page (with and without slash) and the same thing is true for pageviews.

Cache Clearing

Now that I’ve explained a bit how we do the caching, I’ll mention quickly how we clear it. As they say, cache expiration and naming are the two hardest things to do in programming. We opted for the most simple solution that would work for now.

I made a simple site cache clearer module that I include in any model that can affect a site on the front side. It looks something like this.

module SiteCacheClearer
  def self.included(model)
    model.after_save    :clear_item_cache
    model.after_destroy :clear_item_cache
  end
  
  def clear_item_cache
    site.clear_item_cache if site.present?
  end
end

# To use
class Item
  include MongoMapper::Document
  include SiteCacheClearer
end

All it does is remove the entire site’s cache whenever the model is updated or destroyed. Like I said, nothing fancy. Doesn’t check if the thing is published. Doesn’t check what pages it is actually shown on and only removes them. It just blows away cache when things change.

Someday we’ll definitely do something more advanced like a reference-based cache where only the pages that need to be blown away are, but this is working great for now. Hope this is helpful to someone.

The main thing to remember is to use the host and make sure there is only one way to get to the resource.

So what does this all mean to our read heavy application? Well, we end up with Scout graphs like this:

The blue is apache requests and the orange is Rails requests. Notice that as our apache requests go up, our Rails requests stay pretty steady.

I Have No Talent

2010-02-25T22:57:03-0500

The other day someone sent me an IM and thanked me for my open source contributions. They then said something about wishing they had my gem/code creation talents. I didn’t miss a beat and informed them that I have no talent.

It is true. I have no talent. What I do have is a lot of practice. And I am not talking about occasionally dabbling in Ruby on the weekends. I am talking about the kind of practice where I beat code that isn’t working into submission (though often times the code wins).

The kind of practice where all of a sudden I realize that it is 2am and I’m exhausted physically so I should go to bed, but mentally I feel on fire so I let the code have me for another hour or two (I imagine this state to be like a marathon runner or ironman near the end of their race).

The kind of practice that leads to a GitHub profile stuffed with code I regret (and am embarrassed about, but don’t delete to remind me of where I once was) and code I am proud of (not near as much as I am embarrassed about though).

Intelligence

I am also not very smart. I have a good memory (though my wife will tell you it has some missing pieces) and I work really hard. Really hard. I get that from my dad. He is also not very smart (his words, not mine), with a good memory and works really hard. :)

I am sick of hearing people say, “Oh, I love your code, I wish I could do that.” You can. The only reason you can’t is because you don’t practice enough. I used to think that I wasn’t smart enough. I was jealous of those that did crazy code stuff that I couldn’t even comprehend. Then, one day, I ran into something I did not understand and instead of giving up, I pushed through. I sat there in front of my computer for hours and wrestled with class and class instance variables.

That day was a turning point for me. It was the last time I thought that whether or not I was successful depended on my talent or intelligence. It really comes down to hard work people. Ever since then, I have attacked each thing that I do not understand until I understand it.

I will close with this. I still suck. There are still so many people out there who are far better than I am, but that does not stop me anymore. I do not measure myself against the programming greats, but against those projects on my Github profile from years ago.

Live From Harmony

2010-01-08T04:09:37-0500

After more than a year and a half of self-funded work on and off, Harmony is almost ready. It is so close in fact, that we (Steve and I) are beginning to eat our own dog food by moving our own sites into it. Rails Tips is the first to go live (if you can see this post, you are a witness) and I’m not going to lie, it feels good!

As I mentioned previously, we recently switched Harmony to MongoDB. A few have asked what issues we ran into and so far it has only been mental (free your mind matrix-style stuff). Now we will start getting into the nitty gritty. Expect some good posts soon.

Rails Tips is all moved and our company site, OrderedList.com, is in the final stages of the redesign process and should be going live in Harmony later this week.

Redirects

Harmony has redirect support (including regex’s and back references) so I’ve put a few in place to fix the URLs, but there are bound to be some broken things around here. If you run into anything that isn’t working, just let me know with a comment or something.

Technical Details To Come

I am really excited about the future of Harmony and I’ll be posting more on the technical details soon, but for now this lame post will have to suffice. :)

The Balance of Convention and Innovation

2010-01-18T22:53:32-0500

Adam Wiggins just posted a few paragraphs by Tim Lind from his article titled Innovation in Database Technology. Rarely do I just post direct quotes here, but I really thought the paragraphs were insightful, so I’ll share them for those that haven’t had the pleasure yet.

That, is why we have really moved away from sql, it is not for any specific approach to scalability or data storage, but rather just the ability to free ourselves from the standardized ideas encapsulated in the standard query language.
I’m going to say it again, moving away from sql allows us to innovate. I’m sure no one will have ill feeling towards the notion of innovation, and standardization is almost the exact opposite, it is the crystallization of previous innovation, so of course it would be what we stand against.
We do not necessarily stand against any specific idea encapsulated by the sql standardization, rather we just choose to open ourselves up to investigating the elements of the system for the sake of making design decisions which provide innovative solutions.

You can’t innovate in a box of standards, thus the “think outside the box” saying. For whatever reason, I had not thought about the step back from SQL from this perspective.

Several people have asked me what issues I ran into switching to MongoDB. The biggest issue I ran into was freeing my mind from the standards and conventions that swaddled me in bed at night. Conventions and standards are great, but they do seem to be at odds with innovation.

We definitely do not want to be in a constant state of innovation, as I am sure that would lead to chaos. On the other hand, if we always stick to conventions and standards, we will never push forward. There has to be balance. After years of the same with regards to databases, it great to see the new ideas over the past year or so.

Thanks to Tim for the observations and Adam for putting them on my radar.

Getting A Grip on GridFS

2010-01-06T00:37:29-0500

I know it is almost Christmas and your minds are beginning to turn to gifts and family, but hang with me for a few more minutes. My buddy Jon pointed me to Grip yesterday. I liked the idea but not all of the implementation. I’ve been doing some GridFS stuff lately so I decided to take some time and tweak it to be more what I need.

You can get my fork of Grip on Github. Below is a simple example of how it works (you could also check out the tests).

class Foo
  include MongoMapper::Document
  include Grip
  
  has_grid_attachment :image
  has_grid_attachment :pdf
end

image = File.open('foo.png', 'r')
pdf = File.open('foo.pdf', 'r')
foo = Foo.create(:image => image, :pdf => pdf)

foo.image # contents read from gridfs for serving from rack/metal/controller
foo.image_name # foo.jpg
foo.image_content_type # image/png
foo.image_size # File.size of the file
foo.image_path # path in grid fs foo/image/:id where :id is foo.id

The main changes I made were to store name, size and content_type along with the path to the file. I also made it so those are assigned when the file is set so that they can be used in validations. Some other ideas I have for the plugin are adding image resizing with MojoMagick and common validations (much like most of the file upload plugins out there).

What I really like about Grip is its simplicity. A single, small file, an include, a call to has_grid_attachment, and you can start storing files. The great thing about working on this is it gave me some wonderful ideas for how to standardize the process of creating and declaring MongoMapper plugins.

I’ll be adding those to ideas to MongoMapper over the next few weeks as I get time and I think people are going to be stoked about what I’ve come up with (thanks to Brandon for brainstorming the plugin API with me).

Behind the Scenes

As I said, behind the scenes, Grip uses GridFS (more on GridFS), Mongo’s specification for storing files in Mongo.

The good news is that the API for storing files in GridFS using Ruby is nearly identical to using Ruby’s File class. Unfortunately, that is also the bad news, in my opinion, as I find Ruby’s File open, read and close a bit awkward. Here are a few examples pulled almost directly from Grip:

# write foo.jpg to grid fs
file = File.read('foo.jpg')
GridFS::GridStore.open(database, 'foo.jpg', 'w', :content_type => 'image/jpg') do |f|
  f.write(file)
end

# read foo.jpg from grid fs
GridFS::GridStore.read(database, 'foo.jpg')

# delete foo.jpg from grid fs
GridFS::GridStore.unlink(database, 'foo.jpg')

Not horrible but not beautiful. There is a bunch of GridFS related code on Github. To highlight a few interesting ones:

That is all for now. Enjoy the GridFS goodness and the future hope of an awesome MongoMapper plugin interface.

Twizzle Your Deplizzles

2010-01-13T19:52:42-0500

Steve and I have a Twitter account that we send all our commits to. It is all handled by Github and both of us find it really handy. Rain or shine, we get commit updates on our phone which is great for staying in the loop.

For a little while now, I’ve been wanting to add deploy notices to this twitter account and tonight I finally got around to it. It was pretty easy, but I’ll post the code here to save others time. I went the no dependencies route (even though I created the Twitter gem).

set :twitter_username, 'username'
set :twitter_password, 'password'

namespace :twitter do
  task :update do
    require 'open-uri'
    require 'net/http'
    
    url = URI.parse('http://twitter.com/statuses/update.xml')
    request = Net::HTTP::Post.new(url.path)
    request.basic_auth twitter_username, twitter_password
    request.set_form_data('status' => "Deployed #{current_revision[0..6]} to #{rails_env}")
    
    begin
      response = Net::HTTP.new(url.host, url.port).start do |http|
        http.open_timeout = 10
        http.request(request)
      end
      puts 'Deploy notice sent to twitter'
    rescue Timeout::Error => e
      puts "Timeout after 10s: Seems like Twitter is down."
      puts "Use \"cap twitter:update\" to update Twitter status later w/o deploying"
    end
  end
end

after "deploy", "twitter:update"

Just drop this in your deploy file and update the username and password. Notice that I even included the git revision and rails environment so you can easily see what was last deployed and where to. You end up with a nice little message like this:

Deployed 09ea23f to staging

Very handy!

Why I think Mongo is to Databases what Rails was to Frameworks

2010-01-04T11:36:07-0500

Strong statement, eh? The more I work with Mongo the more I am coming around to this way of thinking. I tell no lie when I say that I now approach Mongo with the same kind of excitement I first felt using Rails. For some, that may be enough, but for others, you probably require more than a feeling to check out a new technology.

Below are 7 Mongo and MongoMapper related features that I have found to be really awesome while working on switching Harmony, a new website management system by my company, Ordered List, to Mongo from MySQL.

1. Migrations are Dead

Remember the first time you created and ran a migration in Rails. Can you? Think back to the exuberance of the moment when you realized tempting fate on a production server was a thing of the past. Well I have news for you Walter Cronkite, migrations are so last year.

Yep, you don’t migrate when you want to add or remove columns with Mongo. Heck, you don’t even add or remove columns. Need a new piece of data? Throw a new key into any model and you can start adding data to it. No need to bring your app to a screeching halt, migrate and then head back to normal land. Just add a key and start collecting data.

2. Single Collection Inheritance Gone Wild

There are times when inheritance is sweet. Let’s take Harmony for example. Harmony is all about managing websites. Websites have content. Content does not equal pages. Most website management tools are called content management systems and all that means is that you get a title field and a content field. There, you can now manage content. Wrong!

Pages are made up of content. Each piece of content could be as tiny as a number or as large a massive PDF. Also, different types of pages behave differently. Technically a blog and a page are both pages, but a page has children that are most likely ordered intentionally, whereas a blog has children that are ordered by publish date.

So how did Mongo help us with this? Well, we created a base Item model. Sites have many items. Items have many custom pieces of data. So, we have an Item model that acts as the base for our Page, Blog, Link, BlogPost and such models. Then each of those defines specific keys and behaviors that they do not have in common the other items.

By using inheritance, they all share the same base keys, validations, callbacks and collection. Then for behaviors and keys that are shared by some, but not all, we are creating modules and including them. One such module is SortableItem. This gets included in Page, Blog and Link as those can all be sorted and have previous and next items. The SortableItem module defines a position key and keeps the position order in check when creating and destroying items that include it. Think of it as acts_as_list.

This has been so handy. Steve was building the doc site and said he wished he had a link type, something that shows up in the navigation, but cross links to another section or another site. I was like, so make it! Here it is in all its glory.

class Link < Item
  include SortableItem
  
  key :url, String, :required => true, :allow_blank => false
  
  def permalink
    Harmony.escape_url(title)
  end
end

Yep, barely any code. We inherit from item, include the sortable attributes, define a new key named url (where the link should go to) and make sure the permalink is always set to the title. Nothing to it. This kind of flexibility is huge when you get new feature ideas.

All these completely different documents are stored in the same collection and follow a lot of the same rules but none of them has any more data stored with it than is absolutely needed. No creating a column for any key that could be in any row. Just define the keys that go with specific document types. Sweet!

3. Array Keys

Harmony has sites and users. Users are unique across all of Harmony. One username and password and you can access any specific site or all sites of a particular account. Normally this would require a join table, maybe even some polymorphism. What we decided to do is very simple. Mongo natively understands arrays. Our site model has an array key named authorizations and our Account model has one named memberships. These two array keys store arrays of user ids. We could de-normalize even more and just have a sites array key on user, but we decided not to.

class Site
  include MongoMapper::Document
  key :authorizations, Array, :index => true
  
  # def add_user, remove_user, authorized?
end

class Account
  include MongoMapper::Document
  key :memberships, Array, :index => true
  
  # def add_user, remove_user, member?
end

What is cool about this is that it is still simple to get all the users for a given site.

class Site
  def users
    user_ids = [authorizations, memberships].flatten.compact.uniq
    User.all(:id => user_ids, :order => 'last_name, first_name')
  end
end

The sweet thing about this is that not only does Mongo know how to store arrays in documents, but you can even index the values and perform efficient queries on arrays.

Eventually, I want to roll array key stuff like this into MongoMapper supported associations, but I just haven’t had a chance to abstract them yet. Look for that on the horizon.

4. Hash Keys

As if array keys were not enough, hash keys are just as awesome. Harmony has a really intelligent activity stream. Lets face it, most activity streams out there suck. Take Github’s for example. I will pick on them because I know the guys and they are awesome. They are so successful, they can take it. :)

It may be handy that I can see every single user who follows or forks MongoMapper, but personally I would find it way more helpful if their activity stream just put in one entry that was more like this.

“14 users started watching MongoMapper today and another 3 forked it. Oh, and you had 400 pageviews.”

Am I right? Maybe I have too many projects, but their feed is overwhelming for me at times. What we did to remedy this in Harmony is make the activity stream intelligent. When actions happen, it checks if the same action has happened recently and just increments a count. What you end up with are things in the activity stream like:

“Mongo is to Databases what Rails was to Frameworks was updated 24 times today by John Nunemaker.”

On top of that, we use a hash key named source to store all of the attributes from the original object right in the activity stream collection. This means we do 0, yes 0, extra queries to show each activity. Our activity model looks something like this (obviously this is really pared down):

class Activity
  include MongoMapper::Document
  key :source, Hash
  key :action, String
  key :count, Integer, :default => 1
end

Then, we define an API in that model to normalize the different attributes that could be there. For example, here is the title method:

class Activity
  def title
    source['title'] || source['name'] || source['filename']
  end
end

In order to determine if a new action is already in the stream and just needs to be incremented, we can then use Mongo’s ability to query inside hashes with something like this:

Activity.first({
  'source._id'   => id, 
  :action        => 'updated', 
  :created_at.gt => Time.zone.now.beginning_of_day.utc
})

How fricken sweet is that? Major. Epic.

5. Embedding Custom Objects

What is that you say? Arrays and hashes just aren’t enough for you. Well go fly a kite…or just use an embedded object. When Harmony was powered by MySQL (back a few months ago), we had an Item model and an ItemAttributes (key, value, item_id) model.

Item was the base for all the different types of content and item attributes were how we allowed for custom data on items. This meant every time we queried for an item, we also had to get its attributes. This isn’t a big deal when you know all the items up front as you can eager load some stuff, but we don’t always know all the items that are going to be shown on a page until it happens.

That pretty much killed the effectiveness of eager loading. It also meant that we just always got all of an item’s attributes each time we got an item (and performed the queries to get the info), just in case one of those attributes was being used (title and path are on all items).

With Mongo, however, we just embed custom data right with the item. Anytime we get an item, all the custom data comes with it. This is great as there is never a time where we would get an attribute without the item it is related to. For example, here is part of an item document with some custom data in it:

{
  "_id"   =>..., 
  "_type" =>"Page", 
  "title" =>"Our Writing", 
  "path"  =>"/our-writing/", 
  "data"  =>[
    {"_id" =>..., "file_upload"=>false, "value"=>"", "key"=>"content"}, 
    {"_id" =>..., "file_upload"=>true, "value"=>"", "key"=>"pic"}
  ], 
}

Now anytime we get an item, we already have the data. No need to query for it. This alone will help performance so much in the future, that it alone had the weight to convince us to switch to Mongo, despite being almost 90% done in MySQL.

The great part is embedded objects are just arrays of hashes in Mongo, but MongoMapper automatically turns them into pure ruby objects.

class Item
  include MongoMapper::Document
  
  many :data do
    def [](key)
      detect { |d| d.key == key.to_s }
    end
  end
end

class Datum
  include MongoMapper::EmbeddedDocument
  
  key :key, String
  key :value
end

Just like that, each piece of custom data gets embedded in the item on save and converted to a Datum object when fetched from the database. The association extension on data even allows for getting data by its key quite easily like so:

Item.first.data['foo'] # return datum instance if foo key present

6. Incrementing and Decrementing

A decision we made the moment we switched to Mongo was to take advantage of its awesome parts as much as we could. One way we do that is storing published post counts on year, month and day archive items and label items. Anytime a post is published, unpublished, etc. we use Mongo’s increment modifier to bump the count up or down. This means that there is no query at all needed to get the number of posts published in a given year, month or day or of a certain label if we already have that document.

We have several callbacks related to a post’s publish status that call methods that perform stuff like this under the hood:

# ids is array of item ids
conditions = {:_id => {'$in' => ids}}

# amount is either 1 or -1 for increment and decrement
increments = {'$inc' => {:post_count => amount}}

collection.update(conditions, increments, :multi => true)

For now, we drop down the ruby driver (collection.update), but I have tickets (inc, the rest) to abstract this out of Harmony and into MongoMapper. Modifiers like this are super handy for us and will be even more handy when we roll out statistics in Harmony as we’ll use increments to keep track of pageviews and such.

7. Files, aka GridFS

Man, with all the awesome I’ve mentioned above, some of you may be tired, but I need you to hang with me for one more topic. Mongo actually has a really cool GridFS specification that is implemented for all the drivers to allow storing files right in the database. I remember when storing files in the database was a horrible idea, but with Mongo this is really neat.

We currently store all theme files and assets right in Mongo. This was handy when in development for passing data around and was nice for building everything in stage before our move to production. When we were ready to move to production, we literally just dumped stage and restored it on production. Just like that all data and files were up and running.

No need for S3 or separate file backup processes. Just store the files in Mongo and serve them right out of there. We then heavily use etags and HTTP caching and intend on doing more in the future to make sure that serving these files stays performant, but that is for another day. :) As of now, it is plenty fast and sooooo convenient.

Conclusion

We have been amazed at how much code we cut out of Harmony with the switch from MySQL to Mongo. We’re also really excited about the features mentioned above and how they are going to help us grow our first product, Harmony. I can’t imagine building some of the flexibility we’ve built into Harmony or some of the ideas we have planned for the future with a relational database.

I am truly as excited about the future of Mongo as I once was (and still am) about the future of Rails.

Config So Simple Your Mama Could Use It

2010-01-04T11:36:08-0500

Tonight, Kastner asked me if I had anything to do some simple configuration for something he was working on. I’ve got a simple module and yaml file that I’ve been using so I gist’d it. It then occurred to me that I might as well share it here too.

The Yaml

Below is an example of the yaml file. Basically, I setup some defaults and then customize each environment as needed.

DEFAULTS: &DEFAULTS
  email: no-reply@harmonyapp.com
  email_signature: |
    Regards,
    The Harmony Team
 
development:
  domain: harmonyapp.local
  <<: *DEFAULTS
  
test:
  domain: harmonyapp.com
  <<: *DEFAULTS
 
production:
  domain: harmonyapp.com
  <<: *DEFAULTS

The Module

The module can read and write to the config and even loads the Yaml file the first time you try to read a configuration key.

module Harmony
  # Allows accessing config variables from harmony.yml like so:
  # Harmony[:domain] => harmonyapp.com
  def self.[](key)
    unless @config
      raw_config = File.read(RAILS_ROOT + "/config/harmony.yml")
      @config = YAML.load(raw_config)[RAILS_ENV].symbolize_keys
    end
    @config[key]
  end
  
  def self.[]=(key, value)
    @config[key.to_sym] = value
  end
end

If I wanted to get the domain, I would do the following:

Harmony[:domain]

Nothing fancy, but it gets the job done. I just drop the yaml file in config/ and the module in lib/. Obviously, you would rename the module and yaml file to whatever constant you want, such as App or something related to your application’s name. I know there are gems and plugins to do configuration, but when something this simple gets the job done, I figure why bother.

What do you all use for app configuration? What do you like about what you use?