Viget Extend : The Development Lab

Provisioning & Deploying a LAMP Wordpress Stack with Sprinkle and Capistrano

2009-11-05T16:37:00Z

Recently I had to provision a number of slices for use as LAMP stacks running Wordpress installs. Creating a LAMP stack is not exactly difficult but it's also not that fun, so I decided to use Sprinkle and Capistrano to aid in my task.

In the end, I had a Sprinkle recipe and Capistrano script that saved me a lot of time. Here's what I came up with and how I used it.

Provisioning

After creating your slice / vps and adding a user with sudo privileges for yourself, clone the lamp-sprinkle repository I've created on Github and edit the contained deploy.rb file:

  git clone git://github.com/mig/lamp-sprinkle.git
  cd lamp-sprinkle/
  
  // edit deploy.rb changing:
  role :app, "host.example.com"
  set :user, "username"

Then, issue the Sprinkle command and point it at the lamp recipe:

  sprinkle -c -s lamp.rb

When the script is done running, you should have a fully functioning LAMP stack running on your slice, ready for your Wordpress or any other PHP application.

Deploying

Having used Capistrano with version control for a while now, I couldn't bring myself to deploy any manner of web application using a method like straight (s)ftp. I went searching for people deploying PHP applications with Capistrano and found Duncan's post on doing just that.

Assuming you've got your wordpress install in a git repository, place this Capfile in your local working copy.

Now just run the normal Capistrano deployment commands:

  cap deploy:setup
  cap deploy

All the normal rules apply here, your deploy user will need write permission to the deploy_to directory, and you will need an apache vhost file in your /etc/apache/sites-available turned on.

Other Ideas

You could take these two scripts and run with them in different directions if desired. The Sprinkle script could go inside your wordpress repo, use the Capistrano information in the Capfile, and contain code to configure your vhost file for you. Since I was provisioning a couple slices at the same time, I decided to go with the above, more general approach.

If you've found a nicer approach to provisioning LAMP stacks or deploying PHP applications, let me know in the comments.

(re)-Introducing simple_importer

2009-10-24T18:32:01Z

simple_importer has existed for some time as a lowly little gem that made it easy to create CSV import scripts using the Ruby CSV library. The API was usable but wasn't nearly as "simple" as it could be. Motivated by our upcoming Hackday event, Intro to Ruby Metaprogramming, I decided to rewrite simple_importer using some Ruby metaprogramming techniques. The end result is a gem that makes CSV import tasks easy to create and manage, and hopefully provides some real-world examples of basic Ruby metaprogramming.

The basic use case starts with defining importers using a small internal DSL. Each importer has a name, a csv file, and a block of code that is used to process each row in the file. The row instance that is passed into the block is a fastercsv row and all of the fastercsv configuration options are available.

importer :items do
  file 'items.csv'

  foreach do |row|
    Item.create(:name => row[:name])
  end
end

Once you have created an importer, save it into a directory named importers. This directory can be located in the same directory as your Rakefile, lib/importers or app/importers. Next, pull in the simple_importer rake tasks by adding the following line to your Rakefile.

require 'simple_importer/tasks'

Now, when you do a rake -T simple_importer you'll see a rake task for each importer as well as a rake task that will run all of the importers.

Pretty simple. Although, simple_importer does have a few other neat features such as before callbacks and multiple file processing capabilities. Be sure to check out the documentation which has plenty of examples. You can grab the gem from gemcutter, or check out the code on github.

NoSQL Misconceptions

2009-10-21T13:00:00Z

I wanted to clear up a couple of misconceptions about NoSQL that I've seen much too often lately (what can I say – I watch the Twitter results for 'nosql' obsessively).

Scalability/Performance

NoSQL is just about scalability and/or performance.

This is a particularly attractive claim for the traditionalists because it allows for the hope that these alternatives could be made obsolete in the future by making your RDBMS of choice faster or more scalable.

Unfortunately (for the traditionalist), there's a lot more to NoSQL than just performance and scaling. Most importantly (for me, at least) is that NoSQL DBs often provide better substrates for modeling business domains. I've spent more than two years struggling to map just part of the comic book business onto MySQL, for instance, where something like a graph database would be a vastly better fit.

Database Types

NoSQL is just document databases, or key-value stores, or ...

Most of the anti-NoSQL articles I've seen have addressed only document-oriented databases or key-value stores (though recently, column-oriented databases have started to show up as well). Good arguments can be made against each of those solutions for specific circumstances, but those are arguments against a specific type of storage engine. The problem with pointing to those articles as rebutting the utility of NoSQL in general is that the label is much broader than traditionalists usually understand.

"NoSQL" covers a vast set of applications, from Redis all the way up to Neo4J. Any argument that shows that, say, document-oriented databases fail in a set of situations may very well not apply to key-value stores or graph databases, and the further arguments required to make those cases are rarely (if ever) provided.

FriendFeed & MySQL

I can do NoSQL just as well in a relational database.

Well, sure, you can (and FriendFeed did, apparently) – but you shouldn't. I could roll my own relations in MongoDB, but that doesn't make it a good idea. Different applications are good for different things; relational databases are great for relational data, but why would you want to use them for non-relational data?

Never SQL

NoSQL is a wholesale rejection of relational databases.

Happily, this claim is becoming less common; many proponents of NoSQL have softened their rhetoric, and the traditionalists have become a little less defensive. It seems that we're moving towards a pluralistic approach to storing our data, and that's a good thing. I've suggested 'polyglot persistence' for this approach (though I didn't coin the term), but I also like Ezra Zygmuntowicz's 'LessSQL' as a label, too.

My Tools of the Trade

2009-10-15T14:00:00Z

Inspired by Mike Gunderloy’s tools of the trade post, I made a list of what I use every day for development. My tools don’t necessarily reflect what every developer at Viget uses.

Hardware

2009 MacBook Pro 13”, 2.53 GHz, 4 GB of memory. This machine is a screaming demon of speed and fury. I’m meticulous about keeping junk that it doesn’t need off of it. I love my laptop, although I do wish I had 8 GB of memory in it. Maybe I’ll get the upgrade for its birthday present.
20” Dell 2000FP monitor. It works. It seems giant to me, but I imagine I’d love a bigger one even more. I’ve had my eye on this 28 inch beauty for a while.
MacAlly ICEkey keyboard and Logitech MX518 mouse. Both of these are amazing. I have the same setup at work and at home. The ICEkey is my favorite keyboard I’ve ever used.

Development software

Mac OS X 10.6 Snow Leopard. Pure joy.
Ruby Enterprise Edition 1.8.7. It’s what I use on my servers, and I like to develop in a similar environment.
Homebrew packaging system. I used MacPorts for a long time, and was a big proponent, but the ease of adding new packages to this won me over quickly. Max, the project head, is great about accepting patches.
TextMate. Big surprise, I know. There’s a reason it’s so beloved in the Mac development community.
NetBeans. I’m a wishy-washy person when it comes to my editing environment, so I switch from TextMate to NetBeans often. It’s an excellent IDE for multiple languages, and gets better with every release. If I have to work on anything that isn’t Ruby, I usually use this. If it felt a little snappier and had a little better usability, I’d probably use this for all my code.
Git and GitX. Git’s my version control system of choice. I mainly use it from the command line, but when I need to do some visualization, I pull out GitX.
Mozilla Firefox. It’s my development browser. For casual browsing, I use Safari, as it’s way faster. I was a Firefox-only user for a long time, but frustrations about speed and lack of customizable keybindings has pushed me into Safari’s arms.

Other software

Backblaze. I use this for constant off-site backups. I’d link to them, but they have an awful auto-starting movie on their homepage. Instead, check out this blog post about their storage pods.
Hiveminder. This is a great task management system. My favorite parts are the command-line client and IM interface.
Notational Velocity and SimpleNote. Notational Velocity’s a great super-fast note taking application for OS X, and SimpleNote’s the same for the iPhone. I keep them in sync with a Perl script, so I have my notes everywhere.
I like to keep everything I can online and off my computer, so along with Hiveminder and SimpleNote, I use Gmail for both home and work email, Delicious for bookmarking and making groups of links, and Instapaper for things to read on the can or while waiting in line. I use GitHub and Dropbox to keep all my projects off my computer and accessible.
Fever is my RSS reader of choice. There’s too many high-noise feeds with sporadic great content out there to read them all. Fever’s Hot List makes it way easier to sift through the cruft.
To control my desktop, I use a combination of LaunchBar, MercuryMover, and SizeUp. I can launch anything, do many file manipulations, and move my windows around without leaving the keyboard. LaunchBar’s also great for giving me a global kill ring.

Tell Me What You’re Using

If you’ve got a tool you love, or think my toolset is weak sauce, let me know! I’d love to hear about it.

Deploying the git way

2009-10-14T14:30:00Z

Let’s start with this: Git is mind-blowingly amazing and simple. We’ll get back to this later.

Moving on though, a while back I was combing through my feed reader and found a post that leapt up at me: Deployment Script Spring Cleaning by Chris Wanstrath over at Github. It amazed me that no one had thought of doing this before. That is, using git as the deployment strategy for Capistrano.

A few weeks later I was preparing to get Connect-A-Sketch production ready (it’s in a private beta and you should sign-up if you haven’t yet), and the git deployment strategy jumped into my head. To reiterate what’s so great about using git, instead of rsync with git or any of the other options: it’s fast, lightweight, and can handle all the rollbacks itself with no need for timestamped directories.

The Reflog Rollback

So, I forked defunkt’s gist, and got to work polishing it up for Connect-A-Sketch. The first thing I noticed (as did a few others who commented on the original blog post) his rollback command only rolled back to the previous commit (via the ref name “HEAD^”). What we really want here is to rollback to the commit that we had previously deployed to. As it turns out there’s a ref name for that “HEAD@{1}”. So, we just replace “HEAD^” with that, and we’re all good right? Nope, because we forgot about the edge-case where we want to keep rolling back one after the other, and this can’t handle that. That’s how Capistrano normally works and so our new deployment strategy should work the same.

Well, if you didn’t know, as I didn’t, the “HEAD@{1}” ref name works because git tracks all the commits that HEAD has pointed to in the reflog (this is a really helpful feature that most other version control systems just don’t do). The deploy script uses git reset to force git to set the HEAD to a specific commit not moving through each commit or doing a rebase, or any other modifications to the HEAD reflog. But when we reset to “HEAD@{1}” that is added to the reflog, it doesn’t just pop off the last ref and move HEAD back (and don’t get me wrong this is what git should do, it’s just not what we want to do in our particular case). So we need to pop off what we don’t need from the reflog so “HEAD@{1}” will point to the correct commit now.

Lucky for us git version 1.5.6.6 and later have the ability to remove items from the HEAD reflog and correctly rewrite the reflog so everything still works correctly internally. The command to do this is git reflog delete --rewrite <ref> (see what I mean about mind-blowing!), by using this in our deploy script we can cleanup the reflog so we can continue to rollback one after another as much as we want.

Cap Compatibility

Capistrano and various cap extensions and plugins expect the releases directory and its subdirectories to exist, as well as identifiers for the current and previous versions to be present. The default values for these variables won’t be set correctly given we have no “releases” directory, so we need to make some changes to keep everything compatible.

First, we want to make sure our migrations are run in the correct directory. By default it will run it in the “current_release” directory, which is the last directory in the “releases” directory. We want it to use the “current” directory, to do that we add this line:

set :migrate_target, :current

Next, we want to make sure our deploy script sets the correct values for the various directory variables. To do this we add the following lines to our deploy script:

set(:latest_release)  { fetch(:current_path) }
set(:release_path)    { fetch(:current_path) }
set(:current_release) { fetch(:current_path) }

Finally, we need to make sure that the correct version identifier is supplied to the various revision variables:

set(:current_revision)  { capture("cd #{current_path}; git rev-parse --short HEAD").strip }
set(:latest_revision)   { capture("cd #{current_path}; git rev-parse --short HEAD").strip }
set(:previous_revision) { capture("cd #{current_path}; git rev-parse --short HEAD@{1}").strip }

With all that in place (along with a few other minor changes and additions), we have a git based deployment strategy that’s totally compatible and works just the way we expect Capistrano to work.

The complete deployment script can be found here: http://gist.github.com/176754

Developer Day Boulder Wrapup

2009-10-12T09:31:00Z

Neither snow nor rain nor heat nor gloom of night stays these developers from the swift completion of their appointed conference. As it so happened, we experienced two of those obstacles at Developer Day Boulder this past weekend – Boulder got its first snowfall of the year the night before, and the TechStars bunker got extremely warm as the day progressed (in seeming defiance of the sub-freezing temperatures outside).

We had great attendance despite the difficult conditions, and those who made it enjoyed the event a great deal. Chad Fowler started us off with a discussion of creating a remarkable life, and (judging from the tweets I saw fly by) inspired a number of the listeners to take more control of their experiences.

Chris Perkins followed Chad, and gave us a thorough introduction to TurboGears and the state of the Python web community. From there, we moved on to Rob Sanheim's talk about the cloud, and I saw my first ever live-deploy (live-coding is so last year). Jeremy Hinegardner then closed out the morning session by discussing a number of tools that work well in mixed-language environments.

After lunch, we had a great series of lightning talks. David Eisinger premiered his Introduction to Ruby and Mid-90s Hip Hop for the public, Jason Turner showed off ChaiScript, Ben Reubenstein talked about donating time and talent to worthy causes, Bobby Wilson railed against the current state of templating systems in Ruby, Jess Martin describe the One True Way of CSS, and I talked a little bit about how the comics industry is difficult to model in a relational database.

The afternoon session then continued with David Eisinger, back up to explore email as an interface. Derek Chen-Becker was next up, with an introduction to Scala and a brief demo of Lift, the Scala web framework. We then shifted to the mobile and desktop worlds, with Bill Dudney and a talk about CoreAnimation. Finally, Bruce Eckel closed us out with his perspective of the evolution of features in C++, Java, and Python.

All in all, this was another successful Developer Day, and we're excited to start planning for 2010. If you'd like to see us come to your area and showcase the various developer communities, feel free to leave a comment here or email me at bscofield@developer-day.com. Viva la Developer Day!

* Note: We'll be posting the speakers' slides to their SpeakerRate pages and to the Developer Day Boulder page as we get them in.

Wrapping Rails Session Hash: Follow Up

2009-10-08T14:03:00Z

The other week I wrote about wrapping your Rails session variables in current_object methods. There is, however, a performance related bug in line 5 of the code I posted:

class ApplicationController < ActionController::Base  
  helper_method :current_account  
    
  def current_account  
    @current_account ||= Account.find_by_id(session[:current_account_id])  
  end  
  
  def current_account=(account)  
    session[:current_account_id] = account.try(:id)  
  end  
end

If session[:current_account_id] is nil, current_account will make a useless database query each time it is called. This would happen if you are calling current_account on a public page while a user has yet to log in.

We can fix this in the following way:

...
def current_account
  if session[:current_account_id]
    @current_account ||= Account.find_by_id(session[:current_account_id])
  end
end
...

The above code will only evaluate @current_account if session[:current_account_id] has been set, otherwise it will return nil without hitting the database. Once session[:current_account_id] is set, it will hit the database a single time, memoize the result in @current_account, and return @current_account for the duration of the Rails request cycle. This should help take some load off the database.

If anyone has an even simpler solution, please let me know in the comments!

Database Taxonomy

2009-09-30T10:30:00Z

In my domain modeling/alternative database talk, I usually spend some time talking about taxonomy in biology. It's a fascinating field, with a lot of interesting tangents (ligers and tigons and pluots, oh my!), but in the presentation I focus on how difficult it can be to model in a standard relational schema. I think there's another set of lessons that people interested in databases can draw from taxonomy, however, and that's a way of looking at how databases are related.

Similarity

There are two main techniques in taxonomy for classifying things. Numerical taxonomy is the practice of grouping by similarity - dogs are more similar to bears than they are to cats, so dogs and bears fall into the same suborder (Carniformia, versus Feliformia for cats).

As it turns out, we can group databases by similarity, too. Many lists already do this at a macro level, but they don't do it systematically. I think the right approach here is to identify some axis, and plot the choices along it. Here's an example where the axis is degree of relationality the database provides (with just a subset of databases, of course):

On the other hand, we might also try grouping them based on the degree to which they require data to follow a pre-built schema:

The big changes between the relationality and schema rankings are the move of graph databases (which are schemaless both within individual nodes and for nodes' relationships), the swap of document-oriented and column-oriented databases (which, granted, may just be an opinion), and Oracle's move within relational databases (based mostly on its support for object storage, which is very close to document storage).

Descent

The other main technique in biological taxonomy is cladistics, where relationships between things are identified based on common evolutionary descent. In other words, humans are more closely related to chimps than to lemurs because the common ancestor of chimps and humans occurred more recently than the common ancestor of humans and lemurs (but I bet that common ancestor was darn cute).

In biology, cladistics presents some obstacles – we can't actually go back in time and see the divergence of species, though we've gotten really good at reconstructing lineages based on rates of mutation and the like. With databases, however, this is a lot easier. Heck, half the time it just takes a quick visit to Wikipedia, or maybe an email to the core team for the application.

Regardless of how the history is constructed, however, these trees can provide interesting and useful information. For instance, knowing the genealogy of Cassandra gives you an excellent idea of the sorts of situations for which it is designed:

Doing a full cladistic breakdown of the database landscape is beyond the scope of this article (or of any one person with a job, I think), but I welcome comments and suggestions for relationships!

Using ruby-debug to Find Those Hard-To-Reach Places (bugs)

2009-09-29T17:23:01Z

I've been writing Rails applications for a little over two years now, but only recently has ruby-debug been added to my tool belt. Since then, I've been using it almost every week and have found it super useful.

If you take five minutes to learn the six commands I'll provide for you here, I promise you won't regret it.

The Scenario

I was working with binarylogic's fantastic searchlogic gem in a project recently. I couldn't figure out why mysql "ORDER" parameters were not being produced by searchlogic's search method when chained off of a named scope. For example:

class Order < ActiveRecord::Base
  named_scope :active, 
              :conditions => {:state => "active"}, 
              :order => "delivery_date DESC"
end

Using just this named scope, the DB log would reflect this:

SELECT * FROM "orders" WHERE ("orders"."state" = 'active') ORDER BY delivery_date DESC

That looks like we would expect. However, when executing a command like this:

Order.active.search(:state_is => "onhold").ascend_by_delivery_date

Noting the DB logs, we see this:

SELECT * FROM "orders" WHERE ((orders.state = 'onhold') AND ("orders"."state" = 'active')) ORDER BY delivery_date DESC

Notice that our two states are joined with an AND (that's cool), but check out the "ORDER BY" clause. It still reflects a descending order for delivery_date – not what we wanted at all.

So how do we figure out what's going on here? We could write some tests, but I'm afraid that those tests might just confirm what we already know. There is a bug somewhere, but where?

Setup

There are many ways to go about this. The one I came up with could be clever or could just be a hack. I like to create a rake task that invokes the debugger right before the problem line of code, so I created lib/tasks/debug.rake and added this to it:

namespace :debug do
  require 'ruby-debug'
  
  desc "debug named_scope order issues"
  task :named_scope_issues => :environment do
    debugger
    Order.active.search(:state_is => "onhold").ascend_by_delivery_date
  end
end

Then in terminal:

$ rake debug:named_scope_issues

This will dump us into a debug session.

ruby-debug

What you will see will look similar to this:

rake debug:named_scope_issues 
(in /Users/mig/Desktop/store)
/Users/mig/Desktop/store/lib/tasks/debug.rake:7
Order.active.search(:state_is => "onhold").ascend_by_delivery_date
(rdb:1) where
--> #0 at line /Users/mig/Desktop/store/lib/tasks/debug.rake:7
Warning: saved frames may be incomplete; compare with caller(0).
(rdb:1)

From here we have a ton of commands at our disposal, the most important I've found are: help, step, next, list, and var local

help

Help will give you a run-down on all the other commands available to you, easy enough.

(rdb:1) help
ruby-debug help v0.10.3
Type 'help ' for help on a specific command

Available commands:
backtrace  delete   enable  help    next  quit     show    trace    
break      disable  eval    info    p     reload   source  undisplay
catch      display  exit    irb     pp    restart  step    up       
condition  down     finish  list    ps    save     thread  var      
continue   edit     frame   method  putl  set      tmate   where    

(rdb:1)

step & next

step is where you can get down into the nitty gritty goings-on of the code, it will 'step' into the first method it encounters on its current line.

next is like step, only it will evaluate the current line of code without stepping into the methods within, and simply drop down the next line of code in the current method.

list

list will print the current line of code the debugger is waiting to evaluate, along with the preceding and following five lines.

Here we are buried within some named_scope logic:

(rdb:1) list
[169, 178] in /Users/mig/Desktop/store/vendor/rails/activerecord/lib/active_record/named_scope.rb
   169        private
   170        def method_missing(method, *args, &block)
   171          if scopes.include?(method)
   172            scopes[method].call(self, *args)
   173          else
=> 174            with_scope({:find => proxy_options, :create => proxy_options[:conditions].is_a?(Hash) ?  proxy_options[:conditions] : {}}, :reverse_merge) do
   175              method = :new if method == :build
   176              if current_scoped_methods_when_defined && !scoped_methods.include?(current_scoped_methods_when_defined)
   177                with_scope current_scoped_methods_when_defined do
   178                  proxy_scope.send(method, *args, &block)
(rdb:1)

The => mark indicates the current line (174)

var local

var local is perhaps the most useful of all the debugger commands for me, it will give a value dump of all the local variables. I keep an eye on this every few lines to detect when variable values have changed.

For example, if we called 'var local' on the previous line, we would get this:

(rdb:1) var local
  args => [{:state_is=>"onhold"}]
  block => nil
  method => :search
(rdb:1)

Ahh, we can see that we've passed the evaluation of the 'active' scope, and are now evaluating the scope that search() is creating.

Conclusion

You might be thinking that this is a pretty tedious process, and it often is, but for the bugs buried deep within code it's sometimes a good way to go.

I won't take you though the entire debug session here, as it would take a couple screens, and people would probably lose interest. I will say however, that the bug was not in searchlogic's code, but in Rails itself.

Here is a simple example to demonstrate:

class Order < ActiveRecord::Base
  named_scope :by_delivery, :order => "delivery_date DESC"
  named_scope :by_creation, :order => "created_at ASC"
end

Calling:

Order.by_delivery.by_creation

Would produce:

SELECT * FROM "orders" ORDER BY delivery_date DESC

Hmm...

In the end, the issue was a bug in Rails itself, but this process allowed me to gain a better understanding of how the tools and libraries I use interact with one another. Try giving ruby-debug a whirl in your next project.

Here is a link to the Rails Lighthouse ticket describing this bug.

Hackday: Effective Testing for Rails

2009-09-28T19:01:00Z

This past Saturday we hosted the second Hackday at the Viget Labs office (photos). For this event, Matt and I focused on the tools and techniques for testing that we often use when developing Rails applications. Since the Hackday format is one of collaborative teaching and discussion, we decided to build a micro-app from scratch during a pair-programming session.

The topic for the day was a logical progression from our previous event on controller refactoring as testing is a critical component of any reliable refactoring activity. We spent some time covering the standard testing facilities in Rails and went on to discuss mocking, stubbing, object factories, and other testing libraries for use in Ruby and Rails projects.

The three-hour timeframe was sufficient to scratch the surface of these topics, but didn't allow us to get too deep in any one area. In order to bridge the gap between what we covered during the day and the final application, we've made the full application code and tests available on GitHub. We've also provided a list of links to some of the tools we discussed during the day.

Thanks to everyone who came out to the event – we hope to see you all at the next Hack Night.

North Carolina jQuery Camp Recap

2009-09-21T17:03:00Z

This Saturday Viget South hosted NC jQuery Camp ‘09 which was organized by Rebecca Murphey. The day started off at 10AM with the group making a list of possible presentation topics in typical BarCamp style.

Since some of the attendees were new to jQuery Rebecca did a little jQuery 101 with them while Scott Gonzalez went over creating stateful plugins.

Scott explained that all the jQuery UI widgets and interactions are based off of a widget module. Using $.widget you can easily create your own stateful jQuery plugin. Given that there are a number of use-cases for this style of plugin I’m making sure to keep this in mind for the next time I need to whip up a jQuery plugin.

Next, I took the floor and did an overview of jMapping, how it came to be and the code behind it. I transitioned from that into giving an overview of using Screw.Unit and Smoke to unit test your jQuery code.

Following me, our own David Eisinger captivated everyone with his talk on Perceived Performance. He’s given this a couple of times before but he added something special this time by showing us the jQuery code that runs his whole presentation, jQuinote. There’s a decent amount of JS wizardry going on there and it was interesting to see how it all works.

After lunch we were treated to Rebecca’s talk on organizing your code with object literals. This lead into a group discussion on code organization and the possibility of using jQuery custom events to also organize your code. While I’m not sold on using custom events for code organization, you should definitely look at them if you’re developing a plugin because they make it very easy to add hooks into your code.

Rounding out the afternoon a number of people came up for a “Show & Tell” session where they presented some code and/or functionality they’d written for a recent project. This included our own Clinton Nixon along with Sean Cribbs of Radiant and Lowdown.

Finally, Scott came back up to give us a little jQuery UI 101. He went over all of the various effects, widgets and interactions. He educated us on all the “easings” available for the effects. Previously, I didn’t know so many were bundled with jQuery UI. He then treated us to a preview of the positioning plugin that's going to be released in conjunction with the new jQuery plugin repository that’s in the works.

The positioning plugin is very impressive and I can’t wait for it to be released. It allows you to position an element in relation to another element via a very easy to use API:

$("#some-element").position({
  my: "left top",
  at: "left center",
  of: "#target-element"
});

While still a work in progress source code for the position plugin, an initial example, and some documentation are all available.

Throughout the day everyone had great conversations with other jQuery/JS developers about best-practices, favorite plugins, and new solutions to tough problems. It was a great event and we’re encouraged to hear that Rebecca is considering organizing another event sometime early next year. Hope to see some of you next time!

Keep Your Friends Close, But Your Test Data Closer

2009-09-18T17:38:00Z

Like any Rails developer, you've been indoctrinated into the cult of DRY and are constantly removing duplication whenever you can as you add new functionality to your application. Refactoring is an important part of the development process and improves the maintainability and understandability of your application's code.

While this is good practice for production code, the tests in your application can benefit from refactoring as well. Often, it is the setup phase of the unit testing cycle where you will encounter the most duplication. The following example was extracted from Tweets of Fury (go play now, I'll wait) – I'm using both Shoulda and Matchy for the tests:

context "An instance of the Round class" do
  should "know that there hasn't been a response from the challenger" do
    user   = Factory(:user)
    battle = Factory(:battle, :challenger => user)
    weapon = Factory(:weapon)
    
    round  = battle.rounds.create!(:challenger_weapon => nil, :opponent_weapon => weapon)
    
    round.response_from?(user).should be(false)
  end

  should "know that the challenger has responded" do
    user   = Factory(:user)
    battle = Factory(:battle, :challenger => user)
    weapon = Factory(:weapon)

    round  = battle.rounds.create!(:challenger_weapon => weapon, :opponent_weapon => nil)

    round.response_from?(user).should be(true)
  end
end

A quick refactoring might result in moving the duplication into the setup:

context "An instance of the Round class" do
  setup do
    @user   = Factory(:user)
    @battle = Factory(:battle, :challenger => @user)
    @weapon = Factory(:weapon)
  end
  
  ...
  
end

The tests can now become more focused once the setup code has been extracted:

should "know that there hasn't been a response from the challenger" do
  round = @battle.rounds.create!(:opponent_weapon => @weapon)
  round.response_from?(@user).should be(false)
end

should "know that the challenger has responded" do
  round = @battle.rounds.create!(:challenger_weapon => @weapon)
  round.response_from?(@user).should be(true)
end

Better, but not quite what I want. I've removed the "noise" and can now better focus on the code under test, but it presents me with another issue: the setup that I need to perform is missing some information that is important to keep within the test itself. I want it to be clear to anyone reading either of these tests that the user is the challenger in this particular battle, especially as I continue to add tests for this model.

In order to make this state apparent and remove duplication, I created a helper method that handles the task of setting up the data for a round:

class RoundTest < ActiveSupport::TestCase
  def create_round_with_battle(round_options = {}, battle_options = {})
    ...
  end
end

The implementation isn't particularly important – what it allows me to do is keep the important setup data within my tests:

context "An instance of the Round class with a challenger" do
  setup { @challenger = Factory(:user) }

  should "know that there hasn't been a response from the challenger" do
    round = create_round_with_battle({:weapon_for => :opponent}, {:challenger => @challenger})
    round.response_from?(@challenger).should be(false)
  end

  should "know that the challenger has responded" do
    round = create_round_with_battle({:weapon_for => :challenger}, {:challenger => @challenger})
    round.response_from?(@challenger).should be(true)
  end
end

To me, this strikes the proper balance. It gives me only the context I need while still giving me the required control over the initial state of the object under test. At this point, it may be tempting to tuck the create_round_with_battle method into test_helper.rb, but you should resist that urge. This method is only needed in this test, so it should remain here.

Getting Started with MongoDB & MongoMapper

2009-09-17T09:00:00Z

As part of our NoSQL exploration, I’ve spent some time lately with MongoDB. MongoDB bills itself as a “schema-free document-oriented database.” In using MongoDB, I’ve found it to be an easy transition from RDBMS’s because of the way it organizes document-based data. Here’s the basics:

MongoDB has collections of data, not tables. Unlike CouchDB, which is also a document-oriented DB, Mongo has namespaces for data. These are schema-less, so any data could go in each namespace. In my practice, I’ve persisted objects of one class into each collection, not unlike ActiveRecord with MySQL or any other RDBMS.
MongoDB has indexes. Even though each collection has no schema, you can still index the data in a collection based off a field. Not all documents in a collection have to have this field.
MongoDB has a query language and query profiling. While you can use JavaScript to search through a collection, like CouchDB, you also have access to a rich query language that can filter based on fields, like SQL, and filter based on the contents of embedded documents, which proves to be totally freaking awesome. Instead of a complex join, you can query for all documents in the posts collection that have an embedded comment in the last month.

Given the similarities between MongoDB and a relational database, you’d think it would be easy to use in Ruby in place of ActiveRecord, and you’d be right. John Nunemaker has created a gem called MongoMapper to work as an object mapper to MongoDB. Using MongoMapper, you can create model classes like so:

class Book
  include MongoMapper::Document

  key :title, String, :required => true
  key :author, String
  key :published_at, Date
  key :user_id, String
  timestamps! # HECK YES

  belongs_to :user
  many :chapters
end

You’ll note several things here. Keys are defined in the model, like in a DataMapper model, although they aren’t defining a schema, only a mapping for this particular model. (If the difference seems subtle, that’s because it is: MongoMapper in many ways lets you treat MongoDB as a relational DB.) The keys can be typecast as I’ve done, although they don’t have to be. I’ve defined relationships to other models, and MongoMapper is smart about this. In the case of many :chapters, it looks to see if the Chapter class is embeddable. If so, it will embed Chapter documents in my Book document. If not, it will store them in their own collection.

Just because MongoMapper defines a document with keys, you don’t have to stick to the keys. Because collections are schema-less, you can add new attributes at will, like in this example:

book = Book.new(:title => "Moby Dick 2")
# => #<Book _id: , title: Moby Dick 2, author: >

book.author = "Dan Brown"
book.update_attributes(:author => "J.K. Rowling", 
                       :isbn => '1-2345-6789-0', 
                       :amazon_score => 1.25)
book.save

book = Book.find_by_title("Moby Dick 2")
# => #<Book _id: 4aafe487477a51f0e8000002, 
#         title: Moby Dick 2, 
#         author: J.K. Rowling, 
#         isbn: 1-2345-6789-0, 
#         amazon_score: 1.25>

You can see that I can set keys defined in the class with setters, but I can set any attribute through update_attributes.

MongoMapper’s API is roughly equivalent to ActiveRecord’s, allowing you to use in a Rails application with little difficulty. The only things I’ve had to do are define human_name on model classes and define new_record? on embedded documents.

The only other thing you need to know to get started with MongoMapper is how to tell it what database to use. All you have to do is set MongoMapper.connection and MongoMapper.database. In my sample Rails app, I’ve put a file in config/initializers/ that looks like this:

db_config = YAML::load(File.read(RAILS_ROOT + "/config/database.yml"))

if db_config[Rails.env] && 
db_config[Rails.env]['adapter'] == 'mongodb'
  mongo = db_config[Rails.env]
  MongoMapper.connection = Mongo::Connection.new(mongo['hostname'])
  MongoMapper.database = mongo['database']
end

You can see my database.yml file for more information on setup or check out Ben Scofield’s Rails template for MongoMapper.

That should get you started! I’ve really enjoyed using MongoDB so far. For further information, checkout the MongoDB Ruby driver code, the MongoMapper code, and the code for my sample app on GitHub, and look out for more upcoming posts about how we’ve used MongoDB.

Wrapping Rails Session Hash

2009-09-16T18:42:00Z

The best way to use session data in Rails is sometimes a bit elusive. One technique I see from time to time is using a before_filter to pull data from the session, and store it in an instance variable.

class AccountsController < ApplicationController
  before_filter :find_account
  
  ...
  
  def find_account
    @account = Account.find(session[:account_id])
  end
end

The problem here is that the more places you start using the @account instance variable, the more confusing it gets. Suppose we wanted to reuse this before_filter, and so we move find_account to our ApplicationController. This would further obscure the location where the instance variable is being set.

Another drawback to using a before_filter is having to tediously define where you want the filter to run or not-to-run (using skip_before_filter). It would be nice if we could have that data only where we really needed it without worrying about declaring which actions have access to session data and which don't.

A Better Approach

A pattern I like to use when working with sessions in rails is wrapping session variables in getter and setter current_object methods. I picked this up a while ago while using RestfulAuthentication, I've found since that I use it in almost every project. Observe...

class ApplicationController < ActionController::Base
  helper_method :current_account
  
  def current_account
    @current_account ||= Account.find_by_id(session[:current_account_id])
  end

  def current_account=(account)
    session[:current_account_id] = account.try(:id)
  end
end

I am using find_by_id(id) because it returns a nil value when the record is not found rather than an exception, which is what is returned from an unsuccessful find(id). Keep this in mind, as you may prefer to throw an exception over a silent failure.

Using the conditional assignment method in our getter allows us to lazily load database records. It's also worth noting that the current_account method always returns a model instance, we only store the record's id in the actual session. This keeps our session data as small as possible.

I've declared these methods as helper methods inside the ApplicationController so that I can use them in my controllers and views alike. I find a call to 'current_account' in the view is easier to understand than an instance variable that is set somewhere other than the view's corresponding controller action.

Notice the use of Object#try when calling id on the account argument passed to current_account=. Using Object#try in this situation allows us to set session[:current_account_id] to nil using the same current_account= setter method. This is often seen in logout actions:

class SessionsController < ApplicationController
  ...
  def destroy
    self.current_account = nil
    redirect_to root_path
  end
end

A Gotcha

Remember to prepend the setter method with 'self' when using it in your controllers, else Ruby will make a local variable assignment to 'current_account' the variable, which is different from 'current_account=' the method. Obvious, perhaps, but it trips me up from time to time.

class AccountsController
  def create
    self.current_account = Account.find(params[:id])
    ...
  end
  ...
end

If anyone has other nice ways of handling session variables in Rails, let me know in the comments!

Viget Devs Storm Chicago

2009-09-16T00:58:00Z

This past weekend, Ben and I travelled to Chicago to speak at Windy City Rails. It was a great conference; highlights included Dean Wampler’s discussion of functional programming in Ruby, Noel Rappin’s talk on handling difficult-to-test portions of your applications, and our own Ben Scofield’s very polished presentation about modeling difficult domains. This was the second time I’d seen Yeduda Katz’s keynote on the history and future of the Rails framework, and that the talk has come so far in the last three months says promising things for the progress of Rails 3.

Speaking at Windy City Rails was just as much an excuse to hang out in Chicago as it was a chance to share knowledge with the Ruby community. We knocked out the major food groups on the first day, and I set aside a few days after the conference to wander the city. Big thanks to the conference organizers and attendees. Videos of all of the talks will be appearing on the conference website soon.

Herding Developers

2009-09-10T11:00:00Z

Developer Day has taken the east coast by storm, with great receptions in Durham, DC, and Boston (check SpeakerRate for all the details). As we reflected on the next steps, then, the natural progression was to move west – so that's what we're d oing. On October 10th, Developer Day will be thundering into Boulder, Colorado (home of TechStars, and a number of exciting companies and startups).

We've started to finalize the program, and so far it looks great. There's a huge pool of developer talent in Colorado, and we're aiming to pull in as much of it as we can to speak at the event.

If you'll be anywhere near Boulder around October 10th, you should definitely plan to attend – early bird registration is only $50, which will get you a full day of great talks, smart people to chat with, breakfast, lunch, and a t-shirt. Who could resist that deal? Register today!

Coming Soon: The NoSQL Series

2009-09-02T11:00:00Z

We're always excited about new trends here at Viget, and the most recent phenomenon that's caught our attention is NoSQL - the reaction against traditional relational databases like MySQL. The movement has been characterized by an explosion of interest in key-value stores, document-oriented databases, and other (mostly non-relational) alternatives for data storage. Several of the developers here in the Labs have been experimenting with these emerging systems, and over the next few weeks, we'll be posting our thoughts.

We'll show examples of situations in which, say, MongoDB works particularly well - and we'll also show situations in which you're much better off staying with something more traditional. In fact, that's the conclusion we've drawn from the whole thing: just as many different programming languages can coexist, with each being useful for some particular set of problems, many different databases can coexist - with each being useful for some particular set of domains. Even more interesting to me, though, is the set of complex domains that can be best served with multiple database systems... Hey, if Twitter can rely on Scala and Ruby, why can't we build an application with MySQL on top of CouchDB?

In the course of this series, we'd love to hear your feedback - how are you using these alternatives to traditional databases? Have you abandoned Postgres to Tokyo Cabinet, or tested with Neo4j? How are they working for you?

Look for the first post in the series next week!

Prototyping Java-Processing in Ruby-Processing

2009-08-28T17:53:00Z

It's no secret that I spend copious amounts of free time trying to create art with my laptop. Not too long ago I gave a talk on using Lily and the Monome to generate algorithmic music. More recently, I've been obsessed with learning Processing, an animation and interaction programming language. It has the capability to generate both algorithmic music and visualizations and it runs on the JVM so it's quite fast and portable. The language itself provides a huge library of functions that serve as abstractions to the intricate and tedious world of graphics programming in Java. That said, it isn't really its own language per se. At the end of the day, coding in Processing is coding in Java. Fortunately, Jeremy Ashkenas has created a JRuby interface to the Processing API called ruby-processing. I've been using both, together, at the same time to help improve both my Ruby code and my Java code.

The Concept

The Java based version of Processing runs faster than ruby-processing and this is no surprise as there is an inherent performance cost when translating Ruby to Java. On the other hand, I can code faster in Ruby. I've been combining these two forces and prototyping my Processing applications in ruby-processing and then porting them to Processing. Along the way I've realized that this technique is a powerful aid in understanding object-oriented fundamentals. It has also increased my appreciation of Ruby as it demonstrates how features of the language map to Java code structures with surprisingly less syntax and greater flexibility.

Examples

Here's an example based on a Processing simulation I've been working on. My simulation is based on this video, by Robert Hodgin who blogs at flight404. We'll look at a small subset of the app for demonstration purposes.

Modules and Base Classes

 module Movable
  def move; end
end
class Magnetism
  attr_accessor :particles
  def act; end
end
class Gravity
  include Movable
  attr_accessor :particles
  def act; end
end
class Particle
  include Movable
end

A little explanation is in order here. Magnetism takes a collection of Movables and applies a magnetic field equation in Magnetism#act. Gravity does the same thing except it uses a gravitational equation. Here we see that instances of Particle and Gravity are both Movables. I wanted to be able to have a bunch of small magnetized particles influenced by gravity, as well as a handful of gravitational objects that were magnetized with respect to each other. Watch the flight404 video if that isn't totally clear.

My Java solution is essentially the opposite of what I talked about in an older blog post, Inheritance is Unlikely. I created a Movable base class that Gravity and Particle inherit from. Inside Magnetism#act I cast the objects in the particle collection to the type Movable.

Duck Typing and Interfaces

 class Particles
  def display
    particles.each do |p|
      p.display
    end
  end
  def move
    particles.each do |p|
      p.move
    end
  end
 end

While Particle and Gravity instances get their movement behavior from the Movable module, they both implement different display behavior. Particles, which is the particle collection class, contains an array of instances that respond to both #display and #move messages. My goal was to use this as a collection for both Particle and Gravity instances. In Ruby, this happens automatically. As long as the instance p passed into the #each block responds to #move or #act I should be able to use these delegation methods without any trouble.

interface IDisplayable {
  void display();
}

class Gravity extends Movable implements IDisplayable {
  void display() {
  }
}

class Particle extends Movable implements IDisplayable {
  void display() {
  }
}

class Particles {
  void display() {
    for(int i = 0; i < particles.size(); i++) {
      IDisplayable p = (IDisplayable)particles.get(i);
      p.display();
    }
  }
}

In Java, the solution is a bit more tricky, at least for a Java novice such as myself. I created an IDisplayable interface that defined the display method signature and made Gravity and Particle implement that interface. This particular approach helps concretize the idea that adding public methods to a Ruby object implicitly builds an interface. The duck typed nature of Ruby somewhat hides this aspect but in porting to Java the concept of interfaces becomes explicit.

Lessons

Clearly, these two examples aren't groundbreaking but the value of this approach is in the actual practice. Looking at your Ruby code and thinking about what various interfaces your objects conform to will help you model more flexible and reusable classes. In the event your end goal is flexible Java code, prototyping a class hierarchy in Ruby can help you build in the flexibility of Ruby without having to mess with explicit interfaces until everything is working the way you want. Processing itself is a great environment to try out something like this as the boiler-plate overhead of creating a running Java application is well hidden within the Processing internals.

Regarding Recent Events

2009-08-17T12:30:00Z

We’ve been busy little event organizers here at Viget lately, with a couple of iPhone Hack Days (one at each office) and Developer Day Boston all coming in the last several weeks. In case you missed ‘em, here’s how things went down.

iPhone Hack Days

We’ve been seeing a lot of interest in iPhone development, both from clients and from developers - and there is a flurry of activity going on in the framework space, with Rhomobile, PhoneGap, Titanium, and more all competing to make iPhone development easier. It made perfect sense, then, for me to run a couple of half-day-ish workshops on the platform, both introducing Objective-C and the process for creating native apps (and walking people through the code for my own application in the process), and digging in deeper to a couple of the frameworks.

At the HQ Hack Day, we just had Viget developers (and a lone UX guy) working, but we had a ton of fun. For the Durham version, we invited in a few local developers to bring in people with diverse backgrounds and interests. Brent Collier posted his thoughts about the event, and we got some nice comments on Twitter, as well:

@brentmc79 At Viget Labs hacking around with some iPhone development... http://yfrog.com/2rnhij - link
@rmurphey great day at iphone dev camp @viget -- obj c ugh. titanium and phonegap ftw. - link
@brianjlandau Learning a ton about iPhone development with all the @viget south devs from @bscofield - link

Developer Day Boston

On August 15th, I was up in Boston along with Jess Martin and Stuart Halloway from Relevance to organize (and speak) at our first Developer Day outside of DC and Durham. We were fortunate to have the help of thoughtbot and Microsoft (which allowed us to hold the event in an amazing space at their NERD Center), and with all of that we had a great time. People learned about JavaScript testing from John Resig, about the wonders of CSS 2.1 and 3 from Marc Amos, about git from Nick Quaranto, and more - check out some of the comments:

@vandrijevik @thoughtbot's flutie is sexay! Enjoying @croaky's talk and demo of heroku-suspenders. #developer_day - link
@wrycoder Compelling pitch for document databases by @bscofield. Relationships as first class objects. May even check out some comics. #developer_day - link
@PhilDarnowsky Continuous Integration: writing better applications by harnessing the power of shame. #developer_day - link
@SaraJChipps @LostInTangent right now I'm learning about box-shadow it's freakin rockstar. css 3 stuff. #developer_day - link
@Croaky microsoft mvc framework looking sharp these days. rails-like, cheaper tools than The Old Days, includes jQuery. #developer_day - link
@marcamos Just heard about the Selenium plugin for Firefox, via @jeresig, here at #developer_day. I'm in love. - link
@csspixel #developer_day was awesome. Met some great people. - link
@rkalajian I'm finally home. #developer_day kicked ass. Met some great people, listened to great talks. Can't wait for next year. - link

Every Developer Day adds a little something to the previous ones, and Boston was no exception. We started videotaping talks, and after some review we’re hoping to post them publicly. This is also the first Day where we had shirts for the attendees - take a look!

Plans for the next Developer Day are underway, so keep watching the web site for details - and a new design, coming very soon.

Materialize Views in the Rails Application Layer

2009-08-11T18:04:00Z

In my previous post I talked about how we used a VIEW to act as an auto-updating join between our Group and Entry tables on FeedStitch. After the changes were made there was a concern that as the view grew there would be a significant slowdown in selecting rows.

Identifying our Problem

In general, the results of running EXPLAIN on the VIEW we have set up is that, in order to find a row, the VIEW must be sequentially scanned. This is telling in two ways. First, sequential scans are pretty much the single least efficient way to find a row (unless you're lucky and it's the first row). And second, without indexes, this is the only way to find a row. The primary drawback in using views is that they lack the ability to have indexes, as the data isn't really persisted as we see it.

Moving Forward

The answer to our index problem may be solved with something called "View Materialization". Simply, the view, which contains a complex SELECT would be stored in a table which can be indexed as needed. Sounds simple, right?

The Tricky Parts

Unfortunately, as with any caching mechanism, this one must be expired and updated as the data changes, to keep it fresh. To do this at the database level requires functions and triggers, often a complex task.

The Simplest Thing That Works

Fortunately, in Rails, or with any ORM, we have a sensible place to put this sort of logic. And, with a little SQL written, we can refresh our materialized view with a semblance of ease. The two benefits to this are simplicity and ease of maintenance.

It takes a little bit of planning, but the first thing to do is to understand at what points the data in the VIEW will change. In our case, when we run a rolling update to fetch new feeds we can invalidate and re-materialize the entire thing. This is not very efficient in and of itself. However, it is simple (call destroy_all, only one SELECT) and does what it needs to do. Given that this only happens every few hours or so, this is a feasible option.

An even better solution would be to invalidate (delete) a portion of the materialized view and only SELECT a changed subset. This is better because, at least in our case, we never truly need to re-materialize the entire VIEW. In fact, we never really need to delete any records besides the subset we're working with.

A Little Setup


  ALTER VIEW group_entries RENAME TO group_entries_view; -- Or recreate the view in MySQL
  CREATE TABLE group_entries AS SELECT * FROM group_entries_view;
  ALTER TABLE group_entries ADD PRIMARY KEY (group_id, entry_id);

Updating the Table

We currently have a VIEW which selects a group_id and an entry_id. FeedStitch's updater fetches new entries for a group. Once done, the VIEW will hold an updated set of entry_ids for the group_id we are working with. This must then be materialized into a table. First we delete the rows give the group_id then we INSERT new rows from our VIEW:


  GroupEntry.connection.execute <<-EOSQL
    BEGIN;
    DELETE FROM group_entries WHERE group_id = 1;
    INSERT INTO group_entries SELECT * FROM group_entries_view WHERE group_id = 1;
    COMMIT;
  EOSQL

Though not always applicable to performance issues, materialized views can be useful way to cache frequently selected data. This simple method by which to materialize a view and update from within our application code gives us this functionality in a more accessible, flexible way.