Wood for the Trees

Metaprogramming.apply(Aspects)

2007-10-09T21:51:00-05:00

I’m just briefly breaking my silence (which must continue a bit longer) to raise awareness about an impressive and very useful gem which has just come onto the scene: Aquarium.

Although I don’t follow Aspect Oriented Programming very closely, I know about its benefits and use it when necessary. All Rails developers, for example, employ its principles without even knowing it: before, after and around filters, proxies to alter execution, callbacks and the like. If you don’t know about AOP yet, basically it’s the practice of separating components which appear across the various classes of an application to provide system or class-agnostic functionality—for example, logging, authentication or object persistence.

Having these components intrude on, for example, calculating a bank balance really gets annoying after a while. Imagine every moderately sized script you write needing lots of diagnostic messages, custom event handlers, and the like. It gets very tedious because you know there’s the ideal of reuse just around the corner.

Clean, Ruby-esque AOP… a Pipe Dream?

Imagine methods without calls to a logger, without a litter of ‘if/then/else’ for authentication or authorisation, without calls to save an object and checking if it returns true or false… What you’ll get is the raw functionality of your class or module, which… umm, is as it should be. Wrapping a method (using cuts, wrapping or aspects from AOP) lets you extract common functionality which cuts across objects.

Until now, there’s only been the Facets libraries for AOP (‘advice’, ‘aop’ and ‘cut’ in Facets 2.0.0) and AspectR. The former is buggy at best and the latter is like returning to Java. So I wasn’t very keen on AOP in Ruby at all. Metaprogramming seemed easy enough and didn’t need the structure of AOP to follow.

Enter Aquarium. It’s a glass of clean water in a desert. Ruby-style. The most important thing it offers is a ‘Join Point Model’ which will make you wet. In other words, it is very easy to intercept method calls to wrap them and introduce functionality before or after the original definition.

Sadly, I don’t have the time for an extensive example from my own coding, so I’ll have to rip off the example from Aquarium’s own README which I think illustrates its power best (with a few modifications to show how I use it at the moment): method tracing.

Example: Method Tracing


require 'rubygems'
require 'aquarium'
require 'aquarium/aspects/dsl/object_dsl' # makes the Aspects DSL (before/after/around) system-wide

module Example
  class Foo
    def initialize *args
      p "Inside:   Foo#initialize: args = #{args.inspect}"
    end
    def do_it *args
      p "Inside:   Foo#do_it: args = #{args.inspect}"
    end
  end

  module BarModule
    def initialize *args
      p "Inside:   BarModule#initialize: args = #{args.inspect}"
    end
    def do_something_else *args
      p "Inside:   BarModule#do_something_else: args = #{args.inspect}"
    end
  end

  class Bar
    include BarModule
  end
end

p "Before advising the methods:"
foo1 = Example::Foo.new :a1, :a2
foo1.do_it :b1, :b2

bar1 = Example::Bar.new :a3, :a4
bar1.do_something_else :b3, :b4

# EXAMPLE 1: Wrapping methods on multiple classes using the powerful 'types' matcher.
# It can take constants or regular expressions. Also demonstrates method_options, which
# can specify public, protected, private, inherited or local methods.

module Example
  include Aquarium::Aspects::DSL::AspectDSL
  around :types => [Foo, Bar], :methods => :all, 
  :method_options => :suppress_ancestor_methods do |jp, *args|
    begin
      p "Ex. 1: Entering: #{jp.target_type.name}##{jp.method_name}: args = #{args.inspect}"
      jp.proceed
    ensure
      p "Ex. 1: Leaving:  #{jp.target_type.name}##{jp.method_name}: args = #{args.inspect}"
    end
  end
end

# EXAMPLE 2: Wrapping one or more methods very simply.
module Example
  class Bar
    after :do_something_else do |jp, *args|
      p "Ex. 2: Simply after: #{jp.target_type.name}##{jp.method_name}: args = #{args.inspect}"
    end
  end
end

# EXAMPLE 3: Using the Aspect class without the DSL

include Aquarium::Aspects
Aspect.new :before, :method => :do_it, :type => Example::Foo do |jp, *args|
  p "Ex. 3: Simply before (using Aspect.new): #{jp.target_type.name}##{jp.method_name}: args = #{args.inspect}"
end

# EXAMPLE 4: Contain aspects within a namespace for clarity. These are executed
# upon loading the source.

module Example
  module AllAspects
    include Aquarium::Aspects::DSL::AspectDSL
    before :do_it, :type => Example::Foo do |jp, *args|
      p "Ex. 4: Simply before (using namespaces): #{jp.target_type.name}##{jp.method_name}: args = #{args.inspect}"
    end
  end
end

# And now see the result...
puts
p "After advising the methods. Notice that #intialize isn't advised:"
foo2 = Example::Foo.new :a5, :a6
foo2.do_it :b5, :b6

bar1 = Example::Bar.new :a7, :a8
bar1.do_something_else :b7, :b8

You could argue that the namespace for Aquarium needs a little work and the availability of Aspect.new system wide should be a given, but adding that oneself until Aquarium has it is no big deal. I also didn’t demonstrate pointcuts, another option to around, before and after. With a pointcut, you can encapsulate the type, scope and method names—perhaps as a constant, class variable or class method to reduce dependencies.

Overview

All in all, the DSL is already miles better than anything else out there and Aquarium was only released to the public months ago. To summarise, I think this DSL is killer because it is…

clear about what is affected
easy to separate from what is affected
rich with selectors (type, scope, name, pointcut)
able to choose whether to alter method execution

So, I would definitely suggest giving it a go and seeing how it can modularise some of your ‘cross cutting concerns’. I also suggest the Aquarium README and the reading recommended on its site, http://aquarium.rubyforge.org/.

Distributed SCMs: Be Smart, Use the Bleeding Edge

2007-08-05T00:38:00-05:00

I’ll preface this title by admitting I’ve not paid much attention to my version control software. When you’re smart (cue 16-ton weight), you try to think ahead and choose software or hardware which will vastly reduce the effort of programming. While staying on the edge with most things, for version control I neglected to really think about it. I had a very brief and painful time with CVS and promptly converted to Subversion because it was already installed, everyone used it, it was much easier, it solved the immediate problems with CVS I had, and it did what I needed to do at the time.

Of course, in programming, we all know that “what I needed to do” is never the same as “what I need to do now”. I have a lot more code to manage, I have a lot of projects which I am developing on my own or manage or take a serious part in, and I have more projects which I want to dip my feet into. The number of snippets, little things here and there, and feedback or changes from other people has meant a lot of branches, tags, svn cp and other management issues.

Now, this might sound like a “shiny thing” moment, but it is completely different from shiny thing syndrome. My advice to practically everyone reading this is to keep moving at the cutting edge of version control… and that doesn’t mean upgrading to Subversion 1.4.4. It means that there are genuine advances in version control which go well beyond Subversion and well beyond what it will ever be. I’ve known about distributed SCMs for a while, but haven’t really bothered looking into it. I’m writing this post because I think a lot of people are in the same boat. Subversion is supported, if not as the default SCM, practically everywhere: Sourceforge, Rubyforge, Google Code, Unfuddle, Lighthouse, etc. Every respectable project hosting service uses Subversion and almost every plugin, gem or developer out there at the moment references to Subversion repositories. Well, I’m going to make a serious splash in this department and say: it’s time to move on from Subversion. I have completely grown out of it, and I’m not even managing a codebase as large as Rails (or even ActiveRecord).

The single biggest development in version control over the last umpteen years, but specifically the last 2, is distributed version control. Basically, that means everyone’s working copy is a repository, every commit is local and every ‘commit’ to another repository is a ‘push’ (every ‘update’ from another repository is a ‘pull’)—that means merging is the default behaviour when you communicate with another repository. In other words, every repository is a branch. This way, distributed version control doesn’t force a single source code management architecture. These systems are completely open and the advantages to this model-less model are (obviously) endless:

most importantly, distribution is all about caring more about the coder-to-code relationship than the coder-to-project relationship
Distributed SCMs are orders of magnitude faster in almost every department
they usually use a bit more space
actual commits are almost instant, since they are local
merging is simplified enormously—it’s the daily business of a DSCM
no need for an update—you merge when ready
development can happen anywhere—no connection required
no default model of management or hierarchy
much more intuitive code structure—do what you like
no need for access control, but it’s there if you need it
etc.

You’ll see the benefits just in the development of your own code, without merging with anyone else’s repositories but your own, that distribution of management doesn’t require distribution of code. You can use any methodology, any hierarchy, any paradigm of managing your code that you wish.

Returning to my original point, staying at the bleeding edge doesn’t mean upgrade to the next minor or major version to get the latest feature. It means upgrade, or change software if necessary, to use the latest breakthroughs which will cut costs and save time for your own projects or your company’s projects.

Now, you’re probably asking which distributed SCM to use. I personally go for Mercurial for the following reasons:

SVK is a lie—it is not distributed; it just distributes SVN, which is like a multiplying a herd of turtles
much faster than Arch, Darcs and Bazaar
only Git is faster—but that was developed by Linus Torvalds
Mercurial and Git are relatively younger, meaning they’ve learnt from the mistakes of other DSCMs
Mercurial and Git come from the same background (the BitKeeper drama) and BitKeeper, according to Torvalds, was the only SCM worth using
Mercurial compares feature-for-feature with other DSCMs, and then some
Mercurial uses less space than others
very user friendly CLI—by far the easiest to learn of the DSCMs
decent documentation
a few project hosting sites already (like ShareSource)
ability to have Mercurial support using the HTTP front-end, even where you normally wouldn’t have it (i.e. Sourceforge and Rubyforge)
If you don’t have a feature in Mercurial, find or code a Mercurial extension—it is fully customisable

My Forceful Conclusion

Like the language you are using (Ruby 1.8), the framework (Rails 1.2), the testing suite (RSpec 1.0 and Mocha), the operating system (Mac OS X 10.4 and Linux 2.6), the web server (nginx, LiteSpeed, or lighttpd), the application server (LiteSpeed or Mongrel), the blogging software (Typo or Mephisto), even the hardware (Amazon S3/EC2 or many dual core 2+ GHz servers with many GBs of RAM), you need to use the latest version control software (Mercurial 0.9.4 or Git 1.5). If you don’t, it’s like using an old server with 333MHz & 256MB RAM, or Apache 1.3 & mod_ruby. But unlike an out-moded application server, SVN will make you a slower developer than the one sitting next to you using Mercurial, Git, Darcs or Bazaar. Oh, and I needn’t mention that he’ll get all the babes too.

Positive Reasons for Typo

2007-07-30T20:11:00-05:00

I highly respect quick responses to a post. In Simeon’s post, he laments my lack of a positive reasoning for choosing Typo, so I’d like to quickly outline why I originally selected Typo and why I was drawn back to it.

Typo leverages the modern technologies that are out there and, true to a Rails application, has a love of shiny things without overdoing it. Nevertheless, it is fast (70 req/sec is my entire readership in a second, which makes me feel very warm and cuddly). Despite the speed, it’s not stripped down at all. What it comes with built-in (every social networking/bookmarking sidebar imaginable, multiple feed formats, multiple database support) is easily supplemented by its easy-to-use plugin architecture. It now uses the Rails plugin architecture, which makes it even easier. Adding themes is a doddle. Even editing the core code gives you the result you want, more often than not.

Basically, Typo is written to offer complex features using the simplest code. There are areas you could quibble with me on that score, and I may agree, but generally Typo is a sleek and sexy blogging engine with lots of features and yet which still has limitless potential. Just look at their Trac. The active development suggests a project which is going somewhere and getting there fast. I stopped using Typo after 4.0 and came back at 4.1. Already I see a lot of changes under the hood that would suggest more than just a minor version change. Even with all these changes, reverting back to Typo took me no less than 10 minutes. That says a lot to me.

Sure, WordPress and other blogging engines may have plugins or built-in features which rival those in Typo, but the sheer speed and flexibility of development behind Typo is peerless. I like that. It feels edgy.

I’m no old man and I don’t need Zimmerframe 2.2.1. Give me the bloody nose brawls, the midday groggy writer’s block, the street-vomiting hard nights, the raving ecstasy of Typo any day. She’s the kind of girl you can take out for a walk in the park or on a bruiser’s night and return home proud and glowingly post coital, because she’s always good fun, day or night.

Typo’s no dumb blonde.

Seriously Sick of WordPress -- Back to Typos

2007-07-30T12:02:00-05:00

I should have listened to myself ages ago when I discovered Ruby and said to myself: “I hated PHP before there was an alternative. I hate PHP even more now that there is one.” So why did I convert from Typo to WordPress? I really, really do not understand why there are more posts about converting from Typo to Wordpress than vice versa.

My dear peers, pelt me with rancid tomatoes. I don’t know what I was thinking when converting to a PHP monstrosity. WordPress is such a bloated and badly coded blog engine (and that is much due to the way PHP itself encourages you to code). Every time I try to dig into its code, I become nauseous and frustrated, despite years of PHP experience.

What brought me to this realisation is an innocent attempt to upgrade… the quality of a software distribution and its development practices always reveal themselves when you upgrade, because it is in the care and attention to backwards compatibility which show developers have really considered what they are doing, not just chasing the new shiny thing.

So in trying to upgrade from WordPress 2.0 to 2.2, I found my entire day wasted chasing down errors, discovering none of my plugins will work, finding my theme broken in numerous places, and just plain frustrated by the idiotic process of removing, moving, copying, overwriting and double-checking files and directories. Upgrading WordPress is the most painful form of manual labour I’ve experienced in a while.

So, without further ado, my list of complaints before saying Adieu! to WordPress:

Biggest of all: upgrading is a serious pain in the arse, even with my upgrade.rb script (notice the irony)
Upgrading breaks almost every plugin you use
Plugin compatibility is atrocious. There may be a lot of them in their plugin repository, but look at the fine print and you’ll see they range from pre-1.x to 2.2 compatibility, and that doesn’t mean 2.2 works with 1.5 or vice versa. You need the right version or, more likely than not, it won’t even be recognised. So, really, each version has more like 30 plugins available, not 3000.
With every upgrade, you are welcoming security bugs (the worst kind) and I just don’t like the idea that upgrading to fix one is just introducing a new one. The reason for this is that WordPress is not a framework; its core functionality is part of its higher functionality, and with every upgrade they are mucking with both. I’d suggest to the WordPress development team that they extract the underlying HTTP management and the overlying CMS functionality… but then, why not use something like Django, CakePHP or Rails?
Their plugin architecture is horrendously 90’s
The admin interface is hideous and unintuitive—where do I set pings and trackbacks again? Why do plugins enable new menus in weird places? I can never remember where anything is because nothing is organised properly
Creating and editing posts is so slow I find myself making a 10-cup cafetiere every time.
Does WordPress send pings? I never see my blog appear after a post. I can only assume it doesn’t, even when I tell it to.
The WordPress helper functions are megolithic and cryptic. I can’t cuss them more, really. And they don’t even cover every situation. I’d rather have thousands of little ones to learn that three which do vastly different things, none of which do what I want. That means editing themes is a serious pain.
Lastly, I just despise PHP and giving it support by using it.

But this isn’t just about me whinging. I have positive reasons for moving back to Typo:

It’s written in Rails, which lets me do whatever I want with it, in a pleasurable way. I know Rails well enough to muck around. All I need to do is learn how Typo works, and it isn’t that complicated—nothing written in Rails, that I’ve encountered, is too complicated to grok in a day or two.
Typo 4.2 is going to introduce multiple blogs within a single installation, which I really would like to have. So I’m in for the long haul.
The latest Typo (4.1) uses Rails 1.2, which was one of my concerns with 4.0.
The rumour of an admin interface redesign is rather thrilling, but then, I still really like Typo’s admin interface as it stands. I’m very curious what they’re going to do.
Typo now uses Rails plugins to do everything. I intend to write a FeedBurner plugin that lets you do all sorts of things, like FeedFlare, Feedburner redirects and other nifty FeedBurner stuff. Now that it is easy to write a plugin, I’m keen to test it out and prove its worth.

So, back to Typo 4.x. Let’s hope its developers sorted out some of the issues that put me off it and caused me to so whimsically (and stupidly) change to WordPress.

Note: Over the next week, my dear readers, you may encounter a few problems. These are only temporary as I muck around with Typo 4.1 and juice it for all its worth.

Oh, and thank you for waiting the few months while I was deeply involved with a major project. I’m back permanently now, on freelance. It’s good to be free and back on the Rails.

Project Management and Best Practices in Retrospect-iva

2007-07-27T21:18:35-05:00

I don’t hear a lot about project management, even though there’s a lot about how to manage a project. Testing, deployment and source code management get the most attention, and project management seems to get the least. Maybe that’s because it hasn’t been done properly yet and all the solutions out there only address pieces of the overall problem.

So I’m going to try to clarify, for myself mostly, the kind of project management that is needed and why it is so important in development.

Getting It Right

When I say management, I mean a combination of something like Lighthouse and Basecamp, with a serious overhaul of perpsective. An integral part of good development is developing ideas hand-in-hand with the code. Management is all about keeping this communication as agile as the coding process, sticking to priorities, and addressing the right things at the right time.

Is Basecamp sufficient for managing a project? No. DHH even says it is not meant for managing Rails projects; it’s for marketers and managers. It’s only a piece of the puzzle, because it provides no way to easily track code.

Is Trac sufficient for managing a project? No. It is too much like a big todo list and a bug tracker combined. It is very developer-centric—even when the developer is also the designer and manager, there’s no way to make known the other roles. Trac too is only a piece, because it provides no way to easily communicate ideas. A wiki doesn’t cut it.

Are help desks sufficient? No. They are too customer/support-centric. They have no way to easily communicate the ideas of developers and designers.

What about Lighthouse or Unfuddle? For a hosted solution, Lighthouse and Unfuddle combine Trac and Basecamp. That’s going in a decent direction. Anything which integrates different parts of the development process is addressing the need for management. But it’s not enough, because it has no integration with the customer.

What about [insert megolithic answer to everything]? No. It’s too complicated, has too many options and forms, too much information on each page. Something as complicated as Google Analytics, for example, is pushing the boundary of what is acceptable. Complicated applications get in the way of communication and understanding, even if they integrate everything. Simplicity first.

What all of these solutions lack is a focus on the different kinds of users for a project, ways of easily communicating their needs and ways of addressing those needs. Even when the developer, designer, marketer and manager are rolled into the same person, it is important to separate the roles, make them clear and integrate each one’s concerns at the right points.

It’s all about the development process.

The Development Process

I see that there are 9 stages in the development process:

Management: find out the next need to address
Specification/Testing: specify how the need is addressed
Coding: code until the specification passes
Continuous Integration: combine the efforts of multiple coders
Refactoring: clean up the code
Graphic Design (if needed): make the new feature appealing
Deployment: release the latest revision
Marketing: advertise the latest feature
Customer Feedback: find out what is going well and what isn’t

If you see this process as organic, the importance of management becomes much clearer. In fact, I think management is the most important stage, more important than the code itself, for a number of reasons:

Management is the first step; without management, testing/coding is arbitrary
Management brings everyone into the development process
Management gives everyone an overview so they can see the wood for the trees
Management encourages communication between everyone
Management naturally focuses on the most important aspects
Management reinforces and rewards good development
Management operates organically, reflecting needs and their importance
Management begins the specification/coding cycle
Management draws from and feeds into all other stages of development

Disconnecting from these aspects of development is a serious mistake because it denies the organic element of development. Everything needs to converge at some point and management is the most natural way of acknowledging and converging all of a project’s members, roles, ideas, problems and concerns.

Poor management will try to force the development process into a linear pattern. It will approach things as ‘things to do’, ‘features to have’, ‘milestones to reach’, ‘deadlines to meet’, ‘code to test’. Everything will have its place, need to be addressed by a particular person… in short, it’ll look like a Trac installation. The code will feel strained, regimented and will generally be a rather boring thing to deal with. The developer is being strait-jacketed.

On the other hand, a lack of management will result in the process becoming chaotic. The developer will code whatever takes his fancy. Occasionally e-mails or posts containing feedback will find their way into the code, but mostly the code will diverge from the customer’s interests. The organic element has gone mad in this case because the developer is too isolated.

But good management acknowledges the organic aspect of development and lets the code flow. It translates ideas into specifications just as test-first translates specifications into code. Good project management will create and maintain strong channels of communication between developer, designer, marketer, manager and customer. The real needs for the project will appear of their own accord as different ideas converge in one place.

But that sounds much easier than it actually is. There isn’t yet an application out there which integrates all of those roles together, but some are closer than others. I think Retrospectiva could be the one which gets there first.

What is needed for good management

The three major aspects of project management are development, collaboration and integration. There needs to be a way to develop, track that development, and focus it. There needs to be collaboration and communication surrounding that development. There needs to be a process of integration between management and the other stages of development.

Ideally, a project management system will have the following aspects:

Stories: isolated stories to be resolved (bug, feature, question, idea)
Dynamic properties: status, milestone, persona, feature, assigned user, assigned group
Reinforcement: aspects of the application NOT to change (positive feedback, robust code)
Personae: ability to define personae, like power users, buyers, sellers, novices, etc.
Milestones: rough organisation of stories and deadlines

Messages: site-wide (like Basecamp) and for each milestone
Roles: developer, guest, manager, designer, customer, marketer, administrator
Groups: optional story development by groups
Interfaces: different interfaces for each role and/or group

Cross-referencing: referencing between stories, messages, milestones, source
Testing integration: update stories with progress on tests (e.g. Tesly)
Coverage Integration: stories for area of test coverage, whether 100% covered or not
SCM integration: update stories through commit logs
Continuous Build Integration: create stories for failed builds
Error Notification Integration: create stories for application errors
Customer Integration: create stories for customer feedback (positive & negative)

Most importantly, the interface(s) needs to be extremely clean. Lighthouse goes a long way in making a highly readable, even pleasurable interface. Most of the aspects I list above can exist on their own, meaning the application itself will have many facets, but each very easily understood. Cross-referencing is probably the most important of them all, since it will bring together the various aspects.

The shift in perspective I suggest for project management is to not focus on managing people (like Basecamp) or code (like Trac) or users, but ideas. Those ideas will never disappear from view, unlike tickets on Trac or todo lists on Basecamp. As stories grow and connect with new stories (like associating tickets), everyone will see the evolution and development of features, in which way the project is going, and be able to react better to the movement of the project. In a way, project management also begins to document the project, but more importantly, it shows in black and white how ideas become code and how they evolve. At the centre of the project should be a cloud of ideas which each role can see differently.

Just to give a little hypothetical situation: at the beginning of your project you had a simple user authentication system. Over time, users talked about adding Open ID. Management wanted an authorisation system and an admin interface. Designers wanted a cute widget that pops down with AJAX. Developers wanted to extract it into a plugin. All of these ideas would be associated and appear together in a good management system, showing the time each one was added and completed, the role which initially suggested it, and the group or user responsible for implementing it. All the bugs, notes, support questions, requests and feature stories will clump together and naturally point towards what is needed next, if anything.

This form of project management could very well revolutionise the way development is perceived. Or is it a load of bullshit?

Capistrano 2.0, upgrading & fitting into a size 0 dress

2007-07-27T09:02:09-05:00

The improvements to Capistrano are much welcomed. My deployment recipe is now half the length it used to be and it is much easier to follow what is happening for my many types of deployment. I love the new features added, mostly dealing with manipulating scopes and enhancing the user’s ability to extend the core framework.

Review of new features

namespaces: Like Rake, you can namespace your tasks and group them together more sensibly. This feature alone is worth upgrading for just to make your scripts more sensible and easier to read.

events: Like Rails, you can now perform tasks before or after other ones rather than using the hacky ‘before_something’ and ‘after_something’. Much cleaner and much faster too.

strategies: In addition to checkout, you can now deploy via export and copy and use different strategies for deployment, such as using export for your copy_strategy rather than zips and tarballs.

scoping: All sorts of scoping has been introduced in Capistrano 2.0, from namespacing to single execution of “run” and “sudo”, allowing you to define specific roles or hosts in which your commands run.

help: Capistrano 2.0 now has a more verbose way of explaining tasks with cap -e task_name. You’ll realise how useful this is when you use it for the built-ins as well as your own.

All in all, Capistrano is pretty simple, but it is the way it is written that makes it appear so much simpler than it really is. Capistrano 2.0 takes that to a new level, not groundbreaking perhaps, but definitely a lot cleaner than its previous releases.

Upgrading from 1.4.1

There is no need to change config/deploy.rb out of the box. Capistrano 2.0 is nicely backwards compatible, unlike other things out there, and, at least for me, nothing broke because of the upgrade.

You can look at Capistrano’s instructions for upgrading, if you want to know what is being done, but for the impatient, here are the steps you have to follow before we can start drying up your deploy script.

1. Install the new version of capistrano:

sudo gem install capistrano

2. cd project_root & run capify

~# cd projroot
projroot# capify .

3. Upgrade previous deployments to use the new revision tracking system

projroot# cap -f upgrade -f Capfile upgrade:revisions

4. Rinse and repeat for each of your deployment targets

Getting your deploy.rb into its new size 0 dress

You may now have the very understandable urge to slim down your deployment recipes. With the introduction of Capistrano 2.0, I found my deploy.rb reduced to less than half the size. Below, I cover the areas which you should focus on to get that deploy script into its new size 0 dress.

Anatomy of my deploy.rb

requires: capistrano-ext, mongrel_cluster, etc.
global, stage and custom variables
event chains
rewriting built-ins: web:disable and web:enable
extra tasks: fixing permissions, copying mongrel confs, etc.
custom deploy tasks: long, normal, quick
maintenance tasks: backup, restore

Variables

More than before, variables are the lynch-pin of slimming everything down. The first thing you should do is look over every task rewrite or custom task and see how it can be turned into a simple set :var, true/false/whatever. Capistrano 2.0 will make it very easy to do this.

With Capistrano 2.0, you should use the set command religiously, both for built-in and custom tasks.

I personally set the following at the top of my recipe.

Global variables: stages, deploy_via
Application specific: application, repository, user, scm_username
Deployment specific: deploy_to, rails_env
Custom variables: serving_via, suexec, suexec_user, suexec_group, disable_template

Deployment Strategy

I would personally suggest using xport for your deploy_via strategy unless you have a reason for using heckout or copy.

Using Namespaces

Namespaces make it dead simple to group common tasks, like different restart methodologies. I use a serving_via variable which translates into the reload:whatever task to run for restarting the application. For example:

namespace :reload do
  desc "Default reloading procedure"
  task :default do
    mongrels
  end
  desc "Reload an FCGI application"
  task :fcgi, :roles => :app do
    sudo "#{current_path}/script/process/reaper -a graceful -d #{current_path}/public/dispatch.fcgi"
  end
  desc "Reload an LSAPI application"
  task :lsapi, :roles => :app do
    sudo "/usr/local/litespeed/bin/lswsctrl restart"
  end
  desc "Give the mongrels a bath"
  task :mongrels, :roles => :app do
    restart_mongrel_cluster
  end
end

Note: I warn against using restart as a namespace because it clashes with the built-in task and, in certain instances, results in infinite recursion.

Maintenance Splash

The biggest change in Capistrano you may need to worry about is the removal of delete and render. Don’t despair, though, because creating a maintenance splash is still easy. This is my rewrite:

desc "Generate a maintenance.html to disable requests to the application."
deploy.web.task :disable, :roles => :web do
  remote_path = "#{shared_path}/system/maintenance.html"
  on_rollback { run "rm #{remote_path}" }
  template = File.read(disable_template)
  deadline, reason = ENV["UNTIL"], ENV["REASON"]
  maintenance = ERB.new(template).result(binding)
  put maintenance, "#{remote_path}", :mode => 0644
end

desc "Re-enable the web server by deleting any maintenance file."
deploy.web.task :enable, :roles => :web do
  run "rm #{shared_path}/system/maintenance.html"
end

Using events

Like the before and after filters in Rails, you can now cleanly chain together tasks. I’m a sucker for one-line solutions and these are really so simple that it makes my heart bleed:

before "deploy:restart", "fix:permissions"
before "deploy:migrate", "db:backup"
after "deploy:symlink", "deploy:cleanup"
after "deploy:update_code", "deploy:web:disable"
after "deploy:restart", "deploy:web:enable"

capistrano-ext & multistage

I highly recommend the use of multistage. It comes with the capistrano-ext gem (which has been upgraded to Capistrano 2.0, of course).

Basically, it separates the concerns of different deployments. If, like me, you like having a few other versions of your application out there, like a staging area, a testing area for bleeding edge features, and, of course, the production site, separating these in Capistrano before 2.0 was very irritating. Multistage sorts that out very nicely.

By default, you must specify the stage you wish to deploy. This behaviour can be overridden by setting the default_stage variable, but I like being explicit. This is what using stages looks like:

# cap production deploy

If you don’t provide ‘production’, it’ll complain and abort.

Using multistage is dead easy. Put this at the top of your deploy.rb:

  require 'capistrano/ext/multistage'
  set :stages, %w(staging production testing)

Run the task for generating your stage deploy files:

projroot# cap multistage:prepare

This will create a recipe file for each stage in a new config/deploy directory (exactly like Rails environments). Now, in each stage recipe, add all of your stage-specific tasks and variables. For example:

set :rails_env, "stage"
set :application, "staging.example.com"
set :deploy_to, "/var/www/#{application}"

Now switching between different deployments is a breeze. Just make a new recipe file for it with the necessary variables and you’re set.

Kathy and Chris sitting in a tree...

2007-03-28T22:13:06-05:00

Really hot stuff this Kathy Sierra nonsense. Oh wait, a tasteless pun. Is everyone going to hate me now? Grow up!

I usually don’t wade into political debate because it is full of morons, but it’s the middle of the night and I’m really bored. In fact, annoyed. Like I was with Rails 1.2 (how’s that for continuity!)

Who I Sympathise With and Who I Support

I’ll give, frankly, my sympathy to Kathy (who is manic, slightly weak-willed, has some good ideas, some other ideas a little dodgy, and has all the traits of a modern American woman, for good or ill) but I give my serious support to those calling for uncommon sense. This situation is so cliched and yet has slipped passed a few people I respect that I’m going to kill some time here and point out that Dave Winer, who I never heard of before, is right to be skeptical. He made just one mistake: he wasn’t confident enough to spell out why others should be. Please, please stop fuelling the media. It’s just evil.

So, without further ado, the train of thought:

The gist of my view:

Threats and offensive comments on the internet are newsworthy, therefore not worth my time (unless I’m killing it).
Being a high-profile speaker in the IT world will have its share of ‘acerbic’ commentary
Being a blogger will get you the same
Putting your details online is just dumb
Men abuse each other using sexual language, but we’ve got used to it over the last five millenia and now think it’s kind of amusing

For example, the Bile Blog, which is an institution in slagging shit off. Thankfully it still exists. Poor Hani doesn’t get a BBC article because the Bile stays strictly asexual. But in a post-feminist, emasculated society, you can get really hated for saying anything remotely gender-specific. I’m sorry, but I don’t buy that. It’s proscriptive and ~~- for the same reason I think I should be able to take a pie or bottle of water on a plane even at the risk of being blown up -~~ molly-coddling. Anyone with some self-respect and decency should feel a little spineless and self-indulgent if they coddle Kathy. A mild dose of feminist support is fine, because I like women to be people rather than objects, but indulging in the mass opinion is, well, like getting a labotomy. Some of the points Kathy made were, of course, right: it’s pretty sick some of those things she cited. However, it’s not rocket science to see she is also over-reacting and taking the majestic ‘blogosphere’ with her. Her avid fans who want to hang her are most likely infantile, possibly autistic twits in the middle of nowhere, with the social graces of a louse.

What’s the truth?

The situation is as plain as a Bulgarian pin-up. Kathy takes the net seriously and flips out. Her readers, which are many, believe her. Others who see those believe her think the readers must be right because there are so many of them. A few people are skeptical and they are lampooned, making the case even stronger that Kathy is being persecuted. It goes to the media, which means it must be true!

Qualifications

Please realise that there are minorities of both idiots and intelligent people in the real world just as much as on the internet, with one slight difference: ‘law’ isn’t firmly established online. That’s quite literally fantastical, but true, and we should cherish that infinite freedom. I don’t want a jackboot in my face for telling someone online they can stick their ideas up their cuntflap or hang by their genitals, nor see a person cuffed and imprisoned for behaving as amorally as Sodom. Your morals and theirs may not mix, but that’s neither here nor there. Morality is for religion, not law and the internet is lawless.

All this lovely lawlessness is because of its anonymity. People use that anonymity out of cowardice (yes) but also safety. Uh huh, that’s right. The louse should be protected too, all the way up until he actually clubs Kathy over the head and brings her back to his cave. That’s real free speech. It’s not pleasant sometimes, but it’s very, very important to realise how extreme an idea it is (the original one, not the hashed up piece of shit served by ‘democracies’ around the world).

Summary

Kathy thinks threats are equal to actions, for which she loses my support, though I still hope she pulls out of her mire of self-doubt and back into posting for her readers. Maybe during that period she should read some comforting Spinozan ideas on epistemology and semantics.

Resorting to qualified (that is, reasoned) support and sympathy is quite different from just jumping on the wagon of lynchers, to use Chris Locke’s own words.

Just as a reminder, this is where enforcing morality leads. BBC are really getting lazy; they’ve got two articles on mass hysteria. But they are the news and cater to the lowest common denominator. It’s quite different when respected, intelligent individuals are the cause of the stir. I only hope some of you will give it a little more consideration next time and do the intelligent thing: be as long-winded as I am. Then no journalist could bear reading it.

Initial reactions to Rails 1.2... annoyed

2007-02-18T12:15:27-06:00

I’ve held off my opinion about Rails 1.2 until I had a little more experience with it. First of all, I’m going to say I’m glad Rails has put out this release, because some of the features in Edge I’ve been needing for a while and, in cases where I couldn’t wait, like my simply_restful_backport, I hacked a ported version to 1.1.x. I also had unpublished ports for REST, routing, and a few deprecations to prepare myself. Most of the added features are bang on what is needed when following the Rails methodology of web development: singleton resources, nesting resources, the emphasis on REST, the respond_to(&block), etc.

But my optimism and preparations were in vain. I’m annoyed. My initial reaction is that Rails 1.2 is a fascist. The front-facing libraries, ActiveRecord and ActionPack, are not the cause of my annoyance. It’s the underbelly, ActiveSupport, which has acquired a rank smell.

How Deprecations are handled

I’ll be brief. Backward compatibility is beautiful and when you can’t maintain it, you prepare developers for the transition. Deprecrations are usually the answer: you say it is going to be removed for a long time, encourage people away from it, then finally remove it in either the next major release or the one after that, safe in the knowledge that everyone who will be affected by the change has got the message.

Rails has not held to the time-honoured traditions of deprecation. It does one of three things: fill up the logs with warning messages, raise annoying exceptions that apologise something is gone, or just plain crash because the old API had changed so much. That’s not deprecation. That’s just API upheaval.

And it’s unacceptable to have so many differences in the framework which, first, are undocumented, and second, change development so much. I would expect this behaviour from a 1.3 to 2.0 upgrade, but not between two minor releases. Maybe Rails Core should have re-numbered the release to warn users that Rails has changed quite drastically under the hood. The changes are fine, but it’s the way they came unannounced that is annoying.

Especially after so much time between the two minor versions, there was ample opportunity to tell developers… 1.1.6 was the last opportunity, and a better time was back at 1.1.2, when these additions already existed or were being considered. That is still too quick, but to give no warning, not even a proviso ‘this may change’, until it is too late is just insufferable.

ActiveSupport::Dependencies < ActiveSupport::Devil

For example, I don’t know why an innocuous ObjectSpace block in routes.rb should cause a blocking exception from ActiveSupport::Dependencies. It prevented my application from even loading and took half a day to track down. Or why I needed to alter ExceptionNotifier to require all its libs. Or track down 6 other ‘dependencies’ and make them explicit—I thought Rails was supposed to make this stuff easier, not more of a pain than C’s #include <hardtofind.h>?

It’s ridiculous for an ‘improvement’ release to be so fragile. Rails 1.1.x was pretty flexible on these terms. However, Rails, in its growing focus on encouraging programmers into standardised practices, is looking more like 37signals practices. And to say they are ‘best practices’ or good ‘conventions’ is a bit rich, because Rails is not the cleanest set of libraries. Just run it with $VERBOSE = true and you’ll see what I mean.[1]

In fact, after using ActiveSupport::Dependencies for a while, I’ve come to the conclusion that it is too clever for its own good and I foresee lots of bugs in Rails Core Trac about it. It’s too prying. It’s too observant of everything that is happening. It infects the production environment. It’s a glorified const_set trace_func.

ActiveSupport.include :bloat

What kind of overhead does ActiveSupport::Dependencies have? How many orders of magnitude slower is it than the previous Reloadable module, which just included a method, reloadable?, and was found via ObjectSpace? First of all, it didn’t need a revolution like Dependencies; it was clean, simple, elegant and extendable. It didn’t solve all the problems and had room for improvement, but removing it I think was a mistake. ActiveSupport::Dependencies, unlike it’s predecessor, is a beast. Here’s a quick LOC comparison between activesupport 1.3.1 and 1.4.1 (ignoring hooks in the other gems):

$ grep -v '^\s*#' activesupport-1.3.1/lib/active_support/reloadable.rb | wc -l
28
$ grep -v '^\s*#' activesupport-1.3.1/lib/active_support/dependencies.rb | wc -l
172

$ grep -v '^\s*#' activesupport-1.4.1/lib/active_support/reloadable.rb | wc -l
56
$ grep -v '^\s*#' activesupport-1.4.1/lib/active_support/dependencies.rb | wc -l
538

Not the most reliable comparison, but it gives a good idea of the bloat introduced by, first, dependency tracking and, second, simple deprecating. Most of those lines of code in Reloadable are about how it is not just deprecated, but downright crippled and how evil it was. This is not a good practice for a developing an open source framework which is not easy to upgrade (and we live with it, because Rails is useful). The difficulty of upgrading is precisely because of these badly planned deprecations and backward incompatibilities.

Enough ranting

In this sense, Rails 1.2 is by the far worst gem update I’ve experienced so far. How many blogs have posted about the many little things that needed to be changed to upgrade? How many forward-thinking developers looked in cringing, abject horror when their tests failed left and right after that (apparently) over-optimistic gem update rails? How many people flew to IRC and the mailing lists for help? Let me be so conceited as to introduce a ‘best practice’: discuss major changes with your users.

I realise that Rails is first and foremost a 37signals project, evolved from DHH’s work there and is primarily altered and improved for that company’s purposes, but as an open source project trying to get more users, it’s important to either branch those changes which are special to that company or to be more considerate to the user base. I think Rails development would benefit from:

a separate experimental branch (different from Edge trunk) for API changes—Radiant does this quite nicely
a highly visible discussion board on rubyonrails.org to discuss major changes with a link to it in IRC and regular updates to the Rails mailing lists

Granted, much of my qualms with Rails 1.2 is with Dependencies and with some good old fashioned hacking, they can fix it. However, I think it and the library from which it has spawned are worth my comment. Dependencies is the most invasive addition to the framework I’ve seen to-date. I appreciate the attempt at innovation, but Rails needs to stop doing this kind of widespread deprecation and fundamentalist intolerance to the unconventional.

I wish the development team would take more pleasure in the diversity of uses of Rails and less perverse pleasure in the militant drilling of young, transitional and experienced programmers. I like the framework, but I don’t work (or want to work) at 37signals.

¹ I’ve been aware of my own transgressions for a while. I’m in complete agreement with Mauricio on the topic of warnings. They are there for a reason: to create semantically correct Ruby. We should all be able to run our fantastically short, readable, meta-programmed, overloaded and mixin-loving code with $VERBOSE = true. It isn’t difficult to achieve and it helps in the long-run to make that lovely little code more readable, usable and quiet.

Syspec: System Specifications & Tests using a Rake-like batch file

2007-02-10T15:45:43-06:00

Syspec lets you write a batch file a lot like a Rake file, but in common HTTP-style language, to test the existence and proper functioning of all the domains, subdomains, their pages and responses. This project is still very much in its infancy, but I decided to get it out there to get some feedback while I use it to track production servers. This is a sample Syspecfile:

check :"www.rubyonrails.org" do
  accept 200
  get "/"
  accept 404
  get "/syspec"
  get "/lacrima"
end

The above will check that http://www.rubyonrails.org/ responds with 200 and http://www.rubyonrails.org/syspec and http://www.rubyonrails.org/lacrima respond with 404. You can have multiple checks with the same domain and it does not have to be a symbol, though I prefer it that way.

Currently, Syspec features the following:

accept(*codes) – response code should be among these codes
get, post, delete, put (uses fetch(:request_method, url))

Upcoming features:

login “username”, “password”
body matching
forms

You can grab a copy of it directly from my SVN repository here:

  svn co http://mabs29.googlecode.com/svn/trunk/gems/syspec .

Or download the gem from within the SVN repository (later will be moved to rubyforge):

  svn export http://mabs29.googlecode.com/svn/trunk/gems/syspec/versions/syspec-0.2.1.gem .

Enjoy!

ANSI Strings (2nd Edition)

2007-02-07T01:32:22-06:00

I had a couple of instances where I was adding to the ANSI_CODES constant and creating all sorts of ugly errors. Then a script of mine whispered in my ear its desire for meta-aliases. That prompted a small rewrite of the library.

Ansi codes are now dynamically defined and more easily manipulated without overcomplicating things. Also, I added an RSpec test to the repository for my own piece of mind and to help with refactoring a few things. I don’t know, should I give it a version number? Nah. Just ‘Second Edition’. I don’t envisage any other serious revisions to it any more. What I changed:

dumped the array structure for ansi codes
added a few non-colour ANSI codes
dynamic ansi code definitions: String.define_ansi :alias => “123m”
meta-aliases: String.define_ansi :clear_both => [:clear_line, :clear_screen]
String#method definitions only if it doesn’t exist already
String#method definitions on the fly
spec test

And here’s the code:

$ANSI ||= false

class String

  class << self

    attr_accessor :ansi_codes

    def ansi_codes
      @ansi_codes ||= {}
    end

    def define_ansi(definitions = {})
      self.ansi_codes = ansi_codes.merge(definitions)
      define_shortcuts(definitions)
    end

    def define_shortcuts(definitions = {})
      definitions.keys.each do |meth|
        unless instance_methods.include?(meth.to_s)
          define_method(meth) { with_ansi(meth) }
        end
      end
    end

  end

  define_ansi :normal         => "0m",      :bold           => "1m"
  define_ansi :underline      => "4m",      :blink          => "5m"
  define_ansi :reverse_video  => "7m",      :invisible      => "8m"
  define_ansi :black          => "30m",     :red            => "31m"
  define_ansi :green          => "32m",     :yellow         => "33m"
  define_ansi :blue           => "34m",     :magenta        => "35m"
  define_ansi :cyan           => "36m",     :white          => "37m"
  define_ansi :black_bg       => "40m",     :red_bg         => "41m"
  define_ansi :green_bg       => "42m",     :yellow_bg      => "43m"
  define_ansi :blue_bg        => "44m",     :magenta_bg     => "45m"
  define_ansi :cyan_bg        => "46m",     :white_bg       => "47m"
  define_ansi :clear_line     => "K",       :clear_screen   => "2J"
  define_ansi :go_home        => "0H",      :go_to_end      => "80L"

  # Wrap a string with an arbitrary ansi code and the ansi normal code
  def with_ansi(*codes)
    use_ansi? ? "#{sym_to_ansi(*codes)}#{self}#{sym_to_ansi(:normal)}" : self
  end

  # Just a little metaprogramming shortcut
  def ansi_codes; self.class.ansi_codes; end

  private
  def sym_to_ansi(*symbols)
    symbols.inject("") do |string, symbol|
      if code = ansi_codes[symbol]
        string << (code.is_a?(Array) ? sym_to_ansi(*code) : "\e[#{code}")
      end
      string
    end
  end

  # determine whether we have ansi support or ANSI enabled
  def use_ansi?
    $ANSI && RUBY_PLATFORM !~ /win32/i
  end

end

You can grab the new version from my ‘other’ directory at Google Code:

  svn co http://mabs29.googlecode.com/svn/trunk/other/ansi_strings .

ANSI Strings: your fun-loving, colourful pal (quick-fix colour library)

2007-02-01T02:54:08-06:00

You can make your command line interface look a little better than just a bungle of white on black (or however you or your users set your terminal). In my on-going development of a few little open source projects, all of them with CLIs, I abstracted this cute addition to the Ruby snippets junkyard and decided to air it.

With it, you can wrap any string with ANSI codes like this:

  "I am going to be bold and cyan!".bold.cyan
  "I am going to be underlined. YAY!".underline
  "Oh, for the love of God... please STOP BLINKING!".blink

Make your command line interface glisten with choice and sparing use of ANSI codes not just with the above methods dynamically added to the String class, but also by using with_ansi(*ansi_aliases). It’s really very simple. Also, all application of ANSI is dependent on $ANSI being true, which, by default, is false (as you can at the bottom of this article, in the first line of code). Lastly, ANSI Strings checks the current platform and won’t be applied to strings on a win32 platform—I have no way of testing whether win32 can in certain instances show colour, so I just disable it.

You can download ANSI Strings from my googlecode repository like so:

 svn co http://mabs29.googlecode.com/svn/trunk/other/ansi_strings

Or you can just look at it here, since it is so simple and small:

$ANSI ||= true

class String

  ANSI_CODES = {
    :normal         => [0, "m"],
    :bold           => [1, "m"],
    :underline      => [4, "m"],
    :blink          => [5, "m"],
    :reverse_video  => [7, "m"],
    :invisible      => [8, "m"],

    :black          => [30, "m"],
    :red            => [31, "m"],
    :green          => [32, "m"],
    :yellow         => [33, "m"],
    :blue           => [34, "m"],
    :magenta        => [35, "m"],
    :cyan           => [36, "m"],
    :white          => [37, "m"],

    :black_bg       => [40, "m"],
    :red_bg         => [41, "m"],
    :green_bg       => [42, "m"],
    :yellow_bg      => [43, "m"],
    :blue_bg        => [44, "m"],
    :magenta_bg     => [45, "m"],
    :cyan_bg        => [46, "m"],
    :white_bg       => [47, "m"]
  }

  # Create shortcuts to wrap any string with ansi codes
  ANSI_CODES.keys.each { |meth| define_method(meth) { with_ansi(meth) } }

  # Wrap a string with an arbitrary ansi code and the ansi normal code
  def with_ansi(*codes)
    use_ansi? ? "#{sym_to_ansi(*codes)}#{self}#{sym_to_ansi(:normal)}" : self
  end

  private
  def sym_to_ansi(*symbols)
    symbols.inject("") do |string, symbol|
      string << "\033[%s%s" % ANSI_CODES[symbol] if ANSI_CODES[symbol]; string
    end
  end

  # determine whether we have ansi support or ANSI enabled
  def use_ansi?
    $ANSI && RUBY_PLATFORM !~ /win32/i
  end

end

MicroTest 1.3 / MicroStub 1.1: still under 4KB

2007-01-24T01:33:07-06:00

MicroTest has had a few tweaks, making it more brief and efficient at what it does. Also, I added a setup and teardown interface which can be used. For example, I removed the use of ObjectSpace and squeezed a few percent of performance just from that. On my 1.83GHz, the 6 tests of the self test now consistently take around 500 microseconds (or 0.0005 seconds). Pretty nippy.

MicroStub had a little problem with it—it stubbed instance methods and didn’t return the class back to normal. So, it violated one of the most basic principles of testing: isolated tests. That’s all fixed now, using some Spartan back-and-forth aliasing. Obviously a double stub will fail. Maybe I’ll fix that later.

Lastly, I added a benchmarking test case, really just useful for profiling.

Of course, the library is still under 4KB—in fact, it’s 3940 bytes.

Have a look at the repository (revision 25) for the latest version.

MicroTest 1.2 & MicroStub 1.0: Your microscopic testing pals

2007-01-21T12:49:43-06:00

Just after writing the previous post, I discovered the need for a little bit of method stubbing and promptly fell upon the task with MicroStyle¹. Also, I will officially keep MicroTest under 4 kilobytes².

So, without further ado, here are the latest new features of MicroTest:

colour output
execution time in microseconds
MicroStub: overwrite a method with stub()
twice as fast (no sweat)
a self-test! (SELFTEST=true ruby microtest.rb)
and its own home. Awwww.

MicroTest now lives in my Google Code SVN repository.

  svn co http://mabs29.googlecode.com/svn/trunk/other/microtest/

¹ By the way, if you haven’t noticed yet, I’ve been a little inspired by Camping.

² Unlike Camping (because MicroTest is less complicated), I think it only fair that the 4 kilobytes include comments and the self-test.

MicroTest just got smaller

2007-01-20T07:16:02-06:00

My little library (the smallest ever?) for bootstrapping and lightning-fast testing has just become smaller. I changed the interface a little: now everything is dependent on method name! So you still have two tests, one which expects true and the other false, and which one is determined by using ‘def should_’ or ‘def should_not_’. Oh happy days!


# A super-small testing suite based on method names.
# It has only two types of expectations: should and should_not.
# These two expectations begin the method name, after which you can
# place anything you like. A 'should_...' method will expect true as
# the result of calling the method. A 'should_not_...' method will,
# conversely, expect false.
#
# This is MicroTest's self-test:
#
#   TEST_ROOT = File.dirname(__FILE__) + '/..'
#   require TEST_ROOT + '/lib/microtest.rb'
#
#   class TestMicroTest < MicroTest
#     def should_expect_true
#       true
#     end
#
#     def should_not_expect_false
#       false
#     end
#   end
#
# An example:
#
#   class BinaryMicroTest < MicroTest
#     def should_exit_cleanly
#       `ruby #{MYEXEC}`; $? == 0
#     end
#     def should_not_exit_cleanly
#       `ruby #{MYEXEC} -garbage`; $? == 0
#     end
#   end
#
class MicroTest

  class MicroFailure < StandardError; end

  class << self

    attr_accessor :passed, :failed, :executed, :planned, :plan, :failures

    def run
      init
      execute
      report
    end

    def init
      self.passed, self.failed, self.executed = 0, 0, 0
      self.planned, self.plan = harvest
      self.failures = []
    end

    def harvest
      plan = []
      ObjectSpace.each_object(Class) do |k|
        plan << [k, k.instance_methods(false).grep(/^should_/)] if k < MicroTest
      end
      planned = plan.inject(0) { |c, ary| c += ary[1].size; c }
      [planned, plan]
    end

    def execute
      puts "Running #{self.planned} test(s)..."
      self.plan.each do |klass, tests|
        tests.each do |test|
          expectation = (test =~ /^should_not/ ? false : true)
          instance = klass.new
          r = catch_failures(klass, test) { expect(expectation) { instance.send(test) } }
          if r
            self.passed += 1
            print "."
            STDOUT.flush
          else
            self.failed += 1
            print "F"
            STDOUT.flush
          end
          self.executed += 1
        end
      end
    end

    def expect(expected, &block)
      raise MicroFailure, "should be #{expected}" unless block.call == expected
    end

    def catch_failures(klass, test)
      begin
        yield
        true
      rescue Exception => e
        self.failures << [klass, test, e]
        false
      end
    end

    def report
      puts "\n\n"
      puts "%d planned, %d executed, %d passed, %d failed." %
        [self.planned, self.executed, self.passed, self.failed]
      unless self.failures.empty?
        puts "\nFAILURE REPORT\n"
        self.failures.each { |f| report_failure(f) }
      end
    end

    def report_failure(f)
      puts "Suite: #{f[0]}\nTest: #{f[1]}\nException: #{f[2].class}\nMessage: #{f[2].message}"
      puts ("\t" << f[2].backtrace.join("\n\t"))
    end

  end

end

at_exit do
  MicroTest.run
end

Surprise 10min Benchmark: eval, class_eval, instance_eval, define_method, bind

2007-01-11T03:42:37-06:00

What is the fastest way to dynamically define a method? define_method? class_eval? instance_eval? With strings or procs? Maybe we can pull out some arcane magic and use instance_method and bind (depending on what we want to do).

We all need to use some kind of eval at some point and, while it will not usually effect performance when used sparingly, sometimes it just has to make a serious appearance. When I saw how slow RSpec is in comparison to the venerable Test::Unit, I discovered the way it handles its contexts: each one is individually instance_eval’d into an empty ContextEvalModule. It isn’t the only reason for being slower, but it is one and one which can really mount up on big projects.

So, moving on from RSpec, Test::Unit and my baseline, MicroTest (see previous post), I became perversely interested in benchmarking all the ways of doing it and found some surprises.

Results (Ruby 1.8.4)

                            user     system      total        real
eval/str              112.540000   0.060000 112.600000 (112.615607)
instance_eval/str      99.510000   0.040000  99.550000 ( 99.564661)
class_eval/str        112.750000   0.060000 112.810000 (112.824374)
instance_eval/proc     56.750000   0.040000  56.790000 ( 56.802483)
class_eval/proc        56.440000   0.020000  56.460000 ( 56.464818)
define_method/class   112.420000   0.060000 112.480000 (112.489430)
define_method/send    111.540000   0.050000 111.590000 (111.610502)
def/unbind/bind        73.750000   0.060000  73.810000 ( 73.875800)
def                    52.670000   0.030000  52.700000 ( 52.705886)
method/bind            13.520000   0.010000  13.530000 ( 13.523194)

Analysis

I put it roughly in the order which I originally expected, with the slowest at the top.

The first thing that stunned me was the speed of define_method. It’s as slow as a snail. It’s practically eval, despite using a proc. I’ve been using it in preference before, thinking it was faster than any of the evals (proc or string). Apparently not.

The second revelation was how proc-based evals are about 50% faster than a def/unbind/bind cycle, which only has def as the evaluating part. That any eval is close to def is simply surprising. The instance_eval variety is the one used by RSpec, so I guess they picked the best one already.

The third shock was how slow def is! Granted, this is a lot of iterations, but it should be quicker than that, surely?

So it seems pretty plain that method unbinding as a tuning technique is rubbish, despite not evaluating anything, whereas a proc-based instance_eval or simple def are the clear winners.

But wait… what about Ruby 1.8.5?

Seeing the decrepitude of define_method made me wonder about the latest release of Ruby. I downloaded it, compiled it, got it working and ran the same bench:

Results (Ruby 1.8.5)

                            user     system      total        real
instance_eval/str     109.620000   0.050000 109.670000 (109.685393)
class_eval/str        113.840000   0.060000 113.900000 (113.910749)
eval/str              114.520000   0.060000 114.580000 (114.596238)
define_method/class    70.220000   0.040000  70.260000 ( 70.264506)
define_method/send     68.030000   0.040000  68.070000 ( 68.066688)
def/unbind/bind        73.690000   0.030000  73.720000 ( 73.741549)
instance_eval/proc     59.050000   0.030000  59.080000 ( 59.084445)
class_eval/proc        59.530000   0.030000  59.560000 ( 59.566885)
def                    52.870000   0.030000  52.900000 ( 52.895184)
method/bind            12.470000   0.010000  12.480000 ( 12.484069)

Anaylsis

I guess they fixed the problem and more besides. define_method is now marginally faster than def/unbind/bind (which has not budged an inch). I think I’ll continue to take the performance hit on define_method with 1.8.4, knowing I’ll want to upgrade later in the future and not change techniques.

The old recommendations hold true

instance_eval with a proc for class, instance or singleton evaluation
instance_eval for changing proc bindings
define_method for converting procs into methods
def for everything else
eval when you really know you need it

Note: These findings are only for def meth; true; end and may vary in complex cases.

Benchmark Code

require 'benchmark'

class A; def meth1; true; end; end

# defining using eval/class/instance
str = "def meth2; true; end"
a_binding = A.send(:binding)

# proc for class/instance
proc1 = Proc.new { def meth2; true; end }

# proc for define_method
$proc2 = Proc.new { true }

# unbound method for bind
um = A.instance_method(:meth1)

# number of iterations
n = 12000 * 600

Benchmark.bm(22) do |x|

  x.report('instance_eval/str') do
    for i in 1..n; A.instance_eval(str); end
  end

  x.report('class_eval/str') do
    for i in 1..n; A.class_eval(str); end
  end

  x.report('eval/str') do
    for i in 1..n; eval(str, a_binding); end
  end

  x.report('define_method/class') do
    for i in 1..n; class A; define_method(:meth2, &$proc2); end; end
  end

  x.report('define_method/send') do
    for i in 1..n; A.send(:define_method, :meth2, &$proc2); end
  end

  x.report('def/unbind/bind') do
    for i in 1..n
      class A; def meth2; true; end; end
      A.instance_method(:meth2).bind(A.new)
    end
  end

  x.report('instance_eval/proc') do
    for i in 1..n; A.instance_eval(&proc1); end
  end

  x.report('class_eval/proc') do
    for i in 1..n; A.class_eval(&proc1); end
  end

  x.report('def') do
    for i in 1..n; class A; def meth2; true; end; end; end
  end

  x.report('method/bind') do
    for i in 1..n; um.bind(A.new); end
  end

end

Introducing: MicroTest

2007-01-10T16:04:32-06:00

Something I cooked up in an hour for bootstrap testing a testing framework.

If anyone actually wants me to maintain it, just give a shout and I’ll gemify it.



# A super-small testing suite.
# Just two ways to assert:
#   expect { (whatever must return true) }
#   do_not_expect { (whatever must return false) }
#
# A microtest looks like this:
#
#   class BinaryMicroTest < MicroTest
#     def should_exit_cleanly
#       expect do
#         `ruby #{myexec}`
#         $? == 0
#       end
#     end
#   end
#
class MicroTest

  class MicroFailure < StandardError; end

  class << self

    attr_accessor :passed, :failed, :executed, :expectations, :planned, :plan, :failures

    def run
      init
      execute
      report
    end

    def init
      self.passed, self.failed, self.executed, self.expectations = 0, 0, 0, 0
      self.planned, self.plan = harvest
      self.failures = []
    end

    def harvest
      plan = []
      ObjectSpace.each_object(Class) do |k|
        if k.respond_to?(:superclass) && k.superclass == MicroTest
          plan << [k, k.instance_methods.grep(/^should_/)]
        end
      end
      planned = plan.inject(0) { |c, ary| c += ary[1].size; c }
      [planned, plan]
    end

    def execute
      puts "Running #{self.planned} test(s)..."
      self.plan.each do |klass, tests|
        tests.each do |test|
          instance = klass.new
          r = catch_failures(klass, test) { instance.send(test) }
          if r
            self.passed += 1
            print "."
            STDOUT.flush
          else
            self.failed += 1
            print "F"
            STDOUT.flush
          end
          self.expectations += instance.expectation_count
          self.executed += 1
        end
      end
    end

    def catch_failures(klass, test)
      begin
        yield
        true
      rescue Exception => e
        self.failures << [klass, test, e]
        false
      end
    end

    def report
      puts "\n\n"
      puts "%d planned, %d executed, %d passed, %d failed.\n%d expectations found." %
        [self.planned, self.executed, self.passed, self.failed, self.expectations]
      unless self.failures.empty?
        puts "\nFAILURE REPORT\n"
        self.failures.each { |f| report_failure(f) }
      end
    end

    def report_failure(f)
      puts "Suite: #{f[0]}\nTest: #{f[1]}\nException: #{f[2].class}\nMessage: #{f[2].message}"
      puts ("\t" << f[2].backtrace.join("\n\t"))
    end

  end

  attr_accessor :expectation_count

  def initialize
    self.expectation_count = 0
  end

  def expect(&block)
    self.expectation_count += 1
    raise MicroFailure, "should be true" unless block.call
  end

  def do_not_expect(&block)
    self.expectation_count += 1
    raise MicroFailure, "should be false" if block.call
  end

end

at_exit do
  MicroTest.run
end

Just dump that in a helper file and require it, as per normal Test::Unit practices.

ActiveTest::Redesign < ActiveTest::Examination

2007-01-02T09:55:27-06:00

The following is taken straight from the first two sections of the ActiveTest Redesign Draft. Feel free to comment, ask for features, steer me away from the unnecessary or just say you’re interested.

1.0: ADDRESSING A NEED

This section takes a cursory look at the evolution of ActiveTest.

1.1: The Original Premise

I originally started writing ActiveTest to address problems I had with Test::Unit. It lacked the following for my projects:

inheritance
modularity (adding)
flexible (redefining)
granular verbosity
observing
BDD interface

The most annoying omissions were the first two. I felt that no matter how much documenting my tests were doing, I hated breaking with the habit of not repeating myself. There are simply times when different parts of an application share a common interface or set of functions. That’s all there needs to be said in the tests.

1.2: Problems with ActiveTest 0.1.0

The first beta of ActiveTest was a flop; it was too specific to Rails, especially controllers, and used too many metaprogramming techniques. Consequently, it created doubt about the failures in test cases—were they ActiveTest or real? It mildly discouraged TDD and, as seen in my small series of articles on the mistakes I’ve made, there was too much focus on the One-Liner™.

1.3: Current State of Need

From what I can tell, there are three related schools of thought about testing which have members in the Ruby community: Test Driven Development, Behaviour Driven Development and Story Driven Development. The Agile and XP groups of developers select whichever according to taste. ActiveTest can easily cater to all schools and groups.

Looking at Test::Unit and comparing it to other projects, it is apparent that Rubyists are being left high and dry. Concessions have come in the form of ZenTest for automation, RSpec for BDD, RFuzz for fuzzing, Mocha for mocking, and rcov for coverage. Each of them (except RSpec) are forced to extend an inherently closed model, Test::Unit::TestCase, and the arcane classes which drive it.

2.0: APPROACHING THE REDESIGN

This section explains the purpose and approach for the new ActiveTest framework.

2.1: Purpose

The new ActiveTest is to provide an advanced, backward-compatible replacement for Test::Unit. By this design, it is compatible with many projects already extending Test::Unit, allowing for quick uptake and usage. It is also to be at the forefront of testing theories and advanced testing techniques both within and outside the Ruby community. Ultimately, it’s purpose is to make developers in all other languages very, very jealous.

2.2: Summary

ActiveTest must change from being a Rails plugin to a gem. It must be compatible with other gems and plugins within the Ruby community or, if compatibility is too complicated, incorporate their ideas with due credit. It must take head-on the latest in BDD and SDD styles. It must reuse as much of its old code to this purpose as possible. Finally, it must draw on projects in other languages, most notably Java and Python, to make use of the latest developments.

2.3: Influenctial Ruby projects

Test::Unit
ZenTest (especially autotest and Rails View testing)
Mocha
RSpec
RFuzz
rcov
Systir
Rails Fixtures

2.4: Influential projects in other languages

Testoob (Python)
JUnit (Java)
JUnitPerf (Java)
GroboUtils (Java)
SimpleTest (PHP)
TAP (Perl)
Test::Harness (Perl)
Test::More (Perl)

2.5: Objectives

The following features from Test::Unit:

assertions
setup & teardown
interfaces: Console, Fox, GTK, GTK2, TK
auto-running
filtering
backtrace
text report
unit tests

The following features from Ruby projects are compatible:

mocking and stubbing (through Mocha)
fuzzing (through RFuzz)
coverage reports (through rcov)

The following features incorporated from other projects:

data fixtures (from ActiveRecord)
automation (from ZenTest)
colorized output (from ZenTest & Testoob)
granular test selection (from Testoob)
different report formats (from Testoob)
test threading (from Testoob & GroboUtils)
Test Anything Protocol (from TAP, Test::Harness & Test::More)

The following features are specific to ActiveTest:

co-existence of TDD, BDD & SDD testing styles
inheritance
test type variety: unit, integration, system, performance
granular verbosity (backtrace, debug flags)
interface: Web/AJAX

ActiveTest: Examination, Part V: Redesigning Weak Areas

2007-01-02T09:49:32-06:00

There are three aspects of ActiveTest which really need to be cleaned up: the Subject abstract class, metaprogramming subjects, and the extension of Test::Unit.

The Subject Abstraction

It is pretty obvious now, without a real need for define_behavioral, that ActiveTest::Subject as an interface class is an unnecessary intermediary between Base and the subjects which inherit from it. So trim the fat and move all of Subject’s functionality to Base. The result is a cleaner hierarchy and less temptation to put something in that middle-man model.

class ActiveTest::Controller < ActiveTest::Base
end

This change in design clears up the association between subjects and the base. Base, which was originally a general abstract for any kind of test case, becomes a general abstract for subjects. The change is mostly conceptual. Functionally, it still provides the rudiments of an ActiveTest suite: inheritance and nested setup/teardown. The next logical step, then, is to start using those advantages with a concrete class: Controller, for example.

Subject Metaprogramming

The only concrete subject written so far is Controller and it is a harbinger of doom for the others. It is bloated with metaprogramming, namely succeeds_on and its peers, because of a misunderstanding of where to use macros. Do we need macros in tests? Not to define them—Ruby is clean enough to do it already. If macros clean up readability, then when does something become muddied in a test? Consider the following real situation:

  def test_should_add_quantity_for_product
    @cart = carts(:first)
    @item = @cart.line_items[0]
    assert_no_difference @cart.line_items, :count do
      assert_difference @item, :quantity, 1, :reload do
        @cart.add_product(@item.product.id)
      end
    end
  end

What’s going on? In this example from a simple test on shopping carts, you want to ensure that the quantity of an item in a cart is incremented when a duplicate item is added to the cart. The item and cart objects are associated through the line_items join model. Because the Cart model updates Item, the item instance needs to be reloaded to reflect changes in the database.

Essentially, we want to test changes being propagated down a hierarchy of Record associations. This situation is not so uncommon and we can easily think of more instances. Once we have a repetition of this pattern, we can write a macro to open up the hierarchy:

  expand_hierarchy :cart_to_items, :reload => true do |a|
    a.cart = carts(:first)
    a.line_items = a.cart.line_items
    a.item = a.line_items[0]
  end

  def test_should_add_item_quantity_for_product
    cart_to_items do |a|
      assert_difference a.item, :quantity do
        a.cart.add_product(a.item.product.id)
      end
    end
  end

  def test_should_increment_line_items_for_product
    cart_to_items do |a|
      assert_difference a.line_items, :count do
        a.cart.add_product(a.item.product.id)
      end
    end
  end

The expand_hierarchy macro here helps us remove any extra tests or setup from one test case as well as make it clearer what we are testing for: we want to find, within a hierarchy, the correct change. The easiest way to do that is to expand out the parameters. You can think of it as a ‘setup-on-demand’. Also, when any one instance is updated, the others are out of sync. The :reload flag tells the macro to offer up proxies of each object; when one changes, all of the other objects are refreshed automatically.

expand_hierarchy, then, does two things: it offers a standard interface for accessing hierarchical associations and it removes extra setup within a test case. Both make it easier to read what is happening, I’d say, but don’t effect the documentation quality. In fact, it seems to improve it. So rather than using macros to define the test cases themselves, a better use would be to package non-assertive elements of testing. These programmatic elements do not need to appear in each test case—they belong in a library, which is what ActiveTest can standardise.

Extending Test::Unit

The premise of ActiveTest began to sour when it became a tool to move test cases into stock meta-definitions. The idea of templating is fine, but the meta-programming aspect was a serious flaw. It hid all the guts in a DSL rather than gave a transparent way of writing tests faster and more easily. The reason it went in this direction is that it is the same direction taken by Test::Unit: linear testing.

So the question is: should ActiveTest extend or evolve? When looking at the problems with Test::Unit and the original design of ActiveTest, it may not be so bad of an idea to address the real issue which has been skirted around by many Ruby programmers for a long time now: Test::Unit isn’t flexible enough. I do not wish to trod over Nathaniel Talbott’s work, but rather to address problems that arise in more complicated environments, like Rails, and are not suited to Test::Unit.

The new version of ActiveTest, I think, should live up to its namesake and really be its own testing framework, written to be compatible with Test::Unit, but with a design that can be improved by others without lengthy review of its code. If the only change, in the worst case scenario, is to change Test::Unit::TestCase to ActiveTest::Base, then there is no issue of learning a new DSL. Instead, ActiveTest would use the same language as Test::Unit, but be completely different under the hood and consequently include much more.

Coming Up Next: ActiveTest::Redesign…

ActiveTest: Examination, Part IV: Salvaging Useful Ideas

2007-01-01T23:13:42-06:00

DRY and Abstraction: There is a difference between practices of DRY and Abstraction. When you are being DRY, you are either addressing a new repetition or continuing an earlier practice. The primary focus of being DRY is not to create higher level tools for future use. It is a practice of improving past and present code. It replaces, deletes, compresses, encapsulates—it is passive redesign. Abstraction, on the other hand, is active design. When you try to be DRY without a real model, you are actually abstracting. The process of abstraction is a wider programming activity which can lead quite naturally into over-design, because without real models and implementation, it often fails to address real issues.

With that short sermon, let’s make ActiveTest useful again. As I mentioned in the introduction to this mini-series, there are things to take away from the old version of ActiveTest. We’ll now look at them in full and show why they are still useful.

1. Test::Unit inheritance: filtering classes

One of the few patches in the old ActiveTest is to modify Test::Unit’s very well hidden collector filter. The idea of filtration is critical to making inheritance possible. If you have a situation where many ActiveRecord models share many attributes, for example in a Single Table Inheritance for a ‘Content’ model, and they all have a body, title, excerpt, owner, created_at, created_by, updated_at and updated_by, the tests for these attributes will be identical across all model tests. So why not create a ‘ContentTestCase’ suite which each model test inherits? With Test::Unit you can make a ‘StandardContentTests’ module and mix it in, but then you are looking for a place to put them and later looking around for those abstracted tests. Alternatively, if you want to use inheritance, those tests will be run immediately. You want to filter out the abstract class, but still be able to inherit from it. By modifying Test::Unit’s collector filter, it is possible to put anything in the ActiveTest namespace and it will not be run:

require 'test/unit/autorunner'
class Test::Unit::AutoRunner

  def initialize_with_scrub(standalone)
    initialize_without_scrub(standalone)
    @filters << proc { |t| /^ActiveTest::/ =~ t.class.name ? false : true }
  end

  alias_method :initialize_without_scrub, :initialize
  alias_method :initialize, :initialize_with_scrub
end

I won’t go on about the peculiarity of having the filter in AutoRunner, but the inaccessibility of the filter requires a monkey patch like the one above. Just poking into initialize lets us add to the @filters instance variance on AutoRunner and tell it to ignore the ActiveTest namespace. This technique is more about just reading through the code and finding the right place to patch.

2. Metaprogramming techniques (not all of them)

Wrapping define_method has proven to be pretty pointless other than ensuring an unique method name, but developing a lower-level language to standardise class-level macros is a general idea which ActiveTest could keep with some minor adjustments. Ignoring what define_behavioral does, this is a very short macro definition (from ActiveTest::Controller):

  def assigns_records_on(action, options = {})
    define_behavioral(:assign_records_on, action, options) do
      send("call_#{action}_action", options)
      assert_assigned((options[:with] || plural_or_singular(action)).to_sym)
    end
  end

If define_behavioral is made to register behaviors for a better parsing method than using method_missing, then we give it some meaning, but not really enough to keep it. The rest of the method definition, however, is quite clean for creating macros for Subjects, especially if it is compressed in the way mentioned in Part III. Making test macros, however, needs to be dropped in the way it is being used here, or at least used more sparingly. We could keep the setup macro because it sets up useful instance variables and makes reasonable guesses about the test suite’s environment. So, the macro idea should be kept back for special cases, the current implementation entirely dropped.

3. Nested, self-contained setup methods through sorcery method unbinding & stacking

I can’t help but be partial to the way I nested setup and teardown methods. It is my first actually useful innovation in Ruby—a trickery of method unbinding and stacking. Perhaps this bias makes me think it is still useful, but I honestly think it is a useful way to wrap Test::Unit. Let’s have a look at the way it is done in ActiveTest::Base:

  class_inheritable_array :setup_methods
  class_inheritable_array :teardown_methods

  self.setup_methods = []
  self.teardown_methods = []

  # Execute all defined setup methods beyond Test::Unit::TestCase.
  def setup_with_nesting
    self.setup_methods.each { |method| method.bind(self).call }
  end
  alias_method :setup, :setup_with_nesting

  # Execute all defined teardown methods beyond Test::Unit::TestCase.
  def teardown_with_nesting
    self.teardown_methods.each { |method| method.bind(self).call }
  end
  alias_method :teardown, :teardown_with_nesting

  # Suck in every setup and teardown defined, unbind it, remove it and
  # execute it on the child. From here on out, we nest setup/teardown.
  def self.method_added(symbol)
    unless self == ActiveTest::Base
      case symbol.to_s
      when 'setup'
        self.setup_methods = [instance_method(:setup)]
        remove_method(:setup)
      when 'teardown'
        self.teardown_methods = [instance_method(:teardown)]
        remove_method(:teardown)
      end
    end
  end

The first aspect which is useful is allowing setup and teardown nesting without affecting additions to Test::Unit made before or after loading ActiveTest. This is allowed by the code in method_added. Before ActiveTest is loaded, the methods and aliases already exist on Test::Unit; after ActiveTest is loaded, all setup or teardown methods are stacked in a class inherited array and subsequently undefined from the original class. When setup is finally called on a subclass of ActiveTest::Base, it is the ActiveTest::Base method called first. Here is an example stack:

  class Test::Unit::TestCase
    def setup; puts "a"; end;
  end
  class Test::Unit::TestCase
    def setup_with_fixtures; puts "b"; end
    alias_method_chain :setup, :fixtures
  end
  class ActiveTest::Sample < ActiveTest::Base
    def setup; puts "c"; end
  end
  class SampleTest < ActiveTest::Sample
    def setup; puts "d"; end
  end

Upon execution, it will output: d, c, b, then a. A caveat of this technique is that you must be absolutely certain that each unbound method setup_with_nesting executes must be rebound to the original class or one of its descendants. Because of the way class inherited attributes work, this rule is not violated: SampleTest can run the setup method from ActiveTest::Sample because it is a descendant, but other classes inheriting from ActiveTest::Sample will not have methods from SampleTest.

4. Specificity of breaking down things into Subjects

This is more of a design note than actual programming. Presently in Test::Unit, it is only possible to add to Test::Unit::TestCase, from which every test suite inherits. There is no way to say that, for example, assert_difference for an ActiveRecord test is slightly different to 'assert_difference for an ActionController test. The functionality may be the same (for example, refreshing the Record or refreshing the Controller request), but implemented differently for each. By creating subclasses of a test suite that provide only the methods needed by that kind of test, the developer can easily encapsulate test methods and not leak public methods across different kinds of tests. It’s just cleaner and will definitely form the backbone of test suites in the new ActiveTest framework.

Coming Up Next: Redesigning Weak Areas…

ActiveTest: Examination, Part III: Code Bloat

2007-01-01T22:39:28-06:00

Another problem you can see from the final example in Part II is a hint of code bloat. I somehow managed to add way more lines of code to just define my original test case of 3 lines. Along the way, I made it very difficult to document what any single method did, because they all provided only pieces. If you wanted to find out what succeeds_on :index actually does, you’ll find yourself trawling through more than a handful of methods. This is the terror of too much premature extraction.

Each Subject received the Spartan method ‘define_behavioral’, whose function is solely to ensure a new method’s name is unique:

      def define_behavioral(prefix, core, specifics = {}, &block)

        proposed = "test_should_#{prefix.to_s.underscore}"
        proposed << "_#{core.to_s.underscore}" if core.respond_to?(:to_s)

        while method_defined?(proposed)
          if proposed =~ /^(.*)(_variant_)(\d+)$/
            proposed = "#{$1}#{$2}#{$3.to_i + 1}" 
          else
            proposed << "_variant_2"
          end
        end

        define_method proposed, &block

        proposed
      end

The bloat is in determining such nondescript method names. It is a result of the poor design that any dynamically generated method would clash with another one. Ultimately, it is better if the developer defines the name of the method, because it would be used in a rake agiledox call. After all, it is their documentation. It did not help matters that this problem was glossed over by writing my own agiledox which loads in everything first, then uses ObjectSpace… another rather ~~inelegant~~ hideous solution.

If we remove the unique naming, define_behavioral is useless; it just wraps define_method. Also, if we remove the dynamic call_#{action}_action and instead use an instance method on ActiveTest::Controller, we reduce method_missing and drop a bunch of unnecessary methods:

class << self
  def succeeds_on(method_name, action, options = {})
    request_method = determine_request_method(action)
    define_method(method_name) do
      call_request_method request_method, action, options
      assert_response :success
      assert_template action
    end
  end
end

Using a direct definition removes the flexibility, but all the code bloat is about meeting some unmentioned need of customising. Until an e-mail is received about it, why do it? Also, better solutions which apply to more than just ActiveTest::Controller may appear naturally. All in all, better not to be so flexible so prematurely.

What Was I Thinking?

I really wanted to write a flexible and easily customised macro language. I thought I could make a clean, elegant macro writer for macro writers, with which test cases were created flexibly and efficiently. The flexibility is what caused all the bloat because I decided to use the expedient of method_missing rather than design a register_behavior type of declaration. Also, I thought it would be better to cater to more edge cases than pigeon-hole ActiveTest to only the most common cases (which could potentially be very few, making ActiveTest pretty useless). The mistake was in trying to cater to these two exclusive cases (common and edge) in their specifics (what they include) rather than how they operate (make a request and check a response).

Coming Up Next: Salvaging Useful Ideas…

ActiveTest: Examination, Part II: Abstracting Without Basis

2006-12-31T23:37:50-06:00

My first mistake started the following cascade effect:

move all assert_x_success and assert_x_failure to class methods
begin defining any test case as a class method
extract the definition of an instance method to the superclass level (ActiveTest::Controller)
extract those superclass methods to abstract methods (ActiveTest::Subject)
fragment assert conditions to be passed up the hierarchy (define_behavioural)
extract fragments to the class level (assert_restful_get_success, method_missing, etc.)
rinse & repeat for each Subject

Rather than inheriting from a single class with all its assertions self-contained, I now inherited from ActiveTest::Base -> Subject -> Controller. Along the way there were sprinkled methods on the class and eigenclass level which helped define the test cases (ActiveTest::Subject.define_behavioral) or provide extra assertions. I had extracted so much that it was a nightmare to understand what was actually happening (a rare problem when writing in Ruby).

An Extraction Case Study from 0.1.0 Beta (abridged)

module ActiveTest
  class Controller < Subject

    class << self

      def succeeds_on(action, options = {})
        define_behavioral(:succeed_on, action, options) do
          send("call_#{action}_action", options)
          send("assert_#{action}_success")
        end
      end

    end

    ...

    def assert_restful_get_success(action, options = {})
      assert_response :success
      assert_template action.to_s
    end

    ...

    def method_missing(method, *args)
      options = args.last.is_a?(Hash) ? args.dup.pop : {}
      if method.to_s =~ /^assert_(.*)_(success|failure)$/
        action, sf = $1, $2
        if action !~ /get|post|put|delete/
          request_method = self.class.determine_request_method(action)
          send("assert_restful_#{request_method}_#{sf}", action, options)
        end
      elsif method.to_s =~ /^call_(.*)_action$/
        request_method = self.class.determine_request_method(action = $1)
        send("call_request_method", request_method, action, options)
      else
        super
      end
    end

    def call_request_method(request_method, action, options = {})
      options = expand_options(options)
      send(request_method, action, options[:parameters])
    end

  end

end

As you can see, I left out quite a few secondary methods which are being called. The first problem here is in the design of defining a behaviour: it gives too many hooks to the user that allow them to customise. If I am trying to address common cases, this should have been a red flag to a bull—I was blind. The result was a ton of method_missing calls because I extracted every call into a dynamic method, including the assertions in a dynamically defined test case (twice, in fact). Awful.

What Was I Thinking?

When I first began writing ActiveTest, I focused solely on making test cases a one-line business. ActiveTest contained just a simple class method on ActiveTest::Subject to define any kind of test case and a number of classes inheriting from it that defined common groups of assertions. Then, I abstracted the entire process into defining ‘behaviours’, approaching each test case as a behaviour request-response pair (in fact, it is a context if you use BDD terminology). I went through a number of phases looking for the cleanest way to implement this idea of behaviours (I aped RSpec, Test::Rails and others). None of them seemed appropriate, so my own metalanguage was written that looked clean (but only when I did not provide parameters). I thought the metaprogramming of tests with a descriptive class method call would be self-explanatory enough.[1]

Take away from this one single maxim, by which you should live and die when you refactor: do not abstract without real-life need. If you don’t base your abstraction on something real, you are theorizing—and we all know that the problem with theories is that they have not been tested.

Coming Up Next: Code Bloat…

Footnotes

¹ Recently I encountered a situation where parameters were impossible to pass. This unforeseen situation (where I needed RFuzz:http://rfuzz.rubyforge.org/) contributed to my realisation that ActiveTest’s current design is useless for anything moderately dynamic or complex.

ActiveTest: Examination, Part I: Removing Transparency

2006-12-31T22:34:58-06:00

The first abstraction I made was to change assert_get_success and its equivalents (get_failure, post_success, etc.) into an abstract idea of common test cases. Obviously every application is going to be treated differently, but to make a suite of tools to remove the most common tests seemed like a useful thing to extract—it’s DRY.

Consequently, I had the following in Test::Unit::TestCase:

  class << self
    def test_should_index_items
      define_method(:test_should_index_items) do
        assert_get_success :index
      end
    end
  end

I then abstracted it a little:

  class << self
    def test_should_get_with_success(action = :index, parameters = {})
      define_method("test_should_#{action}_item") do
        assert_get_success action, parameters
      end
    end
  end

This is basic DRYing up. However, I have at this point already lost sight of the simplest benefit of writing out test cases with Test::Unit: documentation. If someone else uses this, the only thing they know is that their GET succeeds… but in what way? I began documenting what each method does and extracting more and more complex cases which may be ‘common’. Documenting these methods hid the general fault, which was making the entire testing process opaque to the user. Tests started to look like this:

class TestSomething < ActiveTest::Base
  test_should_get_with_success :index
  test_should_get_with_success :show, :id => 1
  test_should_get_with_success :new
  test_should_get_with_success :edit, :id => 1
end

So my first mistake was removing transparency without thinking about what I was beginning to lose. This is quite project-specific. In most cases, making methods opaque and useful is a good thing, but in the case of ActiveTest, it needs to be flexible and transparent. It is hard to beat a series of asserts.[1]

This mistake is probably my least offensive, because some, including myself, may like this feature just to remove all those irritating test_should_index_items, especially when you are writing new controllers left and right.

What Was I Thinking?

I thought that the documenting aspect of testing was purely outlining what everything should do, rather than describing what everything should do. My original idea was to make it easier to document and test at the same time by outlining the cases in each suite. Just in terms of line ratios, I now had a 5-line method which let me write a test case in 1 line. Not bad. It quickly paid for itself because it catered to :index, :new:, :show and :edit. However, I lost all the expressiveness of showing what is being tested.

Coming Up Next: Abstracting Without Basis…

Footnotes

¹ It can be done. In fact, the next generation of Active Test, while rather different, frees up the tester to think completely about the behaviour of his code. I will trade in the brevity of test cases for the flexibility of defining them (for anything, whether Rails application, plugin, gem or 60-line file).

ActiveTest: Examination, Introduction: A Mistake and How To Fix It

2006-12-31T17:21:17-06:00

In trying to bring ActiveTest to a state resembling my original article about ActiveTest, I realised that … it’s a piece of crap. It just isn’t the kind of code which many but myself would find useful and now even I don’t find it that helpful anymore. A total cowpat of a project, sadly.

To help both myself and the community, I will be analysing my mistake in full and mercilessly revealing my thinking process, development and design along the way. Because it may get a little… lengthy, I’ll break it down into a number of articles. The last article will be dedicated entirely to making Active Test useful (which itself is three parts: salvaging useful ideas, redesigning weak areas, and changing purpose).

Obituary to Active Test 0.1.0 Beta

In its death-throes, the Active Test Plugin can still be useful: as an example of what not to do. Before I start laying into my sad miscarriage of an idea, I’ll outline useful ideas which came out of it:

Finding a way to extend Test::Unit which is safe and allows inheritance
Learning different ways of metaprogramming (especially ways to wrap define_method)
Extracting a more granular set of things to test in an application
Learning techniques for self-testing a test library

As with many mistakes, more than half of it is about learning rather than providing something useful. That’s kind of the point, I guess! I didn’t receive a single support e-mail or comment that it has helped anyone. Then I began to realise what I was doing was wrong and had managed to air my dirty laundry in the process.

An overview of the mistakes

With that small obituary to ActiveTest in its current incarnation, let’s look at the problems in the order in which I’ll be discussing them:

Removing test case transparency (an advantage of Test::Unit)
Extracting and abstracting without a real-world need or basis¹
Excessive metaprogramming, which introduces ambiguity into tests (the cardinal sin)
Code bloat, plain and simple

The Idea: the original test case

I first came up with ActiveTest when I looked at this test case (from an old revision of one of my projects):

  def test_should_index_items
     assert_get_success :index
  end

At first glance, this looks almost identical to what ActiveTest currently does. I have a case where I want to ensure that a typical GET request receives a HTTP 200 response. I extrapolated HTTP 200 to HTTP GET into a request-response ‘success’ condition. The method was given a parameter (an action to call) that was passed to get and assert_template. All perfectly fine, a little DRY, but not unreadable. However, this test case and a bunch of others exactly like it formed almost the entire basis of my initial version of Active Test.

Coming Up Next: My First Mistake…

Footnotes

¹ For a milder case of this, called premature extraction, see this article

Superfast Testing How To: In-Memory SQLite3

2006-11-26T16:15:49-06:00

Audience: This is for both application and plugin developers. Especially plugin developers. Always use SQLite3 in-memory databases as your default for plugin tests which require a database.

Some of you may have heard of SQLite3, played with it and decided it is too limited or don’t like the files which it creates in your project tree. There is a less-sung alternative which is much, much faster (actual benchmarks vary because of memory speeds, but for me PostgreSQL is about 33% slower and MySQL about 10% slower). Over a large testing library with thousands of assertions, you will feel that meagre 10% by a comfortable minute. Slight disadvantage: you can’t get that coffee during testing anymore!

There are a handful of sites which have spoken of SQLite3 in-memory databases this year, but to get it working you’ll find the information is a little scattered. The following is a step-by-step ‘how to’ on making your application and plugin tests superfast and lightweight, but please read over these pages first:

Set Up

A lot of this doesn’t need much explanation, so I’m just going to dump out the procedure to get you from MySQL/PostgreSQL/whatever testing to SQLite3 in-memory database testing, making you wait less and code more.

Step 1: Install SQLite3 and friends

If you do not have SQLite3 already, you need to go to the SQLite website and download the appropriate package. If you use Mac OS X, you need to install the SWIG library too.

Once you have the latest SQLite3 distribution, you need to install the sqlite3-ruby gem:

  user@host railsproj$ sudo gem install sqlite3-ruby --source code.whytheluckystiff.net

Select the highest ‘(ruby)’ version (1.1.0.1 at the time of this article). You should see it compile a native extension and install without a hitch.

Note: If you already have sqlite3-ruby installed but do not remember seeing a native extension being compiled, you should uninstall (`gem uninstall sqlite3-ruby`) and install it again with the SWIG library.

Step 2: Install the Memory Test Fix plugin

  user@host railsproj$ ./script/plugin install http://topfunky.net/svn/plugins/memory_test_fix

Step 3: Change RAILS_ROOT/config/database.yml

test:
  adapter: sqlite3
  database: ":memory:"
  verbosity: quiet

Note: remove `verbosity: quiet` at the beginning to make sure SQLite3 is loading your db/schema.rb file. If you are not using Migrations or db/schema.rb, you will also need to set `config.active_record.schema_format = :sql` in your environment files and point Rails to the correct schema file.

Step 4: Test it

  user@host railsproj$ rake test

Notes

As TopFunky says: if you use database specific SQL, using SQLite3 will only get in the way.

ActiveTest: TestHelper Bug Fix

2006-11-26T15:46:18-06:00

There was an issue with test_helper variables set on Test::Unit::TestCase not being set properly. Values would always be the default because of the way class inherited attributes are passed down the hierarchy. You now need to require the active_test framework and any other plugins you have installed at the end of RAILS_ROOT/test/test_helper.rb.

Also, plugin developers should never require their libraries through init. Instead, there should be a file with the name of their plugin in PLUGIN_ROOT/lib. For example, active_test_authentication has a file at PLUGIN_ROOT/lib/active_test_authentication.rb. This is the file which is loaded when `require ‘active_test_authentication’` is called at the end of test_helper.

This fix actually alleviates more than google issue 2. When loaded plugins with a name lexicographically earlier than active_test, active_test would automatically be loaded. This is trivial, but not intended. In any case, you must now require both the active_test framework and any plugins you use at the end of test_helper.