Will Leinweber

Quick Guide to Celerity

2009-06-09T04:17:57-05:00

I’ll assume you already know and love Cucumber and are using it. I’ll go even farther and assume you’re writing your steps in Webrat since that’s what cucumber ships with. Now, I really like webrat. It’s a wonderful DSL and fun to work with, but it does have one problem: no javascript, no ajax.

Enter Celerity

Celerity is a headless browser for jruby that can do javascript. Now you can run your cucumber features that need to do ajax calls, and stay on the command line or CI box. It gets better though. The API is compatible with Watir, so you have the option of driving a real browser at any time. It’s not as expressive as webrat, but since you reusing cucumber steps is easy, it’s not a huge deal.

Don’t want jruby?

If you don’t want to run your whole app in jruby, checkout langalex-culerity. It’s a gem that will let you drive celerity from mri. It also comes with a common_celerity.rb file to replace most of your webrat_steps. We ran into a few steps that were missing or worked a little differently, but porting over didn’t take long at all.

That’s it!

Honestly, there’s not much to getting it working, so I’ve skipped doing a detailed step-by-step style guide. The documentation on getting jruby, celerity, and culerity installed is all you need. There weren’t any surprises. If your app has uses a fair amount of javascript, it’s worth your time to check this out.

Acts as Conference Talk

2009-02-09T18:10:57-06:00

I just got back from acts_as_conference and it was great! I promised a few people that I’d put up my slides so here they are:

Thanks again to everyone who made this a fantastic conference.

Video

Update (2/13): Confreaks just posted all the videos for acts_as_conference. There were a lot of great talks so be sure to check them out. The full page for my talk has links for ridiculously high quality versions.

Profiling RSpec

2009-02-01T17:17:57-06:00

If you have anything approaching a large test suite, the time spent running them becomes an important factor. The longer they take, the less often I tend to run them, which is bad. And I’m probably not alone in that. Ideally you’d be refactoring all your tests, removing duplication and such. If you’re just interested in speeding things up, though it’d be nice to know where the slow specs are.

RSpec ships with a profile formatter that, but it leaves a lot to be desired. It only reports the top 10 slowest examples, and with several thousand specs, the top ten just isn’t enough. It also doesn’t provide any sort of statistics which are nice to track over time, say on the CI box. So let’s improve this.

RSpec Formatters

Just run spec by itself, and you’ll see there are about a half dozen formatters. Let’s take a closer look at the existing profile formatter. It records the time taken for each example and then prints the top ten in #start_dump. The recording part is fine, we just need to do a little more work in that dump method there.

def start_dump

   super

   

   example_times</span> = <span class="iv">example_times.sort_by do |description, example, time|

     time

   end.reverse

   

   times = example_times</span>.map {|d,e,t| t}
   mean = times.mean
   stddev = times.standard_deviation
   k = <span class="i">2</span>
   <span class="iv">example_times.reject! { |d,e,t| t < mean + k * stddev }

   

   output</span>.puts <span class="s"><span class="dl">&quot;</span><span class="ch">\n</span><span class="ch">\n</span><span class="k">Top </span><span class="il"><span class="idl">#{</span><span class="iv">example_times.size} slowest examples:\n"

   

   example_times</span>.each <span class="r">do</span> |description, example, time|
     <span class="iv">output.print red(sprintf("%.7f", time))

     output</span>.puts <span class="s"><span class="dl">&quot;</span><span class="ch">\t</span><span class="il"><span class="idl">#{</span>description<span class="idl">}</span></span><span class="k"> </span><span class="il"><span class="idl">#{</span>example<span class="idl">}</span></span><span class="dl">&quot;</span></span>
   <span class="r">end</span>
   
   <span class="iv">output.puts "\n\nStats"

   

   output</span>.puts <span class="s"><span class="dl">&quot;</span><span class="k">Mean:</span><span class="ch">\t</span><span class="il"><span class="idl">#{</span><span class="s"><span class="dl">&quot;</span><span class="k">%.5f</span><span class="dl">&quot;</span></span> % mean<span class="idl">}</span></span><span class="dl">&quot;</span></span>
   <span class="iv">output.puts "StdDev:\t#{"%.5f" % stddev}"

   output</span>.puts <span class="s"><span class="dl">&quot;</span><span class="k">Total:</span><span class="ch">\t</span><span class="il"><span class="idl">#{</span>times.size<span class="idl">}</span></span><span class="dl">&quot;</span></span>
   <span class="iv">output.puts "Slow:\t#{example_times</span>.size<span class="idl">}</span></span><span class="dl">&quot;</span></span>
   
   <span class="iv">output.flush

 end

So what’s going on here? It’s not too much different than the built in one, except we compute the standard deviation and mean of the set of times and use them grab all of the slow specs. This of course assumes that you already have #standard_deviation and #mean on Array. If you don’t or don’t feel comfortable adding those to the class, they can just as easily be methods in this module.

Once we have those, we keep the ones that are 2 deviations from the mean. The choice of two is arbitrary, but I found this gave just enough to get a good look at the slow specs. Finally we print some of the stats we collected. This is great to run on the your CI box. It still checks the whole suite like rake spec, but now there’s the chance to record the stats and see how the tests are evolving over time.

Using the new Formatter

To get this going, we have to make a class that inherits from Spec::Runner::Formatter::ProgressBarFormatter, and that file will also need to require "spec/runner/formatter/progress_bar_formatter". The rest is exactly the same as the built in ProfileFormatter. For example, name this new formatter RobustProfileFormatter, and put it in spec/config. To run it once off the command looks like this: spec -c -r spec/config/robust_profile_formatter.rb -f RobustProfileFormatter. The -c option turns on color output, -r requires the new code, and -f chooses it as the formatter.

That’s a lot to type out, and a rake task would be much nicer. In lib/tasks/rspec.rake there are a bunch of tasks. Inside the spec namespace is a good place to put our new task:

desc "Profile specs"
Spec::Rake::SpecTask.new(:profile) do |t|
  t.spec_opts = ["-c", 
                 "-r spec/config/robust_profile_formatter.rb", 
                 "-f RobustProfileFormatter"]
  t.spec_files = FileList["spec/**/*/*_spec.rb"]
end




desc "Profile specs"

Spec::Rake::SpecTask.new(:profile) do |t|

  t.spec_opts = ["-c", 

                 "-r spec/config/robust_profile_formatter.rb", 

                 "-f RobustProfileFormatter"]

  t.spec_files = FileList["spec///*_spec.rb"]

end




That’s all there is to it, go and speed up your specs!
Stopwatch image from flickr

Switching to Webby

2008-11-12T02:32:44-06:00

I made the transition to webby for the blog. Last time I tried to upgrade typo, the migrations broke, everything blew up, tears and sorrow. I realized I didn’t really need typo, or anything like that.

Webby is, at its core, a set of fancy rake tasks that will compile your site to html from ruby. It can also deploy over ssh or rsync to your site. If you’re the only one writing for your blog, and you know how to code, webby is the way to go. There isn’t really any need for a web UI to write posts, or any of the features (and overhead) you get from a blogging engine. And a huge plus: you can throw the whole thing in git.

Other people have written quality articles and tutorials on webby. Rather than repeating everything, I’ll just link them here:

Mike Clark – Building Static Websites with Webby
Mangus Holm – Building a Website with Webby
github search: ‘webby’ – some people have shared their source for their blogs. Reading code is always a great place to start.

Closures in Ruby

2008-09-15T01:30:11-05:00

I remember reading that Proc.new and lambda weren't exactly the same, but I never ran into any problems so I never bothered to look up exactly what the differences were. I just always used lambda, which as it turns out, is probably for the best. Paul Cantrell wrote a article/script hybrid that does a great job showing and telling all of the differences with the 7 (who knew?) different ways you can have a closure. It's worth a read, check it out. Here's the summary table that's about halfway through.


                                                    "return" returns
                                                    from closure or
                                   True closure?    declaring context?     Arity check?
                                   ---------------  -------------------    -------------------
1. block (called with yield)       N                declaring              no
2. block (&b => f(&b) => yield)    N                declaring              no
3. block (&b => b.call)            Y except return  declaring              warn on too few
4. Proc.new                        Y except return  declaring              warn on too few
5. proc                                <<< alias for lambda in 1.8, Proc.new in 1.9 >>>
6. lambda                          Y                closure                yes, except arity 1
7. method                          Y                closure                yes

Why You Should Use CouchDB

2008-11-16T12:00:00-06:00

New Approach to an Old Problem

Your applications need to store and retrieve data. That’s nothing new. CouchDB, and other document databases like Amazon SimpleDB, are new to the game however. Okay, I’ll admit the general concept is some 20 years old, but it’s safe to say that they’ve recently started to gain popularity over the last year or so.

Learning a document database is worthwhile, even if you don’t end up using it for anything serious in the near future. You’ll gain new insights on how to solve your current problems. Don’t take my word for it, there’s that line everyone quotes from The Pragmatic Programmer: Learn at least one new database paradigm a year, or something like that.

What exactly makes CouchDB so great? Read on.

Stable and Scalable

When you update a document in CouchDB, it doesn’t go in and change the existing document. Instead, it just adds a new version. The upshot is that your data is never in a bad state, so even in the case of a power failure, you’re not going to lose data or spend time verifying against corruption when you start up again. All these extra versions are deleted when you compact the database, so there isn’t much of a storage penalty because of it.

Other databases have low erlangs

It’s written in Erlang, and benefits from the stability and concurrency of the language its design idioms. Also, unlike current relational databases that try and make a single query as fast as possible, CouchDB tries to maintain consistent performance as the number of queries grows.

Another big plus is that it doesn’t have a custom binary protocol. It uses regular HTTP. This means you can use existing http infrastructure tools such as load balancers and clustering proxies. Better still, it’s RESTful, so you interact with it using the CRUD operations we’ve come to know and love.

The way you query data from CouchDB is through views which are written using the map/reduce approach. These views themselves are indexed, so they’re very fast as long as you keep the indices fresh. While it’s not an option now, it’s possible that in the future the mapping step could be distributed amongst several nodes which could make it ridiculously fast.

Deep Structures, No Schemas

Data has it’s own natural shape, and I know I’m not the only one who is sick and tired of fighting that fact. It is often an uphill battle cramming models into relational records. Your data is more complex than a few simple strings, numbers, and boolean values. You have lists, you have hashes, and even small variations between instances of the same object. This is unacceptable in the relational world.

ORMs like ActiveRecord and DataMapper have done a lot to to ease the pain and abstract away the nastiness of SQL. It’s not enough though. It’s like treating the symptoms and not the underlying condition. You still have to worry about joins, normalization, and other artifacts from relational databases. These issues leak their way up into your models where they don’t belong, and obscure more important logic.

All of this isn’t an issue with CouchDB, and that’s the biggest selling point for me. The fields in your documents can be hashes, they can be arrays, they can be arrays of hashes. Anything that can be serialized to JSON. It’s about documents — not records.

As a quick example, rather than having to create an assets table for uploads, you can store all the metadata in a single field. That field can live right on the object that is responsible for it, where it belongs. Or you could store all the comments for a blog post inline with the post itself. There’s a good article comparing inline vs. separate storage that is worth a read.

Downsides

Nothing is without drawbacks, and CouchDB is no exception. For one, it’s still alpha. That’s not to say it’s buggy and unusable — far from it — but there are likely to be changes before they hit 1.0. I can’t say to what extent this will be an issue, but you should be aware of that.

There is no security model. This turns out to not be that big of a deal, but it’s far from ideal. You can lock down the port it runs on to only talk to localhost, which I think happens by default.

The first time you run a view it will be slow. It has to go through every document and build the index. After that though, it only has to go through the new or changed documents and is significantly faster. You can make sure your views are fresh by running them after every couple hundred updates or every 10 minutes or so. But if you’re always running unique views, CouchDB probably isn’t a good choice.

Getting Started

Go ahead and compile CouchDB from the svn head rather than going for the last release. You can find instructions over on their wiki. You’ll also want a persistence layer. So far, I’ve been most impressed with Couch Potato. I’ve been using it with Merb, which has been great. You just need to add it to your config/dependencies.rb like this

dependency "langalex-couch_potato", "0.1", :require_as => "couch_potato"

then add this to your config/init.rb

Merb::BootLoader.before_app_loads do

  CouchPotato::Config.database_name = "appname_#{Merb.environment}"

  require ‘json/pure’

end

You might not need that json/pure fix. I was having some problem where #to_json was declared multiple times. It’d be nice if there was a merb_couchpotato gem so you could just say use_orm :couchpotato, and maybe use database.yml instead, but it’s not really necessary yet.

That’s really all it takes. Go checkout the readme for Couch Potato and its introduction post.

Time to relax.

Faking "or shift left"

2008-09-04T01:30:11-05:00

I've often been annoyed at the lack of ||<< after being spoiled with ||= for so long. The workaround I've typically used is to go ahead and use ||= [ ] to make sure I have an empty array. That's ugly though. You're dropping down a level and writing the "how" instead of the "why".

Today some code Jay Fields posted had the answer I've been looking for. It's not quite as elegant has having the operator itself, but it does the job of tucking away the ugly detail of ensuring you have an empty array to shift on.

1 def method_stack
2   @method_stack ||= []
3 end
4  
5 def method_missing(sym, *args)
6   method_stack << [sym, args]
7   self
8 end

We're still using ||= [ ] but it is hidden away in its own spot, which I find to be much cleaner.

RubyGems and Leopard

2008-01-12T00:30:11-06:00

I just got my replacement iMac (the speakers were shot on the first one) and was going about getting my development environment back together. While trying to instal datamapper, it choked on the json gem.

ruby extconf.rb install json
can't find header files for ruby.

As it turns out, the solution is very simple. I didn't have Xcode tools from the install disc on there yet. Since it's something you just put on once then forget about, it took me a while to realize what was going on. Hopefully this post helps others figure it out quicker than I did if they stumble across it. Also remember to update RubyGems. Leopard comes with 0.9.4 and we're up to 1.0.1 right now.

 sudo gem update --system

More on setting up Leopard

2008-01-15T00:30:11-06:00

After posting about setting up a new mac, I started to put all the other things I found myself missing on a new install. Aside from the standard adium, quicksilver, colloquy, and others there are a few things that are good enough to share.

Blurminal

I'm a long time user of iTerm but I've decided to give leopard's new tabbed terminal a chance. The one big frustrating thing is it lacks iTerm's ability to switch between tabs with command+arrow keys. However, a big plus is Ciarin Walsh's Blurminal. Instead of just having a percentage of transparency, it slightly blurs whatever is behind the terminal.

macFusion

This is a nice gui for the OS X implementation of fuse. This allows you to mount an ssh directory locally right from the menu bar which is really awesome. One annoyance: in tiger, when you mounted something, it would appear on the desktop and the sidebar of finder, which was nice. With leopard though, that's not happening. You can still go up to the menu bar and click on one of your mounts so it's not a huge problem.

skitch

Sktich is a little app that lets you take screenshots, edit them up, and upload them in one go. It's turning out to be really useful. It's still in beta right now, and I'm not sure when it will be finished. One thing it needs is keyed sftp uploading. It has support for sftp, but with usernames and passwords only. They haven't said how much it's going to be once it's out of beta, but they're doing a good job of getting people hooked before that happens. First one is always free.

blue phone elite

The last one I'm going to talk about is blue phone elite. If you have a supported phone, this will let you use your mac as a handsfree (though I've not had great success with this), send and receive text messages through it, flash up caller id on the screen, pause iTunes when you're on a call and resume when the call is over, and a few other neat things. I have it set to run an applescript that syncs my phone when I come in range, which helps a lot because I'd otherwise never remember to sync. I was having some problems with the 2.0 release so I went back to using 1.x. It looks like they've had a few more releases, so I'll give 2.x a try again.

Merb and Datamapper

2008-01-20T00:30:11-06:00

With the recent release of Merb 0.5, I've decided to use it along with datamapper for one of my new projects. It isn't different enough to be completely foreign, but enough to be a refreshing change. I haven't done a whole lot with either yet, but I have had a patch accepted, ran into a huge, annoying problem with autotest, and found out how to watch the SQL datamaper is generating.

stats patch

While I was in #merb earlier today, hassox posted some current benchmarks. Being somewhat a stickler for statistics, I had to point out that means alone were meaningless. Not to get into all of the boring details, two distributions can have the same mean, but be completely different. Here is an image I made in Mathematica and sktich to show why:

They all have the same mean, but a random pick from the distribution with the smaller variance of 0.5 is more likely to be actually be near the mean than with either of the ones with higher variances. If you apply this to load times, two severs could serve pages just as quickly on average. However, the one with the higher variance is going to seem a lot slower, since there are more slow points in the set.

I submitted a patch to add the standard deviations to each of the tests and clean up the output a bit.

AutoTest problem

I was trying to get autotest up and running but kept running into this problem:

/Library/Ruby/Site/1.8/rubygems/custom_require.rb:27:in `gem_original_require': 
no such file to load -- rspec_autotest (LoadError)
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:27:in `require'
from /Library/Ruby/Gems/1.8/gems/zentest-3.5.0/bin/autotest:35
from /usr/bin/autotest:19:in `load'
from /usr/bin/autotest:19

After futilely searching for the problem for a long time, and asking in #merb where someone else was having the same problem, I was about to give up and just run my tests by hand (oh no!). I decided to check the versions of all my gems, and it turned out that zentest was only 3.5.0 when the latest was 3.7.2. I thought that was odd, because I just installed the gem that night. Odder still, updating the gem did nothing to increase the version. It turns out there is zentest and there is ZenTest

$ gem list -r | grep -i zentest
zentest (3.5.0)
ZenTest (3.7.2, 3.7.1, 3.7.0, 3.6.1, 3.6.0, 3.5.2, 3.5.1, 3.4.3, 3.4.2, 3.4.1, 3.4.0, 3.3.0, 3.2.0, 3.1.0, 3.0.0)

edit: As of Jan 20, 2008 this seems to be fixed, and there is only ZenTest

Uninstalling zentest and installing ZenTest fixed the problem.

Watching Datamapper's SQL

This is very easy to do. Just edit your config/database.yml file to include

1   :log_stream: STDOUT
2   :log_level: 0

and you can see the SQL in your merb process.