Continuous development

Zend Framework app's PHPUnit suite 3x faster

Sun, 03 Oct 2010 17:02:00 -0700

The background

My regular readers may have already noticed I don’t even dare writing code without TDD/BDD-ing it from left to right. Unfortunately, my current job is based on Zend Framework. I say “unfortunately”, as (amongst other drawbacks) ZF’s architecture is far from professional in terms of testability.

We currently have 600+ PHPUnit test cases, most of them for models and controllers, but quite some also for view helpers, bootstrap and even some views. The whole suite is taking around 90 seconds to run on my machine against a MySQL database (modelled with Zend_Db* in an active record/table data gateway manner).

In this post I’ll be showing how I managed to decrease the time from ~90 seconds to ~30 seconds.

The problem

As I’ve mentioned before, Zend Framework isn’t really encouraging developers to do TDD (on the other hand, this shouldn’t be a surprise in the PHP community). The application architecture is based on singletons all around (front controller, session, layout, helper broker, registry, etc.), the bootstrap doesn’t “separate the cacheable from the non-cacheable”, there’s no built-in support for dependency injection, the ORM is cumbersome to stub out.

Our approach used to involve calling Our_Db_Table_Abstract::truncate() on each table used by the given test case. Not only was this inelegant, mundane to maintain and easy to forget, but as it later turned out, it was the biggest performance bottleneck. We’d either put it at the end of the test case – which would not be executed if the assertion failed and we’d end up with a dirty db – or we’d put it in tearDown() – which meant it ran even if a particular test didn’t use the given table.

Another problem was controller tests. We needed to explicitly declare, which tables would be used by the test. Even with extra-simple syntax, it was still terribly inelegant. And if we’d forget about one table…

This is how it would look like in model tests:

And in the controller:

As you can see, there’s a lot of verbosity, duplication and noise in the code. We need to do plenty of “plumbing” for the tasks that should be automatic.

The solution

The solution I came up with was inspired by Rails, and other frameworks. It can be summarized as: wrap each test case in a transaction, rolling it back after the test case.

I implemented the solution using a PHPUnit listener, hooking up to startTest() and endTest() callbacks. Here’s the appropriate code fragment:

These callbacks wrap a single test case, so the cycle is:

startTest()
setUp()
test*()
tearDown()
endTest()

And here’s the appropriate fragment of phpunit.xml (we’re using PHPUnit 3.5):

Excellent, that’s an elegant solution with a properly separated concern. I got rid of all the ugly truncate()s and sped up the tests significantly.

Caveats

Of course, life’s not that full of sunshine and bunnies.

Problem #1

First problem I encountered, was bootstrapping the application inside the cycle mentioned above (anywhere within steps 2-4). This happens in controller, view, view helper and bootstrap test cases. Let’s go through a controller test case. We:

startTest(), which begins a transaction
setUp(), which bootstraps the application, which in turn puts a new ‘database’ object in Zend_Registry
…
…
endTest() calls rollBack(), which explodes with No active transaction

This basically means that we’re running beginTransaction() and rollBack() on a different database connection.

One solution for this problem is to use a persistent PDO connection. This will most likely cause both methods to run on two different PHP objects, but same connection resource. To do this, just add the following setting to your database config:

params.persistent = true

I wanted something more robust, though, so I rollBack() manually right before bootstrapping and beginTransaction() immediately afterwards. It’s plumbing again, but at least it’s abstracted in a base class. Here’s an example:

Problem #2

Second caveat was quite obvious. Some of our tables were using MyISAM (which doesn’t support transactions), not InnoDB. I solved it by creating an appropriate migration, like so:

ALTER TABLE Country ENGINE = InnoDB;

Problem #3

Some tests were still failing after all these changes. The problem was that they were assuming primary key values for newly created rows, i.e. expected a new row to have an ID of 1. This was true when we were truncating, but is no longer true for rollbacks, since autoincrement counters are not reset. I refactored it in a following way:

Problem #4

Not really a problem for me, but I should probably mention this. We could observe the performance boost on all developers' machines with Ubuntu. Some of our colleagues are using Windows, and they had a very small gain in speed (around 5 seconds). But their benchmarks are generally tilted, since sometimes the suite would finish under 2 minutes, and sometimes after 5+ minutes. YMMV.

Conclusion

There you go, with a conceptually simple and elegant solution you can achieve quite a dramatic improvement in performance of PHPUnit suite for a Zend Framework application. My next idea is to cache the bootstrap input (i.e. routing). What’s yours?

And here’s the evidence for sceptics:

Before:

$ phpunit
PHPUnit 3.5.0 by Sebastian Bergmann.

............................................................  60 / 630
............................................................ 120 / 630
..............S.SSSS............................S........SS. 180 / 630
S........................................................... 240 / 630
..................................S.S....................... 300 / 630
............................................................ 360 / 630
.........................SS.......F.S...........S........... 420 / 630
.....................S...................................... 480 / 630
..........................I................................. 540 / 630
..............................................S............. 600 / 630
..............................

Time: 01:24, Memory: 136.75Mb

FAILURES!
Tests: 630, Assertions: 830, Failures: 1, Incomplete: 1, Skipped: 17.

After:

$ phpunit 
PHPUnit 3.5.0 by Sebastian Bergmann.

............................................................  60 / 630
............................................................ 120 / 630
..............S.SSSS............................S........SS. 180 / 630
S........................................................... 240 / 630
..................................S.S....................... 300 / 630
............................................................ 360 / 630
.........................SS.......F.S...........S........... 420 / 630
.....................S...................................... 480 / 630
..........................I................................. 540 / 630
..............................................S............. 600 / 630
..............................

Time: 29 seconds, Memory: 131.50Mb

FAILURES!
Tests: 630, Assertions: 830, Failures: 1, Incomplete: 1, Skipped: 17.

Permalink | Leave a comment »

Autotest-like PHPUnit runner for a Zend Framework application

Sat, 28 Aug 2010 05:06:00 -0700

Whenever working on a Ruby project, I’d run autotest with test_notifier to get immediate feedback on my code. Unfortunately, I don’t know a similar utility integrated with Zend Framework and PHPUnit, so I came up with my own, using the watchr gem.

Pseudo-autotest for PHPUnit

My solution is not as well designed and portable as test_notifier, and it doesn’t play as nicely with my Zend application as autotest does with RSpec and Rails. It’s more of a quick hack to cover my needs 80% of the time.

The watchr gem watches changes to files inside a directory and provides a DSL to define tasks to perform on changes of certain files.

First of all, I needed a mapping between my code files and test files for the following rules:

I just want to match files ending with .php, or Test.php for tests/ directory
application code lives in application/, tests are in tests/application
library code lives in library/My, tests live in tests/library

Here are the rules in watchr syntax:

watch '^tests/.*Test\.php' do |match|
  phpunit match[0]
end

watch '^library/My/(.*)\.php' do |match|
  phpunit "tests/library/#{match[1]}Test.php"
end

watch '^application/(.*)\.php' do |match|
  phpunit "tests/application/#{match[1]}Test.php"
end

phpunit is a method defined later that runs the PHPUnit test.

Bubble notifications with Ubuntu’s notify-osd

The phpunit method could as well be as simple as:

def phpunit file
  system("phpunit #{file}") if File.exists?(file)
end

But I wanted not to have to switch to the terminal on every save, instead, I wanted to show a sexy notification bubble. This actually involved more effort than I had previously imagined, but now I have different icons and meaningful messages depending on the test result.

I’m using open3 to capture PHPUnit’s output and echo it without delay on the terminal. Here’s the code:

def say what
  puts "\n   [#{Time.now.strftime('%H:%M:%S')}] #{what}\n"
end

def phpunit file
  if File.exists? file
    cmd = "phpunit #{file} 2>&1" # redirect stderr to stdout
    say "About to run `#{cmd}`"
    _, out, _ = Open3.popen3(cmd) # care only about stdout

    previous = last = nil

    until out.eof?
      previous = last # remember last two lines of the PHPUnit output
      puts last = out.gets
    end

    file_name = File.basename(file)
    image, summary, message = case
    when last =~ /\AOK/ # PHPUnit is green
      ['dialog-ok', file_name, last.gsub('OK (', '').gsub(')', '')]
    when previous =~ /\AOK, but incomplete or skipped tests/ # PHPUnit is yellow
      ['dialog-question', file_name, last]
    when last =~ /\APHP/ # PHP Fatal error, PHPUnit process crashed
      ['dialog-error', 'Fatal error!', last]
    else # PHPUnit is red
      ['dialog-warning', previous, last]
    end

    # `--hint=string:x-canonical-private-synchronous:` is a workaround for `-t`
    `notify-send  --hint=string:x-canonical-private-synchronous: -i #{image} "#{summary}" "#{message}"`
    say "waiting..."
  end
end

I wanted the bubble to hide quickly, and found the -t option to specify expire time, but due to this ridiculous design decision, I had to use a workaround. Now, when my tests pass, I’m seeing:

Yellow is:

Red is:

And PHP fatal error is:

Which tools do you use to make your PHPUnit experience more pleasant?

Permalink | Leave a comment »

View specs with Rails 3, RSpec 2b20 and Webrat

Sat, 28 Aug 2010 02:17:00 -0700

undefined method ‘has_selector?’ for String

After a recent update of my gems (caused by upgrading from Rails 3 RC to Rails 3 RC2), I got quite a lot of this type of errors:

Failure/Error: rendered.should have_selector(:img, :src => @photo.image.url(:preview))
  undefined method `has_selector?' for String
  # ./spec/views/photos/show.html.haml_spec.rb:13

After some digging I found that in rspec-rails 2b20 Webrat is no longer required by default. You can choose between Webrat and Capybara, but you have to add either gem to your Gemfile.

After adding gem 'webrat' to my Gemfile, the have_selector matcher works as expected.

PS. I also changed mysql to mysql2, as recommended by Rails and added the bundler task to Capistrano recipe (require 'bundler/capistrano' on top of config/deploy.rb).

Permalink | Leave a comment »

Upgrading a Rails 3 beta 4 application to Rails 3 RC

Wed, 28 Jul 2010 06:44:00 -0700

As soon as I’ve heard that Rails 3 RC is out, I decided to upgrade my application. I have to admit it was less smooth than I’d imagined, mostly due to Bundler, but after a while I got it to work.

Bundler adventures

First thing I did was changing my Gemfile from

gem "rails", "3.0.0.beta4"

gem "rails", "3.0.0.rc"

and ran

# we're not using production's mysql gem in development
bundle install --without production

getting

No compatible versions could be found for required dependencies:
Conflict on: "bundler":
* bundler (0.9.26) activated by bundler (= 0.9.26, runtime)
* bundler (>= 1.0.0.rc.1, runtime) required by rails (= 3.0.0.rc, runtime)
All possible versions of origin requirements conflict.

I installed new Bundler:

sudo gem uninstall bundler
sudo gem install bundler --pre

and successfully retried bundle install --without production.

First surprise was that Bundler tried to install the gems along other system gems (I was actually hoping to have them in my ~/.bundle/). It seems that ~/.bundle/ is no longer used by Bundler (unless you explicitly tell bundle install to install the gems there). It asked me:

Enter your password to install the bundled RubyGems to your system:

Well, I’m fine with this on my development machine, but what about my production server, where the application user is not in sudoers. Ah, yet another dimension of complexity…

Second surprise – new Bundler left a Gemfile.lock (not really what I expected, kind of annoying – anyone care to explain to me the rationale behind it?).

Changes in default skeleton files

I ran all the specs and got the following:

DEPRECATION WARNING: Calling a method in Rails::Application is deprecated,
  please call it directly in your application constant My::Application.
  (called from method_missing at /usr/lib/ruby/gems/1.8/gems/railties-3.0.0.rc/lib/rails/application.rb:77)

You did not specify how you would like Rails to report deprecation
  notices for your test environment, please set config.active_support.deprecation
  to :stderr at config/environments/test.rb

DEPRECATION WARNING: You are using the old router DSL which will be
  removed in Rails 3.1. Please check how to update your routes file at: http://www.engineyard.com/blog/2010/the-lowdown-on-routes-in-rails-3/.
  (called from initialize at /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.0.rc/lib/action_dispatch/routing/deprecated_mapper.rb:33)

/usr/lib/ruby/gems/1.8/gems/activemodel-3.0.0.rc/lib/active_model/validations/validates.rb:87:in `validates':
  Unknown validator: 'email' (ArgumentError)

Five things here:

calling Rails.application and Rails::Application was deprecated in favour of My::Application, so I replaced it in all config/* files and Rakefile
a new config option config.active_support.deprecation was added
generally the files in config/ have undergone some changes
lib/ directory was no longer autoloaded
routes.draw now expects a block with arity of zero, not one, so I got rid of |map| in config/routes.rb

Below are the most important parts you’ll need to change. For a full list of changes, just generate a new app and diff everything inside the config/ directory.

Rakefile
-Rails::Application.load_tasks
+My::Application.load_tasks


config/application.rb
-Bundler.require :default, Rails.env
+Bundler.require(:default, Rails.env) if defined?(Bundler)

-    # config.load_paths += %W( #{config.root}/extras )
+    config.autoload_paths += %W( #{config.root}/lib )
+    # config.action_view.javascript_expansions[:defaults] = %w(jquery rails)
+    config.encoding = "utf-8"

-    config.filter_parameters << :password
+    config.filter_parameters += [:password]


config/boot.rb
+require 'rubygems'
+gemfile = File.expand_path('../../Gemfile', __FILE__)

-  require File.expand_path('../../.bundle/environment', __FILE__)
-rescue LoadError
-  require 'rubygems'
+  ENV['BUNDLE_GEMFILE'] = gemfile
+rescue Bundler::GemNotFound => e
+  STDERR.puts e.message
+  STDERR.puts "Try running `bundle install`."
+  exit!
+end if File.exist?(gemfile)


config/environments/development.rb
+  config.active_support.deprecation = :log


config/environments/production.rb
+  # Specifies the header that your server uses for sending files
+  config.action_dispatch.x_sendfile_header = "X-Sendfile"
+
+  # For nginx:
+  # config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect'
+
+  # If you have no front-end server that supports something like X-Sendfile,
+  # just comment this out and Rails will serve the files

+  config.i18n.fallbacks = true
+  config.active_support.deprecation = :notify


config/environments/test.rb
+  config.action_dispatch.show_exceptions = false
+  config.active_support.deprecation = :stderr


config/initializers/secret_token.rb
-Rails.application.config.secret_token = '(...)'
+My::Application.config.secret_token = '(...)'


config/initializers/session_store.rb
-Rails.application.config.session_store :cookie_store, :key => '_my_session'
+My::Application.config.session_store :cookie_store, :key => '_my_session'


config/routes.rb
-My::Application.routes.draw do |map|
+My::Application.routes.draw do

Mission complete

After those fixes, all specs went through, I didn’t get any deprecation warnings or errors. Welcome, Rails 3 RC!

I also had some adventures deploying this on production, but that’s a story for a separate article, so stay tuned.

On a side note

Could anyone clear this up for me why some gems' executables end up in /usr/bin/ while others in /usr/lib/ruby/gems/1.8/bin/? I can’t notice the pattern…

Permalink | Leave a comment »

Speccing partials in Rails 3 and RSpec 2

Sat, 17 Jul 2010 16:15:00 -0700

Typically with RSpec 2 and Rails 3, when I spec a view, I’d do something like:

describe 'posts/show.html.haml' do
  it 'should render post title' do
    assign(:post, stub_model(Post, :title => 'Abracadabra'))
    render
    rendered.should have_selector(:h1, :content => 'Abracadabra') # and/or 'view.should ...'
  end
end

In this example, assign(key, value) is used to provide a @post variable to the template.

The big question, however, is how would I assign locals for a specced partial? The answer I found (with my new best friend, bundle open actionpack) is the following:

describe 'posts/_post.html.haml' do
  it 'should render post title' do
    post = stub_model(Post, :title => 'Abracadabra')
    render post, :foo => 'bar'
    rendered.should have_selector(:h1, :content => 'Abracadabra')
  end
end

Here, I’m just calling render identically to how I would in a view, the partial name gets inferred from the model’s metadata, the second argument is an optional hash of other locals. So, in this example, the partial has access to:

post # => #<Post id: nil, title: "Abracadabra", ...>
foo # => "bar"

What if the partial doesn’t have a corresponding model class? That’s simple:

describe 'posts/_form.html.haml' do
  it 'should render an input for post title' do
    post = stub_model(Post)
    render 'form', :post => post, :foo => 'bar'
    rendered.should have_selector('input#post_title')
  end
end

Here, since the first argument of render is a String, Rails treats it as an explicit partial name and just provides the local variables post and foo.

One tiny thing that’s bothering me about this, is having to pass in 'form' to render when it easily could’ve been inferred by the description 'posts/_form.html.haml'. Well, I guess I’ll just have to live with it.

By the way: view helpers not included in the spec’s `view` object

If you’re running into this issue (like me) where your helpers are not include’d in the specced view and you’re getting all these undefined method errors, it’s a known bug. A quick hackish solution is to explicitly view.extend MyHelper in the examples.

Permalink | Leave a comment »

Rails 3 and ZenTest

Sat, 17 Jul 2010 06:05:00 -0700

The background

I used to work exclusively with Rails 2.x until about a year ago, then I took on a PHP project (ugh…), so when my friend recently offered me to do a quasi-startup project together, I didn’t think twice and convinced him to use Rails (I think Rails 3.0.0.beta1 one out at that point).

It took us quite some time to get through all the obstacles related with the general beta-lity of our environment (Rails3, RSpec2, Bundler, Paperclip, some other plugins with undetermined Rails 3 compatibility), especially considering that my friend needed a Rails crash-course and I needed to undust all my Rails skills and working habits.

The problem

One thing I really missed in my workflow was ZenTest and autotest (which I believe used to be autospec for RSpec, but now got unified with autotest).

Unfortunately, a trivial gem install ZenTest didn’t do the trick, since all the app’s gems were managed by Bundler. This is what’s been happening:

$ autotest
loading autotest/rails_rspec2
Autotest style autotest/rails_rspec2 doesn't seem to exist. Aborting.

even though we did have the following in autotest/discover.rb:

Autotest.add_discovery { "rails" }
Autotest.add_discovery { "rspec2" }

We needed a way to run a command with the whole environment set up by Bundler.

The solution

That’s when we stumbled upon bundle exec. The docs say it all – "Run the command in context of the bundle". And in deed:

$ bundle exec autotest
/usr/lib/ruby/1.8/pathname.rb:263: warning: `*' interpreted as argument prefix
/usr/lib/ruby/gems/1.8/gems/bundler-0.9.26/lib/bundler.rb:72: warning: instance variable @setup not initialized
/usr/lib/ruby/gems/1.8/gems/bundler-0.9.26/lib/bundler/runtime.rb:145: warning: method redefined; discarding old path
loading autotest/rails_rspec2
(...)

............*..*..................................................*.........


Finished in 0.57859 seconds
76 examples, 0 failures, 3 pending

Hurray!

Additionally, installing bubble notifications was a breeze – we just added gem 'test_notifier' to our Gemfile. It’s a nifty little gem with multiplatform support and out-of-the-box awesomeness. You can read more about it here.

You’ll need to install appropriate libs and modify ~/.autotest, but it’s all written in the gems' README, so go ahead and check it out.

Here’s what test_notifier gives you:

Finally! No need to switch to the terminal after every save.

Conclusion

After a fair amount of effort spent setting up a comfortable and productive Rails 3 environment, I can say I’m almost there. There’s still room for improvement, but my confidence is slowly coming back, so one day I’ll get there.

Just for the record, here’s my environment:

# Gemfile
gem "rails", "3.0.0.beta4"

group :test do
  gem "rspec-rails", ">= 2.0.0.beta.1"
  gem 'ZenTest'
  gem 'test_notifier'
end

$ bundle show
  * ZenTest (4.3.3)
  * bundler (0.9.26)
  * rspec (2.0.0.beta.17)
  * rspec-core (2.0.0.beta.17)
  * rspec-expectations (2.0.0.beta.17)
  * rspec-mocks (2.0.0.beta.17)
  * rspec-rails (2.0.0.beta.17)
  * test_notifier (0.1.4)

Bonus tip

I fell in love with bundle open to inspect gems' sources. I’ve had so many revelations reading it! Give it a try and you won’t regret it.

Permalink | Leave a comment »

Tricky _flash partial in Rails

Sat, 10 Jul 2010 15:23:00 -0700

In one of my projects, we’re using two different layouts, and I wanted to share the flash rendering between them.

I achieved it with just a typical HAML snippet:

To my surprise the flash was always blank. Fortunately, I had an ‘aha!’ moment when I realized that inside a partial there is a local variable named after the partial (without the underscore). It turned out that flash was a local variable obscuring the #flash method call that I wanted, since I named the partial _flash.html.haml.

A quick

flash = flash()

at the top did the trick. Here’s the final version of the _flash partial:

Permalink | Leave a comment »

Watermarking images with Rails 3 and Paperclip

Mon, 05 Jul 2010 23:37:00 -0700

In one of the projects I’m working on, we needed a way to watermark images for public preview. We’re using Paperclip 2.3.3.

Searching gave only one result.

The included Paperclip processor had some issues, but a little debugging led to making it work properly. Of course, all the credits go to ng for sharing his code.

The model

Here’s how to set up the model to use the watermark processor:

Some comments:

:processors => [:watermark] enables the watermark processor
:watermark_path => "#{Rails.root}/public/images/watermark.png" declared the location of the watermark path
:position => 'Center' defines the position of the watermark on the image

I saved the found watermark processor in lib/paperclip_processors, but…

First obstacle

…this gave me uninitialized constant Paperclip::Watermark. That’s weird, since the Paperclip README says that all files in lib/paperclip_processors should be loaded automatically. I decided to take the short way and just

on top of app/models/photo.rb.

Server restart, and…

Second obstacle

now, when attempting to use the processor, development.log shows:

Notice the weird single-quoting on the lines starting with convert and composite. It turned out that the watermark processor built the command params as a string, and when passed to Paperclip.run, the whole string got quoted. The solution was to pass the params as consecutive arguments:

So, things started to look better after restarting the server. But then…

Third obstacle

As you can see, the single quotes are now as they should be. The issue is that the file names sometimes have this [0] and double-quotes at the end. This was caused by buggy #fromfile and #tofile. Here’s how I fixed it:

Now, after restarting the server, everything worked well.

With some more formatting fixes, here’s the final version of the watermark processor:

Mission complete

Now I need to find a way to serve some styles directly from public/, and to hide some of them behind a controller. Any ideas?

Permalink | Leave a comment »

Welcome to my blog

Thu, 24 Jun 2010 17:25:00 -0700

This blog is to allow me to take and share notes of my experiences on web development and related topics, experiments and findings that I will make during my day-to-day work as a web developer.

These topics will mostly include (but not limit to):

Ruby
- Rails
- Gems
- Heroku
- RSpec
PHP
- Trying to make it suck less (and doomed to fail)
- Zend Framework
- PHPUnit
Web
- HTTP/REST
- HTML5/CSS/JS (mostly jQuery)
- browsers
Workflow tools
- Git
- Trac
- Capistrano
- Cruisecontrol
Sysadmining
- Munin
- Apache
- MySQL
- /etc/*
Ubuntu/Linux/GNU
Netbeans/Vim/gedit
professional development
project/team management
…

This ad-hoc brainstorm list is definitely not carved in stone and will evolve together with my getting more comfortable with publishing.

Editor’s note

Yes, dear reader, this post does feel a bit stiff and official. I hope with time my writing style gets more natural and witty (keep your fingers crossed).

PS. Hey, Posterous, where’s my "Preview before publishing to make sure it looks decent and at least try to find those typos" button?

Permalink | Leave a comment »

Continuous development

Zend Framework app's PHPUnit suite 3x faster

The background

The problem

The solution

Caveats

Problem #1

Problem #2

Problem #3

Problem #4

Conclusion

Autotest-like PHPUnit runner for a Zend Framework application

Pseudo-autotest for PHPUnit

Bubble notifications with Ubuntu’s notify-osd

View specs with Rails 3, RSpec 2b20 and Webrat

undefined method ‘has_selector?’ for String

Upgrading a Rails 3 beta 4 application to Rails 3 RC

Bundler adventures

Changes in default skeleton files

Mission complete

On a side note

Speccing partials in Rails 3 and RSpec 2

By the way: view helpers not included in the spec’s view object

Rails 3 and ZenTest

The background

The problem

The solution

Conclusion

Bonus tip

Tricky _flash partial in Rails

Watermarking images with Rails 3 and Paperclip

The model

First obstacle

Second obstacle

Third obstacle

Mission complete

Welcome to my blog

Editor’s note

By the way: view helpers not included in the spec’s `view` object