I’m interested in having a way to easily create new Rails servers in the cloud (specifically AWS) and to deploy my apps to them. I’m familiar with the bits and pieces that are involved but I hadn’t stitched them together. I started to write a Rails app to manage my AWS assets, partially as a path to learning extjs, but when the AWS Console came out I ditched that and started back on the original problem. Here are the different bits that I planned to use, and how I became aware of them:

• Sprinkle, by Marcus Crafter (http://github.com/crafterm/sprinkle/tree/master) – Sprinkle handles deployment of resources on Ubuntu boxes in a pretty easy-to-understand way. Unfortunately I have a strong preference for Postgres over MySql (blame Simon Harris) and the default Sprinkle deployment recipes all use MySql. Still, it didn’t take me long to make some extensions to install Postgres instead of MySql.
• Passenger-Stack, by Ben Schwarz (http://github.com/benschwarz/passenger-stack/tree/master) – Sprinkles for Apache, Passenger, Mysql, Memcached & Git. I found out about this via a Ryan Allen tweet.
• Amazon Web Services (http://aws.amazon.com/) – well known, but the fairly recent addition of Elastic Block Storage (EBS) gives a way to easily create persistent data that outlives a given instance, say for a Postgres database.

The last piece to this puzzle is a web site called oDesk (http://www.odesk.com/). I was talking to a client last week and she mentioned that she thought Australian Rails developers were the most expensive in the world. When I asked her what she was doing she said that she was using off-shore Rails developers and was finding good people who would work for US$20/hr, from India but also from the USA. It seemed like the recession in the USA was having interesting side affects. I was definitely curious how using oDesk would work from a client perspective.

So I had this small problem that I didn’t have time to work on, and my curiousity about oDesk, and it seemed natural to solve them together. I posted my job on oDesk – produce a script that, given an EC2 instance and an ESB volume, would automatically configure the instance to use Rails with Passenger and Postgres, storing the data on the EBS volume. In addition, provide a test application so I can verify the Capistrano deployment to the instance. I posted this as a fixed price job, and my plan was to use a few different people and compare the outcomes. I posted my job last Sunday (it’s currently Thursday).

I had my first response in a few hours, from within Australia! Twenty four hours didn’t bring any more responses, so I pushed my job in front of a number of Rails developers who had high ratings. Many declined as they were too busy, some thought the suggested price was too low given the need to learn about AWS, and some still haven’t responded. I got two more positive responses, one from an experienced developer in St. Petersburg and one from a less experienced developer on the east coast of the USA. The bids were $333.33 (Australia), $250.00 (Russia), and $166.67 (USA). The expected development times were 1 day, 1 day and 1 week.

I accepted the responses and nominated a starting date of Tuesday. On Wednesday I got my solutions from Australia and St Petersburg. I’ve tested the Russian solution and it works fine. There seem to be couple of issues in the Australian solution but the developer
is being very responsive, and the USA solution isn’t due yet.

What’s more interesting is what else happened in the 24 hours of development. The Russian developer forked passenger-stack on GitHub and hosted the fork as a public repository on GitHub (which was fine by me). However, I found out about this through an email from Marcus Crafter, who knew I was looking for a Postgres solution and pointed the fork out to me! Then I saw these tweets from Ben Schwarz (@benschwarz) “@kouky Just so you know, someone forked passenger-stack overnight rolling postgres support in, I’ll try and pull everything together :)” and “@atnan @kouky, I’ve just blindly merged the PostgreSQL stuff as I’m up in the hills. I’ll check it all out tomorrow :)” The status message for passenger-stack on GitHub now says “Adding support for PostgreSQL. Sprinkle will prompt the user to select MySQL/PostgreSQL server and the relevant Ruby database drivers.”

So it looks like my little experiment ended up sponsoring the change that I wanted in the core implementation of passenger-stack. I suppose I could have asked Ben to do it originally, but that seemed like an imposition and I would have only achieved one of my goals. Now I have a script to configure my EC2 instances automatically, a better feeling for oDesk and off-shore development, and I’ve discovered that you can always sponsor the features that you want in a piece of open source software. Well worth my costs

In case you don’t want to do this yourself, here’s a script that will start Pages, prompt you for the type of export you want, and then prompt you for a destination directory, and it works under Leopard. I told you it was more thorough than my last solution

on open theFiles
	tell application "System Events"
		if process "Pages" exists then
			display dialog "whoops, please close Pages before running droplet!"
		end if
	end tell
	tell application "Pages"
		activate
		delay 1
		close front document
		set theList to {"doc", "rtf", "pdf", "txt"}
		set theType to (choose from list theList OK button name "Select" with title "Pages Export" with prompt "Choose one or more formats to export using Pages" with multiple selections allowed) as text item
		set s to theType as string
		set theLocation to choose folder
		set theLocation to theLocation as string
		
		repeat with aFile in theFiles
			open aFile
			
			if theType contains "doc" then
				set asType to "SLDocumentTypeMSWord"
				set docName to name of front document
				
				-- Remove .pages extension.
				set prevTIDs to AppleScript's text item delimiters
				set AppleScript's text item delimiters to ".pages"
				
				-- Add .doc extension.
				set docNameNew to first text item of docName & asType
				set AppleScript's text item delimiters to prevTIDs
				
				-- Save file to Desktop.
				set docPathAndName to theLocation & docNameNew
				save front document as asType in docPathAndName
				
			end if
			if theType contains "rtf" then
				set asType to "SLDocumentTypeRichText"
				
				set docName to name of front document
				
				-- Remove .pages extension.
				set prevTIDs to AppleScript's text item delimiters
				set AppleScript's text item delimiters to ".pages"
				
				-- Add .doc extension.
				set docNameNew to first text item of docName & asType
				set AppleScript's text item delimiters to prevTIDs
				
				-- Save file to Desktop.
				set docPathAndName to theLocation & docNameNew
				save front document as asType in docPathAndName
			end if
			if theType contains "pdf" then
				set asType to "SLDocumentTypePDF"
				set docName to name of front document
				
				-- Remove .pages extension.
				set prevTIDs to AppleScript's text item delimiters
				set AppleScript's text item delimiters to ".pages"
				
				-- Add .doc extension.
				set docNameNew to first text item of docName & asType
				set AppleScript's text item delimiters to prevTIDs
				
				-- Save file to Desktop.
				set docPathAndName to theLocation & docNameNew
				set s to save front document as asType in docPathAndName
				
			end if
			if theType contains "txt" then
				set asType to "SLDocumentTypePlainText"
				
				set docName to name of front document
				
				-- Remove .pages extension.
				set prevTIDs to AppleScript's text item delimiters
				set AppleScript's text item delimiters to ".pages"
				
				-- Add .doc extension.
				set docNameNew to first text item of docName & asType
				set AppleScript's text item delimiters to prevTIDs
				
				-- Save file to User Specified Location
				set docPathAndName to theLocation & docNameNew
				save front document as asType in docPathAndName
			end if
			try
				close front document saving no
			end try
		end repeat
		quit
	end tell
end open

“In 1957 some of us put forward a partially complete theory of the weak force, in disagreement with the results of seven experiments. It was beautiful and we dared to publish it, believing that all those experiments must be wrong.

In fact, they were all wrong.”

I love that in maths and physics truth and beauty are so frequently aligned, and that misalignment often indicates immature understanding. A new insight can reveal new beauty.

I think the same is true of programming. Your code should be at least as elegant as your problem domain (I can’t do anything about accounting *s*).

I know nothing about AppleScript, but I was able to find a script on the web (thanks again, Google) that did most of what I want. I’ll show you the script, then tell you the parts that weren’t obvious to me and slowed me down

on open theFiles
	tell application "Pages"
		repeat with aFile in theFiles
			open aFile
			set docName to name of front document
			-- Remove .pages extension.
			set prevTIDs to AppleScript's text item delimiters
			set AppleScript's text item delimiters to ".pages"
			-- Add .pdf extension.
			set docName to first text item of docName & ".rtf"
			set AppleScript's text item delimiters to prevTIDs
			-- Save file to Desktop.
			set docPathAndName to (path to desktop as string) & "rtfs:" & docName
			save front document as ".rtf" in docPathAndName
			close front document
		end repeat
	end tell
end open

My first challenge was that I couldn’t figure out how to get this to run! Pressing “Run” in ScriptEditor did nothing. I ended up saving it as an applet (use “Save As”, selecting FileFormat => application), and then I could drag and drop the files I wanted to convert onto the applet. Once I put the applet in the dock, that was pretty convenient.

The second thing that slowed me down was figuring out how to save the results to a folder (the original script saved to the desktop). What I needed was the bit ‘ & “rtfs:”‘ while setting the docPathAndName. Here I’m dealing with a string, and it already has a trailing “:” – I kept trying to add another one, and AppleScript gave me a message about the target not being writable, rather than not existing, which lead me astray for a while.

Anyway, now I can select my Pages files, drag and drop them onto the applet in my dock, and the rtf files appear in a folder on my desktop. Sweet!

The first thing that happened was that we got timeout errors that looked like this (courtesy of ExceptionNotifier):

A Mongrel::TimeoutError occurred in reports#cost_detail_pdf:
Mongrel timed out this thread: shutdown
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:221:in `within'

The support folks at EngineYard suggested that we shouldn’t be running PDF generation from within our controller, and that we should use something like background job, but since we wanted to report status as the job proceeded and we only had a small number of jobs, we went with BackgroundRb instead. (If you care about the difference, look for ‘Backgroundrb and Bj serve different purposes’ in this thread).

BackgroundRb works great once you get it going, but there were examples in the documentation that simply didn’t work for me. Here’s what worked for me in the end:

In my controller:

  MiddleMan.new_worker(:worker => :cost_detail_pdf_worker, :job_key => current_user.id)
  MiddleMan.ask_work(:worker => :cost_detail_pdf_worker, :job_key => current_user.id, 
                                          :worker_method => :build_report, 
                                          :data => [session[:production_id], file_name])

and my worker:

class CostDetailPdfWorker calculated_report, :template => :default)
register_status(“Finished Rendering”)
File.open(file_path, “w”) do |file|
file << pdf end register_status("File available") end end [/sourcecode]

The next part of the adventure was getting this running on our Slicehost slice. Unfortunately, BackgroundRb requires Ruby 1.8.5, and we only had 1.8.4 (on Ubuntu 6.06 LTS). Apt-get for Ubuntu 6.06 doesn’t include Ruby 1.8.5, so I needed to install Ruby 1.8.5 from source using these excellent instructions.

Unfortunately, I also needed to reinstall all our gems to go with the new version of Ruby. This was mostly stragithforward, except for Postgres. The postgres gem wouldn’t build the native extensions, failing to find pg_config. These instructions were close, but before that would work I also needed to do “apt-get install libpq-dev” (you might also need libpgsql-ruby and/or libpgsql-ruby11.8, but I already had those installed).

Ok, looks pretty good now. I started the backgroundrb server manually as a test, and could produce my pdf files, so the next step was to move to our production environment at Engineyard. This was the first time I tried to stop and start the BackgroundRb using Capistrano – unfortunately you can’t simply repeat the command lines in your Capistrano tasks. Fortunately, there’s also a description of some tasks that do work (make sure you use the start task in the comments at the end of post).

All in all, a nice outcome but with much more pain than I would have liked.

What I needed was to be able to associate a piece of code that created or modified some domain objects with the code that cleaned up the changes to the domain model on rollback (assuming that the database transaction handled the persistence parts). I could have used blocks, but I find that unlabelled blocks aren’t very expressive so I decided to use methods (and their names) instead of blocks. Here’s an example of using my method, #within_transaction_with_rollback; in the example :create_reference_objects and :clear_reference_objects are methods defined somewhere else in the class.

  def process
    within_transaction_with_rollback(:create_reference_objects, :clear_reference_objects)
  end

And here’s the implementation of my method, for those who are interested. I’m sure that there are plenty of other possible implementations

module Transactions

    def within_transaction_with_rollback(transaction_method, rollback_method)
      begin
        within_transaction(transaction_method)
      rescue
        self.send(rollback_method)
      end
    end

    def within_transaction(create_method)
      self.class.transaction do
        self.send(create_method)
      end
    end
  
end

1. Active Record ignores finder_sql when you do an eager load of an association (though I can’t find a definitive reference for this, it is the observed behaviour);
2. Active Scaffold uses eager loading by default

I didn’t have any specs that explicitly tested that my model could eagerly load its associations, but everything else seemed to be working, so I was surprised that Active Scaffold was unhappy. The solution was to tell Active Scaffold not to use eager loading on that particular association, following the instructions in the FAQ under “my database is choking” (which kind of describes what was happening to me, if the choking was fatal).

I’m fairly laissez faire about the issue. I’m quite happy to write tests for Rails models that actually invoke ActiveRecord and interact with the database. I’ll also happily set up a network of associated objects in the database rather than mock out everything except the class under test, though I do have some ill-defined limit that makes me uncomfortable.

I’m more strict about controller tests – I want the controller itself to be wafer thin, and I’ll generally only test that the controller sets the right instance variables (along with flash notices etc). The tests end up being fairly small, don’t touch the view at all, and I think even the evangelical TDD folk would call them unit tests.

However, Craig rightly points out a couple of problems with this. I’m typically working with server side functionality (if I had to say what I was best out, I’d initially claim domain modelling), so I write crap view code and I rarely write tests for it (someone’s just going to come along and rewrite it to be presentable anyway). It’s a fairly pathetic defence of my laziness, but anecdotally it seems lots of people don’t write view specs. So there’s a whole area of my application that’s not tested very well.

Craig also points out that one of the biggest sources of defects for him is that his controller doesn’t set up what the view expects (or you can express if from the opposite perspective if you prefer) – the failure occurs during the interaction between the controller and the view. These sorts of failures are quite difficult to find with unit tests alone, and probably won’t get picked up by distinct controller specs and view specs.

We could write separate integration specs, but RSpec gives a simple way to catch at least the most egregious of these problems. Unit testing purists might object, but the laissez faire’sts won’t mind. Put “integrate_views” into your descriptions. When integrate_views is specified, RSpec renders the real view rather than mocking out the rendering, and if objects are missing or badly misconfigured you’ll get a rendering exception.

I remembered the integrate_views option from earlier versions of RSpec – it’s not particularly conspicuous in the current version’s documentation, but it can be very useful.