Steve

I’m interested in having a way to easily create new Rails servers in the cloud (specifically AWS) and to deploy my apps to them. I’m familiar with the bits and pieces that are involved but I hadn’t stitched them together. I started to write a Rails app to manage my AWS assets, partially as a path to learning extjs, but when the AWS Console came out I ditched that and started back on the original problem. Here are the different bits that I planned to use, and how I became aware of them:

• Sprinkle, by Marcus Crafter (http://github.com/crafterm/sprinkle/tree/master) – Sprinkle handles deployment of resources on Ubuntu boxes in a pretty easy-to-understand way. Unfortunately I have a strong preference for Postgres over MySql (blame Simon Harris) and the default Sprinkle deployment recipes all use MySql. Still, it didn’t take me long to make some extensions to install Postgres instead of MySql.
• Passenger-Stack, by Ben Schwarz (http://github.com/benschwarz/passenger-stack/tree/master) – Sprinkles for Apache, Passenger, Mysql, Memcached & Git. I found out about this via a Ryan Allen tweet.
• Amazon Web Services (http://aws.amazon.com/) – well known, but the fairly recent addition of Elastic Block Storage (EBS) gives a way to easily create persistent data that outlives a given instance, say for a Postgres database.

The last piece to this puzzle is a web site called oDesk (http://www.odesk.com/). I was talking to a client last week and she mentioned that she thought Australian Rails developers were the most expensive in the world. When I asked her what she was doing she said that she was using off-shore Rails developers and was finding good people who would work for US$20/hr, from India but also from the USA. It seemed like the recession in the USA was having interesting side affects. I was definitely curious how using oDesk would work from a client perspective.

So I had this small problem that I didn’t have time to work on, and my curiousity about oDesk, and it seemed natural to solve them together. I posted my job on oDesk – produce a script that, given an EC2 instance and an ESB volume, would automatically configure the instance to use Rails with Passenger and Postgres, storing the data on the EBS volume. In addition, provide a test application so I can verify the Capistrano deployment to the instance. I posted this as a fixed price job, and my plan was to use a few different people and compare the outcomes. I posted my job last Sunday (it’s currently Thursday).

I had my first response in a few hours, from within Australia! Twenty four hours didn’t bring any more responses, so I pushed my job in front of a number of Rails developers who had high ratings. Many declined as they were too busy, some thought the suggested price was too low given the need to learn about AWS, and some still haven’t responded. I got two more positive responses, one from an experienced developer in St. Petersburg and one from a less experienced developer on the east coast of the USA. The bids were $333.33 (Australia), $250.00 (Russia), and $166.67 (USA). The expected development times were 1 day, 1 day and 1 week.

I accepted the responses and nominated a starting date of Tuesday. On Wednesday I got my solutions from Australia and St Petersburg. I’ve tested the Russian solution and it works fine. There seem to be couple of issues in the Australian solution but the developer
is being very responsive, and the USA solution isn’t due yet.

What’s more interesting is what else happened in the 24 hours of development. The Russian developer forked passenger-stack on GitHub and hosted the fork as a public repository on GitHub (which was fine by me). However, I found out about this through an email from Marcus Crafter, who knew I was looking for a Postgres solution and pointed the fork out to me! Then I saw these tweets from Ben Schwarz (@benschwarz) “@kouky Just so you know, someone forked passenger-stack overnight rolling postgres support in, I’ll try and pull everything together :)” and “@atnan @kouky, I’ve just blindly merged the PostgreSQL stuff as I’m up in the hills. I’ll check it all out tomorrow :)” The status message for passenger-stack on GitHub now says “Adding support for PostgreSQL. Sprinkle will prompt the user to select MySQL/PostgreSQL server and the relevant Ruby database drivers.”

So it looks like my little experiment ended up sponsoring the change that I wanted in the core implementation of passenger-stack. I suppose I could have asked Ben to do it originally, but that seemed like an imposition and I would have only achieved one of my goals. Now I have a script to configure my EC2 instances automatically, a better feeling for oDesk and off-shore development, and I’ve discovered that you can always sponsor the features that you want in a piece of open source software. Well worth my costs

McDonalds : “Please form a second line here”

Steve : “Why?”

McDonalds : “What?”

Steve : “Why should we form two queues, when one queue is more efficient?”

McDonalds : after 3 or 4 second pause, “Because I say to”

Can’t argue with logic like that….

In case you don’t want to do this yourself, here’s a script that will start Pages, prompt you for the type of export you want, and then prompt you for a destination directory, and it works under Leopard. I told you it was more thorough than my last solution

on open theFiles
	tell application "System Events"
		if process "Pages" exists then
			display dialog "whoops, please close Pages before running droplet!"
		end if
	end tell
	tell application "Pages"
		activate
		delay 1
		close front document
		set theList to {"doc", "rtf", "pdf", "txt"}
		set theType to (choose from list theList OK button name "Select" with title "Pages Export" with prompt "Choose one or more formats to export using Pages" with multiple selections allowed) as text item
		set s to theType as string
		set theLocation to choose folder
		set theLocation to theLocation as string
		
		repeat with aFile in theFiles
			open aFile
			
			if theType contains "doc" then
				set asType to "SLDocumentTypeMSWord"
				set docName to name of front document
				
				-- Remove .pages extension.
				set prevTIDs to AppleScript's text item delimiters
				set AppleScript's text item delimiters to ".pages"
				
				-- Add .doc extension.
				set docNameNew to first text item of docName & asType
				set AppleScript's text item delimiters to prevTIDs
				
				-- Save file to Desktop.
				set docPathAndName to theLocation & docNameNew
				save front document as asType in docPathAndName
				
			end if
			if theType contains "rtf" then
				set asType to "SLDocumentTypeRichText"
				
				set docName to name of front document
				
				-- Remove .pages extension.
				set prevTIDs to AppleScript's text item delimiters
				set AppleScript's text item delimiters to ".pages"
				
				-- Add .doc extension.
				set docNameNew to first text item of docName & asType
				set AppleScript's text item delimiters to prevTIDs
				
				-- Save file to Desktop.
				set docPathAndName to theLocation & docNameNew
				save front document as asType in docPathAndName
			end if
			if theType contains "pdf" then
				set asType to "SLDocumentTypePDF"
				set docName to name of front document
				
				-- Remove .pages extension.
				set prevTIDs to AppleScript's text item delimiters
				set AppleScript's text item delimiters to ".pages"
				
				-- Add .doc extension.
				set docNameNew to first text item of docName & asType
				set AppleScript's text item delimiters to prevTIDs
				
				-- Save file to Desktop.
				set docPathAndName to theLocation & docNameNew
				set s to save front document as asType in docPathAndName
				
			end if
			if theType contains "txt" then
				set asType to "SLDocumentTypePlainText"
				
				set docName to name of front document
				
				-- Remove .pages extension.
				set prevTIDs to AppleScript's text item delimiters
				set AppleScript's text item delimiters to ".pages"
				
				-- Add .doc extension.
				set docNameNew to first text item of docName & asType
				set AppleScript's text item delimiters to prevTIDs
				
				-- Save file to User Specified Location
				set docPathAndName to theLocation & docNameNew
				save front document as asType in docPathAndName
			end if
			try
				close front document saving no
			end try
		end repeat
		quit
	end tell
end open

“In 1957 some of us put forward a partially complete theory of the weak force, in disagreement with the results of seven experiments. It was beautiful and we dared to publish it, believing that all those experiments must be wrong.

In fact, they were all wrong.”

I love that in maths and physics truth and beauty are so frequently aligned, and that misalignment often indicates immature understanding. A new insight can reveal new beauty.

I think the same is true of programming. Your code should be at least as elegant as your problem domain (I can’t do anything about accounting *s*).

I know nothing about AppleScript, but I was able to find a script on the web (thanks again, Google) that did most of what I want. I’ll show you the script, then tell you the parts that weren’t obvious to me and slowed me down

on open theFiles
	tell application "Pages"
		repeat with aFile in theFiles
			open aFile
			set docName to name of front document
			-- Remove .pages extension.
			set prevTIDs to AppleScript's text item delimiters
			set AppleScript's text item delimiters to ".pages"
			-- Add .pdf extension.
			set docName to first text item of docName & ".rtf"
			set AppleScript's text item delimiters to prevTIDs
			-- Save file to Desktop.
			set docPathAndName to (path to desktop as string) & "rtfs:" & docName
			save front document as ".rtf" in docPathAndName
			close front document
		end repeat
	end tell
end open

My first challenge was that I couldn’t figure out how to get this to run! Pressing “Run” in ScriptEditor did nothing. I ended up saving it as an applet (use “Save As”, selecting FileFormat => application), and then I could drag and drop the files I wanted to convert onto the applet. Once I put the applet in the dock, that was pretty convenient.

The second thing that slowed me down was figuring out how to save the results to a folder (the original script saved to the desktop). What I needed was the bit ‘ & “rtfs:”‘ while setting the docPathAndName. Here I’m dealing with a string, and it already has a trailing “:” – I kept trying to add another one, and AppleScript gave me a message about the target not being writable, rather than not existing, which lead me astray for a while.

Anyway, now I can select my Pages files, drag and drop them onto the applet in my dock, and the rtf files appear in a folder on my desktop. Sweet!

Cogent is building a software application to help you organize your personal work, and I’m acting in the sponsor role. The details of the functionality are being sorted out by someone else, but I’m the person focussed on the operational requirements and how we get it to the billable stage as quickly as possible (everyone else in interested in this too, but it’s my particular role). Yesterday afternoon I added operational stories to our list, and then prioritised the list story by story, and got what looked like a reasonable response.

Today I went back and created some themed releases, and gave them names.

1. Homely – Ugly, but interesting. This release is usable, but won’t win any design awards. We want it first because we’re using the application internally now, so functionality is important. This is the release we’ll pass to “friendly” testers for feedback, so it needs to support a small set of concurrent users without getting too slow. Codename : Homer
2. Prettified – Like Homer, but certainly nicer looking and generally more usable. We still won’t be confident in our backup plan, nor have stress tested the application, but it will look professional and be generally usable. Codename : Marge
3. Robust – we need this one to be tough, so it can withstand the abuse of a lot of people. This is the point where we can do an open beta – still not charging, but giving the general public access. Codename : Bart
4. Billable – something that we feel we could reasonably charge money for. Codename : Lisa

With these releases in mind I went back and looked at my priorities again, and found that some of them were quite different. If you are a customer on an agile project, I strongly encourage you to do the same thing – you may be surprised by the difference it makes.

The first thing that happened was that we got timeout errors that looked like this (courtesy of ExceptionNotifier):

A Mongrel::TimeoutError occurred in reports#cost_detail_pdf:
Mongrel timed out this thread: shutdown
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:221:in `within'

The support folks at EngineYard suggested that we shouldn’t be running PDF generation from within our controller, and that we should use something like background job, but since we wanted to report status as the job proceeded and we only had a small number of jobs, we went with BackgroundRb instead. (If you care about the difference, look for ‘Backgroundrb and Bj serve different purposes’ in this thread).

BackgroundRb works great once you get it going, but there were examples in the documentation that simply didn’t work for me. Here’s what worked for me in the end:

In my controller:

  MiddleMan.new_worker(:worker => :cost_detail_pdf_worker, :job_key => current_user.id)
  MiddleMan.ask_work(:worker => :cost_detail_pdf_worker, :job_key => current_user.id, 
                                          :worker_method => :build_report, 
                                          :data => [session[:production_id], file_name])

and my worker:

class CostDetailPdfWorker calculated_report, :template => :default)
register_status(“Finished Rendering”)
File.open(file_path, “w”) do |file|
file << pdf end register_status("File available") end end [/sourcecode]

The next part of the adventure was getting this running on our Slicehost slice. Unfortunately, BackgroundRb requires Ruby 1.8.5, and we only had 1.8.4 (on Ubuntu 6.06 LTS). Apt-get for Ubuntu 6.06 doesn’t include Ruby 1.8.5, so I needed to install Ruby 1.8.5 from source using these excellent instructions.

Unfortunately, I also needed to reinstall all our gems to go with the new version of Ruby. This was mostly stragithforward, except for Postgres. The postgres gem wouldn’t build the native extensions, failing to find pg_config. These instructions were close, but before that would work I also needed to do “apt-get install libpq-dev” (you might also need libpgsql-ruby and/or libpgsql-ruby11.8, but I already had those installed).

Ok, looks pretty good now. I started the backgroundrb server manually as a test, and could produce my pdf files, so the next step was to move to our production environment at Engineyard. This was the first time I tried to stop and start the BackgroundRb using Capistrano – unfortunately you can’t simply repeat the command lines in your Capistrano tasks. Fortunately, there’s also a description of some tasks that do work (make sure you use the start task in the comments at the end of post).

All in all, a nice outcome but with much more pain than I would have liked.