On the new Matthew Herbert Big Band album there’s an achingly beautiful song One Life. I thought it was annoyingly and capriciously glitched. The glitch repeats rhythmically over the track, seemingly detached and irrelevant, just occluding the music. Its not unusual in Herbert’s music, but usually his mistakes enhance tracks.

At the magical Matthew Herbert Big Band gig at the Sydney Opera House last night One Life was the only track where Herbert stood back from his equipment, hands clasped, just listening. I immediately thought that he’d regretted the glitch on the album and was tacitly repenting on stage.

This morning I googled the track; I was wrong. From the liner notes:

one beep from the alarm system of my premature son’s neonatal special care unit. each beep represents 100 people killed in iraq since the start of the war in 2003 to oct 2006.

Rated PoC (you choose the meaning) & alpha.

backstory

I keep running into the same pattern of wanting to keep my app’s data in a set of files in a directory,
rather than in a database.
Its always liberating for a while, but inevitably reaches a point where its more unwieldy than using a
plain ol’ database.

The main benefit seems to be that the data’s data and there’s no reason it shouldn’t be kept in a file
instead of a database anyway.
The unwieldiness and performance angst seems to come not from the meat, but from the metadata: when you don’t need the whole thing
but a small detail or excerpt.

In other words, my solution ends up getting more and more convoluted the further away from the meat of the data I get.

For example, I was writing an app which was used to review Capistrano runs and parse their log files. Why store that stuff in a database? By the time I started considering a rewrite I needed to:

• paginate a list of lot of log files.
• pull out just an excerpt of the log file (say, a comment describing the reason for running the script).
• determine the success of the run.
• determine the time of the run
• etc.

It was boring to code & slow.

This library is an experiment in easing that kind of disparity & angst.

example

The most recent path-crossing I had with the pattern was when I started dreaming of simplifying my blog, a la
http://github.com/toolmantim/toolmantim.

The simplicity of “just having files” is great, but I know I don’t really want to go there in case I regret it, as I have in the past. Instead of giving up, this time I looked deep into my soul to work out what was eating me.

I think this library will probably get me out of the existential bind; an article might look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

require 'rubygems'
require 'dirdb'
require 'pp'

class Article
  # dirdb setting-uppness
  include DirDB::Resource

  index_class DirDB::Index::Basic    # optional, default shown
  glob '*.haml'                      # optional, default '*'

  # optional indexes:
  index :by_created_at do |article|
    File.ctime(article.path)
  end
  index :by_reverse_title_length do |article|
    -article.title.length
  end


  # mandatory scanner; this is where you grab all the file's information
  def self.scan_file(path)
    File.basename(path,'haml').sub(/^\d+_/,'').gsub('_',' ')
  end



  # normal modelness
  attr_reader :title

  def initialize(title)
    @title = title || ''
  end
end

if __FILE__ == $0
  Article.path = '/path/to/your/blog'
  
  @articles = Article.all(:by_created_at)
  
  @how_to_use_files = Article.get('how_to_use_files.haml')
end

scan_file is really the core of it; you could make it much more complex if the fancy took you:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

def self.scan_file(path)
  title = nil
  IO.foreach(path) do |line|
    if line[/^\s*-#\s*title:\s*(.*)$/]
      title = $1
      break
    end
  end

  title || ''
end

The thing to realise is that whatever scan_file returns, initialize gets and must handle.

OK, so it doesn’t really do much, but it does enough and will do more. Maybe I should give it up and just use Sphinx?

indexes

Indexes are only used for sorting at this stage. The block passed to the class method index has the
selfsame semantics as Enumerable#sort_by.

There are two indexes at the moment: Basic and YAML.

• Basic scans & builds indexes into memory, once on first use.
• YAML Is a file-cached version of the Basic index. On first use, it tries to read from YAML, then if necessary scans, builds indexes and writes out the YAML cache.

caveats

• Read-only. Read-write is strongly YAGNI for me, but naively you could get around it by rebuilding the index on write.
• No specs or nuffin yet. Its just a PoC. The included features don’t really match up with the implementation anymore.

TODO

• Sqlite index storage.
• Using git metadata for created and updated at times. This is more of a model implementation thing, but I’d like to include it in the toolkit by default.
• Efficient will_paginate integration.
• rake tasks for offline rebuilding of indexes (YAGNI for now).
• Delta indexing (YAGNI for now).

code

See DirDB on github

and

Please let me know what you think… is there something in this, or is it just a slightly lumpy catharsis?

For some reason an emasculated twitter-debate flared yesterday, on whether code is art.

Of course, the question is intractable. Art is always elusive, otherwise there wouldn’t be any point to creating it in the first place.

However I’ve been thinking on this subject a bit lately. Toby has an insatiable hunger for drawings of animals, to be wrought by whomever is available/closest. This hunger is so strong that he now associates Daddy’s Computer with the drawing of Elephants:

Of course I now have it on good authority that my credentials on the subject of art and code are inadequate. But I’ve still been thinking about it enough to raise a slew of illustrative questions.

Is art available to everyone?

Is a child capable of detecting art? At what age do people become sensible of art? Can all people appreciate art?

If No, then who decides? Is there a gold standard, at least a committee?

If code could be art, who would decide if it actually was?

What is art?

For example, is anything within the realm of industrial design art?

Is a beautifully designed car art? I had this discussion with Dad the other day, and you’d be hard pressed to persuade him that car design is anything lower than the height of human sophistication and art.

Obviously this is completely subjective, even inside art’s own walls.

Is literature art? Its not much to look at, but it can certainly manifest as art inside your head. Is a play art? Is its script art, or only a production thereof? And is an amdram production less arty than a big name one?

Maybe code’s concepts or manifestation could be considered to be art. Or maybe code is only art if you do it like _why.

Science

Programming is part of computer science, but I’m fairly sure we don’t usually treat it as such (though maybe unit tests are moving us back towards the scientific method). There’s abundant beauty in science but it’s not art.

For example, is Flickr more the result of computer science, or the imagination of a small group of people?

Apples, oranges & elephants

Ultimately I think that comparing code and art is an impedance mismatch, but that perhaps the manifestation of code as an application could be considered as art.

And, even if code is art, I think that we’re all still at the stage of kiddies who daub their masterworks with arbitrary parts of their body. Kiddies who appreciate most pictures as art, as long as they have elephants.

RailsCamp 3 just finished, and it was another fantastic weekend of hacking with mates.

what a kerfuffle!

You’ve probably heard by now that Ben (the organiser of RailsCamp) and Kaz’s munchkin Liam came six weeks early. We’d been talking about babies coming early just the night before,
since my son Toby was five weeks early.
It was an eerie shock to hear that nearly the same thing had happened to Ben & Kaz! Liam is doing well, but is still in hospital for a little while.

The upshot of this twist was that Glenn Davy and I ended up running the camp. It was a pleasure as most of the heavy lifting had already been done by Ben.
Thanks so much to Glenn and Ben for making the camp awesome, and to everyone else for being so easy-going over the weekend.

codings

Lack of internet is a key feature of the RailsCamp fun.
To offset this, Lincoln Stoll set up a server chock full of VPS’s for hosting our apps. On top of that it served DNS and hosted gitorious.
At camp it was at http://git.railscamp.net, and now we’re back its still at http://git.railscamp.net. (Some of these apps will no doubt go across to github in time).

A bumper crop of code came home from the camp this year:

• Gitjour refactor

http://git.railscamp.net/projects/gitjour

A group of guys made their dissatisfaction with the current gitjour implementation manifest by refactoring it, waay into early saturday morning.

• Gitnotify

Broadcast a growl notification each time you commit to git.

• Gitman

http://git.railscamp.net/projects/gitman

Manage your git sharing, simply. By Mike Bailey

• Swords

http://git.railscamp.net/projects/swords

A great start on a thesaurus based crossword generator. It has ascii and web interfaces.

• Duke

http://git.railscamp.net/projects/duke

The return of the network jukebox. Gaming the duke voting system provided plenty of fun for all.

Myles Byrne and I were duke’s latest developers (everyone works on duke, some time). I failed to provide events for Myles, and I’m sorry.

• bit-tags

http://git.railscamp.net/projects/bit-tags

Taggable snippets by the Brisbane boys.

• iShare

http://git.railscamp.net/projects/ishare

An app for letting know what you’d like to hear or talk about at railscamp!

• twonklist

http://git.railscamp.net/projects/twonklist

An anti-social networking application.

• Twetter

http://git.railscamp.net/projects/twetter

A local twitter. Linc pointed twitter.com at twetter so that everyone’s Twitterific just worked. Matta writes about it here

Max dropped in and wrote up a Flex based client, too.

• starjour

http://github.com/lachie/starjour/tree/master

I did some work on refactoring starjour. Still not really sure where this app is going, but
it was good to see is list full of *jour services.

If I’ve forgotten any projects, let me know.

jq

I gave a talk on jQuery, which was a bit of a shamozzle as my version of firebug wasn’t playing with firefox 3. Preparation? Next time I suppose.
Thanks to Hugh Evans for having a working ffx and some elegant jq code to show off.

feedback

If you have some feedback on the camp, please let me know: lachie (at) this domain.

One thing I’d like to see is some more emphasis on family at RailsCamp. Brodaigh brought her daughter Crystal to this one and Lone and I brought Toby to the first one.
It adds a mellowing element to the camps that I really enjoy. And, I want a chance to beat Crystal at Set.

saving the day

When people started leaving on Monday morning, we hit a problem: someone had inadvertently taken Phil Oye’s bag.

After we’d finished tidying up and had hit McDonalds, I got a Twitter direct message on my phone from Dr Nic “what’s your ph num? we have someone’s black bag @ airport.”

I was able to put Phil in contact with Greg Fairbrother and it all ended happily ever after.

Twitter FTW

Summer 08 ? Winter 09 ?

So who’s going to step up next? Railscamp happens where ever someone organises it. It may be a structure-less weekend, but its still non-trivial to set one up
If you think you’d like to give it a go, let me know and I’ll put you in touch with people who’ve done it before. lachie (at) this domain.

The guys from Brisbane are keen to put on a camp, probably in Winter 2009.

Who wants to organise an event for Summer 2008/9 ?

A couple of years ago, I designed and hand-printed three t-shirts.

It was a lot of effort, mainly due to having to hand-cut the stencils out of overhead projector slides. The quality was never quite up to snuff. And the paint was smelly and extremely messy. Despite all that, it was a blast.

So now that I have a bit of time off, I find myself wanting to do some designs again… This time I’m eschewing the heady chemical aroma of fabric paint and using Redbubble, a long tail publishing business based in Melbourne. They’ll print cards, prints and shirts from your artwork.

They charge a reasonable base price, over which you can mark up whatever profit you desire. Selling costs nothing, so at the very least you can get your own designs printed for only $21. Niice.

On top of that, the site is really nicely designed and runs on Rails. Our #roro friends Xavier Shay and Justin French work there.

so anyway:

buy my tshirt!

Here

If you don’t get it, have a clue: http://twitter.com

We have such overwhelming abundance as a society; that we continue to squander it brings me close to despair. Acting only as a diluted 2020 herd is abdicating our responsibility.

We’re the top of the pile; the assumption of personal responsibility stands as the only mechanism left to us to achieve the things we need to.

The education system will never tailor itself to your kids. You will need to shoulder the burden of education yourself.

The bulk of society will likely never grasp technology as fully as we do. Its up to us to promulgate and fulfill its promise.

We’re not helpless motes in the plasma of society. We’re the white blood cells.

So there’s only one at the moment, but i really wanted to write this down:

0.5.x make the development logger log DEBUG messages…

Put this in Merb.root/config/environments/development.yml

1

:log_level: DEBUG

0.9.x freezing gems

The way of installing the latest merb release is

1
2

$ cd to/your/app
$ gem install merb -i gems --source=http://merbivore.com

Unfortunately, this command won’t install merb’s dependencies as it won’t be able to look inside the normal gems repository on rubyforge.

This should do the trick though

1
2

$ cd to/your/app
$ gem install merb -i gems --source=http://merbivore.com --source=http://gems.rubyforge.org

Yay!

(Note that this is only a problem while the merb gems aren’t pushed to rubyforge).

point one

Quicksilver

point two

I want to act without doing in web apps.

point three

qs_score.js

For example

I don’t know if its because I’m nosy, or because I’m a control freak, or perhaps both, but I really do like to read the source code. It occurs to me that this is probably why I feel so at home in the open source world.

As much as I like something like Cocoa, it pains me not to be able to trace problems all the way down the stack. Even java solved this problem by shipping with its own source.

Proclivities aside, I really like source, and am always looking for ways to make it easier to get at. cd /usr/lib/ruby/gems/1.8/gems gets old quick.

I started with this little cough gem from Dr Nic. Integrating it with my shell, I could now type

1

$ gems activerecord

and up pops the activerecord gem in Textmate. So some new releases of activerecord came out, and suddenly my naive script is giving me the old version. Without looking too far into it I added gem requirement support so that I could type

1

$ gems activerecord '>=2'

and get a recent activerecord in Textmate. As good as this is, I was really after something with less typing and smarter pants. Well tonight I cracked it and I thought I’d open the source:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#!/usr/bin/env ruby

require ENV['TM_SUPPORT_PATH']+"/lib/ui"
require 'rubygems'

gems = Gem.source_index.latest_specs.collect {|spec| spec.full_name}.sort

TextMate::UI.request_item(:title => 'open a gem', :items => gems) do |selected_gem|
  gem_path = Gem.source_index.specification(selected_gem).full_gem_path
  %x{open -a TextMate #{gem_path}}
end

Honestly, that’s it! Bung that in a command using Textmate’s Bundle Editor and assign it a key (I gave it cmd-shift-G, as I don’t regularly search backwards).

Double click on one and you’re golden… neat.

At Jelly Sydney last Friday, I went to see if my merb gems were up to date. You know:

1
2
3
4
5
6

$ gem list --remote

*** REMOTE GEMS ***

Updating metadata for 114 gems from http://gems.rubyforge.org
...

Each dot was taking aages, and there were to be one hundred and fourteen of the blighters!

In a nutshell

The rubygem server seemed to be having issues serving up incremental metadata. Until the rubygems server got back on its feet, I set my bulk threshold to a lower value:

1

echo :bulk_threshold: 1 >> ~/.gemrc

I won’t leave it there, because it could impact the overall rubygem system’s performance (well maybe if everyone decided to do this all the time).

Detailure

There were about seven peoples on the NerfPalace Wifi by this stage, so it was only natural to ask:

Lachie: “Hey Tim, is the network slow?”

Tim: “Nah aah, girlfriend. You slow. Or yo gems are slow, biatch.”

Well! It turns out that Tim had noticed the same thing and had started digging into the very source code of gems in search of an answer.

Teh Pwobwem

In order not to overtax the rubygem servers’ bandwidth, gem first finds out how many of its cached metadata are out of date. If the number is greater than an arbitrary threshold, it performs a bulk update in which all the metadata is updated at once.

If the number is below the threshold, each metadatum is fetched and updated individually (incrementally you might say).

By default, this number is 1000. Say each update was taking a second and there were 999 gems to update, this process would take more than a quarter of an hour. Presumably the average case is much better than this, but on Friday, each increment of the update was taking forever.

The complete source_cache is currently ~850k gzipped. Although in aggregate this could cost the rubygems people a fair bit of bandwidth, it would never take 15 minutes on today’s broadband.

What we needed was some way of forcing the bulk update, which at the time was much faster than the alternative. The mechanism for this is not very clear, so we got by by hacking the code.

HOWTO

Thankfully, there is a better way or changing the threhold! Although its not well documented or, well, documented at all, you can set the bulk_threshold variable in the .gemrc file kept in your home directory. Its YAML, y’know:

1

:bulk_threshold: 1

See that colon out the front of the bulk_threshold key? Its because its a symbol once it gets inside the ruby.

As I mentioned above, its probably not good manners to leave your threshold at 1, since the authors have built the system in a certain way (even if its not always optimal) and we’re not paying the rubygem bandwidth fees. Just think, it’d be terrible for The Authors to chuck a Zed and shut rubygems down!

Hooray for Rubygems!

Last RORO Sydney meetup I did a talk on DRYing up your views.

I was very pleased with how it went – this really does get easier with practice! The first couple of slides were dropped, but you didn’t miss much of note :)

We’re really starting to refine the workflow for recording talks and I’m planning a write up of our experiences so far.

Big thanks to Tim Lucas for putting this months’ ’casts together.

The Rails log is a very valuable resource. It contains a lot of information and its relatively easy to read.

However, I find that It doesn’t take long to get pretty sick of scrolling around, visually decoding, not to mention that the ansi covered SQL is a pain.

Logeye is my itch-scratcher to make it better and more useful. You point it at either your RAILS_ROOT or directly at a log file, and watch the requests flow past as you develop your app.

Download

Logeye 0.3.6

Features and problems

• In the open dialog you select RAILS_ROOT, not the log file itself.
• Double click a log entry to open the corresponding controller file in your editor (set in preferences)
• You can set window alpha, and make the windows float above other apps (set in preferences)
• The zoom + does something useful.

Upcoming Features

• filtering, searching
• -smarter opening.-
• refined log details view with visual distinction for each kind of line.
• easy copying of SQL.
• easy opening of files referred to in backtraces.
• backtrace filtering.

update 1 2007/08/19

Released Logeye 0.3

changes:

• refined look
• ability to open log files directly
• parsing of intra-request fluff

update 2 2007/08/19

Released Logeye 0.3.5

changes:

• now opens nested controllers
• it remembers which logs it was tailing last time you had it open and reopens them
• more looks

Logeye 0.3.6 is a minor bugfix

Scenarios look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

scenario :apple do
  @apple = Fruit.create! :species => 'apple'
end

scenario :orange do
  @orange = Fruit.create! :species => 'orange'
end

scenario :fruitbowl => [:apple,:orange] do
  @fruitbowl = FruitBowl.create! :fruit => [@apple,@orange]
end

scenario :kitchen => :fruitbowl do
  @kitchen = Kitchen.create! :fruitbowl => @fruitbowl
end

By default, these are loaded from RAILS_ROOT/spec/hornsby_scenarios.rb

…and you use them in specs like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

describe Fruit, "@apple" do
  before do
    hornsby_scenario :apple
  end
  
  it "should be an apple" do
    @apple.species.should == 'apple'
  end
end

describe FruitBowl, "with and apple and an orange" do
  before do
    hornsby_scenario :fruitbowl
  end
  
  it "should have 2 fruits" do
    @fruitbowl.should have(2).fruit
  end
end

Setup

Install the plugin:

./script/plugin install http://rails-oceania.googlecode.com/svn/lachiecox/hornsby

Or piston:

piston import http://rails-oceania.googlecode.com/svn/lachiecox/hornsby vendor/plugins/hornsby

Add the following to spec_helper.rb

1
2
3
4
5
6
7
8

# by default this loads scenarios from RAILS_ROOT/spec/hornsby_scenarios.rb
Hornsby.load

Spec::Runner.configure do |config|
  ...
  
  config.include(HornsbySpecHelper)
end

Advanced Usage

Its just ruby, right? So go nuts:

1
2
3
4
5
6

1.upto(9) do |i|
  scenario("user_#{i}") do
    user = User.create! :name => "user#{i}"
    instance_variable_set("@user_#{i}",user)
  end
end

Rake

If you’d like simply to load your scenarios into a database, use the rake task like so:

$ rake hornsby:scenario RAILS_ENV=test SCENARIO=fruitbowl

TODO

• Make transactional for speed.
• Add scenario namespaces for better organisation.
• Detect scenario cycles.

Credits

Lachie Cox

The code is based on Err’s code found in this post

License

MIT, see LICENCE

seesaw

As you’ve probably heard already, Matt and Max built a sweet little gem called seesaw. It allows zero-downtime, zero-request loss restarting of mongrel clusters. They put their heads together and wrought it before our eyes. I don’t think they noticed anything but the code all day :)

gem install seesaw

javascripter

Pat knocked up a javascript version of his “Styler” plugin. It allows you to include controller- or action-specific javascript in your views. Very useful stuff, as it can add another dimension to your javascripting, and the polish you apply to them.

numbr5

numbr5 got some love from Martin and Harry. They added

Duke

Our perennial favourite web based dukebox “duke” was dusted off by Nick and Dylan, and taught to sing. Well almost. They tried to get it using net/raop but the new airport expresses weren’t really bedded down. Then icecast was attempted. Next Hax day, eh boys?

odds and ends

I made a few tweaks to roro faces, while other guys just soaked up the knowledge, vibe and brownies.

afterword

Some guys stayed around for lovely curry made by Lone.

We’re hoping to have many more Hax Days over the coming months, and I think they’ll become a catalyst for more and more cool, useful stuff.

Well whaddaya know? The second roro Sydney July presentation video has dropped. This time it’s Patrick Crowley from SD Ruby, San Diego talking about his two latest plugins, Headliner and Styler