press play on tape

HTTP... something

2012-06-13T00:00:00+02:00

A survey of HTTP client-side libraries. Too much choice. And my 2 cents.

Note that these are in no particular order and that the list (still) isn’t complete. If you have something that deserves a mention here, let me know. I will include it for novelty, not for repetition.

http (2011), nestful (2010), righthttp (2010), simplehttp (2007), http_request.rb (2009), plain_http (2011)

I have a hard time seeing the differences between these libraries. There sure are some, but they might have well copied the API from one another.


  Http.get 'http://example.com'             # http
  Nestful.get 'http://example.com'          # nestful
  HTTP.get 'http://example.com'             # righttp
  SimpleHttp.get "http://www.example.com"   # simplehttp
  HttpRequest.get('http://www.github.com')  # http_request.rb
  PlainHTTP.get('http://www.google.com')    # plain_http

nestful has probably the most features geared towards JSON.

It looks as if everyone is scratching the same itch. And an itch that itches so bad, you can’t be bothered with google anymore. Some of these are better than others and nestful probably deserves its own category based on features, but there.

Typhoeus (2009)

Typhoeus is a mythical greek god with 100 fire breathing serpent heads.¹ In the case of typhoeus, a head stands metaphorically for a connection – this library allows more than one connection be made at the same time. The documentation at github has a nice ‘getting started’ tutorial that shows you how to do requests.

Typhoeus really will do parallel requests! If you’re someone who calls 100 backend services on every request (thanks, SOA!), you will need this in one of its forms. This library looks very feature complete and advanced, but as you might have guessed, this will also work:


  Typhoeus::Request.get('http://example.com')

An interesting variation here is that you can use Typhoeus as a DSL in your classes, giving them HTTP call perks. See this post from Paul Dix for an in-depth explanation.

httpadapter (2010)

This isn’t a HTTP speaking library, but rather it ‘translates HTTP request and response objects for various clients into a common representation’. The idea seems to be to create a client-side rack.²

http_connection (2009)

This library by RightScale.com seems to attempt to do reliable http connections that do connection retries. It doesn’t have as many features as others, but reliability seems to be something that others don’t address.

Is it something that needs to be handled by your HTTP library?

httpclient (2007)

This seems to be the most complete client like implementation of HTTP. It has things like cookie support. (storing cookies from one request to the next, like a browser does)

The samples are very informative and should be easy enough to pick up.


  client = HTTPClient.new
  client.get 'http://example.com'

httparty (2008)

I’ve never heard of this before today and it seems I am late to the .. well let’s not go there. This library is similar to Typhoeus in that it allows you to create special purpose classes that encapsulate HTTP functionality.

Here’s the simple example from the project page:


  class Twitter
    include HTTParty
    base_uri 'twitter.com'
    basic_auth 'username', 'password'
  end

  Twitter.post('/statuses/update.json', :query => 
    {:status => "It's an HTTParty and everyone is invited!"})

It’s been around for some time and even though it doesn’t seem to be as heavy as Typhoeus is on parallelization, it seems to allow terse special purpose HTTP request classes to be written in just a few lines.

net-https-wrapper (2009)

This library is special in its minimalism. Its (rather short) gemspec has the same amount of lines as its implementation. The author doesn’t seem to think we’d need #get, ever, so it only speaks #post or #put. Is it art?

http_spew (2011)

If one of these libraries gets outlawed, it’s this one. This seems to be the only library I’ve looked at that doesn’t bother with the responses that are sent by the server. Instead, it attempts to generate as much traffic as possible without reading things back. As the author rightfully remarks, this may get you banned from the internet. Or wait. No. Daily business.

fast_http (2010)

Art? Or just plain fast! HTTP? I cannot really tell from the documentation.

net-http-persistent (2010), persistent_http (2010)

These libraries are not about the API, but about making HTTP connections persistent. This is also something a browser does and probably should be a feature of the base library.

persistent_http will allow pooling connections instead of keeping one connection per thread.

net-http-pipeline (2010)

Allows HTTP/1.1 request pipelining:


  require 'net/http/pipeline'

  Net::HTTP.start 'localhost' do |http|
    req1 = Net::HTTP::Get.new '/'
    req2 = Net::HTTP::Get.new '/'
    req3 = Net::HTTP::Get.new '/'

    http.pipeline [req1, req2, req3] do |res|
      puts res.code
      puts res.body[0..60].inspect
      puts
    end
  end

HTTPal (2007)

Similar to httpclient, this keeps cookies and referrers for you. A simple library that seems to either do its job or is abandoned – no releases since the initial few.

httpi (2010)

A common interface to several HTTP backends: This library speaks curb, httpclient and net-http. The idea is that it gives you a simple interface to remember and allows all of these backends to be swapped out, depending on your needs.


  HTTPI.get 'http://www.example.com'    # will work

Conclusion

I’ve used httpclient (for its cookie caps) and httpi (by accident) in the past. No gripes with those. Also, I must have used net-http (stdlib) at some point. The fact that I didn’t write one of these libraries must mean that it didn’t bother me as much as it did others.

I’ve added Typhoeus and HTTParty to the list of libraries I would use. The API idea behind these makes sense and used in the right place, it will simplify things. Typhoeus certainly comes in handy when making hundreds of requests to backends.

Interesting to know about: net-http-persistent and net-http-pipeline.

Write another one?

No.

Perhaps we should all clean up our respective acts and:

Use google before using one of the many project skeleton generators that exist. (That is a list to be made another day…)

Merge some of these? Make better libraries instead of more?

And a parting note

I could make these kind of lists for many problems we Rubyists solve. The authors of these libraries are not to blame. (or rather, they are only to blame for their part of the problem) The Ruby community as a whole is currently experiencing a severe shortness of attention span. If it is not new, it is probably crap, right?

No. We should stop the proliferation of new libraries and start working good ideas into the libraries we already have. Also, a good telling sign whether you need to release a library is: Is the library bigger than the Gemspec?

¹ http://www.pauldix.net/2009/05/breath-fire-over-http-in-ruby-with-typhoeus.html

² http://httpadapter.rubyforge.org/api/

cod is a fiction

2012-03-31T00:00:00+02:00

TL;DR About what cod is not and what it is; and on how a good story builds the kernel of every API.

I am currently underway documenting cod for a broad usage. This is always an interesting process, since writing about a subject seems to be a good way for me to start thinking on other levels than just code.

Let me start by saying what cod is not:

Cod is not a big lump of code It could have turned out to be, but no, it did not. Cod is a really thin layer over the RPC and IPC mechanisms your OS and Ruby already provide.

Cod will not be your only require It is probable that beyond cod, you will need a few extra libraries to do what you want. It doesn’t solve a problem domain all by itself, but it gives you one of the pieces.

So now here’s what cod is:

Cod is really a fiction

Much more than a set of new functionalities that you can enable only by using cod, it is a fiction. It provides you with a way to think about the world of IPC that may very well shape that world for the better.

Stories like these, once shared with others, allow you to refer to a chunk of lifetime by a few words only. Thinking of chunked (‘told’) memories, you can reduce them to what you’ve told yourself what they are. This reduction is a reduction in complexity that we seek in code. In life, too much reduction is dangerous, since we’re also reducing the stuff life is made of. In code, reduction will benefit your life, since you now have time to spend off the hook.

To achieve these goals, cod introduces a new abstraction. It uses the concept of channels, ways for information to flow. These channels have each a #get method that allows you to retrieve information from them. And symmetrically, they have a #put method that allows to put information into them.

This is the basic thing. There is a small universe around those two: Learning how to know when to stop retrieving information, learning how to wait for more information without polling. Together, they form the base dictionary for programs that use cod; it is a small dictionary composed of only a few ideas.

Specifics

And then, once you start using it in real life, you need specific features that your transport provides. An example would be that pipes created by IO.pipe are asymmetric: in most processes, you will either read from or write to such a pipe. I account for this by having special methods in the pipe channel, methods that allow splitting the pipe into two pipes that are read- or write-only.

But wherever possible, I’ll do these things without having you notice. For example: If you create a channel like this:


  channel = Cod.pipe
  fork do
    # here: #put to the channel
    channel.put "This is information"
  end
  # here: #get from the channel
  channel.get

the split will happen behind the scenes and automatically. In daily use, you need not worry.

These specifics still follow the philosophy of the library; they try hard not to introduce new concepts, but rather stay within the fiction. They are like that side-story that you never tell, but for the ones that ask the right question. You went to south-america, but why: for a woman. It went nowhere and is irrelevant to your story, but there: motivation for telling the story & it still fits in the frame.

Environment

Like all good stories, cod takes place in a environment that is not a void. There are useful characters around, and it tries to take them into account. Of course, these links all cost time; so this area of the library will need to grow over time as well.

For example, a useful transport for information in todays world are queue servers. There are a couple of names in this domain: RabbitMQ, beanstalkd, redis, zeromq and … more. A queue fits into the cod fiction by providing a transport for information that allows the participants of the communication to be on several physical/logical machines or to have non-overlapping lifetimes. So cod will, wherever possible and needed, provide channel abstractions of these queues. For example you can already do the following:


  channel = Cod.beanstalk('my_message_tube')
  channel.put [:a, :piece, :of, :information]
  
  # and later
  channel.get # => [:a, :piece, :of, :information]

But cod will never become the library to manage these queues. Like a good story, it fits into the world around it and doesn’t disrupt it; A story that disrupts the world around it is called psychotherapy; let’s leave that to another library and another day.

A seed for more

I’ve talked about how cod is like a story, how it allows to talk about the world in reduction to what is important. I’ve shown how cod deals with nagging questions and the world around it.

I haven’t given you the story, and that was on purpose; if you want the story, you can have a few bits here and there right now. More documentation will be available soon.

Every API should have a kernel that is story. From there on, things fall in place naturally; If the story is crooked, so will the API be; but it will have coherence. If the story is fair on the other hand, your programs will look nice and work together like a chorus of unicorns. Either way, the building principle called ‘story’ is valid and applies.

Introducing the floor_manager plugin

2011-09-28T00:00:00+02:00

One of the decisions a modern development team has to make is: What kind of fixture framework should we use? By now, I hope that YAML fixtures have failed you…

In this blog post, I will tell you about the answer I’ve found and why I think it is useful. But please, do not think that mine is the only answer in the field. Look for factory_girl and others.

floor_manager

As of today, I’ve released the floor_manager gem, which you can install in the usual manner. To be able to tell you how to create fixtures using that gem, here’s a few datamapper models that I’ll be creating a fixture for:


class Person
  has n, :accounts
  has n, :groups
  
  property :full_name, String
  property :age, Integer
end

class Account
  belongs_to :person
  belongs_to :primary_group, model: 'Group'
  
  property :name, String, key: true
end
class Group
  belongs_to :person
  
  property :name, String, key: true
end

This is datamapper syntax for Ruby 1.9. If someone translates this into ActiveRecord, the following examples will all work.

Let’s create a series of fixtures for these three models. Incidentally, they will serve to illustrate almost all aspects of floor_manager.

Factory floors

The line is: “floor_manager doesn’t just handle individual girls, it handles entire factory floors. And it is even less sexistic, since it handles employees, not just ‘girls’”. You say: yes, but what does it mean?

A floor is a namespace of fixture objects. Such a floor contains employees, individual fixture templates or singletons. A really simple floor looks like this:


FloorManager.define(:my_simple_floor) do
  one :employee do
    name 'John Doe'
  end
end

This defines a singleton fixture (more about that later) named :employee, an instance of the Employee class. This employees name field is then set to ‘John Doe’.

Here’s how you use :employee in your specs:


describe "some spec" do
  let(:floor) { FloorManager.get(:my_simple_floor) }
  let(:employee) { floor.employee }
end

This gives you an instance of Employee (imagine this is also a model you have) with ‘name’ filled in, but not saved. In fact, it is aequivalent to calling floor.build(:employee). Here are some other ways to get objects from the floor:


floor.create(:employee) # => a saved model, provided it could be saved.
floor.build(:employee) # => build, but don't persist, see above
floor.attrs(:employee) # => just an attribute hash

As you can see from the above code, floor_manager forces you to name your fixture spaces. It also encourages you to start new fixture spaces per topic. Or maybe even one per spec?

Floor definitions go into normal ruby files that you keep below spec/support. Or anywhere else. You will then require these files in your spec_helper.rb.

Simple fixtures: Singletons

What does the term singletons mean? It refers to what you already do when you use YAML fixtures: Singletons only ever exist once in memory and also only once in the database, if they are created or saved.

Here’s a floor with a singleton:


FloorManager.define(:singleton) do
  one :john_doe, model: Person do
    full_name 'John Doe'
  end
end

When you use floor.create(:john_doe) in your specs, you will trigger one database INSERT and then subsequently just access the instance you generated. This can be helpful when you really want just one john_doe in all your tests, let’s say he’s your system administrator and you want to test with :john_doe specifically, not just a john doe.

Generators

Let’s say you define the floor :generators as follows:


FloorManager.define(:generators) do
  any :person do
    full_name 'John Doe'
  end
end

When you now access :person from the floor through any of the methods on floor (#create, #build, #attrs) you will get a new instance every time. This means that each #create will trigger an INSERT, if the created model is valid.

Generators¹ often need to create random integers or strings for each of their instances. Here’s how you would do that:


FloorManager.define(:generators_random) do
  any :person do
    full_name.string(8)
    age.integer(21..50)
  end
end

I guess that is pretty self-explanatory. This gives you any kind of person that is legally of age but not yet retired.

Note that random generation is not one of floor_managers main strengths. Sorry. Use this:


any :person do
  full_name { ... create a name here }
end

This should get you where you would need to be to use any of the available pseudo-data generation libraries.

Associations

Here’s how you would create a Person instance with Account and Group attached:


FloorManager.define(:full_example) do
  one :person do
    full_name 'John Doe'
    age.integer(20..50)
    
    groups.append :group
    accounts.append :account
  end
  one :group do
    name.string(8)
  end
  one :account do
    name.string(8)
    primary_group.set :group
  end
end

Collections have the #append method for adding objects to them, methods that accept a single object should be filled with #set.

When I now call


  person = floor.create(:person)

I will get a Person instance that has a group and an account attached. The above statement yanks an entire object graph into existence! I think that is pretty cool.

If you’re asking yourself why I didn’t use any in the above example, please think it through and post the answer in the comments. I’ll post the real reason about next week…

Setup, Teardown: Caveats

Like all good things, floor_manager has a few caveats attached, things to remember.

Before you can use fixtures (any kind of fixtures, really), you will need to reset your database to a known state. I mostly prefer this to be an empty database, best achieved with database_cleaner. Have a look at that project and install this properly, then you can forget about floor_manager preconditions.

When you don’t remember this, what will happen is the following: You’ll eventually create an object that has some kind of unique constraint (like Account and Group above) twice. The second time around, the INSERT fails. You get an error message that looks like it’s coming from floor_manager, but really is issued by your ORM system. And the insert fails. And you can’t execute your spec. This means a spec fails because you have a side effect via the database from an earlier spec, something you absolutely want control over/ never to happen. So run already, use database_cleaner, in any of its modes!

Also, since FloorManager retains some (global :() state, you will need to call FloorManager.reset before each spec. This is best achieved with the following bit in spec_helper.rb:


Rspec.configure do |config|
  # other stuff...
  
  config.before(:each) { FloorManager.reset }
end

Works With

Rails & ActiveRecord
Sinatra & Datamapper
Probably a lot of other stuff: It is written to be flexible and environment-agnostic.

Why you should use floor manager

Because it gives you

fixtures as code
separation of fixtures
access to attributes and unsaved objects
flexibility between generator and singletons
yanks entire object graphs into existence!


$ gem install floor_manager

right now! The mgmt.

¹ Generators are what factory_girl calls a ‘factory’. The factories of floor_manager build bigger items.

Parslet Tree Mangling and Transformations

2011-06-29T00:00:00+02:00

A simple parser in parslet will at first only output strings from its parsing stage:


  class SimpleParser < Parslet::Parser
    root :ifthenelse

    rule(:ifthenelse) {
      _if >> _then >> _else.maybe
    }

    rule(:_if)    { str('if') >> space? >> number }
    rule(:_then)  { str('then') >> space? >> number }
    rule(:_else)  { str('else') >> space? >> number }

    rule(:number) { match["0-9"].repeat(1) >> space? }

    rule(:space?) { match["\s"].repeat }
  end
  
  SimpleParser.new.parse(%Q(if 1 then 2 else 3))
  # => "if 1 then 2 else 3"@0

This means that a match has been found at input position 0. That’s all the information you can get from this. But what if this is not enough?

You add tags. Let’s say, we want to know what numbers are in the if, then and else branch, respectively. It would only be natural to assume that tagging number like this:


  ...
  rule(:number) { match["0-9"].repeat(1).as(:number) >> space? }
  ...

would be enough. And it is, in a way:


  SimpleParser.new.number.parse('123')  # => {:number=>"123"@0}

So you launch the full example (without extracting the number parser) and you get this:


  > ruby simple.rb
  Duplicate subtrees while merging result of 
    IFTHENELSE
  only the values of the latter will be kept. (keys: [:number])
  Duplicate subtrees while merging result of 
    IFTHENELSE
  only the values of the latter will be kept. (keys: [:number])
  {:number=>"3"@17}

The warnings that parslet outputs can be shut up with -W0 on the command line, but the result is hardly what we expect. (what about 1 and 2?) So the warnings probably mean something. To explain this, I’d like to dive into what parslet does with the output from its parsing stages (also called atoms).

Output Merging Algorithm (Handwaving Version)

Let’s consider a parser that has no tags (.as(...)) in it. The two basic atoms of such a parser will have strings as output:


  str('a').parse('a')   # => "a"@0
  match['a'].parse('a') # => "a"@0

This is always the case. As shown above, the entire parser will have the input string as output. Here are the stages such an input string goes through before being returned to you:


  parser = str('a') >> str('b') >> str('c')
  input  = 'abc'

  # Stage 1: Sequence matching result (yes, a Ruby Array)
  ["a"@0, "b"@1, "c"@2]
  
  # Stage 2: Merging ajacent strings
  "abc"@0

From this, we’ll deduce merge rule #1: Elements of a sequence or a repetition match are concatenated if they’re strings. This rule is enough to explain the first result we had above where our :ifthenelse rule would return the string we fed into it.

Let’s inspect what happens when we start tagging. The most interesting case is partial tagging, since that is what occurs most often in real life. The things we parse contain delimiters that are not signal, but syntactic noise:


  parser = str('a') >> str('b').as(:important) >> str('c')
  
  # Stage 1: Basic matching
  ["a"@0, {:important => "b"@1}, "c"@2]
    
  # Stage 2: Merge
  {:important=>"b"@1}

Here’s rule #2: In mixed sequences and repetitions, hashes (tagged elements) and arrays (collections of tagged elements) are retained.

Now to add a last element, let’s also look at how multiple tags are handled:


  parser = str('a') >> str('b').as(:b) >> str('c').as(:c)
  
  # Stage 1: Basic matching
  ["a"@0, {:b=>"b"@1}, {:c=>"c"@2}]
  
  # Stage 2: Merge
  {:b=>"b"@1, :c=>"c"@2}

And here’s rule #3: In sequences, hashes mean a named subtree. As such, the branches are merged into one hash.

Why the “Duplicate subtrees” Message?

We now have enough rules for figuring out what the message


  Duplicate subtrees while merging result of 
    IFTHENELSE
  only the values of the latter will be kept. (keys: [:number])

means. Each of the parser atoms :_if, :_then and :_else essentially output {:number => X}. (#2) Those are then merged together into one hash. (#3) Since all of them have a single key :number, parslet cannot merge them together without discarding information. As output we get


  {:number=>"3"@17}
  # result of 
  {:number=>"1"@3}.merge({:number=>"2"@10}).merge({:number=>"3"@17})

How to fix this

To fix this, you will need to provide parslet with more information on how to arrange the information:


  class SimpleParser < Parslet::Parser
    # ...
    rule(:_if)    { str('if') >> space? >> number.as(:cond) }
    rule(:_then)  { str('then') >> space? >> number.as(:then) }
    rule(:_else)  { str('else') >> space? >> number.as(:else) }
    # ...
  end
  
  SimpleParser.new.parse(%Q(if 1 then 2 else 3))
  # => {:cond=>{:number=>"1"@3}, :then=>{:number=>"2"@10}, :else=>{:number=>"3"@17}}

Results are still merged together, but this time the keys are all distinct. Hence no warning and no discarded information.

And Transformation

Let’s consider transforming this into something useful:


  class SimpleTransform < Parslet::Transform
    rule(:number => simple(:n)) { Integer(n) }

    rule(
      :cond => simple(:c), 
      :then => simple(:t), 
      :else => simple(:e)) { [c, t, e] }
  end
  
  SimpleTransform.new.apply(SimpleParser.new.parse(%Q(if 1 then 2 else 3)))
  # => [1, 2, 3]

As you can see, a first rule transforms all :numbers into real Ruby numbers in one single step. A second rule then uses the semantic tags to reorder the information in the tree into a simple triplet of [COND, THEN, ELSE].

This shows that both levels of tags (:numbers and :if/:then/:else) are needed and useful.

Careful readers will have noticed that the above Transformer is missing a rule for the case where ‘else’ is missing. Here’s the full Transformer:


  class SimpleTransform < Parslet::Transform
    rule(:number => simple(:n)) { Integer(n) }

    rule(
      :cond => simple(:c), 
      :then => simple(:t), 
      :else => simple(:e)) { [c, t, e] }
    rule(
      :cond => simple(:c), 
      :then => simple(:t)) { [c, t, nil] }
  end
  
  SimpleTransform.new.apply(SimpleParser.new.parse(%Q(if 1 then 2)))
  # => [1, 2, nil]

Due to the nature of the transformers in parslet the case distinction gets folded into multiple pattern matching rules. This is supposed to remind you of functional programming – Duplication is a good thing, at least here.

Wrapping it up

I’ve tried to show you some of the more technical aspects of parsing and transformation. We’ve seen how parser output gets mangled to throw away all those strings parslet deems unimportant to you. With the three rules given above, you will be able to reason about this stage.

Also, I hope to inspire a ‘more is more’ approach in connection with transformer rules. Rule actions should not handle case distinctions; a properly encoded tree will contain enough variety to be able to match different cases directly. It goes without saying that the code in these blocks should be kept short, since some of it will be repeated.

Introducing Cod

2011-04-21T00:00:00+02:00

Cod is a library for interprocess communication that I am working on in the wee hours these days. My goal is to make things really simple and to overlay sockets, pipes and other means of communication with an interface that is easy to remember.

Cod is open source software and will be released sometime this year. In the meantime, please look at github.com/kschiess/cod to get an idea. Try it out and send me feedback!

UPDATE (2012-03-31): I’ve since rewritten cod from scratch. Some of the stuff talked about here will not work anymore.

Building on small elements

One of the main ideas of parslet was to build on small elements and to construct larger ideas from them. This style of construction pervades my work and never gets old for me. In Cod, I am trying to achieve much the same. All work you do with it will be based on the concept of a unidirectional communication channel:


  channel = Cod.pipe

This is a channel based on IO.pipe. It abstracts all the nitty gritty for you. Here’s how you communicate with child processes:


  control = Cod.pipe
  
  pid = fork do
    loop do
      # do something important & interesting
      break if control.waiting? && control.get == :shutdown
    end
  end
  
  # eventually, deciding that we want to shut down the child: 
  control.put :shutdown
  
  Process.waitall

As you can see in the above sample, Cod defaults to serializing Ruby objects through channels. This will be open for customization later on, but is mostly what you’ll want anyway.

For now, you are forced to ask if messages are queued up (control.waiting?) – however I will be working on allowing a Cod.select as well, implementing mixed #select calls on real FDs and Cod channels. This is real important for the style of process design I am aiming for.

Channels in channels

Cod channels try to be serializable. This sounds trivial, but has really profound implications. Let’s say you want to create a client/server type of interaction: You will have a common server address and open up a channel to it. But to be able to get a response back, you will need two things:

A channel to receive answers in
and to be able to communicate the channel to the server.

Here’s how you would probably do this in Cod:


  server = Cod.beanstalk('somehost:11300', 'server')
  client = Cod.beanstalk('somehost:11300')
  
  server.put ['heiho server!', client]
  answer = client.get

And on the server side of this:


  comm = Cod.beanstalk('somehost:11300', 'server')
  
  message, client = comm.get
  answer = act_on(message)
  client.put answer

But of course, in real life this is slightly more complex. You will want to use Cod::Client and Cod::Service for this kind of interaction; it bundles this functionality and delivers a debugged and robust implementation.

Bleeding edge

The above code will not even work. I am illustrating the idea, see. But it might very soon. Have a look at ./examples – those typically work. For now, assume that nothing else does.

Patterns of distributed computing

Cod will: Capture and encode patterns of distributed computation/communication in a library that allows the programmer to focus on the level above. To allow communication (even cross-communication?) with the newest fad queue server X, you’ll write a channel implementation for X and then profit of all higher order primitives directly.

Nifty. Been chuckling about the simplicity of it for days now. (Insert thunder & lightning)

Parslet and its friends

2011-02-02T00:00:00+01:00

Parslet is a small parser framework for Ruby. It uses the PEG formalism for its grammars, and its not alone to do so. I have been asked to compare parslet to Treetop and Citrus. For raw parsing speed. With complicated grammars.

I’ve generated treetop and citrus versions from rkhs Ansi Smalltalk grammar. As a byproduct, parslet now exposes visitors for the grammar and can generate these two formats as string.

Benchmarks have been run using the above grammar with varying input sizes. This matters a lot for this grammar, since it nests very deep. Speed is not going to vary linearly with size of input in some cases.

I’ve included two versions of parslet, the current one and the very soon to be released (these days in fact) 1.1.0. The difference between the two is about one month of performance optimization.

Treetop and citrus enter the competition with the latest released gems. (1.4.9 and 2.3.4 respectively) I am putting the code that generates these grammars into the next release of parslet, so these benchmarks can be rerun once either one changes.

Here are the pure numbers:

Input size	parslet-1.0	parslet-1.1	treetop	citrus
185 chars	23.644s	0.291s	0.644s	2.503s
361 chars	14m28.970s	0.335s	0.405s	1m16.155s
671 chars	19m2.713s	0.390s	0.421s	1m26.287s
1671 chars	ages	0.581s	0.477s	1m34.930s
9796 chars	ages	2.281s	0.933s	5m8.059s
22186 chars	ages	5.521s	1.630s	7m35.694s

These figures have been measured on my weakly MacBook Air (1.6 GHz, the old one) using Ruby 1.9.2 p136. The exact figures don’t matter much, we’re after a graphic of big O here:

As you can see, treetop and parslet kind of fight it out along the X axis. Here’s another graphic that displays the relevant details of that:

Conclusions

There is a big difference between the 1.0 and 1.1 version of parslet. In fact, I think it is now big enough to warrant a release. I’ll continue to do performance optimization. The numbers indicate that parslet looses out versus treetop for big inputs because it allocates a lot of small objects. I might find ways around that.

One of the ideas I’m tossing around is to optimize not only the way work is being done, but the amount of work itself. If you imagine a tiny part of a grammar that is still not too unlikely to occur in reality:


  str('chunky') >> str('bacon')

It is quite obvious that this could be simplified by a grammar walk to


  str('chunkybacon')

This impacts on several levels. There is now just one parslet to test against, which means less code to execute. Input is read in bigger chunks; this will also benefit performance. And finally, error messages might even be improved by an optimized version of the grammar. Did I mention that you can now visit the grammar tree inside parslet by requiring parslet/atoms/visitor? So things are ready for this step, its just that I want to flush the other changes to the public, so to speak, before starting something new.

Parslet vs…

Comparing parslet against treetop or citrus based on performance alone will always miss the point. Certainly parsing speed matters. But being able to progress with your parser project and not hit roadblocks early matters just as much. I’ve gotten good feedback in that respect; it seems that what works for me also worked for others. Also, I am happy that people on irc (#parslet on freenode) have been helping each other out, (re)creating the friendly atmosphere I associate with Ruby.

I think parslet needs to make further progress in several directions. One direction will certainly be execution speed. And treetops code generation really does a lot, at least so it seems. But realize, treetop does less work as well; parslet keeps all these detailed error messages around for you to peruse. That’s got to count, right?

These are exciting times

And I am happy to be part of it. If you haven’t seen parslets quiet beauty yet, you should head to the project home page and check it out. If you have, please upgrade to 1.1 – it contains bug fixes and a speedy core. Doing so will improve both yours and your users experience!

Stack Traces in Parslet

2011-01-11T00:00:00+01:00

I get a lot of interesting puzzles from users of parslet. You think you’re asking me about a feature, but generally, if you have to ask, I haven’t yet written about it in parslets documentation. And I usually need to, since it is obviously an issue.

This post treats why parslet doesn’t do any compilation; or rather: why parslet doesn’t need to do any compilation.

Why the compilation phase?

One of the reasons other parser frameworks (‘generators’) generate parser code for you is that when the parse fails, the exception will contain a stack trace useful to you. Let’s look at a small sample in the Treetop dialect:


  grammar Simple
    rule addition
      integer (plus integer)*
    end

    rule plus
      '+' space?
    end

    rule integer
      [0-9]+ space?
    end

    rule space
      ' '+
    end
  end

When you apply this to the text ‘1 ++ 2’, Treetop returns nil and inspection of the associated failure reason yields this error message:

Expected   at line 1, column 3 (byte 3) after +

This means that Treetop has been expecting a ’ ’ (space) at byte 3, but saw a ‘+’ (plus). What Treetop could do (but doesn’t currently) at this point is raise an exception at the exact place the parse fails. I’ve inserted the above error message as


fail "Expected   at line 1, column 3 (byte 3) after +"

at the location the parse fails. This gets me the following stack trace:


(...)/simple.rb:202:in `block in _nt_space': Expected   at line 1, column 3 (byte 3) after + (RuntimeError)
  from (...)/simple.rb:197:in `loop'
  from (...)/simple.rb:197:in `_nt_space'
  from (...)/simple.rb:164:in `_nt_integer'
  from (...)/simple.rb:41:in `_nt_addition'
  from (...)/treetop-1.4.9/lib/treetop/runtime/compiled_parser.rb:18:in `parse'
  from driver.rb:8:in `<main>'

Since Treetop compiles your grammar down to ruby, the stack trace contains all rules as method names and indicates where in your grammar you might collide with the input provided. So the stack trace is for debugging. Or rather it could be, if #terminal_parse_failure in Treetop is overridden to raise an error.

Limitations

The stack trace reflects the state of the call stack when the error happens. This stack is linear, and so must be the stack trace. But PEG parsing isn’t linear: you might encounter an alternative (‘a|b’) and ultimately fail in ‘b’ because neither one matches.

Let’s assume that ‘a’ should have matched the input and contains an imperfection. We’ll be interested in all the reasons ‘a’ didn’t match the input. But the stack trace will show only why ‘b’ didn’t match, since it was tried last.

A linear stack trace will never capture why a grammar didn’t consume a particular input. If we could design the ideal kind of error message, we would try to include all alternatives that have failed.

What else would we wish for? There is one last property of stack traces that renders them unsuited for parser error reports: They capture the whole history of that process. If you give Treetop a rule like this one:


  rule rec
    '.' rec / ''
  end

which matches any number of ‘.’s (dots), your stack trace will grow with the number of dots in the input. Yet all those recursive calls wont add to the informational content of the error report. All that it will say is that you were expecting a dot, but got something else. Ideally, this report should only show that the rule ‘rec’ failed, and what it encountered instead of a dot. It should not repeat.

Not Stack Traces, Error Trees

The error reports parslet generates have both properties we’ve seen above. They preserve all relevant error information, not just the last alternative matched, and they never show all levels of recursion.

Here’s how that might look (taken from Getting Started):


  Parsing 1++2: Don't know what to do with ++2 at line 1 char 2.
  `- Unknown error in SUM / INTEGER
     |- Failed to match sequence (INTEGER OPERATOR EXPRESSION) at line 1 char 3.
     |  `- Unknown error in [0-9]{1, } SPACE?
     |     `- Expected at least 1 of \\s at line 1 char 2.
     |        `- Failed to match \\s at line 1 char 3.
     `- Unknown error in [0-9]{1, } SPACE?
        `- Expected at least 1 of \\s at line 1 char 2.
           `- Failed to match \\s at line 1 char 3.

This illustrates both accounts very well:

The alternative SUM / INTEGER produces two error reports, from left to right. The ascii tree helps reading these reports.
All parts are readable and refer our grammar directly. Line numbers cease to mean much when we’re dealing with PEGs.

And beyond

There are many more cool parts to parslet; I suggest you check it out. The upcoming 1.1 release will improve execution speed vastly; of course it also features the cool error reports you’ve just been shown.

It also features tree transformations to be able to act on your grammar, but that is a topic for another post.

Procrastinate away!

2010-12-23T00:00:00+01:00

I’ve just released version 0.2 of procrastinate, the library that is out to stop you from using threads ever again. Indeed, as has been observed elsewhere, threads in programming are evil and belong into limbo at least, if not downright hell.

Opinion to the side. While procrastination will not get a lot done, procrastinate will. You can use it to simply and directly distribute work to a few cores and then collect back the results, all in a few lines:


require 'procrastinate'
include Procrastinate

class Worker
  def fact(n)
    # Insert your factorial implementation here.
  end
end

scheduler = Scheduler.start(SpawnStrategy::Throttled.new(5))
worker = scheduler.create_proxy(Worker.new)

futures = 10.times.map do |i|
  [i, worker.fact(i)]
end

futures.each do |i, fact_i|
  puts "Factorial(#{i}) == #{fact_i.value}"
end

scheduler.shutdown

Get the code here. This uses at most 5 worker ~~threads~~ processes (Throttled.new(5)) to compute factorials behind the scenes and returns the results through an intricate system of pipes to the main process. All of this is hidden.

There are only two really important differences to calling fact(n) directly:

Your methods return value must be able to pass through Marshal.dump and Marshal.load. No returning lambdas, at least not for now.
You cannot share data with the rest of your code. Your methods will get executed in total isolation and any changes to the state of your objects made will just get lost.

And it is precisely that second point that I consider a feature, not a bug. This is real multiprocessing with no data sharing, making your functions as pure as fresh snow.

Status, When is this ready?

Procrastinate is where some of my sweat is invested right now; the library will probably evolve some more before I declare it done. It is today very useful for a number of things.

And just to drive this point home, I’ll show you another application of procrastination: System Administration! Here’s a tool I’ve been also hacking on lately: ndo. Just do a quick


gem install ndo

and you’re almost ready. For this to really work, you have to create a host set first. A host set is a file below ~/.ndo called anything you like. This file contains one line per host you want to execute a command on. Let’s say we want to talk to ‘barbie’ and ‘ken’ at the same time: I’ll call that set ~/.ndo/wonderland:


barbie
ken

With this, I can now see if either barbies or kens disk is full:


> ndo wonderland "df -h"
barbie {
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda1              74G  9.0G   61G  13% /
    none                   32G  532K   32G   1% /dev
    none                   32G     0   32G   0% /dev/shm
    none                   32G  224K   32G   1% /var/run
    none                   32G     0   32G   0% /var/lock
    none                   32G     0   32G   0% /lib/init/rw
    none                   74G  9.0G   61G  13% /var/lib/ureadahead/debugfs}

ken {
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda1              56G  7.1G   46G  14% /
    none                   32G  400K   32G   1% /dev
    none                   32G     0   32G   0% /dev/shm
    none                   32G  200K   32G   1% /var/run
    none                   32G     0   32G   0% /var/lock
    none                   32G     0   32G   0% /lib/init/rw
    none                   56G  7.1G   46G  14% /var/lib/ureadahead/debugfs
    /dev/sr0              3.9G  3.9G     0 100% /cdrom}

This will easily scale over hundreds of servers and give you a scriptable server infrastructure.

Summary

To drive this home: Start doing things with procrastinate now. Send me your ideas and sample applications. Together we’ll eliminate threading from the face of this earth.

And perhaps, execute ‘sudo rm -rf /’ on all of your servers today. Or not. Better not in fact.

Projects, Projects, Projects...

2010-11-23T00:00:00+01:00

Usually when you don’t hear from me here, I am really busy. The policy until now has been not to write about that; but there I went and cracked.

parslet

I’ve discovered that the parser library I’ve written during my brief excursion (that is to be repeated) into the world of compiler writing is valuable on its own. In fact, its probably even more valuable than any of the things I did with the compiler. That’s just the way these projects go.

I’ve thrown together a small website for the project and am working towards an 1.0 release. Encouragements, comments, … are welcome!

picky

On a related note, I’ve been loosely involved in helping to get picky ready for a 1.0 release as well. Picky is a search engine for categorized data, data that is organized in small fields. Like .. for example a database. But who has one of those, right?

Apple Mail woes

At work I was trying to get Apple Mail and Thunderbird to autoconfigure email. My goal was to have the user enter ‘john.doe@example.com’ and figure out the rest by looking at our LDAP database.

And I got it working. At least for Thunderbird which has a documented API and for Microsoft Outlook, which also has an API. Yeah, you read right. Microsoft Outlook. That didn’t go according to plan.

Apple Mail does two things for autodiscovery behind the scenes.

It phones home (apple.com) getting a binary answer that is AFAIK not documented. Try with ‘something@gmail.com’.
It pretends to speak Microsoft and doesn’t honor the answer. It posts the same autodiscovery XML request as Outlook does. Somehow I failed to get it working with the same answer that Outlook accepts. So there.

You probably got by now that this is a tale of FAIL. Indeed I posted to stackoverflow about this, but can I really expect an answer to that? Universe? Surprise me!.

Merged upstream to jekyll

2010-11-08T00:00:00+01:00

For those who use my inofficial fork of jekyll that also speaks haml, good news: I’ve just completed another merge of the new features of jekyll with the changes that enable haml and sass.

This time the merge bridges a gap of two minor versions; a lot of changes had happened upstream. Looks like we might soon be able to close down this fork and get official haml/sass support in jekyll, albeit only in unsafe mode.

Anyway, update today.

last_puppetrun - A small tool for your puppets

2010-06-07T00:00:00+02:00

To complement last weeks post about getting puppet into your datacenter, here’s a small post on a tool I wrote to monitor your puppets.

The tool will look at /var/log/messages and look for the last successful puppet run for all your nodes (as given by


  $ puppetca -al

). It needs to be run as root.

You can download it from github.


  git clone git://github.com/kschiess/last_puppetrun.git

Not the same as `puppetlast`

Connaisseurs of puppet will bark: “But there is already puppetlast”. Yes. There is. The information puppetlast gives you is “When did puppet last run on host X”?. That is not at all the same as “When did puppet last successfully run on host X?” – which is the answer provided by last_puppetrun.

If you have a parse error / a doubly defined resource in your puppet recipes, the tool presented here will allow you to see the problem early.

How to get your own puppet show

2010-05-27T00:00:00+02:00

When I mention puppet to people, a lot of them answer: “Oh, don’t get started on this, I should do that as well.” Yeah, well guess what, you should. Here’s my helping hand to that.

Puppet is a tool that allows you to manage the configuration of your data center in a recipe and then apply it to each machine. It is open-sourceish and developed by a company called ‘Puppet Labs’ (formerly: ‘Reductive Labs’) in Portland, Oregon.

Here’s what you need to get started with puppet:

A puppet master This can be any machine at all, although I would recommend you virtualize this. The puppet master will not be doing a lot at first. It seems like a real box would be wasted on this.

At least one other machine you want to manage.

Of course, the real benefits of puppet will apply when you manage hundreds or thousands of machines with it. But. Knowing puppet, I would even manage my two machine production setup with it. For any kind of setup that has real value, you need a description of some kind on how to reconstruct the machines should you loose them completely. Most people have started entering this step-by-step information into a Wiki of a kind. Puppet can be that as well.

Either way. Here’s what comes next in this article:

Pick a puppet
Install the puppet master
Create an initial configuration
Install puppet on the clients

Pick a puppet

The real trouble with puppet is that it is (still) a relatively young project and under active development. You want to get a really recent puppet version to be able to use new features. On the other hand, puppet is the first thing you need to install on a machine — to do this in an automated way, you want to be able to use what your package management gives you.

I would recommend you look at the puppet versions your package managements give you (you will probably be managing more than one OS/ OS version). Then pick the lowest number. That is the version you will be writing your puppet recipies against. If that is not high enough — you need some other solution.

We’ve gone the other way and have done our own packaging. This way, we can be sure that all puppets talk to each other. I can’t really tell you here how to do that, but a good start is to fetch the latest package for your OS, unpack it, replace the puppet files in it and then repack it. Most often, this will work fine.

Install the puppet master

The puppet master machine should be reachable as ‘puppet.YOURDOMAIN’ or just ‘puppet’. This will avoid a lot of trouble later on.

Your puppet master will need a package called ‘puppetmaster’. This can also be installed via rubygems, but most often the OS package will give you startup/shutdown scripts that the gem doesn’t have.


  root       734  0.0  0.0   5992   604 tty1     Ss+  08:44   0:00 /sbin/getty -8 38400 tty1
  puppet    1044  0.5  0.9 116356 39356 ?        Ssl  08:45   0:00 /usr/bin/ruby1.8 /usr/sbin/puppetmasterd --masterport=8140

As a result of the above step, you should see a puppetmaster binary running and you should have the ‘puppetca’ script available (as root).

You will also need some kind of source control system. I recommend git (‘git-core’).

Create an initial configuration

Here is a sample puppet configuration for you to fill in. This follows the puppet best practices. Just check this in into a git repository and fill in the blanks.

Then have a look at /etc/puppet/ on your puppet master server. (or where your config sits) Some of these files are also worthy to be included in your repository. Candidates are fileserver.conf and puppet.conf.

Once you’ve gone and completed the sample structure, you can make a checkout of it to /etc/puppet/. This is what you will be doing each time you make a change to the configuration.

Install puppet on the clients

Now let’s link up a client that has puppet installed. First, you should request a certificate from your puppet master:


  # puppetd --test
  warning: peer certificate won't be verified in this SSL session
  notice: Did not receive certificate
  notice: Set to run 'one time'; exiting with no certificate

On your server (as root), list the clients that have requested a certificate (but haven’t received one yet):


  # puppetca -l
  myfirstnode.DOMAIN

Sign the certificate:


  # puppetca -s myfirstnode.DOMAIN
  Signed myfirstnode.DOMAIN

Now repeat:


  # puppetd --test
  ...

You should see your initial configuration for the node being applied.

Parting Words

These guidelines have been compiled from documentation at puppet labs and from my own experience. Here are some of the links that I recommend also reading. And now off you go, recruit an army of puppets to reign over!

New Stuff in rspec - a Cheatsheet

2010-05-11T00:00:00+02:00

This presentation puts together a nice overview of features that old-school rspec users like me don’t really use. I have decided to put myself to it and extracted a cheat sheet from the presentation for your perusal. Full steam ahead!

Migrating Request Trackers backend

2010-04-14T00:00:00+02:00

I’ve been stupid enough to try to use Request Tracker with its default (sqlite) backend. Bad idea. Please read that and don’t think I am bashing sqlite, I am not.

As a note to self and to the next guy falling into that trap, here’s how it can be done:

Preparation

Install postgresql and make sure you can access it through a local user using passwords. This might mean you have to tweak postgresqls settings, notably pg_hba.conf.

If you’re using Request Tracker version 3.8.4, you can use the schema behind this link to prepare your database. Otherwise I would suggest using the tool that is provided in the installation (rt-setup-database) to do that. Your command line will have to look something like this:


  $ rt-setup-database --action schema

RT will use its database settings from /etc/request-trackerXXX/RT_SiteConfig.pm to connect to your database, so make sure that points to the new location…

Migration

Now you have two databases that have exactly the same schema, but one contains the data. Turn off your mail gateway and your apache (site is closed for now) and install the prerequisites for this script. This should mostly be the ‘pg’ gem and the ‘sqlite3’ gem. Using aptitude, this would look like this:


  $ aptitude install libpgsql-ruby libsqlite3-ruby

Running the script should be easy: Give it the location of RTs sqlite database as parameter. Adjust the pgsql connection settings inside the script to match your installation.

The script will insert all rows from sqlite into postgresql that are missing there. You can run the script a few times; it will reach a steady state the second time round that looks like this:


  ACL: Source: 26, Target: 26
  ... lots of tables ...

Those are all tables and their corresponding max(id)s.

Correct the sequences

Since all INSERTs are done giving the id of the rows the postgresql sequences will all be off and you wont be able to insert new data. This is easy to correct: Run the following lines against your database:


  select setval('acl_id_seq', (select max(id) from acl));
  select setval('attachments_id_seq', (select max(id) from attachments));
  select setval('attributes_id_seq', (select max(id) from attributes));
  select setval('cachedgroupmembers_id_seq', (select max(id) from cachedgroupmembers));
  select setval('customfields_id_seq', (select max(id) from customfields));
  select setval('customfieldvalues_id_seq', (select max(id) from customfieldvalues));
  select setval('groupmembers_id_seq', (select max(id) from groupmembers));
  select setval('groups_id_seq', (select max(id) from groups));
  select setval('links_id_seq', (select max(id) from links));
  select setval('objectcustomfields_id_s', (select max(id) from objectcustomfields));
  select setval('objectcustomfieldvalues_id_s', (select max(id) from objectcustomfieldvalues));
  select setval('principals_id_seq', (select max(id) from principals));
  select setval('queues_id_seq', (select max(id) from queues));
  select setval('scripactions_id_seq', (select max(id) from scripactions));
  select setval('scripconditions_id_seq', (select max(id) from scripconditions));
  select setval('scrips_id_seq', (select max(id) from scrips));
  select setval('templates_id_seq', (select max(id) from templates));
  select setval('tickets_id_seq', (select max(id) from tickets));
  select setval('transactions_id_seq', (select max(id) from transactions));
  select setval('users_id_seq', (select max(id) from users));

These statements are a copy of what himdel has on his blog.

Done

This should conclude your migration. Start apache and the mail gateway again and browse through your tickets – they are all there. Of course, this is at your own peril. But if all goes well, you will have migrated your Request Tracker instance to PostgreSQL in almost no time.

ZFS backups and other tidbits

2010-04-05T00:00:00+02:00

It seems that OpenSolaris zfs has already got zfs backup and zfs restore commands. Our Solaris release doesn’t yet. Anyhow, backing up is simple, right?

We’ll perform the following steps:

Snapshot the datasets once an hour. This can be done with the simplest of scripts, the hard part being a naming scheme for the backups.


  dataset = 'pool/something'
  name = Time.now.strftime("%Y%m%d%H%M")
  `zfs snapshot #{dataset}@#{name}`

Sync the snapshots to another machine (hot standby) using zfs send and zfs receive with a good combination of the flags -I and -R. This is almost trivial as well. ZFS does a whole lot of work there for you, you just have to come up with what to sync where.

And lastly, to prevent our pools from running full: Weed out snapshots we don’t want anymore, both in the backup and in the active set. This is what the remainder of the article is going to be about.

For all practical purposes, only the Grandfather-Father-Child scheme helps us in this situation. We don’t bother with the Towers of Hanoi even though that might be an algorithmic exercise worth it. It turns out that a generic GFC implementation is worth it as well.

The Grandfather-Father-Child (GFC) scheme sounds easy enough when presented. Let’s assume you want to keep 24 hourly , 7 daily, 12 monthly and 2 yearly backups around. This gives you a solid backlog for your data.

In the above setup, your backups will be named like this:

  pool/dataset@201004042200
  pool/dataset@201004042300
  pool/dataset@201004050000
  (... snippage ...)
  pool/dataset@201004050700
  pool/dataset@201004050800
  pool/dataset@201004050900

But to infuse a bit of real world in this problem description, here’s what’ll happen: Your administrator will snapshot any dataset at random before making an important change. He’ll either name this snapshot @pre_important_change or even @201004050907_important_change.

Another thing that will happen (as in Murphy says it does) is that your cronjob that does the snapshots will do so at irregular moments because of several factors (machine load, pesky users).

So here is a more realistic input to our algorithm:

  pool/dataset@201004042200
  pool/dataset@201004042301
  pool/dataset@201004050010
  (... snippage ...)
  pool/dataset@201004050705
  pool/dataset@201004050800
  pool/dataset@pre_important_change
  pool/dataset@201004050900
  pool/dataset@201004050907_important_change

So I was looking for something simple that could – in this context – apply rules like the ones above (24 hourly backups). After a bit of discussion with one of my coworkers we came up with something surprisingly simple given the irregularity of the problem.

I’ll express the algorithm as a method on something I will call a SnapshotSet, returning a new set containing the snapshots we want to keep. Here’s now you would use this:


  set = SnapshotSet.new([snapshot1, snapshot2, ...])
  
  keep = set.gpc(
    1.hour => 24, 
    1.day => 7)

This reduces the problem to this: Given a list of snapshots (each with a timestamp) and a list of (interval, count) tuples, find which snapshots should be kept.

The following is structured in two parts: What to do for each of the tuples and how to synthesize the resulting keep list.

For each `(interval, count)` tuple…

The image shows a timeline with snapshots in orange. Let’s say we want to find out which snapshots to keep to be able to provide 7 daily snapshots. ((1.day, 7)) To do this, we create a list of sample points starting 1 day ago, with step size of 1 day each. This is also drawn in the picture.

For each of those sample points the snapshot that is closest in time is kept. Look at this picture:

This procedure will yield at most 7 daily snapshots to keep in a best effort manner.

And how to synthesize the result

The resulting list of snapshots to keep can now be assembled by computing the union of all the snapshots that would have been kept for each tuple. This list completely covers what the rules specified.

Except that we probably don’t want to delete all the newest snapshots that the rules say nothing about, like the few ticks between 1 day ago and the now. So I’ll just say that all snapshots between now and the first tick of all tuples must be kept as well. (hand waving)

Conclusion

This three-step backup plan seems to work surprisingly well. If anyone is interested in some code, I think I could put this up on github – but I don’t want to add to the redundancy in the field ;). In this article I’ve shown how a simple Grandfather-Father-Child backup scheme could be made to work with real world input data.

A few Ruby Anti-Patterns

2010-03-18T00:00:00+01:00

Some things you can do with Ruby you just shouldn’t. Often one doesn’t notice until later on, when you are looking for a bug in code.

rescue everything

This is a story from real life. In our recent release of net-ldap we have this snippet of code:


  begin
    @socket = TCPSocket.new(host, port)
  rescue
    raise LdapError, "Something is wrong with your connection."
  end

So when you get the ‘Something is wrong …’ error, where do you go look? Right, you check and recheck your credentials.

Except that wasn’t the problem. We recently reorganized the project and fixed some Ruby 1.9 stuff. This was the reason for our rerelease as well. During that work, we (ok, probably I did) fat-fingered the import of ‘socket’ with the obvious NameError as a consequence.

The above code will tell you that something is wrong with the connection even if nothing is. Look at how I think this should be written:


  begin
    @conn = TCPSocket.new( server[:host], server[:port] )
  rescue SocketError
    raise LdapError, "No such address or other socket error."
  rescue Errno::ECONNREFUSED
    raise LdapError, "Server #{server[:host]} refused connection on port #{server[:port]}."
  end

We could dispute the details here, but the main point is that you should almost never rescue everything. This is an anti-pattern in Ruby and in other languages as well. If you don’t know what happened, make it scream really loud.

There is another big anti-pattern hidden in this war story: Write tests for your code. Because that would have told me what was wrong.

Extending basic Types

Let’s say I have a few derivates of Ruby basic types with strings attached:


  class StringString < String
    attr_accessor :strings
  end
  class StringArray < Array
    attr_accessor :strings
  end
  
  ary = [
    StringString.new('foo'),
    StringArray.new([StringString.new('bar')])
  ]

I want to create a clean YAML representation of this, looking something like


  ---
  - foo
  - - bar

But turning what I have into YAML I get this:


  --- 
  - !str:StringString foo
  - !seq:StringArray 
    - !str:StringString bar

And of course, turning it to YAML is just one of the problems you have. Everywhere you use inspection/reflection in your code, these StringString things won’t behave correctly.

Stop and think about how you would do it for a second. There seems to be no real easy way to do this, even when you own the StringString and StringArray types. Of course, if you knew you only had strings, you could use #to_s / #to_a and be done with it. But polymorphically? What we need is a #detach_strings method.


  def detach_strings(mess)
    mess.class.ancestors[1].new(mess)
  end

  puts detach_strings(StringString.new('bar')).to_yaml
  # => "--- bar\n"

Ugly, isn’t it? Yeah. But in my opinion that is entirely the blame of the extended basic type anti-pattern. I wont spend time on making this beautiful, but rather will transform the code so it doesn’t use extended types anymore.

Your own Anti-Patterns?

Do you have any Ruby anti-patterns from your work? I would be interested in hearing about them.

Notes on CoffeeScript; a javascript replacement

2010-02-22T00:00:00+01:00

This weekend I had the chance to work/play with CoffeeScript, a new language created by Jeremy Ashkenas.

I had been so desperate trying to get jquery.facelist by Ian Tearle do my evil bidding that I decided to scratch and rewrite it. After all, how complicated can things be, its just javascript, right? This was – in the past – a recipe for disaster. I would try a rewrite and end up with something as complicated as the original. And end up feeling worse about it, since this time it was my code.

A change of approach was in order. And since I am a fan of clever code generation, I decided to use CoffeeScript for my rewrite. Here’s some notes about this experiment.

Source to Source compilation

CoffeeScript installs with


  $ gem install coffee-script

Its executable coffee is a source to source compiler that turns CoffeeScript into perfectly valid and well formed Javascript. CoffeeScript is written in Ruby using a treetop parser.

Shorter than Javascript

The source code you write in CoffeeScript is considerably shorter than what you would write in Javascript. Languages either read like noise or they read like signal – and CoffeeScript is definitively signal. To me, Javascript has always seemed like just a tiny bit too complicated to properly remember – and what I wrote turned always out to be lacking in some area.

As an example, remember what you have to do to create your own namespace in Javascript? Yeah,


  (function() {
    // your isolated namespace here
  })

does the job. CoffeeScript will do that for you, no syntax to learn. Turns out, you always want to do that anyway.

Or, another example. I always have to glance at a reference to remember how to do prototype based OOP in javascript. Not just because it is not class based, that part I’ve been able to wrap my head around by now. But because the syntax is so much like blessing references in Perl:


  Foo.prototype.my_method = function my_method() {
    return alert('hi');
  };

What the fail. This is so much typing, it’s almost like Javascript is trying to look like Java. CoffeeScript makes this pleasant:


  Foo::my_method: ->
    alert('hi')

CoffeeScript doesn’t hide Javascript, it just makes it easy. (Here’s the rest of the introduction)

Writing correct Javascript

CoffeeScript also takes the pain out of writing correct Javascript. Maybe I am in this business for too long already, but I distinctly remember fighting with Internet Explorer (was it 4?) for hours on end over a misplaced semicolon. Generations of future programmers will be able to learn how not to do error reporting from that piece of crap. It just told you that you had an error in line XYZ, and whatever number X, Y and Z was, you could be sure that it was unrelated to the line the semicolon was missing in.

JSLint was an improvement. At least you could have some kind of feedback on your code before going into a frustration loop with IE. Mostly JSLint would tell you all the things that would trip up a browser and fixing its warnings could get you that far already.

CoffeeScript takes this one step further. Since its output passes JSLint without warnings, its like that red sports car – faster AND cooler. My experience with it says that when coffee compiles it, it probably runs. And since there is less noise to it, its easier to get right. No fiddling with bits of syntax, just writing code.

Plays well with others

Exhibit A: A piece of HAML code:


  .part
    .name John Doe
    .address Infinity Drive

Exhibit B: Sass


  .part
    color: #fff

And finally (maybe in that same HAML file, thanks to coffee-haml-filter):


  :coffee
    $('.name').each(->
      alert(this.text()))

My ruby liking soul is only slightly worried by the importance of indentation in these languages. Putting that aside, I can really enjoy all of them. (And no, don’t write all of your CoffeeScript inline with your haml!)

The Result

After several attempts, I have been able to complete a rewrite. The resulting code will now be used in the application I am building. You can look at it here: kschiess/jquery.objectlist.

The code I have written reads like a cross between Ruby, Smalltalk and Scheme. The methods have become really short, and some of the data is hidden as lambdas in hooks. CoffeeScript will allow to do all of this: Writing clean OO code, writing functional style and just writing idiomatic .. Javascript.

Writing this has felt good and really really funky. I have focused on the usability of the language in these notes, but I urge you to look at the introduction. CoffeeScript has

list comprehension
terse lambda syntax, with or without this binding
pattern matching
heck, even a small rake replacement!

Do check it out!

The Ruby Object Hierarchy

2010-02-17T00:00:00+01:00

Every self respecting ruby programmer has one of these on his blog. Join the club.

(Click for a pdf of this. Filled triangles are class relationships, empty triangles are super relationships, funky diamonds are funky metaclass relationships, thanks _why!)

Here is the source code for this. A few notes on the code:

The code also graphs the ‘actual class’ as reported here. I didn’t see the use of that, so it’s gone from the version displayed here.

This is probably the most shoddy piece of code I’ve ever released. Note to self: Breathe out, relax. It doesn’t actually hurt.

If you want to run this yourself, call it like this:


  $ ruby extconf.rb && make
  (... snippage ...)
  $ ruby graph.rb RECURSION_DEPTH > mygraph.dot

The code does depth first graph traversal over Ruby’s object graph and outputs a graphviz dot file. It seems that while recursion depths bigger than 3 are sure funky, they don’t add much to the picture.

A few things that are interesting in the above graph (trying not to be redundant with others):

There are really a lot more metaclasses than you think. They might gather in groups and descend upon you in your sleep, even. You think turtles is bad? Metaclasses is way badder.
Class really ties the room together. Without it, Ruby would really be off balance. I guess that’s why its called class oriented programming.
Everything is created out of nothing. It fits with my belief system to have nil sit on the top of that diagram.

Happy hacking!

toamqp 0.3.1 released

2010-02-09T00:00:00+01:00

Just a quick note to let you know that I am releasing toamqp today. I’ve already presented all the novelties in this place, now its time to get the code some action.

This release includes many bugfixes and code cleanups. It also features a TOAMQP::SpecServer helper class that allows easier testing of the code. With this testing becomes a breeze – you set it up like this:


  server = TOAMQP.server(MyHandler.new, TOAMQP::SpecServer)

Then you get two options; either you make all the calls and then call


  server.serve

This will act on all the messages that wait at the broker and then return. When you want to satisfy your mock’s expectations, this is the method to use.

You can also use


  server.serve_in_thread

and then go off and make a bunch of client calls. Just don’t forget to tear down this server thread:


  server.stop_and_join

This will wait for the thread to close down.

Conclusion

Do go off and have fun with this release. And by all means, tell me what doesn’t work.

The new Thrift / AMQP bridge: Explained

2010-01-26T00:00:00+01:00

Abstract

Since this has grown into a really long article, Here’s what its about: I talk about RPC using thrift and AMQP. This can be setup in at least two different ways, one gives you round-robin load balancing, the other allows you to filter messages. At the end I provide you with a project roadmap and an invitation to contribute.

thrift_over_amqp becomes toamqp?

I have reworked the thrift_over_amqp plugin into a real gem that can be used everywhere. To do this, I’ve rewritten the code and cleaned up the javaish API mess. Must’ve been the contact stress with thrift APIs. But this is old news already.

Let me try to give you some new .. well ah .. news. Here’s a graphic list of operation modes for thrift calls when sent via an AMQP broker:

Thrift itself provides you just with the same old point to point procedure call (first schema). Where do the other combinations come from?, you ask. Those are pretty much what AMQP does for you. Using toamqp, you can take advantage of these other operation modes for RPC calls.

To say it with the words of the king – A little less conversation, a little more action please:

One to One example

About the simplest thing you can do with toamqp is to connect two points of your infrastructure via RPC. To do this, you need to write a thrift specification of the receiver:


  service receiver {
    /**
     * Let's say we have a search server somewhere, answering with a list
     * of search results to us:
     */
     list<string> search(1: string word)
  }

This is better than the usual getTime RPC call, since we ‘now what time it is’, right? As you can see, thrift has a rich set of types that can be sent over the wire. You can create new types yourself and even do exception handling.

You may be wondering what the ‘1: ’ is about in the list of arguments of our function. That is thrift’s micro-level versioning. You give all your parameters a unique id and have to make sure you only use that id when you’re talking about the exact same thing. I can rename ‘word’, as long as I use the same id; I can also introduce new parameters unilaterally and be assured that only receivers who have use for them will receive them.

Turning the specification in some code

You can now use thrift to generate Ruby code from that. This generally looks like this:


  > thrift --gen rb receiver.thrift

This only works if you install the whole thrift suite, not just the rubygem. Doing so will give you the compiler for the thrift IDL. I may someday include a new rake task generator into toamqp for making this step real easy. For now you are stuck with the command line.

You most likely get a ‘gen-rb’ subdirectory that contains three ruby files. We’re now just one step away from writing our sender/receiver pair. For the request sender (aka client), we write:


  require 'toamqp'    # requires thrift and bunny
  require 'gen-rb/receiver.rb'

  sender = TOAMQP.client('receiver', Receiver)

The first argument to .client is the name of the exchange that you want to use for all messages. Sender and receiver must use the same name here. Think of it as the address of the receiver.

The second argument is the module that contains all thrift generated classes. TOAMQP will dynamically look up the ‘Client’ class from that module and return you an instance of that client, connected to your queue server. Speaking of which: You can configure the connection attributes using the longish method #connection_attributes= on the connection manager:


  TOAMQP.connection_manager.connection_attributes = {
    :user => 'username', 
    :password => 'password', 
    :host => 'myqueue.mydomain.com'
  }

The default is a connect to localhost.

Server implementation

Thrift makes it easy to write a receiver for our service (a simple one at least):


  require 'toamqp'
  require 'gen-rb/receiver.rb'
  
  class ReceiverHandler < TOAMQP::Service::Base
    exchange 'receiver'   # same as we used above
    serves Receiver
    
    def search(query)
      # search implementation with proper exception handling
      return %w{a few search results}
    end
  end
  
  TOAMQP.server(ReceiverHandler.new).   # create the server
    serve                               # enter its main loop

This will create thrift server instance and call its #serve method that will singlethreadedly (is that a word?) serve requests from the sender. So doing


  sender.search('chunky bacon')

will return .. no chunky bacon anymore, think about the serialization issues, but a ruby array of your search results. This is all thanks to thrift - all that toamqp does at this point is connect the thrift endpoints through your message queue.

I have chosen to make the handler implementation lean towards ActiveRecord-style class declarations. This has some disadvantages: You really have to decide what you’re going to serve and how the exchange is going to be named, for all instances of the class. On the plus side, you are free to meddle with your own constructor – and you don’t have to call super, something I always seem to forget.

Note that there is a second argument to #server that allows you to specify which server class to instantiate. You can plug one of the evented or threaded servers that come with thrift in there, although I must admit that my tests are cruelly lacking in that area. Speaking of tests: toamqp also comes with a server called ‘TOAMQP::SpecServer’ that can help you write integration tests against the whole infrastructure. You should really read the specs in the ‘spec/integration’ subdirectory of toamqp to get an example of usage for that.

Extending to round robin load balancing

This is all very plain thrift stuff – nothing you could not also achieve with tcp transports. Let’s take advantage of your message broker (your queue) a little more. One of the things that a broker does as per specification is round-robin balancing of message receivers.

If we connect more than one receiver to the exact same exchange (a simplification of issues here, more down below), they will each get 50% (in the discret sense of the term, not the quantum mechanic sense) of the load. It’s really as simple as that, just run two servers. And nothing stops you from running 10, on different machines. Let me give you a small example of a forking server that starts more than one server process on one machine:


  $pids = []
  2.times do 
    $pids << fork { TOAMQP.server(ReceiverHandler.new).serve }
  end
  
  Signal.trap("CHLD") {
    Process.wait; 
    $pids.delete($?.pid)
  }
  
  sleep 1 until $pids.empty?

Of course, this is a brutal oversimplification of the issues involved. You might want to look at the threaded server that comes with thrift or/and at unicorn, a forking web server for more information about this.

One small warning at this point: Thrift itself is not thread safe, and neither is toamqp. You cannot just spawn a few Ruby threads and use the client in all of them; you might have to create one client object per thread. And you should really look at the servers that come with thrift, they solve many of the issues with threading you might have.

So road-robin balancing and load distribution over several machines is done, what is next?

Routing messages to the right server

Sometimes you don’t just have many servers of the same kind, but many servers with the same interface of different kind. toamqp supports this directly.

Let’s say you have two places to search for, your desk and your wastebasket. (Since we all know that important documents are stored there, right?) So ideally, you would want to search them both:


  desk        = TOAMQP.client('receiver', Receiver, { :location => :desk })
  wastebasket = TOAMQP.client('receiver', Receiver, { :location => :wastebasket })
  
  [desk, wastebasket].map { |sender| sender.search('important document') }

This will fail brutally after experimenting with the code above. The reason is that for this to work, you will need to redeclare the ‘receiver’ exchange on your broker to be of a different type. Before, it was direct (as in ‘send my messages directly to the receiver’), now it should be headers (as in ‘filter by the headers I send’). Luckily toamqp does this for you, provided you clear the old variant from your servers first. You can either do this by restarting your server or by manually issuing a delete call via ‘bunny’.

For really easy experimentation you should just change the exchange name to ‘receiver_filtered’. This way toamqp will declare a new exchange and your AMQP broker will not throw an error.

I’ll just show you how to implement a wastebasket receiver; the desk receiver should follow from that. Note that since they implement the same interface but have different implementations, you will really need two different classes:


  class Wastebasket < TOAMQP::Service::Base
    exchange 'receiver', match => { :location => :wastebasket }
    serves Receiver
    
    def search(query)
      # ...
    end
  end

Combining the layouts

These two layouts (Round-Robin Load Balancing and Header Routing) can of course be combined. Together these two will help with most if not all communication needs you might have.

Possible extensions, Status, Roadmap

The current released version of the library is 0.3.0. You should not be using that, however. If you really want to help me, please check out the repository from my github account and use the 0.3.1-rc-* branch. This will give you the latest fixes and help me discover issues with the 0.3.1 release. Mind you, there should not be too many now – I am using this code in production and have encountered but few surprises.

In about two weeks time, I will be releasing 0.3.1. This marks the end of the big rewrite. Once that is done, I would like to start working towards at least partial compatibility with txAMQP for 0.3.2.

Of course, many things are left to be done. Here’s my current wishlist:

A specialized rake task for building thrift files (I have something semi-finished laying around)

Wrap the client proxy object in a thread safe way. This could lead to a generic thread-safe-o-matic for client proxies.

Do some real testing with multithreaded server. Find the issues and the solutions.

Broadcasting to many receivers? Easy to do and would enhance the API for announcing presence, sending heartbeats, etc…

Please get in contact if any questions remain (almost impossible, if you read my drivel until here, you must be the expert by now). I welcome all contributions that are accompanied by specs.

Using rake as a component

2010-01-14T00:00:00+01:00

Rake is a useful beast. As one of the most mature ruby projects it makes my life better by allowing me to forget everything I ever knew about make.

Embedded Rake

I would like to show here how you can use rake as a library, not as a tool. The following piece of code shows how you can declare tasks in your own Ruby classes:


  require 'rake'
  class RocketScience
    include Rake::TaskManager
    def task(*args, &block)
      define_task(Rake::Task, *args, &block)
    end

    def init
      task :build_rocket do
        p :build_rocket # rocket building here
      end

      task :launch_rocket => :build_rocket do
        p :launch_rocket # launching here - only works if you build it first
      end
      self
    end
  end

The Rake::TaskManager mixin retrofits task management to any of your classes. It defines a few methods that allow creation, invocation and management of your own tasks. The example uses #define_task to add tasks to the current instance of RocketScience. You can find the default definitions for #task, #file and the whole TaskManager mixin in rake.rb in rake’s source code at the top level.

Now the above example doesn’t do much yet. To show that rake is really useful for your code, here’s how to print something:


  class RocketScience   # reopen the class
    def run
      self[:launch_rocket].invoke
    end

    def options
      options = OpenStruct.new
      options.trace = false
      options.dryrun = false
      options
    end
  end
  
  RocketScience.new.init.run

We need to provide rake with an options method that defines some of the runtime options you can set in the command line tool as well. The above example will print

  :build_rocket
  :launch_rocket

A whole lot, isn’t it?

Now you will say that embedding rake like this is a complicated way of doing the following:


  require 'rake'
  
  task :build_rocket do
    p :build_rocket # rocket building here
  end

  task :launch_rocket => :build_rocket do
    p :launch_rocket # launching here - only works if you build it first
  end
  
  Rake::Task[:launch_rocket].invoke

You would be right. The only thing the first example buys you is that your rake namespace is not a singleton anymore, you can actually have multiple tasks with the same name in different instances of RocketScience.

Remote Rake

One last rake trick before you can go out to play: I recently discovered an heavily underdocumented feature in vlad the deployer: remote rake tasks from within rake. Perhaps this is underdocumented because it is so stupidly obvious to everyone (but me!), but I think it is definitly worth the mention:


  require 'rake'
  require 'rake_remote_task'

  role :server, 'server1'
  role :server, 'server2'

  remote_task :ls, :roles => :server do 
    run 'ls'
  end

  Rake::Task[:ls].invoke

The above example shows how you can execute ‘ls’ on multiple remote servers. You first define your servers as being part of a role, using either the role keyword (as above) or the host keyword:


  host 'multiple-purpose-host', :db, :application

Then you write remote rake tasks against these roles as shown. Note that the example uses the ‘rake-on-a-stick’ technique I’ve shown above and could well be integrated with everything said above. Of course, you could make it a vanilla Rakefile as well.

Since vlad is capistrano built on rake, it should be clear that vlad contains this feature. But I think this is useful for all uses of rake, not just deployments. Using this you can write Rakefiles that execute on multiple servers AND locally at once. Serious coolness. Goodbye Capistrano, welcome to our new russian overlords.

A redesign and a rename in 2010

2010-01-04T00:00:00+01:00

For those of you following the thrift_over_amqp_transport plugin: I have just finished a complete rewrite. The ‘plugin’ is now called ‘toamqp’ and is available through rubygems:


  $ gem install toamqp

The version that is currently published has some minor bugs. I will publish another version real soon now (0.3.1) that fixes those and then some.

The redesign

Starting with the new version it is now really easy to create server and client for a AMQP service. Really, its just a few lines:


  # A real small server: 
  class MyServiceImplementation < TOAMQP::Service::Base
    serves ThriftServiceModule
    exchange :my_service
    
    # Service methods go here
  end
  
  TOAMQP.server(MyServiceImplementation.new).serve


  # A real small (smaller!) client
  client = TOAMQP.client(:my_service, ThriftServiceModule)

Please have a look at the README in the source. I’ve also included a few samples in the ‘examples’ subdirectory of the project – this should get you running in no time.

I will publish a real HOWTO for a few sample use cases here soon.

OpenID explained (while cooking chicken soup)

2009-11-29T00:00:00+01:00

I am going to explain OpenID in just the time it takes to cook a chicken soup. Those two hours will allow me to write just the really simple explanation I have been looking for on the internets recently. My throwing in some Ruby code and some diagrams will allow you to hopefully create an OpenID consumer with some trust in what you are doing and in less time than it took me.

OpenID? What is that?

OpenID will allow you to use multiple sites with just one login. Follow this link to get to know more on the topic. I am going to assume you

know what OpenID is (from a users perspective) and
you already have one.

How does it work?

Keep in mind that we’re running against the clock here and bear with me while I insult your user with the following diagram:

Authenticating your user in 4 easy steps:

1. Your user gives you an OpenID, claiming it’s his.

2. You perform discovery on the OpenID. You find out that in reality, ‘the OpenID provider’ will answer questions about ‘the users OpenID’.

3. So you redirect your user to his OpenID provider. You pass along data that will help ask the question: “Is this guy who he says he is?”

4. You don’t care about how your user identifies to his service. All you care about is that you get your user redirected back to you and that this special redirect carries data proving the users association with the OpenID he gave.

And in Practice

You should install ruby-openid (gem install ruby-openid). Imaging that you have a form that POSTs the users OpenID as ‘:openid’. Steps 2 & 3 above would look like this:


  def start
    # Did the user fill in the field? (maybe step 1 failed)
    render 'form' and return unless params[:openid]
  
    # Perform discovery (step 2)
    openid_request = openid.begin(params[:openid])
    
    return_to = url_for :action => :complete, :only_path => false
    realm     = root_url

    # Send the user to his provider (step 3)
    redirect_to openid_request.redirect_url(realm, return_to)
    
  rescue OpenID::OpenIDError
    # Step 2 (Discovery) failed
    render 'form'
  end

Please keep in mind that this is a really stripped down no-nonsense version of the code that only serves to illustrate the concept simply. I’ll point you to correct (and more complex) code later.

As you can see we redirect the user to an URL that comes straight out of the discovery process. We pass along a realm and a return_to-url, so that when the user has finished talking to his provider, he will come back and call our #complete action.

I am obviously skipping over a lot of detail, including but not limiting myself to where you get the 'openid' instance from. Now there’s something uninteresting I don’t want to talk about! As I said, all the useful stuff will be in the end- and footnotes, as usual¹.

Back to our main narrative: The user finishes authenticating with his OpenID provider and comes back to return_to – meaning our #complete action:


  def complete
    this_url          = url_for :action => 'complete', :only_path => false
    this_urls_params  = request.query_parameters
    
    # Verify the OpenID security answer with what we think it should be (step 4)
    openid_response = openid.complete(this_urls_params, this_url)
    
    if openid_response.status == OpenID::Consumer::SUCCESS
      # All right: The guy is who he says he is.
    else
      # Nope.
    end
  end

We pass the #complete method the url we’re supposed to be at and the query parameters the way Rails sees them before mixing in ‘:action’ and ‘:controller’. This will call back to the OpenID provider once more to check if everything is as it should be.

Once verification of query parameters, of cryptographic hashes and subliminal messages succeeds – we can let the user in. Note that verifying an OpenID is like verifying an email address. We don’t know anything about the users identity yet, only about his association to the OpenID. It turns out that this is often enough. This is standard practice with email after all.

More information

can be found in the sample Rails application provided in the examples subdirectory of the ruby-openid gem. Have a look at the Consumers Controller. Some of that code really is more complex than it needs to be – keep in mind that it just handles a lot of cases in the same bit of code. Most applications wont need to copy that complexity.

The code of ruby-openid is also well annotated. It reads a bit like the Python it is ported from, but apart from that is really ok. Go have a look!

In parting

a few last words: I hope to have simplified OpenID and the use of ‘ruby-openid’. I would be interested to hear your comments and thoughts - leave them below. Also. And most importantly. If you have any really good recipe for chicken soup, I’d be most happy to borrow it from you.

¹ The #openid method should be implemented about the same way the #consumer method in the rails example of ruby-openid is.

Some tricks that can be learnt from the unicorn

2009-11-18T00:00:00+01:00

My general disgust with threaded/evented programming has led me to read up on the good old unix techniques used in unicorn. What a good read it was. The mix of programmers that have constructed unicorns have very different styles. That makes reading fun. I guess anyone could distinguish Zed Shaw’s style from the rest of the pack…

But there are also some unix techniques that I’ve learnt from unicorn and that I will share with you now¹.

The Self Pipe Trick

This trick allows you to handle signals safely in your code. A nice side-effect is that it will also allow you to trigger running of a process without using signalling.

Assume you have two processes, A and B. One process wants to be able to signal the other, awaking it from sleep. Lets assume that they share a FD to a pipe, where read_end is the end of the pipe that is open for reading and write_end the one open for writing to.


  # Process A: Sleep and wait for a signal: 
  IO.select([read_end])

The above code snippet is used to put the process A to sleep, waiting for a signal¹. You would then use the following technique to wake the process up:


  # Process B: Awake process A
  write_end.write_nonblock('.')   # the '.' is arbitrary

Process A wakes up because you’ve just written something to the FD that its waiting on³.

Note that this technique can also be used to wake process A up from a signal handler. So you get safe IPC in just a few lines.

This trick seems to be well known in the unix world, there are even some alternatives (read that link).

Keepalive through an unlinked file

Another thing that I found most enlightening is how unicorn handles sending heartbeats to the controlling process from its workers. This is not as easy as it looks, since actively sending heartbeats will require the master to listen at the same moment. The solution unicorn uses decouples the master from the client in that respect.

The master opens one temporary file per worker and unlinks it right away⁴. The open file handle to that file is then shared with all the worker processes.

Each worker will then do the following to prove its alive and kicking:


  tmpfile.chmod(m = 0 == m ? 1 : 0)

The chmod will use either ‘1’ or ‘0’ each time this gets executed.⁵ (Assign ‘0’ to the local variable ‘m’, compare it to ‘0’. If it matches (m == 0), m = 1, otherwise m = 0. A one line toggle.)

Here’s what the master does to check the worker’s status⁶:


  stat = worker.tmpfile.stat
  unless (diff = (Time.now - stat.ctime)) <= timeout
    # Worker has gone stale and needs to be respawned
  end

Even though the file is unlinked it still has metadata associated with it. Using the file’s #ctime we can check when the file was last mode changed, and thus when the worker was last alive.

This nice gem of a technique uses the unix file system as an external data store. I really like the details that went into this, like the fact that its unlinked already or the small code hack that is explained in the parentheses above. I haven’t been able to find that technique elsewhere, though I am sure it is part of any bearded unix guru’s toolbox.

¹ All references to unicorn code are made against revision 6311cc30b96b08f72fe00a9a0d1e291cf9923e52 of the repository @ git://git.bogomips.org/unicorn.git.

² unicorn.rb, method #master_sleep, lines 422-429

³ unicorn.rb, method #awaken_master, lines 431-438

⁴ util.rb, Util.tmpio, lines 75-86

⁵ unicorn.rb, method #worker_loop, lines 631-639, be sure to read the interesting comment right above.

⁶ unicorn.rb, #murder_lazy_workers, lines 517-527

One-Way RPC using AMQP as transport and THRIFT for communication

2009-11-05T00:00:00+01:00

In this article I would like to present ‘thrift_amqp_transport’, a small library I’ve worked on in the past months. It allows you to do one way rpc (method calls) using AMQP as a transport layer. This will decouple the sender from the receiver completely. I will present some usage examples in the second part of this article.

What is AMQP? Igvita.com has recently featured a nice overview article about AMQP and Ruby. AMQP is a protocol for asynchronous messaging/ queuing over the network.

Thrift (also on igvita.com) is an RPC framework developed by Facebook (formerly) that got made an Apache project. It allows you to do cross language/ cross platform method calls. I am impressed by the clean design that went into this. The intermediary definition language that is used is surprisingly rich, allows you to aggregate data into types and to transmit hashes and lists.

A very basic example of AMQP usage

Your rails application wants to create directories on demand. Using AMQP, you can queue up new directories to be created like this:


  require 'bunny'

  Bunny.start do |connection|
    work_queue = connection.queue('work')

    work_queue.publish('create_directory foo')
  end

A small sample worker might look like this:


  require 'bunny'

  Bunny.start do |connection|
    work_queue = connection.queue('work')

    work = work_queue.pop

    unless work.payload == :queue_empty
      command, *args = work.payload.split

      # Execute the command 
      case command
        when 'create_directory'
          puts "Create directory #{args.first}"
      end
    else
      puts "No work to do."
    end
  end

As you can see, any non-trivial use will implement some form of RPC-mechanism. (even this trivial example does) As long as you’re not transmitting only data, but want to act on that data, you will implement some kind of dispatch mechanism on the other end.

RPC instead of strings

To avoid writing a complex dispatch mechanism, it would probably be wise to use some kind of RPC implementation to reduce complexity at this point. Using ‘thrift_amqp_transport’, the above code would look like this:


  require 'thrift_amqp_transport'
  
  # Yes, its more setup code.
  connection = Thrift::AMQP::Connection.start
  transport = connection.client_transport('work_rpc')
  protocol = Thrift::BinaryProtocol.new(transport)
  client = MyService::Client.new(protocol)

  client.createDirectory('foo')

This layers thrift binary messages atop the ‘work_rpc’ queue on your AMQP host. To your AMQP server, this looks like just another message. To receive work and do something, you would write this code:


  require 'thrift_amqp_transport'
  
  class MyServiceHandler
    def createDirectory(name)
      puts "Creating #{name}"
    end
  end
  
  connection = Thrift::AMQP::Connection.start
  handler = MyServiceHandler.new()
  processor = MyService::Processor.new(handler)
  server_transport = connection.server_transport('work_rpc')

  server = Thrift::SimpleServer.new(processor, transport)
  server.serve    # never returns

As you can see, dispatching has completely gone from the code. You can now extend the service without rewriting any dispatcher/parser code, using thrift’s rich set of intermediate types in your method calls.

You might be wondering where all those new classes (MyService::Processor, MyService::Client, ...) suddenly come from. Those are generated by thrift, starting from the service definition, which would look like this in our case:


  service MyService {
    oneway void createDirectory(1:string name);
  }

Some possible past&future uses of this

I am using this for queuing up work and acting on it like you can see in the example above. This is probably more complex (setup-wise) than other solutions, but allows me to restart workers at any time, since the work is stored on the queue in a persistent manner.

But of course, with the power of thrift and AMQP under the hood, some other uses become possible:

Round Robin Load Balancing

The library might offer the possibility to acknowledge a message once the work is done. This would allow you to push work on the queue and get some guarantees that its really becoming acted upon. If a worker pops a work item and doesn’t acknowledge it, most AMQP implementations should redeliver the work item at one of the next pops. (to the next worker)

Service Discovery (one to many broadcast)

With the current state of the code, you could already have a service broadcast its presence on the network, updating it with heartbeats.

Two-Way Messaging

I would be interested in extending this library to allow two-way messaging.

Status & Limitations

The library is under active development and will be extended in the future. Some of the above uses are not yet possible, patches are welcome.

You can get the source code here.

On sharing your models and still keep the database a secret

2009-10-07T00:00:00+02:00

Your rails application is useful to someone. Now he wants to do more, depend on the data it handles and use it in other parts of his business. The simplest thing to do would be to give him access to your database and just let him have the data, right?

As it turns out: not really. While this is a good short term solution, it will certainly bite you in the long run. Not everyone accessing your data does so in a way that preserves your invariants and integrity constraints. And even though I think that ActiveRecord has found a sweet spot in specifying business rules you need to realize that it wont tolerate other gods beside it. Constraints and higher level rules specified in your Ruby code will only be maintained if you access the data through Ruby code.

The solution to this is obvious: Don’t share your database, share your access layer. Every access to the database has to go through your models. And those will be the keeper of invariants, the maintainer of .. you get my drift.

Here’s the short list of options you have to go about this:

Share model code

This is the first idea that you will have and probably not the best. Somehow bad ideas always come up first.

Following this idea, you would share your model code between various projects (using git submodules or the like) and access the database from all projects, going through the model code.

This has a number of downsides:

Your model code will change over time. That generates deployment issues: How will you update your projects all at the same time? If you don’t, you are back at square one.

Worse: Your database will need to change. But if you do that, all your model code (or the part that is not generated anyway) will possibly become invalid. So doing a migration means updating all your code at the same time. So you stop the world and do the update…

Another minor issue, if you are doing BDD/TDD like me: Where do you keep the tests for your model code? In the project that uses the code, because that project depends on the features that the test demands? Or in your model code project, since the code is there?

If I were you, I would not go down that road.

Distribute the data

Enterprise Rails does mention a few ways to distribute your data to the application that needs the data. The idea is to create a books database (and a books model) that is then used by the ‘book processing system’ locally. Why share when you can own it?

Because sharing is really what makes applications interesting to whomever uses it. And sharing things in a good way really is what application architecture is about.

So data distribution doesn’t work, since there will always be some overlap in data access.

Provide model access as a service

The remaining solution then is: Provide access to your model layer through small services that you publish. Access the model through them and only that way.

When the model changes, the service can stay the same. This solves the deployment issues I’ve shown with the first solution. And because it decouples the consumer of the service from the data, essentially where the data is located doesn’t matter anymore.

Your service will provide an interface for data access. This will hide complexity, allow you to provide abstraction. Its just a good idea™.

And to end on a practical note

I am getting a lot of mileage out of thrift these days. Being able to create service points and not limit myself to one particular language at the same time just has this innate beauty to the language geek in me.

Thrift lets you create services that look like instances of objects on both sides. It will handle marshalling and unmarshalling for you. The internal architecture of thrift really convinces me. It is hackable / more on that later on.

I am really not convinced of REST/SOAP/XML RPC. I don’t want to clutter my controllers with XML generating code. And even though I am using REST style controllers these days (as a way of structuring my application), I don’t want those be an API at the same time. I am really afraid of ending up with the ‘Big Generic Interface To Everything™’ and not being able to change it anymore when I need to. It ties things together that want to be free!

Thrift is small and neat. I can create a service within the space of an hour but still am able to do so explicitly.

A useful addition to your spec_helper.rb

2009-09-29T00:00:00+02:00

Here’s a little something that is really useful to put at the end of your spec/spec_helper.rb:


  # Run away with this. However, if it causes a nightly bug hunt or two, don't
  # blame me. Please. 
  class EscapedOut
    def initialize(old_io)
      @old_io = old_io
    end
    def write(str)
      @old_io.write str.gsub(/</, '&lt;')
    end
  end
  $stdout = EscapedOut.new($stdout)

Afterwards, to quickly debug your specs in TextMate, you just enter


  it "some specification" do
    p some_object  # temporary debug code
    
    # some test here that fails for unknown reasons
  end

and it will just work, outputting the string you’d expect and not a very short line starting with ‘#’ and then nothing. The trick is that the string that is originally output (something like ‘#<Foo:0x128a964>’) is converted into #&lt;Foo:0x128a964> which will output just fine.

You’re welcome.

Joel, duct tape is just one of the tools we have.

2009-09-24T00:00:00+02:00

I used to really like what you write ‘on software’ – now I start really hating it. But that is more useful, because you always get me to think.

The guy with the duct tape – I would hate his guts if he was on my team.¹ The little stunt with prev and next pointers of a linked list rolled into 32bits using XOR – nice. But very immature. If that guy goes missing, you can basically scrap his code and start again. That is not very much better than using multiple inheritance² and multithreaded COM.

You are missing a point. The point is not to deliver software, at least not only, but to deliver working software. There are really just two possible outcomes: Either the thing you’re creating works half-way, or it doesn’t.³ If it works half-way, you (or someone else) will have to work on it for the whole of its lifetime (ambiguity intended). And the customer doesn’t want old bugs cropping up again and again. And he wont listen to you if you talk about duct tape. He doesn’t care.

Duct tape is just one of the tools we have. And as such, it needs to be applied where appropriate. Anyone claiming one tool to be vastly better than all the others is just wrong, whatever the tool. Your smart boy should learn to write code that others can read and maintain. Then he’s got a real future in software development.

I think that every talented programmer starts out as duct-tape guy. And then some of the things he tapes together come crashing down on him (or on the customer). And then he learns. But the dichotomy you’re proposing is wrong, its not either duct-tape or over-engineering. IMHO, there is a lot of middle ground. Just as an example (one is enough) – TDD/BDD practice is to ‘do the simplest thing that can possibly work’. This one sentence has saved me years of mental effort that would ultimately have been invisible to the customer.

The real work is to write working software. Over-engineering and building a second system is just (busy-work) work evasion. But the duct tape approach is evading work as well, by being lazy in the sort of way clever people are.

What I’ve been wondering. Are you doing Jamie Zawinski any kind of justice with your article?

¹ ‘Hate his guts’ is just more colorful than ‘take issues with’ – No, I wouldn’t hate anyones guts. That is very rare.

² I guess that is just a typo.

³ About one third of IT projects get cancelled before completing. The other two thirds sometimes overrun their budget by as much as twice the original estimate. (ie: ‘The Standish Group Report’, The Standish Group, 1995. pdf) We can’t pretend this isn’t happening and we need to feel responsible for the state of the industry.

How to prevent class unloading in Ruby on Rails

2009-09-16T00:00:00+02:00

Let’s assume you have a class that holds a connection to an external system, and you don’t want Rails auto(un)loading to mess with that. The advice the rails documentation gives you in this case is simple and correct: Just don’t let rails autoload your class, instead, require it yourself.


  require 'my_persistent_class'

There is one gotcha to this, and a gotcha that conflicts with the natural ordering tendency that every programmer should have. Let’s say you’re extending a rails model with FeatureX. It will come natural to you to name that module RailsModel::FeatureX. This has consequences.

There is no real way to ‘load’ that RailsModel by using a require – when would you do it? During initialization process, this fails because rails obviously isn’t in place yet. And later on – rails will be in place and the RailsModel class will already be in the fangs of autoloading.

And the next time rails autounloads/ autoloads classes, RailsModel – and with it FeatureX – will be reloaded as well. Darn. The solution to this is really simple, but it took me a while to figure this out: Just don’t include the FeatureX class in the namespace of RailsModel. Then doing this in your environment.rb will ensure that your FeatureX isn’t unloaded:


  config.after_initialize do
    require 'feature_x'
    
    FeatureX.set_some_persistent_attribute :foo => :bar
  end

Btw: Global variables are really evil.

Code Style issues with Real World Haskell

2009-09-14T00:00:00+02:00

I am reading the book “Real World Haskell”¹ – and enjoying it to my surprise. What I find really hard to understand is why the authors chose a code style this opaque. My message to you: Code really is read more than it is written. Here’s an example straight from the book:


  liftP2 q f g w x y z = f w x y z `q` g w x y z

That’s a series of 18 lower case letters in a row, lightened up by a few cryptic line noise like dots and dashes. My growing fondness of the language might come to an abrupt end if I wake up to a community that writes this kind of code…

But to end this post on an upbeat note, I really would like to congratulate the authors of the book. Real World Haskell really is up there with the programming language introductions that make you want to pick up the tool and rewrite an operating system in it. It reads nicely from the start to the end, and if you manage to skimp/skip/plow through the examples like the one I gave above, you’ll find a lot of nice code as well. Worth reading!

¹ Real World Haskell (Bryan O’Sullivan, John Goerzen, Don Stewart)

Perceived Advantages of Writing the Specification first

2009-09-07T00:00:00+02:00

I will try to summarize some of the perceived advantages of writing specifications first. The word ‘perceived’ indicates that I don’t mean to say that the ideas presented here are hard facts. Rather, they are based on the anecdotal evidence of my senses and my experience. I’ll talk about

How Specs are documentation (this post)
How testable code has clean interfaces
How testing forces you to think

Specs are the documentation

If you apply some sort of quality control on your specifications, they will be readable. They should also reflect what you or (to a varying degree) your customer expect from the code. That said: Look at your specs as the only up to date documentation for the code.

Having those specs makes your situation a lot better than having only the code. While you can’t refactor code and keep it at the same time (for documentation purposes on what you already have), you can refactor code and keep the specifications.

Specs encode all the choices you’ve made during writing code. People that don’t write specs have also made these same choices. But only mentally. And then forgotten about it before lunch. It doesn’t feel differently, the code commits might even be exactly the same. Except, some time later, you (the guy who does BDD) will still have a record of what you thought of (or didn’t) when writing the code.

I had to maintain code that looked like this once:


public int someFunction(int p1, int p2) {
  // Some code 
  // Some more code
  
  if (p2 == 23) { // (1)
    return -1;
  }
  
  // Other code
}

The above snippet is anonymized to protect the innocent. The comment at (1) said something like ‘this is a special case’ or ’I’ve had to introduce this, otherwise it wouldn’t work’. While I appreciate the comment, it doesn’t help at all.

My task was to clean up this piece of code and to rewrite some of it. As it happens, no tests/specs were available, making that thin comment everything I could hang myself on. In situations like these, you have two choices: Delete the whole exception and wait for things to break or leave it in and hope someone else breaks it later. Either way, this is a liability that waits to bite you the next time around. Having to remember a functions with a positive image may return a special negative value that indicates mystery going on makes reasoning about the system a complex task.

It would have been an altogether different matter if I had had specifications like these:


describe "someFunction" do
  it "should return -1 (exceptionally) when flab hits the fob" do
    obj.someFunction(1, 2).should == -1
  end
end

In that case, I might have left the code as it was. There’s a clear reason for why this ugly snippet has come to be. And I even have a choice: I let it stand as it is (for working code can’t be all bad) or I rip it out with some confidence in what I am doing.

There’s a guy who wants to fly to the moon without the fuel to get back. My job isn’t like that. I want to be able to go back and revise my choices. That’s why I write them down.

Confession: I am somewhat of a tester

2009-09-02T00:00:00+02:00

I can’t deny it. After only minutes of writing real code I long for the snug comfort of well written behavioural specifications (also known as tests).

Writing specifications first

Any yes, to really get your coder level up, you will need to write specifications before you write the code. This sounds terribly hard and at first it is.

Why is it hard? Because you need to know what you are going to write before you do. But that’s another blog post there.

Frequently seen stereotypes (FSS)

A few stereotypes and my answers:

But .. some things can’t be tested

No. You are not asking the right question. It’s not: “can it be tested automatically?”, its “how much does it cost?”, weighted against the benefit of having tests. That discussion tends to be sharper around the edges.

Also, you might want to consider stubbing and mocking, two – no, really just one – essential tool(s) in any developers toolbox. Then you should look at mockingobjects.com and a few other sites and practice daily.

To be fair: Some things can be pretty hard to test. Be stubborn. If you can show how it pays off to have a test, you can test it. Take a look at the consumer product industry: A simple childs toy is tested more thoroughly than the average piece of code. If all else fails, use a robotic arm that pounds your keyboard like a wild monkey, looking for wear and inflammability.

Its all just a waste of time, I can write my code just right

That probably used to be the case. Or even is the case. It’s just .. I hear this a lot from people who mostly work alone. And on small(ish) projects.

Working in teams is another thing altogether. You will not be able to keep all details in your mind like you used to. Plus, people break stuff. Mostly by making something else better. You need something to tie these forces together, making them visible.

Oh and… if you still think you can write code without tests just fine – go and ask the next guy how much time he spends reading your code before he can modify it. That is: modify it and still be sure that nothing breaks.

To the what and _why of opening a new blog

2009-09-01T00:00:00+02:00

This is a welcome to you, dear potential reader. Again, I take the jump – opening a blog, blossoming like a flower come spring, trusting a new friend, write a new specification.

Or like Hermann Hesse said:

Wie jede Blüte welkt und jede Jugend dem Alter weicht, blüht jede Lebensstufe, blüht jede Weisheit auch und jede Tugend zu ihrer Zeit und darf nicht ewig dauern. Es muss das Herz bei jedem Lebensrufe bereit zum Abschied sein und Neubeginne, um sich in Tapferkeit und ohne Trauern in and’re, neue Bindungen zu geben. Und jedem Anfang wohnt ein Zauber inne, der uns beschützt und der uns hilft zu leben. Wir sollen heiter Raum um Raum durchschreiten, an keinem wie an einer Heimat hängen, der Weltgeist will nicht fesseln uns und engen, er will uns Stuf’ um Stufe heben, weiten! Kaum sind wir heimisch einem Lebenskreise und traulich eingewohnt, so droht Erschlaffen! Nur wer bereit zu Aufbruch ist und Reise, mag lähmender Gewohnheit sich entraffen. Es wird vielleicht auch noch die Todesstunde uns neuen Räumen jung entgegen senden: des Lebens Ruf an uns wird niemals enden. Wohlan denn, Herz, nimm Abschied und gesunde!

And because this is really going to be a blog about programming, here’s a code snippet:


00000000001111111111222222222233333333334444444444555555555566666666666777777777788888888
# This is totally unrelated to anything, but better than foo and bar.
# Voila. I give you: The identity line filter. 
class LineFilter
  def filter(line)
    line
  end
end

Oh and .. like you might have figured out by now: This is a test entry that only serves to populate an otherwise empty design. Like those really shiny new buildings where no one has lived – beautiful but empty, a new shirt.

press play on tape

HTTP... something

http (2011), nestful (2010), righthttp (2010), simplehttp (2007), http_request.rb (2009), plain_http (2011)

Typhoeus (2009)

httpadapter (2010)

http_connection (2009)

httpclient (2007)

httparty (2008)

net-https-wrapper (2009)

http_spew (2011)

fast_http (2010)

net-http-persistent (2010), persistent_http (2010)

net-http-pipeline (2010)

HTTPal (2007)

httpi (2010)

Conclusion

Write another one?

And a parting note

cod is a fiction

Cod is really a fiction

Specifics

Environment

A seed for more

Introducing the floor_manager plugin

floor_manager

Factory floors

Simple fixtures: Singletons

Generators

Associations

Setup, Teardown: Caveats

Works With

Why you should use floor manager

Parslet Tree Mangling and Transformations

Output Merging Algorithm (Handwaving Version)

Why the “Duplicate subtrees” Message?

How to fix this

And Transformation

Wrapping it up

Introducing Cod

Building on small elements

Channels in channels

Bleeding edge

Patterns of distributed computing

Parslet and its friends

Conclusions

Parslet vs…

These are exciting times

Stack Traces in Parslet

Why the compilation phase?

Limitations

Not Stack Traces, Error Trees

And beyond

Procrastinate away!

Status, When is this ready?

Summary

Projects, Projects, Projects...

parslet

picky

Apple Mail woes

Merged upstream to jekyll

last_puppetrun - A small tool for your puppets

Not the same as puppetlast

How to get your own puppet show

Pick a puppet

Install the puppet master

Create an initial configuration

Install puppet on the clients

Parting Words

New Stuff in rspec - a Cheatsheet

Migrating Request Trackers backend

Preparation

Migration

Correct the sequences

Done

ZFS backups and other tidbits

For each (interval, count) tuple…

And how to synthesize the result

Conclusion

A few Ruby Anti-Patterns

rescue everything

Not the same as `puppetlast`

For each `(interval, count)` tuple…