John P Wood discussed some issues they faced with will_paginate and couchrest during the migration of TextMe to CouchDB, but l left out some code to work with. Couchrest itself had some pagination support that got pulled to some extent… This left me wanting, and wondering, since it was my turn to walk down this path.

CouchDB is a different beast, its aggressive use of indexes means that occasionally you loose some functionality that you’ve been accustomed to having in other persistence mechanisms, like the number of rows matching a query. Jan Lenhardt explains on the CouchDB issue tracker in more detail, but it boils down that you need a reduce function to calculate the number of rows. Sounds difficult? Not at all!

In my case I had a collection of announcements to deal with, and the announcement archives is a paginating collection of documents. Standard will_paginate stuff, nothing special.

Those reading in a reader would want to click through to the post to view the embedded gists further down, or view them directly at Github.

Below is a condensed version of the model from our paginating system:

I’ve included only one view and a corresponding class method, as it is enough to proof the principle. Lets dissect.

The map/reduce functions are extremely simplistic, they simply emit the announcer and the date the announcement was created. This allows for easy scoping and ordering of the announcements. The reduce simply counts our returned records. The magic is in the class method that sets up our WillPaginate::Collection with data from our views.

Line 30 creates a new WillPaginate::Collection instance, passing it the page number and total per page as parameters, it gives us back a pager that we can manipulate.

Lines 31 through 38 uses couchrest’s pagination support to pull out data from our view. The most important things to note here are that the page and per_page options are sent to the paginator and we skip the reduce step.

Once we have our records loaded, we ‘replace’  the pager’s collection with our results from our view (line 40).

The final step is to determine the total number of documents available to us, and for this we need the reduce function. On line 42 we call the same view, with the same arguments, except for requiring the reduce step to happen. We use the results (lines 43 to 47) to inform the pager (will_paginate’s pager) how many rows there are in total.

The controller and the views might look something like this:

It worked, and it shows that we don’t loose as much as we might think when moving away from ActiveRecord and the ton of plugins surrounding it.

Thanks to John for documenting the migration of TextMe, just knowing that it was possible to combine couchrest & will_paginate gave me the push I needed to figure this out.

Disclaimer: This code is extracted “as is” from a real life system and might contain idioms/phrases, and even code, that doesn’t make 100% when viewed in a gist. Please wear your thinking cap when applying this lesson to your own projects.

I’ve been lurking in the CouchDB world for quite some time and have spent a lot of time wrestling with how to loosely relate documents in CouchDB to each other. A big part of learning to use CouchDB successfully is to break away from the shackles of the relational world. Relationships between documents is one such a shackle that seems hard to break.

It is unavoidable that data has to be correlated, and I wanted to rethink how to do it. After plenty of discussions in #ruote with John Mettraux we came up with a model based on how the web works. Since CouchDB is “off the web”, the approach feels quite fitting to me and hopefully to you too.

First some insight into my thinking at this stage.

The web has been successful in loosely expressing relationships between documents. Take two examples:

and

For those of you reading this through a reader, click through to see the gist’s above.

Simple as it seems, in both cases we have a document that is somehow related to the page. The nature of the relationship is expressed via the rel attribute, and the target specified via the href attribute. This got me thinking. Since CouchDB is made off the web, can’t these same principles be applied?

Yes, they can. And here is how:

Currently you might be tempted to express relationships link this in your JSON:

Where changing it to this holds the key:

If anything this format, albeit more verbose, expresses the relationships more clearly and in a format that is web friendly. We’ve broken the shackles of relational thinking.

Enter Correlate

Correlate is an experiment in this line of thinking. It is a mixin for CouchRest’s extended documents that allows you to express these relationships:

Correlate generates getter and setter methods for working with your relationships and lot more (review the README). It also includes a compatibility layer for ActiveRecord to help when you’re migrating from ActiveRecord to CouchRest or building a system on CouchDB that needs to access legacy data via ActiveRecord.

The project is still pretty much a moving target, but I’d love to hear how others address the same issues. Correlate does a great job at maintaining relationship information in a web friendly manner and providing you with some convenience around the verbose data structure. Correlate also has a lot of room for improvement, but that will hopefully change over the coming days as I continue integrating it deeply into my existing projects.

Please fork the project on github and join the experiment with me.

Having screwed up my kernel configs for my VirtualBox Gentoo image, I needed a serial console to catch the boot messages scrolling past in order to see if all the required hardware was being loaded by the kernel. I’ve never done this on a physical machine before but I am converted now and will acquire a USB to serial port converter in the near future…

Using this article as a base you need to do the following:

1. Enable serial ports for your virtual machine
2. Select “Host Pipe”
3. Enter /tmp/vboxconsole as the filename
4. Use netcat to read the console: nc -U /tmp/vboxconsole

When booting you need to amend your grub boot line to have the following at the end:

console=ttyS0,38400

Making it look something like this:

kernel=/kernel-2.6.30-r6 root=/dev/sda3 console=ttyS0,38400

Proceed to boot and look at netcat to see the entire boot output scroll past without disappearing into thin air when the kernel panics.

Man, I love virtualization. I tested this on Mac OS X 10.5 with VirtualBox 3.0.6, but it should work on any *nix platform. Some more Gentoo serial console madness can be found on the old Gentoo Wiki.

The worst part of the process was getting the specs to run with rake spec. Using ./script/spec it worked on individual specs and on all the specs worked as advertised, but rake spec didn’t do anything.

After a lot of time spent in the debugger I wasn’t any wiser. The only difference was that in Ruby 1.8 the example groups were fully loaded, and in Ruby 1.9 they were empty. I gave up and started searching relentlessly for some information on the issue. I couldn’t find anything, until I found an indirect solution on the rspec wiki. It seems that if you have any versions of the test-unit gem after the 1.2.3 release installed, your Rails specs will simply not run. For me, removing test-unit 2.0.3 made the difference and the specs ran properly. PowerDNS on Rails has now joined the ranks of my Ruby 1.9 compatible projects.

Ruote in 20 minutes from Kenneth Kalmer on Vimeo.

The original post of the presentation is over here.

After I read this post on Labnotes I decided to “take the plunge” into the world of Ruby 1.9 properly. I mean, I was at RubyKaigi where the biggest lesson of all was “Change!” (and changing to 1.9). Ruby 1.9.1 has been stable for quite some time now, and 1.9.2 is due out on Christmas. And we’re still stuck on 1.8.6 ? WTF

Getting started with Ruby 1.9 is easy (only time will tell how much pain is caused). I have to admit I’m not as brave to wipe my rubygems installation, I might still need 1.8.6 (and I seriously hope I don’t).

So Relevance made this awesome bash script called ruby_switcher.sh that you absolutely have to try. Once you have it installed and working as per their instructions you need to make one small change to the very last line of ruby_switcher.sh. That change looks like this:

#use_leopard_ruby
use_ruby_191

Simple and I can easily toggle 1.8.6 mode if needed.

Yes, I did.

So over the coming days I’ll be working through all 28 of my github repos and making sure they run on 1.9. This is no small task, but I’m looking forward to getting my “house” in order.

At this stage I know acts_as_audited is 1.9 compatible, I fixed it earlier the week while testing ruby_switcher. I had a look at ActiveRecord::Tableless and decided to pull the plug on the project completely (see here).

I’m really looking forward to this adventure and sharing the ups and downs of the road with everyone. Ultimately I would love it if we can get the South Africa Ruby ecosystem on 1.9.X ASAP.

Follow my project updates over on my Ruby 1.9 Progress Report

I wrote daemon-kit to solve two big issues with writing daemon processes in Ruby:

1. Everyone is re-inventing the daemons gem
2. Individual daemons share a lot common code, apart from the daemonizing bits

As for #1, daemon-kit at first wrapped the daemons gem, and later ripped it out completely as it was difficult to wrap up the worst of the daemons gem properly.

#2 seemed to be a twofold problem that daemon-kit has also addressed with great success. The first was addressing the all too common issues of logging, pid file management, umasks, signal traps, config files, exception handling and other “low-level” issues. Everyone was implementing these things to a limited extent in their daemons and this could cause a lot of frustration when done wrong. The second part was the need, even if undiscovered, for higher level re-use and development speed. Daemon-kit addresses this with some limited generators, making it easy to get going with a cron-style daemon, AMQP & XMPP bots as well as the newest addition, Ruote remote participants.

All this is fine and well, and people seem to like the project. The mailing list is getting some noise, we’re over 200 watchers strong on Github and the IRC channel has some folks popping in to say hi.

However, daemon-kit quickly made me lazy and realize there is a couple of things it can do much better. If you’ve used a generator before, you’ll notice the generated code is very much stuck to that type of daemon. Changing from XMPP to AMQP (for illustrative purposes) would be best accomplished by generating a new project and copying over the lib/ folder only. This sucks.

Another problem is people, myself included, would like to have Sinatra-style daemons (i.e. one file) for smaller tasks. Currently this is not possible at all. Another thing I know people are doing behind my back is generating daemons inside Rails projects, which may or may not work, depending on whether you load Rails’ environment.rb.

So, what happens now?

My thoughts are to implement privilege dropping support and tag a 0.2 release. This gives us a feature complete framework, albeit not as good as it can be. I’ll maintain 0.2 as a stable while undertaking the rewrite. A rewrite? Read on.

I posted to the daemon-kit list a suggestion for stackable daemon environments. I’ve discussed this in IRC with a few folks as well. Jordan Ritter gave me an exceptional breakdown of the dangers of doing something because it is “neat” and I’ve taken his warning to heart. However, I cannot seem to argue against stackable daemon environments, it sounds too good.

The idea is pretty simple. Stack-entries can be compared to Rack applications, with two significant differences. The first is that they will be called only once. They have the opportunity to change the environment in which the final code runs. The second is that they can be called at four different stages of the daemon lifecycle: argument processing, pre-daemonization, post-daemonization and shutdown. This differs from Rack’s single call() method.

The stackable nature also gives the stack members easier ways to set conventions and can dramatically minimize configuration. It paves the way for plugins, sinatra-style daemons, rails-based daemons and easier packaging of distributable daemons, and so much more. Looking at the internals of daemon-kit, it would greatly help simplify the existing code as well as help separate utility classes from stack members.

The more I think about it, the more obvious this becomes, and the more possibilities unfold for the framework. This will definitely make daemon-kit a force to be reckoned with, and hopefully I can persuade other library developers to offload their daemonizing code to daemon-kit, just like rack developers offload their HTTP handling to rack.

Undoubtedly my worst time on stage ever! Ubuntu gave me issues with the projector, so I actually had to leave the stage to go to another conference hall to get the setup working again and run back up to the main hall… Once there, I was a mumbling mess, but made it through without the buzzer beating me.

Presenting a lightning talk at a Japanese conference is surely something else, the audience expects fireworks and the presenters deliver. It was truly an amazing experience, humilation and all, and I’m looking forward to the last day of the conference.

But enough blabbering, the ISP in a Box platform is huge, and interacts with various other service providers through a myriad of API’s, some of them as reliable as convicted mobster. Our support teams are constantly double checking the interactions between the autonomous participants and our service providers, and this means a lot of copying and pasting, from our interfaces to theirs.

They simply “double click” to select text in our interface, causing the browser to copy additional whitespace with the selected text. When pasting into a search box they need to prune the additional whitespace off manually, making the process even slower and more tedious than it needs to be. I can’t replicate this with Firefox on Ubuntu, but it happens to the Windows guys…

In the previous version of the system I solved the issue quickly and bluntly by simply placing all the information in text fields that were styled to be transparent. In this day and age of unobtrusiveness I thought it more appropriate to generate the text fields on the fly, since they’re a convenience and have no effect on the functionality of the system.

It took me 5 minutes to get this done in jQuery, and another 10 to figure out how to wrap it all up as a jQuery plugin. I love jQuery…

Ladies and gentleman, I present you with the jQuery copyfriendly plugin.

Usage

  $(document).ready(function(){
    $('.copyme').copyfriendly();
  });

And then mark your “copy friendly” HTML elements like this:

<table>
<tr>
<td class="copyme">Hello</td>
</tr>
</table>

Or inline like this:

<code><span class="copyme">Heya</span></code>

copyfriendly() takes on parameter currently, which is optional:

  copyfriendly( {
    klass: "borderless"
  } );

The ‘klass’ option tells copyfriendly which CSS class to apply to the inputs generated.

Sample CSS

You can make the inputs generated by copyfriendly invisible using this sample CSS:

  input.borderless {
    border: none;
  }

Do be warned that it might take a little more to align the appearance of the remaining text with the surrounding text in the document.

How it works

Actually quite simple. All it does is extract the text from the element and then replaces that text with an input tag of type “text”, and the value is the original text extract from the matching element. The size of the element is dynamically calculated from the string length, so there might be room for improvement here.

Do you externalize your business processes? Or, do you use a state machine or workflow engine?

John Mettraux posted a powerful read on the fact that state machines are not workflow engines, and I share the sentiment completely. I’m a vocal user of John’s ruote, a Ruby workflow engine, but I also use state machines, and I use them in tandem.

I choose early on that I need to externalize my business processes for our ISP in a Box product, to help break the project into manageable components, and this article shares how and why.

Limitations of state machines

I’m not giving any background on what state machines are, or any of the possible uses. I’m going to highlight some limitations of state machines from a business process perspective, especially state machines that hook into ActiveRecord (or database models).

Lack of context

Knowing in what state the object is isn’t always enough. For decision makers (autonomous or human), context plays a definitive part in making a business decision. If a service ends up in a “suspended” state, for arguments sake, decision makers need to know why. Was the service suspended due to bad payment, was it suspended due to abuse of the terms of use, or was it suspended because client relations had a bad day. Having multiple states to disambiguate the precariousness of the suspension is not cool.

Lack of history

Extending on the context topic above, the history of how an object got to a state might be just as important to a decision maker. It is also important for a reporter, or observer in the process. Business processes inherently have multiple paths, it is just never as simple as it seems. Placing an order for a book at Amazon, in theory, fires off business processes unimaginable to us mere mortals. Think about it for a second, they need to check stock, decide from which warehouse to dispense stock, optimum shipping combinations, manage re-orders, handle invalid shipping destinations, and so much more. Calling events on an order object just won’t cut it. Each decision maker in such a huge process needs to know why the work ended up at them. A shipping coordinator in New York might question why he has to ship a book to Seattle, but if he knows that Los Angeles is out of stock he doesn’t mind to do it. I know they’re fully automated, but play along for a second.

Open ended

Another thing about state machines is their open ended nature. For a lot of applications this is brilliant, but not for business processes. Business processes have a definite start and end, with whatever means needed to get from the one to the other. State machines can change from any state to another. This can however be controlled with guards, or setting up a limited transition table, but then the machine gets clunky and difficult to manage. It looses it’s appeal.

Versionless

State machines are versionless by nature, and business processes are constantly evolving and getting refactored. As decision makers gain more experience, business processes get leaner and meaner. Especially in times of financial uncertainty there is a big focus on driving down costs and cutting out unneeded, costly steps from business processes. In a state machine this doesn’t necessarily translate into less states, or less transitions, but it does make it difficult to determine if a process is still part of the old regime, or the new one.

State machines are good at expressing behaviors

And this should be leveraged by the developer, however, behaviors hardly come close to defining a business process as a whole. The behavior of an object can be seen as how it it behaves during a business process, and not the process itself. More on this later when I discuss combining state machines and workflow engines.

Don’t abandon state machines!

To proponents of state machines this post will definitely seem like heresy, and I do apologise. State machines are wonderful tools, and at the end of the article you’ll see how I do use them together with a workflow engine. My argument for the post is that state machines are not the right tool to drive business processes. Business processes are big and bulky and far more complex than what they appear on the surface. They need a suitable environment to run in, something stable to drive them, and a database model with a state machine is hardly the place.

Externalize your business processes

Externalizing your business processes, and running them inside a workflow engine is a big decision that has massive payoffs if you implement it correctly. If you do it wrong, you will most likely hate me for writing this article, and pretty much everyone in the BPM/WFE space. For me, the decision to externalize the process is busy paying off handsomely, albeit I’m still resolving a lot of unanticipated issues (ie scaling the workflows).

Enter ruote

ruote is a Ruby workflow engine, which parses process definitions into expression trees and executes them. Greek? Don’t worry. Process definitions, a ruote DSEL, are Ruby classes that definite the flow of your business process. They are made up off expressions (think methods), and each expression plays its part in the business process. This is akin to a flowchart, where you have a start and an end, or multiple ends to the process. Expressions can be decision points (diamonds), participants that perform work (blocks), looping constructs and plenty of other goodies. ruote is a business process operating system…

Workflow engines are business process operating systems

Bold statement, yes, but not a lie. Workflow engines need to be solid and reliable environments. We rely on Apache to reliably house our running application, or databases to reliably store our information, and the operating system to reliably run all of this. If the environment is stable, we can focus on our own code, without much care. The same holds true for our business processes. A workflow engine doesn’t have office hours, needs to be available at all times and running flawlessly. It also requires a lot of instruments that coordinate our efforts, to keep our business afloat. There is persistence, thread safety and concurrent access of work, schedulers and much more at play.

A part of me thinks that state machines are so attractive to us as Ruby developers because we don’t want to manage all the real work involved in running business processes. We like turning ORM’s into Thor’s hammer, so to speak.

Participation

As your business process is executing it will interact with various participants. Participants can be be fully autonomous, or people. The concepts of participants is foreign at first, because it doesn’t map to users of the system directly. To give you a rough idea of participants, consider a simple domain registration process. Our clients register a new domain in the system, which initiates a new instance of a defined business process. First we add the domain to our DNS servers, then we register the domain with a registrar, then we notify the client. Without automation we’ll have one participant: ActiveRecord participant. This participant simply saves workitems in a table in a database, and we can poll this table looking for work. Using the workitem’s payload we can determine whether the workitem indicates one of three unique pieces of work that needs to be performed:

1. DNS Admin to add domain to servers
2. DNS Admin to register domain with registrar
3. Client message when they log back in again (or view domain, or whatever)

With automation the participation changes to something like this:

History and context

Using the above example of automating processes, history and context becomes quite important when a workitem lands at the support personnel desk. They need to know where the error happened, what lead up to the error, who is involved in the process, and maybe something more. The ability to extract a JSON-formatted expression tree from the engine allows us use ruote-fluo to graph the process on a canvas tag for the support personnel, so they know exactly what happened prior to the error being reported. This knowledge of prior actions is invaluable when deciding on a course of action, which in business is the difference between closing or cancelling a deal.

Time sensitive & long running processes

Automated business processes can usually finish within a couple of minutes, or seconds, depending on their complexity and the number of autonomous participants involved. But business is never as simple as it seems, and time is usually the enemy. A business process might indicate that if a support call has been opened without an initial response for more than 4 hours it needs to be escalated. Others might have indicate that we don’t send SMS messages between 19:00 and 6:00, so we don’t wake our clients. This mixed bag of sleep and timeout expressions are difficult to get done on your own, and a workflow engine should (ruote does) support this out of the box.

Versioned processes

Process definitions in ruote are parsed upon launching the process. Ruote converts the process definition into an expression tree and works with the expression tree from that point forward. Process definitions can then be fine-tuned and altered and all new launches will use the updated definitions. Existing process instances stick to their expression tree in the engine, so they remain unaffected to the changes and will play out exactly as intended. But the expression trees are not concrete either, once in the engine they can be altered at runtime, but I still have to explore those abilities of ruote myself.

Going hybrid, using the best of both

I said earlier that I haven’t abandoned state machines, and that I rather use the two instruments together to accomplish what my system requires. This is in fact very simple to implement and works brilliantly.

ISP in a Box sells services, and these services need to acquired/provisioned for our clients to make use of them. All our models share, a mostly common, state machine. When services are provisioned for the first time, they enter their initial state called “pending”, and the provisioning process for said service is launched. The state machine has no idea what is going on at this stage, but ruote runs and happily executes the process. Each process is usually a mix of mostly autonomous participants and then some human intervention if the automation breaks. Other processes are purely human (like web design).

While the service is “pending”, now changes are allowed to the service. Instead, when the process completes in ruote there is a callback, a “webhook”, that calls back to Rails and activates the service. This “activation” is an event defined in the state machine and transitions from “pending” to “active”. Once a service is active (ie provisioned) the owner of the service is presented with a lot of new options. These options include changing passwords, upgrading & downgrading and even cancelling. Whenever a change is requested by the client we need to fire off another process, and while the process runs we place the service in an “integrating” state. While a service is “integrating” no changes are allowed, and when the process is done it “activates” the service again.

There are plenty of other states and transitions defined, including states for “suspended” and “deleted” (we don’t remove data from the database). Each event fires off a different process in ruote, leaving the service with only one decision to make: Which process to I launch under these circumstances?

This combination works extremely well for us since the state machines are very small and lean, and they do exactly what is expected of them: indicate state and handle/prevent transitions between said states. They are not burdened with making business decisions, the handle state. Simple, powerful, effective.

Wow! How do I get going?

Glad you like it, but be careful. As wonderful as the BPM/WFE world sounds, it is a serious leap to take. First off all make sure that your situation necessitates a workflow engine. I’ve pointed a lot of people to a simpler daemon-kit + AMQP combination for their needs when asking about ruote. In the majority of simple automated tasks a state machine with worker processes will suffice. ruote is all about business processes and automating those processes.

Implementing ruote also brings a lot of changes to the way you develop software. For one, ruote cannot run inside Rails like a normal plugin. It needs one instance to be running, not multiple ones, due to the internal scheduler. We’re trying to work out away around this, but it’s on the back burners for now. This means you’ll be deploying two applications side by side. You’ll probably also start moving some of your code into gems, so you can share it between ruote and your Rails project.

You’ll also need to implement a communication channel between your Rails instances and ruote, or use an existing engine “housing” like ruote-rest, or give me time to finish ruote-kit. Both ruote-rest and ruote-kit offers a RESTful interface to the engine, allowing all your projects in any language to leverage the power of ruote.

State machine versus workflow aside, a major downside to an externalized workflow is the distribution of architecture.  The golden rule for distribution is “don’t”, but that’s unrealistic, because large, complex systems often require it, and it pays off in the form of serious decoupling in these cases.  So if a workflow engine will be the first distributed step you make then you should think very carefully about what that will cost you.  Not to say don’t do it, but it’s not going to fit everywhere, sometimes the “process” is simple enough not to warrant a workflow engine this specialized.  There’s no silver bullet. – Nic Young, a recent reviewer of ruote.

But sometimes the distribution already has occurred because of a service oriented architecture. It is natural to wrap participants around services, or to use participants as facades to services. SOA is usually thought of as “in the bank”, but in our case the applications are already playing with services outside of the building. We had no choice, the distribution was already here. Ruote doesn’t know much about distribution, it just dispatches to participants and sometimes receives workitems back, whether it happend in process, or on the other side of the world.

Another thing potential pitfall is database access. Personally I use ActiveResource to suck data into my participants on a “need to know” basis instead of giving ruote database access and duplicating my models between two projects.

The biggest pitfall of all is learning to write these processes in a concise fashion. The engine and expression language is very very powerful, and you don’t realise at first how to chain things together. Even now, almost a year later, I find myself refining my process definitions to become leaner and cleaner and easier to maintain. The second biggest pitfall, testing business processes is a pain in the ass. I’m definitely making some progress on conceptualizing a testing methodology for process definitions, but it is still some way off.

Is it worth all this trouble?

Yes! As far as I know we might have the biggest implementation of ruote at the moment. I have roughly 70 process definitions in production, and about 20 different autonomous participants. As for human participants, it’s potentially over a thousand, though in ruote terms it is only one, and I map the payload of the workitem back to the user in our systems.

The next best thing that happened to us while implementing business process definitions was it gave us a chance to refactor the business as whole. We’re a small outfit, less than 20 people, so the excercise went well. In a large organization them impact of business process refactoring, or even just formalizing, is a massive undertaking and increases the number of stakeholders in the project dramatically, use more time, and translates into a higher up front cost.

Looking forward

John is working hard on ruote 2.0, which is a complete rewrite of the 0.9 code base. From the outside it will pretty much look the same, but internally it is a whole new beast. I’m taking on a complete rewrite of ruote-rest rest, christened ruote-kit, which will become the prefered means of exposing ruote to Rails applications. After a lot of chatting with Andrew Timberlake and others in #ruote and on the mailing list, it’s become a real neccesity that we provide a framework for testing/specing process definitions. I’ll tackle this once ruote-kit has made it from vaporware into a polished product, and I’m thinking ruote-spec will be its name.

Resources

• pluginaweek/state_machine
• ruote
• ruote in 20 minutes
• John’s article
• Rails Magazine 3
• Wikipedia on Finite State Machines