First off I’d like to thank a few people who have quietly and vocally contributed to or promoted daemon-kit. In no particular order, thanks goes to:

• Josh Owens (@joshowens) for actively testing bug fixes and promoting the project
• Mike Perham (@mperham) for actively reporting bugs, providing patches and promoting the project
• I’ve pulled in patches from the forks of Michael Hutchinson, Milan Novota, Joey Geiger, and John Merrels
• Greg Campbell reported an issue with the AMQP generated daemons only hours after pushing 0.1.8 to gemcutter

The two biggest changes for me has been converting all the generators to use Thor, and using Bundler in the generated daemons. Both of these projects have received some flack, but they both continue to serve me very well and I’m sure they’ll serve daemon-kit well.

The logger got some minor updates, to be more compatible with the standard Ruby logger class. A brand new XMPP generator is included, and it uses Blather for some awesome evented-goodness. Argument handling got fixed up, email exceptions got removed from the project (I might consider implementing them after overhauling the error reporting features of dk). Hoptoad error reporting got upgraded to the use the newer API. The generated rake tasks are more forgiving when rspec/cucumber or other gems are missing. The cron daemon got some love, allowing easier exception logging/notification for when scheduled tasks fail.

I’ll continue chipping away at the project, tidying up loose ends as they are reported, and start developing a proper test suite for everything.

Over the long term I’m hoping to reach the following goals:

• Always running the eventmachine reactor (think node.rb)
• Support for more modular daemons, including ’single-file’ daemons
• Support for daemons inside Rails projects

Some things will probably get thrown out, like generating configs for god & monit. The more I use chef, the more I realize that those responsibilities lie with the infrastructure management (or devops), and not within the project itself. I’m aware this might cause an upset, however I firmly believe that a lighter and smaller daemon-kit will serve the greater community better.

Thanks for making daemon-kit the number 1 daemonizing project. Please share the love and send feedback, and more importantly use Github issues to log issues, feedback, and wishes.

John P Wood discussed some issues they faced with will_paginate and couchrest during the migration of TextMe to CouchDB, but l left out some code to work with. Couchrest itself had some pagination support that got pulled to some extent… This left me wanting, and wondering, since it was my turn to walk down this path.

CouchDB is a different beast, its aggressive use of indexes means that occasionally you loose some functionality that you’ve been accustomed to having in other persistence mechanisms, like the number of rows matching a query. Jan Lenhardt explains on the CouchDB issue tracker in more detail, but it boils down that you need a reduce function to calculate the number of rows. Sounds difficult? Not at all!

In my case I had a collection of announcements to deal with, and the announcement archives is a paginating collection of documents. Standard will_paginate stuff, nothing special.

Those reading in a reader would want to click through to the post to view the embedded gists further down, or view them directly at Github.

Below is a condensed version of the model from our paginating system:

I’ve included only one view and a corresponding class method, as it is enough to proof the principle. Lets dissect.

The map/reduce functions are extremely simplistic, they simply emit the announcer and the date the announcement was created. This allows for easy scoping and ordering of the announcements. The reduce simply counts our returned records. The magic is in the class method that sets up our WillPaginate::Collection with data from our views.

Line 30 creates a new WillPaginate::Collection instance, passing it the page number and total per page as parameters, it gives us back a pager that we can manipulate.

Lines 31 through 38 uses couchrest’s pagination support to pull out data from our view. The most important things to note here are that the page and per_page options are sent to the paginator and we skip the reduce step.

Once we have our records loaded, we ‘replace’  the pager’s collection with our results from our view (line 40).

The final step is to determine the total number of documents available to us, and for this we need the reduce function. On line 42 we call the same view, with the same arguments, except for requiring the reduce step to happen. We use the results (lines 43 to 47) to inform the pager (will_paginate’s pager) how many rows there are in total.

The controller and the views might look something like this:

It worked, and it shows that we don’t loose as much as we might think when moving away from ActiveRecord and the ton of plugins surrounding it.

Thanks to John for documenting the migration of TextMe, just knowing that it was possible to combine couchrest & will_paginate gave me the push I needed to figure this out.

Disclaimer: This code is extracted “as is” from a real life system and might contain idioms/phrases, and even code, that doesn’t make 100% when viewed in a gist. Please wear your thinking cap when applying this lesson to your own projects.

I’ve been lurking in the CouchDB world for quite some time and have spent a lot of time wrestling with how to loosely relate documents in CouchDB to each other. A big part of learning to use CouchDB successfully is to break away from the shackles of the relational world. Relationships between documents is one such a shackle that seems hard to break.

It is unavoidable that data has to be correlated, and I wanted to rethink how to do it. After plenty of discussions in #ruote with John Mettraux we came up with a model based on how the web works. Since CouchDB is “off the web”, the approach feels quite fitting to me and hopefully to you too.

First some insight into my thinking at this stage.

The web has been successful in loosely expressing relationships between documents. Take two examples:

and

For those of you reading this through a reader, click through to see the gist’s above.

Simple as it seems, in both cases we have a document that is somehow related to the page. The nature of the relationship is expressed via the rel attribute, and the target specified via the href attribute. This got me thinking. Since CouchDB is made off the web, can’t these same principles be applied?

Yes, they can. And here is how:

Currently you might be tempted to express relationships link this in your JSON:

Where changing it to this holds the key:

If anything this format, albeit more verbose, expresses the relationships more clearly and in a format that is web friendly. We’ve broken the shackles of relational thinking.

Enter Correlate

Correlate is an experiment in this line of thinking. It is a mixin for CouchRest’s extended documents that allows you to express these relationships:

Correlate generates getter and setter methods for working with your relationships and lot more (review the README). It also includes a compatibility layer for ActiveRecord to help when you’re migrating from ActiveRecord to CouchRest or building a system on CouchDB that needs to access legacy data via ActiveRecord.

The project is still pretty much a moving target, but I’d love to hear how others address the same issues. Correlate does a great job at maintaining relationship information in a web friendly manner and providing you with some convenience around the verbose data structure. Correlate also has a lot of room for improvement, but that will hopefully change over the coming days as I continue integrating it deeply into my existing projects.

Please fork the project on github and join the experiment with me.

Having screwed up my kernel configs for my VirtualBox Gentoo image, I needed a serial console to catch the boot messages scrolling past in order to see if all the required hardware was being loaded by the kernel. I’ve never done this on a physical machine before but I am converted now and will acquire a USB to serial port converter in the near future…

Using this article as a base you need to do the following:

1. Enable serial ports for your virtual machine
2. Select “Host Pipe”
3. Enter /tmp/vboxconsole as the filename
4. Use netcat to read the console: nc -U /tmp/vboxconsole

When booting you need to amend your grub boot line to have the following at the end:

console=ttyS0,38400

Making it look something like this:

kernel=/kernel-2.6.30-r6 root=/dev/sda3 console=ttyS0,38400

Proceed to boot and look at netcat to see the entire boot output scroll past without disappearing into thin air when the kernel panics.

Man, I love virtualization. I tested this on Mac OS X 10.5 with VirtualBox 3.0.6, but it should work on any *nix platform. Some more Gentoo serial console madness can be found on the old Gentoo Wiki.

The worst part of the process was getting the specs to run with rake spec. Using ./script/spec it worked on individual specs and on all the specs worked as advertised, but rake spec didn’t do anything.

After a lot of time spent in the debugger I wasn’t any wiser. The only difference was that in Ruby 1.8 the example groups were fully loaded, and in Ruby 1.9 they were empty. I gave up and started searching relentlessly for some information on the issue. I couldn’t find anything, until I found an indirect solution on the rspec wiki. It seems that if you have any versions of the test-unit gem after the 1.2.3 release installed, your Rails specs will simply not run. For me, removing test-unit 2.0.3 made the difference and the specs ran properly. PowerDNS on Rails has now joined the ranks of my Ruby 1.9 compatible projects.

Ruote in 20 minutes from Kenneth Kalmer on Vimeo.

The original post of the presentation is over here.

After I read this post on Labnotes I decided to “take the plunge” into the world of Ruby 1.9 properly. I mean, I was at RubyKaigi where the biggest lesson of all was “Change!” (and changing to 1.9). Ruby 1.9.1 has been stable for quite some time now, and 1.9.2 is due out on Christmas. And we’re still stuck on 1.8.6 ? WTF

Getting started with Ruby 1.9 is easy (only time will tell how much pain is caused). I have to admit I’m not as brave to wipe my rubygems installation, I might still need 1.8.6 (and I seriously hope I don’t).

So Relevance made this awesome bash script called ruby_switcher.sh that you absolutely have to try. Once you have it installed and working as per their instructions you need to make one small change to the very last line of ruby_switcher.sh. That change looks like this:

#use_leopard_ruby
use_ruby_191

Simple and I can easily toggle 1.8.6 mode if needed.

Yes, I did.

So over the coming days I’ll be working through all 28 of my github repos and making sure they run on 1.9. This is no small task, but I’m looking forward to getting my “house” in order.

At this stage I know acts_as_audited is 1.9 compatible, I fixed it earlier the week while testing ruby_switcher. I had a look at ActiveRecord::Tableless and decided to pull the plug on the project completely (see here).

I’m really looking forward to this adventure and sharing the ups and downs of the road with everyone. Ultimately I would love it if we can get the South Africa Ruby ecosystem on 1.9.X ASAP.

Follow my project updates over on my Ruby 1.9 Progress Report

I wrote daemon-kit to solve two big issues with writing daemon processes in Ruby:

1. Everyone is re-inventing the daemons gem
2. Individual daemons share a lot common code, apart from the daemonizing bits

As for #1, daemon-kit at first wrapped the daemons gem, and later ripped it out completely as it was difficult to wrap up the worst of the daemons gem properly.

#2 seemed to be a twofold problem that daemon-kit has also addressed with great success. The first was addressing the all too common issues of logging, pid file management, umasks, signal traps, config files, exception handling and other “low-level” issues. Everyone was implementing these things to a limited extent in their daemons and this could cause a lot of frustration when done wrong. The second part was the need, even if undiscovered, for higher level re-use and development speed. Daemon-kit addresses this with some limited generators, making it easy to get going with a cron-style daemon, AMQP & XMPP bots as well as the newest addition, Ruote remote participants.

All this is fine and well, and people seem to like the project. The mailing list is getting some noise, we’re over 200 watchers strong on Github and the IRC channel has some folks popping in to say hi.

However, daemon-kit quickly made me lazy and realize there is a couple of things it can do much better. If you’ve used a generator before, you’ll notice the generated code is very much stuck to that type of daemon. Changing from XMPP to AMQP (for illustrative purposes) would be best accomplished by generating a new project and copying over the lib/ folder only. This sucks.

Another problem is people, myself included, would like to have Sinatra-style daemons (i.e. one file) for smaller tasks. Currently this is not possible at all. Another thing I know people are doing behind my back is generating daemons inside Rails projects, which may or may not work, depending on whether you load Rails’ environment.rb.

So, what happens now?

My thoughts are to implement privilege dropping support and tag a 0.2 release. This gives us a feature complete framework, albeit not as good as it can be. I’ll maintain 0.2 as a stable while undertaking the rewrite. A rewrite? Read on.

I posted to the daemon-kit list a suggestion for stackable daemon environments. I’ve discussed this in IRC with a few folks as well. Jordan Ritter gave me an exceptional breakdown of the dangers of doing something because it is “neat” and I’ve taken his warning to heart. However, I cannot seem to argue against stackable daemon environments, it sounds too good.

The idea is pretty simple. Stack-entries can be compared to Rack applications, with two significant differences. The first is that they will be called only once. They have the opportunity to change the environment in which the final code runs. The second is that they can be called at four different stages of the daemon lifecycle: argument processing, pre-daemonization, post-daemonization and shutdown. This differs from Rack’s single call() method.

The stackable nature also gives the stack members easier ways to set conventions and can dramatically minimize configuration. It paves the way for plugins, sinatra-style daemons, rails-based daemons and easier packaging of distributable daemons, and so much more. Looking at the internals of daemon-kit, it would greatly help simplify the existing code as well as help separate utility classes from stack members.

The more I think about it, the more obvious this becomes, and the more possibilities unfold for the framework. This will definitely make daemon-kit a force to be reckoned with, and hopefully I can persuade other library developers to offload their daemonizing code to daemon-kit, just like rack developers offload their HTTP handling to rack.

Undoubtedly my worst time on stage ever! Ubuntu gave me issues with the projector, so I actually had to leave the stage to go to another conference hall to get the setup working again and run back up to the main hall… Once there, I was a mumbling mess, but made it through without the buzzer beating me.

Presenting a lightning talk at a Japanese conference is surely something else, the audience expects fireworks and the presenters deliver. It was truly an amazing experience, humilation and all, and I’m looking forward to the last day of the conference.

But enough blabbering, the ISP in a Box platform is huge, and interacts with various other service providers through a myriad of API’s, some of them as reliable as convicted mobster. Our support teams are constantly double checking the interactions between the autonomous participants and our service providers, and this means a lot of copying and pasting, from our interfaces to theirs.

They simply “double click” to select text in our interface, causing the browser to copy additional whitespace with the selected text. When pasting into a search box they need to prune the additional whitespace off manually, making the process even slower and more tedious than it needs to be. I can’t replicate this with Firefox on Ubuntu, but it happens to the Windows guys…

In the previous version of the system I solved the issue quickly and bluntly by simply placing all the information in text fields that were styled to be transparent. In this day and age of unobtrusiveness I thought it more appropriate to generate the text fields on the fly, since they’re a convenience and have no effect on the functionality of the system.

It took me 5 minutes to get this done in jQuery, and another 10 to figure out how to wrap it all up as a jQuery plugin. I love jQuery…

Ladies and gentleman, I present you with the jQuery copyfriendly plugin.

Usage

  $(document).ready(function(){
    $('.copyme').copyfriendly();
  });

And then mark your “copy friendly” HTML elements like this:

<table>
<tr>
<td class="copyme">Hello</td>
</tr>
</table>

Or inline like this:

<code><span class="copyme">Heya</span></code>

copyfriendly() takes on parameter currently, which is optional:

  copyfriendly( {
    klass: "borderless"
  } );

The ‘klass’ option tells copyfriendly which CSS class to apply to the inputs generated.

Sample CSS

You can make the inputs generated by copyfriendly invisible using this sample CSS:

  input.borderless {
    border: none;
  }

Do be warned that it might take a little more to align the appearance of the remaining text with the surrounding text in the document.

How it works

Actually quite simple. All it does is extract the text from the element and then replaces that text with an input tag of type “text”, and the value is the original text extract from the matching element. The size of the element is dynamically calculated from the string length, so there might be room for improvement here.