BHUGA WOOGA!

Migrating Drupal comments to Disqus

bhuga — Fri, 05 Mar 2010 21:07:38 +0000

After learning about Disqus this week, I became quite enamored of it. Drupal's comment system is swell and all, but I know that I don't have time to set it up as to be as nifty as it could be. So I think it's swell if someone wants to take it off my hands for free. The downside is that if I upgrade to D7, I wouldn't get those swell forthcoming RDF-a comments. But there's a Disqus API, so I can solve that later if it's important to me.

The problem is that having both Disqus and comments enabled makes things confusing. Each post has a link at the bottom with '13 comments 13 comments and 0 Reactions' or something similar. Not swell. I don't want to abandon my existing comments--in particular, my Puppet vs Chef and SPARQL posts got good feedback from their respective communities. The Puppet post is #1 on Google for 'puppet vs chef' largely because the authors of both projects commented there. Hopefully, Disqus would let people track those authors to comments on my site in the future.

So the obvious solution is to just import my comments into Disqus! I'll just download some module or converter and...and...nuts. Wordpress and Blogger have the goods, but not Drupal. So there went another afternoon: I wrote a basic importer in Ruby. Ruby is where the good Disqus library was (even if I did have to fix a bug).

Drupal comments have features that do not map to cleanly to Disqus, so it has weaknesses. All the comments are anonymous, and I really miss markdown. But it worked for me. Hopefully it helps someone else, too.

Rapid and Incremental Infrastructure Development with Puppet

bhuga — Thu, 04 Mar 2010 16:33:10 +0000

So pretty much everyone has at least heard of Puppet by now. And yes, it's awesome. But it can be daunting to get started--generally speaking, configuring something with Puppet takes longer than just doing it, and that means that setting something up requires, if not Big Design Up Front, some Big Work Up Front. You need a puppetmasterd, and some servers, and some config, and it gets a complicated quickly.

What I want is a simple testbed, a dry run setup, where I can run my code repeatedly, just like during normal development. I want to develop incrementally and flexibly. Can it be easier? Of course it can, that's why we have clouds. I've been working on some infrastructure for a new project, and the workflow I'm using is easy and effective. I just boot a community EC2 AMI, get git, and pull down my puppet repo. That repo has a handy script that installs ruby, gem installs puppet (every distro is using a dated version--forget them), and then I'm ready to go.

The directory structure, like most Puppet installations, looks like this:

- etc
  - puppet
    - setup_puppet.sh
    - modules
      - (the goods)
    - manifests
      - nodes.pp
      - site.pp

From here, it's easy. puppet manifests/site.pp will run the config locally, without a server or any other trouble. In my development, I have a branch for the actual puppetmaster, which replaces nodes.pp from a default node that includes everything into something more meaty. Everything else is the same. From here, I can hack away, testing as I go. Add a --noop to do dry runs. Add -d to enable debug mode and see exactly what commands are run.

The setup script is dirt simple:

#!/usr/bin/env bash

# libopenssl-ruby1.8 isn't necessarily required with this, 
# but you do need it for the puppetmaster server.
sudo apt-get install -y rubygems libopenssl-ruby1.8

sudo gem sources --remove http://gems.rubyforge.org/
sudo gem sources --add http://rubygems.org 
sudo gem install puppet --no-rdoc --no-ri

# Debian weird path
export PATH=$PATH:/var/lib/gems/1.8/bin

4 hours of very productive infrastructure work cost me about 35 cents. No puppetmasterd, no existing servers, and no temptation to store meaningful config on the local disk (since I shut these instances down after a few hours, as I'm easily distracted). No messing around. I really like this workflow.

As an aside, this workflow is fast enough that two Ubuntu gotchas are now actually a problem for me. Firstly, the official AMIs from Cannonical now require the initial login to be via the 'ubuntu' user, which is a pain, because now root can't effectively git clone anything without more hoops. Secondly, Rubygems is broken on Debian. It flatly refuses to run gem update --system, and when you force it to with the rubygems-update gem, it manages to lose track of all installed gems, including Puppet. Since any Puppet gathers all information before doing anything, it will read that Puppet is installed, and any code that runs the gem update won't understand that Puppet now needs to be reinstalled. I'm not sure this can be worked around in Puppet at all; it might have to be out of band.

This is a well-known issue, it's more than 2 years old, and I can't find why this is the way it is. Google results are too full of people working around the problem to find an actual discussion of the original issue. Rubygems.org has a faq on the issue, but I did find a the Debian issue in which this appears to have been done instead of simply fixing the problem. I'm not sure if this has to do with a wonky directory setup, or if Debian just assumes I'd put an eye out with that much Power. Either way, tons of people have problems with it, and it seems curious to me that Debian has decided they should suffer. And it is a decision: Rubygems has been updated a number of times since 2007, and the disabling code had to be explicitly upgraded at least once that I found.

Anyways, the next bit of agile infrastructure work I do will be on a RightScale Centos 5.4 AMI. What's the point of being agile if you don't try new things?

Updated: Centos 5.4 is running Ruby 1.8.5. I guess being agile is about trying hopelessly outdated things, too.

Hacking on RDF in Ruby

bhuga — Sun, 28 Feb 2010 07:20:17 +0000

RDF.rb is easily the most fun RDF library I've used. It uses Ruby's dynamic system of mixins to create a library that's very easy to use.

If you're new at Ruby, you might know about mixins in other languages--Scala traits, for example, are almost exactly functionally equivalent. They're distinctly more powerful than Java interfaces or abstract classes. A mixin is basically an interface and an abstract class rolled into one. Rather than extend an abstract class, one includes a mixin into your own class. A mixin will usually require that a given class implement a particular method. Ruby's own Enumerable class, for example, requires that implementing classes implement #each. For that tiny bit of trouble, you get a ton of methods (listed here), including iterators, mapping, partitions, conversion to arrays, and more. (If you're new to Ruby, it might also help you to know that #method_name means 'an instance method named method_name').

RDF.rb uses the principle extensively. RDF::Repository is, in fact, little more than an in-memory reference implementation for 4 traits: RDF::Enumerable, RDF::Mutable, RDF::Queryable, and RDF::Durable. RDF::Sesame::Repository has the exact same interface as the in-memory representation, but is based entirely on a Sesame server. In order to work as a repository, RDF::Sesame::Repository only had to extend the reference implementation and implement #each, #insert_statement, and #delete_statement. Nice! Of course, implementing those took some doing, but it's still exceedingly easy.

RDF::Enumerable is the key here. For implementing an #each that yields RDF::Statement objects, one gains a ton of functionality: #each_subject, #each_predicate, #each_object, #each_context, #has_subject?, #has_triple?, and more. It's a key abstraction that provides huge amounts of functionality.

But the module system goes the other way--not only is it easy to implement new RDF models, existing ones are easily extended. I recently wrote RDF::Isomorphic, which extends RDF::Enumerable with #bijection_to and #isomorphic_with? methods. The module-based system provided by RDF.rb means that my isomorphic methods are now available on RDF::Sesame::Repositories, and indeed anything which includes RDF::Enumerable. This is everything from Repositories to Graphs to query results! In fact, query results themselves implement RDF::Enumerable, and thus implement RDF::Queryable and can be checked for isomorphism, or whatever else you want to add. This is functionality that Sesame does not have natively, and which I wrote for a completely different purpose (testing parsers). Every RDF::Enumerable gets it for free because I wanted to compare 2 textual formats. Neat!

For example, here's what it takes to extend any RDF collection, from RDF::Isomorphic:

require 'rdf'
module RDF
  ##
  # Isomorphism for RDF::Enumerables
  module Isomorphic

    def isomorphic_with(other)
      # code that uses #each, or any other method from RDF::Enumerable goes here
      ...
    end

    def bijection_to(other)
      # code that uses #each, or any other method from RDF::Enumerable goes here
         ...
    end
  end

  #  re-open RDF::Enumerable and add the isomorphic methods
  module Enumerable 
    include RDF::Isomorphic
  end
end

Of course, this just can't be done without monkey patching. Mixins and monkey patching together make for a powerful toolkit. To my knowledge, this is the first RDF library that takes advantage of these features.

It's possible to provide powerful features to a wide range of implementations with this. RDF.rb does not yet have a inference layer, but any such layer would instantly work for any store which implements RDF::Enumerable. Want to prototype some custom business logic that operates over existing RDF data? Copy it into a local repository and hack away. No need for the production RDF store to be the same at all, but you can still apply the same code.

As a counter-example, compare this to the Java RDF ecosystem. There are some excellent implementations (RDF::Isomorphic is heavily in debt to Jena), but they're all incompatible. Jena's check for isomorphism is not really translatable to Sesame, or anything else. RDF.rb, in addition to providing a reference implementation, acts as an abstraction layer for underlying RDF implementations. The difference is night and day--with RDF.rb, you only need to implement a feature once, at the API layer, to have it apply to any implementation. This is not a knock at the very talented people behind those Java implementations; making this happen is a lot of work in a language without monkey patching, and RDF.rb is only as good as it is because of the significant influences those projects have been on Arto's design.

The end result of the mixin-based approach is a system that is incredibly easy to extend, and just downright fun. It would be a fairly simple task to extend a Ruby class completely unrelated to RDF with an #each method that yields statements, allowing it to work in RDF::Enumerable. Voila, your existing classes now have an RDF representation. Along the same lines, if one is bothered by the statement-oriented nature of RDF.rb, building a system which took a resource-oriented view would not require one to 'break away' from the RDF.rb ecosystem. Just build your resource-oriented model objects and implement #each, and away you go--you can now run RDF queries and test isomorphism on your model. Build it to accept an RDF::Enumerable in the constructor and you can use any existing repository or query to initialize your model.

RDF.rb is not yet ready for production use, but it's under heavy development and already quite useful. Give it a shot. You can post any issues in the GitHub issue queue.

The Promising Future of the Unlicense

bhuga — Wed, 13 Jan 2010 17:35:41 +0000

So the public launch of the unlicense movement on January 1st has gone better than expected. Arto's post hit top of the list for 'most controversial' on Reddit for a while, and unlicense.org itself is seeing decent traffic. Since the target audience of the site is developers, even a few thousand eyeballs is a good number, and that number is being handily beaten.

There are three main concerns that seem to keep appearing in discussions in regards to releasing software into the public domain. I'd like to briefly offer a response to them, and then provide an example of why public domain might be what you're looking for for your software.

Concern 1: Only the GPL preserves Freedom!

This is amazing to me. There is an incredible amount of worry, it seems, that faceless corporate Cthulhu-associated entities are lurking in the shadows, just waiting to pounce on vulnerable code unprotected by the GPL's armor. These faceless horrors have crushed promising startup projects with vulnerable licenses like Apache (the world's most-installed web server), SQLite (the world's most-installed SQL server) (and public domain, not actually licensed at all, by the way), BIND (the world's most-installed DNS server), and other sad stories. I find this concern so unrealistic it boggles the mind.

But just for fun, let's start a real flame war. Numerous folks out there claim, with a certain sort of correctness, that the GPL keeps software free from the lockdown of derivative works. This is called Freedom (capital F). However, that software's Freedom is enforced by a copyright, which restricts the actions of people by proscribing certain kinds of copy and use. These proscriptions are enforced via a system with far reaching effects, which prevent me, for example, from purchasing a DVD player that disregards region codes.

The inevitable conclusion is that the GPL is about valuing the 'Freedom' of bits over the freedom of humans.

You, dear reader, are far more important than my code, regardless of your choice of license. I have no time for a moral system that makes such claims on your autonomy. I will avoid that system as much as possible: by using the unlicense.

Concern 2: Statutory Law vs Common Law

Statutory law, such as that of many European states, often fails to specify a process for explicit public domain donations, leading many to wonder if such a thing is even possible. Folks better educated than I seem to have differing opinions. But I'll note that if you're concerned about it, there is precedent. The original w3 server, which was owned by CERN (guess what the 'E' stands for?), was placed into the public domain in 1993.

Apache and Netscape both trace their heritage back to European public-domain software. You'd be foolish to accept this post as legal advice, of course, but in my mind, those are perfectly acceptable European counter-examples.

Concern 3: Moral rights cannot be relinquished

A true concern, many jurisdictions specifically prohibit the relinquishment of 'moral rights' of authorship, namely, the right to be the named author of a work, and the right to not have works you did not author attributed to you. To my mind, this a not a problem of copyright, it's basically statutory encoding of the fact of authorship. The issue is muddied by several states conflating copyright enforcement with moral rights enforcement. While again not legal advice, I'd say that simple attribution covers you.

Concerns 2 and 3 really bother me, because legal arguments against them are complicated and require specialized knowledge. I'm not qualified to argue them in a bulletproof manner. But most software will never become big enough to have a license (or unlicense) issue, and you can issue someone an explicit license if it's ever a problem (as SQLite does).

How public domain lets your software grow

"CERN's decision to make the Web foundations and protocols available on a royalty free basis, and without additional impediments, was crucial to the Web's existence. Without this commitment, the enormous individual and corporate investment in Web technology simply would never have happened, and we wouldn't have the Web today."

Tim Berners-Lee, Director, WWW Consortium

The public domain is the best way that others can take your ideas and run with them. CERN's public-domain dedication is probably the best example of that, but if you want your software to change the world, you need to allow others to use it as freely as possible. I'll give you a little personal example.

A few days ago, I published promising future, a Ruby gem that adds Scheme-style promises and futures to Ruby. I did this because I happen to love promises and futures, and it drives me absolutely nuts whenever they are not available.

If my goal is (and it is) to always have promises and futures available, the ideal would be that it were in the Ruby core library, or even a language future. I could start a rallying cry on a mailing list somewhere, and may yet, but my odds are slim. But what if a much better known author, with a much more popular library, wants to use these lovely little things in his code? Well, they could add a gem dependency, but that's not a popular option for various reasons. If the licenses work out, they could incorporate the code, usually requiring a note of attribution.

But my promises and futures code has the maximum possible flexibility. Anyone, with or without any license, can copy/paste my promises and futures into their code, without attribution, and be done with it. Problem solved. My code lives on, or at least inspires the creation of equivalent functionality implemented in a better way. And maybe, one day in a promising future, every Rubyist everywhere will have promises and futures available. That SQLite can be embedded in other software is a huge factor in its unparalleled adoption.

How could my software be more free? How could I be more free? How could you be more free? What could be better? Public domain promises a very promising future indeed.

Quantity.rb: first-class quantities for Ruby

bhuga — Tue, 05 Jan 2010 10:07:48 +0000

I've just put out a first release of Quantity.rb, which scratches an itch I had and much more.

Quantity.rb provides first-class Quantity objects, like '12 meters', '1 liter', or '1 dozen'. More significantly, it supports things like '12 meters * 1 kilogram / 2 seconds**2'. It was an outgrowth of an attempt to do some automated unit conversions of a project I am working on involving some monitoring, and I wasn't happy with what was out there. In particular, I wanted to eventually provide the ability to divide one time series of data points by another, regardless of units. It needed to be something more than 'meters to feet'. Maybe it didn't need to be this involved, but it's the right way to do it: anything can be built on top of this.

It's not the first attempt, and perhaps not even the first success. Quanty is the earliest one I can find, and it does most of what I want. Unfortunately, it uses yacc, which I have no intention of learning, and the English docs are sparse. There's something called the Quantity Management Framework, but I can't find much info about it.

Besides, I figured it would be fun. I would learn something, and sometimes it's good to have a project with a well-defined scope so that you can Finish It. Especially when you have a handful of muddy projects mixed with a handful of very long term ones. So it was the charge of the light brigade. And I did learn something. Earlier versions used some class inheritance features that made me learn far more about Ruby's metaobject system than I had ever hoped to. That was kind of like this for me.

Anyways, there's more to do, but I'm pleased with the results so far. Some of the things you can do, from the README:

require 'quantity/all'
1.meter                                                 #=> 1 meter
1.meter.to_feet                                         #=> 3.28083... foot
c = 299792458.meters / 1.second                         #=> 299792458 meter/second

newton = 1.meter * 1.kilogram / 1.second**2             #=> 1 meter*kilogram/second^2
newton.to_feet                                          #=> 3.28083989501312 foot*kilogram/second^2
newton.convert(:feet)                                   #=> 3.28083989501312 foot*kilogram/second^2
jerk_newton = newton / 1.second                         #=> 1 meter*kilogram/second^3
jerk_newton * 1.second == newton                        #=> true

mmcubed = 1.mm.cubed                                    #=> 1 millimeter^3
mmcubed * 1000 == 1.milliliter                          #=> true

[1.meter, 1.foot, 1.inch].sort                          #=> [1 inch, 1 foot, 1 meter]

m_to_f = Quantity::Unit.for(:meter).convert_proc(:feet)
m_to_f.call(1)                                          #=> 3.28083... (or a Rational)

It's made my IRB shell quite the handy calculator. Try it out for that, if you're CLI-inclined.

This whole affair was also an excuse to release something meaningful via the unlicense (I also did a growl-amqp thingee but it hardly counts). The unlicense is a framework for releasing code not with a license, but as public domain. Public domain is something that old timers remember: what used to older copyrighted works. Originally some pithy few years, copyright these days now lasts for an author's lifetime + 70 years, and it's been several years since anything entered the public domain in the US due to numerous extensions. Some countries have gone so far down the rabbit hole that one cannot dedicate things to the public domain.

This is all the more ridiculous when one considers that most people now believe copyright is bunk. Eventually, legal frameworks will respect how the world is, and not how it was. A lot of people won't release software under the public domain because of the spotty legal status. A few years ago, people were equally afraid of the GPL until some court cases affirmed the common-sense interpretation of the license. Let's release some public domain software and push the issue of what happens when you don't have a license at all. I was hoping to release this on the first of January for public domain day, but it needed more work. I guess it's not much of a holiday since nothing enters the public domain anymore anyway.

Anyways, 'gem install quantity' and have fun.

Is W3C going the wrong direction with SPARQL 1.1?

bhuga — Tue, 03 Nov 2009 02:02:11 +0000

The W3C SPARQL working group (previously the Data Access Working Group) has recently released their first versions of the updated SPARQL standards, or SPARQL 1.1. The group's roadmap has these finalized a year from now, but they have asked for comments and I suppose these are mine.

I believe that these documents are a step further down a wrong path for SPARQL and, to a lesser degree, for RDF in general.

The latest round of changes includes a number of changes to SPARQL, including aggregate functions, subqueries, projection expressions, negations, updates and deletions, more specific HTTP protocol bindings, service discovery, entailment regimes, and a RESTful protocol for managing RDF graphs (the last one is not really just SPARQL, but it's in the updates).

So I'll start with my comments, which are mostly critical.

To start, an RDF-specific complaint, not really related to the rest of the post. Why would the one mandated format to be supported in the new RESTful RDF graph management interface be RDF/XML? What would it take for a the semweb community to move on from this failed standard, which has had known issues for more than 5 years? (those two issues were raised in 2001 and are currently marked 'postponed') Why should such an increasingly irrelevant standard as RDF/XML be chosen instead of the widely-supported and easy to implement N3, N-Triples, or Turtle?

As for SPARQL, the 1.1 standards continue to give named graphs first class citizen status, both in the web APIs and in more SPARQL syntax than they had before. It's not so much triples as quads these days. Other meta-metadata, such as time of assertion or validity time, are not covered. While named graphs are admittedly a particularly often-found case, why does it need to invade the syntax of SPARQL? Not every use case needs named graphs, but every SPARQL implementor must support them. The 1.1 standard now includes precedence rules when for named graph and base URIs when they conflict in HTTP query options and inside the query itself, attempting to solve this self-created problem.

How about subqueries? What about variables during insertions? What about subqueries during insertions? Do we really need implementors to consider these kinds of things for every SPARQL endpoint on the web?

None of these things is really all that bad by itself, but one must consider the bigger picture. SPARQL 1.0 was released in January of 2008 (with some comment period before that) and there is still no implementation of a SPARQL engine in PHP or Ruby (exceptions apply, see [1]). One does not increase the participation of that ecosystem by adding a selection of entailment regimes to the standard.

While a SPARQL implementation exists for the excellent RDFLib in Python, it's only one of the current big 3 (with Ruby and PHP) in web development, and there's only one. The fact that no SPARQL engines exist for Ruby or PHP should be considered a failure of the standard. Why are we adding complexity when there is no SQLite for SPARQL? Why are there at least 3 monolithic Java implementations (Jena, Sesame, Boca), all financially sponsored to some degree or another, but so little 'in the wild'? How long can RDFLib herd 16 cats as committers on the project? While I don't have a lot of direct experience with RDFLib, I pity the project 'leads' (I cannot find evidence that the project is sponsored or that anyone is 'in charge') trying to look towards the future of implementing 6 working papers of new standards.

One of the biggest success stories for semweb in widespread use is the Drupal RDF module, which has found wide acceptance in the Drupal community and started an ecosystem of modules. Drupal 7 will output RDFa by default and Drupal 6 supports a ton of wonderful features, including reversing the RSS 1.0 to 2.0 downgrade back to RDF. But Drupal remains a producer of simple triples and a consumer of SPARQL queries generated by other endpoints. Data in those sites remains locked down. Why? Because implementing SPARQL in PHP is nontrivial, and in a chicken-egg problem, nobody's paying for it before someone has a need for SPARQL.

I could go on, but these are symptoms (well, not that RDF/XML thing, I don't think there's a good reason for that). I feel that the working group is attempting to solve the wrong problem. Namely, it is attempting to define a somewhat-human-readable query language, SPARQL that works for almost all use cases. But why must the whole 'kitchen sink' be well-defined? Such a standards body should be attempting to define the easiest possible thing to implement and extend, not the the last tool anyone would ever use.

The SPARQL 1.0 standard's grammar was well-defined as a context free grammar. It also had extension functions, which were uniquely defined by URIs. Why the distinction between CFG elements and extension functions? Why not make syntax elements like named graphs and aggregate functions as discoverable as extensions? Well, the reason is that it's hard to write a parser of a human-readable format and make those things optional and discoverable. (Here's a SPARQL parser implementation in Scala, a language with powerful pattern matching features for good parsing, and it's 500 lines of code. It compiles to S-expressions, the parsing of which is about 30 lines. Hmm.)

If the protocol had been defined as S-expressions, the distinction would not exist and the syntax could be as expandable as the current functions (the current syntax would just be more functions). The new 1.1 service discovery mechanism is excellent and extendible and would allow the standard to grow dynamically instead of becoming bogged down in features for particular use cases. New baseline implementations of SPARQL would be easy to implement and grow incrementally, and the current human-readable format can be implemented in terms of these expressions.

The web of ontologies has grown with ad-hoc definitions created by people used to fill their needs. Standards grow organically around the ones that are needed most, others languish. Why should SPARQL functions have this kind of flexibility, but not the syntax? The distinction makes implementation overly difficult and is slowing the expansion of the Semantic Web.

In fact, it turns out that Jena has been parsing to S-expressions for some time. If you're an implementor, why would you do it any other way, especially when the standard can change as much as it does in 1.1? Any implementation will have to come up with something equivalent to S-expressions if you are going to be able to upgrade your engine implementation to meet standards like this when they are finalized. If people are doing it anyway, why not just make it the standard?

The SPARQL Working Group should be working on a definition for a function list and discovery protocol for S-expressions, and not for what we currently call SPARQL. What we call SPARQL is something that should compile to a simpler standard if various vendors want to implement it. S-expressions allow maximally simple parsing maximally simple serialization, and the ability to do feature discovery on core features of the language, not just portions which are blessed with the ability to be extended. S-expressions are easier for machines to generate for wide variety of automated use cases, far wider, I would venture, than the set of use cases for the human-readable queries.

Please, please, please do not doom the world to write the SPARQL equivalent of SQLAlchemy and ActiveRecord for the next 20 years! We can define a standard that machines can use natively. Now's the time.

At any rate, that's my beef in a nutshell. The working group won't come up with a successful standard until it's easy enough to implement it that workable implementations appear in the languages that are defining the web today. And when people can use those languages to implement that standard without an army of VC-funded engineers.

The SPARQL 1.1 proposals make the standard better than before, but it's not the standard we need. The SPARQL algebra is what needed expansion and specification, not the syntax.

[1]: The PHP ARC project has an implementation, but it attempts to directly convert SPARQL to an SQL query on particular table layout in MySQL, and is difficult to convert to general use. Despite SPARQL's complexity, ARC managed to implement this in just 6400 lines of code. The parser alone is 2000 lines and the engine another 4400. The serialization/parsing libraries, however, are fine, and were integrated successfully into the Drupal RDF module. The PHP RAP project has also done some good work and is perhaps more wrappable than ARC, but implements only a subset of SPARQL.

Puppet vs Chef

bhuga — Sat, 19 Sep 2009 17:17:55 +0000

I spent a fair bit of time this week messing with Chef, which I had somehow missed in my previous rounds of scooping up system administration tools. It was kind of a late-in-the-game evaluation--I was already in the process of deploying Puppet when it was brought to my attention. But it was only almost too late, not too late, so I threw up some VMs and did some testing.

I was not really enthralled by Chef. Not, at least, for the particular use case I was solving.

It seems pretty awesome. In this post over at Engine Yard, the problem with puppet is summed up rather nicely: Puppet’s biggest flaw is its configuration language that is not quite ruby and not quite turing complete. So you end up wrestling with it to get it to do anything. This is completely true--much like you end up doing cuts for the simplest of things in Prolog, Puppet's DSL is declarative and starts as hassle and never really stops. It seems like a better idea than it is. There are a lot of cases where people need to run puppet twice to get dependencies to work out right.

Inadvertently, however, the sentence before that quote is the big problem with chef: The big advantage Chef has over Puppet is that the codebase is 1/10th the size and it is pure ruby, including the recipe DSL. I am, unfortunately, not measuring the quality of my configuration management system based on how many lines of code it is. Nor am I measuring it's 'ruby purity'. These are, however, Chef design goals. This zealous hunting of unquantifiable elegance feels, to me, to be endemic in the ruby world. It's a fine goal, but it also tends to encourage one to re-invent components that did not need reinventing. It's usually foolish to think that if you start a project over that it will be any better than what you're replacing. I don't think that Chef is really any better than puppet for most situations, syntactic elegance notwithstanding. There are important exceptions, though, so keep reading.

My first experience with Chef was an uneasy installation process. There's no description of how to install Chef. Instead, there are Chef scripts that install Chef for you. That might not be so bad, but if there are stupid problems, it can be difficult to follow.

After a while, I figured out that Chef is just another Merb application with a CouchDB backend. It's a Merb application which requires you to have an OpenID to log in to it, and which requires 2 web servers so that you can have an OpenID server separate from the Chef server (I think). But okay, it's Merb. I've played with it before.

But honestly, that is about as far as I got in terms of making it do things. I never made the server push a useful config to a client. I would not even have gotten to the web interface without Meerkat, because it uses Apache's SSL instead of letting me have the option of using an SSL concentrator. By comparison, puppetmaster's client-server model, which probably uses a lot of those precious, precious lines of code, works pretty much flawlessly from any number of RPM sources, or from gems.

I spent a while going over recipes, and comparing them to Puppet. For example, here's some code to manage sudo for Chef. The Chef code was written by Chef's authors; the Puppet code was written by myself. The Chef code is spread across 3 files.

# recipes/default.rb:
package "sudo" do
  action :upgrade
end
 
template "/etc/sudoers" do
  source "sudoers.erb"
  mode 0440
  owner "root"
  group "root"
  variables(
    :sudoers_groups => node[:authorization][:sudo][:groups], 
    :sudoers_users => node[:authorization][:sudo][:users]
  )
end
# attributes.rb:
authorization Mash.new unless attribute?("authorization")
 
authorization[:sudo] = Mash.new unless authorization.has_key?(:sudo)
 
unless authorization[:sudo].has_key?(:groups)
  authorization[:sudo][:groups] = Array.new 
end
 
unless authorization[:sudo].has_key?(:users)
  authorization[:sudo][:users] = Array.new
end
# metadata.rb:
maintainer        "Opscode, Inc."
maintainer_email  "cookbooks@opscode.com"
license           "Apache 2.0"
description       "Installs and configures sudo"
version           "0.7"
 
attribute "authorization",
  :display_name => "Authorization",
  :description => "Hash of Authorization attributes",
  :type => "hash"
 
attribute "authorization/sudoers",
  :display_name => "Authorization Sudoers",
  :description => "Hash of Authorization/Sudoers attributes",
  :type => "hash"
 
attribute "authorization/sudoers/users",
  :display_name => "Sudo Users",
  :description => "Users who are allowed sudo ALL",
  :type => "array",
  :default => ""
 
attribute "authorization/sudoers/groups",
  :display_name => "Sudo Groups",
  :description => "Groups who are allowed sudo ALL",
  :type => "array",
  :default => ""

Here's more or less the same thing for Puppet:

class sudo {

  package { ["sudo","audit-libs"]: ensure => latest }

  file { "/etc/sudoers":
    owner   => root,
    group   => root,
    mode    => 440,
    content => template("sudo/files/sudoers.erb"),
    require => Package["sudo"],
  }
}

Both Chef and Puppet then take this information and output it through an ERB template, which is an exercise for the reader, since it's basically the same for both.

There's a few things worth noting here. First of all, Puppet has zero metadata available. If you want to set sudo-able groups, you need to know those variable names ahead of time and set them to what you want. Both your template and whatever code sets your sudo-able groups must magically 'just know' this information. Since the Puppet DSL is not even Ruby, you have *zero* ability to perform any kind of metadata analysis on these attributes in order to make code more generic.

Chef gives you complete metadata about the variables it's using. This is powerful and indeed critical in my imagined use domains for Chef (keep reading). That metadata comes at a cost of a lot of boilerplate code, though. Chef comes with some rake tasks to generate some scaffolding. I'm always uncomfortable with scaffolding like this; I think this kind of code generation is a bad way to do metaprogramming.

Chef spreads this information across 3 files, named a particular way. Puppet has a similar scheme of magically named files, but it's basically just a folder structure, a file called init.pp, and templates/source files. For a fairly simple task, Chef requires you to know a folder structure and 3 file names, and which data goes in which files. This is congruent with the Ruby world's (perhaps specifically the rails/merb world's?) general practice of 'convention not configuration'. This is in addition to all of the 'you just have to know' parts of the Chef system which are taken from Merb, such as where models and controllers live, though you would not need to edit those save for pretty advanced cases.

Lastly, Chef provides you with an actual data structure that is fed to the sudoers template. Puppet simply uses available dynamically-scoped variables in its template files. This is *awful*, and a big loss for puppet. I administrate Zimbra servers, for example, which require extra content in sudoers. I cannot add this to the zimbra module unless the zimbra module were to be the one including the sudo module. There are solutions to this, of course, but this is a really, really simple use case and we're already shaving yaks. Chef's method is undeniably superior.

All 3 of these are part of the same core difference between the two: Puppet is an application, and Chef is a part of one.

Chef is a library to be used in a combined system of resource management in which the application itself is aware of the hardware it's using. This allows certain kinds of applications to exist on certain kinds of platforms (particularly EC2) that simply couldn't before--an application using this system can declare a database just as well as it can declare an integer. That's fundamentally powerful, awesome, amazing.

Puppet is an application which has an enormous built-in library of control methods for systems. The puppet package manager, for example, supports multiple kinds of *nix, Solaris, HPUX, and so forth. Chef cookbooks can certainly be written to do this, but I imagine by the time you supported everything puppet does I don't think Chef would get a smiley-face sticker for being tiny and pure with extra ruby sauce. Puppet's not a fundamental change, it's just a really nice workhorse.

I picked puppet for the project I'm working on now. It made sense for a lot of reasons. Probably first and foremost, there are 3 other sysadmins working with me, some split between this project and others. None of us are ruby programmers. We don't write rake tasks like we configure Apache, we don't want to explain to new hires the difference between a symbol or a variable, or where the default Merb configuration files, or 100 other ruby-isms. Meanwhile, most puppet config, silly folder structure aside, is not any harder to configure than something like Nagios. I think it would be a mistake for an IT shop with a lot of existing systems running various old-fashioned stateful applications like databases or LDAP to suddenly declare that sysadmins need to be Merb programmers.

Puppet's much deeper out-of-the-box support for a lot of systems provides the kind of right-now real improvements that a lot of IT shops and random contractors desperately need. System administration is depressingly rarely about being elegant or 'the best' and much more frequently about being repeatable and reliable. It's just the nature of the business--if the systems ran themselves, there would be no administrators. Having a bunch of non-programmers become not just programmers but programmers specializing in a tiny subset of the ruby world is a lot of yaks to shave for an organization. This is not some abstract jab at my colleagues: I am most certainly not a Merb programmer, and even if I were, I have too many database copies to make, SQL queries to run, mysterious performance problems to diagnose and deployments to make to give this kind of development the attention it requires. How many system administrators do you know that use the kind of TDD that Merb can provide for their bash scripts? What would make one think that's going to happen with Chef?

The other big reason I picked Puppet is that it's got a sizable mailing list, a friendly and frequently used google group for help, and remains in active development after a couple of years. I don't think Reductive Labs is going away, and if it did, there have been a lot of contributors to the code base over those 2 years.

It's worth noting, though, that the Chef guys come with an impressive set of resumes. It seems to be somehow tied in with Engine Yard (several presentations about Chef include Ezra Zygmuntowicz as a speaker). I worry, though, that they are working the typical valley business model, namely to explode about a year after launch. Chef was released about 8 months before I write this. The organization I am installing Puppet for does not have the Ruby talent base required to ensure that they can fix bugs as required in the long term if Opscode goes away, or if they get hired on to Engine Yard and they make Chef into the kind of competitive differentiation secret it could be.

Chef currently manages the EC2 version of Engine Yard, and that's just the kind of thing I cannot imagine using puppet for: interact with a giant ruby application to manage itself. If you have a lot of systems joining and leaving the resource pool as required, Chef's ability to add nodes dynamically is going to save you. The ability to define resources programmatically is very powerful--one could easily imagine reducing the number of web server threads if a system's CPU use goes over a certain threshold, for example. I would not try that in puppet! But note that this is an application built from scratch to expect such a command and control system to exist. If you're just managing a bunch of LAMP stacks and samba servers, this is more power than you need. One of the Opscode founders has some slides that talk about this kind of model.

And Chef is powerful for that model, sure, but is that even the model you want for your applications? Applications should not have to worry about the hardware they use. Making an application's own hardware use visible to itself encourages programmers to spend time thinking about issues they should be trying their hardest to ignore. A better model is App Engine's, where the system just scales forever without developer intervention. Even Azure's service configuration schema model is better, in which different application roles (web, proxy, etc) are described as resources and given a dynamic instance count, and transparently scalable data stores are available. The number of 'nodes' in the system is never an issue for either model.

Chef is what you'd use to build that auto-scaling backend. Engine Yard uses it for, well, Engine Yard--scalable rails hosting, transparently sold as a service to folks who can then just blissfully program in rails and never think about Chef. Very few organizations are making that infrastructure, and most of them that are are shaving really big yaks and need to stop and use one of the available clouds.

Meanwhile, a very many organizations are running 6 kinds of *nix to maintain tens of older applications built on the POSIX or LAMP paradigms, or hosting virtual machines running applications made who knows when. For these organizations, Puppet is probably the easiest thing that could work, and thus probably the best option.

I'm sure there are sysadmins out there who think I'm completely wrong, and that you just can't beat the elegance Chef provides. There are a lot of people better than me out there, and I'm sure they have a point. But in my experience, bad system administration happens when sysadmins try and do everything for themselves. For a given situation in system administration, it's highly unlikely a sysadmin can do a better job than an available tool. Puppet's sizable default library is what most organizations need, not the ability to write their own.

And all of the above aside, one thing is clear: there is little excuse for an organization with 3 or more *nix servers not to be using Puppet, Chef, cfengine, or *something*. I would argue that about 80% of the virtualization push is dodging some of the core questions of system administration, making systems movable to new resources indefinitely rather than making their configuration repeatable, but that's a topic for another post. Especially since nobody got this far on this one anyway.

Session Proposal for Drupalcon Paris

bhuga — Mon, 29 Jun 2009 23:01:01 +0000

I've submitted my session Proposal for Drupalcon Paris 2009. After some consternation about the title, I went with 'Replacing your Internal Apps with Drupal'. I think that's where this hits home--we're replacing an entire backoffice of work-process and communications tools with Drupal, and we think it's great.

This is what we've been doing at Openband for the last few years. I've written a number of small, internal apps for a lot of small companies over the last few years and I would do them all in Drupal if I could do it all over again. It's too easy to get data in and out without buying some $20,000 piece of software to link sharepoint and your work-process system.

I gave Openband's presentation at Drupalcon DC 2009, and we got a lot of feedback that what we did was a lot more impressive than the title and session description gave away.

This talk will be a bit of a continuation of this topic. This platform is our baby, and we have a lot to say now that we've started using it internally. We're hoping to aim this at a slightly broader audience, and hopefully the session description is a bit more attractive this time. And even if you were at the session last time, we've come a long way in the last 6 months.

Anyways, go vote for the session!

RDFa video from Drupalcon

bhuga — Mon, 16 Mar 2009 19:23:49 +0000

scor did the hard work on putting together a presentation on RDFa for the keynote at DCDC. Just like last year, I did the legwork on the screencast while smarter, more productive people did the hard parts. It would seem that I am the voice of RDF in Drupal.

But alas, due to timeframe it was not to be. Fortunately, Boris played a version during his RDFa presentation at Drupalcon DC 2009, and scor ran two BoFs which I couldn't really participate in, having to give Openband's presentation on Thursday and being cut down by food poisoning on Friday.

Now scor did a writeup on the semweb group, and Dries noted it on his blog. I won't belabor the point when others have already worded it better by reiterating it all here, but the more exposure RDFa gets, the better! More credits for the work are in both of those links. You can view the video in the above links flashy style, and I've also put up the higher-res original here.

Import/Export BoF at Drupalcon 2009 DC

bhuga — Sun, 08 Mar 2009 18:55:32 +0000

My food poisoning at DCDC (don't eat rare meat in DC!) abated just long enough for me to attend an excellent BoF towards the end of Friday at Drupalcon DC 2009. The goal was workable configuration management in Drupal, and in no particular order, attending were:

Adrian Roussouw, author of Forms API
Bala Bosch, Chrys Bryant and Sarva Bryant, authors of Patterns
Greg Dunlap, author of the deployment framework
Young Hahn and Jeff Miccolis, who gave us the context and spaces projects
Bevan Rudge, who wrote config to code

In terms of configuration management, having these folks in a room is a bit like the G7, but with Voltron attending.

In addition, one or two more interested parties showed up, including myself (I've tried to limit rattling off names to people who contributed to the projects listed; if I missed one, sorry). Openband is trying to stand up our stack with more than 200 modules, and configuration management is probably our biggest challenge right now. We'll be applying some resources to the problem and want to make sure that whatever solution happens is one that provides us a workable upgrade path for the future.

Probably the biggest trick about all of this is separating what's needed from the use cases. It's difficult to separate what really needs to happen from the end result, and the community is so quick to jump on any potential solution to this widespread problem that it's hard to work on something that enables solutions without talking about solutions. The short version of what needs to happen is that 'each module needs to do its own part', but there's no straightforward answer.

Going over my notes, the discussion had two main themes: requirements and implementation.

Requirements

Context and Dependencies

The context module does something really sweet, in that it provides a 'context' for a page load or a feature (a 'space' from the spaces module is more or less a context definition for a feature). This is an important idea, in that it maps settings to functionality and not to modules. It was generally agreed that this idea needs to be incorporated in any end solution, but perhaps not agreed that it was core's job to deal with it. Along related lines, there was also some question on whether or not core should handle ideas like dependencies, and I believe the general consensus was no.

Roadmap

Most everyone agreed that nobody wants to throw D6 to the wolves, and that the solution should come in the form of a hook/hooks added to D6 in contrib which can hopefully be added to D7 core. In either case, anything that builds on these hooks is a contrib space thing--this hook needs to be implemented by core, but won't be called by it. So the degree to which D6/D7 get supported after the hook's implementation can be left up to whoever has the resources and inclination to do it.

Limitations

Related to roadmap, we need to be cognizant of what we're doing. It would be easy to replace a Data API by being too broad--the truth is that config and content have no inherent difference (what's a group, after all?). The Data API (or something else) should eventually take care not only of this problem, but of the content problem as well. Thus, this is not something that will live forever, nor is it something that will be perfect. It will pick some low-hanging fruit in the problem space, and do it broadly, but it's not going to be all things to all people. This is frustrating, as where to draw the line between 'config' and 'data' is something that varies from site to site and it's going to be a problem for people for whom the line is drawn just a tad bit off.

Implementation

Hooks in contrib6/core7, the rest in contrib

It was generally agreed upon is that there needs to be some kind of hook, or set of hooks, that *each module* can implement. The work needs to be associated with each module, so that it only need be done once. The exact nature of that hook has some argument left; there are two basic paths, outlined below. There's also consideration for a hook_default_config hook.

A self-consuming API?

One idea is that modules simply export a configuration and import it: a module can eat whatever it outputs. But this turns out to be a Godzilla task for most modules to implement. Forms API has a couple of layers of validation in a couple of places, some of which, such as what kind of data something is, or whether it's in a list of allowed values, is never implemented by a developer. Such a high-level import/export API would force modules to write specific validation for everything, and a lot of modules would need to refactor custom validation from form code into reusable code. But even that would not be enough--thanks to the magic of form_alter, modules do not even necessarily know what consists of a valid configuration for themselves! To match existing functionality 100%, we'd have to have a hook_export_alter and hook_import_alter to allow modules to clobber each other as much as they already do. Thus, not only would each module need to implement the hook, it would have to re-alter every module it already does. Fun stuff!

Is Forms API the API?

The only API that *every* drupal module supports is Forms API. Everyone's a bit nervous about this, since the Forms API does not technically support macros, and even if it did, it's not the best way to visualize configuration for a module; a pretty form does not automatically transfer to a data structure. But as things stand, this is the only API that is 100% compatible.

Current Implementations

Context / Spaces

Spaces is a module that provides context for a feature, with import/export functionality. I understood that it implements its config import/export mostly on its own, without using Forms API. Associating features with a context allows them to be quickly enabled or disabled, or for modules/themes to change their behavior based on the context , and probably more. I need to play with this more.

Patterns

Gravitek Labs have written Patterns, which convert snippets of YAML and XML into Form API calls. They have support for most of core, plus views and cck, and it's fairly trivial to write patterns for modules that don't support it by identifying fields in their forms.

They have also started some work on something called the Configuration Framework for D6, which appears to be a standardized way to write data for Forms API and some magic for processing it. It provides some hooks for modules to implement which are a bit like import/export, but designed to provide input to their forms. It's also got the idea that modules should be able to ship with their default config in a text file. It uses patterns as the representation of config, which means it can be XML or YAML (with more to come).

Deployment Framework

Greg Dunlap uses both methods (for some things he used drupal_execute, others not). Much of the deployment framework is solving another problem, however, content, and I think it was generally agreed that this system should not attempt to create a layer that would be used for passing around content. It's a significantly more complicated problem anyway, as Greg discovered when he added a lot of things I suspect the Data API folks will end up having to do anyway, in particular indexing content by both auto-incremented id's *and* unique identifiers.

All the rest

Other issues were mentioned, including, but not limited to, the D7 variables patch, and context and how important it is (solving, for example, the global uid problem),

At any rate, we agreed that the next phase of deliverables are:

An import-export API for discussion
A best-practices document to describe how to write exportable modules