The Madstop

Summer Operations Conferences

2009-06-05T15:20:07Z

I’ll be at three conferences in quick succession in June, speaking at two of them.

First I’m on a configuration management panel run by James Turnbull at the Open Source Bridge in Portland, OR. I haven’t been on a multiple-tool panel in a while, so this should be interesting. I’m also excited to be back in Portland - I absolutely love the town, so much that I’m moving back there this summer. And for those non-Americans who think we don’t make great beer: Get thyself to Portland, and see how wrong you are. I should also be running a Puppet BoF there.

From there I fly down to San Jose to give a workshop on Puppet at the O’Reilly Velocity conference. This is its second year, and while it claims to have a web operations/performance focus, I think it stands to quickly become the best general operations conference around. LISA is too academic and just moves too slowly, and there isn’t a whole lot of other competition.

Last year’s was great, and this year’s is a day longer and looks to be even better.

Finally, the day after Velocity, I attend Om Malik’s Structure. This conference is more focused on operations executives, which is a very different focus for me, but I met a lot of interesting and knowledgeable people there last year, including some from completely outside the technology space. I normally diss executive events, but this one seems to stand out.

Hopefully you’ll have a chance to track me down at one of these events.

RESTful Puppet and You

2009-06-04T22:34:22Z

No, I’m actually not dead, as much as it might appear from my lack of updates. Fortunately there are other means to verify my vitality.

Anyway, the goal here is to help you understand how the imminent release of Puppet 0.25 affects you. And by “you” here I am, of course, making some assumptions - if you don’t use or care about Puppet, um, not so much. But if you use Puppet, or, even better, if you care about how well Puppet solves your problems, then this post is for you. I hope.

I’ve no short-term memory and don’t feel like planning this blog post out, so I’m going to just do what I can to ad-lib the reasons that Puppet’s new-found RESTfulness kicks ass. And stuff.

Smaller and Faster

First and foremost, 0.25 does not have many what you would call features. It’s mostly a refactoring release. This refactoring in general results in a faster system that takes less memory. Experiments have borne this out: People are seeing 10-40x speedups in fileserving, at least 30% less ram consumed on client and server, and much faster run times on the client.

So, look not for features but for benefits.

What is REST?

REST is more of a principle than a standard, but the things that matter in it for Puppet are:

All data transferred is treated as a stand-alone ‘resource’ (and here ‘resource’ means something different than a normal Puppet resource). For instance, Puppet 0.25 treates catalogs, file content, file metadata, certificates, certificate requests, fact collections, and probably a bit more as resources.
All resources have a unique URI. E.g., a given host’s catalog can be found at ‘http://puppet/$environment/catalog/$hostname’.
Standard HTTP infrastructure is used as the ‘api’ - HTTP ‘put’, ‘get’, and ‘delete’ are the primary operations, and HTTP content-type handling is used for serialization negotiation.

Of course, none of these are really helpful to you, whomever you are. What’s helpful is how we’ve made these basic ideas useful in Puppet. So here are some of the things you get:

Faster and Smaller

The big problem with our current network protocol (XMLRPC, which is essentially XML over HTTPS) is that everything has to be considered XML, which means that anything that isn’t actually XML (which is, um, everything in Puppet) has to be encoded and escaped so it can act like XML. For file serving, this means that every file has a minimum of three copies in memory - plaintext, encoded, and escaped. This is time- and memory-expensive.

Oops I Did It Again

Sorry, couldn’t resist. One of the benefits of switching to REST is that we’re breaking compatibility, so we can fix some design problems we (i.e., I) made early-on. The biggest benefits have been around fileserving and the catalog.

In pre-0.25 releases, recursive file-serving involves a minimum of one call per directory and one call per file. This is obviously slow and bad. 0.25 fixes this so it’s one call for an entire directory structure, with possibly one call for each file that actually needs to be transferred. The combination of this plus the lack of encoding and escaping has resulted in most people seeing a 10x speedup in recursive fileserving.

For catalogs, well, pre-0.25 releases don’t really talk about them. They transfer this serialized recursive tree structure, rather than the whole catalog (which is really mostly a graph). This matters a bit in 0.25, because it’s allowed us to remove some extraneous stupidity that resulted from the limitations of the tree structure (relationships no longer propagate to all contained resources - see the ticket for more information), but it’s going to be a bigger deal going forward as we can start relying on the complete graph being passed around.

Both of these refactors could have been done with XLMRPC, but they would have required breaking compatibility, which is always a difficult step to take, so doing it all at once makes the most sense.

Easier Integration

The big deal in our use of REST is really an internal system that enables REST - this is the all-powerful Indirector. It’s essentially a plugin system that makes it easy to add new sources or destinations for data we care about. For example, when we wanted to queue the storage of catalogs to a database (rather than forcing the client to wait until it’s done), we had to develop the queueing infrastructure, of course, but integrating it with Puppet was less than 100 lines of code that knew how to accept catalogs and send them to the queue.

This ease of integration opens up all kinds of possibilities, like moving our CA to be database-backed, but the main thing is that if you come up with some cool integration, it’s similarly easy for you — you’re not dependent on us.

We’re still missing one piece - a user-configurable routing system to help you configure which plugin is used - but we’ve got that mostly done and it’ll be in the release after 0.25.

Note that this integration was often possible in earlier releases, but it was much harder because you had to know a lot more about every individual class you were integrating with. Because we use the same methods and the same interface and the same location pattern for everything, you don’t have to know as much.

Enforced Good Citizenship

Without the Indirector at the heart of everything, and an assumption that other people are going to be integrating with our code, we are forced to build more decoupled, portable components. This isn’t much, but it does always enforce a kind of good citizenship, which is something.

Using ‘git rebase’ to clean development histories

2009-04-13T06:54:11Z

In general, development in the Puppet world is a series of essentially disconnected batches of commits. We do a pretty good job of applying related commits all at once, so it’s obvious when a set of commits is related, but otherwise, we don’t have to worry.

Sometimes, though, multiple series of commits are related to each other, which can easily get lost. Even worse, multiple series of commits developed in tandem can cause downright painful development histories.

For example, we’re currently working on refactoring Puppet’s ActiveRecord integration while at the same time we’re adding the ability to queue the database store operations. We’ve had two development teams (of 1 to 3 people each) working on each feature, constantly publishing and rebasing against each others’ work. You could certainly argue that this isn’t the right model (we should have worked in serial rather than parallel, probably), time constraints didn’t allow this.

On Friday, we got to the point where we’re nearly done, and I started sending my code to the -dev list for review. That was straightforward enough, because I’d done my development in a separate branch and I still had those branches. The other team, though, had been pushing code around like made, tuning and modifying their code over time, so sending their commits out for review was harder. To top it off, our final branch was already a mixture of the different efforts.

So, I decided to see if I could clean up the development history. We had 80 commits spread all around, and I needed to reorder and squash them so they made easy sense during code review. (This is a pseudo-process without actual code, because the reality was too messy to reproduce here.) First I needed a list of the commits in the branch; git rebase is our tool through this process, and using it interactively (with ‘-i’) is the key. So, I made a new branch and started my rebasing:

git checkout dev

git checkout -b clean_dev

git rebase -i 0.24.8

This opens up my editor with an ordered list of all of the commits in my branch that aren’t in 0.24.8. There are three things you can do with commits in this list: Leave them alone, combine them with another commit, and delete them (which deletes them from the branch). Because I knew this would be a long complicated process, I saved the whole list to a separate file to start.

Some of the commits in our branch were backports of fixes we needed from the ‘master’ branch, so these were the only commits I left in the commit list in my first rebase. This added about six commits, and was pretty easy to merge. All of the other commits were just deleted from my clean branch.

The next step was to add my indirected ActiveRecord code. This was only four or five commits, but should have been collapsed into fewer than that (e.g., one of the commits fixed a misspelling in a test). There are multiple ways you could do this, including cherry-picking, but rebasing is definitely the most powerful. I created a new temporary branch in which to do my rebase:

git checkout -b clean_indirected_activerecord

git rebase -i clean_dev

This actually results in a noop, because I’m rebasing against a branch that’s a complete duplicate of my current branch. However, because I’m in interactive mode, I can do whatever I want from here, so I pasted in my four or five commits, and s/pick/squash/ where appropriate. Once I dealt with any merge conflicts, I then merged back into my clean branch:

git checkout clean_dev

git merge –no-ff clean_indirected_activerecord

I decided to force the merge commit to exist because, um, actually I don’t have a good reason. It seemed like a good idea to have a clear milepost saying that a given branch is merged, like that first email in a code review describing a patch series.

So now I’ve got a branch that has a series of patches that prepare the branch for us (with backported fixes, mostly), then a series of patches providing the first set of development. Now I just need to repeat this process for the other three development stages, one done by me and two done by others.

I had no problem with my other code; it was 8 or so patches, but I wrote them all, so I could easily handle merge conflicts. I also had no problem with one of the other chunks of code, because it was only three commits, so simple cherry-picking would have sufficed.

The last bit is where rebasing eventually broke down. In the end, I had 35 commits that I thought contributed something to the code (we had some duplicate commits in there, somehow, and some other commits that got cancelled out by later commits), but it looked like they should have been reduced to as little as four or five commits, because that’s about how many components were added in the code. However, I don’t know this code as well as the people who wrote it, so I decided to punt here and told them they needed to clean up their development path and send me some patches that had no duplication and no code that gets deleted in a later patch. My expectation is that they’ll create entirely new commits from the current state of the files, because the current commit history reflects a process rather than the desired state.

This whole process made me think of a discussion we had a while back on the -dev list (and a related thread started by Brice Figureau). Apparently Linux development maintains every commit series separately, and only merges them when it’s time for release. Or rather, the release involves a final merge. They maintain multiple ongoing development branches; one of them has all of the proposed patch series, but it’s never merged directly into the release branch. Instead, when a given patch series is accepted, it’s merged separately into the main branch.

This rebasing I did above made me realize the benefit of that approach - if the four development chunks had each remained separate branches, rather than merging early and merging often, it would have been *much* easier to keep them clean, and the developer responsible for a given chunk could always easily rebase just his or her own commits without affecting anyone else. Then, when it was time for release, I could just merge them all in the appropriate order and release.

This is pretty easy for four patch series, but obviously gets more complicated as we have tens of sets. I think for now, it’s too much work to maintain the patch sets in appropriate merge order without actually merging, but I think at some point, it really will make sense. At the very least, this process has taught me the value of rebasing early and rebasing often.

Puppet makes it into MacPorts

2009-03-29T21:56:19Z

Nigel Kersten just let me know that Puppet and Facter are now in MacPorts. That’s one more distribution (or in this case, pseudo-distribution) that Puppet’s a part of. Thanks Nigel.

RailsMachine Releases Puppet Rails Tool

2009-03-23T01:44:26Z

RailsMachine has announced their project Moonshine, which provides a pure Ruby interface to Puppet and is essentially custom-built to simplify Rails deployment and management:

One of the things that separates Moonshine from other solutions like Chef and Sprinkle is that out of the box, Moonshine comes with recipes for the same Ubuntu/Ruby Enterprise Edition/Apache/Passenger/MySQL stack that’s in production use at Rails Machine.

We’re pretty excited about this, for multiple reasons. It’s another Ruby company developing in and around Puppet, and it’s a great, simple way for Rails developers to take advantage of both Puppet and the RailsMachine stack.

The next step is to get the ShadowPuppet pure Ruby interface imported into Puppet. It will complement the existing language rather than replacing it, but we haven’t really settled all of the details yet.

Puppet wins Fukuoka Ruby Award

2009-03-23T01:22:22Z

Puppet was one of the winners of the Ruby Award handed out by the Fukuoka Prefecture in Japan. The Climate Information Toolkit won the top prize, and Puppet was one of three to win the second tier of prize.

Unfortunately, we could not travel to Japan to receive the award in person, but I was able to give a talk via Skype the night of the award ceremony. It was around 2am my time on a day I’d traveled with my wife and twins, so it was a long day, but it was worth it. I only hope my talk was worth anything.

Andrew did all of the hard work around getting the submission in and organizing the talk itself, so I definitely have him to thank for that.

Now I just need to figure out how to get my bank to accept those postal money orders from Japan.

The Most Free(tm) Way to Make Money from Open Source

2009-02-28T07:44:47Z

Tarus Balog is on a one-man campaign against open-core licensing, or really, any company that produces both open source and closed source software:

Of course, in the open core model there must be “commercially-available extensions” in order to get companies to sign a “commercial license”. Why is this? Because the open core product has been intentionally hobbled to force companies to buy the closed software product in order to get it to do the things that customers need it to do, and thus to generate revenue for the software company.

I find this article interesting, because he seems to have taken the opposite tack from me in terms of deciding what causes the most dilution of an open source product.

I’ve always figured that requiring copyright attribution is a greater sin than providing commercial add-ons, with the strong assumption that the product completely stands on its own without those add-ons.

As far as I see it, requiring copyright attribution restricts the developer community, while providing commercial add-ons doesn’t restrict anyone anywhere, it just says that some of my code isn’t free.

Sure, I can see if I produce a crippled, useless free product that *requires* the commercial add-ons that it’d be evil, but if the OSS portions actually do kick ass on their own, then what’s the issue?

To me, community is the big differentiator. If a given OSS project doesn’t actually care enough about a community, then open core is basically a sales tool. But if it *really* cares about community, then open core always has to worry that the community can thrive without the commercial portions. I basically think of this as requiring that the open core company always open source anything that’s required by the project or that’s essentially commoditized, but it leaves plenty of room for using a commercial license on those projects that not everyone needs, and especially on those areas that are clearly not commoditized and not many people need.

I think Tarus has a special perspective on OSS success because he’s in what amounts to an entirely commoditized space - there are so many monitoring tools with such a similar feature set that it’s crazy to think anyone would choose anything but an entirely open source solution. In less commoditized spaces, it likely doesn’t make quite as much sense to stick to entirely open licenses.

This bit in particular sticks out:

If you like your users why not provide all of the features as open source? Red Hat does it, JBoss does it, OpenNMS does it. The answer will always be that their business model can’t survive unless they sell closed source software. In that aspect, I can see little difference between open core companies and Microsoft.

Well, I can tell you that the reason that Reductive Labs is considering producing commercial software is that we can’t afford to produce much more software unless someone pays for the development, and at this point, we have a thriving, healthy community that largely gets huge benefit from Puppet without ever needing help from us. So, our options are to grow so slowly that all of the interesting opportunities pass us by, or to start producing software that allows us to, at the least, recoup our development costs.

I agree with Tarus’s basic sentiment: A lot of these open core companies aren’t actually producing open source software that you could reasonably use on its own. But that doesn’t invalidate the model, in my opinion.

His model of providing supported binaries, as Red Hat and others do, only works for those who use compiled languages, which means it’s right out for us. Maybe we should stick some C in there, just to make it easier to charge for support?

So what do you think: Is it a greater sin to only accept patches to your product if the contributor is willing to assign copyright to your commercial company, or to produce some closed-source code?

Summary of February 2009 Puppet Developer call

2009-02-05T18:13:16Z

We had another developer call last night, and until I can get the audio posted, here’s a basic summary.

Development Workflow

We led the discussion with a conversation about how the development workflow will change now that we’re finally releasing the code in master as 0.25. After much discussion, we largely concluded that the least-surprise solution was to use the ‘master’ branch for stable development, and have something like a ‘next’ branch for new features and major refactorings.

The main goal is for new developers to be able to clone our repository and get to developing code immediately without having to worry about switching branches or any such thing. More experienced developers will know where their development should be done, so it’s mostly a question of least-surprise for newcomers.

There was also discussion of fly-by-night developers, people who show up, produce a patch or two, and wander off. James Turnbull (who handles 99% of the merging and release management) said he was fine accepting patches over email, rather than forcing people to publish everything via git.

Other than those two changes, our development workflow has worked pretty well. However, the fly-by-night patches led into discussion of…

Core vs. Modules

Puppet’s core is starting to get many non-core features, like Nagios and Zenöss integration. It’s essentially impossible for the core team to maintain all of these extentions, but many of them were written as a one-off solution and won’t be maintained by their authors. Because of this, we as a project need to develop a means of splitting Puppet’s core away from all of these modules.

Discussion was had of using a Nagios-style ‘plugins’ repository, but I don’t like that idea because it leaves basically the same problem - a centralized repository that no one wants to maintain.

Instead, we all basically agreed that we want to focus on modules as the means of adding functionality to Puppet, but the big problem there is that the only way to do so is to add package-like behaviours to modules, and none of us wants to make our own package manager. Nonetheless, it was agreed that this was the only real option, so it just remains to do it.

Roadmap and Release Status

Against all odds, it looks like we’ll get 0.25 out in February - I’ll be merging in the last major refactor this week, and then it’s just a question of getting as many bug-fixes in as we can and dropping the release.

In addition, there are (annoyingly) a few important bugs in 0.24.7, so we’re unfortunately going to have to be put out a 0.24.8 release. One thing that’s become clear in the last couple of releases is that we need to get more community testing of release candidates - many of the bugs would have been caught by basic testing in the community.

Facter Design and Shadow{Facter,Puppet}

RailsMachine released ShadowPuppet and ShadowFacter in the last few weeks, and it’s led to a flurry of design conversations. We didn’t cover too much of it in the dev call, other than my reiteration that I’m excited by them and would like to merge them into Puppet and Facter.

Conclusion

And that was it. Please join us on the next call if you can.

Golden Image or Foil Ball?

2009-02-04T04:23:50Z

The essential basis of running services in “the cloud” is that they run in virtual machines, which come with their own idioms and practices for managing them. One of the mainstays of managing virtual machines (’VMs’) uses what’s called ‘golden images’. These are images built up complete with the services they’re supposed to run, so that you can just start them and they’ll immediately join the network and do whatever it is they’re supposed to do.

As the post title suggests, though, I think ‘foil ball’ is a more appropriate term.

You have significant problems when you rely on golden images: Image sprawl, updating your images, and image state vs. running state.

Image sprawl is what you get when the number of images (not running virtual machines) you have grows to an essentially unmaintainable figure. Let’s start with a simple LAMP stack: At the least, you’ll have a separate image for your web, database, and application servers. Oh, except you probably need a load balancer image. If you have any support services like DNS, you need an image for those. And so on. You soon find that you have a separate image for every service you provide.

Now that you’ve got this image sprawl, you run into the next issue: Updating these images is relatively expensive, and nearly always results in redundancy. It’s expensive because even trivial changes require a full image rebuild, which is itself a bit complicated. The redundancy comes because you *still* have to do some work on the image once it’s booted as a server, even if it’s minimal. So now you’ve got this complicated image generation process that has some kind of overlap with a simple on-server management process. Another kind of redundancy arrives when you make a change that affects multiple images (e.g., upgrading the same package, or performing the same configuration change): you have to make this change to each of these images separately.

Oh, and by the way - this updating process is usually completely unrelated to the process you use to update your non-image machines. Because hey, if a little bit of redundancy is good, then redundant redundancy is especially awesome.

Say you managed all of that, though, and all of your images are correctly updated all of the time. Great, now you just have to reboot every machine on your network to take advantage of the new changes. Of course, this isn’t exactly feasible for every machine all the time, which means you’ve got drift between the desired and actual configuration state.

This is why I think maintaining these images is more like managing a foil ball: It’s difficult to pull apart, difficult to press back together, and if you get too many of them they just get into the way.

If, instead, you use a single, base image for all of your work — I call these images stem cell images for what are hopefully obvious reasons - and then use a tool like Puppet to configure them once they’re running, you avoid all of the above problems: You have one image to maintain and it’s necessarily simplistic, you use the same tool and the same configuration base across all images, and Puppet keeps your machines updated within 30 minutes of any central change.

So, if someone tries to sell you a golden image, don’t buy it - instead choose a tool you can use for every machine in your organization, and push every configuration operation possible into that tool, rather than spreading tasks around to your provisioning, image management, and configuration management tools. This is just as true for tools like Jumpstart and Kickstart - they should do as little as possible, and hand off immediately to a tool like Puppet; well, really, just Puppet.

Puppet on the IT Management Podcast

2008-12-22T15:46:44Z

I was a guest on last week’s IT Management Podcast again last week, and we ended up talking a lot about Puppet and the difficulties in running an open source software company. As always, John Willis and Coté are informed and interesting. Give it a listen, and maybe subscribe to the whole series.