bitfluent

Creating a Custom dpkg Search Index for Chef 0.7.x

Tue, 03 Nov 2009 23:14:40 +0800

I wanted to search across all my nodes that they have the latest Chef client package installed. The default indexer doesn’t index installed debian packages, so I whipped out a custom dpkg indexer based on the sample code from the wiki.

Here it is:

It’s a lot longer than the sample because I had to do the whole client authentication dance (most of it was copied directly from lib/chef/client.rb). Good news is that authentication will change dramatically in the upcoming Chef 0.8.x release which should simplify creating custom clients.

Once you run the indexer, your search screen on chef-server will now see a new query form for dpkg. Here’s an example of querying for packages that match the name attribute “chef” and returning only the “version” and “status” attributes.

Railscasts Theme for RubyMine

Sun, 27 Sep 2009 14:07:22 +0800

I ported the Ryan Bates’ Railscasts TextMate theme to RubyMine because I love it so damn much. You’ll notice some differences due to how RubyMine’s lexer works, but it’s mostly serviceable.

My wishlist for RubyMine:

add more scopes for finer-grained highlighting
support setting opacity
use the Mac’s native color picker

Get the RubyMine Railscasts theme

Screenshots comparing the theme on TextMate vs. RubyMine.

Ruby

ERB

Javascript

CSS

YAML

Get the RubyMine Railscasts theme

Using Chef Server Indexes as a Simple DNS

Fri, 25 Sep 2009 23:11:00 +0800

I’ve been setting up a small cluster of nodes for a client on Amazon EC2. For each node that is brought up, an internal hostname is attached to it to make it easier to address, for example, db0, db1, admin and so forth. To manage all this, I had been manually editing /etc/hosts on each node whenever new nodes and/or roles are added to the mix. As you can imagine with EC2, it is all too easy to spin up new instances when the need arises, so I needed a more sustainable solution.

The common approach is to run a private internal DNS server. As these machines are configured with Chef, 37signal’s djbdns::autozone recipe looked like a good starting point. It uses Chef Server Indexes to seed djbdns by querying all nodes and extracting the IP Address, FQDN and a custom DNS Aliases attribute. Nodes running chef-client keeps the Chef Server Indexes up to date everytime it connects to the server on a predefined interval. This makes it possible for djbdns to be automatically maintained as new nodes are spun up and added to Chef Server.

I also came across a much simpler solution in Tim Dysinger’s Using Amazon EC2 Metadata as a Simple DNS. In this approach, he crons a query to Amazon EC2’s metadata to rebuild /etc/hosts every hour. I like the simplicity of it (very similar to what I’ve already been doing by hand) and not having to run yet another daemon. However, it required the EC2 secret and key which my client may or may not be willing to share.

The solution I came to is a hybrid of the two. Instead of querying EC2’s metadata, I queried the Chef Server Indexes. Instead of cron, I relied on chef-client’s interval. Instead of djbdns, I used the hosts file.

Here’s how it looks like:

cookbooks/hosts/recipes/default.rb

#
# Cookbook Name:: hosts
# Recipe:: default
#
# Copyright 2009, Bitfluent
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# 
#     http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

hosts     = {}
localhost = nil

search(:node, "*", %w(ipaddress fqdn dns_aliases)) do |n|
  # node own's record, store in localhost
  if n["ipaddress"] == node[:ipaddress]
    localhost = n
  else
    hosts[n["ipaddress"]] = n
  end
end

template "/etc/hosts" do
  source "hosts.erb"
  mode 0644
  variables(:localhost => localhost, :hosts => hosts)
end

cookbooks/hosts/templates/default/hosts.erb

# Auto-generated by chef for <%= @node[:fqdn] %>
#

127.0.0.1 localhost <%= @localhost["fqdn"] %> <%= (@localhost["dns_aliases"] || []).sort.join(" ") %>
<% @hosts.keys.sort.each do |ip| %>
<%= ip %> <%= @hosts[ip]["fqdn"] %> <%= (@hosts[ip]["dns_aliases"] || []).sort.join(" ") %>
<% end %>

A few things to note:

I use a hosts hash because for some reason, the server indexes were returning duplicate nodes
I output sorted IPs and dns aliases to keep /etc/hosts stable. Hashes are themselves not sorted. I didn’t want Chef to replace the files if no new nodes or aliases were added.

Happy cooking!

Installing CouchDB 0.9.0 Ubuntu Karmic Package on Jaunty

Fri, 04 Sep 2009 14:53:00 +0800

Add Karmic into /etc/apt/sources.list.

deb-src http://us.archive.ubuntu.com/ubuntu karmic main restricted universe multiverse

Run apt-get update.
```
$ sudo apt-get update
```
Because Karmic is so bleeding edge, the 0.9.0 package has been superseeded by 0.10 snapshots. You can locate the binary builds for 0.9.0-2ubuntu5 at https://launchpad.net/ubuntu/+source/couchdb/0.9.0-2ubuntu5

Download the package.

$ wget http://launchpadlibrarian.net/29926161/couchdb_0.9.0-2ubuntu5_i386.deb

0.9.0 requires libicu40. Jaunty only ships with licicu38. So we need to build and install libicu40 from Karmic’s source.
```
$ sudo apt-get build-dep libicu40
$ sudo apt-get -b source libicu40
$ sudo dpkg -i libicu40_4.0.1-2_i386.deb
```

Karmic’s couchdb package requires Erlang R13B1. Jaunty ships with R12B5.

$ sudo apt-get build-dep erlang-base-hipe=1:13.b.1-dfsg-2
$ sudo apt-get -b source erlang-base-hipe=1:13.b.1-dfsg-2
$ sudo dpkg -i erlang-base-hipe_13.b.1-dfsg-2_i386.deb erlang-crypto_13.b.1-dfsg-2_i386.deb erlang-mnesia_13.b.1-dfsg-2_i386.deb erlang-public-key_13.b.1-dfsg-2_i386.deb erlang-runtime-tools_13.b.1-dfsg-2_i386.deb erlang-ssl_13.b.1-dfsg-2_i386.deb erlang-inets_13.b.1-dfsg-2_i386.deb erlang-xmerl_13.b.1-dfsg-2_i386.deb

Install the last few extra packages needed. Thankfully, Jaunty has them at the required versions.

$ sudo apt-get install dictionaries-common hunspell-en-us libhunspell-1.2-0 libjs-jquery libnspr4-0d libnss3-1d libpython2.6 libstartup-notification0 xulrunner-1.9

Finally, install the couchdb package.

$ sudo dpkg -i couchdb_0.9.0-2ubuntu5_i386.deb

Git Tip: Ignoring Icon\r in .gitignore

Fri, 28 Aug 2009 17:34:28 +0800

If you have a git repository at the root level of a writable disk image, you may notice that Mac OS X litters it with various junk.

MBP:foo(master) kamal$ git status
# On branch master
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#   .DS_Store
#   .VolumeIcon.icns
#   .fseventsd/
#   "Icon\r"

Three out of four of the files above is an easy .gitignore entry away. The last one, "Icon\r", on the other hand, is quite a bitch. I tried various permutations of it in .gitignore with no luck. Icon, "Icon", "Icon\r", nothing worked.

Solution

The trick is to use the literal ^M control character in the name. Yup, Apple decided that the file name should contain an actual CRLF! But there’s a trick to this trick: you need two ^M characters. Credits go to this post for the solution.

Here’s how your .gitignore should look like in vi:

.DS_Store
.VolumeIcon.icns
.fseventsd
Icon^M^M
~
~
~
~

A quick check,

MBP:foo(master) kamal$ git status
# On branch master
nothing to commit (working directory clean)

All clean!

Those Are Not Caret-Ms

It took me a while to figure out how to create a control character. Everything I searched on Google was concerned with removing them. I even booted up Windows XP in VirtualBox to create a text file in Notepad in the hopes of copy-and-pasting it.

Finally, in a fit of desperation and being the respectable Rubyist that I am, I fired up irb and wrote that mofo to the .gitignore file directly.

>> f = File.open(".gitignore", "a+") # append
=> #
>> f.write("Icon\r\r")
=> 8
>> f.close
=> nil

There! Done! Err, maybe someone can tell me the easy way to do it. What’s the keystroke in vi?

Anyway, if you can’t be arsed, let me save you the trouble. Here’s a copy of my .gitignore. Download, rename and stick it in your repo.

Make It Global

Here’s a little tip: create the .gitignore in ~ to make it apply across all repos. It’s additive — git status will use the contents of both the local .gitignore and ~/.gitignore. My strategy is to ignore project-specific junk like *.log and database.yml in the repo itself and general annoyances like above in ~/.gitignore. In a later post, I’ll put up examples of each and explain them line by line.

What’s With the Disk Image in the First Place?

Being paranoid, I’ve begun to store client work on encrypted disk images. I can’t stress this enough: if you’re a developer, it’s your responsibility to secure your client’s source code in the event of theft! Can you imagine the shit storm and liability you’ll face if your laptop got stolen?

To manage the disk images, I’m currently trialing Knox. It does cost some money even though it uses the same backend you get for free via FileVault and Disk Utility. However, the difference with Knox-created images is the disk images grow on demand and it makes it really easy to switch between images. FileVault, on the other hand, applies it to the entire home directory while Disk Utility images take up the entire space at creation time.

Hope this entry helps someone out there.

Intro to Ruby on Rails Charity Tutorial Survey

Tue, 25 Aug 2009 13:00:33 +0800

Here’s the plan: I’m going to run a 1-day Introduction to Ruby on Rails tutorial. I won’t charge a single cent for this tutorial. Instead, you will be asked for a minimum donation to be given to a charity. I haven’t worked out the details yet, but most likely I’ll hold this in October in conjunction with the FOSS.my 2009 Conference.

Would you be interested in this? Help me out by filling in this short survey so that I can plan and customize the tutorial for you. Much thanks!

Retweet this

Tethering over Celcom 3G on the iPhone 3.0

Thu, 18 Jun 2009 17:42:00 +0800

Open this link in MobileSafari - my_celcom.mobileconfig and install the profile. Alternatively, you can download this file and mail it to yourself and open it up in the iPhone Mail.app.

Credits go to http://help.benm.at/help.php. I downloaded the settings for SG telcos from that site to see which parts were customizable and pretty much found you just need to change the APN to use celcom3g.

Also, for you people on DiGi: DiGi.mobileconfig courtesy of Snuffykl from LYN.

EDIT: I’m on the RM68/month unlimited data plan. Is it the cheapest unlimited 3G plan around?

EDIT: I can confirm that tethering and MMS does not work on the iPhone 2G (1st-gen) right now. It will probably require jailbreaking which should be out on Friday, 19th June.

EDIT: “Does not work” means the MMS and tether options are nowhere to be found in Settings.

Are You Coming To #barcampkl?

Tue, 31 Mar 2009 16:26:11 +0800

#barcampkl is happening this weekend, 4-5th April 2009, at Inti College Subang. Some pretty interesting talks have been scheduled ahead of time although the organizers have promised to implement a “real” unconference where the talks are voted at the opening of each day.

So, are you coming? Register here.

Secure and Signed AuthSub Requests in Ruby

Wed, 18 Mar 2009 23:21:00 +0800

I pulled the trigger in my last blog post about solving Google AuthSub’s warning. It solved only half the problem - the not-showing-the-warning half. I was too quick to assume it would work since Google returned a token. Actually performing GET on the contacts feed would yield me a 401 Unauthorized.

So, what did I get wrong? A few things.

Firstly, I didn’t have to set the sig in the request authorization URL. A simple secure=1 would have been sufficient. I confused this concept with other request schemes like Facebook’s which signs the request and appends the sig to the URL parameters.

Secondly, for secure AuthSub requests, you need to set special signed headers. Regular non-secure AuthSub requests only need a short header.

Authorization: AuthSub token="COy2q9qdGRDWm7iMAm"

However, if you want secure requests, you need to sign the request. It’ll look something like this.

Authorization: AuthSub token="COy2q9qdGRDWm7iMAm" sigalg="rsa-sha1" data="GET http://www.google.com/m8/feeds/contacts/default/thin?max-results=200 1237370831 18094396511823580603" sig="XOQNfKpQ8VPCN2Yp+Zt="

I thought I nailed it but it still returned me 401 Unauthorized. I downloaded the Python GData client to compare the sig values and indeed the sig I generated was off.

So the final piece of the puzzle is how do I correctly generate the signature. google-authsub (and my contacts commit) got it wrong. This is what it was doing:

digest = OpenSSL::Digest::SHA1.new(data).hexdigest
sig = [@@pkey.private_encrypt(digest)].pack("m")  #Base64 encode

After much googling, I found the answer. Immad Akhund posted the solution on the Google Data Protocol mailing list in June 2008. The correct way to sign data is to use the OpenSSL::PKey::RSA#sign method and remove the newlines after Base64 encoding.

sig = @@pkey.sign(OpenSSL::Digest::SHA1.new, data)
sig = [sig].pack("m").gsub(/\n/, "") #Base64 encode

So, there you have it. I’ve pushed a new commit to the contacts library with all the fixes. I’ll be forking google-authsub and pushing fixes there too in hopes that another person wouldn’t need to spend a day figuring it out like I did.

Solving "This website has not registered with Google to establish a secure connection for authorization requests"

Wed, 18 Mar 2009 01:47:00 +0800

I recently ran into an ominous warning on the Google Contacts Access Request landing page.

After some googling, the solution is to register the requesting domain and upload a self-signed X.509 certificate.

Once completed, I retried the request and got a slightly less threatening warning.

The solution was to add two additional parameters to the request: secure=1 and a signature. Fortunately, I found how to generate the signature in the google-authsub gem. A few minutes later, I added support for signing AuthSub requests to Mislav’s contacts library and got the results I wanted. Commit.

Update: I got it wrong about appending the signature to the parameter. I posted a follow up on how to correctly perform signed AuthSub requests in Ruby.

#rubinius

Thu, 22 Jan 2009 17:28:00 +0800

evan: i'm curious, how do you know the JVM is deopt'ing in certain cases?
headius: it tells us
evan: twitter?
evan: us post?
evan: collect call?
headius: reaches out of the screen and slaps us sideways
evan: hah
headius: there's a bunch of debug options for hotspot
evan: NOT SO FAST BUCKO
headius: some in the released jdk, some require a debug build
headius: LogCompilation, PrintInlining, PrintAssembly
evan: -XXSlapRatio=1persecond

Sharing Contracts

Tue, 23 Dec 2008 12:41:00 +0800

I love it when companies that provide professional services share the contracts they use when dealing with clients. Many people consider contracts to be part of their secret sauce and competitive advantage over their competition so it’s understandable that these documents are not discussed much (plus they also cost money having to go through legal counsel).

However, Obie Fernandez of HashRocket and Andy Clarke of Stuff and Nonsense have been awesome by sharing the contracts they use in their daily business:

Master Services Agreement Part 1
Master Services Agreement Part 2
Contract Killer (via @cheeaun)
Flavert Media Lab’s Contract (thanks @tekong!)

I’d love to collect more of these kinds of write-ups. Do you have some? Twitter me!

Test autoposting from Posterous

Sat, 20 Dec 2008 15:14:31 +0800

I never knew posterous could autopost to other services as well. That’s pretty rad.

Posted via email from kamal’s posterous | Comment »

Full List of 1,339 Rails Contributors

Fri, 31 Oct 2008 15:42:00 +0800

One of the biggest plus points of using git for open source projects is the preservation of the original author of the patch. Here’s how it looks like when displayed in GitHub:

In the Subversion days, patches into Rails were attributed by adding an arbitrary combination of the author’s name/email/nick at the end of the commit messages. Makes it kind of hard to keep track of the number of distinct contributors over the lifetime of the project.

Fortunately, Xavier Noria whipped up a script to parse the legacy commit messages to extract the number of commits per author using these rules:

First extract authors from commit message
If empty, check changelogs via git show id
If empty, author is the committer

The full list is can be found on the Rails Contributors page.

Tumblr Client On The iPhone

Sat, 09 Aug 2008 23:43:54 +0800

Would having a tumblr client on my iPhone see me posting more frequently? Obligatory test post.

Posted with LifeCast

On the Beauty of Rubinius' Design (I Wish I Had Rails.new)

Mon, 02 Jun 2008 17:13:00 +0800

In Rubinius, you can spawn off a brand new complete VM by simply calling Rubinius.new. It’ll behave exactly as though it was invoked directly from an rbx binary sitting in your $PATH, complete with STDIN/STDOUT (which you can override). I believe this is one of the basis of how Rubinius’ multi-VM architecture works.

Anyway, I bring this up because I really, really wish Rails was architected in a similar fashion. I am building a CMS on top of Rails and would love to get my hands on a Rails.new if there ever was one. Here’s why.

In a CMS setting, very little of what Rails offers out of the box is usable. You don’t have access to Rails routes so from the very beginning, you don’t have Rails automatically invoking the right controller, the right action and rendering the right view. This doesn’t make sense anyway - you don’t expect your CMS users to start writing controller code in your web editor, do you? (Unless you are Heroku.)

On a slight tangent, the assumption when writing a CMS is that when you ship, you would have written every conceivable controller and model there is (views don’t fall into this because users are generally familiar with the concept of customizable templates). One strategy to extend your “frozen code base” is via the use of widgets and third-party apps (like Facebook) so that you can create seemingly new pages served by custom controllers.

So how do you design a CMS? I’ve been prototyping something for the past week and came up for a breather to check out how other people have solved it. I am delightfully surprised to find out that Radiant does it very close to what I have. In particular, Radiant has one single controller that accepts all requests (lets ignore the entire admin portion for the time being). Based on the path array (provided by the globbed route), it decides what to do / where to dispatch. It takes care of locating the page that corresponds to the URL (it uses a Page model), rendering it and returning the result to the user. It is interesting to note that Radiant directly manipulates the request and response objects, something that Rails developers almost never had to reach for in a regular app. On the other hand, I am exploring the use of serializing the templates to disk and simply calling render :template on it.

Wait a minute. Holy cow, we just built (a simplified) Rails on top of Rails!

What I’d love to see here instead is a Rails.new method just like Rubinius. Boom, a full blown MVC at your fingertips. Configure it right and there you have your very own CMS with minimal work. Or maybe there is. Lazyweb?

An update in a really long while

Fri, 02 May 2008 03:53:52 +0800

Woah, one month with no updates.

I started contracting for a startup in Seattle, WA since beginning of April and have been neglecting to update stuff. It’s pretty nice here plus I get to access things that people here take for granted like attending Seattle.rb hack nights, Pandora, Hulu and this weekend, Barcamp Portland!

See you guys in a bit.

http://dev.rubyonrails.org/ticket/11491

Tue, 01 Apr 2008 12:03:00 +0800

http://dev.rubyonrails.org/ticket/11491:

Finally, render :partial => some_collection will pick the right template based on each object in the collection. I can now retire my workaround:

<% some_collection.each do |item| %>
  <%= render :partial => item %>
<% end %>

Best part is that the partial_counter is maintained across the different partials.

Updating Counter Cache in Migrations

Sun, 30 Mar 2008 01:34:00 +0800

I’m nearly complete with an app I’ve been working on, so I thought I’d dedicate some time for optimization. One of the first things I did to my models was to add counter caches so that performing counts on my association were fast.

First try,

class AddCommentCounterCacheOnTopics < ActiveRecord::Migration
  def self.up
    add_column :topics, :comments_count, :integer, :default => 0
  end

  def self.down
    remove_column :topics, :comments_count
  end
end

Very standard. However, it initializes the column to 0, effectively “losing” all my comments (they’re there in the DB, but Rails adds an additional optimization whereby it won’t fetch the association if the counter cache is 0). Looks like I need to update the comments_count column to whatever it was at the time of migration.

Second try,

class AddCommentCounterCacheOnTopics < ActiveRecord::Migration
  def self.up
    add_column :topics, :comments_count, :integer, :default => 0

    Topic.find(:all).each do |t|
      t.comments_count = t.comments.count
      t.save!
    end
  end

  def self.down
    remove_column :topics, :comments_count
  end
end

Two things to note: First, I’m using the #count method in t.comments.count because calling #size will use the value in t.comment_count which is 0. #count on the other hand will perform a SQL count(). Second, this won’t work! Why? Because counter cache columns are set to attr_readonly.

To work around this, a little hack. Final result,

class AddCommentCounterCacheOnTopics < ActiveRecord::Migration
  def self.up
    add_column :topics, :comments_count, :integer, :default => 0

    def Topic.readonly_attributes; nil end # A little evil hack

    Topic.find(:all).each do |t|
      t.comments_count = t.comments.count
      t.save!
    end
  end

  def self.down
    remove_column :topics, :comments_count
  end
end

Excellent panel on scaling customer support. Definitely...

Mon, 24 Mar 2008 11:56:00 +0800

Excellent panel on scaling customer support. Definitely something to take into account whenever you think of launching a new product out there. Are you too caught up in the idea and the development, forgetting the most important ingredient of your success: the users?