A Single Programmer's Blog

Puppet Debugging: --noop

2012-03-09T00:05:43-05:00

I do more and more Puppet, and I really like it. It’s a simple way to declare the desired state of the world. As I do more and more with Puppet, I needed to debug my manifests: check that they were doing the expected thing. I was looking for some kind of dry run option, similar to apt-get‘s or other tools, but puppet help apply didn’t provide any helpful hints. Until I stumbled upon Puppet dry run.

A direct run with --noop --test failed because --test was not recognized, but --noop was!. A sample run with --noop looks like this:

1 # puppet apply puppet/hosts/db03.staging.internal.pp  --noop
2 notice: /Stage[main]//Postgresql::Database[svreporting_staging]/Exec[createdb svreporting_staging]/returns: current_value notrun, should be 0 (noop)
3 notice: Postgresql::Database[svreporting_staging]: Would have triggered 'refresh' from 1 events
4 notice: /Stage[main]//Cron[replace svreporting_staging with svanalytics_staging]/ensure: current_value absent, should be present (noop)
5 notice: Class[Main]: Would have triggered 'refresh' from 2 events
6 notice: Stage[main]: Would have triggered 'refresh' from 1 events
7 notice: Finished catalog run in 10.86 seconds

Check the wording: Would have triggered, and should be. These are great ways to know what Puppet will do for you. There is only one caveat: when you need to spend more than a few minutes debugging your recipes, disable the automatic run. The scheduled run from puppet agent may go behind your back and apply your recipe, which you’re just debugging. In my case, this is very easy to do since I host my puppet manifests in a Git repository, which is pulled hourly and applied.

I’m only waiting for my pull request to be applied so you don’t have to go through the same process as I did.

Scala Case Classes and State Machines Using Akka Actors

2012-02-29T21:04:54-05:00

At Seevibes we use the Twitter Streaming API to harvest what people say about television shows. The Twitter Streaming API docs state that:

Upon a change, reconnect immediately if no changes have occurred for some time. For example, reconnect no more than twice every four minutes, or three times per six minutes, or some similar metric.

From Updating Filter Predicates

The way my system is architected, whenever we change the list of keywords, a ShowDataErased message is sent, followed by a bunch of ShowKeywordsReplaced events, one per show. The system uses CQRS where each change that occurs is reflected in the system through one or more events, hence the two events above.

Here’s my state machine in all it’s gory details:

What the machine does is:

When streaming, and time passes, nothing happens;
When streaming and we change keywords (either reset or add keyword), we enter a state where we’re pending some changes;
When we’re pending, and we change keywords (reset or add again), we stay in the pending state;
When we’re pending, and insufficient time has passed, then we must stay in the same state, in case we receive more keyword change events;
Finally, if we’re pending and sufficient time has passed, we just go back to the streaming state.

The obvious first test is to start with the fact that when we’re streaming, we must still be streaming. Here’s an implementation of this test.

 1 import org.junit.Test
 2 import org.scalatest.Assertions
 3 import org.joda.time.Instant
 4 
 5 class StreamingStateMachineTest extends Assertions {
 6     val KEY2 = Set("b")
 7 
 8     @Test
 9     def whenStreaming_andReceiveTick_thenShouldStillBeStreaming() {
10         expect(Streaming(KEY2)) {
11             Streaming(KEY2).tick(new Instant())
12         }
13     }
14 }

For the Rubyists out there, case classes are a kind of class that has a few nice properties:

They act like methods, thus we can call the Streaming method (which the compiler turns into a call of Streaming.apply());
They have an unapply() method, which enables pattern-matching (which I won’t use in this case);
They are serializable, externalizable, clonable, have a free equals() and hashCode() implementation that is correct and consistent with the Java equals() and hashCode() guidelines.

Case classes are an ideal vehicle to transport data with one or more methods. One thing that you must realize when using case classes is that you must use only immutable objects as parameters to your case classes. If you used a mutable object, your case class would suddenly be mutable, and the equals() and hashCode() methods would be useless for you. The Scala compiler has no way to enfore this.

Getting back to our business, the obvious implementation is a no-op:

1 import org.joda.time.ReadableInstant
2 case class Streaming(keywords: Set[String]) {
3     def tick(now: ReadableInstant) = this
4 }

One event down, 6 to go! Let’s change state : go from the streaming state to the pending keywords change state. I’ll start with the simpler case: resetting the keywords list. Here’s an implementation:

1 @Test
2 def whenStreaming_andReceiveReset_thenShouldBePending() {
3     expect(PendingKeywordsChange(Set.empty[String], now)) {
4         Streaming(KEY1).resetKeywords(now)
5     }
6 }

Notice that I’m passing explicit times to my methods. This is to make testing much easier. It doesn’t really bother me to send the current time along. It’s certainly easier than sending a factory which returns the current time, which would add a level of indirection, and be more complicated.

The implementation is also pretty simple:

1 def resetKeywords(now: ReadableInstant) =
2     PendingKeywordsChange(Set.empty[String], now)

And so on for the other implementations of Streaming. The guts of the state machine are really in the PendingKeywordsChange class. First, a test:

 1 @Test
 2 def whenStreaming_andReceiveAddKeyword_thenShouldBePending() {
 3     expect(PendingKeywordsChange(KEY1 + "b", now)) {
 4         Streaming(KEY1).addKeyword("b", now)
 5     }
 6 }
 7 
 8 case class PendingKeywordsChange(keywords: Set[String],
 9                                  lastChangedAt: ReadableInstant) {
10 
11     def addKeyword(keyword: String, now: ReadableInstant) =
12         PendingKeywordsChange(newKeywords + keyword, now)
13 
14 }

When I wrote the code, I was really excited by how concise and clear my intents shined through the code. The final method I’d like to show is the tick() method on PendingKeywordsChange:

 1 @Test
 2 def whenPending_andReceiveTick_andInsufficientTimeHasPassed_thenShouldStayPending() {
 3     expect(PendingKeywordsChange(KEY1, oneMinuteAgo)) {
 4         PendingKeywordsChange(KEY1, oneMinuteAgo).tick(now)
 5     }
 6 
 7     expect(PendingKeywordsChange(KEY1, twoMinutesAgo)) {
 8         PendingKeywordsChange(KEY1, twoMinutesAgo).tick(now)
 9     }
10 
11     expect(PendingKeywordsChange(KEY1, twoMinutesAgo)) {
12         PendingKeywordsChange(KEY1, twoMinutesAgo).tick(twoMinutesAgo)
13     }
14 }
15 
16 @Test
17 def whenPending_andReceiveTick_andSufficientTimeHasPassed_thenShouldBeStreaming() {
18     expect(Streaming(KEY1, now)) {
19         PendingKeywordsChange(KEY1, twoMinutesAgo.minus(TimeUnit.SECONDS.toMillis(1))).tick(now)
20     }
21 }
22 
23 // in class PendingKeywordsChange
24 def nextChangeAt = lastChangedAt.toInstant.plus(TimeUnit.MINUTES.toMillis(2))
25 
26 def tick(now: ReadableInstant = new Instant()) =
27     if (now.isAfter(nextChangeAt))
28         Streaming(newKeywords)
29     else
30         this

The final part is where and how I regularly call the tick event. Because the rest of my infrastructure is tied to Akka actors, I used the Akka scheduler API to send a message to an actor, which hid my state machine behind a nice and consistent facade:

 1 import akka.actor.Actor
 2 
 3 case object Tick
 4 case object ResetKeywords
 5 case class AddKeyword(keyword: String)
 6 
 7 trait State {
 8     def resetKeywords(now: ReadableInstant): State
 9     def addKeyword(keyword: String, now: ReadableInstant): State
10     def tick(now: ReadableInstant): State
11     def keywords: Set[String]
12 }
13 
14 case class Streaming(keywords: Set[String],
15                      lastChangedAt: ReadableInstant) extends State {
16     def resetKeywords(now: ReadableInstant) =
17         PendingKeywordsChange(Set.empty[String], now)
18 
19     def addKeyword(keyword: String, now: ReadableInstant) =
20         PendingKeywordsChange(Set(keyword), now)
21 
22     def tick(now: ReadableInstant) = this
23 }
24 
25 case class PendingKeywordsChange(keywords: Set[String],
26                                  lastChangedAt: ReadableInstant) extends State {
27     def resetKeywords(now: ReadableInstant) =
28         PendingKeywordsChange(Set.empty[String], now)
29 
30     def addKeyword(keyword: String, now: ReadableInstant) =
31         PendingKeywordsChange(keywords + keyword, now)
32 
33     def nextChangeAt = lastChangedAt.toInstant.plus(TimeUnit.MINUTES.toMillis(2))
34 
35     def tick(now: ReadableInstant = new Instant()) =
36         if (now.isAfter(nextChangeAt))
37             Streaming(keywords)
38         else
39             this
40 }
41 
42 class KeywordsStateMachine extends Actor {
43     private var state: State = Streaming(Set.empty[String], new Instant())
44     private var currentKeywords = Set.empty[String]
45     private val stream = ... // Twitter4J Streaming API
46 
47     def receive = {
48         case Tick =>
49             state = state.tick(new Instant)
50             if (state.keywords != currentKeywords)
51                 // reset stream with new keywords
52 
53         case AddKeyword(keyword) =>
54             state = state.addKeyword(keyword, new Instant())
55 
56         case ResetKeywords =>
57             state = state.resetKeywords(new Instant())
58     }
59 }
60 
61 object Main {
62     def main(args : Array[String]) {
63         val keywordsStateMachine =
64             Actor.actorOf[KeywordsStateMachine].start()
65 
66         // Send the Tick message to the keywordsStateMachine now,
67         // and every minute thereafter
68         Scheduler.schedule(keywordsStateMachine, Tick, 0, 1, TimeUnit.MINUTES)
69     }
70 }

A few things contributed to the clarity of the implementation:

The Joda Time library is really easy to use and understand. You don’t need to use java.util.Date or java.util.Calendar : ditch them as soon as you can;
Scala’s case classes reduced boilerplate: no new operator in sight;
Public by default enhances clarity by removing keywords where they don’t matter;
Libraries, yes, but not when something simple is required. Clarity of implementation before reuse.

Note that this state machine does not have to deal with dropped connections, or rate limiting or anything of the sort, because it’s all hidden behind Twitter4J’s interface. The resulting state machine is much easier to understand and reason about.

In the interest of fun, profit and learning, I’ve made available a GitHub repository with similar code to what’s here at streaming-state-machine. The code is different because it doesn’t actually connect to Twitter, and thus only needs to demonstrate how changing the states works. The time scale was changed from minutes to seconds.

Scala Hackaton and SQL Workshop Wrap Up

2012-02-06T14:19:51-05:00

A great time was had by all!

Seriously, the Scala workshop was very cool. The actor model was new to almost all attendants and needed some explaining. Looking back, I should have explained a bit more about how the this works. From what I saw, nobody stumbled upon the Scala syntax that much. The people that were there were all experienced programmers, thus may not have needed much hand-holding regarding syntax.

Martin Provencher, organizer of Montreal.rb, and his teammate, Olivier Melcher, wrote an actor-based histogram producing word counter that counted which programming languages were most spoken of. In their tests, Ruby and PHP were the most frequent ones.

The SQL Workshop was more difficult for all. The first exercise was much harder than I anticipated. The first question was:

Calculate the top 5 shows, ordered by social impressions, for the period between 2011-10-17 and 2011-10-23. Return an answer under 3 seconds.

First, people had to get familiar with the data model. I asked people to get the data in under 3 seconds, thus was more of an optimization problem, rather than just getting the correct query. I allowed about 40 minutes for this exercise, and most of the time was used in getting a correct query. Nobody had time to get to the optimization part. Looking back, I’d make this question be two parts: first write the query, then optimize it, or provide the correct query and let people optimize it.

I hope everybody learned a lot by coming to the hackaton.

A big “Thank You” to the event’s sponsors for the beer and pizza:

Notes for Hackaton organizers

If you can, provide a starter application or package. All attendees were happy they could get up and running with little to no fuss. It saved a lot of time because they didn’t have to resolve dependency nightmares.

Move around the room, look over everybody’s shoulders. Without fail, when I walked up to a team, they’d have a question for me.

Speaking of teams, it’s useful for people to work in teams. I’d say this harkens back to Pair Programming with a Novice / Novice pair. Two novices working together can get further along than a single novice. One of the members will have an insight that the other member didn’t have, and both can move forward from there.

Quick Puppet Tip

2012-01-24T09:46:14-05:00

While testing my Puppet recipes, I usually boot a new instance then apply configuration manually. I have a base configuration, which is extended with other things. I just noticed I could apply a manifest using STDIN:

1 # ( echo "include sysbase" ; echo "include rabbitmq" ) | puppet apply -v

Viva the Unix tradition!

If you do any kind of configuration management, I highly recommend Puppet.

Needium Joins Seevibes for Beer

2012-01-23T10:02:05-05:00

I just learned this morning that Needium will join Seevibes in sponsoring the beer at the Scala Hackaton and SQL Workshop. Both events are on February 2nd. If you haven’t already done so and wish to join us, please register at EventBrite:

Hope to see you there!

Announcing Scala and SQL Event on February 2nd in Montreal

2012-01-18T22:14:54-05:00

I would like to invite you to attend one or two events on February 2nd: Analyzing Twitter Social Data using Scala and Akka Actors and Social Media Metrics using SQL Engines.

Schedule

2:00 PM Doors open
2:30 PM Hackaton — Analyzing Twitter Social Data using Scala and Akka Actors
5:00 PM Beer & Pizza, sponsored by Seevibes
6:30 PM Workshop — Social Media Metrics using SQL Engines
9:00 PM Socializing

The event will be at the Notman House, 51 Sherbrooke W. We have a limited number of places available, so be sure to reserve your seat now. There are two events, both are free to attend, and both are bilingual French and English.

Registration

Reserve for Analyzing Twitter Social Data using Scala and Akka Actors
Reserve for Social Media Metrics using SQL Engines

Setup

Before you come in, please be sure to follow these instructions to get you started:

Scala Hackaton

The Scala Hackaton is an event where you’ll build whatever you wish: word counter, word frequency, hashtag frequency, etc. You get to choose. There will be people more familiar with Scala and Akka at the event which can help you. The first 30 minutes of the event will be reserved for a quick introduction to Scala.

Clone the scala-hackaton Git repository, or download a ZIP
In the repository / project, run mvn test and mvn exec:java -Dexec.mainClass=seevibes.HelloActor

The two Maven steps are to download all necessary dependencies. If you don’t, you’ll lose a lot of time at the event downloading your dependencies.

If you are unfamiliar with Java and Scala, I strongly recommend you use an IDE, which will help with code completion and syntax awareness. I happen to prefer JetBrains’ IDEA, but this is like Vim vs Emacs. You can use Eclipse if you prefer. If you use Eclipse, be sure to use the Scala IDE extension. In the case of IDEA, download and install the Scala plugin.

If you have any issues, please email me, François Beausoleil, and I’ll help you out. I’ll post updates to this page if common errors pop up.

SQL Workshop

The SQL workshop will be a series of directed examples:

1. I will present a problem, a report or a question we want answered, and some details on how you can accomplish the goal;
2. You will answer the question with the knowledge you have;
3. I’ll ask people to present their solutions;
4. I’ll present my solution and discuss specifics

I have 6 exercises planned out, from 15 to 45 minutes each. The topics range from indexing to joining to using intersections and unions and ending with windowing functions. The workshop is for people who wish to learn more about SQL and how to more effectively use thecapabilites of their favorite SQL engines.

Install PostgreSQL 9.1 (latest is currently 9.1.2)
Load this PostgreSQL database dump svworkshop.sql.bz2 (315 MiB) in your cluster using the following command:

1 bzcat svworkshop.sql.bz2 | psql

The dump file expects to create a new database named svworkshop using your default user.

Familiarize yourself with the entity-relationship diagram (PDF) of the dump

Hope to see you there!

Daemontools Best Practices

2011-09-12T08:33:02-04:00

I sometimes have to do sysadmin work, such as when I’m the sole technical person on a probject. When I need to keep a service running, I usually turn to daemontools. Daemontools was written by D. J. Bernstein, a mathematician and author of many UNIX utilities.

From daemontools’ home page:

daemontools is a collection of tools for managing UNIX services.

daemontools home page

What this means is daemontools is designed to run, and keep running, a service. Daemontools also includes other utilities which I find useful, such as setuidgid, envdir and multilog,. I searched for an article such as this, but couldn’t find it. If you find a factual error, please let me know immediately. If you have your own best practices, let me know so I can expand on the list.

Read the articles themselves here:

Running ScalaTest Tests Under IntelliJ IDEA

2011-08-15T14:09:54-04:00

I had lots of difficulties running my tests under IDEA. The exact error message was:

Error running All Tests:
Not found suite class.

Where All Tests was the name of my Run configuration.

I finally ended up with the right incantations. In my POM, I have the following:

 1 <dependencies>
 2     <dependency>
 3       <groupId>junit</groupId>
 4       <artifactId>junit</artifactId>
 5       <version>4.8.1</version>
 6       <scope>test</scope>
 7     </dependency>
 8     <dependency>
 9       <groupId>org.scalatest</groupId>
10       <artifactId>scalatest_2.9.0-1</artifactId>
11       <version>1.6.1</version>
12     </dependency>
13     <dependency>
14       <groupId>org.mockito</groupId>
15       <artifactId>mockito-core</artifactId>
16       <version>1.8.1</version>
17       <scope>test</scope>
18     </dependency>
19 </dependencies>

Then, I had to extend org.scalatest.junit.JUnitSuite for my test classes, like this:

1 import JUnitSuite
2 import org.junit.Test
3 
4 class GivenAnEmptyQueue extends JUnitSuite {
5   @Test def thenItShouldNotHaveAnyElements() {
6     assert(new Queue.empty)
7   }
8 }

Finally, I had to verify that both the JUnit and Scala plugins were enabled and at their latest versions in the IDE itself. After that, I was able to run my tests from within the IDE.

Simple-Build-Tool: Error Getting Started

2011-07-30T22:50:09-04:00

I’m starting in Scala, because Seevibes’ code is in Scala. Scala has a tool named simple-build-tool for managing your projects. sbt is similar to Ruby’s Bundler and Clojure’s Leiningen in that it manages dependencies and helps build a project for you.

Unfortunately, I had problems getting started. After following the Setup instructions, I was consistently getting this error:

 1 Getting org.scala-tools.sbt sbt_2.8.1 0.10.1 ...
 2 
 3 :: problems summary ::
 4 :::: WARNINGS
 5     [NOT FOUND  ] commons-logging#commons-logging;1.0.4!commons-logging.jar (5ms)
 6 
 7   ==== Maven2 Local: tried
 8 
 9     file:///Users/francois/.m2/repository/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.jar
10 
11     ::::::::::::::::::::::::::::::::::::::::::::::
12 
13     ::              FAILED DOWNLOADS            ::
14 
15     :: ^ see resolution messages for details  ^ ::
16 
17     ::::::::::::::::::::::::::::::::::::::::::::::
18 
19     :: commons-logging#commons-logging;1.0.4!commons-logging.jar
20 
21     ::::::::::::::::::::::::::::::::::::::::::::::
22 
23 
24 
25 :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
26 download failed: commons-logging#commons-logging;1.0.4!commons-logging.jar
27 Error during sbt execution: Error retrieving required libraries
28   (see /Users/francois/Projects/project/boot/update.log for complete log)
29 Error: Could not retrieve sbt 0.10.1

I hit #scala and RSchulz pointed me to ~/.ivy2. After I rm -rf’d ~/.ivy2 and ~/.m2, sbt ran to completion. I was unable to find any mentions of the errors above, so hopefully this may help someone else.

Sequel's #set_all still restricts the primary key

2011-07-04T10:57:00-04:00

I ran into a little gotcha today, using Sequel. I’m writing an importer, you know the kind: read record from database A, apply some transformations, write to database B. No rocket science required. But, Sequel has a little gotcha that stumped me for a bit. My script looked like this:

 1 DBa = Sequel.connect "..."
 2 DBb = Sequel.connect "..."
 3 
 4 class APerson < Sequel::Model(DBa[:people])
 5 end
 6 
 7 class BContact < Sequel::Model(DBb[:contacts])
 8 end
 9 
10 contacts = Hash.new
11 APerson.all.each do |person|
12   contact = BContact.create(
13     :name        => person.last_name + ", " + person.first_name,
14     :tenant_code => ENV["TENANT_CODE"],
15     :updated_by  => "import",
16     :updated_at  => Time.now)
17   contacts[ person.id ] = contact.contact_id
18 end
19 
20 # Now I can map A's IDs to the correct value in database B, such as
21 # for attaching email addresses, phone numbers, etc.

The Contact model in the B database is declared like this:

1 create_table :contacts do
2   column :tenant_code, :integer,       :null => false
3   column :contact_id,  :serial,        :null => false
4   column :name,        "varchar(240)", :null => false
5 
6   primary_key [:tenant_code, :contact_id]
7   foreign_key [:tenant_code], :tenants
8 end

Notice tenant_code and contact_id are part of the primary key. I don’t write to contact_id because I want the sequence’s value to be returned to me. But I must write my own value to the tenant_code column. I was receiving an exception on the #create call:

 1 /Users/francois/.rvm/gems/ruby-1.9.2-p180/gems/sequel-3.25.0/lib/sequel/model/base.rb:1491:in `block in set_restricted': method tenant_code= doesn't exist or access is restricted to it (Sequel::Error)
 2   from /Users/francois/.rvm/gems/ruby-1.9.2-p180/gems/sequel-3.25.0/lib/sequel/model/base.rb:1486:in `each'
 3   from /Users/francois/.rvm/gems/ruby-1.9.2-p180/gems/sequel-3.25.0/lib/sequel/model/base.rb:1486:in `set_restricted'
 4   from /Users/francois/.rvm/gems/ruby-1.9.2-p180/gems/sequel-3.25.0/lib/sequel/model/base.rb:1077:in `set'
 5   from /Users/francois/.rvm/gems/ruby-1.9.2-p180/gems/sequel-3.25.0/lib/sequel/model/base.rb:1456:in `initialize_set'
 6   from /Users/francois/.rvm/gems/ruby-1.9.2-p180/gems/sequel-3.25.0/lib/sequel/model/base.rb:764:in `initialize'
 7   from /Users/francois/.rvm/gems/ruby-1.9.2-p180/gems/sequel-3.25.0/lib/sequel/model/base.rb:134:in `new'
 8   from /Users/francois/.rvm/gems/ruby-1.9.2-p180/gems/sequel-3.25.0/lib/sequel/model/base.rb:134:in `create'
 9   from /Users/francois/.rvm/gems/ruby-1.9.2-p180/gems/sequel-3.25.0/lib/sequel/model/base.rb:248:in `find_or_create'
10   from script/import:65:in `block (2 levels) in <top (required)>'
11

I was very much stumped, and turned to the excellent documentation. I eventually found my way to #set_all, and changed my code accordingly:

1 APerson.all.each do |person|
2   contact = BContact.new.set_all(
3     :name            => person.last_name + ", " + person.first_name,
4     :tenant_code     => ENV["TENANT_CODE"],
5     :last_updated_by => "import",
6     :last_updated_at => Time.now)
7   contacts[ person.id ] = contact.contact_id
8 end

Even though the Sequel RDoc says #set_all ignores restricted columns, I was still receiving the same exception. I was now doubly stumped, until I found a reference to #unrestrict_primary_key. I added the declaration to BContact and was able to proceed:

1 class BContact < Sequel::Model(DBb[:contacts])
2   unrestrict_primary_key
3 end

You know the drill though: where you import one model, you’ll have more to import shortly. Ruby to the rescue!

1 class Sequel::Model
2   # Open classes win every time!
3   unrestrict_primary_key
4 end

Problem solved!

Counters: An Easy Way to Gather Metrics From Your Ruby Code

2011-04-24T23:07:35-04:00

Sometimes, you need to know what your program’s doing, or how long it’s taking to do something. You could always log to a file, then use a combination of grep, awk and/or wc to gather the statistics yourself, but why bother? There are many tools out there which will do exactly what you want, just use them: Cacti, Graphite or plain-old RRD.

For instance, at Yatter, we need to know how fast our ranking algorithms are running, and we must know how long the ranking takes compared to the number of users and pages we have on hand. Graphing is the perfect solution for that, and Graphite fit the bill just fine for us. But Graphite alone won’t do all that we need: we also needed a way to instrument our code, hence the Counters library:

 1 require "counters"
 2 require "sequel"
 3 
 4 DB = Sequel.connect "jdbc:postgres://127.0.0.1:5432/db"
 5 Counter = Counters::StatsD.new(:url => "udp://127.0.0.1:8125", :namespace => "ranker")
 6 
 7 users = Counter.latency "fetch.users" do
 8   DB[:users].all
 9 end
10 
11 pages = Counter.latency "fetch.pages" do
12   DB[:pages].all
13 end
14 
15 Counter.magnitude "count.users", users.length
16 Counter.magnitude "count.pages", pages.length
17 
18 Counter.latency "ranking" do
19   entropy = 1.0
20   while entropy > MIN_ENTROPY
21     Counter.hit "iteration"
22     # Reduce entropy
23   end
24 end

At the end of the day, we’ll have hierarchical counters in Graphite which will give us all kinds of statistics. From the API above, you can gather that values are stored under hierarchical keys separated by fullstops. If you’re interested in the code, make yourself at home with the Counters GitHub repository.

Counters is certified to run on JRuby 1.6.0 in 1.8 mode, and MRI 1.9.2.

JRuby, PostgreSQL and Bulk Copy Operations

2011-04-10T21:09:00-04:00

 1 # JRuby 1.6.0 required
 2 require "sequel"
 3 require "jdbc/postgresql"
 4 
 5 DB = Sequel.connect "jdbc:postgresql://127.0.0.1:5432/mydb"
 6 
 7 table = DB[:mytable]
 8 time = Benchmark.measure do
 9   (1..10000).each do |n|
10     table.insert(:a => n, :b => 2*n)
11   end
12 end
13 
14 puts time

How much time do you think this is going to take? This is the most inefficient way to insert many rows to a database. Remember each call to #insert will do a round trip to the database. Even if your round trip time is 1ms, you’ll still pay for 10 seconds of round trip time, time which your program could be doing something much more useful, such as generating revenue (somehow).

Instead, you should use the bulk copy feature of your database engine. In my case, that’s PostgreSQL. Since I’m using JRuby, I have to turn to the JDBC world, but that’s all right: everything has been implemented already, by someone, somewhere. I’ll refer you to the relevant pages:

And the relevant code would be:

 1 # JRuby 1.6.0 required
 2 require "sequel"
 3 require "jdbc/postgresql"
 4 require "java"
 5 
 6 DB = Sequel.connect "jdbc:postgresql://127.0.0.1:5432/mydb"
 7 
 8 time = Benchmark.measure do
 9   DB.synchronize do |connection|
10     copy_manager = org.postgresql.copy.CopyManager.new(connection)
11     stream = copy_manager.copy_in("COPY mytable(a, b) FROM STDIN WITH CSV")
12 
13     begin
14       (1..10000).each do |n|
15         # Don't forget we're streaming CSV data, thus each row/line MUST be terminated with a newline
16         row = "#{n},#{2*n}\n".to_java_bytes
17         stream.write_to_copy(row, 0, row.length)
18       end
19     rescue
20       stream.cancel_copy
21       raise
22     else
23       stream.end_copy
24     end
25   end
26 end
27 
28 puts time

This will execute a single round trip to the database server: you’ll pay the latence cost only once.

On an unrelated note, this is the first time ever I use an else clause on a begin/rescue. If an exception is raised, we want to cancel the copy (the rescue clause), but on the other hand, if nothing is raised, we want to end the copy (the else clause). One or the other must happen, but not both.

If you’re curious what difference bulk copying makes, here are the benchmark results:

10000 INSERT statements
  7.012000   0.000000   7.012000 (  7.012000)

1 COPY FROM STDIN statement
  0.848000   0.000000   0.848000 (  0.848000)

The numbers speak for themselves: 8× faster. Not too shabby, and remember this ratio will simply increase as the number of rows increases.

DataDumper: Copy Your Test Data to Your Development Environment With Ease

2011-03-31T22:08:13-04:00

The README says it all:

You’re knee deep in a debugger session, and you can’t understand why something’s wrong. You wish you could fire up your application against the test database, but sadly, the process which is running the tests is within a transaction, and thus the actual data is opaque. What can you do?

1 # Somewhere deep in your tests
2 test "the frobble touches the widget" do
3   assert_equal 42, frobble.widget_id
4 end

You’ve been on this assert_equal call for the past hour wondering. Frustration’s been mounting, because you don’t understand why the frobble doesn’t touch the widget. Clearly, there’s something wrong with the fixtures, but you can’t understand what it is. Time to fire up the debugger and dump the data:

 1 [814, 823] in test/unit/widget_test.rb
 2    814          frobble.save!
 3    815        end
 4    816
 5    817        test "the frobble touches the widget" do
 6    818          debugger
 7 => 819          assert_equal 42, frobble.widget_id
 8    820        end
 9    821
10    822        test "the widget touched the frobble in turn" do
11    823          assert widget.touched_by_frobble?
12 test/unit/widget_test.rb:819
13 => 819          assert_equal 42, frobble.widget_id
14 (rdb:112)

Since the data_dumper gem is already declared in your Gemfile (if not, declare it, bundle install, then run your tests again), type:

1 (rdb:112) File.mkdir(Rails.root + "dump")
2 (rdb:113) DataDumper.dump(Rails.root + "dump")

Then, quit your failing tests, and from the trusty command line:

1 $ rails console
2 > DataDumper.load(Rails.root + "dump")
3 > exit
4 
5 $ rails server

Any and all data from your test database will be loaded in your development environment. You can now explore your model with your trusty application, to find out what’s really going on.

Development Environment Benefits From Config File Tweaking as Well

2011-03-17T11:05:01-04:00

If you’re like me, you installed PostgreSQL (or MySQL or whatever) locally for development purposes. Then you copied data into your instance, and along you went.

Eventually, you wrote a migration that touched nearly all rows in your database, and you waited, and waited, and waited some more (grab some coffee!) for the migration to finish.

In my case, I had a migration using pure Ruby which took 30 minutes. The migration ran along the lines of:

 1 def self.up
 2   add_column :segments, :pathname, :string
 3 
 4   # There are 200,000 segments in the DB
 5   Segment.find_each do |segment|
 6     segment.pathname = segment.full_pathname
 7   end
 8 
 9   change_column :segments, :pathname, :string, :null => false
10 end
11 
12 class Segment < ActiveRecord::Base
13   belongs_to :parent, :class_name => "Segment"
14 
15   def full_pathname
16     full_name = []
17     m = self
18     while m
19       full_name << m.name
20       m = m.parent
21     end
22 
23     full_name.reverse.join(" / ")
24   end
25 end

The intent being that we’d cache the segment’s pathname in each record (think a directory tree). Since we had 200,000 segments, and each of those would walk up the chain, it took around 30 minutes to run that migration. I rewrote the migration to use CTE, which PostgreSQL supports out-of-the-box. My migration now ran in 17 minutes or so. I warned my teammates about this migration, and let it be. Eventually, a colleague took my branch and ran the migration. When he reported the migration took 30 seconds for him, at first I thought his SSD was the major difference between our machines. It turns out it wasn’t…

1 # PostgreSQL installed through Homebrew
2 # /usr/local/var/postgres/postgresql.conf
3 # Your file location will vary
4 # ...
5 shared_buffers = 2400kB
6 # ...

Changing that single line from 2400kB to 96MB let me run the migration in 30 seconds as well… I was very impressed at my stupidity. Let it be a lesson learned!

PostgreSQL, Foreign Keys and Deadlocks

2010-11-23T08:36:40-05:00

What happens when you mix foreign key constraints, PostgreSQL, ActiveRecord and cached counters (search for :counter_cache)? Deadlocks.

My colleague, Mina Naguib, spent some time while away from the office to dig and diagnose some problems. The underlying issue is probably not related to PostgreSQL itself, but cached counters, which Rails makes very easy to do, sometimes result in deadlocks. We’ve seen those backtraces in Hoptoad, but had never really investigated them: they were rare, meaning nobody complained about them. They were just a thorn in our side.

In a nutshell, this flow is problematic:

Adding or modifying a record in a table, when:

That table is a child (using a foreign key contraint) of another table

and then modifying the parent record in the parent table

Using transactions

Concurrently

Sometimes (race condition)

From Mina’s PostgreSQL transactions wrapping child+parent modifications, deadlocks, and ActiveRecord

Read the full text: it’s well worth it. Unfortunately, Mina doesn’t have a resolution, but he wants more people to be aware of the issue, in the hopes that somebody will find a clean solution.

Presenter or Command?

2010-10-09T22:54:25-04:00

The more I think about it, the more I believe Commands and Presenters are intertwined. If you don’t know what a Presenter is, I suggest reading these articles:

I’m building a gem to abstract the Command pattern in your applications. It’s not Rails-specific, but does know about ActiveRecord. Anyway, as I was writing my sample application, I noticed some duplication:

app/controllers/invitations_controller.rb

1 class InvitationsController < ApplicationController
2   def show
3     @user = User.invited.with_token(params[:id]).first
4     raise ActiveRecord::RecordNotFound unless @user
5     render :action => :confirm
6   end
7 end

app/commands/confirm_invitation_request_command.rb

 1 class ConfirmInvitationRequestCommand
 2   Komando.make_command self
 3 
 4   mandatory_steps do
 5     @user = User.invited.with_token(params[:id]).first
 6     raise ActiveRecord::RecordNotFound unless @user
 7 
 8     @user.activate!(@attributes)
 9   end
10 end

Notice the first two lines of InvitationsController#show and ConfirmInvitationRequestCommand#mandatory_steps: identical. Then I thought, what if the Command was also a Presenter? Then, I could refactor appropriately:

app/commands/confirm_invitation_request_command.rb

 1 class ConfirmInvitationRequestCommand
 2   Komando.make_command self
 3 
 4   def user
 5     @user ||= begin
 6       User.invited.with_token(params[:id]).first.tap do |user|
 7         raise ActiveRecord::RecordNotFound unless user
 8       end
 9     end
10   end
11 
12   mandatory_steps do
13     @user = User.invited.with_token(params[:id]).first
14     raise ActiveRecord::RecordNotFound unless @user
15 
16     @user.activate!(@attributes)
17   end
18 end

app/controllers/invitations_controller.rb

1 class InvitationsController < ApplicationController
2   def show
3     @user = ConfirmInvitationRequestCommand.new(:token => params[:id]).user
4     render :action => :confirm
5   end
6 end

Notice how the business logic of finding a user by token is nicely tucked away.

Of course, while writing this post, I realized that finding a User by token should live in my model, not in the controller or the command, but still: bear with me.

The Presenter pattern makes some decisions for the view: how to get at the objects to be shown, how to arrange them, how to save them as a group. Turns out my Command does essentially the same thing. Maybe we can think of Presenter as a special-case of Command? Such that Presenter is a Command to view an object? That seems logical to me.

Unintented Consequences: The Pitfalls of ActiveRecord Callbacks

2010-09-28T08:42:17-04:00

At Bloom, we have a largish Rails application: 12 000 lines of model code, alone. Yesterday, we found a major problem, and the cause was very non-obvious.

Bloom Digital Platforms has a product called AdGear. AdGear is an ad distribution platform. The obvious models are Ad and Campaign: a Campaign has many ads. From a UI perspective, it’s nice to show how many active ads are in the campaign. In the campaign’s index view, we have something like this:

1 <%# app/views/campaigns/_campaign.html.erb %>
2 <tr>
3   <td><%= link_to(h(campaign.name), campaign) %></td>
4   <td><%= pluralize(campaign.active_ad_units_count, "active ads") %></td>
5 </tr>
6

Notice how we’re using an attribute named active_ad_units_count. This is a cached value. We keep this value updated by doing the following:

 1 class Ad < ActiveRecord::Base
 2   belongs_to :campaign
 3   after_save :update_campaign_cached_values
 4 
 5   private
 6 
 7   def update_campaign_cached_values
 8     campaign.update_cached_values
 9   end
10 end
11 
12 class Campaign < ActiveRecord::Base
13   def update_cached_values
14     update_attribute(:active_ad_units_count, ad_units.active.count)
15   end
16 end

Pretty reasonable, you’ll say. And I agree: good names explain the intent, without needing to read all the details.

Let’s throw a monkey wrench in there:

 1 class Campaign < ActiveRecord::Base
 2   before_save :ensure_consistency
 3 
 4   attr_accessor :conditions
 5 
 6   def conditions
 7     return @conditions if @conditions
 8     @conditions =|| serialized_conditions ? YAML.load(serialized_conditions) : nil
 9   end
10 
11   private
12 
13   # Method renamed to protect the innocent
14   def ensure_consistency
15     if some_condition then
16       self.serialized_conditions = conditions ? conditions.to_yaml : nil
17       return true
18     end
19 
20     false
21   end
22 end

Can you spot the issue? Note this code is in Campaign, not Ad. It’s pretty obvious when boiled down to 20 lines of code. When you’re wading through 12 000, it’s pretty tough.

If you read the documentation for #before_save, you’ll notice this gem:

1 # == Canceling callbacks
2 #
3 # If a <tt>before_*</tt> callback returns +false+, all the later callbacks and the associated action are cancelled. If an <tt>after_*</tt> callback returns
4 # +false+, all the later callbacks are cancelled. Callbacks are generally run in the order they are defined, with the exception of callbacks
5 # defined as methods on the model, which are called last.

Excerpt from activerecord/lib/active_record/callbacks.rb

In the example above, some_condition might return false, which would cancel the save, but let’s unwind the stack:

AdsController#update
Ad#update_campaign_cached_values
Campaign#update_cached_values
ActiveRecord::Base#update_attribute
Campaign#ensure_consistency – returns false

Because we were calling #update_attribute, which returns false when it fails to update the object, we ended up not updating the Campaign’s cached values. And we ended up with a stale cache. And ads which didn’t get updated. A careful tower of convoluted code tumbled down upon us.

Why did this happen? It’s obvious the developer who wrote the #ensure_consistency method didn’t know about ActiveRecord’s #serialize, which did exactly what he wanted to. He also didn’t know about the dangers of returning false from a callback. These two point to relative inexperience / lack of knowledge about Rails & ActiveRecord.

We’re obviously not the first ones to hit this kind of problem:

James Golick also ranted off about something similar a while back: Crazy, Heretical, and Awesome: The Way I Write Rails Apps

This points to a misfeature of ActiveRecord: a seemingly innocuous change had unintended consequences in a very unrelated portion of the code. Rails has a lot of magic / implicits. You have to be very careful. Ignorance is no excuse. The more I think about it, the more I like the service / command oriented way James offers, and it boils down to the discussion we had at the last Montreal.rb: implicit (Rails/Ruby) vs explicit (Django/Python).

Bundler & Continuous Testing Scenarios: Usage Tips

2010-08-19T10:30:24-04:00

I just pushed a change for my teammates: we’re now using Bundler on our Rails 2.3.5 application. Things went rather well, but our continuous testing server choked a bit on that.

First, I had forgotten to run bundle install after pulling changes. Then I had to ignore our cucumber environment, because it depends on rb-appscript which is a native gem on Mac OS X. Our CT machine is a Linux box, hence the change. Then, bundle install would fail installation because it didn’t have sudo access:

 1 Fetching git://github.com/bloom/rails.git
 2 Fetching git://github.com/francois/active_url
 3 Fetching git://github.com/francois/safariwatir.git
 4 Fetching source index for http://rubygems.org/
 5 sudo: no tty present and no askpass program specified
 6 sudo: no tty present and no askpass program specified
 7 Using rake (0.8.7) 
 8 sudo: no tty present and no askpass program specified
 9 sudo: no tty present and no askpass program specified
10 /usr/lib64/ruby/site_ruby/1.8/rubygems/format.rb:38:in `from_file_by_path': Cannot load gem at [/usr/lib64/ruby/gems/1.8/cache/SystemTimer-1.2.gem] in /home/francois/adgear-admin (Gem::Exception)
11   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/source.rb:77:in `fetch'
12   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/installer.rb:45:in `run'
13   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/spec_set.rb:12:in `each'
14   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/spec_set.rb:12:in `each'
15   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/installer.rb:44:in `run'
16   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/installer.rb:8:in `install'
17   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/cli.rb:185:in `install'
18   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/vendor/thor/task.rb:22:in `send'
19   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/vendor/thor/task.rb:22:in `run'
20   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
21   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/vendor/thor.rb:246:in `dispatch'
22   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/lib/bundler/vendor/thor/base.rb:389:in `start'
23   from /usr/lib64/ruby/gems/1.8/gems/bundler-1.0.0.rc.5/bin/bundle:13
24   from /usr/bin/bundle:19:in `load'
25   from /usr/bin/bundle:19
26

I was busy writing a message to the Ruby Bundler Google Group when I remembered bundle install has an optional parameter which specifies where the bundle should be installed. I changed my CT script to run bundle install --without=cucumber vendor/gems and bundle did it’s magic, and my CT box started running tests.

Much rejoicing ensued.

For the record, here’s our testing script:

 1 #!/bin/bash
 2 cd /home/francois/adgear-admin
 3 
 4 if [ -z $1 ] ; then
 5         echo "Can't proceed if I don't know which branch to test"
 6         exit 1
 7 fi
 8 
 9 TSTAMP=`date +%Y-%m-%d-%H:%M`
10 BRANCH=$1
11 LOG=/home/francois/build-logs/$BRANCH-$TSTAMP.log
12 
13 (
14         git fetch origin                                           && \
15         git reset --hard origin/$BRANCH                            && \
16         echo =======================                               && \
17         git show                                                   && \
18         ( bundle install --without=cucumber vendor/cache || true ) && \
19         nice rake db:adgear:production:update parallel:prepare parallel:test
20 ) > $LOG 2>&1
21 
22 if [ "$?" = "0" ] ; then
23         /usr/bin/curl -s -o /dev/null -d "{\"message\":{\"body\":\"Test Suite Passed -- $BRANCH -- http://ct.local:9001/logs/$BRANCH/$TSTAMP\"}}" -HContent-Type:application/json -u 12345:X http://oursite.campfirenow.com/room/12345/speak.json
24 else
25         /usr/bin/curl -s -o /dev/null -d "{\"message\":{\"body\":\"Test Suite FAILED -- $BRANCH -- http://ct.local:9001/logs/$BRANCH/$TSTAMP\"}}" -HContent-Type:application/json -u 12345:X http://oursite.campfirenow.com/room/12345/speak.json
26 fi
27 
28 # vi: sw=8 ts=8 expandtab

This script is run from cron.

ActiveRecord STI + abstract_class? have a gotcha

2010-07-25T22:44:25-04:00

To try and ease my pain, I am refactoring a Rails 2.3.5 application using Hobo from multiple tables to a single table. I’m talking about models which are essentially the same: companies, people and employees. I went from this:

 1 class Company < ActiveRecord::Base
 2   has_many :employees
 3   has_many :people, :through => :employees
 4 end
 5 
 6 class Person < ActiveRecord::Base
 7   has_many :employees
 8   has_many :companies, :through => :employees
 9 end
10 
11 class Employee < ActiveRecord::Base
12   belongs_to :company
13   belongs_to :person
14 end

To an STI-enabled class hierarchy:

 1 class Addressee < ActiveRecord::Base
 2   self.abstract_class = true
 3 end
 4 
 5 class Company < Addressee
 6   has_many :employees
 7   has_many :people, :through => :employees
 8 end
 9 
10 class Person < Addressee
11   has_many :employees
12   has_many :companies, :through => :employees
13 end
14 
15 class Employee < Addressee
16   belongs_to :company
17   belongs_to :person
18 end

Can you spot the mistake? Here’s where it breaks down:

1 $ script/console
2 > PersonAddressee.count
3 SQL (9.9ms)   SELECT count(*) AS count_all FROM "addressees"
4 => 10231

When the classes are defined with abstract_class? set to true, calling Company#count executes the following:

ActiveRecord Company#superclass, which answers Addressee
ActiveRecord calls Addressee#abstract_class?, which answers true
Because Company#base_class == Company#superclass, no STI is involved, and ActiveRecord queries the addressees table with no STI hints (type column).

When removing the abstract class declaration, here’s how it works:

ActiveRecord Company#superclass, which answers Addressee
ActiveRecord calls Addressee#abstract_class?, which answers false
ActiveRecord Addressee#superclass, which answers ActiveRecord::Base
Because Company#base_class != Company#superclass, STI is involved and ActiveRecord adds the type hint to the addressees table query.

The documentation is quite clear though:

abstract_class?()

Returns whether this class is a base AR class. If A is a base class and B descends from A, then B.base_class will return B.

The documentation on #base_class is much better though:

base_class()

Returns the base AR subclass that this class descends from. If A extends AR::Base, A.base_class will return A. If B descends from A through some arbitrarily deep hierarchy, B.base_class will return A.

And current HEAD documentation is the best of them all:

base_class()

Returns the base AR subclass that this class descends from. If A extends AR::Base, A.base_class will return A. If B descends from A through some arbitrarily deep hierarchy, B.base_class will return A.

If B < A and C < B and if A is an abstract_class then both B.base_class and C.base_class would return B as the answer since A is an abstract_class.

Luckily, I found this before users got a chance to report bugs. And no, I would never have thought to add a test for this: I’m not in the habit of testing base ActiveRecord functionality.

Updated Mongo Explorer to v1.0.2

2010-07-16T23:52:04-04:00

Damien Caselli, one of the first brave souls to use Mongo Explorer, reported an issue with time travelling in Mongo Explorer. Luckily, it wasn’t anything serious: just a misunderstanding between me, Cocoa and JavaScript.

To celebrate the No Time Travelling Issues, I’ve released version 1.0.2 of Mongo Explorer. Get yours: mongo-explorer-v1.0.2.zip

Yes, I’m aware of Sparkle, and yes, it will eventually make it into Mongo Explorer. That, and a million other little things.

Mongo Explorer: Explore your MongoDB

2010-07-14T23:44:03-04:00

In the past few months, I was looking for an excuse to write a native Mac Cocoa application. We use MongoDB at Bloom Digital to handle our analytics needs. I had tried mongohub, but my itch wasn’t scratched: I wanted something that displayed my data in tables, not as some kind of outline. Thus was born Mongo Explorer.

Click to enlarge

If you tell me “Why didn’t you fork mongohub and add your code”, I’ll tell you I was intimidated by a larger Cocoa application. I wanted something simple from which I could learn everything I wanted: from the grounds up. Of course, once I’m “done” (for some measure of done-ness), the code base will be much larger.

One useful piece of code that might be useful for others is a Mongo Driver for Objective-C. Underlying Mongo Explorer is an abstraction for Mongo connections that’s similar to the Ruby driver. It’s not yet a framework, since I don’t yet know how to do that, but hey, this is a learning exercise for me. One nice thing about this abstraction is that it’s built directly atop the official Mongo C driver: thus it inherits it’s abilities, but some of it’s quirks as well.

What can you expect from Mongo Explorer today? You can open a connection to any local MongoDB instance (manually open an SSH tunnel if you want a remote instance), browse the server’s databases and collections, and list your collection’s documents. For each document, you’ll get what I call the flat view. The flat view is a dot-separated list of keys, followed by the value. It’s useful to view nested objects:

1 obj = {"rp" => {"a" => 1, "b" => {"c" => "d"}}}
2 # Flat view
3 # rp.a   => 1
4 # rp.b.c => d

The flat view was my biggest impetus to write Mongo Explorer.

What’s broken? First, this needs some kind of icon. Also, don’t expect to browse multi-million document collections just yet. Cocoa wants to fully materialize the NSTableView rows, and I don’t know how to handle that yet. I do have an MEArray class that helps somewhat, but it’s not perfect yet. Learning exercise, remember? I’ve successfully viewed collections with slightly less than 20,000 document with no issues.

If you’re interested in following, you can play around on GitHub, or even download the first official release: Mongo Explorer, v1.0.1

UPDATE: Pushed 1.0.1, as 1.0.0 didn’t include it’s frameworks, and was 10.6 only.

Using the Mongo C Driver

2010-06-06T15:30:58-04:00

I’m writing a Cocoa application to browse and administer a Mongo DB. This is a fun side-project. I’m using the Mongo driver from Objective-C, hence there might be differences for you.

First off, we need to connect to Mongo, and to do that, we need to tell Mongo about the connection information, namely the host and port:

1 mongo_connection_options opts;
2 strcpy(opts.host,
3        [[self.connectionInfo objectForKey:MEHost] cString]);
4 opts.port = [[self.connectionInfo objectForKey:MEPort] intValue];
5 /* mongo doesn't have a default port constant or define */
6 if (0 == opts.port) opts.port = 27017;

connectionInfo above is an instance of NSDictionary. The available keys are declared in NewConnectionController.

After we’ve prepared the connection options, it’s time to connect:

1 connection = (mongo_connection *)malloc(sizeof(mongo_connection));
2 if (!connection) return;
3 
4 mongo_conn_return result = mongo_connect(connection, &opts);
5 if (mongo_conn_success == result) return;

connection above was declared as a pointer to a mongo_connection structure in DatabaseController. If the connection succeeds, I simply continue. If the connection fails, I fall-through and inform the user using an NSAlert, shown as a sheet.

I haven’t yet needed to authenticate to Mongo, but it seems simple to call mongo_cmd_authenticate().

Don’t forget the tests available in the mongo driver itself.

Autosave Associations and validates_uniqueness_of: Don't Do That

2010-06-02T15:08:20-04:00

We received a nasty Hoptoad notification today at AdGear :

1 ActiveRecord::StatementInvalid: PGError: ERROR:
2 duplicate key value violates unique constraint "index_placement_rules_on_type_and_placement_id_and_bookable_typ" :
3 INSERT INTO "placement_rules" ("kind", "created_at", "placement_id", "bookable_type", "bookable_id", "updated_at") VALUES(E'PlacementInclusion', '2010-06-02 19:29:24.479983', 2320, E'Site', 162, '2010-06-02 19:29:24.479983') RETURNING "id"

This manifested itself when a customer called and told us he’d found our 500 error page. Ooops.

Our problem turned out to be a misunderstanding of autosave association validations.

Autosave associations are a wonderful beast: with a single #save call, the nested entities are saved within the same transaction, without any intervention on the developer’s part. Unfortunately, validations behave differently than what we expected.

For the sake of argument, let’s use these models:

 1 class Order < ActiveRecord::Base
 2   has_many :items
 3 end
 4 
 5 class OrderItem < ActiveRecord::Base
 6   belongs_to :order
 7 
 8   validates_presence_of :product_number
 9   validates_uniqueness_of :product_number, :scope => :order
10 end

If you build an Order and add two items, you’ll have troubles, unless you wrap your save in a transaction block:

 1 > o = Order.new
 2 > 2.times { o.items << Item.new(:product_number => "A-113") }
 3 > o.save
 4   # Validates entities
 5   Item Load (0.2ms)   SELECT "items".id FROM "items" WHERE ("items"."product_number" = 'A-113' AND "items".order_id IS NULL) LIMIT 1
 6   Item Load (0.1ms)   SELECT "items".id FROM "items" WHERE ("items"."product_number" = 'A-113' AND "items".order_id IS NULL) LIMIT 1
 7   Order Create (0.7ms)   INSERT INTO "orders" ("customer_number", "created_at", "updated_at") VALUES(NULL, '2010-06-02 19:15:57', '2010-06-02 19:15:57')
 8   # Validate 1st entity again
 9   Item Load (0.1ms)   SELECT "items".id FROM "items" WHERE ("items"."product_number" = 'A-113' AND "items".order_id = 1) LIMIT 1
10   Item Create (0.1ms)   INSERT INTO "items" ("created_at", "order_id", "updated_at", "product_number") VALUES('2010-06-02 19:15:57', 1, '2010-06-02 19:15:57', 'A-113')
11   # Validate 2nd entity again, but fails this time around
12   Item Load (0.1ms)   SELECT "items".id FROM "items" WHERE ("items"."product_number" = 'A-113' AND "items".order_id = 1) LIMIT 1
13  => [false, true] 
14 > o.items.map(&:errors).flatten.map(&:full_messages)
15  => [[], ["Product number has already been taken"]]

A partially saved order is rarely what you’re looking for. Adding the :autosave => true option…

1 class Order < ActiveRecord::Base
2   has_many :items, :autosave => true
3 end

Generates a very different SQL trace:

 1 > o = Order.new
 2 > 2.times { o.items << Item.new(:product_number => "A-113") }
 3 > o.save
 4   # Validate associated entities...
 5   Item Load (0.2ms)   SELECT "items".id FROM "items" WHERE ("items"."product_number" = 'A-113' AND "items".order_id IS NULL) LIMIT 1
 6   Item Load (0.1ms)   SELECT "items".id FROM "items" WHERE ("items"."product_number" = 'A-113' AND "items".order_id IS NULL) LIMIT 1
 7   Order Create (0.4ms)   INSERT INTO "orders" ("customer_number", "created_at", "updated_at") VALUES(NULL, '2010-06-02 20:03:12', '2010-06-02 20:03:12')
 8   # Then save them, even though they're invalid.
 9   Item Create (0.1ms)   INSERT INTO "items" ("created_at", "order_id", "updated_at", "product_number") VALUES('2010-06-02 20:03:12', 2, '2010-06-02 20:03:12', 'A-113')
10   Item Create (0.1ms)   INSERT INTO "items" ("created_at", "order_id", "updated_at", "product_number") VALUES('2010-06-02 20:03:12', 2, '2010-06-02 20:03:12', 'A-113')
11  => true

This is even worse: instead of a partial save, ActiveRecord told us that everything was good. This is how ActiveRecord behaves:

1 if order.save then
2   order.items.each(&:save) if order.items.all?(&:valid?)
3 end

What we need instead is to hook into the autosave callback chain:

 1 class Order < ActiveRecord::Base
 2   has_many :items, :autosave => true
 3 
 4   private
 5 
 6   # items is the has_many name. If you had posts, it would be
 7   # validate_associated_records_for_posts. If you have multiple
 8   # autosave associations, each can have a method such as this
 9   # one to handle it's validation needs.
10   def validate_associated_records_for_items
11     product_number_errors = false
12     items.group_by(&:product_number).reject {|_, group| group.length <= 1}.each do |product_number, group|
13       product_number_errors = true
14       errors.add_to_base "Product #{product_number} used #{group.length} times on the order"
15     end
16 
17     errors.add_to_base("Each product number can be used at most once per order") if product_number_errors
18   end
19 end

With this validation in place, validations will run on the full collection in a single pass, rather than piecemeal.

1 > o = Order.new
2 > 2.times { o.items << Item.new(:product_number => "A-113") }
3 > o.save # Will run in-memory validations
4  => false 
5 > pp o.errors.full_messages
6 ["Product A-113 used 2 times on the order",
7  "Each product number can be used at most once per order"]

Notice I didn’t say anything about the validates_uniqueness_of validation in Item. I left it in place. It doesn’t harm anything, and if you ever create items without going through the parent model, your code is ready to take care of itself.

Bookmarks Are King, Even on Applications

2010-05-24T09:54:46-04:00

I built an application for a non-profit in my region. It allows them to track donations and statistics about their work. One thing that’s very important for them is to know if / when they called a previous donor, to see if they will give again this year. They also need to emit receipts for those people that donated. And they must not emit the same receipt twice, etc. You know the drill. I have a page in the application where they can build such a query:

On this page, they can choose which campaign they’re working on, and different event types that occur during a campaign. They can also choose how to present the result: the date at which the event occured, or the sum of money or quantity they acquired. It’s a large form, since there are almost 15 event types that can occur in a single campaign.

René, my customer, asked me if they could save queries between sessions, so they can go home in the evening, and come back the next day. I said “Sure, I’ll just need a Search model here, and … Hold on a minute: I have a much simpler solution: bookmark the result page.”

You see, I built this application on REST principles. The query itself is idempotent: it either shows events, or not. And since this is idempotent, a GET query is just what the doctor ordered. And browsers are nice enough to allow bookmarking GET requests. Yup: 10 hours of work just disappeared because I used what’s already available. My customer was happy: he didn’t have to wait a minute to get the functionality he needed. I was happy because I had 10 hours of work NOT to do, and I don’t have to maintain that code: it’s in somebody else’s hands. TSTTCPW

This was a real win-win situation.

What Does Your Workday Look Like?

2010-05-23T10:51:23-04:00

I spend 70% to 90% of my day in a full-screen iTerm, in Vim. I find it ironic that I use a 3kCAD machine to do what green screen terminals did 30 years ago… My iTerm’s color scheme is green text on a black background. I read somewhere that this color combination was the easiest on eyes, but can’t find the reference right now.

When I’m not in Vim, I’m usually in Safari or Firefox. What does your day look like?

FAQs Reduce Support Requests

2010-05-21T08:31:54-04:00

I’m a happy DreamHost customer, for many years. Recently, they moved my data to a new server. That in itself wasn’t a problem, but this blog went down with their “bad_httpd_conf” error. DreamHost support has always been prompt to answer my questions. But more importantly, when a customer asks for support, they have answer right where it matters.

As part of the support process, DreamHost asks what site you have a problem with. Then they run some diagnostics on the network and report those as warnings.

Click to view full size

After they’ve done that, DreamHost asks for what type of problem you have. And the different categories have tips you can apply to try to correct the problem, before you have to wait for DreamHost support.

Click to view full size

Another thing that’s very nice: after logging in, one of the 8 main options is, you guessed it, Support. DreamHost doesn’t hide their support. It’s very visible. I’m very satisfied with the services I use.

Using Watchr to Check on Your Tests

2010-05-20T21:52:51-04:00

I started using Watchr in a couple of projects. An interesting use of Watchr is to check on your tests. I have been using this script with great success so far:

test.watchr

 1 def failed
 2   system("growlnotify --name adgear-reporting-tests --image /Applications/Mail.app/Contents/Resources/Caution.tiff -m 'Oops, tests failed'")
 3   system("say 'failed'")
 4 end
 5 
 6 def succeeded
 7   system("growlnotify --name adgear-reporting-tests -m 'Green tests'")
 8   system("say 'pass'")
 9 end
10 
11 watch('(lib|test)/.+\.(js|rb)') do |_|
12   system("rake")
13   $?.success? ? succeeded : failed
14 end

Notice I’m using Mac OS X application paths, but it works just fine for me. And the regular “pass” and “failed” messages keep me abreast of what’s going on. But this style of continuous testing only works when your tests take a few seconds at most. When my tests take much longer than that, I start fiddling, opening Google Reader, checking news, whatever.

Your mileage may vary.

Use Rdebug When Debugging Statements Don't Take Effect

2010-03-15T12:00:51-04:00

I was meta-meta-programming, and had an issue when an #included method was called:

 1 anon = Module.new
 2 anon.class_eval do
 3   def self.included(base)
 4     debugger
 5     if some_method then
 6       # code
 7     else
 8       # more code
 9     end
10   end
11 end

The code above resulted in:

1 /Users/francois/Projects/project/lib/extensions.rb:214:in `included': undefined local variable or method `some_method' for #<Module:0x103305b20> (NameError)
2         from /Users/francois/Projects/project/lib/extensions.rb:245:in `include'
3         from /Users/francois/Projects/project/lib/extensions.rb:245:in `send'
4         ...
5

The debugger statement above just wouldn’t take: Ruby ran right over it. I happened to look at ruby-debug’s Rubygem spec file:

1 --- !ruby/object:Gem::Specification 
2 name: ruby-debug
3 ...
4 executables: 
5 - rdebug
6 ...

Oh, had never noticed the executables before… Sure enough, I managed to run under debugger control immediately:

 1 $ rdebug -I test test/functional/ad_spots_controller_test.rb [-4, 5] in /Users/francois/Projects/bloom/adgear-admin/test/functional/ad_spots_controller_test.rb
 2 => 1  require "test_helper"
 3    2  
 4    3  class AdSpotsControllerTest < ActionController::TestCase
 5    4  
 6    5    def setup
 7 /Users/francois/Projects/bloom/adgear-admin/test/functional/ad_spots_controller_test.rb:1
 8 require "test_helper"
 9 (rdb:1)

From there, I was able to use (C)ontinue to end up on my debugger statement. Happy times ensued!

For the curious, the NoMethodError is because #some_method is defined on base, not on self. self in this context is the included module, while base is the place where we’re including it into.

Hobo Fields: Lots of ActiveRecord Goodness

2010-03-03T15:31:05-05:00

What? You’re still generating migrations manually? I don’t anymore. I have tools which do it for me.

Hobofields is a set of extensions to ActiveRecord that gives us lots of goodies, including the ability to generate migrations automatically. The key to achieving this automatic property declaration is schema declaration in the model.

1 class Post < ActiveRecord::Base
2   fields do
3     title :string
4     published_at :datetime
5     timestamps
6   end
7 end

If you ever used DataMapper, the idea is similar. Where Hobofields shines though is in reading your model and inferring many, many conventions:

1 class Comment < ActiveRecord::Base
2   belongs_to :post
3 end

Running the hobo_migration generator results in:

 1 $ script/generate hobo_migration
 2 
 3 ---------- Up Migration ----------
 4 add_column :comments, :post_id, :integer
 5 
 6 add_index :comments, [:post_id]
 7 ----------------------------------
 8 
 9 ---------- Down Migration --------
10 remove_column :comments, :post_id
11 
12 remove_index :comments, :name => :index_comments_on_post_id rescue ActiveRecord::StatementInvalid
13 ----------------------------------

If you use acts_as_list, Hobofields also knows to add a position column (or whatever name is needed):

1 class Comment < ActiveRecord::Base
2   acts_as_list
3 end

Running the migration generator, you get:

 1 $ script/generate  hobo_migration
 2 
 3 ---------- Up Migration ----------
 4 add_column :comments, :position, :integer
 5 ----------------------------------
 6 
 7 ---------- Down Migration --------
 8 remove_column :comments, :position
 9 ----------------------------------
10 What now: [g]enerate migration, generate and [m]igrate now or [c]ancel?

Another thing I love about Hobofields is rich type support. Need Textile?

1 class Post < ActiveRecord::Base
2   fields do
3     body :textile
4   end
5 end

 1 content = <<EOTEXTILE
 2 This is some Textile content
 3 
 4 h1. Automatically formatted
 5 
 6 To your:
 7 
 8 * specifications,
 9 * using standard tools
10 EOTEXTILE
11 
12 Post.new(:body => content).body.to_html
13 #=> "<p>This is some Textile content</p>\n<h1>Automatically formatted</h1>\n<p>To your:</p>\n<ul><li>specifications,</li>\n<li>using standard tools</li>\n</ul>\n"

Or maybe you need an enumeration?

1 class Comment < ActiveRecord::Base
2   fields do
3     status enum_string(:spam, :ham), :default => :spam, :required => true
4   end
5 end

 1 $ script/generate hobo_migration
 2 
 3 ---------- Up Migration ----------
 4 add_column :comments, :status, :string, :default => "ham", :required => true
 5 ----------------------------------
 6 
 7 ---------- Down Migration --------
 8 remove_column :comments, :status
 9 ----------------------------------

1 >>Comment.new.status
2 #=> "ham"
3 >> Comment.create!(:status => "foo")
4 #=> ActiveRecord::RecordInvalid: Validation failed: Status must be one of spam, ham

Admittedly, this last example could be better handled with Hobo’s lifecycle (state machine) implementation, but still, it’s nice that the validation already exists. Using :null => false, :required => true creates the appropriate validation as well:

1 class Comment < ActiveRecord::Base
2   fields do
3     author :string, :required
4     email  :string, :required
5     status enum_string(:spam, :ham), :required => true, :default => "ham"
6     timestamps
7   end
8 end

And Hobofields is smart enough to know when no changes are required:

1 $ script/generate hobo_migration
2 Database and models match -- nothing to change

But with this model definition, we now have new validation automatically:

1 >> Comment.create!
2 #=> ActiveRecord::RecordInvalid: Validation failed: Author can't be blank, Email can't be blank

Oh wait, just realized that the email address should be an email address:

1 class Comment < ActiveRecord::Base
2   fields do
3     email :email_address, :required
4   end
5 end

1 >> Comment.create!(:email => "a")
2 #=> ActiveRecord::RecordInvalid: Validation failed: Author can't be blank, Email is invalid

I really enjoy Hobofields for all the goodies it brings to the table. I haven’t even started on all the automatic scopes goodies, the lifecycle methods (which I haven’t used yet), or scoped associations.

Until we meet again tomorrow, I suggest you read Hobofields Reference, as well as Hobofields rich types to get the full goods.

Using Hobo: Info From the Trenches

2010-03-02T11:41:47-05:00

Recently, a friend asked me to help him rewrite the Access database they use to track donations into something better. I offered to rewrite as a web app (surprise!), and chose to use Hobo.

I’m not a designer, and it shows. My apps are horrible, and have bad choice of color. Red on blue? Check. Green on orange? Check. Wide margins and misaligned text base? Check. Name any design mistakes, and I can make it happen.

I had seen Hobo used by another developer, and I was impressed with the ActiveRecord extensions. There was also the fact that Hobo has a pleasing design that could serve as an initial draft, until external help can be acquired.

Hobo isn’t really different from Rails: hobo name_of_new_app. Done.

Things I liked immediately: Declarative declaration of fields and validations, automatic generation of migrations, permissions built-in. You know, stuff that covers the 80%

The thing I didn’t like: DRYML. Don’t get me wrong: DRYML is a very nice tool, and I didn’t have any problems understanding the implicit context. All I had was the age-old problem: learning a new API. After 2-3 weeks (very part time), I was more comfortable with the different tags, and how to use them.

I really enjoy writing this:

1 <table-plus fields="this, city.name, toll_free_number, main_number"/>

and receiving this in return:

Click for larger view

Right now, I’ve had a very positive experience with Hobo. Having good documentation helps a lot, and Hobo hasn’t disappointed: http://cookbook.hobocentral.net/

Over the next few days, I’ll discuss Hobo in more details. Hope you follow along!

Looking for Best Practice on Using Bundler With Multiple Ruby Implementations

2010-02-24T18:56:34-05:00

At Bloom, we’re looking into using JRuby instead of MRI. To support the two implementations, some gems are different. Case in point: RMagick and the JSON gems:

config/environment.rb

 1 # More config.gem declarations elided
 2 
 3 if defined?(JRUBY_VERSION) then
 4   config.gem "activerecord-jdbc-adapter", :version => "0.9.2",  :lib => "jdbc_adapter"
 5   config.gem "json-jruby",  :version => "1.2.0",  :lib => "json"
 6   config.gem "rmagick4j",   :version => "0.3.6",  :lib => "RMagick"
 7 else
 8   config.gem "json",        :version => "1.2.0"
 9   config.gem "rmagick",     :version => "2.12.2", :lib => "RMagick"
10 end

I created the following in Gemfile:

Gemfile

 1 source :gemcutter
 2 
 3 gem "rails", "= 2.3.4"
 4 gem "pg", "~> 0.8.0" if !defined?(JRUBY_VERSION)
 5 
 6 if defined?(JRUBY_VERSION) then
 7   gem "activerecord-jdbc-adapter", ">= 0.9.2",  :lib => "jdbc_adapter"
 8   gem "json-jruby", ">= 1.2.0"
 9   gem "rmagick4j", ">= 0.3.6"
10 else
11   gem "json", ">= 1.2.0"
12   gem "rmagick", ">= 2.12.2"
13 end
14 
15 # Ruby's 1.9 stdlib replaced CSV with the implementation of 1.8's FasterCSV, but
16 # the upgrade path is problematic: we have to change how we're loading the library.
17 gem "fastercsv" if RUBY_VERSION =~ /^1\.8/

Notice I target 3 versions of Ruby: MRI 1.8, MRI 1.9 and JRuby. If I bundle lock from one Ruby implementation, then the other ones won’t be able to use the locked bundle:

 1 [jruby-1.4.0]$ bundle unlock && bundle lock
 2 The bundle is now unlocked. The dependencies may be changed.
 3 The bundle is now locked. Use `bundle show` to list the gems in the environment.
 4 
 5 [jruby-1.4.0]$ rvm use 1.8.7
 6 Now using ruby 1.8.7 p248
 7 
 8 [ruby-1.8.7-p248]$ bundle check
 9 Could not find gem 'rmagick4j (= 0.3.7, runtime)' in any of the sources.

I’m using Bundler in the context of a Rails 2.3.4 application, but we’ll eventually move to Rails 3. Anybody has any experience using Bundler in this way?

Nestor: A Butler for Your Tests

2009-11-18T09:38:47-05:00

At yesterday’s Montreal.rb I presented Nestor, an autotest-like framework. This is it’s official release announcement.

Nestor is different in that it uses an explicit state machine, namely Aaron Pfeifer‘s StateMachine. Nestor also uses Martin Aumont’s Watchr to listen to filesystem events. But the biggest difference is that the default Rails + Test::Unit is a forking test server. Nestor will load the framework classes—ActiveRecord, ActionController, ActionView, plugins and gems—only once. This saves a lot of time on the aggregate versus running rake everytime.

Click for larger version

This release of Nestor is 0.2 quality: it’s not ready for large projects. It only supports Rails + Test::Unit, probably doesn’t run on 1.9 or JRuby, but it’s a solid foundation for going further. In the coming days, I will blog on the internals of Nestor and how StateMachine allowed me to add plugins with very little effort.

Installation

1 $ gem install nestor
2 $ cd railsapp
3 $ # edit config/environments/test.rb to set cache_classes to false
4 $ nestor

I already have a plugin that enables Growl notifications. Install and use:

1 $ gem install nestor_growl
2 $ cd railsapp
3 $ nestor start --require nestor/growl

The —require option is where plugins are loaded. This is an Array of files Nestor will require on startup.

Notes

You must set cache_classes to false in test mode for now. This is a limitation of how Rails boots. With cache_classes set to true, Rails will load the controllers and models when it boots. Since this happens before forking, the code under test would never get reloaded. Did I say it was 0.2 quality?

Piston 2.0.8: minor bugfix release

2009-11-18T09:38:47-05:00

Piston 2.0.8 is here. This is a minor bugfix release:

piston status with no path would not check any status. Thanks to Michael Grosser for the heads up;
The ActiveSupport gem deprecated require “activesupport” in favor of “active_support”. Thanks for Michael for reporting this as well;
Scott Johnson reported and fixed a problem where a git mv would fail because a parent directory was missing.

Thanks to all contributors!

Installing

1 $ gem install piston

Entering the Enterprise Space: Moving to Bloom Digital Platforms

2009-09-29T10:31:31-04:00

After a few years as a consultant, I received an offer from Bosko Milekic from Bloom Digital Platforms Inc. Starting on October 5th, I will be a permanent Bloom employee. Marc-André and Gary are moving outside the enterprise, while I’m moving inside.

At Bloom, I will be responsible for the AdGear API, enabling applications to easily connect to their platform for ad serving and management. We even have an open source Ruby client available on GitHub as ad_gear_client.

This will be a welcome change of pace for me.

See you at the next Montreal.rb, at my new office!

RestClient + Multipart Form Data: See My Branch

2009-08-12T11:05:38-04:00

I just merged Rick Olson’s (technoweenie) rest-client multipart_streaming branch with RestClient 1.0.4. You can see the results at francois/rest-client. I sent a pull request to Adam, so if he pulls it in, everyone will benefit from this code.

Using ActiveResource With Digest Authentication: Branch in Progress

2009-08-04T16:04:11-04:00

I have a work-in-progress branch for using Digest authentication with ActiveResource.

So, how did I do it? It wasn’t too hard actually. When I spiked, I changed ActiveResource::Connection#request to handle authentication itself. I ended up with a big mess: a new rescue clause, 10 lines of code to calculate the digest and so on. But I knew it would work. So, I git checkout . and started with tests, as it should.

The way ActiveResource is built, if a username / password is sent in, ActiveResource will send those automatically in an Authorization header, using the Basic authentication method. I need a way to turn this off. Thus grew #use_basic_authentication= and #use_digest_authentication=.

Next up, actually being able to calculate the Digest. A quick search turned up code by Eric Hodel in the form of a Ruby module: An implementation of HTTP Digest Authentication in Ruby

After a bit of cleanup and rewriting, I have a branch of ActiveResource that’s ready to be commented on. Please see francois/ar_digest and leave comments there.

An example of using Digest would be:

 1 require "logger"
 2 require "activeresource"
 3 require "pp"
 4 
 5 ActiveResource::Base.logger = Logger.new(STDERR)
 6 
 7 ActiveResource::Base.site     = "http://adgear.local/api"
 8 ActiveResource::Base.user     = "francois"
 9 ActiveResource::Base.password = "my-funny-new-password-which-you've-never-seen-before"
10 ActiveResource::Base.timeout  = 30
11 
12 # Don't attempt Basic authentication, but be sure to use Digest
13 ActiveResource::Base.use_basic_authentication  = false
14 ActiveResource::Base.use_digest_authentication = true
15 
16 class Site < ActiveResource::Base
17 end
18 
19 pp sites = Site.find(:all)

This work was sponsored by Bloom Digital Platforms, as part of my work on their AdGear API.

Piston 2.0.6: minor bug fixes

2009-08-01T13:21:14-04:00

I just pulled in a couple of patches from outside contributors. These are all minor bug fixes, but are important:

Chris Gibson: When forcing the repository type, Piston would break because it called #downcase on a Symbol. 1bcc16bf8
Terry Heath: Subversion‘s —non-interactive would prevent OS X’s keychain from kicking in. 93d9a957
Florian Aßmann: In certain cases, the revision would be a String, and other times it would be an Integer. Normalize before comparing. 40c0bc4e

All users are advised to upgrade:

1 $ sudo gem install piston

Clear and Concise Explanation of RESTful Resources

2009-07-22T14:30:23-04:00

I just found Your Web Service Might Not Be RESTful If… by Paul Sadauskas. This article is a very clear and concise explanation of how to do real RESTful services.

His arguments are:

1. Present a single top-level resource;
2. All resources should be accessible by followling links from the top-level resource;
3. Don’t put API tokens in the URL or headers (use representations instead).

A very enjoyable read, and something we should all be following.

Piston 2.0.4: minor fixes

2009-07-20T11:18:11-04:00

I just release Piston 2.0.4. This is a minor fix to enable Piston to work with more Git versions.

Git 1.6.0.1 accepts checking out a new branch without specifying the remote’s name:

1 $ git checkout -b my-2-3-stable 2-3-stable
2 # implies origin/2-3-stable

Versions prior to that expect the full name of the remote branch:

1 $ git checkout -b my-2-3-stable origin/2-3-stable

Installation is the usual incantation: sudo gem install piston

What is Piston?

Piston is a utility that eases vendor branch management. This is similar to svn:externals, except you have a local copy of the files, which you can modify at will. As long as the changes are mergeable, you should have no problems.

MongoDB or Redis? The Choice Is Yours

2009-07-15T09:35:54-04:00

In the past couple of weeks, I had the opportunity to look at MongoDB and Redis in more details. Both are fast, but serve different needs. Here’s my take on them both.

MongoDB

MongoDB is a document oriented database. MongoDB stores structured data, as documents:

1 {
2   'name': 'François Beausoleil',
3   'email': 'francois@teksol.info',
4   'year-of-birth': 1973,
5   'tags': ['friendly', 'ruby', 'coder', 'father']
6 }

Since MongoDB stores structured data, it has a richer query language:

 1 // Find people that are fathers OR coders
 2 db.people.find( { "tags" : { $in : ['father', 'coder'] } } );
 3 
 4 // Find people that are fathers AND coders
 5 db.people.find( { "tags" : { $all : ['father', 'coder'] } } );
 6 
 7 // Index the email member of documents
 8 db.people.ensureIndex({email: 1});
 9 
10 // Query using the index
11 db.people.findOne({email: 'francois@teksol.info'})

Redis

Redis is a key-value store that is strongly typed. Redis has three types of values: strings, sets and lists. Out of the box, you cannot store structured data in Redis. You have to build the index manually, storing yet other keys.

Redis is conceptually simple. All operations are either stores or fetches:

1 PUT key value
2 GET key

You can think of Redis as a persistent Hash or Dictionary. The fact that Redis is strongly typed makes some operations very interesting.

 1 # Store other data as serialized JSON object, but we won't be able to query the value itself
 2 SET "people:francois" "{'name': 'François Beausoleil', 'email': 'francois@teksol.info', 'year-of-birth': 1973, 'tags': ['friendly', 'ruby', 'coder', 'father']}"
 3 
 4 # Build ourselves an index on the email value
 5 SET "email:francois@teksol.info" "francois"
 6 
 7 # Build another index for tags
 8 SADD "tags:friendly" "francois"
 9 SADD "tags:ruby"     "francois"
10 SADD "tags:coder"    "francois"
11 SADD "tags:father"   "francois"
12 
13 # During authentication, we would find the person using the email address
14 GET "email:francois@teksol.info"
15 
16 # Get the rest of the data
17 GET "people:francois"
18 
19 # Find the keys that have both father AND coder as tags
20 SINTER "tags:father" "tags:coder"

Conclusion

So, which one should you choose? The short answer is “it depends”.

The longer answer is to use the best tool for the job. Storing documents is different than storing simple key values. Of course, both can be used for the same job, but Redis will be easier if you need a simple Hash/Dictionary. Use MongoDB for storing structured data.

Cliaws 1.5.0: cliec2 commands implemented

2009-07-07T20:48:44-04:00

Please grab cliaws from Rubyforge:

1 $ gem install cliaws

What is Cliaws?

Cliaws is a replacement for the Amazon EC2 API tools, but uses Ruby, and thus does not suffer from a long boot time. Cliaws is also easier to setup:

1 $ AWS_ACCESS_KEY_ID=# Your Access ID
2 $ AWS_SECRET_ACCESS_KEY=# Your secret ID
3 $ # Setup is done, enjoy!
4 $ clis3 list YOUR_BUCKET
5 $ cliec2 launch AMI --keypair KEYNAME

Please view the README for some details. The implementation is still the best place to get information about the options you can pass.

We've Been Doing TDD for Centuries, You Know..

2009-06-20T21:55:29-04:00

Today, I was working on my lawn. Spreading new soil to sow new grass. I was looking at the ground, and I was pretty sure drainage wasn’t going to be good. I decided to make a test: I got out the garden hose and let it flow. Sure enough, water was pooling instead of draining. Back to the drawing board (or at least, a couple more rakes to spread things around).

As I was working the soil, a thought hit me: I’d say nearly every profession has been doing test driven design (not development) since pretty much the dawn of time:

the toolsmith in the ancient tribes would hit the stone, see if it did work or not, and repeat as appropriate;
the chef tastes his food before letting other people eat it;
the plumber tests the pipes before letting the water flow in;
the soldier tests his equipment before setting off on the battlefield;
naval engineers are simulation testing their latest aircraft carrier in an effort to ensure nothing is out of place;
certain programmers write automated tests for their code, to ensure it works as designed.

If nearly every profession has been doing it for thousands of years, why aren’t you doing it today?

Piston on Windows: Some Pointers

2009-06-07T07:34:20-04:00

If you need to run Piston on Windows, BoxCycle wrote some information about it at: Rails Plugin Updates, SVN, and Piston 2.0.2 on Windows

How to Parse HTML Documents Under JRuby Using XOM, TagSoup and Nux

2009-05-01T16:03:16-04:00

It’s not that hard, but it still took me 2 hours to do it. I had a couple of false starts, and I pored over documentation for a while until I hit upon the excellent Nux library.

I won’t let you go through the same failures I had. Here’s the code:

 1 # Demonstrates how to parse a local HTML document using XOM,
 2 # TagSoup and Nux, under JRuby.
 3 #
 4 # http://www.xom.nu/
 5 # http://home.ccil.org/~cowan/XML/tagsoup/
 6 # http://acs.lbl.gov/nux/
 7 
 8 include Java
 9 mydir = File.expand_path(File.dirname(__FILE__))
10 
11 # This is how you require libraries without touching your
12 # CLASSPATH from JRuby. I put the required files in vendor/.
13 # Nux includes it's dependencies (XOM and saxon), so I didn't
14 # have any other libraries to add.
15 require File.join(mydir, "vendor", "tagsoup.jar")
16 %w(nux.jar saxon8.jar xom.jar).each do |filename|
17   require File.join(mydir, "vendor", "nux", "lib", filename)
18 end
19 
20 import "org.ccil.cowan.tagsoup.Parser"
21 import "nu.xom.Builder"
22 
23 builder = Builder.new(Parser.new)
24 
25 # XOM's Builder expects a full URL, so tell it where to find the
26 # document.
27 doc = builder.build("file://#{File.expand_path(File.join(mydir, ARGV[0]))}")
28 puts doc.toXML

Extra! Add XPath querying

Continuing from above, you can add XPath querying:

1 import "nux.xom.xquery.XQueryUtil"
2 
3 # Must use '*:p'.  '*' stands for any/default namespace.
4 results = XQueryUtil.xquery(doc, "//*:p")
5 p results.size
6 results.size.times do |index|
7   puts results.get(index).toXML
8 end

Why am I going through these motions? Because I wanted to use my 20% for fun. Besides, I need to process large quantities of HTML as quickly as possible for a cool project I’m working on, and JRuby seems to be the fastest implementation, according to my unscientific benchmark.

But the real reason was that both Nokogiri and Hpricot wouldn’t load/run under JRuby 1.2.0.

Actually, let me rephrase that: Nokogiri did load, but crashed while requiring the library:

 1 $ jruby -w test.rb data.html 
 2 /Users/francois/Library/Java/JRuby/jruby-1.2.0/lib/ruby/gems/1.8/gems/nokogiri-1.2.3-java/lib/nokogiri/xml/node.rb:180: undefined method `next_sibling' for class `Nokogiri::XML::Node' (NameError)
 3         from /Users/francois/Library/Java/JRuby/jruby-1.2.0/lib/ruby/gems/1.8/gems/nokogiri-1.2.3-java/lib/nokogiri/xml/node.rb:31:in `require'
 4         from /Users/francois/Library/Java/JRuby/current/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
 5         from /Users/francois/Library/Java/JRuby/jruby-1.2.0/lib/ruby/gems/1.8/gems/nokogiri-1.2.3-java/lib/nokogiri/xml.rb:3
 6         from /Users/francois/Library/Java/JRuby/jruby-1.2.0/lib/ruby/gems/1.8/gems/nokogiri-1.2.3-java/lib/nokogiri/xml.rb:31:in `require'
 7         from /Users/francois/Library/Java/JRuby/current/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
 8         from /Users/francois/Library/Java/JRuby/jruby-1.2.0/lib/ruby/gems/1.8/gems/nokogiri-1.2.3-java/lib/nokogiri.rb:10
 9         from /Users/francois/Library/Java/JRuby/jruby-1.2.0/lib/ruby/gems/1.8/gems/nokogiri-1.2.3-java/lib/nokogiri.rb:36:in `require'
10         from /Users/francois/Library/Java/JRuby/current/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
11         from test.rb:2
12

I have reported this bug to the proper authorities.

Hpricot is another matter entirely. When I tried to use it earlier, I hit a roadblock because JRuby couldn’t install the native extensions. I tried again just now, and if you specify the version to be ~> 0.6.1, it works. Specify any other version, and you’re a sitting duck:

 1 $ jruby -S gem install -v '~> 0.6' hpricot
 2 Building native extensions.  This could take a while...
 3 ERROR:  Error installing hpricot:
 4         ERROR: Failed to build gem native extension.
 5 
 6 /Users/francois/Library/Java/JRuby/current/bin/jruby extconf.rb install -v ~> 0.6 hpricot
 7 
 8 
 9 Gem files will remain installed in /Users/francois/Library/Java/JRuby/jruby-1.2.0/lib/ruby/gems/1.8/gems/hpricot-0.8.1 for inspection.
10 Results logged to /Users/francois/Library/Java/JRuby/jruby-1.2.0/lib/ruby/gems/1.8/gems/hpricot-0.8.1/ext/hpricot_scan/gem_make.out
11 
12 $ jruby -S gem install -v '~> 0.6.1' hpricot
13 Successfully installed hpricot-0.6.164-java
14 1 gem installed
15 Installing ri documentation for hpricot-0.6.164-java...

Keeping Yourself DRY in Shoulda Tests

2009-04-29T22:02:42-04:00

When I’m testing admin controllers, I often have tests that follow this form:

test/functional/admin/orders_controller_test.rb

 1 class OrdersControllerTest < ActionController::TestCase
 2   logged_in_as :active_user do
 3     context "on GET to :index" do
 4       setup do
 5         get :index
 6       end
 7 
 8       should_deny_access
 9     end
10   end
11 
12   not_logged_in do
13     context "on GET to :index" do
14       setup do
15         get :index
16       end
17 
18       should_deny_access
19     end
20   end
21 end

Well, this is all Ruby, right? And Ruby has wonderful blocks, and blocks can be passed around…

test/functional/admin/orders_controller_test.rb

 1 class OrdersControllerTest < ActionController::TestCase
 2   deny_access_tests = lambda do
 3     context "on GET to :index" do
 4       setup do
 5         get :index
 6       end
 7 
 8       should_deny_access
 9     end
10   end
11 
12   logged_in_as :active_user, &deny_access_tests
13   not_logged_in, &deny_access_tests
14 end

This is valid for any block of code that you want to test again and again:

test/functional/admin/orders_controller_test.rb

 1 class OrdersControllerTest < ActionController::TestCase
 2   successful_index_render = lambda do
 3     should_respond_with :success
 4     should_render_template "new"
 5     should_assign_to :orders
 6   end
 7 
 8   logged_in_as :admin do
 9     context "", &successful_index_render
10   end
11 
12   logged_in_as :sub_admin do
13     context "", &successful_index_render
14   end
15 end

Alternatively, and it might be easier in the end, you could use methods:

test/functional/admin/orders_controller_test.rb

 1 class OrdersControllerTest < ActionController::TestCase
 2   def self.should_render_successful_index_response
 3     should_respond_with :success
 4     should_render_template "new"
 5     should_assign_to :orders
 6   end
 7     
 8   logged_in_as :admin do
 9     should_render_successful_index_response
10   end
11 
12   logged_in_as :sub_admin do
13     should_render_successful_index_response
14   end
15 end

Note thought that you must define your methods at the top of your test case. Remember that Ruby executes a class definition, so when you suddenly call should_render_successful_index_response, the method definition has to be available, or else Ruby will complain with a NoMethodError.

Ain’t Ruby sweet?

What if You Didn't Have Access to If/Then/Else?

2009-04-17T11:33:16-04:00

Image a Ruby where if/then/else isn’t available:

1 class Account
2   # This method returns the balance in words, negative or positive, ready for display in the UI
3   def balance_in_words
4     # Err...  How do I do that?
5   end
6 end

If you were a strict Object-Oriented person, you’d do it this way:

 1 class PositiveBalanceAccount
 2   def balance_in_words
 3     "Positive"
 4   end
 5 end
 6 
 7 class NegativeBalanceAccount
 8   def balance_in_words
 9     "Negative"
10   end
11 end

But then, your objects would have to change class whily-nilly. Pretty darn hard. But multi-method dispatching gives us a nice solution:

 1 class Account
 2   def initialize(balance)
 3     @balance = balance
 4   end
 5 
 6   defmulti :balance_in_words,
 7     lambda { @balance < 0 } => "Negative",
 8     lambda { @balance > 0 } => "Positive"
 9 end
10 
11 Account.new(15).balance_in_words # => "Positive"
12 Account.new(-5).balance_in_words # => "Negative"

For the observant amongst you, you might have noticed I forgot the nil balance case. This is really a programming error, so it should be treated as such:

1 Account.new(0).balance_in_words
2 lib/defmulti.rb:46:in `balance_in_words': #<Account:0x197222c @balance=0> received balance_in_words but did not have a guard clause that matched and no else clause. (Defmulti::MissingGuardClause)
3         from test.rb:16
4

For the even more observant, yes, there exists an implementation that does exactly what I have described above. It’s called defmulti, it’s a gem, and it’s on GitHub: http://github.com/francois/defmulti.

What is this useful for?

This library is a thought experiment. When you lose the familiar tools, what can you do? Of course, this library works atop the existing Ruby implementation and to be truly useful, syntax would have to be provided to make this much less verbose. It’s interesting nonetheless to see what can be done without the syntax extensions.

Anyway, what would you use this for? Multi-method dispatching is a tool that helps writing code without conditionals. The conditionals are specified outside the block of code that executes. The examples above are pretty thin, but looking at Java.next #3: Dispatch, I can provide another solution to his Ruby example:

 1 defmulti :letter_grade,
 2   lambda {|grade| (90..100).include?(grade) || grade == "A"} => "A",
 3   lambda {|grade| (80...90).include?(grade) || grade == "B"} => "B",
 4   lambda {|grade| (70...80).include?(grade) || grade == "C"} => "C",
 5   lambda {|grade| (60...70).include?(grade) || grade == "D"} => "D",
 6   lambda {|grade| ( 0...60).include?(grade) || grade == "F"} => "F"
 7 
 8 letter_grade 60  # => "D"
 9 letter_grade "A" # => "A"
10 letter_grade nil # => Defmulti::MissingGuardClause

Again, I’m struck by the clunky syntax, but if we ignore that for a second, could this be even better than Stuart Halloway’s example? Ruby’s case statement is very, very powerful and very easy to use. Is this useful? Not at the moment. But it’s a thought experiment I thought I’d throw out there.

Beware of Git's Content Tracking

2009-04-15T00:31:52-04:00

A while ago, I was wondering how rename tracking in Git worked. I was told that renames didn’t really exist in Git, as Git tracked content, not files themselves. Fair enough.

But, I just stumbled upon something:

 1 $ gdc vendor/plugins/acts_as_money/LICENSE 
 2 diff --git a/vendor/plugins/acts_as_money/LICENSE b/vendor/plugins/acts_as_money/LICENSE
 3 index e69de29..a273c73 100644
 4 --- a/vendor/plugins/acts_as_money/LICENSE
 5 +++ b/vendor/plugins/acts_as_money/LICENSE
 6 @@ -0,0 +1,4 @@
 7 +one:
 8 +  user: active
 9 +  name: name
10 +  description: description

Tell me, does that look right? I’ll manually fix that file, but that just decreased my confidence level in Git.

The original LICENSE file was empty, and there apparently was another file (a fixture file) that was empty too, and the latter saw some content added.

Why did this happen? LICENSE was updated as part of a merge—from a branch in which the LICENSE file doesn’t exist.

Returning CSV data to the browser - revisited

2009-04-06T10:23:30-04:00

Back in 2006, I wrote Returning CSV data to the browser. The method I was using back then is way obsolete. Let’s use the correct way to do it. If you want to follow along, just browse over to returning-csv-data-to-the-browser-revisited and read on.

First of all, the correct way to return non HTML data to the browser is to use respond_to. Let’s do so here:

app/controllers/reports/timelines_controller.rb

 1 class Reports::TimelinesController < ApplicationController
 2   def show
 3     @timelines = Timeline.all
 4     respond_to do |format|
 5       format.csv do
 6         response.headers["Content-Type"]        = "text/csv; charset=UTF-8; header=present"
 7         response.headers["Content-Disposition"] = "attachment; filename=timeline-report.csv"
 8       end
 9     end
10   end
11 end

The rest is pretty easy. Generating the data is easy enough:

app/models/timeline.rb

 1 require "fastercsv"
 2 
 3 class Timeline < ActiveRecord::Base
 4   default_scope :order => "started_at ASC"
 5 
 6   def to_csv
 7     FasterCSV.generate_line([
 8           started_at.to_s(:db),
 9           ended_at.to_s(:db),
10           project_id]).chomp
11   end
12 end

And rendering the view? Look how easy this gets:

app/views/reports/timelines/show.csv.erb

1 Started At,Ended At,Project ID
2 <%= render :partial => @timelines %>

Notice the file’s name? show.csv.erb? The csv in the filename is what connects the respond_to call with the view.

And the final piece, the timeline partial:

app/views/reports/timelines/_timeline.csv.erb

1 <%= timeline.to_csv %>
2

Now, on to the next obsolete article in the bunch…

Building the SQL WHERE Clause Dynamically - Updated

2009-04-02T09:29:11-04:00

Back in October 2005, I wrote an article entitled Building the SQL WHERE Clause Dynamically. This article is obsolete and I’m replacing it with this one instead.

What do you do with a FORM that has multiple conditions? Something like this:

The way I would do it today is using named scopes:

app/models/person.rb

 1 class Person < ActiveRecord::Base
 2   is_taggable
 3 
 4   named_scope :in_region,
 5     lambda {|region| {:conditions => {:region => region}}}
 6   named_scope :with_salary,
 7     lambda {|lower, upper| {:conditions =>
 8       {:salary => (lower .. upper)}}}
 9 
10   # Elided, but something similar to the above
11   named_scope :tagged_with,
12     lambda {|tags| {:conditions =>
13       ...}}
14 end

Then you need your controller to translate between what the view provides and what the model expects:

app/controllers/searches_controller.rb

 1 class SearchesController < ApplicationController
 2   def show
 3     root = Person
 4     root = root.in_region(params[:region]) unless params[:region] =~ /all/i
 5     unless params[:salary] =~ /all/i
 6       params[:salary].gsub!("K", "000")
 7       lower, upper = params[:salary].split("-")
 8       root = root.with_salary(lower, upper)
 9     end
10     root = root.tagged_with(Tag.parse(params[:tags])) unless params[:tags].blank?
11     @people = root.paginate(:page => params[:page])
12   end
13 end

For simpler forms, the approach above would be sufficient. For more complex forms, I suggest you look at plugins that aim to provide more structure around your SQL clauses, such as:

I also suggest looking at Searching and Queries Plugins – Agile Web Development to find other plugins.

Actions With Dashes - Updated

2009-04-02T09:00:00-04:00

Back in October 2005, I wrote an article entitled Actions with Dashes. This article is perfectly obsolete and uses a trick instead of the router as it was intended.

Let’s say you want an action where the URL is http://myproject.com/contact-us. The best way to do that currently is to use Rails’ router:

config/routes.rb

1 map.contact_us "/contact-us", :controller => "pages", :action => "show", :page => "contact_us"

app/controllers/pages_controller.rb

1 class PagesController < ActionController
2   def show
3     render :action => params[:page]
4   end
5 end

The PagesController above is also good for any other static pages you want to serve on your site.