In my graph, I have several lines; Highcharts lets the user show/hide those lines manually. I want to make the data in the grid reflect what’s been shown or hidden in the graph.

I first tried to bind the Highcharts series via Knockout, somehow, to what should be shown in the grid. Maybe it’s possible, but I couldn’t figure out how to get Knockout to bind to variables deep inside the Highcharts object structure.

But Javascript and jQuery have event listeners, right? Hopefully Highcharts is nice enough to use those, so I can get an event when the series visibility is changed.

I looked in the Highcharts source, and down in the Series’ setVisibility method (more than 10000 lines deep), I found this:

fireEvent(series, showOrHide);

(The “showOrHide” variable is always either “show” or “hide”.)

That looks promising, but what does fireEvent do?

fireEvent = function (el, type, eventArguments, defaultFunction) {
    var event = jQ.Event(type),
        detachedType = 'detached' + type;
    extend(event, eventArguments);

    // Prevent jQuery from triggering the object method that is named the
    // same as the event. For example, if the event is 'select', jQuery
    // attempts calling el.select and it goes into a loop.
    if (el[type]) {
        el[detachedType] = el[type];
        el[type] = null;
    }

    // trigger it
    jQ(el).trigger(event);

    // attach the method
    if (el[detachedType]) {
        el[type] = el[detachedType];
        el[detachedType] = null;
    }

    if (defaultFunction && !event.isDefaultPrevented()) {
        defaultFunction(event);
    }
};

Most of that is necessary stuff that I don’t care about, but the key line is this one:

jQ(el).trigger(event);

That means that if I grab the Series object and wrap it with a jQuery call, I can call “bind” on it and be notified any time “trigger” is called. So I did that, for each series object:

var listener = function() {
    MyGrid.updateGrid(MyGraph.visibleSeries());
};
for (var i=0; i < this.chart.series.length; i++) {
    $(this.chart.series[i]).bind("show", listener);
    $(this.chart.series[i]).bind("hide", listener);
}

I created a function and saved it to a variable, and bound it to the “show” and “hide” events on each of my Series. I could have passed it in as an anonymous function, but I would have had to define it twice. I could have created it as an actual function on my object, but this way it’s scoped right here and won’t be accessible anywhere else.

On my MyGraph object, I need to be able to check which series are currently visible:

function visibleSeries() {
    var selected = [];
    for (var i=0; i < MyGraph.chart.series.length; i++) {
        if (MyGraph.chart.series[i].visible) {
            selected.push(MyGraph.chart.series[i].name);
        }
    }
    return selected;
}

That returns a list of series names, which I can then pass to the MyGrid object to update which lines it should display:

var updateGrid = function(events) {
    var newData = {};
    newData.page = 1;
    newData.total = 1;
    newData.rows = [];
    for (var i=0; i < MyGrid.gridData.rows.length; i++) {
        if ($.inArray(MyGrid.gridData.rows[i].cell[1], events) != -1) {
            newData.rows.push(MyGrid.gridData.rows[i]);
        }
    }
    newData.records = newData.rows.length;
    MyGrid.updateData(JSON.stringify(newData));
};

I’ve stored “gridData” in MyGrid, which is all the data for the grid that was returned from the server. It doesn’t get updated here, which means I can look at the full dataset again each time the visible series changes, and update the grid accordingly.

Victory! Now, whenever the user hides a series in the graph, it disappears from the grid. When they add it back, it reappears in the grid. The grid also updates its display of total records available, and the number of pages, and always jumps back to the first page.

I couldn’t find much documentation about how to use these, especially through the CLI (which is where I typically start out when trying to investigate a new feature). So here goes with an explanation of what I’ve found.

Create your “counter” column family:

create column family MyCounters with default_validation_class=CounterColumnType and comparator=UTF8Type;

Note that you don’t have to create the column family with a “default_validation_class” of CounterColumnType, but by doing so you get a few major advantages:

1. You can create arbitrary column names as counters, rather than having to add them explicitly and being required to know ahead of time what you’ll be counting

2. I couldn’t figure out how to actually do that, despite the fact that the documentation I read seemed to indicate it should be possible; instead, when I attempted to update column family with some column_metadata, it failed, like so:

   [default@MyTest] update column family MyCounters with column_metadata = [{column_name:thing-1, validation_class:CounterColumnType}];
   org.apache.thrift.TApplicationException: Internal error processing system_update_column_family
   

That’s all the information you get in the CLI. Apparently, when you see “Internal error”, that means you need to look in /var/log/cassandra/system.log, where you’ll see:

ERROR [pool-2-thread-1] 2012-01-26 09:41:01,705 Cassandra.java (line 4038) Internal error processing sys
tem_update_column_family
java.lang.RuntimeException: org.apache.cassandra.config.ConfigurationException: Cannot add a counter col
umn (thing-1) in a non counter column family
        at org.apache.cassandra.thrift.CassandraServer.system_update_column_family(CassandraServer.java:
1049)
        at org.apache.cassandra.thrift.Cassandra$Processor$system_update_column_family.process(Cassandra
.java:4032)
        at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
        at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer
.java:187)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.cassandra.config.ConfigurationException: Cannot add a counter column (thing-1) in a
 non counter column family
        at org.apache.cassandra.config.CFMetaData.validate(CFMetaData.java:994)
        at org.apache.cassandra.config.CFMetaData.fromThrift(CFMetaData.java:684)
        at org.apache.cassandra.thrift.CassandraServer.system_update_column_family(CassandraServer.java:
1045)
        ... 6 more

So that’s why it’s important to create column family as a “counter column family” from the start. Because of this, I intend to use these counter column families as what are essentially extra, metadata-only denormalized column families; ie, I won’t be mixing actual data in with these counters. Instead, the actual data will live elsewhere, and the counters will live apart, by themselves. Remember, in Cassandra, this sort of denormalization is considered not only acceptable, but required.

Okay, enough of that diversion. Let’s continue with using the CLI to make and use our counter column family.

I usually tell the CLI to use UTF-8, rather than raw hex bytes (I don’t know why it defaults to expecting you, a human, to be able to read and write raw bytes):

assume MyCounters keys as utf8;

Now you can use incr to increment your counters:

incr MyCounters['key-1']['thing-1'];
incr MyCounters['key-1']['thing-2'];
incr MyCounters['key-1']['thing-1'];
incr MyCounters['key-2']['thing-1'];
incr MyCounters['key-2']['thing-3'];
incr MyCounters['key-2']['thing-3'];
incr MyCounters['key-1']['thing-4'] by 5;

And you can pull out the whole thing with list:

[default@MyTest] list MyCounters;
Using default limit of 100
-------------------
RowKey: key-1
=> (counter=thing-1, value=2)
=> (counter=thing-2, value=1)
=> (counter=thing-4, value=5)
-------------------
RowKey: key-2
=> (counter=thing-1, value=1)
=> (counter=thing-3, value=2)

2 Rows Returned.
Elapsed time: 8 msec(s).

Or you can get the values for a single RowKey:

[default@MyTest] get MyCounters['key-1'];
=> (counter=thing-1, value=2)
=> (counter=thing-2, value=1)
=> (counter=thing-4, value=5)
Returned 3 results.
Elapsed time: 8 msec(s).

You can also get the value of a single counter within a column:

[default@MyTest] get MyCounters['key-1']['thing-1'];
=> (counter=thing-1, value=2)
Elapsed time: 8 msec(s).

As you can see, counter columns are not actually complicated or difficult to use. This got me around some of the initial issues I encountered, so hopefully it helps someone else.

One last thing

When using counters, there are some things to consider when it comes to your consistency level:

it’s important to understand that unlike normal columns, a write to a counter requires a read in the background to ensure that distributed counter values remain consistent across replicas. If you write at a consistency level of ONE, the implicit read will not impact write latency, hence, ONE is the most common consistency level to use with counters.

I haven’t addressed this yet, but I’ll keep it in mind.

Today, I want to demonstrate something that it makes very easy — and something that I wouldn’t know how to adequately test without it.

In our OCSP responder, we rely on a Redis database and can configure (at startup) whether we want to copy the “nonce”* from the request into the response.

* The idea of the “nonce” is that it increases the browser’s ability to trust a response from the server, because not only is the response signed, the signed response contains the timestamp that was given in the request. It makes it much, much more difficult for a replay attack to work against the service. This is mostly irrelevant to my discussion of Dependo, I just wanted to explain it (a little) so you’re not too confused.

But in our unit tests, we need to mock the Redis database (because we don’t want to have to actually run a database in order for our tests to pass) and re-configure that “copy nonce” behavior between tests.

First, before each test, we set up our Dependo::Registry with the values we’re going to want. We clear it out from the previous test, set the Logger to not log anything (we can change that if we want to see logging to help figure something out, but generally we don’t want to log anything during tests), mock Redis, set a default “copy nonce” setting (defaults to false here), and read our config file (I’d really rather not do it like this, we’ll get around to it later, I hope).

before :each do
    # clear the dependo before each test
    Dependo::Registry.clear
    Dependo::Registry[:log] = Logger.new(nil)

    # we always want to mock with a new redis
    @redis = double("redis")
    Dependo::Registry[:redis] = @redis

    # default value for :copy_nonce is false (can override on a per-test basis)
    Dependo::Registry[:copy_nonce] = false

    # read the config.yaml
    Dependo::Registry[:config_pool] = R509::Config::CaConfigPool.from_yaml("certificate_authorities", File.read("config.yaml"))
end

Now, we haven’t finished setting up each test. In our config.ru, we also add an R509::Ocsp::Signer object to the Dependo::Registry. The Ocsp::Signer is constructed from the other items in the Dependo::Registry that are configured, in production, in the config.ru file. So where did we put it?

def app
    # this is executed after the code in each test, so if we change something in the dependo registry, it'll show up here (we will set :copy_nonce in some tests)
    Dependo::Registry[:ocsp_signer] = R509::Ocsp::Signer.new(
        :configs => Dependo::Registry[:config_pool].all,
        :validity_checker => R509::Validity::Redis::Checker.new(Dependo::Registry[:redis]),
        :copy_nonce => Dependo::Registry[:copy_nonce]
    )
    R509::Ocsp::Responder
end

I define it in my spec file’s #app method because of the order of execution. That order is:

1. The code in “before :each”
2. The code in the individual test
3. The code in #app

Why is that relevant?

Because, in the case where we want to override the default value of Dependo::Registry[:copy_nonce], we need to be able to do that in the individual test code, and then read it in the #app method where the R509::Ocsp::Signer is built.

it "copies nonce when copy_nonce is true" do
    @redis.should_receive(:hgetall).with("cert:/C=US/ST=Illinois/L=Chicago/O=Ruby CA Project/CN=Test CA:872625873161273451176241581705670534707360122361").and_return({"status" => R509::Validity::VALID})

    # set to true for this test (this works because the app doesn't get set up until after this code)
    Dependo::Registry[:copy_nonce] = true

    get '/MHsweTBSMFAwTjAJBgUrDgMCGgUABBQ4ykaMB0SN9IGWx21tTHBRnmCnvQQUeXW7hDrLLN56Cb4xG0O8HCpNU1gCFQCY2eXAtMNzVS33fF0PHrUSjklF%2BaIjMCEwHwYJKwYBBQUHMAECBBIEEDTJniOQonxCRmmHAHCVstw%3D'
    request = OpenSSL::OCSP::Request.new(Base64.decode64("MHsweTBSMFAwTjAJBgUrDgMCGgUABBQ4ykaMB0SN9IGWx21tTHBRnmCnvQQUeXW7hDrLLN56Cb4xG0O8HCpNU1gCFQCY2eXAtMNzVS33fF0PHrUSjklF+aIjMCEwHwYJKwYBBQUHMAECBBIEEDTJniOQonxCRmmHAHCVstw="))
    ocsp_response = R509::Ocsp::Response.parse(last_response.body)
    request.check_nonce(ocsp_response.basic).should == R509::Ocsp::Request::Nonce::PRESENT_AND_EQUAL
end

Without Dependo (or a framework with similar capability), it’d be more difficult to test this behavior. Frankly, I don’t know if you could even do it, much less do it elegantly.

So the first time a tough spot like this came up, I found myself glad to have Dependo.

DI frameworks are unnecessary. In more rigid environments, they have value. In agile environments like Ruby, not so much. The patterns themselves may still be applicable, but beware of falling into the trap of thinking you need a special tool for everything. Ruby is Play-Doh, remember! Let’s keep it that way.

Others have apparently taken that to mean that dependency injection is simply unnecessary in Ruby, despite the fact that that’s explicitly not what Jamis Buck is saying here.

So, is there no room for DI in Ruby? There definitely is. I use DI nearly every day in Ruby, but I do not use a DI framework. Ruby itself has sufficient power to represent any day-to-day DI idioms you need.

He talks about the ways he injects his dependencies, using Ruby without any framework:

• Factory method that takes an optional class and instantiates an object for you
• A second factory method that you can override in a subclass for testing
• Pass in either classes or implementations in a constructor

But let’s say you’ve got the following problem:

• You’re writing a web app, using Sinatra (or anything else, really)
• You’re using an ORM library, like Sequel
  • Your models have methods on them
• You want some sort of logging
  • You want to define just one Logger object and use it in your Sinatra app and your Sequel models

Passing the Logger instance from the Sinatra app to the Sequel models either in a constructor or in each method is an ugly hack, and a non-starter. So what do you do?

Enter: Dependo!

Dependo lets you register your Logger object. I do this in my config.ru:

require "dependo"
Dependo::Registry[:log] = Logger.new(STDOUT)

Then, you can just include everything in the Registry as methods in your Sinatra app, and use them as if they’re instance methods:

class MyApp < Sinatra::Base
    include Dependo::Mixin

    get "/?" do
        log.info "I'm logging!"

        "Hello, world."
    end
end

You can also include the methods in your Sequel models:

class MyThing < Sequel::Model
    include Dependo::Mixin
    extend Dependo::Mixin

    def self.do_some_query
        log.info "I'm querying with a class method"
        self
    end

    def perform_some_other_action
        log.info "I'm calling an instance method"
    end
end

Note that in the model class, we used both include and extend. We do that so we can get the Dependo functionality in both instance methods (via include) and class methods (via extend).

Now, the Dependo::Registry doesn’t care what you put in it. You’ve already seen that we can put an object in it (like a Logger, as above). You can also put in a number,

Dependo::Registry[:my_important_number] = 7

or a string,

Dependo::Registry[:my_important_string] = "seven"

or even a function,

Dependo::Registry[:my_important_proc] = Proc.new { |x| x + 7 }

or a lambda,

Dependo::Registry[:my_important_lambda] = lambda { |x| x * 7 }

and you’ll use those like so:

class DemoClass
    include Dependo::Mixin

    def do_things
        puts my_important_number
        puts my_important_string
        puts my_important_proc.call(5)
        puts my_important_lambda.call(6)
    end
end

* I don’t know how to call a Proc or a lambda in this case without using #call on them. I’d much rather the syntax be something like “my_important_lambda(6)” … so, does anybody have any ideas? Thanks!

Do I think dependency injection is necessary, even in Ruby? I do. It helps with separation of concerns, so each of your classes can do only the things it cares about; it helps with unit testing, so you can mock your classes’ dependencies rather than test them all the way through (as you might in an integration test). You also don’t have to muck up your constructors or method definitions; your code’s API can stay the way you want it, and having to “include” and/or “extend” a mixin is a small price to pay (I think) for what you get.

I’m using Dependo with some success. Let me know if you do too — or, more importantly, if you find something about it that sucks, so I can try to fix it.

You’re accustomed to dealing with subject components by their shortname, things like “CN” or “O” or “C”, etc. But underneath the shortname is a different representation; the longname can be something like “1.3.6.1.4.1.311.60.2.1.3″ instead.

Now, you’re allowed to create an OpenSSL::X509::Name object with an unknown OID like this.

name = OpenSSL::X509::Name.new [["CN", "vikinghammer.com"], ["1.3.6.1.4.1.311.60.2.1.3", "US"]]

And if you now get the subject line, all is well.

name.to_s
#=> "/CN=vikinghammer.com/1.3.6.1.4.1.311.60.2.1.3=US"

But sometimes you want to get that array (like the one you passed into the constructor) back out and deal with it directly. We’re doing that in one of our projects, where we want to wrap OpenSSL::X509::Name in a friendlier interface. But what happens if you do it?

name.to_a
#=> [["CN", "vikinghammer.com", 12], ["UNDEF", "US", 12]]

UNDEF?! What help is that, you guys? I mean, after all …

OpenSSL::X509::Name.new name.to_a
#OpenSSL::X509::NameError: invalid field name
#    from (irb):8:in `initialize'
#    from (irb):8:in `each'
#    from (irb):8:in `initialize'
#    from (irb):8:in `new'
#    from (irb):8

Yeah, you can’t pass the output from OpenSSL::X509::Name#to_a back into the constructor; if you try, it’ll blow up.

But I kind of need to do that; when we parse a CSR or a certificate, it contains an OpenSSL::X509::Name, and we need to be able to read it. When we do that, we read the array out of #to_a, because parsing the subject line string out of #to_s is dangerously fragile (because the delimiters, “/” and “=”, can be used unescaped in the subject component values).

Enough complaining. What’s the solution?

We staged a two-pronged attack on this problem. The first step was to patch Ruby, to fix the bug in its OpenSSL bindings. My teammate Paul Kehrer did this.

short_name = OBJ_nid2sn(OBJ_ln2nid(long_name));
if (strcmp(short_name,"UNDEF") == 0) {
    return_name = &long_name;
} else {
    return_name = short_name;
}

It’s a fairly simple fix (once you’ve, you know, gotten deep enough into Ruby’s C source). It used to attempt to translate the longname into a shortname, and if no translation was found, it’d just leave the shortname as UNDEF. Instead, he checks for UNDEF and if that’s the shortname, he resets it to be the (original) longname. That way, the OID is preserved in the OpenSSL::X509::Name object. Victory!

But … that’d mean our project would only work on a custom patch of Ruby 1.9.3, and we were hoping to maintain compatibility across both 1.8 and 1.9, and we certainly can’t rely on people to use Paul’s patch rather than the actual release of Ruby. And even if (hopefully “when”) Ruby accepts the patch, that’d mean we’d have to require Ruby (at least) 1.9.4, which is pretty close to unacceptable. We need an interim solution — where “interim” is taken to mean “useful for at least a few years, potentially indefinitely”.

That’s where the second prong of our attack comes in. Sanitizing the OpenSSL::X509::Name#to_a right in Ruby!

# Sanitize an X509::Name. The #to_a method replaces unknown OIDs with "UNDEF", but the #to_s
# method doesn't. What we want to do is build the array that would have been produced by #to_a
# if it didn't throw away the OID.
class NameSanitizer
    # @option name [OpenSSL::X509::Name]
    # @return an array of the form [["OID", "VALUE], ["OID", "VALUE"]] with "UNDEF" replaced by the actual OID
    def sanitize(name)
        line = name.to_s
        array = name.to_a.dup
        used_oids = []
        undefined_components(array).each do |component|
            begin
                # get the OID from the subject line that has this value
                oids = line.scan(/\/([\d\.]+)=#{component[:value]}/).flatten
                if oids.size == 1
                    oid = oids.first
                else
                    oid = oids.select{ |match| not used_oids.include?(match) }.first
                end
                # replace the "UNDEF" OID name in the array at the index the UNDEF was found
                array[component[:index]][0] = oid
                # remove the first occurrence of this in the subject line (so we can handle the same oid/value pair multiple times)
                line = line.sub("/#{oid}=#{component[:value]}", "")
                # we record which OIDs we've used in case two different unknown OIDs have the same value
                used_oids << oid
            rescue
                # I don't expect this to happen, but if it does we'll just not replace UNDEF and continue
            end
        end
        array
    end

    private

    # get the components from #to_a that are UNDEF
    # @option array [Array<OpenSSL::X509::Name>]
    # @return [{ :index => the index in the original array where we found an UNDEF, :value => the subject component value }]
    def undefined_components(array)
        components = []
        array.each_index do |index|
            components << { :index => index, :value => array[index][1] } if array[index][0] == "UNDEF"
        end
        components
    end
end

This solution, as you may have noticed, is a little bit more complicated. I’ll try to explain.

First, we take advantage of the fact that OpenSSL::X509::Name#to_s contains all the data we need, despite the fact that #to_a throws it away. So we loop over #to_a to find all the subject components whose name is UNDEF and record their index in the #to_a array, as well as their value (we don’t need to record their names, because we know they’re always UNDEF).

Then, we loop over all those UNDEF subject components, and we determine what the original OID was from the subject line string from #to_s. For that, we take advantage of the fact that these unknown OIDs will be of the form 1.2.3.4.5, or something like that; ie, our regex looks for anything with digits and periods in the place of the subject component name, with the same value as the undefined subject component.

Note that we also rely on the fact that #to_s and #to_a maintain the order of subject components, so as we loop over the unknown OIDs, the current one will always correspond to the first match from the subject line.

After updating the subject component name in the array, we remove the subject component from the subject line (but only the first occurrence of it), and we record that we’ve used that OID. Both of those steps are to ensure that if the same OID occurs multiple times, or if two different OIDs have the same value, that we handle that properly and correctly maintain the order of the components.

And now …

sanitizer = NameSanitizer.new
sanitizer.sanitize(name)
#=> [["CN", "vikinghammer.com", 12], ["1.3.6.1.4.1.311.60.2.1.3", "US", 12]]

OpenSSL::X509::Name.new(name.to_a)
#OpenSSL::X509::NameError: invalid field name

OpenSSL::X509::Name.new(sanitizer.sanitize(name))
#=> /CN=vikinghammer.com/1.3.6.1.4.1.311.60.2.1.3=US

It works! As long as we pass the original name to the sanitizer, we can get the original OIDs in the array structure we need (the same one returned from #to_a).

This will work across Ruby 1.8 and 1.9, and if Ruby accepts Paul’s patch, it’ll also work in 1.9.4; if the bug is fixed, NameSanitizer#undefined_components will not find any UNDEF subject components, and none of our complex code will ever be touched.

I think this exercise demonstrates a few things:

• It’s considerably simpler to fix this kind of thing at a lower level, if possible.
• Even if a bug in your language gets fixed, you should still try to figure out a backwards-compatible fix.
• Use open source languages! It’s awesome to have the option of fixing Ruby and submitting a patch.
• If you’re working on an open source project, you can write posts like this about it.

This turned a horrifyingly annoying Tuesday evening (when Paul discovered the bug) into a pretty fun Wednesday morning (when we fixed it).

The concept of this class is that it looks up an SSL certificate serial number in a Redis database, and returns some status information. Those details are mostly irrelevant to the question of mocking, but I thought I’d get it out of the way.

module R509::Validity::Redis
    class Checker < R509::Validity::Checker
        def initialize(redis)
            raise ArgumentError.new("Redis must be provided") if redis.nil?
            @redis = redis
        end

        def check(serial)
            raise ArgumentError.new("Serial must be provided") if serial.nil? or serial.to_s.empty?

            hash = @redis.hgetall("cert:#{serial}")
            if not hash.nil? and hash.has_key?("status")
                R509::Validity::Status.new(
                    :status => hash["status"].to_i,
                    :revocation_time => hash["revocation_time"].to_i || nil,
                    :revocation_reason => hash["revocation_reason"].to_i || 0
                )
            else
                R509::Validity::Status.new(:status => R509::Validity::UNKNOWN)
            end
        end
    end
end

If you just write tests for this, and your constructor looks like:

R509::Validity::Redis::Checker.new(Redis.new)

then you’re going to have to actually have a Redis database running for your tests, and you need to have the correct data in it in order for the tests to pass. Obviously, that sucks. You want to avoid that kind of environmental dependency in your unit tests.

Enter mocking. Here’s a pair of rspec tests that makes sure it returns an UNKNOWN status if nothing is found:

it "gets unknown when serial is not found (returns {})" do
    redis = double("redis")
    checker = R509::Validity::Redis::Checker.new(redis)
    redis.should_receive(:hgetall).with("cert:123").and_return({})
    status = checker.check(123)
    status.status.should == R509::Validity::UNKNOWN
end
it "gets unknown when serial is not found (returns nil)" do
    redis = double("redis")
    checker = R509::Validity::Redis::Checker.new(redis)
    redis.should_receive(:hgetall).with("cert:123").and_return(nil)
    status = checker.check(123)
    status.status.should == R509::Validity::UNKNOWN
end

(I have two tests here because I’ve seen the Redis driver return {} if a Hash isn’t found, but I want to make sure my code still works if it returns nil.)

The first key line is:

redis = double("redis")

It’s called “double” because you’re creating a “test double” object instead of an actual connection to the Redis database. The test double doesn’t actually do anything, but if it receives a method call that you didn’t tell it to expect (or if it doesn’t receive a method call that you did tell it to expect), it’ll give you a failing test.

The next key line is where you tell your test double what to expect (and what it should do):

redis.should_receive(:hgetall).with("cert:123").and_return({})

Let’s break this down. You’re telling your test double that it “should_receive” a call to the “hgetall” method (you use the symbol :hgetall for this), “with” the single parameter “cert:123″, and it should return an empty hash object {}.

The test double verifies that our Checker#check implementation does actually make this call (and only this call) to the Redis object, and it helpfully returns {}, which then exercises the “nothing found in the database” code path. Thus, we can then check that the returned status should equal R509::Validity::UNKNOWN (which would only happen if we reacted properly to the data we told the test double to return).

And what about testing something else, for example that the serial number does exist in the database? Here’s the spec for a certificate that’s been revoked:

it "gets revoked with revocation time and reason" do
    redis = double("redis")
    checker = R509::Validity::Redis::Checker.new(redis)
    redis.should_receive(:hgetall).with("cert:123").and_return({"status" => "1", "revocation_time" => "789", "revocation_reason" => "5" })
    status = checker.check(123)
    status.status.should == R509::Validity::REVOKED
    status.revocation_time.should == 789
    status.revocation_reason.should == 5
end

Here, we say that our test double should receive the “hgetall” method with the single parameter “cert:123″, and it’ll return a populated hash with some certificate information (its status, revocation time, and revocation reason). We then verify that the status should be REVOKED, and that the revocationtime and revocationreason are correctly translated to integers.

Now, so far we’ve only seen expectations where the method should receive only one parameter. Expecting multiple parameters is exactly what you’d expect: just pass multiple parameters to should_receive. Here’s an example of a spec for the R509::Validity::Redis::Writer class, where our test double will expect to receive multiple parameters:

it "when reason isn't provided" do
    redis = double("redis")
    writer = R509::Validity::Redis::Writer.new(redis)
    redis.should_receive(:hmset).with("cert:123", "status", 1, "revocation_time", Time.now.to_i, "revocation_reason", 0)
    writer.revoke(123)
end

Here, our test double should receive the “hmset” method, with a slew of parameters (obviously, it maintains the type and order of the parameters, in addition to their values). Note also that we don’t specify any “and_return” here, since we don’t care about the return value from this method call.

Hopefully that helps to explain rspec mocks, in the interest of getting you to make them yourself. It beats the hell out of having a database running for your unit tests to work.

One of the big features, in my opinion, that my MLB Scoreboard app has over MLB’s official apps (or any other third party scoreboard apps that I know of) is that it converts the given Eastern times into your local time. I’ve just started using that same code in my new NFL Scoreboard app (which I’ve decided to make open source, for some reason).

I want to be able to use this essentially like so:

LocalTimeCalculator calculator = new LocalTimeCalculator("12:30", "PM");
// if I'm in Central time (which I am), this will yield 11:30 AM
Date localTime = calculator.getLocalTime().getTime();

I want it to return a Calendar instead of a Date so I can easily set the date (year/month/day) if I want to; I do that sometimes, but I didn’t feel that it should be part of the local time calculation.

So here’s my LocalTimeCalculator:

public class LocalTimeCalculator {

    private String mTime;
    private String mAmpm;

    public LocalTimeCalculator(String time, String ampm) {
        mTime = time;
        mAmpm = ampm;
    }

    public Calendar getLocalTime() {
        String[] timeArray = mTime.split(":");
        int hour = Integer.parseInt(timeArray[0]);
        int minute = Integer.parseInt(timeArray[1]);

        hour = convertToHourOfDay(hour, mAmpm);

        Calendar eastern = Calendar.getInstance(TimeZone.getTimeZone("America/New_York"));
        eastern.set(Calendar.HOUR_OF_DAY, hour);
        eastern.set(Calendar.MINUTE, minute);

        Calendar local = Calendar.getInstance();
        local.setTimeInMillis(eastern.getTimeInMillis());

        return local;
    }

    private int convertToHourOfDay(int hour, String ampm) {
        if ("AM".equalsIgnoreCase(ampm)) {
            if (hour == 12) {
                return 0;
            } else {
                return hour;
            }
        } else {
            if (hour == 12) {
                return 12;
            } else {
                return hour + 12;
            }
        }
    }
}

First, I split the “12:30″ time into hour and minute, and then convert the hour to 24-hour time based on the AM/PM field. Then I get a Calendar instance set to Eastern time (“America/New_York”), and set its time. I get a new Calendar instance, which will default to local time,* and set the time on it based on the original Eastern Calendar.

* On my Android phone, this updated as soon as I changed to a different timezone. It was working fine for me for the first few months of the MLB season before I finally went somewhere else (my trip to Las Vegas for DEFCON) and got to test that it was working in a timezone other than Central. Exactly what I was hoping for.

I don’t know if local time conversion was a big reason anybody wanted to use MLB Scoreboard or NFL Scoreboard rather than the official apps, but I do think it’s the kind of thing all apps should do. Your phone knows what timezone it’s in; you shouldn’t have to keep translating times in your head.

It’s not complicated, and it makes things better for your users. If you’re making Android apps, I think you should be doing this.

The Obvious

I figured that when I’ve downloaded the new list of stuff I want to show, that I could just create a new adapter and stuff it into the list.

EventLogAdapter eventLogAdapter = new EventLogAdapter(mContext, events);
mEventListView.setAdapter(eventLogAdapter);

This works, in that the list gets updated. But it always scrolls all the way back up to the top. Sometimes that might be what you want, but most of the time you’ll want to maintain your scroll position.

The Naive

I tried getting pixel-level scroll position using getScrollY(), but it always returned 0. I don’t know what it’s supposed to do. I ended up going with a solution that got close to maintaining your scroll position.

int firstPosition = mEventListView.getFirstVisiblePosition();
EventLogAdapter eventLogAdapter = new EventLogAdapter(mContext, events);
mEventListView.setAdapter(eventLogAdapter);
mEventListView.setSelection(firstPosition);

This figures out the first item in the list you can see before resetting the adapter, and then scrolls you to it. This maintains your scroll position within some unknown/arbitrary range, and can cause you to jump around in the list a little bit when you refresh.

If you’re scrolled halfway through a list item, it’ll snap you to the top of it so it’s completely visible. Unfortunately, if you’re scrolled halfway through a list item, that probably wasn’t the one you were paying the closest attention to.

The Elegant

There had to be a better way!

And, of course, there is. Romain Guy, an Android developer who haunts Stack Overflow and Google Groups dropping golden hints when people ask questions, pointed out:

The problem is that you are creating a new adapter every time you reload the data. That’s not how you should use ListView and its adapter. Instead of setting a new adapter (which causes ListView to reset its state), simply update the content of the adapter already set on the ListView. And the selection/scroll position will be saved for you.

There are two problems with Romain Guy: 1) he doesn’t have a central repository of these hints/answers so I can learn what to do before doing everything wrong first, and 2) they really are just hints, in that they point you in the right direction without getting you all the way there.

In this case, yes, updating the adapter without creating a new one every time will maintain your scroll position. Except that you’ll frequently get an “IllegalStateException: The content of the adapter has changed but ListView did not receive a notification. Make sure the content of your adapter is not modified from a background thread, but only from the UI thread.”

It turns out that you need to call notifyDataSetChanged() on your adapter after changing its contents; fortunately, that “only from the UI thread” bit was a red herring, because I didn’t really want to do any processing that could/should be asynchronous on the UI thread.

I added a refill() method to my adapters:

public void refill(List<EventLog> events) {
    mEvents.clear();
    mEvents.addAll(events);
    notifyDataSetChanged();
}

And I call it when my download is complete:

if (mEventListView.getAdapter() == null) {
    EventLogAdapter eventLogAdapter = new EventLogAdapter(mContext, events);
    mEventListView.setAdapter(eventLogAdapter);
} else {
    ((EventLogAdapter)mEventListView.getAdapter()).refill(events);
}

If the list doesn’t already have an adapter, then I create one. But if it does have an adapter, then I just refill it. And it maintains my scroll position exactly!

PersistenceManager pm = pmFactory.getPersistenceManager();
try {
    // do something with the pm here
} finally {
    pm.close();
}

So that’s five lines of crap, in every method. But the only thing that actually matters, that I actually want to write in my DAO methods, is the meaty part, the part that says “do something” in that comment there. So, I wrote a little library to let me do just that: it’s my vh-dao project at Github.

It starts with a DAO interface, which says that every DAO class has a store method; I’ve been on too many projects where sometimes that method is called “store” and sometimes it’s “save” and sometimes it’s “storeWidget” or “saveWidget” (where “Widget” is the name of the model class). Enough.

public interface VHDao<T> {
    public void store(T model);
}

Then, I have an abstract base class that implements a few things that’ll be the same for every DAO; including that store method, so when you create a new DAO you never have to worry about that.

public abstract class BaseVHDao<T> implements VHDao<T> {

    @Autowired
    protected PersistenceManagerFactory pmFactory;

    @Override
    public void store(T model) {
        PersistenceManager pm = pmFactory.getPersistenceManager();
        try {
            pm.makePersistent(model);
        } finally {
            pm.close();
        }
    }

    protected List<T> list(VHQuery vhQuery, Object... args) {
        PersistenceManager pm = pmFactory.getPersistenceManager();
        try {
            Query query = pm.newQuery(vhQuery.getClazz());
            if (vhQuery.hasFilter()) {
                query.setFilter(vhQuery.getFilter());
            }
            if (vhQuery.hasOrdering()) {
                query.setOrdering(vhQuery.getOrdering());
            }
            if (vhQuery.hasRange()) {
                query.setRange(vhQuery.getRangeStart(), vhQuery.getRangeEnd());
            }
            List<T> list = (List<T>)query.executeWithArray(args);
            list.size();
            return list;
        } finally {
            pm.close();
        }
    }

    protected T first(VHQuery vhQuery, Object... args) {
        PersistenceManager pm = pmFactory.getPersistenceManager();
        try {
            Query query = pm.newQuery(vhQuery.getClazz());
            if (vhQuery.hasFilter()) {
                query.setFilter(vhQuery.getFilter());
            }
            if (vhQuery.hasOrdering()) {
                query.setOrdering(vhQuery.getOrdering());
            }
            query.setRange(0, 1);
            List<T> list = (List<T>)query.executeWithArray(args);
            if (list.size() > 0) {
                return list.get(0);
            } else {
                return null;
            }
        } finally {
            pm.close();
        }
    }

}

As you can see, there are two methods that we can use in the subclass: list() and first(). They get a PersistenceManager, set up a Query based on the VHQuery you pass in, execute the Query based on all the (optional) parameters you give, and then close the PersistenceManager.

That VHQuery object is there to define the filter/ordering/range that I need to enter into the actual JDO Query object; its reason for being is twofold:

• The JDO Query object is created by calling the PersistenceManager, which I won’t have access to in the DAO subclass since I want to instantiate it in the base methods
• Prevent a leaky abstraction of requiring the subclass to know about the JDO Query object

I originally had a method that built and returned the Query object that I could then fill in and execute, but I couldn’t close the PersistenceManager after executing it, which would have led to annoying memory leaks. This seems nicer.

Here’s what a method looks like that just wants to grab a single item from the database:

public Greeting getLatestByAuthor(User author) {
    VHQuery query = new VHQuery(Greeting.class);
    query.setFilter("author == :author");
    query.setOrdering("date desc");
    return first(query, author);
}

It just sets up the VHQuery and asks the base class to give it the first result that matches the given query parameters.

And here’s one that returns a list:

public List<Greeting> mostRecent(int count) {
    VHQuery query = new VHQuery(Greeting.class);
    query.setOrdering("date desc");
    query.setRange(0, count);
    return list(query);
}

As you can see, your DAO methods now have no extraneous lines that don’t have anything to do with the query you’re trying to execute. That’s all handled generically for you, and you don’t have to worry about it.

Note: I’ve written this so I can use it with the Appengine datastore, and haven’t tried it with an SQL database. JDO is supposed to work just fine with an RDBMS, so this will probably help. I guess I’ll find out when I get around to that.

I’m guessing there was already something that does this, because I can’t imagine that everyone using JDO is okay with typing all that boilerplate in every method; I couldn’t find it, though, and am open to someone pointing me in the right direction. I’m also sure there will be more additions to this DAO abstraction as I find needs — what do you think I’ve missed?

I’m using Ruby here, along with the awesome Sequel ORM library.

My idea is that whenever a record is deleted from our database, I want to create an identical record in a “slum” database. These slums won’t be used for anything, hopefully, but if we need to use them they’ll have everything — including the id’s — that we deleted.

At first, my models each had methods that supported this … and it got pretty cumbersome. Then I remembered, “hey wait a minute, this is Ruby, and I can probably do something awesome!”

Indeed, you can do awesome things.

First, I created a “SlumBuilder” module that adds a class method called “build” to all the classes that extend it. In Ruby, when you extend a module its classes are included as class methods, and when you include a module its classes are included as instance methods. That’s an important distinction, because we’ll be doing both.

Here’s SlumBuilder:

module SlumBuilder
    def build(original)
        object = self.new
        original.keys.each do |field|
            object[field] = original[field]
        end
        object.save
    end
end

As you can see, it instantiates a new object of whatever type has extended SlumBuilder, and fills it with all the fields (keys) in the given model object (original).

A Slum model object might look like this:

class SlumCoolThing < Sequel::Model(:cool_thing)
    extend SlumBuilder
    self.db = $slum_db
end

That’s it. It gets the available fields from the database (thank you Sequel!) and all its functionality from SlumBuilder. We just need to set the database it uses, since we’re using two different databases.

The regular model object, the one we’ll need to delete some records out of, would look like this:

class CoolThing < Sequel::Model(:cool_thing)
    include ModelDestroyer
    self.db = $regular_db
end

Simple enough. As you can see, I went with the convention that the slum’s model class is the same as the regular model’s class with “Slum” prepended. This is important, as you’ll see here in ModelDestroyer:

module ModelDestroyer
    def slum_class
        Object::const_get("Slum#{self.class}")
    end

    def destroy
        begin
            self.slum_class.build(self)
        rescue
            # failed to build a slum, probably because the class doesn't exist, but we'll just continue on with the deletion
            puts "Failed to build slum: #{self.inspect}"
        end
        self.delete
    end
end

This is where most of the magic happens. The “slum_class” method uses Ruby’s cool “get the constant, like a class, that has the given name” functionality to get the class that has the same name as whichever one has included our ModelDestroyer with “Slum” prepended.

Our “destroy” method gets that Slum class and calls its “build” class method, discussed above, that copies all the fields and saves a new record to the slum database. It then calls Sequel’s “delete” method to remove the record from our database.

So if we had a variable, cool_thing, that we pulled out of the CoolThing database and we wanted to delete it, we now have two options:

cool_thing.delete

Which just removes cool_thing from the database. The other option is:

cool_thing.destroy

Which removes cool_thing from the database AND inserts a perfect copy of it into a separate slum database.

You’re probably not going to need to do exactly this, with inserting duplicate copies of a row upon deletion. But being able to inject functionality like SlumBuilder and ModelDestroyer into a class is really cool — we couldn’t use inheritance for this, because our model classes already inherit from Sequel::Model.

It’s bending my mind right now to think about how I’d do this in Java. I think it might be a bit more verbose … if it’s even possible.