set_trace_func

Installing Wifi without Internet

2009-06-21T19:37:00-04:00

I picked up a new machine this week to turn into a home media server, and had a small hoop to jump through with the Broadcom wifi card I picked up. Problem being, office and all my monitors were upstairs, wifi was downstairs, and it's difficult to install new drivers without net access.

But, Linux Mint popped up a "new hardware" dialog, saying it wanted to use b43-fwripper to set up the drivers. Good news, it gave me the url it failed to download from. Manually download the deb, open it up on my other machine using Archive Manager (rather than GDebi), and navigate through to the postinstall script, at data.tar.gz/./usr/share/b43-fwcutter/, and I find a reference to two other files.

After downloading those, copy all three files to the new box. Copy the .deb to /var/cache/apt/archives, and then apt-get install b43-fwripper. Ignore the failure, then go to the dir with the other two files, and manually run the 3 useful lines from the install script (don't forget to add sudo to the last one):

b43-fwcutter -w /lib/firmware wl_apsta-3.130.20.0.o
tar xfvj broadcom-wl-4.150.10.5.tar.bz2
b43-fwcutter --unsupported -w /lib/firmware broadcom-wl-4.150.10.5/driver/wl_apsta_mimo.o

Reboot, and voila - local wifi networks show up in the tray applet. I presume these steps will work for an Ubuntu install as well, since Mint is based on Ubuntu, but I haven't tried it as yet - in fact, the required .deb is on the Ubuntu 9.04 livecd. I also don't know if these files will work for other Broadcom wireless adapters, but if your modern linux system can do a hardware detect and tell you what driver it's trying to pull down, that should be a good starting point.

Fib for Fun

2009-02-21T22:31:41-05:00

A little math can be a dangerous thing. Here's a quick benchmark for Fibonacci numbers (Y axis in seconds, 20,000 iterations):

The line on the left, marked A, is the typical naive recursive solution, based as directly as possible on the mathematical definition of the function:

def fib_r(n)
  return 1 if n <= 2
  fib_r(n-1) + fib_r(n-2)
end

Note the telltale exponential curve - this is what we call Very Bad. The line there stops at Fib(10).

A little jiggery lets us convert the recursive solution into an iterative one, line B:

def fib_i(n)
  x = y = 1
  (n-1).times do
    x, y = y, x+y
  end
  x
end

My personal favorite Fib function is (ab)using a hash's default value function to act as a memoizer. Line C is the fastest of the bunch (constant time of ~0.012s) but does have some extra memory overhead.

def fib_h(n)
  @h ||= Hash.new{ |h,k|
    if k < 2
      h[k] = k
    else
      h[k] = h[k-1] + h[k-2]
    end
  }
  @h[n]
end

If we were in an embedded system or something, the memory overhead would probably be bad, but fear not! Math can save us!

Line D is also constant time, but about 0.037s. It's a fun little bit of math mentioned in SICP, and turns Fibonacci numbers into a very simple (if unintuitive) bit of math:

SQRT5 = Math.sqrt(5.0)
PHI = (1 + SQRT5) / 2
PSI = (1 - SQRT5) / 2

def fib_m(n)
  ((PHI ** n - PSI ** n) / SQRT5).to_i
end

geoip_city on Leopard

2009-02-20T12:27:50-05:00

I've been trying to figure out a minimal way of getting the geoip_city gem installed on my mac, and having seen a few ideas on google, I'm gonna consolidate into a minimal solution. I originally wanted to work with the macports version of libgeoip, but had absolutely no luck with that, so here's something a bit more manual:

# we need to be root for the ARCHFLAGS later to stick
sudo su

mkdir -p /usr/local/src
cd /usr/local/src

# Download and install latest geoip
curl -O http://geolite.maxmind.com/download/geoip/api/c/GeoIP-1.4.5.tar.gz
tar -xzf GeoIP-1.4.5.tar.gz
cd GeoIP-1.4.5
./configure
make
make check
make install

# install the gem
export ARCHFLAGS='-arch i386'
gem install geoip_city

The main sticking point for me was that setting ARCHFLAGS doesn't continue through to a sudo gem install, so we need to do the whole thing as root.

But, now that we've done that, we can grab the GeoLiteCity database and point our app at it:

curl -O http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz

Going Functional

2009-01-17T10:00:00-05:00

So, some Hacker News users recently decided that it'd be a good idea to work through the SICP together and make a kind of book club out of the deal over irc.

When I heard of it, I thought it'd be a good excuse to work through the text myself (since it was just sitting on my bookshelf), and also stretch my brain a bit more teaching myself Clojure. I'm almost done the reading and exercises for section 1.1, and I'm already finding it a valuable exercise.

The third exercise asks us to "define a procedure that takes three numbers as arguments and returns the sum of the squares of the two larger numbers." At either extremity are two solutions. The straightforward solution a new programmer can generate based solely on the first 20 pages of the text looks something like this:

(defn sum-largest-squares [a b c]
  (cond (and (< a b) (< a c)) (+ (* b b) (* c c))
        (and (< b a) (< b c)) (+ (* a a) (* c c))
        (and (< c a) (< c b)) (+ (* a a) (* b b))))

Some tests to prove correctness, returning 2^2 + 3^2, or 13, are:

(sum-largest-squares 1 2 3)
(sum-largest-squares 2 3 1)
(sum-largest-squares 3 1 2)

An interesting (and enlightening) exercise is to take our naive function, and just start randomly refactoring, and see where it takes us. One step at a time, making sure we are always passing our handful of test cases. The goal is to decompose and abstract the function into more manageable pieces, with less repetition. To start:

(defn sum-largest-squares [a b c]
  (cond (and (< a b) (< a c)) (+ (square b) (square c))
        (and (< b a) (< b c)) (+ (square a) (square c))
        (and (< c a) (< c b)) (+ (square a) (square b))))

The square function is trivially pulling out (* x x), and conveniently is already present in Clojure.

Second step is to pull out the three conditional clauses into sum-of-squares:

(defn sum-of-squares [a b]
  (+ (square a) (square b)))

(defn sum-largest-squares [a b c]
  (cond (and (< a b) (< a c)) (sum-of-squares b c)
        (and (< b a) (< b c)) (sum-of-squares a c)
        (and (< c a) (< c b)) (sum-of-squares a b)))

Next, we can probably abstract the conditionals themselves, since what we're looking for is the smallest of three elements (min is also provided by clojure):

(defn sum-largest-squares [a b c]
  (cond (= a (min a b c)) (sum-of-squares b c)
        (= b (min a b c)) (sum-of-squares a c)
        (= c (min a b c)) (sum-of-squares a b)))

Now, we've got a much more parallel structure here, so we should be able to collapse the conditional, and process the values first.

I'll stop here, but there are a few different ways to get started with the next step - most involve transitioning our three arguments into a list at some point. I've found it instructive to play around with the different ways of decomposing the problem (including a few side steps along the way to the route I've followed above). I'm thinking that as I continue with the SICP exercies, I'll try and make a habit of not stopping with the first solution that springs out of my text editor, but to try and explore the structure of the code through refactoring.

Also, while I never managed to figure out refactoring steps that would produce it, I think the most straightforward functional solution (rather than the semi-iterative solution we started with above) winds up being this - it has a certain elegance to it, no?

(defn sum-largest-squares [a b c]
  (apply + (map square (rest (sort (list a b c))))))

Gem Cleanup

2008-11-25T19:00:00-05:00

The latest version of rubygems seems to be more gung-ho about trying to clean up old versions of gems. This is fine, except that OSX seems to be very protective of its bundled gem directory, refusing to uninstall gems located there even though I'm running the gem command via sudo.

As a result, while gem list tells me that I've got activerecord v 2.2.2 and 1.15.6 installed, gem cleanup fails miserably:

jamie@juliet ~> sudo gem cleanup
Cleaning up installed gems...
Attempting to uninstall activerecord-1.15.6
ERROR:  While executing gem ... (Gem::InstallError)
    Unknown gem activerecord = 1.15.6

I appreciate Leopard not wanting to break the bundled versions of gems, but having a functioning gem cleanup is helpful to me. Turns out it's pretty easy to just prevent rubygems from checking the Ruby.framework gem path, just create (or edit) ~/.gemrc and include the following:

gempath:
  - /Library/Ruby/Gems/1.8
  - /Users/jamie/.gem/ruby/1.8

Of course, you'll need to use your own username on the last line there, and you might want to double-check the existing GEM PATH values from gem env output so you don't accidentally clobber anything else. Also, if you've got a previously-generated .gemrc, the gempath key needs to be a string, not a symbol, or else it won't get picked up. Whee.

Async-Observer with RabbitMQ

2008-08-08T16:00:00-04:00

Update: This code is stale, I've extracted a gem of it and posted on github.

Async-observer is great. Fast, easy to use API, and it Just Works. The downside is that the backend, Beanstalkd, doesn't support persistent messages in case the server crashes. I hear it's on the roadmap, though.

However, there's another messaging backend, RabbitMQ, that seems just as easy to get set up, and does support persistent messages. So, how to get these two bits of tech working together? Well, if you're hosting your app on Thin (or another app server that runs in EventMachine), it's pretty straightforward.

First, install the amqp ruby library to connect to rabbit, and then add a tiny bit of setup.

In config/environment.rb:

require 'mq'
class BeanstalkPoolImpersonator
  def initialize(opts={})
    @opts = opts
  end

  def connect
    connection = AMQP.connect(@opts)
    @channel = channel = MQ.new(connection)
  end

  def use(queue)
    @queue = MQ::Queue.new(@channel, queue)
  end

  def yput(obj, pri, delay, ttr)
    p [obj, pri, delay, ttr]
    @queue.publish(YAML.dump(obj))
  end

  def last_server
    :last_server_stub
  end

  def subscribe(*args, &blk)
    @queue.subscribe(*args, &blk)
  end
end

Then, instead of connecting via Beanstalk::Pool.new, do this:

AsyncObserver::Queue.queue = BeanstalkPoolImpersonator.new()

You can pass an options hash to the new call, providing user, pass, vhost, host, or port as necessary.

Then, in your workers, load up the async_observer worker class, and extend like so:

class RabbitWorker < AsyncObserver::Worker
  def run()
    EM.run do
      AsyncObserver::Queue.queue.connect
      AsyncObserver::Queue.queue.use('1.0')
      AsyncObserver::Queue.queue.subscribe do |headers, msg|
        job = OpenStruct.new(:ybody => YAML.load(msg), :body => msg, :stats => [])
        job.id = headers.properties[:delivery_tag]
        safe_dispatch(job)
      end
    end
  end
end

Create the new worker the same way you would for the AO::Worker, and you're set:

RabbitWorker.new(binding).run()

Note: I'm maintaining a merb port of async-observer on github.

Note 2: This worker is somewhat fragile, if the RabbitMQ server goes down it will just hang forever waiting for more jobs. I'll need to figure out a solution to that before we move this into production (and I wrap it up in a gem), but I thought I'd get this out and about now.

Specing Layouts

2008-07-16T16:37:00-04:00

Just a quick little snippet for those trying to write specs for their layout.

Create your spec in spec/views/layout/application.html.erb_spec.html, and add this class to it:

class Layout < Application
  layout nil
  def application; render; end
end

Then just test it as you would any other ordinary view.

Porting Rails Plugins to DataMapper

2008-07-15T15:59:00-04:00

As a follow-up to my previous post, here's some gotchas to be aware of if you're looking to support both ActiveRecord and DataMapper in a Merb (and/or Rails) plugin.

The Strategy

The best way I've found to handle multiple ORM support in your plugin is not to start monkeypatching around to make one ORM handle like another. I've done it, and can tell you that wrapping one ORM's backend into another is ugly.

The better way is to localize the points where your plugin interacts with the data model, with an eye to swapping them out. For the above example, I would be better off taking the method that used the reflection method and putting it inside an ActiveRecord-specific module. Then, create a DataMapper-specific module that defines the same method, but instead relies on the DM backend to get at the association information. Finally, when the plugin was loaded I could just include one of the modules based on which ORM was loaded into the runtime.

Basic Translation

Now that we have a plan, we can start translating our extracted functions from AR bits to DM bits. There's a bunch of fairly straightforward transformations we can make.

It's unfortunate that there isn't more unity between the two, as from a library-developer's perspective it would make this sort of thing much easier, but the DM team is pretty vocal about wanting the best API they can get, and not worrying about being hobbled by how AR does things. I don't particularly disagree.

ActiveRecord	DataMapper
.find(:all, ...)	.all(...)
.find(:first, ...)	.first(...)
.find(id)	.get(...)
.find_all_by_id(id)	.all(:id => ids)
.table_name	.storage_name
.primary_key	.key.first.name
.connection	repository.adapter

Raw SQL

If you're running raw SQL queries, firstly, I'm sorry. Secondly, you want to run .query instead of .execute. Thirdly, if you care about getting the results of the query back, AR returns an array of arrays, DM returns an array of hashlike objects, so you want to map them for their values array. The hashlike object in question is order-preserving, so you'll get things out in the right order. If you're concerned, grab one of the result objects and verify that the keys array is in the correct order.

Hooks

ActiveRecord defines a few hook points, along the lines of before_create and after_save. DataMapper uses (a modified version of) the Extlib gem, allowing it to hook pretty much any method. The syntax is like before(:create) and after(:save). AR's hooks pass in the object to work with, DM's have the object available as self.

In before hooks, the AR hook chain stops if your method returns false, in DM you must throw :halt.

Things You Shouldn't Be Doing Anyways

If you're manually setting @attribute values in your AR code, you'll need to use instance_variable_set for DataMapper. I recommend writing manual accessor methods to wrap it for abstraction.

If you're wanting some arbitrary data structures back, I recommend using OpenStruct (require 'ostruct') to pass structured data back and forth. This was especially handy when I wanted some results from DM to look like AR, because I was just doing an adapter (bad me!) and the client code wanted to interact with the AR object. Just be aware that OpenStruct doesn't quite clear out all its methods, so you might want to define some custom readers anyway. I had problems with type in particular, which is a deprecated alias of class - redefining the method to return @table[:type] fixed that up nicely.

Porting Rails Plugins to Merb

2008-07-15T00:55:00-04:00

We're about to start up a new project at work, and we've decided to go with Merb (yay!) rather than Rails. Before we get started, though, we wanted to make sure that we'd be able to integrate well with the various plugins available for Rails.

The first two libraries we wanted to use were ultrasphinx, an interface to the Sphinx fulltext search engine, and async-observer, an abstraction library using the beanstalkd work queue library to delay actions to be processed at a later time, rather than while processing the page.

The good news is that both of the projects are available on GitHub, which means easy forks, and easy contributions back to the source if my changes are as good as I think they are.

So, today I'd like to talk about the basic changes you'll need to make to a Rails plugin so that it will play nicely with Merb, and a few of the extra hooks that come into play. In the near future, I hope to provide a bit of a primer on adapting a plugin that uses ActiveRecord so that it will also work with Datamapper. Preview tip from that post, if you're calling any methods provided by ActiveRecord, please make an adapter class/module to pass those methods through, as it makes ports like these much easier.

Basics

To get your plugin to get picked up properly, Rails requires an init.rb file in the plugin root. The equivalent for Merb is a file named after your plugin, in the /lib directory. Copying /init.rb to /lib/async_observer.rb worked for that one, Ultrasphinx is somewhat better behaved in that its init.rb just required ultrasphinx, so both Rails and Merb worked for me out of the box.

If you depend on anything in Merb, you'll need to add to the docs that applications should add the plugin dependency inside a Merb::BootLoader.before_app_loads block - otherwise nothing in Merb is defined yet. This is of particular importance if you want to switch behaviour based on whether the Rails or Merb constants are defined. As Rails handles loading plugins itself, there's no concern for keeping things special for Rails.

If you plan on dealing with the app directory structure or environment, an easy way to do it is:

if defined?(Rails)
  ROOT = RAILS_ROOT
  ENV = RAILS_ENV
elsif defined?(Merb)
  ROOT = Merb.root
  ENV = (Merb.env == 'rake' ? 'development' : Merb.env)
end

The additional rake environment transparently proxies to the development db connection, so if you just want to compare your plugin's interpretation of ENV the above will make that cleaner.

Rake Tasks

Rails automatically loads any files matching tasks/*.rake in the plugin dir.

Merb needs to be told explicitly, relative to the lib directory. The canonical example from the docs is:

if defined?(Merb::Plugins)
  Merb::Plugins.add_rakefiles "merb_sequel" / "merbtasks"
end

Unfortunately, it seems that it wants specified a single file with a .rb extension, which is incompatible with the Rails Way. The easiest fix I've found is to add a file called tasks.rb under /lib, inside which you just manually require the individual rake files:

load File.expand_path(File.join(File.dirname(__FILE__), '..', '..', 'tasks', 'merb_sequel.rake'))

Do note that the load is necessary, require doesn't pick the file up properly.

Finally, if you have any tasks that depend on environment, the easiest way to get compatability with both frameworks is to add task :environment => :merb_env to your merbtasks.rb file.

Generators

I haven't looked into generators in too much depth, but the API between Rails::Generator::Base and Merb::GeneratorBase seem different enough to warrant not reusing the generation script. If you conditionally define a generator based on the defined?ness of those two base classes, you should be able to reuse all your generation templates, and both Rails and Merb look in the same place for generators, so that should be the only adaptation necessary.

ORM Integration

Check back next time, as I write up my experience writing an ActiveRecord shim for the latest DataMapper. It's vaguely ugly. I highly recommend if you're working on a plugin now that interacts with models to abstract any access to the database into a module, and include it appropriately. This goes double if you're using any AR magic ;)

Two Questions

2008-07-14T17:54:00-04:00

Question #1: Why is Symbol#to_proc so popular?

class Array
  alias old_map map

  def map(method=nil, *args, &blk)
    return old_each(&blk) if block_given?
    old_map do |e|
      e.send(method, *args)
    end
  end
end

ary = %w(hello world)
puts ary.map(:reverse).join(" ")
puts ary.map(:*, 2).join(" ")
puts ary.map(:slice, 1, 3).join(" ")

It should be trivial to do this for the other common enumerable methods, the only places I see Symbol#to_proc used, anyway.

I posit that the answer is question 2.

Question #2: Why do I need to do this method hackery in Array instead of Enumerable? I can understand Array having its own definitions of the standard collection methods for performance reasons, but in a properly OO system, that should be transparent to me.