Emil Loer

The Rake task-killer

2011-10-15T00:00:00+02:00

If you have a lot of private gems that are packaged using Bundler then there is always the fear of someone accidentaly running rake release and pushing all of your fancy enterprise code to RubyGems. Let’s prevent this from happening!

First, add the following to your Rakefile, just below the require statements.

class Rake::Task
  def kill!
    @actions.clear
    prerequisites.clear
    enhance { puts "This task is not allowed!" }
  end
end

This snippet adds a kill method to all Rake tasks, even those that are defined outside of the Rakefile.

Now, whenever you want to prevent a task from being used you can kill it using the following invocation:

Rake::Task[:release].kill!

When you invoke the release task after calling kill! on it you will be presented with a warning:

$ rake release
This task is not allowed!

Of course you can use it to kill nested tasks too:

Rake::Task[:"db:drop"].kill!

In this example the Ruby on Rails db:drop task is killed.

Replacing tasks

Normally, when you create a task entry for a task that already exists the given block will be appended to the task. This means that vanilla Rake only has the ability to add work to a given task, not replace it.

However, if we clear the actions first and then append a new block to the list of actions (this is what the enhance call does) we can still manage to replace the entire task.

Here’s an example:

class Rake::Task
  def replace &block
    @actions.clear
    prerequisites.clear
    enhance &block
  end
end

Rake::Task[:release].replace do
  puts "Put your own release code here"
end

The one true Objective-C brace style

2011-08-24T00:00:00+02:00

Whenever you create a new project in Xcode there are a couple of Objective-C files automatically generated, such as the application delegate. One thing in particular that bothers me about these files is the messy coding style.

One example of this is the inconsistent placing of the opening curly braces. On methods the opening brace is placed on a new line below the method declaration, and on if statements and the like they are placed on the same line.

Most of the templates also have a lot of trailing whitespace, and the Xcode editor itself is rather sloppy when it comes to cleaning that up.

Because I primarily use Vim for my editing needs, I wrote a little function that cleans things up. You can put it in your .vimrc.

function! TransformObjC()
    " correct the code style of an Xcode Objective-C file
    silent! execute "%s/- (/-(/"
    silent! execute "%s/+ (/+(/"
    silent! execute "%s/\\n{\\n/ {\\r/"
    silent! execute "%s/\\ \\+$//"
endfunction

This function does the following:

Remove the space between the plus or minus and the return type parenthesis on method signatures.
Place the opening curly brace on the same line as the method signature.
Clean up any trailing whitespace.

Because these regular expressions only do minor corrections you can call the function as an auto command when you load a new Objective-C file. This can be done as follows in the .vimrc:

autocmd! BufRead *.m call TransformObjC()

Now you can easily mix edits in Vim and Xcode without being annoyed by sloppy formatting. Just remember to reload buffers when going from Xcode to Vim.

Dealing with project.pbxproj in Ruby

2011-08-15T00:00:00+02:00

For Mac and iOS development the weapon of choice is Xcode. And it’s pretty good for what it does (if you disregard the current stability issues with Lion) when you are just making simple apps. In my own projects, however, things usually get more complex and I need to put a lot of stuff in static libraries and juggle these around in all the projects that use them. It turns out that using Xcode for this library management is quite cumbersome, so let’s see what we can do to automate this.

When you look at a typical Xcode project you see a bunch of code assets and a projectname.xcodeproj file. This file is actually a directory in disguise and contains all the project metadata. Most of this metadata is just settings for the Xcode user interface and is not directly related to our code.

The file that we are interested in is called project.pbxproj. This file contains the entire Xcode project structure. The project is described as an object graph, meaning that every entity in the project is a node that has an identifier, a type and zero or more references to other entities. The ideal representation for such a graph is a list of key-value pairs, and indeed the pbxproj file resembles this. Users of document oriented databases such as MongoDB should feel right at home.

Now when you open the pbxproj file in a text editor you see that the file has a syntax that is somewhat reminiscent of JSON. What you are looking at is in fact a plist file in the ancient (and even deprecated) NeXTSTEP ASCII format. Why Apple still uses this format for Xcode remains a mystery to me.

There are a couple of Ruby gems available to parse and manipulate pbxproj files. However, I find all of these gems too bloated for what they do. We can apply a bit of DRY by utilizing two existing tools.

On OS X there is a command line utility called plutil that, accoording to the manual page, is supposed to be used for syntax checking and plist format conversion. Of notable interest is that it can convert plist files to JSON, which we can parse using Ruby’s own JSON parser. Another interesting thing is that it can read all types of plist formats, including the NeXTSTEP format we are interested in.

This means that a simple pbxproj parser can be shrunk down to a single line of code:

def parse_pbxproj filename
  JSON.parse(`plutil -convert json -o - "#{filename}"`)
end

This function returns a Hash object containing the entire object graph which you can then manipulate in whatever way you want. When the time comes to save the changes you can make use of these extra methods:

class String
  alias to_plist to_json
end

class Array
  def to_plist
    items = map { |item| "#{item.to_plist}" }
    "( #{items.join ","} )"
  end
end

class Hash
  def to_plist
    items = map { |key, value| "#{key.to_plist} = #{value.to_plist};"
    "{ #{items.join} }"
  end
end

def save_pbxproj filename, hash
  File.open(filename, "w") do |file|
    file.write hash.to_plist
  end
end

Now remember that the only types that you should use in a plist are strings, arrays and hashes. If you have anything else, like an integer, you will have to convert it to a string.

The methods above have been tested on about ten different projects using Xcode 4.1 and 4.2 beta and work perfectly. For best results reload your Xcode project after manipulating the pbxproj file with an external program, especially when developing for iOS.

Of course, this is just the beginning. I have used this parser myself to build a tool that can synchronize Xcode projects and GNU makefiles with good success. What will you do to accelerate your workflow?

Const function substitution in Clang

2011-07-17T00:00:00+02:00

I am currently working on a synthesizer for the iPad. An important thing here is that the sound rendering callback must be as fast as possible so that you can reduce the audio buffer size and thus get a lower latency.

This means that, if you want to do anything more than a stupid non-bandlimited monosynth with horrible sound quality (such as the currently trending free Core Synth, but there are many others), you will have to spend a lot of time optimizing the renderer. This is especially important when you want to build a polyphonic synthesizer, where rendering time scales linear with the amount of concurrently playing notes.

Recently I discovered that Clang optimizes differently depending on whether you use single or double precision floating point numbers. The result of this is something you would not expect. Today I want to focus on a single aspect which might slow a lot of things down. Suppose you have this piece of code:

double y = exp(123.456);

Any sane programmer would convert this to machine code by evaluating the exponential function during compilation and writing down a single load-constant operation in the compiled program. In other words, the compiler would treat the exponential function as a const function. That is a function that produces an output value that only depends on the input, meaning that there are no side effects and every invocation of the function with a constant input value will always result in the same output value.

However, it turns out that Clang can only optimize away some of these functions when they are called with constant parameters. So I’ve made a little table which tells you what functions you can safely use with constants. The functions that have a “no” will be compiled to a normal function call even when the input is a constant, so they should be replaced with precomputed constants if possible.

So here’s the table. Measurements were done using the Apple LLVM 3.0 compiler which is found in the XCode 4.2 beta.

Function	float	double
sin	yes	yes
cos	yes	yes
tan	no	yes
asin	no	no
acos	no	no
atan	no	yes
atan2	no	yes
sinh	no	yes
cosh	no	yes
tanh	no	yes
exp	no	yes
exp2	no	no
expm1	no	no
log	no	yes
log2	no	no
log10	no	yes
log1p	no	no
fmod	no	yes
fabs	yes	yes
sqrt	yes	yes
pow	no	no
ceil	no	yes
floor	no	yes
rint	no	no
round	no	no
trunc	no	no

When you write your own functions you can give Clang a hint that this function is a const function and has no side effects. This helps Clang to optimize away instances which are called with constants. In order to help Clang you have to modify your function declaration to be as follows:

int __attribute__ ((const)) myfunction(int arg) {
    return 42 + arg;
}

This trick originates from GCC, so more documentation about function attributes can be found in the GCC documentation.

Plotting command line output

2011-07-09T00:00:00+02:00

Consider the following piece of Python code:

import math

class RCFilter:
    def __init__(self, decaytime):
        self.output = 0.0
        self.coeff = 1.0 - math.exp(-1.0 / decaytime)

    def apply(self, input):
        self.output += self.coeff * (input - self.output)
        return self.output

This is a simple one-pole lowpass RC-filter, frequently used as a smoother for parameters that change over time but should never be changed instantly. You instantiate the class with a time coefficient which indicates how many samples it it should take to reach a ratio of 1 / e between the old and the new value. After that you can call apply to smooth out an input signal.

Now, we want to test the behaviour of this filter with a given signal. This can be done as follows:

filter = RCFilter(10.0)

for i in xrange(40):
    print filter.apply(1.0)
for i in xrange(80):
    print filter.apply(0.0)

Now we have generated 120 samples, but from the numbers alone it is hard to see what is happening. However, we can use some command line magicks to turn this into a plot. For this we are going to use Gnuplot, which should be available in your favourite package manager.

In order to make the plotting as simple as possible, we want to invoke it as a command line script that reads numbers from standard input and presents a plot on screen. This way we can pipe any program we write directly to our plot script.

The hard part is getting Gnuplot to read stuff from the standard input stream of the invoking script. This is not trivial because Gnuplot flushes its own input stream and assumes that standard input is the same stream as the one that is used to enter the plot commands. That means we can only get this working if we concatenate the script and standard input. The Gnuplot commands to plot data from what Gnuplot considers the input stream are like this:

set term pdfcairo enhanced color solid \
    font "Helvetica Neue,8" lw 5 size 8,6;
plot "-" with lines title "";

These commands tell Gnuplot to produce a PDF file which has been rendered using Cairo. This gives us the best visual result. If you want to use PNG output you can use the pngcairo engine instead, although I recommend using a tool like ImageMagick for the conversion because it generates better antialiased results and can handle transparency if you need it. We also set some good looking appearance settings, such as font, line width and output size (in inches at 72 dpi). The second line is the actual plot line and can of course be modified if you want a bar chart instead of a line chart or whatever.

Using these commands directly will not work because we have not yet concatenated anything. To do that we will wrap the Gnuplot commands in a Bash script, like this:

#/bin/bash
(
    echo "
        set term pdfcairo enhanced color solid \
            font \"Helvetica Neue,8\" lw 5 size 8,6; 
        plot \"-\" with lines title \"\""; 
    cat -
) | gnuplot | open -f -a Preview

Now we have our plot commands concatenated with the script’s standard input and pipe that to Gnuplot. This produces a PDF file in standard output which we can pipe to our favourite display program. The open command is Mac OS X specific and giving it the -f -a Preview argument tells it to write standard input to a temporary file and open that file with Preview. For Linux you can use display, xdg-open, eog or whatever.

Now when we run our Python filter script and pipe it to this plot script we get the following window:

Because the generated PDF file contains a vector representation of the plot, you can resize the plot to whatever size you need without any loss of detail.

Searching the news with Tapir

2011-05-04T00:00:00+02:00

Today I read an announcement that Tapir had been launched. Tapir is a new search service for static page blogs. Because my own blog is based on a static site generator (Jekyll) I thought this was an interesting thing to check out.

Tapir enables you to build your own search by using their API to fetch results, thus staying in control when it comes to appearance. Quite the difference when compared to Google Site Search, which maintains its own layout and such.

So here’s how it works. You supply an RSS feed and email address to them and you get an API token for that in return. Then you can hit the API URL with the token and a search query and it will give you a JSON serialized list of results containing amongst other things a title, description and search relevance score. Using some jQuery magic (or whatever) you can display these results however you like.

Now this is all cool and such, but a service like this can be used for many other purposes than searching in programmer blogs.

To explore this I built a little news search engine. I created several API tokens, one for each in a set of news sites like CNN or the Dutch NOS. For each of the tokens the engine hits the API with a search. Now we have several result sets that we can concatenate and sort by relevance.

Because Tapir will continue to poll RSS feeds every 15 minutes the total amount of news headlines you can search in will keep on growing. I wonder how useful such a thing would be after a couple of weeks.

Anyway, you can see my news search engine at http://emilloer.com/newssearch. Have fun!

Fixing column alignment with jQuery

2011-02-12T00:00:00+01:00

Lately I have been working a lot with Kramdown, which is a superset of Markdown. One of the cool features it adds is support for tables. Here is an example:

| Product      | Price |
|:-------------|------:|
| Franziskaner |  3.20 |
| Erdinger     |  2.50 |
| Paulaner     |  3.00 |

If you look carefully you can spot two colon characters in the header separating the header and body. These colons are used to specify the text alignment of the column.

If the colon is on the left then it should be left-aligned. If it is right then it should be right-aligned. When you see a colon on either side the text should be centered. No colons at all means we should leave the text as-is.

So in our little example table we have a left-aligned and a right-aligned column. Now let’s see how this table renders to HTML:

<table> 
  <col align="left" /> 
  <col align="right" /> 
  <thead> 
    <tr> 
      <th>Product</th> 
      <th>Price</th> 
    </tr> 
  </thead> 
  <tbody> 
    <tr> 
      <td>Franziskaner</td> 
      <td>3.20</td> 
    </tr> 
    <tr> 
      <td>Erdinger</td> 
      <td>2.50</td> 
    </tr> 
    <tr> 
      <td>Paulaner</td> 
      <td>3.00</td> 
    </tr> 
  </tbody> 
</table>

Oooh, that’s nasty! Kramdown specifies table alignment with the align attribute of the col tag. This was deprecated way back in HTML 4.01 and made obsolete in ~~HTML5~~ HTML. And apparently the only browser that currently implements it is Internet Explorer. So let’s fix this by using a litle bit of jQuery magic.

To do this we will have to find all col tags and extract the alignment attribute from it. Then we will inject the value of the attribute in a CSS text-align attribute. The trick here is that we need to find the cells that belong to each column.

By iterating over the jQuery object containing the columns of a given table we are given an index. We can use this index in a :nth-child selector to find the cell which belongs to the column. These cells always exist in row elements so we can limit our search to direct descendants of tr tags that are either a th or td element. This gives us the final selector tr>(th|td):nth-child(index) which we can apply to our table. Do note that the :nth-child selector counts from 1!

Here is the full code:

// For each table
$("table").each(function() {
    // Store the jQuery object to the current table for later use
    var table = $(this);

    // Iterate over the <col> tags in the current table
    table.find("col").each(function(index, column) {
        
        // Get the value of the align attribute
        var align = $(column).attr("align");
        
        // Apply alignment if necessary
        if (align) {
            table.find("tr>(th|td):nth-child("+(index+1)+")")
                 .css("text-align", align);
        }
    });
});

Serving static files from within a Rack app

2011-02-01T00:00:00+01:00

Here’s a common pattern you might run across when developing a custom Rack application or middleware. Suppose you are developing some kind of web application that should serve everything from a certain directory except for the root index file, which we want to generate programmatically.

Now, what I see very often is something like this:

if path == "/"
  generate content here
else
  read file from disk
end

However, in order to do it properly you will also have to implement decent error handling, proper header generation, permission handling and all that jazz. So the “read file from disk” step is actually a large chunk of code if done right.

Luckily, we can apply some of the DRY principle here by using the built-in Rack::File application class. A simple scaffold for the described application then comes down to just a few lines of code:

class Application
  def initialize(opts = {})
    @files = Rack::File.new(Dir.pwd)
    # Put your own initialization here
  end

  def call(env)
    # Get the request path
    request = Rack::Request.new(env)
    path_info = request.path_info

    # Check for paths we want to override
    if path_info == "/" or path_info == "/index.html"
      # Generate your content here
      [200, {"Content-Type" => "text/html"}, ["Hello World"]]
    else
      # Delegate to Rack::File
      @files.call(env)
    end
  end
end

And that’s all there is to it. If you also want to serve directory indexes then you can replace Rack::File with Rack::Directory, which applies a similar technique under the hood.

Hashfeedr is back

2011-01-17T00:00:00+01:00

About six months ago me and my good friends Michiel Heijkoop and Pieter Noordhuis teamed up to participate in the 2010 edition of the Django Dash.

The Django Dash is a contest in which up to three developers have 48 hours to create a new Django application from scratch. Our contest entry was called Hashfeedr and can be found at hashfeedr.com. A brief description of our application (from the site itself):

Hashfeedr is a real-time Twitter tracker designed to run on a big screen (e.g. a beamer) to provide the necessary “twentertainment” at your next party or display what the customers are tweeting about your company by running hashfeedr at the office.

After the contest the site went offline becase we had no proper place to host it anymore, and nobody had the time to do anything about it.

However, today I am happy to announce that Hashfeedr is back online, powered by a new fast server, and that I will continue to maintain it here until bit rot eventually makes it fall apart. But let’s hope that moment will not be anytime soon.

By the way, Hashfeedr is open source! You can find the source in our GitHub repository.

Preventing CSRF in Sinatra

2011-01-15T00:00:00+01:00

Recently I have been doing a lot of development work with the Ruby based Sinatra framework. For very small applications it is an excellent choice. However, Sinatra is very minimal and does not venture far in the land of security. So if you’re dealing with sensitive form data it’s a good practice to add some sort of protection against Cross Site Request Forgery (CSRF) attacks.

We can add CSRF protection in our Sinatra app by using the Rack::Csrf middleware. You can install it from RubyGems:

gem install rack_csrf

Now, to use the middleware we have to add the following snippet to our Sinatra application file. Note that we also add a session middleware. This is a dependency which is required by the CSRF middleware and is used to store a unique anti-forgery token per visitor.

require "rack/csrf"

configure do
  use Rack::Session::Cookie, :secret => "some unique secret string here"
  use Rack::Csrf, :raise => true
end

I added the :raise option so that the middleware raises an exception in case of an attack. That enables us to see it in the error log.

After restarting your app the middleware will check all POST, PUT and DELETE requests for the token. Any of these requests not having the token will be blocked, which means we will have to patch all our forms to supply the token. We can get the token from the middleware by calling the Rack::Csrf.csrf_token method with our environment as the argument. There is also a Rack::Csrf.csrf_tag method which works similarly but returns a string with an HTML input tag.

I don’t feel like typing this long method call over and over in my templates, so let’s make some helper functions to keep our templates clean.

helpers do
  def csrf_token
    Rack::Csrf.csrf_token(env)
  end

  def csrf_tag
    Rack::Csrf.csrf_tag(env)
  end
end

If you have worked with Rails these methods might seem familiar, and indeed they do similar things.

Here is an example ERB snippet with the token inserted into a form:

<form method="post" action="/tweet">
  <%= csrf_tag %>
  <input type="text" name="message"/>
  <input type="submit" value="Submit a tweet!"/>
</form>

And there you have it! Your Sinatra app is safe from any CSRF attacks. For more information about the Rack::Csrf middleware check out its GitHub project page.

Control surface

2010-12-20T00:00:00+01:00

I present you the results of two hours of photoshopping:

These knobs are for a super secret project which was in desperate need of some fancy visuals. It was a first attempt at using only shape layers and layer styles to create depth effects, and I think it turned out pretty well.

Checking for dependencies in Bash

2010-12-09T00:00:00+01:00

Here is a quick Bash trick to check for external command dependencies and install them if they aren’t available.

function depcheck {
  type -P $1 &>/dev/null && return
  echo "Installing $2"
  sudo apt-get -y install $2 && return
  echo "Error installing $2"
  exit 1
}

The type internal command prints how a shell command is interpreted by Bash. Specifying the -P argument causes it to enforce a shell path search, bypassing any aliases, functions and internal commands. A nonzero exit status is returned when the argument can’t be interpreted.

Now, if the command exists all is well and we can return. When it isn’t, we are going to install it using sudo. For this you can use whatever package manager you have available. In this snippet I am assuming you are running a Debian based system, which is in my case Ubuntu. Because the command you are checking for might be different from the package name it is in (e.g. git in git-core) we supply a package name as the second argument of our function.

Using our function is easy. For example, this line adds a dependency for the previously mentioned git:

depcheck git git-core

I use this function myself in my dotfiles bootstrapping script. This script clones my dotfiles repository, initializes git submodules and does some other small chores to prepare the account for some serious coding business. But it cannot do so without making sure Vim is available. ;)