Mark Hansen

Visualising NZ's Stolen Cars, Part 2

2011-12-21T00:00:00+11:00

I had a lot of fun making this visualization. d3.js + Backbone.js + CoffeeScript is an amazing combination.

It’s all totally standards-compliant HTML5. Boxes are drawn with inline SVG, ~~and transitions are run in CSS for hardware acceleration~~ (edit: they aren’t by default, you have to do a little extra work for that, see comments). Popovers are Twitter Bootstrap.

Choose a Region

It’s crazy how much the number of stolen cars has picked up in the last few days approaching Christmas.

A Code Walkthrough

As an example of how to do these things with coffeescript, d3, and backbone, here’s the code used to generate the weekDays graph, showing the number of stolen vehicles per weekday.

The Backbone Bits

The whole widget is in one javascript class that takes care of itself - WeekGraph. It extends from Backbone.View, and its constructor hooks it up with an HTML element - @el, and a data model - @model. The view then listens for ‘change’ events coming off the model, and rerenders whenever the model changes. In this way, the view and model stay in sync.

WeekGraph = Backbone.View.extend
  height: 150
  width: 420
  initialize: ->
    @model.bind "change", => @render()

That was the wiring (decoupling and dependency injection if you’re into that).

The d3 Bits

The render() method is where the real magic happens, rendering that data model into the HTML element, so we can see it on screen.

First, separate the stolen vehicles data by weekday. This is accomplished with d3.nest().

  render: ->
    weekDays = d3.nest()
      .key((d) -> d.dayOfWeek)
      .entries(@model.get "selectedVehicles")

We need to separate the weekdays on the graph horizontally across the x axis.

x is a function that does this. You could just use a linear function, but d3 has scale function helpers to make changing the domain and range of these functions really easy, without having to manipulate equations when you want to change the width.

The d3 Scales

x = d3.scale.linear()
    .domain([0, 6]) # weekdays
    .range([0, 6 * (cellPadding + cellSize)])

Also, the bars need to have a height, decided by the number of vehicles stolen that day. y is a linear scale that turns vehicles stolen into a pixel height for the bar.

I’m also making the bar darker when there’s more vehicles stolen that day. d3 can also make scales that interpolate between colors. This seemed a bit magic to me, but really d3 just has a list of color strings, and turns them into RGB colors and interpolates them.

maxWeekday = _.max(weekDays, (d) -> d.values.length)
maxPerWeek = maxWeekday.values.length

height = d3.scale.linear()
    .domain([0, maxPerWeek])
    .range([0, @height])

color = d3.scale.linear()
    .domain([0, maxPerWeek])
    .range(["white", "darkblue"])

The d3 Data-DOM Join

Now let’s select the SVG element, set the width and height, and do a relational join between our weekDay data and the bar DOM elements. I’ll explain in a second.

bars = d3.select(@el)
    .attr("width", @width)
    .attr("height", @height)
    .selectAll("rect.weekDay")
    .data(weekDays)

This data() relational join operator gives us three selections:

The Updating Elements

1: update. data that already existed in the DOM, but might have changed.

So we update the DOM. Use a nice transition of 1000ms. We can pass functions that have (d, i) arguments. d is the data associated with that DOM element, i is the data index. Transitions are all interpolated nicely.

bars
    .transition()
    .duration(1000)
    .attr("x", (d, i) -> x(i))
    .attr("y", (d, i) => @height - height(d.values.length))
    .attr("height", (d, i) -> height(d.values.length))
    .attr("fill", (d) -> color(d.values.length))

The New Elements

2: enter(). Like an actor entering the stage. Data that doesn’t have a DOM element yet. So we append elements to the DOM for each new datum, and set the location, size, class, and fill according to the data values.

bars.enter()
    .append("svg:rect")
    .attr("x", (d, i) -> x(i))
    .attr("y", (d, i) => @height - height(d.values.length))
    .attr("height", (d, i) -> height(d.values.length))
    .attr("width", cellSize)
    .attr("class", "weekDay")
    .attr("fill", (d) -> color(d.values.length))

The Exiting Elements

3: exit(). Like an actor exiting the stage. Elements that don’t match up with any data we have. The elements used to match up with some data, but they don’t any more. So we remove the elements from the DOM.

bars.exit()
    .remove()

The Popovers

Finally, I hooked up a twipsy overlay on mouseover. This uses twitter bootstrap’s twipsy plugin. It demonstrates how each DOM element contains a property __data__ that contains the d3 data.

@$ is this.$, a jQuery selector scoped to the local backbone element. It’s handy to make sure one view doesn’t accidentally attach handlers to another view’s elements.

    @$("rect.weekDay").twipsy
      title: ->
        weekDayName(@__data__.values[0].date) + "<br>" +
        @__data__.values.length + " Reports"
      html: true
      placement: "above"

I hope this has given you a taste of how general and powerful d3 is. It’s a really enjoyable library to use, and it combines well with Backbone and CoffeeScript. Highly recommended.

Visualising New Zealand's Stolen Vehicle Database

2011-12-10T00:00:00+11:00

Stolen Vehicles, by Make

Mouseover the bars for a visualisation of the colors of the stolen cars.

The Database

A few days ago, the NZ Police launched a public database of stolen cars. It’s a snapshot taken from the Police Vehicle of Interest database, updated 3x daily.

"It will be a resource for security guards, insurance companies, moteliers, scrap metal dealers and community policing patrols."

...

"It also has obvious benefits for people buying second-hand vehicles, garages that service vehicles and service stations where petrol thefts can regularly be associated with stolen vehicles."

...

"Potentially, it gives police many more pairs of eyes out there. People can do their own checking and then report it to Police."

This sounds like a goldmine of interesting data. I grabbed the list of all the cars stolen in the last six months. It comes in a handy CSV file. For example, here’s the first four vehicles:

Plate,Color,Make,Model,Year,Type,Date,Region
007579,Blue,Yamaha,TTR230,2007,Trail Bike,2011-07-14,Bay of Plenty
0BOOST,Blue,Honda,ACCORD,2003,Saloon,2011-06-19,Counties/Manukau
1045Z,Orange,Trailer,HOMEMADE,1972,Trailer,2011-11-07,Eastern
1079Y,Silver,Trailer,ABEL K8SSA,2001,Trailer,2011-07-14,Waitemata
...

So I decided to have a crack at visualising the data. Above is the first interactive graphic I’ve published - the most popular makes of stolen vehicles.

Visualisation Tech

This visualisation was a great excuse to play around with some new tech. Rendering is done with SVG, controlled by the excellent d3 visualisation library. Popovers are care of Twitter Bootstrap, which rocks, although I did have to put in a pull request with fixing the popovers for SVG.

Sorry Android 2.3 users, your browser doesn’t do SVG, so you’ll need to use a desktop browser to see.

I was very impressed with d3’s flexibility and conciseness, and I’m looking forward to using it on more projects. I may blog about how awesome the d3 experience was later. If you’re into making visualisations, I heartily recommend checking d3 out.

Automatically Compiling Haml in Vim

2011-12-02T00:00:00+11:00

I don’t write a lot of straight HTML any more - I prefer the rapid prototyping you can do with the likes of Haml, without having to worry about always matching your start and end tags.

However, the main disadvantage of haml is that it needs to be compiled to HTML. This is an extra step in development that slows down iteration.

Previously I used a watchr script, spinning up a process that watches for new haml files and automatically compiles them. This isn’t as seamless as I’d like, because I have to remember to spin up this process every time I want to write some haml. Also, watchr won’t pick up new files without a restart.

But I don’t want to auto compile every haml file. My rails or jekyll templates are supposed to be substituted into templates, not rendered as-is. I want a way to opt in to auto-compilation, preferably only once per directory.

To opt-in, my script checks if a file named “.autohaml” exists in the current directory. Create a file named “.autohaml” to opt in.

I’ve added this autocompile code to my vimrc. I’ve tried to use the minimum of vimscript possible, doing most of the logic using python.

" Auto compile .haml files on save, but only
" if there's a .autocompilehaml file in the cwd.
" Depends on a `haml` executable. `sudo gem install haml`

au BufWritePost *.haml call HamlMake()

function! HamlMake()
    py << ENDOFPYTHON
import os
import vim

in_file = vim.current.buffer.name
dirname = os.path.dirname(in_file)
if os.path.exists(dirname + "/.autohaml"):
    out_file = in_file[0:-5] + ".html"
    os.system("haml %s > %s" % (in_file, out_file))

ENDOFPYTHON
endfunction

A Visualisation of Your Last.FM History

2011-10-31T00:00:00+11:00

Today, I’m launching Scatter.FM, a new visualisation that shows Last.fm users how their listening habits change over time.

If you’ve been a Last.fm user for any length of time, you can use Scatter.FM to visualise your entire listening history.

Oh yeah... I remember that time when we were listening to that!

I associate memories with the music I was listening to at the time. It’s possible to go the other way: unlocking ‘forgotten’ memories of good times with friends by seeing what music you were listening to.

I remember the first time I heard that song...

This is my favourite thing about Scatter.FM, remembering good times from the past. Using Scatter.FM, I can see parties where my iPod DJ’d, study sessions leading up to exams, late night hacking sessions, roadtrips, holidays (sparse areas on the graph), and times overseas (the graph is time-shifted).

I remember when I had that album on repeat for three days straight...

Big life changes often precipitate changes in listening habits - you can see these changes in the graph, reflecting your story.

What does it do?

Scatter.FM extracts the list of songs you’ve listened to from Last.fm’s servers, and plots every song on a scatterplot, so you can see how your listening habits evolve over time.

Scatter.FM plots every song in a Last.fm user’s account as a dot in a graph, plotted by days (x-axis) and time of day (y-axis).

You can hover over any point to see information about the song, the exact time you listened to it, and album art. The graph will show you other times you played the same song by flashing the other dots of the same song.

Your top artists are assigned distict colors so they stand out, and you can filter the displayed data by artist, album, and track using the ‘Filter Songs’ box at the top.

If you have an extra-long listening history, you can zoom and pan in on specific days by double-clicking.

Conclusion

I hope you enjoy! This has been a fun project, and helped bring back memories of a lot of good times. Hopefully it can do the same for other people.

[Scatter.FM]

Unity+XMonad in Ubuntu 11.10 Oneiric

2011-10-17T00:00:00+11:00

I love XMonad. For years I’ve been using it inside Gnome 2, but with the release of Ubuntu 11.10 Oneiric, Gnome 2 is on the way out, and Unity is the way forward. Here’s how to run XMonad with Unity-2D:

1) Install XMonad. Open a terminal and enter

sudo apt-get install xmonad

2) Configure XMonad to interact happily with unity. This involves floating the laucher over the top of the active window, and not trying to tile the top panel.

Make a file ~/.xmonad/xmonad.hs, and put in it:

import XMonad
import XMonad.Config.Gnome

myManageHook = composeAll (
    [ manageHook gnomeConfig
    , className =? "Unity-2d-panel" --> doIgnore
    , className =? "Unity-2d-launcher" --> doFloat
    ])

main = xmonad gnomeConfig { manageHook = myManageHook }

3) Compile this config file in the terminal

xmonad --recompile

4) Tie it all together with an alternative “XMonad Unity” XSession.

Make a file /usr/share/gnome-session/sessions/xmonad.session with these lines:

[GNOME Session]
Name=Xmonad Unity
RequiredComponents=gnome-settings-daemon;
RequiredProviders=windowmanager;panel;launcher;
DefaultProvider-windowmanager=xmonad
DefaultProvider-panel=unity-2d-panel
DefaultProvider-launcher=unity-2d-launcher

Make a file /usr/share/xsessions/xmonad-unity-session.desktop with these lines:

[Desktop Entry]
Name=XMonad Unity
Comment=Tiling window manager
TryExec=/usr/bin/gnome-session
Exec=gnome-session --session=xmonad
Type=XSession

And you’re done! Logout, and log back in, selecting the “XMonad Unity” session. The Unity panel and launcher will happily coexist with XMonad. Choice!

A New Map of New Zealand's Wireless Networks

2011-08-24T00:00:00+10:00

Mt Te Aroha Antenna Mast (Photo Credit: Joel Oughton)

I’ve just launched a new visualisation: a map of New Zealand’s terrestrial Radio/Wireless network.

It’s a map that draws lines showing the links between antennae like the one shown at the right. You might have seen these towers on top of hills while driving. I always wondered what they were used for, and where they connected to. Now you can see where the links go, and who they’re for!

It’s not just huge antenna mast towers like the one shown on the right. Hospitals, big businesses, cell towers, weather stations, and offshore oil rigs are connected with these radio links.

I’m really excited about being able to visualise this data - the map shows over 17,000 locations, and 26,000 point-to-point links over the whole country!

Here’s the visualisation: NZ Wireless Map. Check it out!

Here’s just a few interesting things I found exploring the data:

Lake Rotorua

Lakes are awesome to transmit across! They stay flat, and usually have good hills on either side. Lake Rotorua demonstates this really well - here you can see how densely it’s criscrossed with links.

Auckland Skytower

The busiest place I could find, with the most links converging on one location, was the Skytower in Auckland. Here dozens, if not hundreds of wireless links converge from all over the Auckland region.

Hamilton

Hamilton’s a very flat city, and there’s not much in the way of mountain ranges nearby. I imagine this would make it difficult to transmit signals to the roofs of buildings in town. On the map, you can see how transmitting companies have dealt with this flatness - the few high buildings that do exist are used extensively:

The Hamilton Lake Water Tower (shown)
Waikato Hospital
WEL Building

Signals spread all over town from these vantage points.

Taranaki Oil/Gas fields

You don’t usually notice New Zealand’s offshore gasfields on maps, but they really stick out on a map of wireless links. You can see the Maui A and Maui B platforms communicating through a host of links on the Taranaki shore, and the Kupe Gas field south of the mountain.

White Island

We even have a wireless link set up to an active volcano. Awesome!

Views

Exploring the map, I realised that it inadvertently highlights the best views in the country. Antenna masts are specifically placed for great views in all directions. Lots of research must go into finding the places with these great views before spending vast amounts of money erecting a tower.

Why not take advantage of all this research into great views next time you’re deciding where to go hiking?

Getting the data on the map

If you’ve got a nerdy streak, keep reading. This is where I explain the programming behind the visualisation. It’s all open source, available on Github.

A friend showed me BarTender’s work making maps of cell tower sites he made using Radio Spectrum Management’s database. BarTender explains how he extracted the data from the Microsoft Access Database dump, into a SQLite database with mdb-sqlite, then used SQL queries and XML tools to manipulate the data into KML files.

I managed to dump the SQL out of the Access Database and import it into SQLite using these instructions. Then the data was all mine to mess around with!

Exploring The Database

I found the important tables (here’s the full schema):

Licence: A licence to use the airways granted to a company. Usually this has one or more ReceiveConfigurations and TransmitConfigurations.
ReceiveConfiguration and TransmitConfiguration. These tables give information on locations, antenna equipment, direction, and height.
Location: This table gives each location a name, height. However, you need to join with GeographicReference to get Latitudes and Longitudes.
GeographicReference: NZ uses a whole lot of geodetic datums, and this table holds the location of each antenna in every different format. We’re only interested in WGS84, the worldwide format that Google Maps uses. This table will give us the Latitude and Longitude of each location.

I crafted an SQL query to select pairs of towers, but it was incredibly slow. I put a few indexes on ID fields of the tables I used, and then the query ran in less than a second. Awesome.

Plotting the lines

I wrote a python script to manipulate these transmit-receive antenna pairs into KML, showing the pairs as lines on a map. I hadn’t used KML before, but, aside from being XML, it was very easy. The format is well documented, the only problems came when one of the names had an 0x19 “ASCII device control 3” character, which XML apparently can’t represent. I removed the character, and everything worked swimmingly.

However, the network was far too crowded with fake links, with far too many links going out into the middle of the sea. Upon investigation, these links were all the corners of a “Receive Protection Area” polygon - not an actual point-to-point link. I had to reverse engineer the database to figure out that point-to-point links are identified by locationtypeid = 4, and then the map showed the real structure of the network.

Huge lines were also stretching up to the equator, stretching out to geosynchronous satellites. These lines were really interesting, but they obfuscated the terrestrial links, so I removed the satellite links. I’d like to add them back in somehow, maybe by using another marker on the ground where the uplinks are.

Putting it on a website

This was all great, but I can’t expect people to download a KML file and open it in a desktop application. People expect data to be available in a web-based tool these days, and if it isn’t, many people just won’t bother.

First of all, I tried hosting the KML on a server, and searching for the URL inside google maps, just like BarTender has done with his cellphone maps (here’s an example). However, Google Maps has specific limits on the number of features it will plot: 1000 data points, and 10MB of KML. I had over ten thousand features, and 20MB of KML. This wasn’t going to work.

Fusion Tables

There’s a way around this, by using Google Fusion Tables. Fusion Tables is one of my favourite tools for visualisation. [Disclaimer: I will be working at Google, but I don’t know anyone on the Fusion Tables team. I just think the work they do is awesome]

Fusion Tables is like a spreadsheet program, but with specific support for geographic data types: placemarks, routes, polygons. It comes with a half-dozen visualisation methods, including “Map” view, where it can plot KML on a map. With Fusion Tables, there’s no limit to the amount of points you can plot on the map. Perfect!

I repurposed my KML exporter into a TSV exporter, and uploaded the data to Fusion Tables. You can then specify what information you want in the popup when a user clicks on a line - I made it show the receiver, transmitter, licence id, and licencee. and made the lines a bit thinner and greener.

Fusion Tables makes it almost criminally easy to make interactive geographic visualisations. I’m a huge fan.

Thanks

Many thanks to the people at Radio Spectrum Management in the Ministry of Economic Development. They’ve made public a very high-quality dataset containing all the radio licences in New Zealand. Usually when dealing with datasets with tens of thousands of points, there’s a few problems, but this dataset is pristine. This wouldn’t have been possible without their hard work. Cheers!

Have a look, go exploring, and let me know in the blog comments if you find something interesting. Feedback is much appreciated!

[NZ Wireless Map]

Visualising Auckland's Public Transit

2011-06-17T00:00:00+10:00

Chris McDowall has made a wonderful visualisation of Auckland’s Transit data. He’s taken the Google Transit Feed data from Maxx, and mashed it up using Python/Matplotlib/Basemap/FFmpeg/Vimeo to deliver a birds-eye view of Auckland’s transit system over time.

I’m keen to do something similar for Hamilton’s transport network. Hamilton has recently integrated their bus system with the Google Transit Feed API, so I know there must be the data I need sitting publicly available on some web server somewhere, in delicious plain CSV format. I’m just not sure where. I’ve asked Busit! if they can point me in the right direction, so watch this space.

Chris has made a lot of other excellent visualisations, they’re pretty cool. Check them out.

Hamilton's Bridges in Fog

2011-06-13T00:00:00+10:00

Lights have been attached to the Bridge St Bridge, and they look amazing in Hamilton’s signature thick fog. The fog’s usually pretty thick, but it’s even thicker along the banks of the Waikato River, which these bridges cross.

I got a chance to snap some pictures of Bridge St bridge a few nights ago, along with the Fairfield bridge and The Base.

Click to embiggen:

Fog photos are a small obsession of mine. This time last year, I took some fog photos of Waikato University. Do you know any other areas in Hamilton that are amazing in the fog?

Visualising the New Zealand Government Budget

2011-05-19T00:00:00+10:00

I never used to care about the government budget. Every year, I’d see a storm of Budget stories on the news, and I wouldn’t understand any of the numbers they were throwing about. $50 million cuts to education. $2 billion for fibre broadband. $200 million increases for student loans.

I knew these were big numbers, but I can’t reason about them. Are they huge? Or are they pocket change, compared to the government balance? Are they going to bankrupt our country? I just didn’t know. I didn’t have any context, so I couldn’t reason about these huge numbers. I got frustrated, and I didn’t pay attention.

It’s a shame, because the Budget is arguably the most important thing the government does. It decides funding for every aspect of the government, where the money comes from, and where the money goes. It distributes billions of dollars, impacting every one of us, and shaping the future of the country.

And yet, it’s inscrutable. The data is available, but it’s buried deep in spreadsheets - which is great if you want to do hardcore analysis of the data, but not for interactive exploring.

Presenting: A Visualisation

Today, with the announcement of the 2011 Budget, we’re launching Where’s My Taxes? , a visualisation of what the goverment’s spending your money on.

With this website, we let you explore goverment expenses, and bring huge numbers into context by showing the the per capita levels of spending of each item.

Picking an example at random, it’s very difficult for me to reason about a $250 million spend on Pacific aid. Is it high? Is it low? I don’t know! But when you convert it to per capita spending - it comes out at $57 per person - that’s a number that’s easy to reason about. “That’s two boxes of beer!”.

As another random example, look at the Student Allowance. $660 million spending per year, according to the budget. That’s got very little context, but when you work it out to be $148 per person, again, it’s a lot easier to reason about the costs/benefits, when you can see the average cost to you.

It’s also a lot easier to see the relative importance of each government Department when you plot it as a pie chart. It’s easy to see, for example, that the Police cost less than the Defence force, and the Department of Conservation is a tiny fraction of the cost of Health and Education.

Conclusion

I hope that this website helps people understand where their taxes are going. Better informed voters can only lead to better governent.

Thanks

Thanks to Luke Bjerring, my partner in this assignment, and Ryan Delaney, for whipping up an excellent design on very short notice.

We took a lot of inspiration from the DataVizChallenge, a challenge to visualise US goverment taxes. There are some amazing visualisations of the US budget, and I applied some of these techniques to visualise the New Zealand data. In particular, I want to thank Where Did My Tax Dollars Go?, an excellent visualisation that provided a lot of inspiration for this project.

This was developed as a project for the Information Visualisation class at the University of Waikato. Cheers to our lecturer, Mark Apperley, for giving us the freedom to pursue whatever visualisation we like.

[Where’s My Taxes?]

I'm interning at Google!

2010-10-27T00:00:00+11:00

I’m interning at Google Sydney!

I’ve just accepted their internship offer for three months over the summer. I’ll be relocating to Sydney till February next year.

This is pretty much a dream come true for me - Google’s got a huge reputation as a great place for developers to work.

Good times are expected!

My preparation for the interviews:

I checked out as many books as I could carry (go the uni library!). Books on networks, algorithms, methodology, languages, parallel programming - anything I could get my hands on.

In the interviews, we talked about designing for scale, languages, algorithms (lots of algorithms) - all the juicy bits of Computer Science. We even did some live coding via Google Docs. It was really challenging, and I’m lucky to have got through.

I’m pretty excited. This is shaping up to be a great summer!

Awesome Small Software: Watch

2010-09-03T00:00:00+10:00

I’m starting a small series where I do shout-outs to my favorite bits of software - software that’s just awesome. Software that will make your life easier. Software that you want to know about.

As a political statement, I’ll be focussing on small software. I particularly like software that does one thing, and does it well. I love small software. It’s more maintainable, easier to understand, and more likely to be correct.

Small software is inherently more flexible than large software, because it’s easier to understand the small software to the point where you can change it.

One of my favorite small pieces of software is watch.

Ok… what does watch do?

In general, you need to run a bit of code, and you need it running often, sometimes in the background while you’re working. watch lets you run arbitrary code, repeatedly every few seconds. watch program runs program every 2 seconds, and displays the output.

Give me some examples, man!

Sure thing!

You’re working with javascript, and you’re getting sick of syntax errors. It seems every time you reopen your site, things just aren’t working as you’d expect. you’ve missed a semicolon somewhere, or misspelled a function. Stupid mistakes. And you’ve got to switch over to the browser console to find these errors, then copy the line number back into your editor, navigate to the line, and look through the line to try to spot your error.

If only there was some way to let me know immediately as soon as i made a mistake! watch and jslint to the rescue!

# jslint knows when you make mistakes in javascript
watch jslint *.js

Of couse, that last example was silly. You’ve got 100% coverage with your test suite on all your code, because you’ve diligently practiced TDD. (right?) Better to actually run your code, exercise it, and flush out the real bugs along with the syntax errors.

Doing this manually sucks - leaving your editor, running the tests, then coming back to your editor takes valuable time. It’d be great if the tests could be run continuously - then you’d catch bugs as you make them, and you won’t have to backtrack/search to find where the bug is.

watch runtests.rb

You’re working in a templating language (TeX, markdown, haml, whatever), and there’s this stupid error - you’re not escaping some character somewhere, and everything’s been thrown off. You want to see the compiled output as you’re working on the file, so you know immediately when you’ve fixed the problem:

Of course, you could edit, exit your editor, recompile the document, open the result, see what’s changed, then switch back to your editor, clean up some more, exit your editor, recompile again, open the result again, see that it’s still not fixed…

But you don’t hate yourself: you want to shorten this whole write-compile-debug loop. So you open up a terminal next to your editor, firing up your good friend watch:

# continuously compile my markdown report and display the html
watch markdown COMP301_final_report.markdown

You’re waiting on this massive download of (completely legal, creative-commons licensed) music. You want to know the second it appears finished in your download folder:

watch ls -l ~/Downloads

You’re on a shared box, and it’s going dog-slow. What the hell are the other users up to?

# programs that aren't mine, and aren't system
watch 'ps aux | grep -v mark | grep -v root'

You’re debugging an HTTP API, and you want to see the full response to a request as you modify the backend. This is a DELETE method, and your browser only supports GET and POST. Refreshing in-browser isn’t going to cut it. This would be better served as a unit test, but you’re doing exploratory programming and want to see the result of what you’re doing in real time.

# keep on firing DELETE requests!
watch 'curl -X DELETE http://testserver.markhansen.co.nz/api/testpage'

1-day is running one of their crazy ‘new items every few minutes’ sales, and you want to know immediately if rock band drum kits come on sale, because you’ve been wanting to learn the drums for ages, but you’ve got no room for a real drumkit. Whatever - the electronic version is way fun anyway. You fire up watch the keep an eye on the site

# gimme them bargains!
watch 'curl www.1-day.co.nz | grep -i "rock band"'

You keep tabbing around in your trusty IDE, (vim, naturally) but sometimes the jump-to-function fails - your tag cache is stale! You’re sick and tired of exiting to the console and rebuilding to the taglist every time this happens, so you fire up a background terminal and call on your trusty friend watch to do it for you, every 2 seconds:

# ctags, work your magic analyzing my code!
watch ctags my_code_directory -R

Conclusion

It’s really useful to get (almost) real time status updates on almost anything. I use it whenever there’s a long write-compile-debug cycle where the recompiling could be automated.

It really shines with interpreted languages, where there’s no compile wait and you get almost instant feedback, running your tests continuously.

The default setting for watch is to check your command every 2 seconds, but you can change this with the -n option.

# we have a lot of files, it's going to take a while to make the tags db
# give it 30 seconds
watch -n 30 ctags ~/awesomeproject/ -R

Just knowing watch exists, you’re at a massive advantage whenever you need some task repeated. I’ve found so many places where watch is a natural fit. I bet you will too!

iPhone as Numberpad

2010-08-01T00:00:00+10:00

I made a simple HTML5 iPhone app that displays a numeric keypad.

My laptop doesn’t have a numberpad, and I needed one for some of Blender’s keyboard shortcuts. I was about to buy a USB numeric keypad, when I thought: If I call myself a programmer, I should be able to make a numeric keypad. Screw paying for one! So I made one.

When you touch one of the keys, it POSTs the key to a Sinatra server on my laptop, where xdotool sends the X11 event for the keypress to the active window.

There’s no force feedback, so this wouldn’t do for serious data entry. But for triggering simple keyboard shortcuts, this works perfectly.

Thoughts:

I’m really grateful someone wrote xdotool so I don’t have to get my hands dirty with the X11 C++ API.
I love Sinatra. It’s never been so easy to make a webapp, especially simple ‘glue’ applications like this one. It really is a superbly simple framework, and it gets right out of your way. The whole serverside code was less than 50 lines, most of which my validation of keycodes:

#!/usr/bin/env ruby
require 'rubygems'
require 'sinatra'

configure do
    # make haml compile to html5 
    set :haml, :format => :html5
end

get '/' do
    # compile views/index.haml 
    haml :index
end

post '/press/:key' do
    # get the keycode for the numberpad key
    keysym = keysyms[params[:key]]
    # invoke xdotool to send the keypress to the active window
    system("xdotool key #{keysym}")
end

# keycode reference is at /usr/include/X11/keysymdef.h
# all of these are keypad events, so they start with 'KP_'
keysyms = {
    'zero' => 'KP_0',
    'one' => 'KP_1',
    'two' => 'KP_2',
    'three' => 'KP_3',
    'four' => 'KP_4',
    'five' => 'KP_5',
    'six' => 'KP_6',
    'seven' => 'KP_7',
    'eight' => 'KP_8',
    'nine' => 'KP_9',

    'multiply' => 'KP_Multiply',
    'divide' => 'KP_Divide',
    'add' => 'KP_Add',
    'subtract' => 'KP_Subtract',

    'decimal' => 'KP_Decimal',
    'enter' => 'KP_Enter'
}

Haml is excellent. It took 15 minutes to learn, and it made it so much faster to prototype new layout ideas. It was tricky to get the calculator aligned properly, but Haml made it easy to move bits of layout around, and change CSS classes:

%html
  %head
    %meta(charset='utf-8')
    %meta(name='viewport' content='width=320; height=480; user-scalable=0;')
    %title Numpad
    %link(rel='stylesheet' href='/site.css' type='text/css' media='screen' title='default style sheet')
  %body
    #content
      .row
        .nothing &nbsp;
        #divide /
        #multiply *
        #subtract -
      .row
        #seven 7
        #eight 8
        #nine 9
        #add.tall +
      .row
        #four 4
        #five 5
        #six 6
      .row
        #one 1
        #two 2
        #three 3
        #enter.tall ↵
      .row
        #zero.wide 0
        #decimal .
    %script(src='http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js')
    %script(src='site.js')

Adding a class was a simple matter of adding a ‘.classname’ to the element. Removing an element, I didn’t have to find and remove it’s closing element, I just delete a line. The same for adding an element. It feels a lot nicer than HTML. I might redo this site in Haml sometime.

Try It

If you’re on an X11 system, and have a phone with a browser, head over to the github page for installation instructions.

Your Language Influences your Thinking

2010-07-27T00:00:00+10:00

There’s a great article over at the Wall Street Journal about how language isn’t a dumb, passive medium of data transport. Rather, language actively shapes our thoughts and ideas. It’s a great article, go read it. I’ll wait here.

All this new research shows us that the languages we speak not only reflect or express our thoughts, but also shape the very thoughts we wish to express. The structures that exist in our languages profoundly shape how we construct reality.

The same idea, expressed in different languages, is no longer the same idea. Each language carries different connotations:

English speakers tend to say things like "John broke the vase" even for accidents. Speakers of Spanish or Japanese would be more likely to say "the vase broke itself."

Each language embeds different details in its grammar:

In Turkish, you would have to include in the verb how you acquired this information. For example, if you saw the chubby fellow on the wall with your own eyes, you'd use one form of the verb, but if you had simply read or heard about it, you'd use a different form.

These subtle differences shape your thinking - different languages emphasise different aspects of the situation.

You can see the same thing with programming languages! Functional languages like Haskell or Clojure favor verbs over nouns. Both these languages offer a multitude of functions that act on a small number of basic noun types. Using a functional language, you’re likely to approach a problem by thinking about the functions needed - the data transformations, the procedures. The data takes a back seat - the act is important, rather than the actor.

Other languages offer a more noun-based approach (e.g. Java, C#). These languages offer first-class support for a large number of nouns (objects), where the object is the first-class citizen. Functions are only valid in the context of an object. Functions only exist as the children of objects. The actor is important, not the act.

Other languages offer a hybrid approach (e.g. JavaScript, Scala), offering a more-or-less equal footing between verbs and nouns. For example, Underscore.js (a JavaScript library) offers the same operations with your choice of noun-first or verb-first grammar. I love this choice - it lets me choose the way I approach a problem. Some problems seem predominantly noun-based (e.g. modelling and simulating), but some problems seem more verb-based (e.g. dataflow, mathematics), and I get to choose the most readable approach for my domain.

It goes further than functional/object-oriented grammar differences. For example Scheme programmers focus on building large solutions out of many smaller problems. The whole language embodies this - Scheme only defines a minimal set of primitives, and the rest of the language is built from combining these primitives.

The good news: as developers, it’s easy to gain access to these different worldviews! It’s relatively easy for us to pick up a new language - A new way of looking at the world. It’s definitely a lot easier to learn a new computer language than to learn a new human language.

Try something more exotic:

Prolog - see problems in terms of logic!
Erlang - gain a glimpse into their actor-based worldview.
Clojure - learn to live without changing any state.
Haskell - live without any nouns at all! (while, hopefully, still getting stuff done)

Any languages offering compellingly different worldviews that I left out? Leave a comment.

See Also: - Steve Yegge’s epic ‘Execution in the Kingdom of Nouns’. If you’re thinking ‘nouns are all that we need anyway’, you need to read this.

Fog @ Waikato University

2010-07-26T00:00:00+10:00

Hamilton gets heavy fog at night time.

It plays with the lights at the university, turning downward-facing lamps into bright cones of light, turning spherical lights into ambient glowing orange, and turning groups of streetlights into aurora.

Click to embiggen:

UPDATE June 2011: I took some more fog photos, this time of foggy Hamilton bridges.

Porting BASIC to JS

2010-05-30T00:00:00+10:00

A few weeks ago, I heard one of my friends was trying to port some old BASIC code to PHP. (Why? I’m still not sure.)

Somehow, the mention of BASIC triggered some nostalgia for the days of cutting my teeth programming on the Apple IIc, typing in BASIC programs from a book. I had to have a go.

So I ported the old code to JavaScript.

It was a lot easier than I expected - mostly because both languages have global scope by default.

JavaScript gives you the option of global or local scope every time you declare a variable - if you declare it with var it is local to the function.

var a = "a function-scoped variable";
b = "a global variable";

I never thought I’d see global scope as a feature, but it really helps porting from BASIC :)

The code tells you sunrise and sunset times anywhere in the world, so it uses a bit of trigonometry. BASIC defines the SIN, COS, SQR, ATN, and SGN functions. Plugging these into JavaScript was easy:

var SIN = Math.sin;
var COS = Math.cos;
var SQR = Math.sqrt;
var ATN = Math.atan;
function SGN(num) {
    if (num < 0) return -1;
    if (num > 0) return +1;
    return 0;
}

The rest of the code was just converting

GOSUB -> function calls
GOTO -> if/else statements
colons -> semicolons
array indexing from one-based to zero-based
removing line numbers.

Easy.

Of course, the code is up on GitHub. I’m a little disappointed GitHub doesn’t have syntax highlighting for BASIC. ;)

Gnome+XMonad in Ubuntu 10.04 Lucid

2010-05-16T00:00:00+10:00

XMonad is a tiling window manager for X11. I’ve only used it for a day, but I’m addicted. I can’t see myself going back to a non-tiling WM:

I love not having to resize windows to put them next to each other.
I love the extra screen space of not having window borders
I love being able to run tons of programs on my small laptop screen, but still be able to see them all
As a developer, I love that xmonad is small (just a thousand lines of Haskell), AND it’s been run through a theorem prover!

However, you probably don’t want to run xmonad as a top-level window manager. If you run xmonad inside a gnome session, you get all the good stuff gnome usually does for free. Things like message notifications, wireless network connectivity, graphical multiple monitor support, password-management, volume control, printer support, power management, and device management. You could set up all that independently of gnome (and people do), but I don’t consider it a good use of time.

Here’s how to run XMonad as a drop-in replacement for gnome’s default window manager (Metacity):

1) Install XMonad. Open a terminal and enter

sudo apt-get install xmonad

2) Configure XMonad to interact happily with gnome. Make a file ~/.xmonad/xmonad.hs, and put in it:

import XMonad
import XMonad.Config.Gnome

main = xmonad gnomeConfig

Compile this config file by typing xmonad --recompile at the terminal.

This is the crucial step most guides leave out. If you don’t do this, xmonad will try and tile the two gnome panels, and then the nautilus desktop application will steal full the whole screen, and you won’t be able to switch to your other programs, and you will swear a lot.

3) Tell gnome to use xmonad instead of metacity. At the terminal, enter:

gconftool-2 -s /desktop/gnome/session/required_components/windowmanager xmonad --type string

And you’re done! Logout, and log back in. The gnome-panel will happily coexist with the tiling window manager.

Log to the Error Console from a Firefox Extension

2010-05-13T00:00:00+10:00

To log to the console from inside a firefox extension’s javascript:

Application.console.log("Hello from my Firefox Extension!");

Get at the output from Tools -& Error Console.

If you’re doing Firefox Extension development, you can’t log to the Firebug console using console.debug(object) like you’re used to (‘console is not defined’).

At the time of writing, the ChromeBug XUL debugger (developed to debug firebug) is horribly buggy, and doesn’t have a console. So it’s either this, or alert() statements.

Bad Benchmarks

2010-05-10T00:00:00+10:00

I saw someone ask today if Chrome really is faster than Firefox. I’ve always felt that chrome was ‘snappier’ than Firefox, but I thought I’d run a little (unscientific) test, to see if it really was.

I compared Chrome and Firefox for startup time: from launch (from the terminal) till when the window appears and I click the close button.

moon@Moon-Satellite ~> time google-chrome
0.56user 0.26system 0:03.03elapsed 27%CPU (0avgtext+0avgdata 246176maxresident)k
2304inputs+112outputs (3major+32511minor)pagefaults 0swaps

moon@Moon-Satellite ~> time firefox
1.80user 0.18system 0:09.21elapsed 21%CPU (0avgtext+0avgdata 206864maxresident)k
64648inputs+160outputs (233major+15989minor)pagefaults 0swaps

Awesome! My little test had shown that Chrome started in 3 seconds, compared to Firefox’s 9 seconds. That’s well outside the margin of error I give for between me seeing the window is open and closing it. Hypothesis confirmed!

Not so fast. I reran the tests, and the results were telling:

moon@Moon-Satellite ~> time google-chrome
0.56user 0.21system 0:02.12elapsed 36%CPU (0avgtext+0avgdata 240384maxresident)k
0inputs+216outputs (0major+42590minor)pagefaults 0swaps
moon@Moon-Satellite ~> time firefox
1.57user 0.12system 0:02.70elapsed 62%CPU (0avgtext+0avgdata 199936maxresident)k
0inputs+112outputs (0major+15669minor)pagefaults 0swaps

With a warm cache, Firefox manages to start in 2.70 seconds, and Chrome starts even faster, at 2.12 seconds. This is inconclusive - the difference (half a second) is probably within the time it would take me to notice the window had popped up, and click the close button, telling time to stop the clock.

Of course, the first time you run firefox, the operating system has to go to the disc to get the firefox application, and all its configuration. The next time you run it, the operating system has a cache of all that data in RAM. RAM is just a straight read off a register chip, using only electricity, and electricity moves at almost the speed of light. You can only access the hard disk at the speed it spins (about the speed of a car). Big difference.

When benchmarking, beware of confirmation bias, and cache effects (even when the test is unscientific, and just for a bit of fun!).

Vim slow startup opening HTML

2010-05-09T00:00:00+10:00

Editing files earlier, I’d noticed it took ages to load a file into vim. The screen would display the filename at the bottom, wait for about 10 seconds (pegging the CPU at 100%), and then display the file. This only happened for Markdown files, so I opened markdown.vim.

My first instinct was a runaway regular expression, backtracking all over the code

syntax match  htmlH1       /^.\+\n=\+$/ contains=@Spell
syntax match  htmlH2       /^.\+\n-\+$/ contains=@Spell

These regular expressions match Markdown headings that look like

Heading 1
=========

Heading 2
---------

I was concerned with the .\+ part of the regular expression. . means ‘match any character’ and \+ means match as many of the previous identifier as possible. \+ grabs as many characters as possible - it is a greedy operator.

If you tell Vim to match as many of ‘any character’ as possible, it will first match the whole document, then have to backtrack through the whole thing to find a match for \n=\+$ (a newline containing only = characters).

Turns out this wasn’t a problem, because in vimscript, the ‘match any character’ doesn’t match newlines, so the regex only goes to the end of the line. This is exactly the behaviour you want for the markdown parser. This wasn’t the code that was pegging my CPU at 100%.

Commenting out various lines of markdown.vim, I found the line causing the slow loads:

runtime! syntax/html.vim

There was a bug in the html syntax file that was causing this trouble. I tried commenting out the VBScript part of the html syntax file (who embeds VBScript in HTML any more?), and then the CSS section. Turning off CSS fixed the problem right away.

The problem was this nifty little vimscript I downloaded a week ago to display CSS color strings like “#e0f300” in the color they represent: css_color.vim.

Removing this plugin bought me back to the happy world of sub-second startups. It’s a shame, though, because css_color.vim seems really useful to visualise colors as you’re writing CSS. If someone could fix it up to not murder my startup time, that’d be awesome! UPDATE 2011-10-02: Someone’s done just that - there’s a much, much faster version available.

Lessons Learned at Reddit

2010-05-09T00:00:00+10:00

I learned a lot about building scalable websites reading Steve Huffman’s ‘Lessons Learned at Reddit’.

I’m super grateful they provided a transcript of the video, because Flash sucks big time on Linux. I tried to play it full screen, and the video would stutter, even though it had already loaded. I can’t wait for the native rendering speeds promised by the <video> tag.

Extreme Testing

2010-04-02T00:00:00+11:00

I hear again and again that SQLite’s source code is among the best to learn from - high praise. But I had no idea how extreme their testing procedures are.

I have never heard of a project being this extensively tested.

From How SQLite is Tested:

SQLite has 100% branch test coverage on over 67 thousand source lines of code, with 46 million source lines of test cases.

That’s 679 times as much test code as production code. Wow.

And it doesn’t stop there. SQLite has:

Out-of-memory tests
I/O error tests
Crash/Power loss tests
Fuzz tests for invalid input
Boundary value tests
Malformed database tests
Memory usage tests (valgrind)

The article dives deep into the challenges of such extensive testing. For example: How do you get 100% branch coverage when you have ‘defensive code’ that should never be exercised? (the answer: leave the defensive code in, but remove it with the preprocessor before testing.)

It’s well worth a read: Testing in SQLite.

Updated node-XMLHttpRequest

2010-03-29T00:00:00+11:00

I had an evening off from assignments, so I updated node-XMLHttpRequest to reflect the extensive changes in the latest node.js API (0.1.33).

node-XMLHttpRequest is a port of the browser’s XMLHttpRequest object to node.js, to help ease porting of browser-based javascript to the node.js platform.

Race Conditions

I thought I was going crazy when the HTTP Client silently failed to connect to the HTTP server. Eventually I found a race condition was set up in some test code between starting an HTTP server and starting the HTTP Client. It seems the http.createServer call is made asynchonously (like everything in node), and there’s no event listener for when the server has binded to a port, so you just have to guess when it’s ready to take your connection.

That’s easy enough, as a quick hack: good old setTimeout delayed invocation of the client for 100 milliseconds.

http.createServer(/*SNIP*/).listen(8000);

setTimeout(function() {
    /* set up a client to connect to the above server */
}, 100)

HTTP Headers are case-insensitive

The old tests had an assertion, which was failing:

var headers = "Content-Type: text/plain\r\nContent-Length: 11\r\nConnection: close";
assertEquals(headers, this.getAllResponseHeaders());
// assertion failed: 
// expected: 'Content-Type: text/plain\r\nContent-Length: 11\r\nConnection: close'
// actual:   'content-type: text/plain\r\ncontent-length: 11\r\nconnection: close'

I did some research on why node.js lowercases header names and was surprised to find that the RFC 2616 defines HTTP Headers to be case-insensitive.

HTTP agents are free to manipulate the case of headers, so agents shouldn’t depend on the header having a certain case. Oops.

So why does node.js automatically lowercase these headers, instead of leaving them unchanged? Let’s look at these options.

Pass on the headers to the user of the code, unchanged.

To be RFC 2616h compliant, the programmer would have to do a case-insensitive match on the headers. Not trivial when the headers are given as keys to an object:

var contentType; 
for (var header in req.headers) { 
    //case insensitive match
    if (header.match(/Content-Type/i) { 
            contentType = req.headers[header]; 
            break; 
    } 
}

Enforce strict lowercase or uppercase on headers.

Clients can access the headers with:

var contentType = request.headers['content-type']

Put like that, it’s kind of a no-brainer. I’m grateful to node.js for helping me make my code not only compliant, but also liberal in what it accepts. And I don’t even have to lift a finger. Awesome!

Trust the Profiler

2010-03-27T00:00:00+11:00

I was totally wrong.

I’ve been working on an LZW data compressor for my Algorithms paper with a partner.

We thought we were being smart about it - we designed our data structures to use a multiway trie on the compressor, and a simple lookup table on the decompresser. Runtime should be strictly O(n) to compress any n-sized file.

Our code worked wonderfully on small files, but as soon as we turned it to the 5MB Brown Corpus, after a few minutes it had only chewed through 20% of the file, and it was getting slower as it went. Definitely not O(n) behavior.

Every iteration, we were careful to only make one change to the data structure, and one move in the data structure.

My partner thought that we should profile the program to find out where the bad behavior is, but I didn’t think that’d be necessary - we’d be able to find the flaw pretty quickly just by looking through the code, right? - there were only a few accesses to our data structures. We must have been doing something wrong there.

Well, I tried looking, and he profiled the program. He easily got the answer in seconds, and it wasn’t something obvious just from looking at the code, at all.

By running java -Xprof Compressor < brown.txt the built-in Java profiler was invoked, which samples that stack every 10ms, to see which function it’s in. He found over 90% of computation time was spent inside Vector.toString()…

boolean DEBUG = false;
private void log(String s) {
    if (DEBUG) {
        System.err.println(s);
    }
}

/* ...SNIP... */
log(myVector.toString()); //on every iteration

However, even though debugging was disabled, java was still evaluating myVector.toString() every iteration. Of course, toString() has to iterate through every value in our data structure. All this computation was occurring, even though log() was effectively a no-op.

So, now I’m sold on two things:

Profilers. Especially java’s builtin profiler - could it be easier to profile your code? I don’t think so. Just add the -Xprof flag.
Lazy Evaluation. Why compute data you’re not going to do anything with?

Rainy Nights

2010-02-27T00:00:00+11:00

Love the sound of a rainy night? I know I do! Well, there’s a site for that.

The best way to stay cool

2010-02-26T00:00:00+11:00

We were partying at a dance club, it was busy, and I was overheating. So I asked for some water - but what they gave me was 90% ice, topped off with water. It did an awesome job at cooling me down.

Later on, I got thinking - How much better is ice at cooling you down than water? (Hey, I was a chemistry student in a past life).

While water does an admirable job cooling you down at a dance party, clearly ice is better. But how much better?

To make this fair on water, we’ll skew our test a bit in water’s favor, comparing:

the coldest possible liquid H₂O (water at 0°C)
to the warmest possible solid H₂O (ice at 0°C)
for cooling down a hot person (slightly over 37°C)

Water

Water cools you down in two ways:

Water is usually less than 37°C when you drink it. Once it is inside your body, its temperature rises.

Temperature rises need energy, and the energy is taken from your body heat, cooling you down.
You can sweat the water out, where it vaporises (turns to gas). Vaporisation requires energy, some of which is taken from your body heat, cooling you down.

As anyone who’s ever been to a concert can testify, sweating is useless when there’s no wind, and humidity is at 100% (e.g. in a dance club). So we can ignore its effects here.

It takes 4.18J to heat one gram of water by 1°C. Ingesting water at 0°C, you will raise its temperature by 37° to body temperature.

4.18J/g x 37° = 155J/g

The energy to heat up the water comes from your body heat, so you will lose 155J per gram of water you drink.

Ice

Ice melts, forming water (obviously). It takes a lot of energy to melt the ice - you have to partially break hydrogen bonds - one of the strongest bonds in chemistry.

It takes 334J to melt one gram of ice. That’s almost twice the energy you lose in heating that same gram of water from 0°C to 37°C!

And, once that gram of ice is melted, you’re left with a gram of water at 0°. Which, as we’ve established, cools you down by 155J.

melting energy + heating energy = total
334J/g         + 155J/g         = 489J/g

Conclusion

Drinking one gram of 0°C water will cool you down by 155J. Drinking one gram of ice at 0°C will cool you down by 489J.

So, at minimum ice cools you down over three times as well as water. If the ice is colder, or the water is warmer, ice does even better. Awesome!

Hofstadter on Intelligence

2010-02-24T00:00:00+11:00

Reading GEB today, I came across this nice definition of intelligence I had to share:

No one knows where the borderline between non-intelligent behaviour and intelligent behaviour lies; in fact, to suggest that a sharp borderline exists is probably silly. But essential abilities for intelligence are certainly:

to respond to sitations very flexibly;

to take advantage of fortuitious circumstances;

to make sense out of ambiguous or contradictory messages;

to recognize the relative importance of different elements of a situation;

to find similarities between situations despite differences which may separate them;

to draw distinctions between situations despite similarities which may link them;

to synthesize new concepts by taking old concepts and putting them together in new ways;

to come up with ideas which are novel.

– Douglas R. Hofstadter, in Godel, Escher, Bach: An Eternal Golden Braid

Ruby from Java

2010-02-23T00:00:00+11:00

10 Things Every Java Programmer Should Know About Ruby (2006), by Jim Weirich.

This is a concise guide to the major differences between Ruby and Java (or java-like languages, e.g. C#). Reading it, I finally feel like I fully understand mixins and messages. The real-life examples for everything helped a lot.

I can recommend these slides for anyone coming from a Java or C# background who wants to learn about Ruby.

Selling Static Analysis

2010-02-20T00:00:00+11:00

The people behind Coverity, a commercial static analysis bugfinding tool, have shared their experiences in selling business software. It starts off a little slow, but it’s a fascinating read, exposing many counter-intuitive problems in the way software is developed:

Some managers refused to upgrade to the latest version of the bugfinding tool. That sounds like a no-brainer to me:

you get the latest version
you get a better analysis of your code
you find more bugs
you fix them before your customers find them

But if you’re a manager and your bonus depends on the bug number going down, the last thing you want is a new version of the tool to push bug numbers back up.

At company X, version 2.4 of the tool found approximately 2,400 errors, and over time the company fixed about 1,200 of them. Then it upgraded to version 3.6. Suddenly there were 3,600 errors. The manager was furious for two reasons: One, we “undid” all the work his people had done, and two, how could we have missed them the first time?

Gah!

The article’s a great read. Check it out.

Installing JSON.vim

2010-02-18T00:00:00+11:00

Download Json.vim to ~/.vim/syntax/json.vim
Make a new file ~/.vim/ftdetect/json.vim and write in it:

au BufRead,BufNewFile *.json set filetype=json

How does this work?

When you put a file in ~/.vim/syntax/{name}.vim, that syntax highlighting file will be loaded whenever the filetype variable is set to {name}.
The second step sets filetype=json whenever you load a file ending with .json, triggering the syntax highlighting.

JSON.vim is a syntax highlighter for JSON code, but the install instructions are a little light on detail.

Other blog posts tell you to explicitly source JSON.vim from some explicit path. This will work for one computer, but will fail if you like to use your vimfiles in different directories on more than one computer. (e.g. my vimfiles are in ~/.vim/ on unix, and ~/vimfiles on windows)

NZ Phone Numbers

2010-02-17T00:00:00+11:00

Setting up a VoIP router today, I came across an interesting setting: ‘Dial Plan 1’.

(*xx|#xx|1xx|082210S0|[2-9]xxxxxx|0[3469]xxxxxxxS0|0800xxxxxxS0|00[1-9]x.S3|0[12578]x.S3|0900xxxxxxS0)

That looks suspiciously like a regular expression, matching New Zealand phone numbers! Cool! Let’s analyze it. I love a puzzle.

Quick Regex Primer

[abc123] matches any one of the symbols inside the bracket. Called a ‘character class’.
[1-9] is the same as [123456789].
(pattern1|pattern2|pattern3) is the basic OR of regular expressions: match pattern1 OR pattern2 OR pattern3. So in this expression, it looks like there are about 10 different types of general phone numbers.
In this regex flavor, it looks like x is used as a stand-in for any number: a shorter way of saying [0-9].

Splitting the Regex

*xx
Usually, * in a regular expression means “match any number of the previous token”. Here, * is the first token, so it can’t mean that. I’m guessing that it literally means the * button on the phone, followed by two numbers.
#xx
Hash, followed by two numbers.
1xx
One, followed by two numbers. Example: 111 (emergency services)
082210S0
Probably my ISP’s voicemail service. I wouldn’t know: I forgot the PIN number shortly after signing up and haven’t used voicemail since :)
[2-9]xxxxxx
In New Zealand, a local number is dialed with 7 numbers. I never knew that they weren’t allowed to start with a 1.
0[3469]xxxxxxxS0
0 is the national access code. If a number starts with 03, 04, 06, or 09, it is a national call. Notably absent is 07 the area code for where I live.
0800xxxxxxS0
0800 numbers are national freecall hotlines.
00[1-9]x.S3
00 is the international access code. It’s strange that there’s only one x here, as there will probably be many more numbers after the country code. e.g. to call the US, you need to dial 00 for international, 1 for US, then the seven-digit US phone number.
0[12578]x.S3
Aha! Here are the other area codes. These are invalid numbers - these area codes don’t exist in New Zealand, except for 07 - my area code, which is an error if you use it for a local call.
0900xxxxxxS0
0900 numbers: They charge you by the minute to call them.

Unsolved Mysteries

What is the purpose of this regex?
S0 and S3 suffixes at the end some regexes. What do they mean?
What do . dots mean?
Why is there nothing distinguishing the valid numbers from the invalid 0[12578]x.S3] form?

This format is probably documented somewhere. This regex was found on a Linksys AG310 ADSL2+ Gateway with VoIP. But I know from experience that searching through a supplier’s website for documentation is usually a horrible experience, so I think I’ll let these mysteries lie. :)

Git inside Git

2010-02-11T00:00:00+11:00

UPDATE 2011: I don’t use this technique any more, because Git submodules kinda suck. Instead, I use felixge’s technique.

How do you keep another Git repository inside a folder that’s already under version control with Git? Why would you do this?

I keep my vim config files under revision control, and I tried cloning other people’s vim modules from github into subdirectories of my repository.
I use underscore.js inside some of my javascript projects. Both are kept in git.

If you git clone, git will happily comply and clone the repository into your folder, but you won’t be able to add the .git folder into revision control. If you try to git add subrepository/.git, your request will be ignored, as if the .git folder doesn’t exist at all:

git add subrepository/.git
fatal: pathspec 'subrepository.git' did not match any files

And that’s just as well - keeping track of a repository’s history inside another repository would be horribly wasteful. And every time the subrepository updates, the parent repository will need to update both the working directory files and the subrepository history files, complicating the diffs. And it’s just not necessary: git has a great way of keeping git repositories inside git.

Submodules

Submodules are a way of importing code from other git repositories into your source tree, and keeping them up to date after they are imported.

cd my_git_repo_root
git submodule add <remote repository> <local path>

This will checkout the remote repository into the local path, and records the remote repository in a .gitmodules at the root of your repo.

Then, later on, if a new version of a submodule is released, you can update your copy with

git submodule update

That’s the basics. For more info:

Don't debug with JSON.Stringify

2010-02-10T00:00:00+11:00

I’ve been having some problems using JSON.Stringify to inspect objects while debugging inside node.js. It seems like a great choice:

var obj = { 
    "prop1" : "value1", 
    "prop2" : 5
}
print(JSON.stringify(obj));
//prints: {"prop1":"value1","prop2":5}

But there are gotchas:

var infinity = 1 / 0;
var func = function() {};
func.innerproperty = "value";
var obj = {
    "prop1" : func,
    "prop2" : undefined,
    "prop3" : infinity
}
print(JSON.stringify(obj);
//prints {"prop3":null}

Oh no! JSON.Stringify has

ignored prop1, a function
ignored func.innerproperty
ignored prop2, an undefined value
replaced prop3’s Infinity with null.

Yikes! This is because JSON, being a pure data interchange format, has no notation for functions. Or undefined values. And it replaces nonfinite numbers with null.

In node.js, there’s a dedicated function for stringifying objects for debuggings: sys.inspect(object). See the API docs. This fares somewhat better for the last example, outputting:

{
 "prop1": [Function],
 "prop2": undefined,
 "prop3": Infinity
}

~~However, note that it doesn't check for function properties. `prop1.innerproperty` is ignored.~~

EDIT: I opened an issue, and it was fixed in less than a day. Functions with properties are now listed too. Thanks, creationix!

This is pretty important, because it’s very common to use functions as property holders in popular javascript libraries.

jQuery uses $ both as a selector function, and a library object
underscore.js uses _ as a wrapper function, and a library object

Benchmark your DNS!

2010-02-08T00:00:00+11:00

Every time you browse to a website, your computer needs to translate the human-readable URL (www.google.com) into an IP (66.102.7.103). It does this by asking your DNS servers. The speed of your DNS server can really impact the speed of your internet browsing.

There’s a lot of choices for DNS servers. Chances are you’re just using your ISP’s one, and this may not be the fastest. I was using my ISP’s DNS servers, and it turns out that on average another ISP’s DNS servers were over twice as fast - from 258ms down to 112ms!

You can benchmark your DNS servers with namebench, a free tool for Windows, Mac, and UNIX. Namebench is smart - it pulls data from your browser history to ensure your DNS is optimised for the sites you frequent. Namebench checks the best DNS servers in your region, as well as global services like Google Public DNS, OpenDNS, and UltraDNS. Namebench even checks for incorrect results from DNS Servers!

I’m very impressed with how simple, smart and effective namebench is. Kudos to the developers! Thanks for the 50% reduction in DNS wait times.

Mouse Hiding in MinTTY

2010-02-03T00:00:00+11:00

I love MinTTY - it’s the best terminal emulator I know of for Windows. It’s so much better than the default cmd.exe window that Cygwin uses by default. You can’t resize cmd.exe, and has almost no scrollback.

But I’ve been having an intermittent problem with it - my mouse cursor keeps disappearing. I move my mouse, hoping to find it, but it’s invisible until I click somewhere (and I can’t see where I’m clicking). It’s panicing!

Today I did some tests - the problem occurs when you move the mouse outside the MinTTY window, and then type. The mouse disappears, and doesn’t reappear until it moves back into the MinTTY window.

So I looked through the MinTTY source code to see what’s causing this. That’s easy enough. It’s open source, and hosted at Google Code.

svn checkout http://mintty.googlecode.com/svn/trunk/ mintty-read-only

The source code is rather small, for a program written in C. With a little help from grep and ctags it was easy to find the bit concerning mouse input - in wininput.c.

The mouse is hidden whenever a keypress notification arrives. Keypress notifications arrive when the mouse is both inside and outisde the window. The mouse is shown whenever a mouse click/release/move notification arrives. Mouse notifications are only delivered when the mouse is inside the window area.

So it seems the mouse can be hidden by a keypress event while it is outside the window, but it won’t reappear, even if you move the mouse, until it reenters the window. Or you click onto another program.

There’s a simple solution: check if the mouse is inside the window before hiding it. I looked up a few Windows API calls, found the ones I needed were GetWindowRect() and GetCursorPos(). I made the change, recompiled, tested, and it works!

I checked the edge cases (literal edge cases - the four window edges), and it even works there: I was worried that moving from just inside the window to outside wouldn’t send a mouse-move notification to the window, but happily it does.

So, here it is - my first submitted patch to an open-source project. It’s not much, but it’s a start. :)

static bool
mouse_inside_window() {
    RECT w; POINT m;
    GetWindowRect(wnd, &w);
    GetCursorPos(&m);
    return
        (w.left <= m.x && m.x < w.right) &&
        (w.top <= m.y && m.y < w.bottom);
}

void
hide_mouse()
{
  if (mouse_showing && mouse_inside_window()) {
    ShowCursor(false);
    mouse_showing = false;
  }
}

UPDATE: The MinTTY maintainer Andy Koppe was totally awesome helping with my problem, but he can’t reproduce the problem at his end, and after I restarted my computer I can’t reproduce it either.

However, the problem has been troubling me for a few months, so I’m sure it will be back. And when it does, I’ll be ready. ;)

Git in Color

2010-02-01T00:00:00+11:00

By default, git doesn’t use color. However, you can enable color for interactive use:

$ git config --global --add color.ui true

This adds a line to your ~/.gitconfig file, enabling colors output for all git operations on all repositories.

Awesome! Now git-diff will shine in radiant reds and greens, and it’ll be oh-so-easy to see what’s changed. git-branch will show your current branch in green, git-status will show added files in green and deleted files in red.

And git’s smart enough to check if the output is going to the terminal or to a pipe, where color codes could break things. Git only sends color output to the terminal, where your eyes can enjoy it. I can’t see any reason not to enable color in git. Enjoy!

Optional parameters in Javascript

2010-01-19T00:00:00+11:00

I’ve been reading a lot of JavaScript code lately, and it’s been a joy. I’m picking up on some common JavaScript idioms too, and I thought I’d share.

Introduction to Optional Parameters

Optional parameters are a nice language feature - function parameters that are given default values if not used when calling a function. Optional parameters are great for simplifying code, and hiding advanced but not-often-used functionality. If 90% of the time you’re calling a function using the same values for some parameters, you should look into making those parameters optional to avoid Repeating Yourself.

Compare this C# code for connecting to a server, first without optional parameters:

public void connect() 
    : connect("localhost") {}

public void connect(string hostname) 
    : connect(hostname, 80) {} 

public void connect(string hostname, int port) 
    : connect(hostname, port, "HTTP") {}

public void connect(string hostname, int port, string method); { 
    ... 
}

Now, with the new C# 4.0 optional parameters, things are a lot simpler:

public void connect(string hostname = "localhost", 
                    int port = 80, string method = "HTTP") { ... }

Method 1: Undefined arguments

At first glance, javascript has nothing like this available. However, javascript lets you call functions omitting some parameters, filling in the other parameters with the value undefined.

connect("www.google.com");
function connect(hostname, port, method) {
     // inside the function, 
     // hostname === "www.google.com", 
     // port === undefined,
     // method === undefined 
}

It’s easy to check if a value is undefined and fill it in with a default parameter if it is:

function connect(hostname, port, method) {
    if (hostname === undefined) hostname = "localhost";
    if (port === undefined) port = 80;
    if (method === undefined) method = "HTTP";
}

However, there’s a prettier shortcut:

function connect(hostname, port, method) {
    hostname = hostname || "localhost";
    port = port || 80;
    method = method || "GET";
}

The short-circuit OR operator || returns the left side if the left argument is truthy (evaluates to true in conditionals), otherwise it checks if the right argument is truthy, returning it. We can use this shortcut because undefined is falsy: in conditionals, undefined evaluates to false.

This shortcut approach is a very common idiom, but it does have a disadvantage: You can’t use for any argument that could accept a falsy value: false, 0, null, undefined, the empty string "", and NaN.

Using the || shortcut will override any falsy input value. If you expect a falsy value, you must explicitly check for argument === undefined.

This method only allows the last arguments to be optional - you cannot make an optional first parameter, middle parameter, or combination of parameters optional. The next methods let you position optional arguments anywhere.

Method 2: The arguments variable

All javascript functions get passed an implicit arguments variable when they’re called. arguments is an array containing the values of all the arguments passed to the function. Here’s a function that takes any arguments and alert()’s them:

alertArgs("arg1", 12.3, true, "arg4");
function alertArgs() {
    alert(JSON.stringify(arguments));
    /* alerts '[ "arg1", 12.3, true, "arg4" ]' */
}

function connect() {
    hostname = arguments[0] || "localhost";
    port = port arguments[1] || 80;
    method = arguments[2] || "GET";
}
/* example: do a GET request to www.example.com on port 80 */
connect("www.example.com");

Method 3: The object literal

Javascript has a cheap and easy object literal syntax, so why not use this to make flexible and readable optional arguments?

function connect(hostname, options) {
    var options = options || {};
    var port = options.port || 80;
    var method = options.method || "GET";
}
connect("www.example.com", { port: 8080, method: "POST" });
connect("www.google.com");

Note that you need to ensure that an object is passed into options, replace it with an empty object {}, or you will get errors trying to read properties from undefined.

A combination of all three methods

Here’s an example, from node.js’s process.watchFile(filename, options, listener) function. It’s used to watch filename for changes, calling listener when it does. In this function, the options argument is optional:

process.watchFile = function (filename) {
  var options;
  var listener;

  if (typeof arguments[1] == "object") {
    options = arguments[1];
    listener = arguments[2];
  } else {
    options = {};
    listener = arguments[1];
  }

  if (options.persistent === undefined) options.persistent = true;
  if (options.interval === undefined) options.interval = 0;
  
  /* SNIP: main function logic */
}

/* example: watch file '/etc/passwd' for changes */
process.watchFile('/etc/passwd', function() { 
   log("passwd file changed");
}

Peter Norvig's "Design Patterns in Dynamic Programming"

2010-01-15T00:00:00+11:00

Today I read something that’s been sitting in my bookmarks for a few months now: Peter Norvig’s 1996 slides on Design Patterns in Dynamic Programming.

I’ve been toying around with dynamic languages a lot lately, and I’ve been excited at the flexibility and simplicity offered by features like first-class functions and types. These features are noticeably absent from traditional languages like Java and C++.

It was a bit tricky to understand the slides where the examples are written in Dylan, a language I’d never heard of before today. Dylan is a very Ruby-like language developed in the early 1990s by Apple.

The slides make a good case for using languages with first-class types and first class functions (i.e. you can pass functions and types as variables at runtime). Common patterns like Factory, Strategy, and Observer become simpler. Here’s a few examples:

Strategy Pattern

You don’t need full-blown classes just to hold a function/routine. Just pass the function!

var bird = { 
    observers: [] 
};
var birdObserver = { 
    onBirdChange: function() { 
        alert("The Bird Moved!"); 
    } 
};
bird.observers.push(function() { 
    birdObserver.onBirdChange();
});

Factory Pattern

The Factory Pattern defers deciding the type of an object instatiation till runtime. This is trivial with first-class types:

var carType = getTypeFromConfigFile();
var car = carType.createCar();

Observer Pattern

The Observer pattern calls a function somewhere when an object changes. Of course, this gets simpler when functions are first-class. Without first-class functions, observers need to inherit a common base class. With first-class functions, you can maintain a list of observers as a list of functions, and one object can observe many others easily.

Changing the Minix3 Bootimage

2009-11-05T00:00:00+11:00

(or: how to add a device driver to Minix3 without bricking your bootimage)

In this tutorial, we will duplicate the Minix memory device driver. The memory driver is a very small driver, and can serve as a simple working implementation of the driver interface. Once we have duplicated the driver, we will test it by modifying it, adding a new minor device to it. This driver will retrieve the system uptime. Note: I will be using Minix 3.1.4 for the examples.

Background

When you read from a file in Minix3, execution passes through the file system and device driver:

read(fd, &buf, bytes) call from a user-space program (e.g. /bin/cat) passes a message to the filesystem (/usr/src/servers/vfs)
the filesystem dispatches to a device driver, based on its table mapping major device numbers to device driver process numbers in /usr/src/servers/vfs/dmap.c.
The device driver recieves a message from the filesystem, and performs the read call.

We must add information about the driver to files in /usr/src/:

drivers/Makefile - organises compilation of drivers
kernel/table.c - this defines the programs in the bootimage
tools/Makefile - defines the location of files in the bootimage
include/minix/com.h - defines process numbers of drivers
servers/vfs/dmap.c, - defines device major-number -> device process number mappings

1: Duplicate the memory driver

cd /usr/src/drivers/
cp -r memory mydriver

Now, add a line to the /usr/src/drivers/Makefile, in the all install depend clean: section, telling it to compile whatever’s in the mydriver directory. Put it in just before the memory driver, as the memory driver must be last for some reason.

cd ./mydriver && $(MAKE) $@

Also, we want our driver to run when the computer boots - we want it in the bootimage. So add a line to the bootimage section of the makefile, again, just before the memory driver:

cd ./mydriver && $(MAKE) build

You can test that this works by observing the make program enter your mydriver directory:

cd /usr/src/drivers
make
make image

2: Add the driver to the bootimage

This is a big step, and if something goes wrong, your new system image will not boot. Don’t worry! Minix gives you the option to boot into the original, working image instead, if something goes wrong.

First, give your new driver a process number. Edit /usr/src/include/minix/com.h, and find where INIT_PROC_NR is defined. These task numbers are indexes into the bootimage table defined in kernel/table.c, and INIT must be last. So increment INIT_PROC_NR and insert MYDRIVER_PROC_NR just before INIT

#define MYDRIVER_PROC_NR    9
#define INIT_PROC_NR        10

Now, edit /usr/src/kernel/table.c, and add in the new driver just before INIT in the process table, again. The last three lines of my bootimage image[] look like:

{VM_PROC_NR,        0,SRV_F, 32,      2, 0,  SRV_T, SRV_M, c(vm_c), "vm"    },
{MYDRIVER_PROC_NR,  0,SVM_F,  4,      3, 0,  SRV_T, SYS_M, c(mem_c),"mydriver"},
{INIT_PROC_NR,      0,USR_F,  8, USER_Q, 0,  USR_T, USR_M, no_c,    "init"  },

I just copied the fields from the memory device’s line, changing the process number and name. The order is important - it must be consistent throughout the files dealing with the bootimage (kernel/table.c, tools/Makefile, and include/minix/com.h), or your system will not boot.

Next, edit /usr/src/tools/Makefile. This file specifies the parameters passed to the program that constructs the bootimage, in the PROGRAMS variable. Add a line to this, just before ../servers/init/init:

../drivers/mydriver/memory

memory is the name of the driver program produced by the Makefile in mydriver. You’ll probably want to change this at some point. When you do, change the reference in tools/Makefile too.

The driver’s now added to the bootimage. Try recompiling the system, and rebooting into the new boot image.

   
cd /usr/src/tools
make install
shutdown

If all goes well, your system should reboot happily, and as the kernel starts, listing the programs in the bootimage, you will see mydriver before init. If not, try booting into the old (working) bootimage, and run a make fresh install from /usr/src/tools, which will recompile everything from source (and take a long time).

Excellent! There are now two memory drivers running. However, we can’t use the new one - the filesystem doesn’t know about it yet - it doesn’t have a major device number.

3. Add the device to the File Server

Edit /usr/src/servers/vfs/dmap.c. In this file, there is a struct dmap dmap containing a mapping from major device number to process number, and other info. Find a major device number (index in the table) that is marked ‘not used’ (on my system, #9 was not used), and replace it with data copied from the memory device driver above, changing MEM_PROC_NR to MYDRIVER_PROC_NR and "memory" to "mydriver".

DT(1, gen_opcl, gen_io, MYDRIVER_PROC_NR, 0, "mydriver")

Now, calls to that major device number will dispatch to the handlers defined in /usr/src/drivers/mydriver. You can modify these routines to execute arbitrary code, and return anything.

To make a special file to access this new driver, use mknod to make a character special file:

mknod myspecialfile c [major number] [minor number]

You can then access your driver through this special file, as you would any other file.

cat myspecialfile

Now you can modify your second memory driver to return whatever you want on a read() call.

Dodgy Pizza

2009-10-24T00:00:00+11:00

With some friends last night, I got some pizza. As the pizza delivery boy answered the door, he asked to see the credit card we had used to order the pizza, remarking “We have had 70 cases of credit card fraud”. This, from a store that just opened two weeks ago, is crazy. There must be a reason why Hells Pizza is getting so much fraud!

Quick! What’s wrong with this picture?

Here’s a clue:

See that link “Click here to verify merchant details”? Here’s what happens when I click it:

Hells Pizza’s website:

Uses unsecured HTTP (plaintext transmission!)
Has no certificate to ensure I’m actually at the real hells pizza website
Has a link to a validation website telling me their website is invalid.

These are serious problems for a company taking their customer’s credit card number over the internet.

Unsecured HTTP

This means everything is transmitted in plain text. Anyone on the connection between my computer and hell.co.nz can watch it, inspect it, or even change it. If they’re transmitting my credit card number like this, it’s also against VISA’s terms and conditions.

No certificate

Certificates say “A trusted authority has vouched that this is indeed the real website of hell.co.nz, and not some imposter”. Since there is no certificate, I have no way of knowing that I’m looking at the real Hells Pizza website! It could be running off a completely different server, anywhere in the world and I’d have no idea that I was actually giving my credit card number to someone in, say, Nigeria.

Sounds farfetched? There are many ways I could be redirected to a different server:

It’s being transmitted over unsecured HTTP, so anyone between my computer and hell’s pizza’s server could intercept the communication, reply as if they were hell’s server, and issue a redirect request to their server.
DNS cache poisoning. Any of your DNS caches could be poisoned to redirect www.hell.co.nz to a fake host. Your browser, your operating system, your router, and your ISP all operate their own DNS caches. Any of these could be poisoned.
HOSTS file poisoning. Every time your computer visits a website, it checks the hosts file. And if the hosts file says “hellspizza.co.nz is at this ip address”, my request (and credit card number) will be sent to that ip address.
Hell’s Pizza forgets to renew their domain name, and someone else snaps it up.

Sniffing

I popped open my favorite network sniffer Wireshark to have a closer look at the network traffic leaving my computer, and whether it was encrypted when I sent my credit card number.

The whole website is done as a Flash application, and internally, the application does use HTTPS to transmit credit card numbers. This means that not I, nor anybody else on the network, can listen in on the credit card numbers in transit.

Conclusion

For now, there’s no way for me to know if I’m dealing with the real Hell’s Pizza website when I’m entering my credit card details.

I know it’s encrypting my credit card details before sending them, but I can’t be sure who it’s sending them to. This is not good enough for a website handling customer’s credit card numbers.

Hell’s Pizza needs to:

serve up their content using Secure HTTP, so that nobody can alter their website in the network.
get a certificate for their website, so that I know I’m looking at the real Hell’s Pizza website.
Investigate why their paymark ‘site validation’ link tells me the site is Not Valid.

Till then, I know who I’ll be ordering from:

Private methods and variables in Javascript

2009-08-24T00:00:00+10:00

Till yesterday, I believed there was no way to encapsulate private methods and variables in javascript:

There are no ‘private’ or ‘public’ keywords, like in Java or C#. There are no classes as such, just objects inheriting directly from other objects. And all fields on any object are public.

But, it turns out there is a way to make private members in javascript, using closures.

What’s a closure?

If an outer function returns an inner function, the inner function still has access to the variables of the outer function, even after the outer function has returned.

function outerFunction(id) {
    var outerFunctionVariable = id;
    return function innerFunction() {
        return outerFunctionVariable;
    }
}
 
var a = outerFunction(5);
a() //returns 5

We are calling the inner function after the outer function has returned, but the inner function still has access to the outer function’s methods. Magic!

How can you access the local variables of a function after it has returned, the stack frame will be popped right off and the variables will be lost, right?

Not necessarily. When you use a closure, the stack frame is not popped off. It’s saved somewhere safe, and inner functions can still get at it.

Morris Johns has some excellent examples illustrating this.

Here’s an example of a plain vanilla javascript object.

var car = {
    "odometer" : 100000,
    "drive" : function(distance) { 
        if (distance > 0) this.odomoter += distance; 
    }
};

I have a drive function that increases the odometer, and guards against winding it back. Except odometer is a public member, so anyone can change it. Uh-oh.

Functions in javascript create a new scope, so we can ‘hide’ odometer inside a function. We still need to read the odometer reading, so we can return a function getOdometer(). Through closure, the inner function getOdometer() has access to the odometer member of the outer function makeCar().

function makeCar(initialOdometer) {
    var privateOdometer = initialOdometer;
    return {
        "getOdometer" : function() { 
            return privateOdometer;
        }
        "drive" : function(distance) {
            if (distance > 0) privateOdometer += distance;
        }
    };
}

(Note: We are not using this to access the odometer reading any more. this is not needed - we have a direct reference to privateOdometer in the closure)

We can then instantiate a really old car, setting its private odometer:

var reallyOldCar = makeCar(180000);

We can’t get or set the odometer directly: there’s no privateOdometer field on the object we returned:

reallyOldCar.privateOdometer; //undefined

However we can use getOdometer()

reallyOldCar.getOdometer(); //180000

We can drive the car halfway around the world, and recheck the odometer.

reallyOldCar.drive(20000); 
reallyOldCar.getOdometer(); //200000

Then we can complete the journey, circling the world.

reallyOldCar.drive(20000); 
reallyOldCar.getOdometer(); //220000

Brilliant! Private variables in javascript. You can apply exactly the same technique for private methods.

props to Douglas Crockford for explaining this pattern.