GFX::Monk

Cool projects I learnt about at lca2012

2012-01-21T00:00:00+11:00

I went to linux.conf.au this week, and learnt about some pretty awesome tech (as well as hearing some entertaining, inspiring, touching and terrifying talks from the likes of Paul Fenwick, Bruce Perens, Karen Sandler and Jacob Appelbaum).

So here’s a quick dump of cool projects I learnt about, with links where I could find them.

browser id:

Openid is not so great, due to:

usability / confusion (requiring a url as an id)
reliability (if your provider goes down, you can’t log in)
lock in, (hard for the user to migrate providers)
privacy (your provider knows every url you log into, every time)

Browserid serves one simple purpose: to prove you own an email. It’s distributed. The browser, (not a third-party server) is the login intermediary.

As an example: The login process to gmail generates a shortlived (in the order of hours or maybe days, I guess) cryptographically signed statement that you own that email, which your browser stores. Other sites just need to grab gmail’s public key and then they can themselves verify that the assertion you sent them proves you own (or can log in to) you@example.com.

Demo site: myfavoritebeer.org

Not awesome yet: There’s no browser or email provider support. But it’s developed by mozilla, has scaffolding in js to work already in all browsers. Certs are stored on browserid.org for now, until browsers implement native support (so they are just as bad as a server intermediary, but only as a stopgap).

mediastreams processing api:

Very cool demos, which are online (although they require a dev build of firefox).

css calc()

Computation in css. Very cool. There are implementations in IE and Firefox so far, and work on webkit is in progress. This is not just a convenience like SASS and friends - it allows for previously impossible mixing of units like 100% - 10px.

News on arbitrary metadata in git:

Apparently many people feel this is important for VCS interop (git-svn, fastimport, etc), and there are proposals for adding support (but it’s difficult to figure out how to do right in a way that won’t prohibit other useful things in the future).

openstack

Uses orchestra by ubuntu for deployments.

gerrit for code review. Use Jenkins plugins, pushes state notifications to launchpad. openstack wrote a patch submission tool: git-review. Rebases & pushes patchset to gerrit.

testing ctdb

Autocluster is a tool for testing clustered samba (and other things, presumably). Virtual kvm based clusters. Uses guestfish to manage volumes inside the kvm guests.

dbench for automated performance tests. Can describe any kind of IO workload.

Assessing the potential impact of package updates

Tools for spelunking package relationships (on debian at least): - Recursive depends, reverse depends, apt-cache, germinate.

misc projects / pointers

sozi is an inkscape plugin for chopping up vectors into slide presentations.

fossil scm is a VCS that tracks docs, wikis, bugs, etc in with the source code. Cool idea, apparently some bad implementation decisions though.

safe-rm: replaces rm and has a blacklist of folders you probably don’t really want to remove, such as /usr/. No more bumblebeeing (can’t find the link, but there was once a bug in the bumblebee uninstall script that removed all of /usr as root).

handbag makes android accessory dev easy(er)

libvirt-sandbox: new library and command-line tools for app-level sandboxing with LCX and/or KVM. Should be coming in Fedora 17.

Vsualisation: gapminder.org

freedombox: 100% free tiny personal server project. Uses all open-source & federated social software like diaspora, buddycloud

instamorph: malleable plastic that’s solid at room temperature, for making ad-hoc hooks, connectors, docks, etc. The aussie equivalent is apparently called “Polymorph”.

Mentally ignoring files in git

2011-11-15T00:00:00+11:00

For unfortunate reasons, at work we have intellij module files that are:

checked into git (because they are useful and hard to recreate)
changing all the time for no good reason (because intellij is a bit like that)

For the most part, the changes are just noise because paths, versions, etc. differ slightly across dev machines. Sometimes the differences are completely meaningless (unordered elements in a file being rearranged). But sometimes we do want to check in new versions of these files, say when we add a new module.

Since they are checked in, we can’t just add them to .gitignore. But we can still mentally ignore them, with this rather ridiculous hack:

$ git status | sed -E -e 's/\x1b\[[0-9][12]m(.*\.iml)/'"`tput setaf 3`"'\1/'

This changes the normally red (or green, or blue) lines in the git status output to yellow instead when they refer to an intellij .iml file. It’s completely hacky, but it’s better than breaking the build because you forgot to commit a file amongst all the noise.

You can put this in a script you call instead of git status - writing git status takes too long anyway, I just call my script g.

You’ll also need to make sure you have enabled colour “always” in git status output, otherwise you’ll get no colours at all:

$ git config --global color.status always

If anyone has a better way of ignoring files while having them checked in, I’d love to hear it. Do other VCSs allow you to deal with this any better?

Obligate.js

2011-09-11T00:00:00+10:00

I’ve been doing some browser-side javascript lately, and getting frustrated at the mess that is browser-side modules.. So here’s a tiny library that, given a tree containing javascript files, will give you a single javascript file containing all of the modules, as well as a commonJS require method to use in browser-side code. No more accidental globals, your variables are local and you just add your module’s public interface to properties on the module-local exports object.

Of course, it also includes a tool to grab all the javascript code of your own and that of your recursive dependencies, specified in a zero install feed.

(view link)

Ruby's split() function makes me feel special (in a bad way)

2011-09-04T00:00:00+10:00

Quick hand count: who knows what String.split() does?

Most developers probably do. Python? easy. Javascript? probably. But if you’re a ruby developer, chances are close to nil. I’m not trying to imply anything about the intelligence or skill of ruby developers, it’s just that the odds are stacked against you.

So, what does `String.split()` do?

In the simple case, it takes a separator string. It returns an array of substrings, split on the given string. Like so:

py> "one|two|three".split("|")
["one", "two", "three"]

Simple enough. As an extension, some languages allow you to pass in a num_splits option. In python, it splits only this many times, like so:

py> "one|two|three".split("|", 1)
["one", "two|three"]

Ruby is similar, although you have to add one to the second argument (it talks about number of returned components, rather than number of splits performed).

Javascript is a bit odd, in that it will ignore the rest of the string if you limit it:

js> "one|two|three".split("|", 2)
["one", "two"]

I don’t like the javascript way, but these are all valid interpretations of split. So far. And that’s pretty much all you have to know for python and javascript. But ruby? Pull up a seat.

Special cases.

a.k.a surprises.

irb> "one||two||three".split("|")
=> ["one", "", "two", "", "three"]

Seems reasonable. But:

irb> "one  two  three".split(" ")
=> ["one", "two", "three"]

Note there are double spaces in the input - ruby’s split has different behaviour if the separator is a single space character!

But if you don’t like that, you can use a regex!

irb> "one  two  three".split(/ /)
=> ["one", "", "two", "", "three"]

Theoretically this has the same meaning, but you get different behaviour!

The stupid thing about this example is that there is absolutely no need to treat ' ' as a special case - you can achieve the same behaviour explicitly if you actually want it, by using /s+/ as the separator.

And how about:

irb> "||one||two||three||".split("|")
=> ["", "", "one", "", "two", "", "three"]

ruby’s split treats leading blanks differently to trailing blanks.

unless you include a negative number as the num_splits argument - in that case it has no effect but its sign changes the behaviour of split to include trailing blanks:

irb> "||one||two||three||".split("|", -2)
=> ["", "", "one", "", "two", "", "three", "", ""]

(note that there is no way to enable this behaviour and limit the number of splits performed).

I can see no useful purpose for this special case. And again, if I actually wanted this behaviour, it would be easy enough to implement. In fact I would still implement it in order to correctly convey intent instead of relying on poorly-known edge cases of the ruby standard library.

By the way, these are not accidents - they are documented.

For bonus points, the two special cases outlined here can interact in fun ways. I didn’t mention it above, but splitting on a single space also ignores leading and trailing whitespace (also known as strip, if you actually want that behaviour). If you wanted to preserve those, you might try the negative number trick from above. But what does that give you?

irb> "  a  b  c  ".split(" ", -1)
=> ["a", "b", "c", ""]

What a mess.

So, have you got all that? Are you going to remember it next time you use split? I doubt it (I just wrote about it in depth, and I’ll still probably forget). And don’t forget to think about how these special cases could interact with the myriad of other special cases and surprising behaviour in ruby.

You might think that I’m just picking on ruby - this could happen in any language. But one of the reasons I like python so much is that not only is the language itself simple and elegant, but the community largely agrees upon some very fundamental ideas - the zen of python. These are more guiding principles than laws, but they are usually respected unless you have a good reason not to (and you probably don’t).

These points in particular all seem pretty relevant:

Simple is better than complex.
Special cases aren’t special enough to break the rules.
If the implementation is hard to explain, it’s a bad idea.

But I guess if you disagree with those, ruby might well be for you.

Node.js child processes

2011-09-02T00:00:00+10:00

me: Hey node, how can I tell if my child_process.spawn() call failed before it even runs the desired command? Is there a return value? An exception? Maybe there’s an error event I can listen for?

node: Even better! Here’s an example from the documentation to do exactly that:

var spawn = require('child_process').spawn,
    child = spawn('bad_command');

child.stderr.setEncoding('utf8');
child.stderr.on('data', function (data) {
  if (/^execvp\(\)/.test(data)) {
    console.log('Failed to start child process.');
  }
});

me: Thanks, that’s… That’s just stunning.

(For extra points, this doesn’t even work if the child process inherits stderr, as my actual code does. Is this just a silly example from the documentation, or is there really nothing better?)

Stereoscoper and the Depth of Awesomeness

2011-08-29T00:00:00+10:00

A few days ago I got a shiny new toy: a 3d camera from thinkgeek (I’d link to it, but it seems to have disappeared from their catalogue). I’m a massive fan of 3d photos / video, so it’s pretty cool to have a device that allows me to take stereoscopic photo pairs simultaneously (you can do it manually with a static scene, but those get boring).

Sadly (although not surprisingly), the quality is not great. The limited resolution is not really an issue given how you’re likely to view them, but the pictures come out awkwardly stretched to half the expected horizontal resolution. They are also pre-combined in a single JPEG, they are the way around for cross-eyed viewing, and the colour balance is frequently off between the two sensors (which can be really jarring).

Seeing a lot of manual photo fixing in my future, I set out to automate it. And thus stereoscoper was born, as a way to bulk-convert stereo images to other formats. Aside from the obvious geometry changes (the horizontal resolution and image placement), I also learnt all about histogram matching in order to make the colour balance consistent across stereo pairs. And in order to make animated gifs that match up nicely, there’s even an interactive mode where you can fine-tune the alignment of the image pairs.

In the wiggly-animated spirit of 3ERD (note: some images there are NSFW), here’s some fun we had in the park with my new toy:

(click to toggle each animation. It’s off by default to save your brain from having a fit ;)

(stereo)

Update: Click the (stereo) link under each image for a cross-eyed viewing version.

Experiment: Using Zero-Install as a Plugin Manager

2011-08-02T00:00:00+10:00

So for a little while now I’ve been wanting to try an experiment, using Zero Install as a Plugin Manager. Some background:

Zero install is designed to be used for distributing applications and libraries.
One of its greatest strengths is that it has no global state, and doesn’t require elevated privileges (i.e root) to run a program (and as the name implies, there is no installation to speak of).

Since there’s no global state, it seems possible to use this as a dependency manager within a program - also known as a plugin system.

Happily, it turns out it’s not so difficult. There are a few pieces required to set it up, but for the most part they are reusable - and much simpler to implement (and more flexible) than most home-rolled plugin systems.

(note: this article makes use of features in zero install version 0.52. If you are trying to run the examples, you will need that version or later. Fortunately, 0install can be easily bootstrapped by running 0launch http://0install.net/2007/interfaces/ZeroInstall.xml <interface>)

Note to program authors: This article is an explanation and rationale of the techniques I’ve used to manage plugins with zeroinstall. If you simply want to enable plugins for your own zeroinstall feed, you may want to skip to the documentation for zeroinstall-plugin-manager.

Background:

nosetests is an extensible test runner for python. I think it’s great, and have written a number of plugins for it. It has a built-in plugin system that relies on setuptools entry points, which annoys me because it means I have to package and install my plugins using setuptools, which is not a very good dependency manager for at least these reasons:

it only works for python
packages are installed globally, and there can be only one “active” version¹
it can’t uninstall anything (this is pretty rubbish)

That’s why I want to use 0install to publish and manage nosetests plugins.

The basics - every user gets their own feed:

A zero install feed has two important parts: The implementation, and the dependencies. For nosetests itself, it doesn’t really have dependencies. But if you consider a user’s own set of preferred plugins, that’s really just a list of packages that you want nosetets to know about. Nosetests isn’t directly dependent upon them, but the conceptual package of “Tim’s preferred nosetests plugins” does.

So we’ll start with a local 0install feed to represent the user’s preferred set of plugins. This allows us to list nosetests as the “runner”, while depending on whatever additional plugins we want nosetests to know about. This will be specific to the user, and will never be published (it doesn’t even need a URI).

Here’s the relevant excerpt from my own such feed:

<group>
  <command name="run">
    <runner interface="http://gfxmonk.net/dist/0install/nosetests-plugin-resolver.xml"/>
  </command>
  <requires interface="http://gfxmonk.net/dist/0install/rednose.xml"/>
</group>

That just says “use the nosetests-plugin-resolver to run this feed, and make sure the rednose plugin is made available”. Observant readers will notice that I originally said we could use nosetests itself as a runner, so what’s this plugin resolver thing about?

Making plugins known to nosetests at runtime

Setuptools provides an entry point mechanism for python programs to discover plugins that have registered an implementation for certain well-known entry points. This is done using a setuptools-specific API, and mandates that all plugins must also use setuptools. In other words, we can’t re-use that. But it’s not too hard to make our own.

Zero Install allows packages one main way of allowing packages to know about other packages: environment variables. It’s a simple mechanism, and works well with existing systems - for example, if you write a python library, you just have to add it to $PYTHONPATH. If you want something to be runnable, you add it to $PATH.

So all we need to do is:

Put our plugin’s name into an environment variable
Make a plugin loader that can use this environment variable to load plugins (this takes the place of setuptools’ entry point loader)

The first one is trivial. In the feed for rednose, we just include:

<environment name="NOSETESTS_PLUGINS" value="rednose/RedNose"/>

That will tell our plugin resolver to perform the equivalent of:

from rednose import RedNose
plugins.append(RedNose())

The implementation of the plugin loader (#2) is reasonably unexciting. It simply parses a PATH-like $NOSETESTS_PLUGINS variable, imports each plugin (using __import__), and hands control over to nosetests itself. For the sake of reusability I’ve pulled the entry-point-like mechanism of instantiating objects from an environment variable into a helper library.

Note that if support for loading plugins from environment variables can be added to nosetests proper, there will be no need for this nosetests-plugin-resolver wrapper. And indeed if you are using this mechanism for your own program, you shouldn’t need one either.

Making it useable

So now we have all of the functional pieces, but it’s not very user-friendly. We don’t want the user to have to be an expert at editing zero-install xml feed files, so we’ll need a simple runner that puts together the feed for the user. This program must:

load a persisted list of plugins (implemented as a file in ~/.config with one URI per line)
add or remove plugins from this set (if –plugin-add or –plugin-remove options are given)
compose a local XML feed, using a template and adding in a <requires> element for each requested plugin
ask 0install to launch the feed

The runner could modify a 0install file directly, but I think managing a simple file with one URI per line is a much simpler proposition.

In order to help with this, I’ve created zeroinstall-plugin-manager. If your requirements aren’t terribly demanding, you can simply use it as a <runner> for your own feed and move your actual functionality to the core command of your feed (instead of the default run). If you require more control, it’s an importable python module you can use to do the heavy lifting of maintaining the configuration and generating the user-specific feed file.

Summary: the final setup

So after all this, we have three pieces to our puzzle:

nosetests-runner.xml - this is the feed that is actually run by the user, and contains the code to load plugins from environment variables and then pass control to nosetests proper.
zeroinstall-plugin-manager.xml - this is the <runner> for nosetests-runner.xml, and it processes the plugin options before passing control back to nosetests-runner (with user-specific plugins as additional dependencies).
nose.xml - this is just a zero-install feed for the official nosetests package

Again, if you can add the plugin-loading features from nosetests-runner to nosetests itself, there would be no need to publish two separate feeds - it’s just needed in the case where the original program can’t load plugins from environment variables.

Benefits

So what do we actually gain from using zero install?

Specifically for nosetests, we gain a lot of freedom:

You can use and publish nosetests plugins without having to use setuptools.
- You don’t have to use zero-install either, as long as your preferred system can add to $NOSETESTS_PLUGINS.
Plugins are not installed globally, and are only loaded when required by either the program under test or the user’s preference.
- This reduces the chance of conflicting plugins.
You can easily stop using a plugin temporarily to diagnose problems.

And in general, for any program that uses 0install to manage plugins, you get:

A robust dependency manager that can deal with not just inter-dependent plugins, but also plugins that require native libraries or libraries written in another language.
Developers can use a feed that runs your application with the required plugins for their project as dependencies, saving end-users from having to manually install / remove plugins.
0test can test multiple versions of your app automatically, even if different versions have conflicting plugin dependencies.

Results

I may not be able to convince the author of nosetests to use this new system instead of the setuptools-based approach (since setuptools is so widely used in the python community), but nonetheless as an experiment it’s shown that zero install can work quite well as a plugin manager. No knowledge of zero install is required to use it, and developers get the benefits of a general-purpose dependency manager instead of the quirks of rolling your own plugin system. I strongly suggest that anyone looking to add a plugin system to an application take a look at zero install.

Extending to other software: Gnome Shell

To show that this is really not a hard thing to apply to other projects, I’ll give another example using Gnome Shell, the new user interface for Gnome 3. I have a vested interest in using zero install for gnome shell, as I have published an extension (shellshape) that relies upon a custom fork of mutter (which the existing extension system cannot support).

So, how hard was it to add zeroinstall extensions to gnome shell? Pretty easy - since gnome-shell is already a good citizen using $XDG_DATA_DIRS to find extensions, there was no code modification needed. It’s also an excellent example of the benefits of zero install, since the manual (user-level) installation process is downright awful for extensions that need their own gsettings schemas (it’s possible, though I’ve never seen anything but discouragement for those who would like to).

To use this system without any modifications to gnome-shell, you can simply run:

0launch \
	http://gfxmonk.net/dist/0install/zeroinstall-plugin-manager.xml \
	--plugin-command run \
	http://gfxmonk.net/dist/0install/gnome-shell.xml

Obviously, this is better used as a <runner>² - the above script is just to demonstrate that this mechanism requires no code modifications. I’ve also started using this runner for shellshape, because that’s become my gnome-shell replacement.

Two examples that use this mechanism already are my own shellshape.xml, and the much simpler gnome-shell-updateindicator.xml feed I created for updateindicator.

        Virtualenv and friends can help with non-global python packages and I fully recommend doing so, but it's still a hack (and requires too much effort from the user).

↩

      I'd add the `<runner>` to gnome-shell itself, except that it's a package implementation and for technical reasons these can't make use of `<runner>`s.

↩

Tame.JS: Async Flow Control

2011-07-19T00:00:00+10:00

If you are interested in my defer work with async control flow in CofeeScript, you’ll probably be interested in Tame.JS. The guys from OkCupid have a history with this sort of thing, apparently they have been using a similar mechanism they built for C++ for years.

Tame allows for more explicit control over parallelism than defer, and is a pretty simple mechanism. Contrasted to Stratified JS it seems to be simpler and more interoperable with existing javascript codebases, but also has fewer features - Tame.js is at a similar level to defer, while Stratified JS offers additional features like promise values (strata), parallel composition, alternative composition and more.

Regardless of which you prefer, it’s good to see people tackling the problem despite the common wisdom seeming to be that there is no problem (or worse, that it can be adequately addressed with libraries alone).

As usual, there’s some good commentary going on at hacker news. There are even a bunch of people wondering when such a useful mechanism will arrive in CoffeeScript ;)

(view link)

Personas: The Missing Piece of Google+ Relationships

2011-07-09T00:00:00+10:00

I’ve been playing with Google+ for a little while, and think that circles (and the emphasis on using them to manage your relationships) is an excellent approach. But it still seems to me like half of the equation is missing.

In the Beginning, Friends Were Binary (and Mutual)

In Facebook, you are friends or you are not. And worse, you can only be friends if both participants agree to be friends. That might well represent close friends reasonably well, but that’s not what Facebook is used for a lot of the time, and it makes it pretty awkward when you have to decide whether to refuse someone friendship status just because you don’t really want to be inundated with their updates.

Asymmetric, Contextual Relationships

Google+ circles allow me, the viewer, to group you. I can group you by context (work, geek, etc) or by closeness (friend, acquaintance, etc). I can put you in multiple groups. I don’t have to get your permission, and you don’t even know what groups I’ve put you in. When I post, I can share with groups, and when I read posts, I can filter by group. This is all pretty splendid, because it supports a more nuanced view of relations more than just a boolean “friend” status.

But it’s only half of the equation. I can say that you are my close friend and that you are a poster of excellent geeky links, but that simply drops you wholesale into both of those contexts. I see everything that you share with me, in any context that I have placed you.

People aren’t like that. People share a number of things, for a number of reasons. I post ~~dumb~~ humorous quips. I post boring status updates about my sandwich and its contents. I post about political issues. I post timely banter filled with inside references to my close friends. And most frequently of all, I post about technology.

Now, I know for a fact that not everybody wants to see all of that. My high-school friends don’t care about inside jokes, but may well care about big events like my career, travel adventures, moving, etc. People who follow me for my relentless but insightful bashing of ruby and javascript probably don’t give a damn about my personal life, and may or may not care about my views on current political events. Not to mention that the current system places the same importance on “I have a sandwich” as “I have a baby” (I’m sure the latter would receive more attention, but the posts alone aren’t represented any differently).

I could use circles for this, in theory. I could send my humorous quips to those who find me funny, my geeky links to those who would be interested in them, and my discussions of sandwich contents to those I presume to be exceedingly bored. But that’s close to impossible for me to get right (even if I do, people’s preferences are bound to change without them telling me) and doesn’t deal with twitter-style followers at all - what if my geeky links are so great that people I’ve never added to a circle wish to read them? I have no problem with that, and yet circles encourages me to share those things with only those who I think would like them (or else post them too broadly, annoying my friends who don’t care for masses of geeky links).

Even those friends who do care about all of my posts probably don’t want to see them everywhere. If you’re checking up what’s going on in the lives of your friends, you don’t really want to see links to ranting articles about the state of the patent system or discussions over which is the true circle constant. And yet maybe you do want to see that when you’re cruising through your tech friends’ posts.

Personas: Two-Way Sharing Control

Blogs and other sites will often address this issue with tags or categories. If you have great opinions on programming but are an insufferable git when you ramble on about sports, I will happily just subscribe to your programming tagged posts. Personas serve the same purpose, on a personal level. A blog has categories, I have personas. Instead of forcing me to carefully pick my audience, let the choice be two-way. I’ll share as broadly as I feel comfortable, and you can view my updates only so far as they interest you. You get a more relevant stream of information, and I don’t have to self-censor so as to not be that guy who drowns out everyone else’s stream.

For a filtering example, take Vic Gundotra. He’s in my “following” circle because he posts relevant news about google+ itself that I want to be kept up to date on. He also shares photos of him and his wife, and general life news. No offence to Vic, but I don’t really care for that other stuff, and to be honest I feel a little invasive looking at pictures of him and his wife. I would very much like to see only Vic’s “google-news” persona.

Even if I want to see all of your updates, I don’t want to see them in the same place. If I want to catch up on what’s going on in the lives of the people closest to me, I don’t want to see all of my best friend’s links to geek news and neat programming ideas. I still want to read such things, but I’d much rather see them in my “geeky news” stream when I feel like catching up on that sort of thing.

Actual implementation

I imagine personas would work pretty much like tags do on a blog. You would typically use one or two, and you’d typically keep the number quite low. There’s a pragmatic issue about what happens when you add new tags, and I think the answer is for new tags to be included by default. If my best friend suddenly starts sharing heaps of neat magic tricks, it’s much easier for me to ignore that tag (once) than it would be for me to be constantly checking whether any of my friends had suddenly developed wizardly tendencies.

In practice, it seems likely there would be a relation between tags and circles. For example, I would typically share my “geek links” and “thoughts on current events” with everyone, “life updates” with friends & acquaintances, and “random babblings” with just my friends. I don’t know whether this link should be explicit (by you assigning correlations in the UI) or heuristic (I’ll bet google could make some good circle suggestions based on a set of tags, if your behaviour is fairly consistent). In the common case, it should end up little more effort for the poster than selecting circles currently is.

I’ve been told in the past that my ideas about obsessive categorisation (see: irank) are really good, but that nobody actually cares enough to put as much effort as I’m willing to. So I suspect there are a lot of people who would not be at all enthused by this idea, and the effort and complexity of adding it wouldn’t be worth it. I really hope that’s not true though, because I’d honestly love to see this implemented.

The most annoying chrome bug arrives in firefox

2011-06-27T00:00:00+10:00

Ugh. I loathe the feature of chrome that hides "http://" from the address bar. You can argue that it makes the address bar less scary, but that doesn’t change the fact that it is an incomplete representation of a URL, and after 12 months they still can’t do it right and won’t provide an option to disable it.

When chrome implemented it the response to user frustration was “if something is broken, it is a bug and will be fixed”. But I’ve encountered a number of broken things that the developers either cannot fix or refuse to fix, so clearly that’s not true. Not to mention the unnecessary effort fixing all the bugs they actually did fix (this thing was horribly broken when it first landed). Perhaps firefox will actually make sure it works properly, but I doubt it.

(view link)

Shellshape: A Tiling Window Manager for Gnome Shell

2011-06-26T00:00:00+10:00

Today I released the first version of shellshape, a tiling window manager plugin for gnome-shell. It’s definitely pre-alpha software, and currently requires a custom fork of the mutter window manager. I’ve had some trouble getting it running due to awful packaging things (I now know far too much about dynamic linking path resolution on linux), but it should work, at least on Fedora 15. Please give it a go if you use gnome-shell - and if not, there’s more information and a demo video at the above link.

It’s got a long way to go - there are certainly bugs, and some features aren’t done right yet. But it works, and that’s pretty exciting to me after working towards it on and off for about four months.

Node.js is backwards

2011-06-24T00:00:00+10:00

(view link)

daglink: organise your system configuration

2011-06-10T00:00:00+10:00

I just released daglink, a tiny program for managing your system configuration. The readme has most of the information, but as a quick overview daglink allows you to maintain symbolic links from well-known locations (e.g /etc/apt/sources.list.d/) into locations of your choosing (mine live in ~/dev/app-customisations/).

But more than just creating symlinks, it allows you to create the appropriate links based on tags of your choosing, for example distro, release and keyboard type (yes, I do actually have multiple of each of those that I frequently use).

I’ve seen others use git or puppet for this, and until now I just had a hand-rolled script and some well-named files. So I figured I might as well write something proper. Did I mention it has built-in support for zero install URLs instead of paths, if you’re into that sort of thing? I’m not sure how useful that is (or should be), but it could turn out useful for some very niche cases.

Ruby's unicode treatment

2011-05-05T00:00:00+10:00

I recently came across this enlightening post on the changes to strings and encodings in ruby 1.9. As a python lover who has only used ruby 1.8 so far, it’s interesting to see the different approaches to very similar problems in python 3 and ruby 1.9.

I may be biased, but ruby’s implementation sounds like it will lead to a lot of pain and bugs, while python’s implementation will lead to a little more pain as you are forced to learn about encodings, and a lot less bugs (as you are forced to learn about encodings). Let me explain why:

Ruby conflates the concept of strings and bytes

This has been the biggest problem with encodings in most languages for a long time. It saddens me to see that ruby is still not fixing it. In python 3, there are bytes and there are strs. Trying to do the wrong thing (without converting one to the other by explicitly encodeing or decodeing) will always result in a TypeError. In ruby, doing the wrong thing will sometimes raise an error.

Yes, both languages are dynamically typed - but the difference is still important. In python, you will get an error as soon as you run bad code. In ruby, you will get an error as soon as you run bad code with input data that is non-ASCII (and sometimes only if you use multiple encodings). And how much of your test data is ASCII? If your native tongue is english, probably most of it. That means your test cases will not catch encoding errors - your users will.

Note that neither language can prevent bugs that are due to incorrectly specifying the encoding of a given string. But in python, you have to specify this a lot less often (see my next point) and the conversion is represented at a type level, therefore it should be easier to spot and a more obvious test candidate.

I liken this approach to weak typing. In perl, 1 is less than 2. And 9 is less than 10. Unless one of your arguments is actually a string, at which point perl silently converts them both to strings. What’s worse, you will still think it works while you happen to use values 1 through 9 - only once you reach 10 will perl happily declare “10” to be less than “9”.

Ruby and python are both strongly typed and will not allow you to compare a number to a string¹, because that doesn’t make sense. The same goes with strings and bytes. It’s far more confusing since in the most common case (ASCII), they happen to equal the same thing, but combining the two in any operation (without explicit conversion) simply does not make sense. It may happen to work in a bunch of common cases, but that’s not how you write robust software.

Ruby makes you think about the encoding of every single string

If I read that article correctly, there is no String-like type in ruby that allows you to arbitrarily concatenate two strings, regardless of their original encoding. Rather the normal concatenation functions will happen to work² if the two strings are of the same encoding, or if one value happens to be only ASCII. If you want to properly concatenate two strings, you either have to:

check the encoding at each concatenation operation (incredibly cumbersome), or
ensure your entire application (or at least the part that you wrote) converts strings to your preferred encoding at every boundary. This includes not only your interfaces to the outside world (HTTP, files on disk) but also potentially any string that you receive from a library.

And if you’re writing a library, you’ll have to resort to option (1), since you don’t get to control the rest of the application.

In python, the only times you have to deal with encodings are when going to or from bytes, which is exactly when you need to know about encodings. So you still need to specify encodings for network traffic, or data on disk. But once you have a string, you never need to know its encoding, and it will combine just fine with any other string. What’s more, you will never be under the mistaken impression that your data is a string, when in fact it is really just binary data (or at least, you’ll find out as soon as you run that code).

Python libraries that don’t need to encode or decode data to or from bytes simply deal with the unicode str type, and never have to care about encodings. Whereas it sounds like erb and other ruby templating libraries will either have to do a lot of encoding checks / conversions, or will only work on strings with the correct encoding.

A caveat: perhaps unicode is not sufficient?

According to the original post, ruby cannot use a single unicode type because some encodings (specifically SHIFT-JIS) cannot be losslessly converted to unicode. This is the first I’ve heard of it, and information is scarce on the matter (and the article itself provides no further clues). I did eventually find this thread on hacker news, which suggests the article is referring to Han unification. I don’t know enough about asian character sets to know how big of an issue this actually is, but it’s an interesting one (and new to me). It would be a huge shame if unicode were not sufficient to represent some character sets, given that universality was the entire point of the unicode effort. Even so, I’m sure there are ways to deal with this situation that don’t have the unfortunate effects I’ve listed above - there is still no excuse for failing to properly separate Bytes from Strings, for example.

Apparently python 2 does not forbid this due to a weird implementation detail (although it at least gives less misleading results; a string is always greater than an integer). Thankfully python 3 will raise a TypeError.↩
Note that things which merely happen to work will almost universally stop doing so in painfully inconvenient (and often confusing) ways↩

Awesome light drawing video

2011-05-05T00:00:00+10:00

Courtesy Ned.

(view link)

From Ubuntu to Fedora

2011-05-03T00:00:00+10:00

I recently moved from Ubuntu to Fedora. It seemed like the right time for it. I just got a new computer, so I’d have to do a fresh install of something anyway. And on the day that my computer arrived, the Fedora 15 beta was released. Perhaps it was meant to be?

Why switch?

I have a number of reasons for trying Fedora, but the biggest few are:

To hack

Most simply, I want to hack on gnome-shell, and Fedora is currently the only distro to include it. It’s entirely possible that I’ll switch back to Ubuntu in 6 months time, when gnome-shell is supported. But I’d rather not put my project on hold until then.

To see what’s out there

I’d like to see how different distros actually are. Ubuntu was the first linux distro I used at home, and I feel it’s worth knowing what things are different / better / worse between distros. I’m pretty adept at maintaining my own preferred system configuration, so in theory it should be reasonably painless to switch.

Ubuntu is a bit of a control freak

Ubuntu has a reputation for being heavily patched, which can interfere with submitting upstream bug reports and such. I heaven’t really experienced that either way, but many recent changes in natty are Ubuntu-specific patches that I disagree with. It is starting to feel to me like Ubuntu is becoming the HTC of linux - HTC provides great android phones, but they keep sticking their foot in and messing with the software, which in my experience has led to more bugs, slower software and a general muddying of the android experience.

I have far greater faith in canonical’s ability to produce quality software than I do for HTC (and canonical does actually contribute things upstream), but the analogy still holds to some degree. Ubuntu wants to differentiate itself from upstream and other distros, but personally I would prefer a less opinionated distro. I have heard that Fedora is closer to upstream sources, and is governed more by the community and less by Mark Shuttleworth (who I’m sure is a great guy, but that doesn’t always make him right).

So, having some of my reasoning out of the way, let’s delve into how it’s going so far. Note that this is only a comparison to Ubuntu, not other distros - I haven’t actually used anything else (yet).

How’s Fedora?

Overall, it’s pretty nice. I’ve had a few issues with system configuration and kernel modules that Ubuntu makes easier, but I’ve also had a bunch of bugs disappear and things work better than they did in Ubuntu. I don’t think either distro is wildly better than the other, but so far I’m personally happier with Fedora.

Package management

For most users, the biggest difficulty in setting up a new system is simply the package manager, its interface, and actual package names. There’s pretty much the same selection of software, it just might have different names. This probably affects developers more than end-users, as most of the different names are for libraries.

I think I like yum a little better than apt, as it’s command line interface seems more clean and logical. But really, they do the same thing and they both manage it just fine. rpm commands are just as archaic as dpkg ones, but this cheat sheet covers all I really need for both of them.

Yum also does a lot more automatic refreshing, which is convenient because you don’t have to remember to do it yourself (I’ve had some bizarre apt sessions due to stale package data), but can be annoying when you forget the -C (cache) option for a simple search or info operation that gets blocked by a metadata refresh.

The Ubuntu software center is a really nice UI, and something that I hope appstream will bring to the other distros soon. But the packagekit-based UI in Fedora is more than enough for me. One neat feature is that after installing a package, it will list the user-facing applications that were installed, tell you their menu heirarchy locations, and allow you to run any of them. Much better than hunting in the “Applications” menu looking for that thing you just installed.

One of the benefits of Fedora using PackageKit (or maybe it’s yum’s doing?) is that the package manager isn’t so blocking. Operations can still only proceed one at a time (although read-only operations seem to work while other things are happening), but it’s not like on Ubuntu where you can’t run any apt commands while you have synaptic open.

One random note, yum info tells you what repository an installed package comes from. I can’t believe I haven’t found the apt or dpkg invocation to do the same, but I’ve looked, and it shouldn’t be that hard.

PPAs and private repositories

I believe Fedora has a similar mechanism to launchpad PPAs, although it’s not as widely used and possibly not as easy. That doesn’t really bother me, as I prefer to use 0install for as many non-distro applications as I can. I could do a whole post about this, but PPAs solve the problem for one step (“I have nonstandard software that depends only on standard libraries”). It does not solve the problem in general (“I have nonstandard software with arbitrary dependencies”), nor does it attempt to address the very important issue of “I want to use different versions of libraries that system components depend upon” (a.k.a dependency hell). It all seems like a lot of effort for a reasonably limited solution. I strongly believe that 0install has it right, so am unlikely to ever embrace PPAs or whatever it is that Fedora has in their place.

Kernels and encryption and security, oh my!

This is where the Fedora transition was not so smooth. Ubuntu has a neat facility for encrypting your home directory, or a Private folder within it. It uses ecryptfs, which is available on Fedora, but not default and not really exposed in the UI. I did get it installed, but had troubles with the kernel modules not being loaded automatically.

SELinux is probably a great thing, but I’m not sure I’ll get much use out of it. It put itself into terrified mode when I tried to run things with my /home directory copied over from my old Ubuntu disk, as evidently it was lacking important SELinux metadata. But once I found out how to tell SELinux to relabel my hard drive, it hasn’t made a peep. I’m still not sure if I’ll get any benefit from it, but it sounds like a good thing to have.

I also had problems with VirtualBox not having the right kernel modules loaded, although it did work once (I think I upgraded my kernel since). I’m running a beta, so this hopefully isn’t normal behaviour.

On a positive system-level note, from everything I have read systemd sounds like a much better boot system than upstart, although I have little experience with either.

Free as in beards

Fedora, like debian, doesn’t ship any non-free codecs (mp3, h.264, etc) or other patent-encumbered technology (flash) by default. Ubuntu does via a separate repository that can be enabled at install time (or after). This is much nicer for users, although I completely understand and sympathize with the puritan position.

Of course, I still need the damn codecs, and I need flash. So there is more work in Fedora to get the appropriate repository set up (rpm fusion), and I ended up installing flash manually because of dependency troubles. That’s possibly related to the fact I’m running a beta.

Fedora also doesn’t package chromium in the default repos, for what sound like pretty sane reasons. It sounds like google is more to blame than Fedora on that front, but again the user is the one who has to deal with the slight inconvenience of adding a third-party repo.

Miscellaneous bugs

These are presumably not common to many users, but gee they annoy me.

Pros for Fedora:

gpg-agent actually works out of the box
my keyboard’s volume_up key is no longer irrevocably bound to activate the screensaver. For 3 Ubuntu releases straight I had to manually cripple the gnome-screensaver system binary in order to use my keyboard properly.
I can change the GVim icon with a theme (it seems to be hard coded into the binary in Ubuntu)
Pretty much all the things I mentioned in my Natty UI Fumbles post.

Cons for Fedora:

I can’t seem to enable rgba subpixel smoothing.
Guake crashes on startup. Probably a general gnome bug.
nautilus keeps forgetting my background image, and turning the screen white. I’m using a gnome-shell hack to get it to display the desktop, so that could be why ;)

Equally broken:

Both unity and gnome-shell fail to raise the window you alt-tab to if you have enabled focus-follows-mouse and your mouse does not happen to already be over the desired window. That is going to annoy me.
Suspend resets keyboard layout. This bug probably exists in Ubuntu, but I could never get my old computer to come back from suspend anyway (probably a hardware / kernel thing).

The shell

Oh, the shell.

This hopefully won’t be such a hot topic in 6 months time, but right now it’s probably the biggest factor anyone needs to consider when upgrading / installing either of these distros.

Overall, I much prefer gnome-shell. It seems reasonably well designed and calm (for the most part), whereas my experience with unity made it seem more haphazard. But overall, there seem to be good ideas in both, and I hope that the best ideas from each will eventually make it into the other. Right now I have plans to work on gnome-shell extensions, so there is simply little point in me running an operating system that prevents that.

Particular notes, while I’m on the subject:

I can’t believe gnome-shell doesn’t display the contents of your Desktop folder. You can enable it with gnome-tweak-tool, but it’s a terrible default (and it keeps losing my background image).
The choice to not have an actual “shut down” option in the session menu is pure crazy. Easily fixed with an extension, but still a terrible default.
Alt-tab seems much better thought out (in combination with alt-`) than unity.
Launching multiple apps at once is tedious, because the dock is only visible in overlay mode, and overlay mode disappears whenever you launch an app. So you have to keep tapping the windows key.
I didn’t think I would, but I actually quite like the automatic workspace management now that I’m used to it. I still have trouble if I want to swap two workspaces (If you move everything out of a workspace, it disappears!) but other than that it’s treating me well.
There’s no way for applications to dynamically add things to the top panel. I hope this will change in future releases, but the persistent notification / applet problem in general seems like it has simply been swept under the rug (or hidden in the bottom-right corner ;)) for now.
Speaking of issues that have been ignored, I’m baffled by the treatment of minimised windows (the button has been removed, but the functionality is still there. They don’t go anywhere, and still clog up the overview mode). I don’t know if that will change in future releases, but I feel something has to change…

If you’re new to the shell or don’t know how it works, the cheat sheet is a good starting point.

I still think the gnome shell is a much improved interface over gnome 2, and hopefully it’ll keep getting better over the next few releases. Plus, it’s bound to be heaps better once I’ve added tiling window management to it ;)

A Team player

There’s a little bit of a theme in this post. My personal preference is for distributions not to try too hard to differentiate, and rather focus on bug fixing and integration work, as well as improving upstream software rather than forking it. For example, Fedora sticks to gnome 3 and PackageKit, which are both cross-distro initiatives. While Ubuntu has written its own desktop environment (Unity) and, still doesn’t have a working packagekit UI.

I’m particularly biased with the packagekit thing because 0install uses it to install dependencies that are only available as system packages (because they have not been ported to 0install). 0install is my absolute favourite piece of software (well, maybe a tie with vim). It’s a dependency manager for cross-distro, cross-platform, decentralised applications and libraries. I find it ridiculous that every distro has it’s own name for the exact same software, and I have started using tiny 0install feeds to simply abstract over that for the packages that I need to depend on for my own software and scripts. Happily, almost all of the 0install feeds I’ve published (90+ at last count!) worked just fine on Fedora without additional effort on my part.

Summary

Overall, I’m currently happier with Fedora then I was with Ubuntu. I’ve listed a lot of issues here, but I’ve fixed a lot of them (and ubuntu had its fair share). It was definitely a little harder to get set up the way that I want, but that’s a one-off effort. And a lot of it is due to very new and very recent decisions (like the shell), which I hope will flatten out in further releases so that users are able to run whatever software they prefer. I’m thankful for the work Ubuntu has done to invigorate the linux desktop space, but right now I’m enjoying a distro that doesn’t feel such a frantic need to differentiate.

Tip: Transient Notifications in notify-send

2011-04-25T00:00:00+10:00

In gnome-shell (and in contrast to Ubuntu), notifications are persistent by default - they don’t go away until you dismiss them.

I have been using notify-send for some time to pop up quick messages that do not need to stick around, so I’m happy to have found how you can make individual notifications transient. If you want to make a transient notify-send message, you can do the following:

notify-send --hint=int:transient:1 [standard-arguments ...]

I’ve created a notify-send script in my ~/bin directory that sets this by default, although that depends on you having added ~/bin before /usr/bin in your $PATH

New GVim Icon

2011-04-25T00:00:00+10:00

The official gvim icon (left) is showing its age. The palette seems to be from the days of 256color, there is no anti-aliasing to speak of in many variants, and it’s just generally bland. I have found a few more modern variants, most notably the tango version, but none of them looked terribly great to me.

So I set out to create a more modern rendering of the classic logo. I exchanged the round serifs for a round diamond, added some psuedo-3d shading, gradients and shadows. All in all I’m quite pleased with how it’s turned out: 128px / svg

If you’d like to use this for gvim, you should do something like this¹:

mkdir -p ~/.icons
cd ~/.icons
wget http://gfxmonk.net/misc/gvim-icon-gfxmonk.tgz
tar zxf gvim-icon-gfxmonk.tgz
rm gvim-icon-gfxmonk.tgz

Then go to your system’s theme selector and pick the “gfxmonk” icon theme.

If all the other (non-gvim) icons have now reverted to the system default and you didn’t want that, you will need to edit ~/.icons/gfxmonk/index.theme and set the inherits value to the name of your preferred icon set².

Note: the svg icon may need the norasi font installed to display properly.

These instructions have been tested to work on fedora, but I fear other distros may have wildly different mechanisms for overriding icons.↩
Forgive me if there is a better way for packaging/installing icons, this is my first attempt.↩

Natty UI Fumbles

2011-04-24T00:00:00+10:00

While my main computer is out of action, I’ve been trying beta 2 of ubuntu 11.04 (Natty Narwhal). The biggest change in this version is the introduction of unity, but there are also a number of other UI changes. Here are some of the bad bits:

The new scrollbars break a few interactions that I am rather fond of:
- Sometimes the grabber is inside the window, sometimes it’s outside.
- If you scroll up or down by pressing the buttons a few times, and then pause to read, and then try to click the arrow again, it may have disappeared - leading to awfully surprising results.
- You can’t use the neat trick of middle-clicking anywhere in the scrollbar pane to immediately bring the scrubber there (instead you have to find its current location, then drag it to where you want it to be).
- If the orange scrubber is taller than the pair of arrows and you want to hit one of the arrows, it remains a moving target until you actually leave the window’s boundary.
- After scrolling with the mouse wheel, the orange scrubber turns practically invisible, making it impossible to tell how much scrolling remains.
The new terminal theme makes it really hard to tell which tab is active.
Whenever I press alt (for example, when starting alt-tab or another keyboard shortcut) I get really distracted by the global menu flashing in and out of my peripheral vision.
Alt-tab raises windows but does not focus them (bug).
Focus-follows-mouse (my preferred setting) is incompatible with the new global menu. You can remove the global menu, but that didn’t work for me. For now, I’m having to get used to pressing F10. There’s a proposal to fix the global menu behaviour when using focus-follows-mouse, but nobody seems interested in implementing it.
There’s no way to make “fullscreen” mean “everything but the dock”, as it does by default for other non-auto-hide docks. I do like the combination of titlebar & top-bar though.

Given my progress on a tiling window manager for gnome-shell (more on that when it’s working, I hope), I may well end up switching to gnome-shell soon. I’ll probably try out fedora 15 in the process, I wonder how much of ubuntu I’ll miss.

Arrows and Haskell

2011-04-23T00:00:00+10:00

For a little while I’ve been wondering what exactly arrows were and what they’re used for - ever since I came across their syntax in my window manager’s somewhat-indecipherable configuration file (which I don’t claim to understand).

So anyway, this article is probably the best thing I’ve found on the subject. I’m still not entirely sure what I’d use them for (the parser example that everyone seems to give is a little abstract for my likes), but this article was the easiest introduction to arrows that I’ve found, and now I think I finally get what they are, at least…

GFX::Monk

Cool projects I learnt about at lca2012

browser id:

mediastreams processing api:

css calc()

News on arbitrary metadata in git:

openstack

testing ctdb

Assessing the potential impact of package updates

misc projects / pointers

Mentally ignoring files in git

Obligate.js

Ruby's split() function makes me feel special (in a bad way)

So, what does String.split() do?

Special cases.

Node.js child processes

Stereoscoper and the Depth of Awesomeness

Experiment: Using Zero-Install as a Plugin Manager

Background:

The basics - every user gets their own feed:

Making plugins known to nosetests at runtime

Making it useable

Summary: the final setup

Benefits

Results

Extending to other software: Gnome Shell

Tame.JS: Async Flow Control

Personas: The Missing Piece of Google+ Relationships

In the Beginning, Friends Were Binary (and Mutual)

Asymmetric, Contextual Relationships

Personas: Two-Way Sharing Control

Actual implementation

The most annoying chrome bug arrives in firefox

Shellshape: A Tiling Window Manager for Gnome Shell

Node.js is backwards

daglink: organise your system configuration

Ruby's unicode treatment

Ruby conflates the concept of strings and bytes

Ruby makes you think about the encoding of every single string

A caveat: perhaps unicode is not sufficient?

Awesome light drawing video

From Ubuntu to Fedora

Table of contents:

Why switch?

To hack

To see what’s out there

Ubuntu is a bit of a control freak

How’s Fedora?

Package management

PPAs and private repositories

Kernels and encryption and security, oh my!

Free as in beards

Miscellaneous bugs

The shell

A Team player

Summary

Tip: Transient Notifications in notify-send

New GVim Icon

Natty UI Fumbles

Arrows and Haskell

So, what does `String.split()` do?