dywypi.org blog

Back on Linux

2012-02-11 00:00:00

About a month ago I moved to Stockholm, Sweden and started working at Spotify. A month before starting I was asked about my preferred setup for working. Having used OS X for year as the primary operating system at work the choice was clear, I wanted to go back to Linux, in this case to Debian to be exact.

Working with Macbook Pro, a year in retrospective, the hardware

There's one thing that really stands out of Apple's laptops, the build quality. The aluminium unibody is manufactured with precision, there's no air intakes in the bottom of the machine (this is really useful if you use a laptop in your lap) and the peripherals, like the touchpad, are top quality. I really miss those and keep on wondering why there's no PC manufacturers that would have the same overall quality of the hardware.

The choices Apple has made also have drawbacks: The mini-DisplayPort requires you to carry around a load of adapters, the amount of USB ports is pretty limited and due to the missing of proper air intake the laptop can get pretty hot, but that was really rare in my use.

Operating system OS X excelled in the hardware drivers: when you have limited amount of hardware you need to support, building good drivers for it is much easier. Feature that I probably miss the most was the somewhat smart handling of external displays, along with the brilliant multitouch gestures of the touchpad.

...and the software

However, the operating system was in no way par with the hardware. Biggest grief was the memory usage. Judging from the interwebs, the most developers purchased 8 gigabytes of memory on their machines to start with. I had four. For web development, Python programming and such four is more than plenty. Except if you're running OS X, apparently.

Completely failed memory management

OS X has a feature called inactive memory. This is memory that was recently used by an app you closed and can be quickly made to active memory if you resume to use that app. A nice concept, that fails miserably. OS X's documentation says, that this memory may be freed at any moment. However in practice, it just keeps on accumulating until you run out of free memory. In this case a sane option for the OS would be freeing the inactive memory. Instead the OS X decides to swap the inactive memory on the disk. So when running out of free memory and having a 1,5 gigabytes of inactive memory left, your OS starts paging the unused inactive memory to disk instead of freeing it for applications to use. Not only this causes your computer to slow down, it also is counter-intuitive in the terms of the original idea of inactive memory: when it's on disk, it definitely is not made active quickly.

I managed to find out that this memory can be freed with combination of XCode's purge-command and repairing disk permissions. First usually freed around 200MB of memory while latter freed almost every bit of inactive memory. Eventually this became a daily routine. When arriving to work the first thing was to hit repair disk permissions button and do something else than actually use the computer for the next five to ten minutes. Sigh.

Messed up software installation

There are at least four different ways of installing software on OS X. Download a DMG image, drag the icon from there to Applications folder, run an installer, install stuff from Mac App Store or compile it yourself. There seems to be no standard way how to do this properly. Of course, being a software dev, I ended up using the last alternative a lot.

However, someone had come up with a solution for manual installing: a package distribution system that takes care of all requirements for you. There are multiple flavours of those, I ended up using homebrew.

When you have pretty much unified hardware (32/64-bit Intel) with pretty limited number of operating system versions, one could think that distributing binaries would be the way to go. Apparently it is not. Using homebrew is like using Gentoo, except that it's even more screwed up. There's no central repository for sources, it tries to download everything from the projects sites using different methods. After that, it compiles everything. Want to install library X? Please wait, compiling and downloading tons of crap from the interwebs, might take tens of minutes of wall time.

And of course this does not support installing Python, Ruby, Perl on any other software that has its own way of distributing software. And in case you mix up MacPorts and homebrew, you're deeply screwed.

And don't even get me started with the closed ecosystem thing Apple and OS X are heading towards with rapid speed.

Why Debian?

I'm a long time Ubuntu user, but this time I decided to go with Debian. Why? Mostly because our servers are Debian and because latest updates of Ubuntu have mostly focused on breaking the desktop environment. With Debian Squeeze I get the old and reliable Gnome 2.30 with the ability of pinning newer packages from either backports or from testing and unstable distributions.

Most notably I get the ultimate software installation tool, apt-get. A tool that installs binary packages with their dependencies. A tool, that's universally supported by Google et al., so I don't even have any problems installing newest Chrome, for example.

Linux in general brings other improvements with it, like working command line tools. GNU version of common tools like find and grep are much more powerful when compared to the BSD alternatives available on OS X.

Do I miss something?

Sure. Even though Linux in modern times mostly works out of the box, there's still slight issues with external displays, for example I can't set the 30" Dell monitor at work to be the only display without doing some xrand magic. I guess that's really the only thing I'm missing from OS X, a sane and automatic way of handling external displays. I also miss the Macbook's superior touchpad, the joystick of Lenovo is nothing in comparison.

Would I consider switching back? No. I got working virtual desktops, package manager and properly working memory model now, currently using 3 gigabytes out of total eight, and it includes running four virtual machines 512 megs of memory each.

So long OS X. You were a nice experiment and I miss some parts of you, however we definitely were not a match made in heaven.

Rewinding during rebase

2011-10-11 00:00:00

A fellow pythonaut asked a question on #python.fi IRC channel: would it be possible to "rewind" during git rebase --interactive? He wanted to go back and edit a few commits during his massive rebase operation and then continue the operation from the same point.

Initial situation

I assume you know in general what rebase does (too bad the kernel.org has no longer the man pages, or anything to that matter, served). If not, I suggest to check that previous link or to run git help rebase on your commandline.

Now, let's assume you have following repository structure:

      D - E - F (*another-branch)
     /
A - B - C (master)

You want to rebase another-branch on top of the master and edit every one of those commits on the way. Naturally, you use git rebase --interactive master while checked in another-branch. Now you edit commits D and E, transforming them to D' and E' respectively, and you have a following repository structure (in middle of the rebase):

      D - E - F (another-branch)
     /
A - B - C - D' - E' - F' (HEAD)
        ^
      master

Now, while editing F, you notice that you forgot to add something in D while rebasing it. Traditionally, the two options here are to abort rebase, rebase again and correct mistakes or to skip remaining commits and manually cherry-pick them back after rebase. But let's assume you've modified a load of commits and need to modify yet another load of commits in this rebase, so you don't want to abort. The solution? Let's just mimic how the rebase works!

Magic part: Rewinding in rebase!

First: Let's create us a point to return to by creating a temporary branch in the F commit: git branch continue. So you have following structure now:

      D - E - F (another-branch)
     /
A - B - C - D' - E' - F' (HEAD/continue)
        ^
      master

Next we'll check out the commit we want to modify: git checkout D' and hack it for the parts needed. After hacking, we'll replay the rebase: Technically rebase with edits is just cherry picking of commits, so well do that. First let's see what to cherry-pick by running git log continue, which returns something like this:

486106d edit edit (F)
ea857b6 Moar edits (E')

While repository looks like this:

      D - E - F (another-branch)
     /
A - B - C - D'' - E' - F' (continue)
        ^   ^
   master   HEAD

Now we just replay the rebase by issuing git cherry-pick ea857b6 and git-cherry-pick 486106d. Now we're back at the point where we rewinded to the D' and can continue rebase normally! Just do the edits on the F you need to do and say git rebase --continue and you'll get on with your rebase, and finally the tree looks like this:

A - B - C - D'' - E' - F (HEAD/another-branch)
        ^
   master

Word of caution

This is more of a proof of concept trick and has worked on one real life scenario (and I've tested it on simulated environment), but I recommend you to back up your working directory / .git before trying this at home.

Or (thanks Akheron) true warrior just digs through reflog and does not waste extra disk space for puny backups!

Mobile Web Development and Maps

2011-07-28 00:00:00

Lately I've been working on Helsinki Festival mobile site (in Finnish). The main idea of the site is to provide mobile information for users, including location and maps of the venues. While implementing the venue locations and other, yet upcoming, features, I've run in to a series of problems of which some should be shared to the wild.

Supporting wide range of devices

Due to heterogeneity of attendees the site should support a wide range of devices. At Taiste we sport a few test devices: iPhone 4, HTC Desire with original firmware and another one rooted with Cyanogenmod and two Nokia devices, the flagship N8 and an older E75. Pretty decent coverage of mobile platforms in circulation in Finland.

I'm not going to discuss about rendering woes and missing <meta name="viewport"> support here, even though they are challenges that just need to be handled when building a site like this. The problem I want to focus on is the functionality of maps and navigation on a mobile web site.

Displaying a map

I started with a dual-mode map: A static image, courtesy of Google, for devices without proper JavaScript support and a fancy embedded Google Map for devices with JavaScript support. However, this turned out to be a bit problematic.

First caveat: scrolling. On touch screen phones the scrolling of venue pages, which have embedded map, was misleading; if user started scrolling (i.e. placed her finger) on the map, she would end up scrolling the map instead of the site. By scrolling the map user would also lose sight of the marker, placed on the map. On Google Maps API this is fairly easy to disable just by passing draggable: false to options of the constructor of a Map object.

Second caveat: The marker. On the embedded map we drew a nice yellow marker. However, the Nokia N8 failed to display this marker. According to my debugging sessions the problem resides somewhere in the Google's Map Marker API, though I can't say for sure. The phone does not report any JavaScript errors, so it must be rendering the Marker to a wrong place or underneath the map element (if it renders it at all). Even with furious debugging applied, the marker could not be seen on the map.

Now the second problem was a bit harder to solve: there didn't seem to be any way we could solve it. The N8 sports a Symbian browser 7.2, which is equipped with Webkit version 525, the same version Chrome 1.0, iPhone 2.2 and Android 1.0 had in 2008, except that somehow it's crappier than those. Luckily, the new Symbian Anna upgrade should bring Browser 7.3, which has a newer webkit and it even includes viewport meta-tag support!

To overcome these two problems I wrote a custom template tag that just outputs an img tag that has an URL to a 240x240 map picture (240 is the smallest horizontal resolution we support) showing the location. The img tag is then later on accompanied with a small JavaScript that transforms the src attribute to point to a wider map picture. JavaScript is needed to sniff the width of the containing div element by using element.clientWidth. Oh, and while doing this I also discovered that Google Maps API limits the image width to 640 pixels. Damn.

Not only the static image overcame these both problems but it also renders a neater marker, with drop shadows and all!

Providing links to the internal navigator application

To ease the finding of a venue site provides a link to an internal map application. Or more specific: tries to provide one. We started with linking directly to Google Maps with proper longitudes and latitudes in the URL. This works mostly fine: on iPhone and on Android it, in almost every case, opens up the native Google Maps application. Almost.

HTC woes

During last firmware update HTC rolled out a new version of their customized Android Browser (which they call "Internet"). This updated Internet contains a small, but pretty annoying bug: it seems to break URL handling so that the Google Maps links are now opened in browser by default and there's no way to change this default. I haven't explored if it does the same with other URLs. My hypothesis is that this could be related to their own navigation software, Route 66, which is shipped with newer HTC Android devices. It could possibly open by default the Google Maps links in this application. However, the app is not available for Desire. Apparently I'm not alone with this. Funny thing is that if I type maps.google.com in the Google Search app (not the website), it asks me politely if I want to open Maps or Browser. So HTC has definitely broke something while modifying the browser.

Needless to say, the link works fine on the Cyanogenmod Desire, as it sports nearly vanilla Android browser by Google. The bad thing here is that there's no other way to send a link that suggests the phone to open Maps application with correct coordinates.

N8 woes and yays

The Nokia Browser documentation developer.nokia.com does not say a thing about opening Ovi Maps from a website. After some excessive googling I found a forum post that mentioned briefly about an undocumented services URI scheme. By calling that URI scheme with a properly formatted URL, in this case services://C2A?C2Aid=2305&Version=1.0&2305=Landmark(lat=XXX&long=YYY), it opens up the Ovi Maps application with a marker at the latitude and longitude specified. You can even supply a name with additional name parameter given to the Landmark "constructor"!

...and the issues with Series 60

Even though Nokia E75 has the same generation browser as N8, it does not support services URI scheme, at least not in the same format. I also failed to find any documentation or experience about opening Ovi Maps application from a link, so the built-in navigation is a no-go with E75. With S60, we're stuck with mobile Google Maps URL and the experience is not optimal. You could even craft a URL that gives you directions, but that would require knowing your current location and E75s browser does not support Geolocation API, so that's a no-go too.

Conclusion

If it would be satsifying for us to serve only iPhone/iOS and Android devices (sans current incarnation of HTC Desire, possibly other HTC Android devices too) the mapping would be easy: Either display a JavaScript map (pretty usable on those devices) or use static image provided by El Goog with a properly formatted link to maps.google.com.

However, if you need to support other OSs, it might require tremendous amounts of googling around the interwebs. I don't even have a clue how Bada or WP7 works, but luckily they both are a small minority in the browser market. I also couldn't find anything for Series 40, so the Nokia seems to keep consistent line of undocumenting through the product line.

My suggestion to mobile phone manufacturers would be a single page of documentation describing the supported URI schemes of the built-in applications to tell the developer what's possible and what's not. With examples, of course.

P.S. At the moment of writing this, we yet haven't launched the new maps and map links in to the wild, but we're in the process of doing so.

rxvt-unicode and OS X

2011-04-05 00:00:00

When I started at my current job at Taiste I got 15" Macbook Pro as my primary computer at work. As a long time Ubuntu user I was faced with series of problems, mainly due to differences in platforms. Package management is a pain I just have to live with, Emacs works almost out of the box as it should, but the terminal requires some heavy tweaking.

Available alternatives

OS X has few alternatives available. The system ships with Terminal.app, which has been redone for Snow Leopard (aka 10.6). It's decent, looks like a terminal, but has at least one minor and one major drawback. Minor drawback is the lack of proper color support (yes, I need colors in terminal). Terminal.app supports only 16 colors. 16, Six-teen. 70s called and they want their terminal back.

Major drawback is that you simply can't override cmd key functionality in any sane way to make it work as a Meta key in terminal. In practice, your forced to use ESC as Meta key, which just does not cut it.

Another option is iTerm2. Feels like the Swiss army knife of OS X terminals. Has load of configuration options, even supports, in latest versions, rebinding cmd keys as Meta keys! So after all it looks that even this milky white cloud has a silver lining.

But unfortunately, the iTerm2 also has a major drawback: If you rebind either of cmd keys as Meta, it wont let cmd+Tab (aka the Alt-Tab equivalent of OS X) go through to OS X level (Emacs knows how to do this, can't be that hard...). So basically you can't switch between applications with keyboard when using iTerm, unless you bind Option key to cmd, which makes option+tab switching possible, which on the other hand breaks every single option+key combo (like parenthesis)...

Back to the One That (Mostly) Works

I was a happy user of rxvt-unicode on Ubuntu, so I started searching if it would cut on OS X too. First caveat is that it needs to be run under X11, or XQuartz. I've been told that that's the final frontier no true OS X user wants to confront.

However, everything went smoother than expected. First thing was to install rxvt-unicode. I used homebrew for that:

  brew install urxvt

After compiling a while (binary packages, anyone?) rxvt-unicode got installed. Butt ugly as usual, but there it was. After configuring colors (see here for example) it was time to make it more Mac-compatible.

Copy & Paste

As every true programmer knows, when deadlines hit the tool that Gets Job Done is Copy & Paste. So it was essential to get copy & paste functionality in urxvt. I use a slightly modified clipboard script on urxvt to do the magic. It's available as a gist.

To get it working, you need to define it (and the shortcuts) in .Xdefaults with few lines:

  URxvt.perl-ext-common: macosx-clipboard
  URxvt*keysym.M-c: perl:macosx-clipboard:copy
  URxvt*keysym.M-v: perl:macosx-clipboard:paste

After this, I got traditional copy & paste with cmd-C and cmd-V in urxvt!

Opening urls

Bart Trojanowski's excellent mark-yank-urls comes to help with URLs. For OS X use, I've modified it a bit. In addition to the script, I use a custom applescript to launch URLs in background in Chrome:

  on run argv
    tell application "Google Chrome"
      set taburl to item 1 of argv
      set myTab to make new tab at end of tabs of window 1 with properties {URL:taburl}
    end tell
  end run

Applescript is launched via bash script (yo dawg!), which just calls osascript <path-to-applescript> argument. Next step is just to add this script and keybinding in .Xdefaults:

  URxvt.perl-ext: selection,mark-yank-urls
  URxvt.keysym.M-u: perl:mark-yank-urls:activate_mark_mode

After this, Meta-U goes to URL selection mode, C-p/C-n scroll through URLs, y yanks (that's copying, you lesser emacsers) and Return opens URLs in background in your Chrome.

The impact of document IDs on performance of CouchDB

2010-11-11 00:00:00

(This text is copied from blog.inoi.fi to my personal blog, as I no longer work there and think this is a keeper)

One major part of our product is a data acquisition framework. The framework is responsible of gathering data on devices and sending it to the server. The server then processes the data and eventually stores it in CouchDB.

CouchDB storage is pretty simple. It's just a single database where the data gets inserted with a random ID. Removals or modifications are hardly done to the data after it has been inserted there, its only function is just to be read later and to be displayed as graphs and such

Things started to get weird

However, the data server started to behave strangely. First symptoms were slow inserts and huge disk usage of the database. For the slow inserts we first suspected the reason was that we were not using CouchDB's bulk document API for inserting the data but after fixing that we only got a minor speed increase. The compaction of the database didn't really help, as it compacted the database too slow, peak rate being 300 documents per second and the average about 100 doc/s. With 8 million docs that takes a while. Worst thing was that the compaction didn't even reduce the disk usage. The database ate 26 gigabytes of disk containing only 8 million documents. That's a whopping 3 kilobytes per document, and the documents were really about 100 bytes in size.

What was even weirder we didn't really see anything strange in the way the server performed. IO ops were low, CPU usage was low and there was a plenty of free memory. Heck, we even could write zeros (dd if=/dev/zero of=/var/somefile.img) on the disk at the rate of 50Mb/s. Even killing most of the other services didn't help: CouchDB just kept being slow. We even upgraded the CouchDB in our test environment to try if it helped, but we only gained small performance gain from that operation.

Let's not be that random

As you might have heard, the problem with the random is that you never can be sure. So after a week or so pondering the issue we stumbled upon on a chapter about bulk inserts and monotonic DocIDs in the excellent CouchDB guide. With a tip from the awesome folks at #couchdb we rebuilt our system using sequential IDs instead of purely random IDs.

We copied the idea of sequential IDs straight from CouchDB which uses them for its auto-generated DocIDs. We just needed to implement it in Python and ended up with the following:

The

SequentialID is a random 32 character hexadecimal string. The first 26 characters is a prefix that stays the same for approx. 8000 subsequent calls. The suffix is increased monotonically. After around 8000 calls the the prefix is regenerated and the suffix is reseted to a small positive value. We also built another one using ISO-formatted UTC-timestamp with few random digits suffixed.

Now you might think that we would have collisions with the

SequentialID, especially if multiple processes are writing to the same CouchDB database. However, we're not since the prefix is a random generated string and the entropy (i.e. string length) is big enough to make the collision highly unlikely. Plus the SequentialID is never shared between processes it is regenerated for every single one instead.

Performance rocketed through the roof

No fix would be good without performance metrics so Petri wrote a small benchmarking script.

This showed us the huge performance increase we got. The write speed is almost four times as fast with sequential IDs than with random IDs. Not to mention that the database takes a one seventh of the space on the disk! We didn't try the compaction speed, but everything indicates that should be a lot faster too.

How does it work?

The reason why this worked is due to the design how CouchDB stores it data on the disk. Internally CouchDB uses B+-tree (from now on referred as B-tree) to store the data of a single database. Now if the data that is inserted in the B-tree (in this case, the DocIDs) is random, it causes the tree to fan out quickly. As the minimum fill rate is 1/2 for every internal node, the nodes are mostly filled up to the 1/2 (as the data spreads evenly due to its randomness) generating more internal nodes than before.

More the internal nodes the more disk seeks the CouchDB has to perform to write the correct leaf to write the data in to. This is why we didn't see any IO ops stacking up: the disk seeks do not show up in the iostat! And this was the single biggest cause to slow down both reads and especially writes of our CouchDB database.

The problem aren't even limited to the disk seeking. The randomness of the DocIDs causes the tree structure to be changed often (due to the nature of B-trees). When using append only log the database has no other way to do than rewrite the whole new structure of the tree to the end of the append only log. As this gets more common and common as more random data gets poured in, the disk usage grows rapidly, eventually hogging a lot more disk than the data in the database is actually requiring. Not only this, this slow down compaction too, as the compaction needs to constantly rebuild the new database.

The B-tree is why using the (semi-)sequential IDs is a real life saver. The new model causes the database to be filled in orderly fashion and the buckets (i.e. leafs) are filled in instead of leaving them half full. Best part here is that the auto generated IDs by CouchDB (which were not an option for us) already use the sequential ID scheme, so using those IDs you don't really need to worry a thing.

So remember kids: if you cram loads of data in your CouchDB, remember to select your document ID scheme carefully!