Sounds From The Dungeon

Links for 2009-07-01 [del.icio.us]

Thu, 02 Jul 2009 00:00:00 PDT

[whatwg] Codecs for <audio> and <video>
Great overview of the (sad) situation

Links for 2009-06-30 [del.icio.us]

Wed, 01 Jul 2009 00:00:00 PDT

Michael Bay Facts
Like Chuck Norris Facts :-)

Links for 2009-06-23 [del.icio.us]

Wed, 24 Jun 2009 00:00:00 PDT

Pay Apple US$9.90 to fix 46 security holes in your iPod Touch
Food for thought

New Toys

Mon, 15 Jun 2009 11:37 +0800

This past Saturday, I went down to the PC Show 2009 and got some new toys to play with.

The first is the Acer Aspire Timeline 3810T notebook, sporting Intel's new ultra-low voltage SU9400 Core 2 Duo processor and promising a battery life of 8 hours!

Some specs: 13.3" (1366x768) LED-backlit display, 4GB DDR3 RAM, 500GB HDD. Price: SGD 1470.

I really like the design of this laptop. Very clean without any of the gimmicky buttons or a hundred flashing lights that seem to be so prevalent on all the laptops these days. But what really made me settle on this model was the combination of insanely long battery life coupled with an amazingly light body weight of just 1.6kg!

It is a bit weak in the graphics department but it plays HD videos from vimeo.com and youtube just fine which is enough juice for my needs.

The other toy I got was a replacement for my ageing N73 phone: a sleek black Android powered HTC Magic!

This was a bit of a surprise since I hadn't expected the Magic to be on sale and even if it was, all news I had heard about it indicated that it would be retailing at SGD 1000+ - something out of my budget. As it turned out, my current service provider M1 had quietly launched the Magic and I was able to get one for SGD 508 after trading in the N73 for a 100 dollar rebate.

All in all, the geek in me is super happy playing with the new toys :)

Links for 2009-06-08 [del.icio.us]

Tue, 09 Jun 2009 00:00:00 PDT

Introducing Android Scripting Environment
I really must get an Android phone now!

Logo Wars!

Fri, 05 Jun 2009 12:05 +0800

Exhibit #1

Exhibit #2

Something I noticed plastered on a bus today:

I wonder which came first!

Links for 2009-05-31 [del.icio.us]

Mon, 01 Jun 2009 00:00:00 PDT

TechnoBabble Podcast #0, a.k.a. 'Hello, World'
wherein we babble about Google Wave

Links for 2009-05-24 [del.icio.us]

Mon, 25 May 2009 00:00:00 PDT

Do Not Play the Lottery Unless You Are a Millionaire
Can someone run the numbers for Singapore's 4D?

Links for 2009-05-22 [del.icio.us]

Sat, 23 May 2009 00:00:00 PDT

Size does not matter!
being in the right place does :)

Giving up on NextBus

Fri, 15 May 2009 00:01 +0800

My bus arrival timings web-app, NextBus, has been performing very poorly for the past several weeks. I've tried everything I can to fix it but unfortunately I can't. Given the circumstances, I have decided to stop further development on the app.

I'll try and describe why the site is broken but the summary is this: I have reason to believe that SBS Transit is rejecting network connections from my app to its servers. In the face of this, there is no way that I can keep NextBus functional.

To regular users who have come to rely on NextBus, I am sorry for letting you down. I have also let down the folks behind third-party apps such as iSingeo and SG Buses which relied on NextBus. I feel really bad that they spent time integrating their applications with NextBus which is now a waste.

I'll leave the site running and it will at least work for the 200 odd stops for which arrival information is available via PublicTransport@SG. iPhone users: you may want to try tranSGuide, an app that shows bus arrival timings. It's developed by a friend and it still works :-)

The Details

As you know, the core functionality of NextBus is to provide you with arrival information of all buses at any bus stop. To do this, it has to communicate with the SBS Transit mobile iris website which provides the information, although in a rather inconvenient manner.

Starting sometime in the last week of March, several connection requests to the SBS Transit servers from the Google App Engine environment (the infrastructure that hosts NextBus), started timing out. I didn't notice it initially since the stops that I personally use NextBus for are handled by the Public Transport@SG service, which NextBus also supports. When I did notice the problem from the server logs, I didn't have sufficient time to investigate the cause since I was to leave town for a week. So the site suffered while I was away.

When I got back, it took me a day to figure out what the problem was (requests to SBS servers timing out) and another day to implement a work-around. My work-around was simple - I implemented a simple web proxy that just fetches pages from SBS servers and returns them as is. I put up this proxy on an Amazon EC2 server instance hosted in the US. I modified the NextBus code to request pages from this proxy instead of requesting them directly from SBS servers. This worked! SBS pages were flowing back to NextBus via the proxy. So I quickly put this new proxy-powered code into production and the NextBus site was working reliably again. I thought it was kind-of weird that direct connections from the App Engine environment were not working but connections from Amazon servers were working. But hey, it was working now! So I didn't give it much thought.

Until a week later.

A week after implementing the proxy, requests from the Amazon EC2 server hosting the proxy also started timing out! I then moved the proxy service to my shared hosting server; the same server that shows you this blog. So now requests were going from NextBus to my shared hosting server to SBS servers and all the way back. NextBus was working again.

This lasted for three days.

Interestingly enough, when the proxy service on my shared hosting server stopped working, it wasn't because of connections to SBS timing out. No, the connection attempts were simply getting dropped! They still are, as a matter of fact.

Then I asked couple of my friends to help me out. They kindly agreed to host the proxy service on their servers. You can guess how that went.

Given what I've seen so far, I think the reasonable conclusion to draw is that the folks at SBS are denying connections from NextBus. It may be intentional or it may just be an automated response by some network monitoring tool that is banning any IP that originates an unusually high number of requests. Either way, I don't have an endless supply of proxy servers at my disposal to try and work around the connection bans. And even if I did have that supply, I have no intention of actually doing it since it makes no sense to go on with NextBus when SBS does not like what I am doing here.

So to all of NextBus' users, so long and thanks for all the fish!

Talk on Google App Engine

Fri, 27 Feb 2009 11:28 +0800

I gave an introductory talk on Google App Engine yesterday at RIAction, a Rich Internet Application platforms focussed day-long event here in Singapore. I am embedding the slides here or you can click here to view in fullscreen or download the deck. Sorry about the lack of any notes!

The event went well for a first effort. Thanks to the organizers & the sponsors for putting this thing together!

On a humourous note, the goodie bag included a first-aid kit sponsored by Microsoft. Feel free to speculate on why MS would be handing out band-aids to developers ;-)

Gothere script update

Fri, 23 Jan 2009 10:16 +0800

Some time back, I wrote about my tiny little greasemonkey script for gothere.sg - all it does is replace links to SBS Transit with links to my NextBus app. Back then, I also wrote that time permitting, I'd update the script to fetch the bus arrival timings & show them within the bubble itself, instead of triggering a new page load. As you've already guessed, I found the time :)

To install/upgrade, just click here: NextBus & gothere.sg. Remember, you must have Greasemonkey already installed in Firefox. If someone feels like packaging the script as a Firefox extension, please do :-)

Update: Okay, here's the NextBus & gothere Firefox Extension.

Finally, a note of thanks to the gothere folks. This update wouldn't have been possible without some code changes on the gothere site. I wrote to them about it and in just a few hours, Dominic got back saying he had already implemented my request! After that, updating the script took no time. Thanks Dominic!

Boxee + Wiimote = ♥

Sun, 18 Jan 2009 18:23 +0800

I've been using Boxee on my Linux system at home for several months now and I am really happy with it. However, one thing that is kind of a pain is having to get up and reach for the keyboard just to control Boxee. Ever since I got a Wii a month ago, I had resolved to correct this shortcoming and the recent upgrade to Ubuntu Intrepid seemed like a perfect time to dive in. So herewith are my short notes on how to control Boxee with a Wiimote!

The basic idea here is that the Wiimote communicates with the Wii console using bluetooth. So if your computer supports bluetooth, it can start talking to the Wiimote over bluetooth using some custom bits of software. Note that this does not involve the use of the Wii's IR sensor bar. Leave that connected to your TV!

First thing you'll need to do is to install a bunch of packages that enable interaction with the Wiimote.

$ sudo apt-get install wminput wmgui lswm

Once these packages are installed, run lswm to see if your Wii got detected.

$ lswm
Put Wiimotes in discoverable mode now (press 1+2)...

Follow the instructions and you should soon see lswm print out your Wiimote's bluetooth MAC address. This would be something like 00:1E:35:09:9A:9A. If lswm exits before discovery is complete, just re-run it till you see the MAC address. Make note of the address.

Next, to verify that all the Wiimote button presses can be detected by your computer, run wmgui and use the File > Connect option to connect to the Wii. You may be asked to put the Wii in discoverable mode again. Once the Wiimote is connected, just press the buttons on your Wiimote & wmgui will show what buttons you pressed.

Now that this basic test procedure is done, we are ready to use the Wiimote as a mouse. This requires the use of a kernel module called uinput. Load it up:

$ sudo modprobe uinput

Now enable mouse mode by typing:

$ sudo wminput 00:1E:35:09:9A:9A

Take care to replace the bluetooth address in the above command with your own Wiimote's address! With wminput running, you should be able to control the mouse (in a bizarre way) by tilting your Wiimote & pressing the A or B buttons. Once this works, hit Ctrl+C to exit wminput.

If you noticed, in the default mouse mode, the Wiimote's A button was mapped to left click while the button B was mapped to right click. The wminput tools allows you to map the buttons in whatever manner suits you. We'll setup a key mapping that makes sense to use with Boxee.

$ cd /etc/cwiid/wminput
$ sudo cp buttons boxee-buttons
$ sudo vim boxee-buttons

In the boxee-buttons file, you'll find the default key mappings for the Wiimote as well as the Nunchuck & the Nintendo Classic controller. Ignore the latter two and replace the Wiimote's key mapping lines with these:

#buttons
Wiimote.A       = KEY_KPENTER
Wiimote.B       = KEY_ESC
Wiimote.Up      = KEY_UP
Wiimote.Down    = KEY_DOWN
Wiimote.Left    = KEY_LEFT
Wiimote.Right   = KEY_RIGHT
Wiimote.Minus   = KEY_KPMINUS
Wiimote.Plus    = KEY_KPPLUS
Wiimote.Home    = KEY_H
Wiimote.1       = KEY_SPACE
Wiimote.2       = KEY_S

The above mapping should be obvious. You can get a list of supported key codes from the wminput source. With the mapping file in place, re-run wminput but specifying this custom mapping file.

$ sudo wminput -c boxee-buttons 00:1E:35:09:9A:9A

Once wminput is running, fire up Boxee and you should now be able to control it using your Wiimote!

It is time to make all these settings permanent. First, the loading of the kernel module is automated by adding the module name uinput to a new line at the end of the file /etc/modules

If you noticed, we were running wminput with superuser privileges. We'll remedy this by adding a udev rule that lets any user in the plugdev group access the uinput module. Create a file (as root) named 50-cwiid-input.rules in the /etc/udev/rules.d directory with the following line in the file:

KERNEL=="uinput", GROUP="plugdev"

Typically, your user will be in the plugdev group. If not, replace the group with something else like admin, etc.

Finally, create a startup script that starts wminput when you log in to your X session. For KDE, this would mean creating a script in ~/.kde/Autostart

$ cd ~/.kde/Autostart
$ echo '#!/bin/sh' > wminput.sh
$ echo 'wminput -dc boxee-buttons 00:1E:35:09:9A:9A' >> wminput.sh
$ chmod +x wminput.sh

That's it! Once you restart your computer, your Wiimote will be ready to use! If you find it's not working, try putting it in discoverable mode and the already running wminput daemon will pick it up.

Have fun!

A brief announcement

Sat, 22 Nov 2008 17:48 +0800

I'm off to India today for about two weeks starting today. Top item on the agenda for this trip is to get married next Saturday :)

If you happen to be in Hyderabad, you are invited!

Adios.

Mitter 0.4

Wed, 12 Nov 2008 19:10 +0800

After a long time, we have a new Mitter release. What's Mitter you say? Mitter is a sweet and simple Twitter client for Linux desktops. You can use it as a regular graphical app that sits in your system tray till you need it or as a CLI app tucked away in one of your half a dozen terminal windows.

There isn't much new feature wise in this release. The Changelog is there for you to peruse. Just think of it as a release that rolls up all the current fixes and lets us focus on the next set of features :-)

If you are a Twitter user running Linux, do give Mitter a try! It's really nice, and that's not just me saying it :-)

Source packages can be downloaded from Mitter's project page while Ubuntu users can use Sugree's PPA to get the latest version.

Incidentally, the release names for the 0.4 cycle will be based on quotes from Jurassic Park, in honour of Michael Crichton.

NextBus meets gothere.sg

Thu, 06 Nov 2008 18:23 +0800

Without a doubt, gothere.sg is one of the best Singapore-focussed map sites today. It may not be the best in terms of content but it's surely the best in terms of usability.

They have this feature where they show little bus stop icons on the map which you can click to bring up a list of bus services operating at that stop. They also include a link to SBS Transit's mobile iris website to check when the next bus is arriving. You can see where this is going :-)

I just put together a tiny Greasemonkey script which replaces the link to iris with a link to my NextBus app. If you have Greasemonkey already setup in your Firefox, you can install the script by clicking here: NextBus & gothere.sg

Or just install the NextBus & gothere Firefox Extension

When I find some time in the future, I'll update the script to fetch the arrival timings and show them within the bubble itself, instead of having to load a new page. Update: Done!

As always, bug reports are welcome. :-)

SBS NextBus: The Next Release

Thu, 30 Oct 2008 18:11 +0800

I've just pushed to production a new version of NextBus, a web-app that I created for tracking bus arrival information at bus stops in Singapore. Read more about it in my earlier post announcing NextBus. Now before I get into what's new, let me first provide you with a list of things that provided the impetus for this new release.

Firstly, NextBus is way more popular than I expected it to ever be. Currently, the site receives well over a thousand requests per day! Considering that I spent zero energy in marketing it, I am quite happy with the uptake and have to thank everyone for using it as well as for spreading the word about its existence! What this also means is that I now have an incentive to keep maintaining it and making new releases!

Secondly, nearly 40% of the site's usage is via Jon's integration of NextBus with isingeo.com and singeo.com.sg. That's quite a few people who are accessing the app only indirectly. And this is despite the fact that the mashup that Jon created is a very crude one relying on just embedding the sbsnextbus.appspot.com pages within his sites. Could I do something to improve the quality and scope of such mashups?

Thirdly, the demographic that uses the app is quite different from my initial expectations. When I first released the app, I had assumed that the majority of requests would come from mobile phones and the app's design & implementation was based on this assumption. But a look at the logs pulled from Google Analytics reveal that the reality is quite different:

There's a caveat to these numbers: the tracking code is only used on the home page. If you visit only the results pages or use the app via isingeo.com, etc., your usage won't show up in the data above. Nevertheless, as you can see, over 45% of usage is from full-fledged desktop browsers. If you count the iPhone too, the proportion of visitors using a fully capable browser goes up by another 18%. The conclusion from all this data? Although I designed the NextBus app for browsers that face network latency & bandwidth constraints - like my Nokia N73 browser - there's significant usage from browsers that face no such constraints. So a substantial number of users are getting a sub-optimal experience from the site.

And finally, the launch of PublicTransport@SG couple of weeks ago. This new site is maintained by the Land Transport Authority and includes real time bus arrival information for quite a few bus stops around town. The obvious benefit over SBS Transit's iris service is that LTA's information includes timing information for buses operated by SMRT. Apart from providing arrival information on the PublicTransport@SG site, LTA has also launched an SMS enquiry service similar to iris. However, there is no trace of a mobile optimized version of the site. Thus, there is scope for me to do some work after all :-)

So these were the things I had in mind while working on the three big features in this new release.

Optimized site for capable browsers If your browser can handle it, it will be presented with a results page that is much faster to load. This is done by querying for all the different services at your particular bus stop in parallel using AJAX. Once the page is loaded, you can refresh the results without causing a full page reload. You can also refresh results for just one particular service by clicking its link. For other (mobile phone) browsers which can't handle AJAX, the old style static html results page will be served. I've tested this in Firefox, IE 6/7 and Opera on the desktop. The AJAX powered results also work on the iPhone. If there's something broken with this feature, please do get in touch! Note: if you have saved the results pages as bookmarks, you'll not get the AJAX effect since the URLs have changed. Please update your bookmarks.

JSON API With this release, I am launching a publicly accessible API to provide developers with access to bus arrival data in a clean format. The whole AJAX experience is itself powered by this API. I hope the release of this API will spur the creation of lots of great mashups! The details of the API are documented further down this post and I welcome developer feedback for this feature. Also, this feature finally makes NextBus Web 2.0 compliant. ;-)

PublicTransport@SG integration As you may have guessed (if you actually read the preamble!), NextBus now integrates results from LTA's site. If the bus stop you query for happens to be one of the (as of now) 213 stops that PublicTransport@SG provides results for, NextBus will pull results from there. By virtue of this, arrival info for SMRT buses is also available for these 213 stops.

That's about it! I urge you to try out the new release and let me know if you find anything that's broken or doesn't work as expected. If you do find NextBus to be more useful than ever, please talk about it! Marketing!

JSON API

The API is very simple. There are two URI end points that the API provides:

http://sbsnextbus.appspot.com/api/v1/<busstopnumber>/ -> Returns the stop description (if available) and a list of bus services operating at the queried stop.
http://sbsnextbus.appspot.com/api/v1/<busstopnumber>/<service>/ -> Returns arrival timings for the queried bus service & stop combination.

Examples: http://sbsnextbus.appspot.com/api/v1/92049/ and http://sbsnextbus.appspot.com/api/v1/92049/15/

The response is in the JSON format. The response schema should be pretty obvious looking at the results to the above query.

To facilitate building of browser based mashups that use the API, I've also provided support for JSONP callbacks using the callback=foo parameter, like so: http://sbsnextbus.appspot.com/api/v1/92049/?callback=myCallbackFunction

Things to watch out for:

In the API response, the code attribute value should be 200. If it isn't, then something went wrong. If possible, the cause of the error will be sent back as the value of a message attribute. Example: http://sbsnextbus.appspot.com/api/v1/00000/
There's some inconsistency between SBS iris & PT@SG regarding the use of service numbers. While PT@SG truncates leading zeros (service 15 instead of 015), iris demands that no truncation take place. Hence, you should use the service number strings exactly as returned by the API without any modifications between successive calls.
The backend caches the arrival timing information for one minute. Hence, there's no point in making calls for the same service & stop combo more than once a minute.
Don't treat NextBus as an authoritative source of bus services & stops in Singapore. When the API returns a list of services operating at a stop, it only means that the API can provide arrival info for those services. It does not mean that that list of bus services is the complete list of services operating at that stop.

Do get in touch if you have any concerns about this API. If you go ahead and build something using the API, I would love to hear about it!

Bizarro Mail

Tue, 23 Sep 2008 18:52 +0800

When I first read the subject Recurrent/Occult hernias, I figured this was just another spam mail which had escaped GMail's filters. Now GMail's filtering is so good that everytime something escapes their trap, it indicates a new approach by the spammers in their never-ending quest to bypass the filters. So whenever I encounter a spam message in my inbox, I have to read it to see what's this new trick up the spammers' sleeves. Thus, intellectual curiosity forced me to read what is truly one of the more bizarre mails I've ever received.

Here's the thread (start reading at the bottom). Folks with a weak constitution should avoid reading further!

From: "Deepak Sarda" <dsarda@gmail>
To: jd@yahoo
Subject: Re: Recurrent/Occult hernias

No.. this is 'dsarda' while you want 'desarda' - notice the extra 'e'?

On Tue, Sep 23, 2008 at 1:57 AM, John Doe <jd@yahoo> wrote:
> Isn't this email coming from the correct address?
> This mail is supposed to go to a surgeon in India
>
> Sincerely
>
> --- On Mon, 9/22/08, Deepak Sarda <dsarda@gmail> wrote:
>
> From: Deepak Sarda <dsarda@gmail>
> Subject: Re: Recurrent/Occult hernias
> To: jd@yahoo.com
> Date: Monday, September 22, 2008, 5:41 PM
>
> Dear John Doe
>
> I empathize with your situation. However, I don't think I can help you.
>
> But perhaps 'desarda@gmail' can help you?
>
> Cheers.
>
> On Tue, Sep 23, 2008 at 1:03 AM, John Doe <jd@yahoo> wrote:
>>
>> Dear
>>
>> Dr. Desarda,
>>
>> I have had trouble with mesh retraction/shrinkage and last year had a
>> surgeon take out and replace Dr. Goodyears prolene PHS dual layer mesh with
>> a Ultra Pro flat mesh via a laparocopic TAPP repair.  Prior to this I had a
>> lichtenstein mesh which folded up which Dr. Goodyear took out, and several
>> attempts by a Shouldice surgeon to keep the mesh flat via stitches which
>> helped but didn't work.  But this TAPP repair worked for 9 months and now
>> the pain is coming back.  I felt an odd pain which was probably the mesh
>> shrinking up.
>>
>> Can you check for any hernia and then do your repair?  Would it be necessary
>> to take out the mesh placed via a laparscopic TAPP method?  What is the
>> cost?
>>
>> Sincerely
>> John Doe
>> Seattle WA
>>
>
>

Dropbox without Gnome

Wed, 17 Sep 2008 12:22 +0800

Dropbox, the 'it just works' file syncing service, released their Linux client recently. Unfortunately, it has a heavy Gnome/Nautilus dependency. If you don't use Gnome, you can get Dropbox to work without that baggage with these steps:

Download the closed source Dropbox Linux client from http://www.getdropbox.com/download?plat=lnx.x86 (x86_64 for 64 bit)
Extract the contents and you should get a .dropbox-dist folder out of the archive. Move the folder to $HOME
Run ~/.dropbox-dist/dropboxd.

The first time you run the dropboxd daemon, a wizard will prompt you to configure the client for your machine. By default, Dropbox syncs the contents of the ~/Dropbox folder and as long as the dropboxd daemon is running, it will transparently sync that folder with your Dropbox account. To ensure that the daemon runs whenever you use your computer, just add a symlink to it in your ~/.kde/Autostart/ folder or equivalent location.

You don't actually need to do anything beyond that since it 'just works'. But if you feel like getting hold of some of the info that the Nautilus client provides, you can download this command line utility that some kind soul has written. Hopefully, someone out there is working on KDE integration even as I write this!

The only thing that I am syncing right now with my Dropbox account is my Firefox bookmarks and history information. Firefox 3 stores this data in a file called places.sqlite which lives in the Firefox user profile folder. On my work and home computers, I just moved this file to my ~/Dropbox folder and symlinked to it from the profile folder. It's been working ok for me so far without any conflicts. Let's see if this turns out to be a better solution than Weave which has never ever worked for me.

SBS NextBus: Behind the Scenes

Mon, 08 Sep 2008 18:52 +0800

I must admit that I am slightly surprised by the positive reception to the SBS NextBus app that I announced in my previous post. I thought it would languish in obscurity like most of the other toys that I've created over the years but a positive recommendation from Mr Brown turned on a firehose of traffic that, after almost a week now, still hasn't died out.

The attention also reminded me of how old this blog is now. When I started it, Sounds from the Dungeon sounded like a cool name and had some meaning to boot. Now I am old and jaded and it just grates on my nerves whenever someone writes words like, Deepak of Sounds from the Dungeon says.. It's high time I streamlined the whole antrix.net experience, getting rid of all the archaic and disconnected stuff around here, reorganizing the content into a cohesive whole and getting rid of the whole dungeon metaphor in the process.

Anyway, getting back to topic, I do hope that the traffic to the SBS NextBus app converts to a steady userbase; there's nothing like your users screaming at you to provide motivation to maintain the app!

Speaking of maintaining the app, I committed a couple of feature additions to SBS NextBus last night and with those, I think the app is basically done. The only updates it is likely to see are when the parsers break because SBS Transit decided to change something upstream. I do hope such situations would be really rare!

With the app basically done, I figured it would be worthwhile to describe here some of the technical bits involved in building the app. If you are the kind that enjoys such geekery, read on :-)

As is obvious from the domain - sbsnextbus.appspot.com - the app is running on Google App Engine. The basic application is fairly straight-forward but there are a couple of twists in its implementation that are worth noting. First of all, the core logic behind the app is dead-simple. When a user requests for bus arrival info at a stop, the app must:

Fetch page from the mobile iris site for a bus stop and parse that page for services operating at the stop.
For each service, fetch page from iris that shows arrival info at that stop.
Collate results and render output page.

I had the basic code structure and rough-cut implementation of this done in about an hour or so, most of which was spent in writing the html parsing bits which I wrote based on html served by the mobile iris site to Firefox. This was a mistake. When I actually started fetching pages from App Engine's development server, the parsers broke down because iris was serving different html to these requests! One quick way to fix this would have been to emulate Firefox's User Agent in the requests to iris. Unfortunately, App Engine's urlfetch currently doesn't provide any way of overriding the user agent identifier. So I had to basically rewrite the parsing bits to adapt to the different html. It didn't take much time but still counted as wasted effort.

Once the basic app was ready, I had to tackle what I knew would be a problem going in. For those of you who are not aware of its design, App Engine does not allow for long running requests in apps and requires that each HTTP request to your app be served within a short time. If you exceed this short time interval at your disposal, you app process will be killed and the user's request terminated. Although Google does not document the exact duration, looking at App Engine's app monitoring dashboard, I can say that this time limit is around 8 seconds, i.e. every HTTP request to your app has to be handled and a response sent back within 8 seconds. Since App Engine also forbids spawning of threads or processes, you are essentially limited to less that 8 seconds of sequential processing from request to response. For the NextBus app, if it needs to query the iris site for 10 bus numbers at a particular bus stop, all those requests have to happen sequentially and there's no telling how long each remote request would take to complete. Thus, it is quite likely that the 8 second limit is exceeded and your app process is killed for taking too long to complete. The fact that App Engine's urlfetch call does not support specifying request time outs just exacerbates the problem.

Thankfully, the App Engine environment doesn't just summarily kill your app on exceeding the alloted time quota. Instead, your app receives a DeadlineExceededError exception and is given a few more cycles to finish cleaning up before it is killed for good. So we can exploit this last chance at redemption in a manner similar to this psuedocode:

try:
    for service in services:
        if service not in memcache:
            # fetch page from iris
            # parse html and extract timings
            # add to memcache
        else:
            # fetch timings from memcache
        # add timings to http response
    #return http response
except google.appengine.runtime.DeadlineExceededError:
    # send as response a HTTP redirect to ourselves:
    # 'http://domain/stop/?number=[stop number]&random=[random int]'

So what's happening here? The core idea is that in the event that our application process receives the DeadlineExceededError exception, we save our state and redirect the user to make the same request one more time. In the subsequent request, we continue from where we left off last time and hopefully in the span of this new request, we'll be able to finish processing the request. If not, rinse and repeat!

State communication between requests can be done via url request parameters or via saving/reading from the Datastore or via memcache. For this app, I chose the memcache approach. Once a bus service's arrival timing is known, it is immediately saved to memcache with an expiry time of 1 minute since after a minute, the arrival timings would be practically useless.

Thus, this trick of redirecting to self and spreading the processing load between multiple requests allows us to effectively work around the 8 seconds limitation imposed by App Engine. However, there's one final issue to be resolved. The class of the exception thrown seems to vary with environment. After a bit of googling and testing, I ended up with something like this:

try:
    from google.appengine.runtime import DeadlineExceededError
except:
    # on the development server 
    from google.appengine.runtime.apiproxy_errors import DeadlineExceededError

try:
    # this was noticed in the logs after deployment
    from google3.apphosting.runtime import DeadlineExceededError as DeadlineExceededError2
except:
    DeadlineExceededError2 = DeadlineExceededError

# some app code

try:
    # app logic
except (DeadlineExceededError, DeadlineExceededError2):
    # serve redirect

Hopefully, the inconsistency in the exception class resulting in the boilerplate above will be fixed in a future App Engine update.

That wraps up all that I remember as worth writing about the SBS NextBus app. But before I end, a few words on the general experience of working with App Engine. After oohEmbed.com, this is the second app I've built that runs on App Engine. For these kind of apps with bare-bones functionality, using the App Engine is a no-brainer simply because the deployment is so damn easy! For the NextBus app, after I had everything working on the local development server, I just had to go online and register the app id (sbsnextbus), upload the code and visit sbsnextbus.appspot.com. The code just worked and I went from development to deployment in under one minute! The only code change I had to make subsequently was regarding the import of the DeadlineExceededError exception as noted above. Compare this to the process of configuring Python scripts to be served under cgi, fcgi or mod_python in traditional hosting environments where getting the deployment right often takes as long as developing the entire app.

From these two app building experiences, I can confidentally say that if you are writing simple Python web apps, you should strongly consider using the App Engine environment. However, this is a very qualified recommendation since I haven't yet used the Datastore and have no idea how easy or difficult it would be to adapt to that data model.

Improving SBS Transit's iris NextBus

Mon, 01 Sep 2008 15:12 +0800

If you happen to use public transport in Singapore and also happen to surf the internets on the go, chances are that you've heard of SBS Transit's NextBus information service, iris. I am a frequent user of this service and although I like it, there is one particular aspect of it that really annoys me. Allow me to illustrate.

This above is the home page of the iris service on which you enter the bus stop number & the bus service number for which you wish to know the arrival time. Although it isn't obvious from the page, you can leave the bus service field empty and submit the form. When the bus service number is not specified, you get a page that lists all the services operating at the specified bus stop. Like this:

Now once you get this page, you can click on any of the service numbers to get the estimated arrival times. Clicking on 93 above would get you:

And this is what annoys me. It is quite likely that I would be interested in more than one of these bus numbers. But to get information on multiple service numbers, I have to navigate back & forth between the service numbers listing & the individual service numbers arrival info.

To fix this annoyance, I made sbsnextbus.appspot.com. When you visit this new site, you are greeted with this page:

As earlier, you just enter the bus stop number and hit Fetch. If all goes well, the resulting page will look something like this:

There, all the arrival information in one easily accessible page! No more annoying back & forth navigation! So if you use public transport in Singapore and also happen to surf the internets on the go, please test out sbsnextbus.appspot.com and let me know how it works (or not!) for you.

Oh, here's the QR Code to the site, if your phone supports reading these digital hieroglyphics:

Update: (4/Sep/2008) Hello to everyone coming here via Mr Brown's blog! Thanks for your interest and for trying out this app! Your usage has revealed a few corner cases and bugs which I have now fixed. I've also made a few cosmetic changes suggested by Eric (in comments) and Jon (of iSinGeo.com) so that the pages display better on an iPhone.

Update: (7/Sep/2008) This app is now integrated with SinGeo as well as iSinGeo.com! Read the details on the SinGeo blog.

Update: (8/Sep/2008) The app now shows bus stop descriptions.

Update: (9/Sep/2008) Read the blog post describing some of the tech behind the app.

Update: (30/Oct/2008) New release with new features.

Computing Mysteries

Wed, 27 Aug 2008 15:11 +0800

From time to time, I'm confounded by computers. I have come to know them reasonably well over the years, at least at a superficial level if not at a more intimate level. However, they can still throw up some error, some noise or some silence that leaves me scratching my head.

Consider this: we bought a commodity server for work some time back. It's nothing fancy, just Linux running on a Core 2 Duo sitting on a Gigabyte mainboard. The curious thing about this machine is the kernel loading phase. You see, the kernel on this machine apparently doesn't like USB keyboards very much and simply refuses to boot up if it doesn't detect a PS/2 keyboard! Of course, this preference is limited only to the initial bootstrapping stage; once the init image is loaded up, I could not only disconnect the PS/2 keyboard but also shred it to bits and the kernel wouldn't care. It would just use the USB keyboard like nothing ever happened between them. So now whenever I reboot this server, I have to remember to plug in the PS/2 keyboard. Yes, the same one which I haven't shred to bits.

You can imagine what process of trial and error led me to solve the case of the spurned keyboard.

Then consider my desktop computer at home which feels like it has to compete with my work machines to get my attention. This computer doesn't even get to the kernel loading stage. It hangs before loading GRUB, the OS bootloader. The culprit? My external Seagate USB hard drive which is always connected to the PC. Apparently, the BIOS on this computer doesn't like to be connected to this external disk on boot. So I've to power off the Seagate before turning on the PC. Of course, it isn't as simple as that. If I forget to power on the Seagate before Linux starts booting, then the init scripts detect the missing disk and throw a fit. So I am left with co-ordinating this intricate boot up dance sequence:

Power off the Seagate external drive
Power on PC
Wait for GRUB to show up
Press Esc to enter GRUB menu
Power on the external drive
Wait for the beep from the BIOS indicating recognition of the drive.
Press Enter in GRUB to load Linux

Thankfully, this workaround wasn't that tricky to figure out.

Finally, there's my old IBM Thinkpad which I recently turned on after leaving it sitting in a cabinet for almost an year. Well, tried to turn on. When powered up, the Thinkpad does nothing but emit four loud beeps repeated four times in case you miss them the first, second or third time. Lenovo says this beep pattern means System board (Security chip). I am not sure that's very meaningful.

Thankfully, this kind of stuff doesn't happen often enough for me to consider alternate careers - like that MBA everyone keeps talking about. :-)

Shared stuff, APIs, feeds, oh my!

Wed, 09 Jul 2008 17:02 +0800

There's no doubt about the fact that we are generating lots and lots of content as part of our online activities. We blog, leave comments, bookmark sites, upload photos and videos, provide status updates and what not. There's so much activity that it's hard to keep track of it all. RSS/Atom feeds have helped but it's just too much work to track down, subscribe and manage a dozen or more feeds for every person that you know.

On a personal level, I've tried to provide feeds for as many of the things I create/share as possible. There's this blog, the tidbits blog, the tumblelog, photos on flickr, code commits, travel updates, status updates and more. Recently was my attempt at bringing all of these bits and pieces of info back to a single location on antrix.net. Although slick (bias!), it really isn't what a central clearing house of a person's activities should look like. As it stands, Recently is rather limited: it doesn't provide permalinks nor feeds and does it allow interaction in the form of comments, etc.

Enter stage left: FriendFeed.

I've used FriendFeed on and off for some time now, most of the it spent trying to figure out what exactly it does. I still haven't so I will not try to explain. What I will do is to tell you that as of now, I've settled on letting FriendFeed track as much of my publicly accessible info as possible. So everything I mentioned above, with the exception of twitter updates, is now available to you - dear readers - on a single FriendFeed page. Love it or ignore it!

If you are on Facebook, a smaller helping of this stuff should also appear in your Facebook news feed. Of course, only if you are my 'friend'.

There's exactly one thing that I did towards this shuffling of data that's worth noting here. Regular readers will know that I've long been using del.icio.us as a link blog. I use it so much that I even wrote Dumble (and now oohEmbed) as a friendly tumblelog style front end to del.icio.us. So del.icio.us has to remain as the definitive link archive in these quarters.

With that in mind, you can understand why I've been so slow on the uptake of Google Reader's shared items feature, although I've been using Reader exclusively since perhaps late 2006. Simply put, I didn't want to fragment my shared links under two different sources - one of which wouldn't play ball with Dumble. The horror! The only reasonable thing to do was to write some code that took shared items from Google Reader and posted them to my del.icio.us account. And so I wrote yummy. My last blog post alluded to this wee bit of code and indeed, yummy does incorporate feedback from you, dear readers. Where by readers, I mean Harish!

On the whole, I think I'm happy with the current situation. Of course, I'd much happier if we were in the year 2012 with all this drama behind us - a distant memory of less civilized times. :-)

Now, I shall go pack my bags since, as Dopplr says, I'm scheduled to be in Mumbai tomorrow!

Python: constrained containers

Fri, 04 Jul 2008 18:04 +0800

As it happens, I was playing with the del.icio.us API today because of reasons which I shall hopefully elaborate upon in the coming few days. While cooking up some code to interact with the API, I saw the need to model a basic del.icio.us post entity. For my needs, this entity wouldn't have to be much more than a dumb dictionary with keys like url, description, etc. So I started out that way but pretty soon I thought, wouldn't it be nice to be able to write post.url instead of post['url']? Further, wouldn't it be nice to constrain the keys in the dictionary to those required and prevent typo errors such as post['descrption'] instead of post['description']?

This seemed like the perfect opportunity to put into practice some of the stuff I'd learned about "new" style classes not too long ago. So I wrote the following class which implements the requirements laid out in the preceding paragraph. Since I'm just getting the hang of this stuff, please critique the code!

class Post(object):
    """Class to model a del.icio.us Post. It works like a struct/dict 
    like object which limits keys to a retricted subset, i.e. those
    used in a del.icio.us post."""

    __slots__ = ['description', 'url', 'extended', 'tags']

    def __getitem__(self, key):
        if key in self.__slots__:
            return self.__getattribute__(key)
        else:
            raise KeyError

    def __setitem__(self, key, value):
        if key in self.__slots__:
            self.__setattr__(key, value)
        else:
            raise KeyError('Given key is not allowed in class %s'\
                                % self.__class__.__name__)

    def __contains__(self, key):
        try:
            self.__getitem__(key)
        except:
            return False
        return True
    
    # urllib.urlencode() just needs this beyond the basic stuff above
    def items(self):
        return [(k, self[k]) for k in self.__slots__ if k in self]

And here's the usage of the class.

>>> p = Post()
>>> p.url
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: url
>>> p.url = 'http://google.com'
>>> p.url
'http://google.com'
>>> p['url']
'http://google.com'
>>> p['desc']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "yummy.py", line 17, in __getitem__
    raise KeyError
KeyError
>>> p['description'] = 'Google homepage'
>>> p['description']
'Google homepage'
>>> p.items()
[('description', 'Google homepage'), ('url', 'http://google.com')]
>>> p.tags = 'search'
>>> p.items()
[('description', 'Google homepage'), ('url', 'http://google.com'), ('tags', 'search')]
>>> 'extended' in p
False
>>> p.extended = 'homepage of the world'
>>> p.items()
[('description', 'Google homepage'), ('url', 'http://google.com'), ('extended', 'homepage of the world'), ('tags', 'search')]
>>>

So, what am I doing wrong? :-)

Announcing oohEmbed!

Sun, 01 Jun 2008 22:19 +0800

In my last post here, I mentioned some items on my ever growing TODO list and on top of that list was: Finish the web app that I've started working on.

Well, it's time to scratch that one off the list since oohEmbed.com is now live!

A bit of background before I get to what is oohEmbed. As you know, I've been working on and off on Dumble for the past several months. Now Dumble is a purely browser based application with antrix.net serving up just a bunch of static files. Because of this non-involvement of server-side programming and the web browser's same origin policy restrictions, there's a limit to how much smart URL-content-inferring can be achieved in Dumble. With the current design, Dumble can go only as far as the number of websites out there that support JSON APIs.

So on one side, I was toying around with adding a server-side component to Dumble's design to alleviate this problem. On the other, I was looking to kick the tires around Google App Engine. In between the two, the oEmbed specification was announced and thus was born oohEmbed.

So what exactly is oohEmbed? In the simplest terms, oohEmbed acts like a oEmbed API compatible proxy service between you (the developer) and target websites. For more details, visit the oohEmbed.com web site where I've described it a bit more with some examples.

Of course, I've re-written parts of Dumble to use this new oohEmbed service. You can find the forked version at oohembed.com/dumble/. If things go well, I'll designate this as the authoritative version and switch/redirect the version at antrix.net/dumble/ to it.

Thanks to the folks behind oEmbed for drafting the spec and to Google for creating App Engine. Although the App Engine environment is restrictive development wise, there's something to be said for a stack which allows one command - appcfg.py update myapp/ - deployments to the cloud! And it's all Python so I couldn't be any happier!

Please test out oohEmbed and let me know what you think. If you really want to please me, go build something on top of it!

TODO List

Wed, 28 May 2008 14:47 +0800

My TODO list just keeps growing. In no particular order, I have to:

Finish the web app that I've started working on
Start the web app that I've planned out
Finish processing photos and clear that backlog
Get photo backups in order
Print a few photos as an experiment
Implement a major feature that I've planned for Mitter
Finish reading the four books in various stages of completion. Perhaps only three; this one is too irritating to finish.
Buy an Xbox 360 and play GTA 4, Bioshock

And this is not counting all of the stuff from work! I'm just too lazy these days and need to get my act together.

Dumble Update #3

Wed, 28 May 2008 14:36 +0800

Alright, time for another Dumble update. For those tuning in late, Dumble is a web-app that auto-magically creates a tumblelog out of someone's delicious bookmarks. More background in this introductory post from last November.

So what's been cooking since the last Dumble update which was all the way back in December? For starters, Dumble's source code repository is now online so you can follow Dumble's progress as it happens!

Looking at the changes since December, apart from the bug-fixes and speedup related commits, I've added support for three more sites — Metacafe, Twitter and Wikipedia — while improving support for Amazon. While the Metacafe support just relies on the standard video embed widget, the Twitter and Wikipedia support leverages their respective APIs.

There are a few minor UI changes. You can find a feed subscription link to the currently viewed user's delicious bookmarks.

Finally, there's one change explicitly to support embedding of Dumble on other websites. Let's say you have a website and wish to have your own copy of Dumble running over there tumbling your delicious bookmarks. One way to do this is to just copy all the Dumble code and host it on your site. While I'm perfectly happy if you do that, there is a drawback to that approach. As and when I make any changes to Dumble, you'll have to keep up by modifying your copy of Dumble.

An easier way is to just embed http://antrix.net/dumble/ onto your site. Just copy the following HTML snippet into an index.html file and place it somewhere on your website. Be sure to modify the delicious username and optionally the tag parameters.

<html>
<head>
    <title>Dumble : auto tumble your delicious links</title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
    <style type="text/css" media="screen">
    html, body {
        margin: 0;
        padding: 0;
        border: 0;
        outline: 0;
    }
    </style>
</head>
<body>
<iframe src="http://antrix.net/dumble/?u=USERNAME&t=TAG&title=My delicious tumblelog" height="100%" width="100%" frameborder="0" scrolling="auto">
Since your browser doesn't support IFrames, please <a href="http://antrix.net/dumble/?u=Ravages">visit Dumble directly</a>.
</iframe>
</body>
</html>

Did you notice the title parameter in the iframe src URI? That's something I added just to enhance this kind of embedding of Dumble. The value of that title parameter will become the window/browser title.

You can see an example of such an embedded Dumble over on Selective Amnesia. In industry parlance, this would be called a reference installation. ;-)

As usual, please test Dumble on your browser/OS combo and let me know of any stuff that breaks. The Wikipedia support is especially hacky and I would appreciate some feedback on that. :)