Boris Smus

Backbone and Ember

2012-02-15T09:00:00Z

Backbone and Ember

Posted on Feb. 15, 2012

Ember and Backbone are both promising JavaScript frameworks but have completely different philosophies. In this post, I'll compare the two, both from a practical and philosophical perspective. I'll defer to real world experience with Backbone and SproutCore (Ember's predecessor), as well as basic experiments with Ember (full disclosure: haven't built a large Ember app yet). I'll also supplement claims with quotes from a fantastic conversation from Freenode #documentcloud on February 3rd, 2011. For quote context, wycats is Yehuda Katz, one of the lead developers on Ember, and jashkenas is Jeremy Ashkenas, one of the lead developers on Backbone.

Before I go into too much detail, it's pretty clear that both frameworks have the same goal: to help developers build apps.

1:09 PM wycats backbone apps are apps

So we have a roughly apples-to-apples comparison. Let me dive in and talk about some philosophical differences:

Backbone at a glance

Backbone is a minimalist framework that builds on top of ideas from jQuery to give some structure to web applications. It introduces concepts of views, models, a restful sync interface, routers, etc, in surprisingly little code. Backbone is the darling of microframework lovers, who highly value small framework size and don't want to commit to a single "full stack" solution.

Backbone is incredibly flexible, and doesn't impose how you views and models should actually interact. The main benefit is that it adds some structure to the app and provides convenient ways of listening to DOM events and turning them into application events.

1:15 PM wycats backbone is 600loc
1:16 PM wycats "here's how you should think about your app structure"

Since Backbone is so small, it leaves a lot of decisions up to the developer. This is both a blessing (flexibility, works for me, etc) and a curse, since support for many things is missing.

12:32 PM wycats jashkenas: backbone is 600 lines of code
12:32 PM wycats jashkenas: the idea that there are things missing in it that are common should not be controversial

Ember at a glance

Ember has a very different history. It's an evolution of SproutCore, which is a complete web application solution. Ember takes the core parts of SproutCore - two-way data binding, computed properties, tight template integration, and strips the rest off into sub-modules. Things that come in module format are a data serialization layer (via data), routing (via routes). Ember's out of the box functionality is actually smaller than Backbone's, but it provides a higher level abstraction.

I used SproutCore back in 2008 while working as an engineer on iWork.com. It was a SproutCore pre-1.0 release, and things were a bit rough. Ember seems to have come clean in many ways, presenting a more consistent template data binding solution via Handlebars, and being a lot less monolithic SproutCore once was.

Ember aims to provide a full solution in an opinionated way. Thus, to get the most of the framework, developers must do things in a certain style. While you're not forced to use Handlebars, it's the only way to get some of the compelling features the framework provides (eg. for two-way data binding). Many of the auxillary modules such as ember-data are designed to fit will into the existing structure of the framework.

1:22 PM wycats tbranyen: ember is an end-to-end framework built on top of more modular components

There is a sentiment from many people, including Backbone's founder that much of what Ember provides is over-engineered:

12:28 PM jashkenas It's Backbone's take that Ember's more complex data binding model, intermediate controllers, run loop etc. ... are all interesting approaches, but are not usually helpful in building a real site.

Different audiences

12:30 PM jashkenas wycats: and I find quite the opposite -- self-selecting sample pools, as you'd expect.

So to summarize at a high level, there are two camps (and excuse me while I generalize a lot).

Lovers of micro-frameworks. JavaScript hackers seeking extra structure in their slightly complex apps. These people are comfortable mixing and matching frameworks, solving problems as they come, and just want to get started quickly.
Software Engineers that are used to a deep abstraction layer and a full service stack. These people are probably coming from native app development and want to write very complex applications on the web.

Basically, these two developers have different needs, come from different programming cultures, and are maybe even writing different applications.

12:31 PM wycats jashkenas: so then it's fair to say that for some people, Ember's approach is overkill

Andrew says it best:

12:36 PM andrewdeandrade It's really all about values. Occasionally I get frustrated by things backbone.js doesn't do and occasionally I get frustrated by things rails does that are hard to undo. My personal preference is to have a framework not do something and implement it myself than have a framework do something and figure out how to do the opposite. That's me. I know people who feel differently

A more detailed comparison

After engineering with SproutCore, writing an app with Backbone, and writing small amounts of Ember (mostly samples to get a feel for data bindings, etc), I've got some sense of the issues that you will run into when developing a moderately complex app. I'll go over some, and supplement them with quotes from framework authors.

1:28 PM wycats jashkenas: do you disagree that the pattern "listen for these properties, and when any of them changes, trigger observers" is very common?
1:26 PM jashkenas that's correct -- you listen for changes to the source data and render computed values. not hard.
1:26 PM wycats jashkenas: yes… a pattern that happens sufficiently often that it's good to abstract

In general, Ember seeks to find common problems that developers face, and solve them in an opinionated way. Backbone, on the other hand, leaves it to developers solve their own problems in the way that works for them.

Templates

While Ember in theory lets you pick which templating engine to use, you lose a lot of benefits if you're not using Handlebars.

1:29 PM jashkenas forcing your users to use logic-less templates is incredibly constraining.
1:30 PM tomdale jashkenas: ember supports any templating language you'd like. you just don't get auto-updating
1:30 PM jashkenas right, exactly.

In practice, my last project used Mustache because it was logic-less. Such templating engines are my preference, and I'm happy to be constrained to a pretty good templating system if that means access to powerful features, cleaner application code and less boilerplate.

I don't think that Ember should be toting choice of template as a big benefit. Like any application framework, Ember is a contract between developer and framework author. You write in our style and we'll give you powerful features. In an ideal world, this lets developers focus on their app instead of dealing with middleware.

Views

view decomposition is one of the first tasks a front end developer faces. They need to decide which parts of their app will be implemented with what view. Should each list item be its own view? Should the list be a single monolithic view?

In my experience writing Backbone apps, views are very primitive and tend to cause issues. There's no support for any sort of view nesting, which is totally critical for large applications with complex UIs. In contrast, Ember provides an easy way of nesting views inside one another.

The other thing you'll notice with Backbone is that there's a lot of micro-management required when building Backbone views. They need to be properly cleaned up by hand, otherwise you end up with zombie views bound to events, or events that don't fire at all.

Some backbone projects, such as the layout manager, aim to remedy some of these limitations by creating a Layout abstraction, that allows nested views and handles a lot of the rendering. This is a very interesting project, but I haven't tried it yet.

MVC?

Ember is a traditional MVC framework, where it's clear which parts are the view, the controller and the model. Backbone on the other hand, is explicitly not an MVC. It never even claims to be!

In fact, if you read the main page, you'll notice that there's no mention of controllers anywhere. Don't get me wrong: Backbone still gives your applications structure, but has no opinion on the gluing layer between data and presentation. I think Backbone once had controllers, but they were renamed to routers, designed primarily for handling URLs and history/pushState.

In my experience, there's very much a need for a controller when writing even moderately complex apps. You're presented with several options:

Write controller code in views
Write controller code in models
Write controller code in a router
Write your own controller infrastructure

If you care about separation of concerns, none of these options are really acceptable.

Data and servers

Backbone packs a huge punch in a small package. It comes with Backbone.sync, which lets you fully customize how you want to interact with the server.

Following the CRUD pattern, Backbone lets you specify the response format of an Read, but unfortunately doesn't allow you to fully customize how you would like to serialize Create and Update and Delete payloads for calling your server-side API. For my last project, I ended up just completely redefining Backbone.sync, which is very powerful, but I needed to write a lot of boilerplate to make it work reasonably.

The problem with Backbone's approach is that it doesn't separate two parts of data stores: internal collection management and the interface with the backend API.

Ember data, on the other hand, has the concept API adapters, which let you specify the interface the server in response to the inner workings of the adapter. And, additionally it has a DataStore model. I haven't experimented with this in Ember yet, but SproutCore's version of this worked quite well.

Performance considerations

When I worked on iWork.com, we definitely had some issues with SproutCore performance for large amounts of data. The problem was that we couldn't really optimize our code because we were locked into the way SproutCore does things, so our way forward was to patch SproutCore itself to address some of the performance issues, or break out of the framework and implement the performance intensive part manually.

Jeremy voices this concern here:

1:33 PM jashkenas finally, and perhaps most importantly, embers tight coupling of handlebars-ui-with-very-specific-bindings-to-ember-models is trouble, performance-wise. You can't build the really intensive parts of your UI with that level of binding / dom tweaking.

And I agree with him. It's very legitimate concern for large apps, and Yehuda didn't answer adequately in my opinion.

1:34 PM wycats jashkenas: exposing the performance question to the user is trouble
1:34 PM wycats over the long haul Ember will be able to heuristically decide how bulky to update
1:34 PM jashkenas sufficiently smart compiler, eh?
1:34 PM wycats jashkenas: :P

I'm a bit concerned about this point and would love to hear a less handwavy answer from Yehuda.

Final thoughts (Backbone)

My conclusion from writing an app with Backbone, and attending a Bocoup training workshop, is that Backbone by itself is not sufficient for building complex web apps. You will invariably go one of two directions:

Engineer a lot of stuff on top of it, or
Use existing Backbone plugins of various maturity and hope they work well together.

Based on my experience with jQuery, and the mess of ensuing plugins, The latter seems overly optimistic. So basically, be prepared to write a lot of extra boilerplate code. But if you're a JavaScript developer, you can handle that!

In practice, it's very difficult to remain productive if you're writing both an app and a framework at the same time. Unfortunately this was my tendency when using Backbone. I hate reinventing the wheel. Especially if it's kind of lopsided.

That said, Backbone is fantastic for mid-to-low complexity applications that want to maintain structured code.

Final thoughts (Ember)

Ember, on the other hand, forces you into its way of doing things. This is a framework with opinions that gives you less flexibility. However, if you grit your teeth a little bit and buy in, you'll be exposed to a well thought out set of libraries that work well together.

I really like the features Ember offers, and its philosophy of finding common problems developers will face and solving them. One thing I anticipate is that many of the non-core Ember modules are immature. However, just having these modules that are designed to work together is a boon for serious application developers.

I'm still concerned that Ember applications may be stuck if dealing with particularly hairy custom view situations, or large amounts of data, but I'll have a better sense of the limitations soon.

PDF Conversion and Client-side Scraping

2012-02-08T09:00:00Z

PDF Conversion and Client-side Scraping

Posted on Feb. 8, 2012

One of my ongoing projects is called smusique, a mobile web-based sheet music viewer. The application was designed for tablets, enabling a rich browsing experience through sheet music. Though some sheet music is encumbered by licenses, most of the classic stuff is legally available online via sheet music databases such as the Petrucci Library at imslp.org.

While the focus of my project was to design and develop a delightful tablet-based interface for sheet music, I needed to solve some technical problems:

Figure out how to deal with PDFs in the browser.
Create or find a database with a large enough corpus of searchable sheet music.

Read on to find out how I solved these problems using the AppEngine conversion API and through a Chrome extension that does client-side scraping.

Dealing with PDFs

Since digital sheets are largely in PDF format, I needed some solution for showing PDFs inside a web user interface. Unfortunately showing PDFs inline is pretty much impossible using today's web. Ideally, the browser would have support for something like the following:

<img src="something.pdf" page="3" />

But this functionality simply doesn't exist in a widely supported fasion. Given how important showing PDFs is for my application, I needed a workaround. A few options came to mind:

Attempt to show PDFs in an iframe, and programatically scroll to the right position based on the page number.
Convert PDFs on the client side using something like PDFJS
Build my own PDF conversion server

I experimented briefly with the first approach and unsurprisingly found mixed levels of browser support for PDF rendering inside iframes. Some mobile browsers allowed PDFs to be opened within iframes, but exhibited unexpected scaling behavior, could not be programmatically scrolled, or both.

Did not seriously consider PDFJS as a solution, judging that running PDF conversion in a mobile web browser would be prohibitively slow.

Just as I was bracing to take the plunge and write my own ImageMagick-based conversion API running on Django/Slicehost, I discovered the very new AppEngine conversion API.

The conversion API

Turns out that AppEngine provides a new experimental file format conversion API. It allows you to map from a variety of input file formats to a variety of outputs, including PDF to PNGs (one per page), which is exactly what I wanted. The full list of conversion paths is available in the docs. Some of the more exciting features include image to text conversions via OCR, and generating images from HTML.

My conversion was really straight forward to implement:

def convert_pdf(self, pdf_data):
    """Converts PDF to PNG images. Returns an array of PNG data."""
    asset = conversion.Asset('application/pdf', pdf_data, 'sheet.pdf')
    conversion_request = conversion.Conversion(asset, 'image/png')
    result = conversion.convert(conversion_request)
    if result.assets:
        return [asset.data for asset in result.assets]
    else:
        raise Exception('Conversion failed: %d %s'
                % (result.error_code, result.error_text))

Note that the API methods recently changed (maybe in AppEngine 1.6?) from conversion.ConversionRequest to conversion.Conversion. I should have expected breaking changes since the conversion API is still experimental, but it stumped me for a little while anyway.

The API works pretty well, but doesn't provide much meaningful feedback if running in the dev server. So far I've only managed to overload the conversion service a few times with very large (eg. ~100 page) PDFs. That said, a lot of sheet music is incredibly long, especially in an orchestral setting. So, if I was doing this for production, I would probably need to build a custom solution.

One other caveat with this approach is that I have no idea how much money I would be charged for using this service. Conversion is quite intensive, and given the recently increased rates, I would be wary.

Once the images are converted, I upload them to an Amazon S3 instance where I keep my data. To do this, I use S3.py, a really simple library for interacting with Amazon S3:

def upload_helper(self, path, data, contentType):
    conn = S3.AWSAuthConnection(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
    options = {'x-amz-acl': 'public-read', 'Content-Type': contentType}
    response = conn.put(DEFAULT_BUCKET, path, S3.S3Object(data), options)
    return URL_FORMAT % {'bucket': DEFAULT_BUCKET, 'path': path}

The full code for the AppEngine server is open source on github.

Client-side scraping

Unfortunately IMSLP doesn't provide a useful API out of the box. I wasn't ambitious enough to create a full API for it, but needed some interim solution. I figured that for a demo, it wouldn't be such a terrible experience to have to seed the database with the repertoire you were interested in if it was easy enough to do.

Ordinarily, one might write a little scraper utility in their favorite scripting language. However, scraping HTML from python (or any other scripting language for that matter) is really not my favorite activity. Additionally, relying on the command line excludes the target audience (musicians) for this application.

As a workaround, I came across a potentially interesting idea: client-side scraping with a Chrome extension. Let me explain.

Suppose you want to scrape part of a corpus of data that's available through a website, but want to let the user decide which parts they are interested in. Simply use a Chrome extension that injects code into the target page, fetches the interesting part of the DOM using selectors and perhaps jQuery for convenience, and then uploads the data to some server. I used content scripts for this purpose. In the manifest, the entry looks as follows:

"content_scripts": [{
  "matches": ["http://imslp.org/wiki/*"],
  "js": ["imslp.js"],
  "css": ["imslp.css"]
}]

Then, imslp.js does the scraping, which can happen automatically as a user navigates through a page, or by adding extra elements to the page. This simple IMSLP scraper creates a "send to smusique" button beside each PDF on IMSLP:

Once clicked, imslp.js gets the URL to fetch, extracts all of the meta data of the current piece via CSS selectors, and makes a cross-domain request to the converter with all of this information.

This approach is much harder to prevent than traditional, server side scraping. Although setting UserAgent restrictions is futile, many sites use JavaScript to render their content, and serve up a content-free page. In contrast, there's very little a content provider can do to protect themselves from client side scraping, making this a very powerful technique. If the content is in the DOM, it can be extracted.

The source of the extension is somewhat messy but available also on github.

A mobile web application tech stack

2012-01-18T09:00:00Z

A mobile web application tech stack

Posted on Jan. 18, 2012

There are many technical decisions to make when writing web applications. I've come back to writing modern web applications lately, and wanted to consolidate some scattered thoughts that I’ve recorded over the course of my development cycle. In particular, this post is about the set of frameworks that I found to be instrumental in developing my most recent project. I'll go over some of the most important framework types, each of which could be expanded into an article in its own right. This is not meant to be an extensive comparison of existing offerings, just a slice of technologies that I experimented with most recently.

Although my focus is on mobile first, I think that this set of technologies can be applied to web apps in general. All of my decisions and data points were made with a few requirements in mind:

JavaScript only (CoffeeScript, Dart, are definitely worth a serious look, but cause an explosion in choice which I wanted to avoid)
Must work well in modern browsers (iOS 5, Android 4)

Picking an MVC

The model view controller pattern has been in use in native UI app development for decades. The basic idea is to separate the data layer (storage, communication, data) from the presentation layer (UI, animation, input). There are other similar patterns such as MVVM (Model View ViewModel), but the main idea is to have well-defined separation between the presentation and data layers for cleaner code and ultimately long-term velocity:

There are tons of offerings of JavaScript model view controller frameworks. Some, such as Backbone.js and Spine.js are written in pure code, while others like Knockout.js and Angular rely on DOM data attribute binding. Relying on HTML5 data DOM attributes feels wrong for an MVC system, whose whole point is to separate view and data. This excludes the Knockout and Angular frameworks. Spine.js is easier with CoffeeScript, which we exclude based on my initial requirements.

Backbone.js has been around for longer than most (except perhaps JavaScriptMVC, seems like a dead project), and also features a growing open source community. For my app stack, I went with Backbone.js. For more information about picking an MVC, check out TodoMVC, which implements the same Todo application using different MVC frameworks. Also see this MVC framework comparison, which strongly favors the Ember.js, a relative newcomer to the scene. I haven’t yet had a chance to play with it, but it’s on my list.

Picking a templating engine

To build a serious application on the web, you inevitably build up large DOM trees. Rather than using JavaScript APIs to manipulate the DOM, it can be much simpler and more efficient to write HTML using a string-based template instead. Generally speaking JS templates have evolved to use this at-first strange convention of embedding the template content inside script tags: <script id="my-template" type="text/my-template-language">...</script>. The basic pattern of use for all template engines is to load the template as a string, construct template parameters and then run the template and parameters through the templating engine.

Backbone.js depends on Underscore.js, which ships with a somewhat limited templating engine with verbose syntax. There are other options available, including jQuery Templates, Handlebars.js, Mustache.js and many others. jQuery Templates have been deprecated by the jQuery team, so I did not consider this option. Mustache is a cross-language templating system, featuring simplicity and a deliberate decision to support as little logic as possible. Indeed, the most complex construct in Mustache is a way to iterate an array of objects. Handlebars.js builds heavily on Mustache, adding some nice features such as template precompilation and in-template expressions. For my purposes I didn’t need these extra features, and chose Mustache.js as my templating platform.

In general, my impression is that the existing templating frameworks are quite minimal and comparable in features, so the decision is largely a matter of personal preference.

Picking a CSS Framework

CSS frameworks are essential tools that extend CSS’s feature set with conveniences such as variables, a way to create hierarchical CSS selectors, and some more advanced features. This essentially creates a new language: an augmented version of CSS (let’s call it CSS++). For development ease, some frameworks implement a JavaScript CSS++ interpreter in the browser, while other frameworks let you monitor a CSS++ file and compile it whenever there are any changes made. All CSS frameworks should provide command line tool to compile CSS++ down to CSS for deployment.

As with templating languages, there are many choices all of which do very similar things. My choice was motivated by personal syntax preference, and I prefer SCSS because it avoids weird syntax like @. One drawback of SCSS is that it doesn’t ship with a JavaScript interpreter (there is an unofficial one that I haven’t tried), but does come with a command line watcher. Other similar CSS frameworks include LESS and Stylus.

How to layout views

HTML5 provides a variety of ways to layout content, and MVC frameworks provide no opinion about which of these layout technologies to use, leaving the sometimes difficult decision to you, the developer.

Generally speaking, relative positioning is appropriate for documents, but falls apart for apps. Absolute positioning should be avoided, as should tables, clearly. Many web developers have turned to the float property to align elements, but this is suboptimal for building application views, since it’s not optimized for app-like layouts, which results in many odd problems and infamous clearfix hacks.

After much experimentation with various web layout technologies over the years, I think that a combination of fixed positioning and flexbox model is ideal for mobile web applications. I use fixed positioning for UI elements that are fixed on the screen (headers, sidebars, footers, etc). The flex box model is great for laying out stacked views on the page (horizontally or vertically). It’s the only CSS box model explicitly optimized for interface design, quite similar to Android’s LinearLayout manager. For more information about the flexbox model, read Paul's article and note that this spec is being replaced by a new, non-backwards compatible version.

Adaptive Web Apps

One final section on this matter: I’m a strong proponent of creating device-specific user interfaces. This means re-writing parts of your view code for different form factors. Luckily, the MVC pattern makes it relatively easy to reuse a single model for multiple views (eg. tablet and phone).

Flipboard for iOS demonstrates this idea very well, giving tablet and phone users a highly tailored experience for each device form factor.

The phone UI is optimized for vertical swipes, allowing single hand use.

Tablet UI works well for two hands holding the device on opposite sides.

Input considerations

On mobile, the main way users interact with your application is by touching the screen with their fingers. This is quite different from mouse-based interaction, since there are 9 additional points to track on the screen, which means developers need to move away from mouse events when writing mobile apps. In addition, mouse events on mobile have the problem of clicks being delayed by 300ms (there is a well-known touch-based workaround). For more information about using these events in mobile browsers, see my touch events article.

It’s not enough to just s/mousedown/touchstart/ all of your event handlers. There is a completely new set of gestures that users have come to expect on touch devices, such as swipes to, for example, navigate through lists of images. Though Apple has a little-known gestures API, there is no open spec for doing gesture detection on the web. We really need a JavaScript library to do gesture detection, for some of the more common gestures.

How to make it work offline

For an app to work offline, you need two things to be true:

Assets are available (via AppCache, Filesystem API, etc)
Data is available (via LocalStorage, WebSQL, IndexedDB, etc)

In practice, building offline apps on the web is a difficult problem. Generally speaking offline functionality should be built into your app from the beginning. It’s especially difficult to offline-ify an existing web application without significant code rewriting. Additionally, there are often unknown storage limits for various offline technologies, and undefined behavior for what happens when those limits are exceeded. Finally, there are problems with technologies in the offline technology stack, most notably AppCache, as I outlined in a previous post.

A very interesting approach to write truly offline-capable apps is to go “offline first”. In other words, write everything as if you have no internet connection, and implement a syncing layer that synchronizes data when an internet connection exists. In the Backbone.js MVC model, this can fit nicely as a custom Backbone.sync adapter.

Unit testing

It’s hard to unit test your UI. However, since you’re using an MVC, the model is completely isolated from the UI and as a result, easy to test. QUnit is quite a nice option, especially because it allows to unit test asynchronous code using it’s start() and stop() methods.

Signing off

To summarize, I used Backbone.js for MVC, Mustache.js for templating, SCSS for a CSS framework, CSS Flexbox to render views, custom touch events and QUnit for unit testing to write my mobile web application. For offline support, I’m still experimenting with various technologies and will hopefully follow up with more information a future post. While I strongly believe in the need for each class of tool (eg. MVC) outlined here, I also believe that many of the specific technologies I described here are interchangeable (eg. Handlebars and Mustache).

One more thing: yesterday (on January 17th, 2012), Thorax was announced. This is a Backbone-based set of libraries very similar in spirit to what I describe in this post. I've yet to investigate it in any depth, but the name is great :)

Use a similar set of frameworks? Have a personal favorite? Think I’m missing an important type of framework? Let me know!

Hybrid Operating Systems

2011-11-28T09:00:00Z

Hybrid Operating Systems

Posted on Nov. 28, 2011

I've been thinking a bit about native and web applications, and how they can (or can't) coexist in the future. This topic has been steeping in my head for months now, so here is a brain dump of some of my thoughts.

On hybrid apps

Hybrid apps embed a browser, using a web view for part of their UI. This is a very flexible definition for a very flexible beast. At one extreme are native apps that embed a Web View to render a small widget (written in HTML/CSS/JS) in the UI. On the other hand, you could have a web application that provides just a native frame around the content which is entirely implemented as a web app. The hybrid app spectrum looks something like this:

native <--------------- hybrid apps ---------------> web
Mail.app                GMail for iOS              GMail

Pieces of the web stack have long been useful as building blocks for applications, but lately there has been a bloom in features under the HTML5 moniker. As a result, many developers are gravitating towards the right end of the hybrid app spectrum.

There are several frameworks which target this niche by providing thin native frames, such as PhoneGap and Fluid. There are three benefits to these frameworks, helping developers to:

Make money: capitalize on AppStore and Market earnings.
Use more features: get access to features not available from the web.
Provide good platform-specific native integration.

The first two are relatively well understood, so for the purposes of this post, I'm more interested in the third. Android PhoneGap apps are launched from the home screen, Fluid Mac apps from spotlight or your dock, etc. Then you can switch between these apps as if they were regular OS X applications.

Even though the browser is a single app, it runs tons of applications like GMail, twitter, etc. The browser allows people to do two fundamentally different things: view content on the web (sites), and do things, like listen to music, play games and create documents (apps). For a deeper discussion on the distinctions between web apps and web sites, check out James Pearce's article Of Sites and Apps.

A pure web application does not rely on platform-specific native code and runs inside the browser, but still has to rely on a native frame solution to reap the native integration benefit frameworks like PhoneGap and Fluid provide.

Hybrid operating systems

Some browsers today have an explicit notion of apps. Without the proper OS integration, this is confusing. Now your device (laptop, tablet, phone, tv) has apps, one of which is a browser, and the browser has apps too. So to launch an app, users have to open the browser, and then launch what they want. This reinforces the distinction between web and native apps in people's heads and makes for a very inelegant solution.

One approach is for the OS to make the browser special in the operating system, allowing it also to manage installed web applications. Another approach is for operating systems to implement their SDKs using HTML, CSS and JavaScript, but with non-standard APIs specific to the platform, such as WebOS. The extreme of this approach is Chrome OS, where browser and OS are indistinguishable.

Like apps, operating systems vary in how much they embrace web applications. Here is an interesting spectrum to think about:

web agnostic <------------ hybrid OS ------------> apps are webapps
iOS           Windows 8              Web OS               Chrome OS

Let me define what I mean by "hybrid OS". A hybrid OS is aware of the presence of not just native, but also web applications, and provides ways of managing web apps, possibly alongside native ones. Windows 8 is an example of a hybrid OS, employing the new Metro UI, while also supporting Windows 7-style UI. Palm's Web OS, has no native mode at all (except via PDK plugins), and all apps are web apps (though they require the use of a special set of JavaScript libraries).

Web apps today are actively used, in some cases, even surpassing native apps in popularity. Unfortunately, users of web apps are stuck in their browser which is confusing and limiting, given the current landscape of native applications. Hybrid operating systems make explicit the idea that in the end, web apps are just apps.

Future

The second goal of PhoneGap is for the project to cease to exist. This is not a nihilistic sentiment... -- Apache Callback Proposal

My goal is for web apps to become compelling enough to force OS creators to hybridize their platforms. In other words, I'd like to see rightward movement in both the app and OS spectrums. As more hybrid operating systems emerge, we get closer to PhoneGap's second goal.

Stack Exchange question notifier

2011-11-15T09:00:00Z

Stack Exchange question notifier

Posted on Nov. 15, 2011

I've been a member of the Chrome developer relations team for the last 9 months, where one aspect of my job is developer support. I actively monitor the chromium-extensions@chromium.org list, answering questions about developing Chrome extensions. Thankfully, there are many others that help me in this endeavor, both from the Chrome team and from within the developer community itself, notably a certain PhistucK, the current record holder for the number of messages posted to the list.

Over the last years, another channel of questions has emerged: Stack Overflow (SO). SO is a great Q&A community for software developers, providing many advantages over ordinary discussion groups. Google has recognized this and sponsored SO tags, such as google-chrome, and google-chrome-extension. As I outlined in an mail to the group, we are now including SO as an official Chrome extension developer support channel.

As part of optimizing the support workflow, I wrote a Chrome extension which monitors questions with certain tags on SO and other Stack Exchange sites. This extension implements a browser action which gets badged with the number of unreviewed questions.

The extension UI uses the same visual style as other Google apps, and blends well into Chrome. I think it works reasonably well from a design perspective, both in the options page and in a browser action:

If you're curious to see how the extension is implemented, the source is available free of charge! This extension was implemented in Closure-style JavaScript.

If you support developers on Stack Overflow, please install it, give me constructive feedback and consider monitoring the google-chrome-extension tag :)

UIST 2011 Greatest Hits

2011-10-20T09:00:35Z

UIST 2011 Greatest Hits

Posted on Oct. 20, 2011

I went to UIST 2011 in Santa Barbara and presented our research on CrowdForge.

Here's a sample of some of the great work that was presented this year, in the 3 research areas that interest me most: crowdsourcing/human computation, mobile physical computing, and music.

Crowdsourcing

Notable work in both workflow-oriented approaches (Jabberwocky, CrowdForge, PlateMate) and synchronous collaborative approaches (Crowds in seconds, Collabode, Real-time).

PlateMate

Presents a workflow-based crowdsourcing nutritional analysis from food photographs.
Implemented in Django/Python, same as CrowdForge
Found out about CrowdForge 90% of the way into the research.

Real-time crowd control

Nice research from Jeff Bigham's group.
Compares different strategies of merging input from different users.
Reminded me of my twitter mindstorms project.

Crowds in two seconds

Great research from MIT crowdsourcing folks (Michael Bernstein, Rob Miller)
Retainer: keep workers "on tap" - ready to work by paying $0.30/hour.
Rapid refinement: crowd algorithm to narrow search space to accelerate.
Cool application: take movies, crowdsource the best moment.

The Jabberwocky programming environment

Really ambitious MTurk like project.
Combine different worker types from different spheres (social groups, paid workers, machines)
Full runtime stack (Dog high level language, ManReduce low level language, Dormouse runtime)
Great potential to mix different spheres of workers, but a rather intimidating project... scope creep!

Real-time collaborative coding in a web IDE

More awesome work from MIT (Max Goldman, Greg Little, Rob Miller)
Web-based Java EtherPad editor for collaboration
Main problem: others leave code in semi-working state. When to sync?
Idea: automatic error-aware integration. Auto-sync code when it compiles (or - tests pass)

Mobile physical computing

Interesting work in mobile virtual and projected UI spaces.

PocketTouch

Great work from Microsoft Research to let you use your phone through fabric (while in the pocket)
Since orientation in-pocket is unclear, there's an orientation setting gesture (not sold on this)
Unfortunately typical front pocket jeans work rather poorly.

Multi-user Interaction with Handheld Projectors

Most projector-based systems require a fixed place. This one is fully mobile.
System produces visible projection (for image) and invisible projection (for data)
Camera tracks location of nearby projections
Cool applications: virtual boxing, content transfer, 3D model viewing.

To get a better sense of the project, take a look at this video.

Imaginary phone

Lets you control a mobile phone device without taking it out of your pocket.
Maps iPhone controls to the palm of your hand similar to Guidonian hand
Touchscreen progression: fingers replaced styli. Next step: palm replaces phone (bit of a stretch)

OmniTouch

Shoulder mounted depth camera/projector.
Works for all sorts of surfaces. Heavy math for tracking and projecting.
Tracks "hover" (haven't seen this notion before) and "touch" states.

Music

I had the pleasure of hearing Ge Wang speak about some of his older projects, including Ocarina, Leaf Trombone, the Stanford Mobile orchestra, and others. Notably I hadn't really seen the ChucK language in action, and would be interested in seeing if the Web Audio API could support this sort of thing. Browser-based ChucK, anyone?

The last piece of research that I really enjoyed was called "onNote: Playing printed music scores as a musical instrument". The idea here was to use OMR techniques for markerless tracking of sheet music using positioning and using a finger for pointing. The other neat application here was to support compositional remixing by literally cutting up sheet music and splicing it back together.

This was my first UIST, and though I'm unlikely to have new research to submit for UIST 2012, I'm seriously considering going anyway, just to stay on top of the great work that this vibrant community generates.

Screen video capture for Chrome OS

2011-10-04T09:00:00Z

Screen video capture for Chrome OS

Posted on Oct. 4, 2011

This post is about video capture in Chrome that doesn't rely on any external dependencies like Flash (no fun), NPAPI (not supported on Chrome OS) and Native Client (not yet supported on Chrome OS).

I take screenshots all the time for bug reporting, image editing, etc. On OS X, this functionality is conveniently built in, and available through Command - Shift - 4. As a web denizen, I find it very useful to auto-upload these captures to a remote server, so I wrote this minimal image uploader which replaces the default behavior on OS X to capture the screenshot, and also uploads it to a picture hosting service.

Taking video capture of various UIs is also immensely useful for showing demonstrations, complex interactions, and subtle bugs. I recently re-discovered that QuickTime on OS X comes with this functionality built in. Prior to that I used (paid) ScreenFlow, which also has very nice dimension cropping and time dilation features.

What if we're on a web-only device, such as a Chromebook running Chrome OS? There is a still screenshotting API, but capturing video is less trivial. I've released an extension that captures and play backs video captures inside Chrome, and also lets you share stills to Picasa (using the OAuth 2 extension library). It's available on the webstore, and the source is on github. Read on to learn how it works, and see how you can help.

Screenshots in Chrome

Chrome provides the captureVisibleTab extension API for taking a screenshot of a tab. It requires host permissions on the page, but as usual the permission will enable the API across all pages (with some exceptions). A few successful extensions, such as Awesome Screenshot, use this API and allow cropping, annotation and sharing of screen grabs.

What if you want to capture video of a tab? Chrome provides no pre-existing API for this purpose, however, we can piggyback on the still screenshot API, executing it repeatedly from the background page for every frame we want to capture:

var images = [];
var FPS = 30;
var QUALITY = 50;
timer = setInterval(function() {
  chrome.tabs.captureVisibleTab(null, {quality: QUALITY},
    function(img) {
      images.push(img);
    });
}, 1000 / FPS);

As we capture, we store the base64-encoded strings representing video frames in an array. Once we're done capturing, we can simulate video playback by rapidly swapping the images in and out:

var background = chrome.extension.getBackgroundPage();
timer = setInterval(function() {
  if (currentIndex >= images.length - 1) {
    pause();
    return;
  }
  setIndex(currentIndex + 1);
  updateSliderPosition();
}, 1000 / background.FPS);

This approach turns out to be surprisingly efficient, with the extension being able to capture at 30 FPS on a MacBook Air, and 10 FPS on a Chromebook without too much noticeable slowdown.

Note that we rely on a fixed FPS for ease of implementation, however one could imagine using requestAnimationFrame and tracking the variable frame rate so that the playback speed is reasonable. However, there are definitely precision issues with JavaScript's timers, so this is a much more challenging approach.

So we can capture and playback videos inside the browser, but getting it out of the browser is another matter entirely. As a temporary measure, my colleague Ido Green built a screen stitching service which encodes multiple images into a movie using ffmpeg. Ideally, of course, we would encode in the browser. Perhaps a JavaScript video encoder could be implemented, though the performance may be too poor for practical use. Alternatively, a ffmpeg Native Client-based approach might be suitable, especially given that ffmpeg has already been ported.

Free ideas

There are a few logical next steps for this sample. As already mentioned, encoding video in the browser is a top priority, but there are a slew of other interesting directions, some of which can be seen as features, and others as separate products.

The captureVisibleTab API doesn't track the mouse cursor. This could be done by injecting an overlay onto the current page and tracking mousemove and click events. This data could then either be drawn onto a canvas context, or encoded separately as mouseData, and then drawn with JavaScript at playback time.
Cropping the video dimensions, modifying the video time schedule (speedup, slowdown, truncation) and annotation are all desired video-editing class features that could be implemented by treating images as canvases.
A compelling use case for this technology is creating screen sharing sessions for demos and presentations. Thus, it would be very useful to stream the video to a server, and broadcast it to multiple clients in real time. Binary websockets are now available in Chrome, and this could be a great application.
Audio annotations on screen captures make perfect sense, and are widely supported by desktop applications. APIs for sound capture have been a long time coming, but finally we may have an answer via the WebRTC ecosystem, and the getUserMedia call.

By the way, I've switched to exclusively using Markdown for all of my published writing, and wrote an markdown preview for Chrome to make my life a bit easier.

Loading Large Assets in Modern HTML5 Games

2011-09-22T15:00:36Z

Loading Large Assets in Modern HTML5 Games

Posted on Sept. 22, 2011

HTML5 games are here today, and rapidly increasing in complexity. Impressive demos are everywhere, and prominent titles like Gun Bros and Angry Birds prove that it's possible to create competitive gaming experiences in the browser. Games like these are possible thanks largely to the modern web stack which includes WebGL, the Web Audio API, Web Sockets and others.

Often forgotten, however, is the less sexy story of loading game assets. As the web platform progresses and allows for increasingly complex games, game assets (ex. textures, movies, music and images) grow in size and number, and asset management becomes a sticking point for game developers.

Let me share with you some truths:

Modern games require gigabytes of assets (textures, movies, etc)
Gamers don't like waiting for their game to load
Browser gamers want to be able to play regardless of internet connectivity

"But wait," you say, "I know! Just use the Application Cache and yer done!". Not so fast, dear reader... As described below, there are problems with this approach, and I propose some solutions.

Problems with Application Cache

So you've started implementing your awesome asset loading solution using AppCache. The good news is that there are some useful tools to help you debug if you have taken this difficult route:

You can get basic information about the site's app cache through the Developer Tools' resource panel.
You can view (and remove!) caches stored in Chrome by navigating to chrome://appcache-internals/.

But let me be blunt: AppCache is annoying to deal with. If you've made a small error in your cache manifest file, you'll quickly hit a brick wall. I ran into an issue where I forgot to include a NETWORK: fallback clause, and wasted hours trying to figure out why all of my XHRs were responding with status 0.

Part of what makes AppCache difficult to debug is its very limited JavaScript API. Aside from letting you inspect the status of the entire cache with window.applicationCache and the updateready event, AppCache doesn't give us much to work with. There's no way to tell if a particular resource we're dealing with is cached or not and no programmatic way of clearing the cache.

AppCache takes a fully transactional approach to asset loading. Either the cache is fully loaded, or fully unloaded. Compounding this issue, it's impossible to resume the download of an AppCache. Thus, if you have a large amount of assets, your user will have to wait a long time for everything to be loaded, and if they reload, they will need to restart their cache download.

Lastly, you can only include one cache manifest per page, making it impossible to group assets into multiple bundles. There are hacks that use multiple iframes with different cache manifests to work around this limitation (used in Angry Birds), but these are ugly!

Ultimately, what we need is a well-thought-out Application Cache enhancement or replacement. Given how quickly web standards bodies move, I've started thinking a bit about a transitional solution.

Designing a game asset loader

An ideal asset loading solution requires some of these features:

Granular asset loading. Load all, in groups, or individually.
No asset size limits.
Offline capability.
Programatic control over assets.

It makes sense to group assets in bundles and let the loader take care of the details. We can even create a custom manifest format, for example, in JSON format:

{
  "assetRoot": "./media/",   // The root of the assets.
  "bundles": [{
    "name": "core",          // A bundle definition.
    "contents": [            // The contents within.
      "theme.mp3",
      "loading.jpg"
    ]
  }, {
    "name": "level1",        // Multiple bundles defined.
    "contents": [            // Note: order implicit since bundles
      "L1/background.jpg",   // objects are stored in an array.
      "L1/blip.wav"
    ]
  }, {
    "name": "level2",
    "contents": [
      "L2/intro.mov"
    ]
  }],
  "autoDownload": false      // If true, download all in order.
}

With this manifest format in mind, sample API usage might look like this:

// Load the asset library.
var gal = new GameAssetLoader('/path/to/gal.manifest');

// Read the manifest and other good stuff.
gal.init(function() {
  // When ready, download the bundle named 'core'.
  gal.download('core');
});

// When the core assets are loaded.
gal.onLoaded('core', function(result) {
  if (result.success) {
    // Show a loading indicator.
    document.querySelector('img').src = gal.get('loading.jpg');
  }
});

// Check the progress of the download.
gal.onProgress('core', function(status) {
  console.log('status:', status.current/status.total, '%');
});

Note that although I've been using the name Game Asset Loader, this approach can be used for loading any large non-game assets, such as for example, a video or photo gallery.

Implementation details

Luckily, the modern web stack enables us to create a custom solution to address all of these requirements. By leveraging technologies such as the HTML5 Filesystem API or Indexed DB, we have programmatic access to a storage mechanism that we can use to build an asset loader described here.

I used the Filesystem API to implement a version of the asset loader. The code requests a large amount of persistent storage using the Quota API, which is undocumented, but works anyway:

// Get quota.
storageInfo.requestQuota(window.PERSISTENT, quota,
  onQuotaGranted, onError);

// Callback when the quota API has granted quota
function onQuotaGranted = function(grantedBytes) {
  // Save grantedBytes in the adapter
  that.grantedBytes = grantedBytes;
  // Once quota is grantedBytes, initialize a filesystem
  requestFileSystem(window.PERSISTENT, grantedBytes, onInitFS, onError);
};

// Callback when the filesystem API has created a filesystem.
function onInitFS = function(fs) {
  // Create a directory for the root of the assets.
  fs.root.getDirectory(ROOT_DIR, {create: true}, function(dirEntry) {
    that.root = dirEntry;
  }, onError);
};

The approach fetches assets with XMLHttpRequest, and stores them in the filesystem. All files in the filesystem are accessible via the filesystem:// schema, and can be used as any other resource. This filesystem URL is returned by the library in the get(path) call.

Note that the writable HTML5 filesystem API is currently available in Chrome only, but that it's quite possible to use IndexedDB (supported in Firefox and IE10) as the data store.

Usage scenarios

The following section briefly describes what the game asset loader (GAL) does in several scenarios.

Player goes to game.com which uses the game asset loader. The game calls gal.download('core') to download core assets and gal.download('level1') to load the first level into the player’s filesystem. While the core bundle loads, the game displays a loading indicator. Once core is loaded, the game displays the main menu. As soon as the first level is loaded, the "Play now" button is enabled. As the player plays, the GAL downloads more of the levels in the background.

Next time, the player tries playing offline. He goes to game.com, whose code is cached via AppCache, and loads GAL again. This time GAL knows it’s offline, looks up its manifest stored on the filesystem and doesn’t try to download new assets. The old assets still work though.

Player is still offline, making good progress, and beats level 5, but there are no assets downloaded for level 6. Luckily, before starting each level, the game calls gal.download('levelBundle') to make sure that the contents of that bundle are downloaded. The callback returns an error and the game displays an error telling the player that he needs to be online to download the next level.

So the player goes online and tries again. GAL re-downloads a manifest. Next, GAL tries re-downloading every asset that the JS requests. Luckily most of these assets are still in the browser cache, and won't be re-downloaded. The loader then saves all of the assets in the filesystem, clobbering old files indiscriminately. (This is bad, and needs to be fixed. Read on!)

Future work

In particular, re-downloading every asset while online is not desirable behavior, and we can't always rely on the browser cache for this. For smaller files, we can probably rely on ETag and Last-Modified headers and hopefully the browser won't re-download the files. However, the asset loader will still overwrite the asset in the filesystem, even if it's unmodified. This needs to be fixed. Large files are not likely to be cached by the browser, so we will need more intelligent caching built into the asset loader itself.

There are other edge cases that need to be considered, such as what happens when an asset is removed from a manifest. Ideally if this occurs, it should also be removed from the filesystem, but this is not currently implemented.

I'm happy to release the source under the permissive Apache 2 license and provide unit tests and a sample project for your perusal. It's well documented and should be reasonably easy to understand. I've also made provisions to separate the core library interface from the Filesystem-based implementation, making it even easier to implement an Indexed DB adapter.

Before I go, let me reiterate that this library isn't quite production ready, but a step in the right direction for facilitating real games on the web. Please comment below if you have feedback on the idea, or are using the library to write a game of your own!

Developing Multiplayer HTML5 Games with Node.js

2011-08-30T09:00:36Z

Developing Multiplayer HTML5 Games with Node.js

Posted on Aug. 30, 2011

One day I had some friends over at my house introducing me some cool iPad games. One of the games was Osmos, developed by an Canadian indie studio called Hemisphere Games. You control a little blob that floats in 2D space, and the only thing your blob can do is shoot pieces of itself in a given direction, which propels it in the opposite direction. The rules of the game are simple, the main rule being that when two blobs collide, the larger one will consume the smaller one. The rest of the rules pretty much follow directly from conservation of mass and energy. See for yourself - it's way better than it sounds!

Osmos really caught my attention because of its simple but engaging gameplay, meditative pace and distinct lack of multiplayer support, which struck me as a potentially very interesting problem to tackle. And so, osMUs (mu for multiplayer) was born as a browser-based multiplayer Osmos clone.

How it works

When a browser navigates to the osmus landing page, the server sends the new client the current state of its universe, which is composed of blobs with randomized velocities. At this point, the client can passively watch the game progress, but of course, can also join the game as a player controlled blob. Once a player joins, he can click or tap (on mobile devices) the canvas to shoot off a new blob.

As the game progresses, the server decides when someone (possibly one of the autonomous blobs) is victorious, at which point, players are notified and the game is restarted.

The rest of this post is about some development-related details So if you just want to try it out, go ahead. Note, however, that Osmus works on Chrome Stable (version 13) and iPad.

Game architecture

I wrote osmus to be split into distinct, loosely coupled components both to make the codebase more approachable for other contributors, and to make it easy to experiment with interchangeable technologies.

Osmus uses a shared game engine that runs in both the browser and on the server. The engine is a simple state machine whose primary function is to compute the next game state as a function of time using the rules of physics defined within.

Game.prototype.computeState = function(delta) {
  var newState = {};
  // Compute a bunch of stuff based on this.state
  return newState;
}

This is a pretty narrow definition of a Game Engine. In the game developer world, what's typically meant by a game engine may include anything from a renderer, sound player, networking layer, etc. In this case, I've made very clear divisions between these components, and the osmus game core only includes just the physical state machine, so that both client and server can compute the next states and be reasonably synchronized in time.

The client is composed of three main components: a renderer, input manager and sound manager. I built a very simple canvas-based renderer that draws blobs as red circles, and player blobs are green ones. My colleague Arne Roomann-Kurrik wrote an alternative three.js based renderer with some epic shaders and shadows.

The sound manager handles playback of both sound effects and background music (taken from 8-bit Magic). The current implementation uses audio tags, with two <audio> elements, one for the background music channel, and one for the sound effects. There are known limitations of this approach, but given the modularity of my implementation, the sound implementation can be swapped out for one that uses Chrome's Web Audio API, for example.

Finally, the input manager handles mouse events, but can be replaced with one that uses touch instead, for a mobile version. In the mobile context, it will likely make sense to use CSS3 transformations instead of canvas, since CSS3 is hardware accelerated on iOS, while HTML5 canvas still isn't, and WebGL is not implemented.

Speaking of mobile, I was happily surprised that osmus works pretty well on iPad, especially on an iPad 2 running the latest iOS version. This is really great, and one of the tangible benefits of writing games for the open web.

Networking is hard

From a networking perspective, a game is a rather ambitious project that requires seamless real-time synchronization between clients. Because of this, bidirectional client-server communication is essential. In the modern web stack, this is provided by Web Sockets which supply a thin layer above TCP and hide a lot of gory details from the implementer. To further hide network stack details, I use the socket.io library, which provides a dead simple event-driven abstraction for the whole thing. Unfortunately there's currently no support for binary data, which would greatly compress message size, perhaps by an order of two magnitudes in the case of osmos.

From a bit of research which included this nice talk from Rob Hawkes, it became clear that to have any sort of shared experience, the simplest model is to have the true game state on the server, and have clients periodically sync with it. The main trade off here is synchronization quality vs. network traffic required.

On one extreme, a game can be written by having the game logic entirely on the server and sending updates (or perhaps even simply screenshots) to the client at 60 FPS, but this is generally not feasible due to the sheer amount of bandwidth required for this model. On the opposite extreme, you can imagine a network architecture in which clients connect, get initial state, and are then largely autonomous.

In practice, there is a happy medium in which many multiplayer games fall, which means replicating non-trivial code in both the client and the server. Luckily, now that we're in the ubiquitous JavaScript era, there is no longer a need to duplicate functionality, but can instead share code by writing the game engine in JavaScript, and then running it in both a browser on the client, and in node.js on the server.

There's a lot more to be written about writing the multiplayer bits of osmus, which will hopefully turn into a more detailed article at some point in the future.

Shared JS modules

As mentioned earlier, osmus uses a physics engine that's shared between clients and the server. One might imagine that sharing JavaScript code between the two would be a breeze, but it's not that simple.

Module loaders are a mess. There's the CommonJS spec, RequireJS library and node.js require system, none of which play nicely together. If you want to share code between client and server (one of the big wins of JS on the server) without a module loader, you can use this somewhat hacky pattern:

(function(exports) {

var MyClass = function() { /* ... */ };
var myObject = {};

exports.MyClass = MyClass;
exports.myObject = MyObject;

})(typeof global === "undefined" ? window : exports);

This hack relies on the fact that node.js defines a global object while the browser does not. With the hack, node.js require() will be happy, and you can also include the file in a <script> tag without polluting your namespace, assuming of course, that no other JS pollutes your namespace with a window.global object!

Unfortunately this approach only works well for one shared module. As soon as you have multiple modules depending on each other (via requires in node-land, and globals in browser-land), the difference between node's namespacing and browser's inclusion becomes painfully apparent and requires more hacky workarounds.

Another approach is to use browserify to bundle all JS and emulate requires in the browser. This approach relies on node.js to serve the generated JS, which is not ideal, since static files should be served by a webserver optimized for the purpose. However node.js + browserify can be configured to compile JS that can be served statically without relying on node to serve it. This approach introduces some overhead:

Extra build step for deploying.
Performance overhead of whatever mechanism browserify uses to support require() calls.

Overall this approach sounds better to me, and I hope to try it out in a future version of osmus.

Your turn

Today I'm releasing osmus as a completely open source HTML5 game. Feel free to fork it to your heart's content. Oh, and for other game related goodness, check out this article on HTML5 canvas performance recently posted on html5rocks.

Global Chrome Media Keys with Key Socket

2011-07-28T09:00:35Z

Global Chrome Media Keys with Key Socket

Posted on July 28, 2011

I just skipped to the next Google Music track without leaving vim. Wanna play? Here's the app and Chrome extension. To learn how it works, read on!

Script injection limitations

Last time around I implemented keyboard bindings by injecting a content script into every tab in Chrome, capturing key events and sending them to a background page. This approach has some serious performance drawbacks:

Content scripts injected into each page.
Background pages don't perform very well.

And functional limitations:

Won't work on special URLs like chrome:// and file://.
Won't work when the omnibox is focused.
Requires chrome to be in the foreground.

Global key bindings and websockets

What we really want is global key bindings. I don't care where my keyboard focus happens to be right now, I just want to switch to the next freaking song! This sort of thing requires OS-level event capture, which is functionality most browsers don't come with. To get around this, I run a standalone app to capture global keys and run a websocket server to send these events to the browser. Note that this approach generalizes well to other use cases where functionality is not available in a browser, but can be more readily implemented natively.

The obvious drawback to this approach is that it requires the user to run a separate process to capture events.

Media key bindings in Cocoa and Python

Rogue Amoeba, maker of some popular OS X audio utilities, has a nice post on their blog on capturing media keys from an OS X application. The basic idea is to subclass NSApplication and override the sendEvent: selector:

- (void)sendEvent: (NSEvent*)event {
  if( [event type] == NSSystemDefined && [event subtype] == 8 ) {
      // Event processing
  }
  [super sendEvent: event];
}

Which in PyObjC results in the following equivalent code:

def sendEvent_(self, event):
    if event.type() is NSSystemDefined and event.subtype() is 8:
        # Event processing

    NSApplication.sendEvent_(self, event)

It's pretty neat to be able to implement Cocoa apps without having to write a single line of objective C. Writing a statusbar app with no dock item was surprisingly simple (though I have doubts that this works well for Lions and Tigers and Bears). All that's required is to set LSUIElement to true in the Info.plist.

To package the whole PyObjC application, I wrote a setup.py script and used py2app, which generates a Mac OS X .app bundle which, from a user's perspective is indistinguishable from an OS X app written in Objective C.

A WebSocket server in python

In addition to spawning off a Cocoa application and capturing events, of course I create a WebSocket server. WebSockets use a pretty simple protocol which can easily be implemented using python sockets. I based my implementation heavily on this one.

Since a Cocoa application runs its own event loop which captures the main thread, the websocket listener needs to run in a separate thread:

class KeySocketServer(Thread):
    def __init__(self):
        self.server = websocket.WebSocketServer('localhost', 1337, KeySocket)
        Thread.__init__(self)

    def run(self):
        self.server.listen()

The WebSocket standards are still evolving and implementers are, as ever, scrambling to catch up. The good news is that this means binary support is coming, which is a boon for games and other intensive network consumers. The bad news is that the latest Chrome canary (at the time of writing), requires the response to contain the Sec-WebSocket-Accept header, conforming to the draft-ietf-hybi-thewebsocketprotocol-06, which is incompatible with the python WebSocket code I'm currently using.

On my wishlist is a robus python WebSockets implementation that supports multiple versions of the spec while it's still in flux.

Injected scripts

On the Chrome extension side, a script is injected into the web player application, which creates a WebSocket client that connects to the python server on port 1337. When media keys are pressed, the python server sends messages to the JS clients and the injected JS simulates DOM events in the web player application, controlling music playback.

Try it out

If you listen to Google Music, thesixtyone or Grooveshark in Chrome on OS X and want global key bindings, please install the extension and application. If you're feeling generous, contribute your time and love! I'd gladly take fixes for

Key Socket servers for Linux and Windows
Content scripts to control other web audio players
Web Socket implementations that work with the new spec

And of course, here's the source.

Extending Chrome Developer Tools

2011-07-11T15:00:00Z

Extending Chrome Developer Tools

Posted on July 11, 2011

Chrome’s developer tools (also known as the WebKit inspector) are super useful for web developers. If you aren't ramped up already, take a look at this cheat sheet overview. Also check out some of many online resources, such as the official documentation, which is quite readable, and this 12 tricks screencast from Paul Irish.

But let’s not get sidetracked here. I wasn't going to write about using the developer tools, it’s about developing developer tools. There are two ways to go:

You can extend the tools by writing chrome extensions that use the new chrome.experimental.devtools APIs. These recently created APIs are still subject to change, and marked experimental, so it’s a bit more difficult to distribute and install them compared to other chrome extension APIs (more on this later).
You can quite easily hack the devtools code yourself, to customize it to your needs. Then, if you do a really good job, you can contribute your changes back to the chromium project and feel better for being such an awesome person.

Let’s dive in, starting from the beginning.

Developer tools extensions

Chrome provides three separate APIs for extending the developer tools:

Audits - add a new audit to the Audits tab in the developer tools:
Panels - add a whole new panel to the developer tools:
Resources - access a HAR file containing known resources

There are several extension samples available as starting points, which cover all of the devtools extension API. The API is geared towards Audit-oriented extensions (HTML/CSS/Javascript validators, performance analyzers) and Panel-oriented extensions (framework specific tooling, etc). You can also write extensions that integrate more loosely with the developer tools (perhaps only using the resources API).

My JSHint audit extension

I wanted to get my hands dirty, and wrote a JSHint-based Javascript validator extension. This extension uses the devtools audit API to create an audit that checks all scripts (linked and inline) on the current page. Errors are shown per-script in the audit results view. I wanted to go further, and allow the user to click on the error and jump to the resource, but this wasn’t possible with the developer tools extension API. To get around this limitation, I patched the developer tools with a pretty quick fix. More on modifying the devtools themselves later.

Check out the JSHint audit extension on github.

Writing an extension

The first thing you should know about developer tools extensions, is that (at the time of writing), they are experimental. This means that you will need to launch chrome with a special flag to use them. I use a shell script called chrome-devtools in my ~/bin (which is in my $PATH of course):

/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary \
  --user-data-dir=/Users/smus/.chrome-devtools \
  --enable-experimental-extension-apis \
  --debug-devtools-frontend=/Users/smus/devtools_frontend

Let me explain these switches:

--user-data-dir: specifies a custom profile for your chrome (so you have a fresh profile to deal with and don’t accidentally clobber something important in your main chrome profile).
--enable-experimental-extension-apis: turns on experimental extension APIs.
More on the last switch later.

By the way, all of chrome’s switches are explained on this page, provided by Peter Beverloo.

Developer tools extensions are based on a devtools page, which gets loaded when the devtools open. You can specify this page in the manifest like other pages (such as background and options pages):

{
  // manifest start
  "devtools_page": "devtools.html",
  // manifest end
}

The API favors extensions that are built around an Audit or a Panel, and you are free to use the resources API, as well as the rest of the chrome extension features.

One important note is that the devtools page has very limited access to chrome extension APIs, so you need to use messaging and a background page to access the full chrome API.

devtools.html:

// Send request to the background page
chrome.extension.sendRequest({}, function(response) {
  // Handle response
});

background.html:

chrome.extension.onRequest.addListener(function(request, sender, callback) {
  // Call some chrome extension APIs. For example,
  chrome.tabs.getSelected(null, function(tab) {
    // Etc... and then callback to devtools.html
    callback({data: tab.id});
  });
});

Debugging devtools extensions

When writing Chrome extensions, you have the power of the Chrome developer tools at your disposal. You can debug content scripts by inspecting the page into which the Javascript has been injected. Background pages can be viewed by running chrome://extensions in developer mode, and clicking on background.html for your extension.

However, when you write a developer tools extension, you rely on this devtools_page as well as the rest of the extension ecosystem. Debugging this page can get a bit meta – just inspect the devtools with the devtools!

Now you can inspect your devtools.html page, and debug away!

Changing the product itself

In addition to being highly extensible via Chrome extensions, the devtools are also pretty easy to modify and tinker with. As you noticed earlier, the tools are a web application written in Javascript, CSS and HTML, and thus inspectable by the devtools themselves. This web application is hidden inside Chrome, but luckily Chrome makes it easy to run a custom version of it. Basically, it takes three steps:

Download a devtools frontend zip (pre-packed version of devtools). You will need to drill down into a directory for your platform.
Extract the zip to some directory, DIR.
Invoke Chrome with the --debug-devtools-frontend=DIR switch, specifying the same directory as in the previous step.

Since the devtools_frontend.zip has some dependencies on Chrome, your zip file and chrome version should be pretty close, or you may run into problems. You can run the exact same version by also downloading the chrome-platform.zip.

Note: the Chromium project has a more detailed instruction set for getting started.

Making it your own

Now that you have your custom devtools frontend, you can make tweaks to the JS/CSS/HTML source, and see those changes in your development version of chrome after you manually restart the browser.

For example, a bunch of people wanted to get rid of the yellow highlight that appears overlaid in the browser window when hovering over DOM elements. At the time of writing, the method responsible for this is in inspector.js called WebInspector.highlightDOMNode. By applying this small patch, you can disable the default behavior.

1150,1151c1150
<     // Do not highlight the DOM node.
<     //this.highlightDOMNodeForTwoSeconds(nodeId);
---
>     this.highlightDOMNodeForTwoSeconds(nodeId);

You can also easily make cosmetic tweaks to the devtools by changing the CSS. A lot of the base styles are implemented in inspector.css. For instance, for presentation purposes, the devtools are a bit small, so I find it useful to increase the size of the devtools, which is a one line CSS change.

There's already a ton of useful features packed into the devtools, but I bet you can come up with some more awesomeness to add as an extension or chromium patch. Let me know if you do!

OAuth 2.0 from Chrome Extensions

2011-07-07T09:00:00Z

OAuth 2.0 from Chrome Extensions

Posted on July 7, 2011

Applications that access online services often need to access a user's private data. Chrome Extensions are no different. OAuth has emerged as the standard way of letting users share their private resources across sites without having to hand out their usernames and passwords. There is already a very nice OAuth library for Chrome Extensions that aims to simplify some of the pains that developers face when authorizing against OAuth endpoints.

Since this library was written, the OAuth standard enjoyed a version bump (OAuth 2.0) which greatly simplifies the flow by no longer requiring cryptography in the client. Also, some adventurous companies (notably Google, Facebook and others) have actually implemented OAuth 2.0 endpoints. At the time of writing, OAuth 2.0 is still a draft spec, but is nearing completion, and Chrome Extensions need some love.

You may be wondering why you even need an OAuth 2.0 library in the first place. As Aaron Parecki pointed out in his Current State of OAuth 2 presentation at Open Source Bridge in Portland, OAuth 2 is very much a moving target. The spec is not yet finalized, and there are 16 versions of it (although the most popular seems to be v10). Also, today's OAuth 2 implementations diverge from the spec in varying degrees, adding to the developer pain. The reason this library needs to be chrome extension-specific is that unfortunately Chrome extensions can't directly use the OAuth 2.0 server-side or client-side flows because they live at chrome-extension:// URLs.

When writing the OAuth 2.0 library for Chrome extensions, I had some goals in mind:

Support a variety of OAuth 2.0 providers that implement the spec
Allow one app/extension to use multiple different OAuth 2.0 endpoints
Avoid background pages for performance reasons

The OAuth 2.0 Library

There's a bit of setup involved if you'd like to create a Chrome extension that connects to an OAuth 2 endpoint. This brief tutorial will guide you through connecting to Google's APIs.

Register your application with an OAuth 2.0 endpoint that you'd like to use. If it's a Google API you're calling, go to the Google APIs page, create your application and note your client ID and client secret. For more info on this, check out the Google OAuth 2.0 docs. When you setup your application, you will be asked to provide redirect URI(s). Please provide the URI that corresponds to the service you're using.

Here's a table that will come in handy:

Adapter	Redirect URI	Access Token URI
google	http://www.google.com/robots.txt	https://accounts.google.com/o/oauth2/token
facebook	http://www.facebook.com/robots.txt	https://graph.facebook.com/oauth/access_token
github	https://github.com/robots.txt	https://github.com/login/oauth/access_token

Step 1: Copy library

You will need to copy the oauth2 library into your chrome extension root into a directory called oauth2.

Step 2: Inject content script

Then you need to modify your manifest.json file to include a content script at the redirect URL used by the Google adapter. The "matches" redirect URI can be looked up in the table above:

"content_scripts": [
  {
    "matches": ["http://www.google.com/robots.txt*"],
    "js": ["oauth2/oauth2_inject.js"],
    "run_at": "document_start"
  }
],

Step 3: Allow access token URL

Also, you will need to add a permission to Google's access token granting URL, since the library will do an XHR against it. The access token URI can be looked up in the table above as well.

"permissions": [
  "https://accounts.google.com/o/oauth2/token"
]

Step 4: Include the OAuth 2.0 library

Next, in your extension's code, you should include the OAuth 2.0 library:

<script src="/oauth2/oauth2.js"></script>

Step 5: Configure the OAuth 2.0 endpoint

And configure your OAuth 2 connection by providing clientId, clientSecret and apiScopes from the registration page. The authorize() method may create a new popup window for the user to grant your extension access to the OAuth2 endpoint.

var googleAuth = new OAuth2('google', {
  client_id: '17755888930840',
  client_secret: 'b4a5741bd3d6de6ac591c7b0e279c9f',
  api_scope: 'https://www.googleapis.com/auth/tasks'
});

googleAuth.authorize(function() {
  // Ready for action
});

Step 6: Use the access token

Now that your user has an access token via auth.getAccessToken(), you can request protected data by adding the accessToken as a request header

xhr.setRequestHeader('Authorization', 'OAuth ' + myAuth.getAccessToken())

or by passing it as part of the URL (depending on the server implementation):

myUrl + '?oauth_token=' + myAuth.getAccessToken();

Note: if you have multiple OAuth 2.0 endpoints that you would like to authorize with, you can do that too! Just inject content scripts and add permissions for all of the providers you would like to authorize with.

I've provided some sample extensions that use this library to help you get started.

Varying OAuth implementations

Writing this library for one OAuth 2.0 endpoint was pretty straightforward. The issues came when branching out to support multiple OAuth 2.0 server implementations which comply to various degrees with differing versions of the spec.

Facebook was the worst offender here. They claim to be an OAuth 2.0 implementation in line with v10, but are actually quite far from it. Here are some of the issues:

Token request method is GET instead of POST.
Token response is some strange form encoded format instead of JSON.
List of scopes (aka "extended permissions") was really hard to find.
Apparently to get a user's favorite music, you need the user_likes permission. Facebook, please fix your docs.
No refresh tokens but they have an offline_access permission which makes your access token expire later. This is ridiculous!

Twitter doesn't even have an OAuth 2.0 API. @Anywhere does not count. Some good questions on quora about this.

Still there are a lot of services that DO implement OAuth 2.0, such as Foursquare, Gowalla, Windows Live, Salesforce, Soundcloud and many others.

Extending the Library

To mitigate differences between OAuth 2.0 implementations, I implemented the Adapter pattern. Doing this encapsulates protocol differences in a separate adapter module for each server implementation.

The library comes with adapters for Google, Facebook and Github. These adapters are located in the adapters directory here. If you would like to contribute your own adapter, please take a look at the sample adapter and then fork the project, submit a pull request, and I'll try to add it to the project.

Also, please let me know if you experience problems using this library and we'll sort them out! The best way to do this is via github or twitter.

Brazil Trip

2011-06-21T09:00:00Z

Brazil Trip

Posted on June 21, 2011

Last month I took a work trip to São Paulo, Brazil. I gave four Chrome/HTML5 presentations and talked to many engineers and designers over the course of the week, trying hard not to sound like a broken record. Luckily Brazil has a lot of people so the audiences were different each time! Here's a link to my slides, that I tweaked slightly depending on the audience.

I'll be dumping all of my slide decks and/or talk videos to smustalks on AppEngine. I've made the switch from building slides in Keynote to using this fantastic HTML5 slide deck template and my favorite text editor.

Food

Delicious fruits of all varieties: exotic cashew, persimmon, star fruit, guyava, and way tastier bananas.
Ridiculously massive portions of sashimi at japanese restaurants.
The Rodizio we went to was super exquisite. Largest salad bar ever, including caviar (srsly).

Life

SP is a concrete jungle like nothing I've ever seen. The whole city is built up within a 60km radius.
Chaotic traffic patterns and insane motorcyclists abound. Everyone loves to drive between lanes and honk gratuitously.
Super bumpy roads date back to the dictatorship days. If you take the wrong exit, prepare to jump around.
Helicopters fly all over town and sometimes land precariously close to unsuspecting window cleaners dangling from highrise roofs.
Beware foul smelling rivers and slums on the way to Friday morning meetings!

Work

Super friendly people but my Portuguese skills failed to topple the language barrier.
Long process to get into any office building. Some ask for ID, others ask for passport. Takes half an hour just to get in...
Somehow, Google Brazil still feels like Google!

After my work was done, I took some travel photos and heard Samba da Minha Terra live, and by so doing, joined the ranks of Don and Magdalena!

I've never sailed the Amazon,
I've never reached Brazil;
But the Don and Magdalena,
They can go there when they will!

            - R. Kipling

Dynamic Chrome Extension Icons

2011-06-06T09:00:36Z

Dynamic Chrome Extension Icons

Posted on June 6, 2011

Extension developers aren't given much freedom to modify Chrome's browser chrome. Without resorting to changing the page itself, or using the new devtools extension APIs, there are two main ways of doing this. Page actions, which reside in the omnibox, and browser actions, which are positioned to the right of the omnibox both of which are simple buttons with icons, click actions and hover states. Chrome conveniently provides an API to dynamically change the icon of these buttons.

You can do this by creating image data by hand or using the canvas API's getImageData function.:

var canvas = document.getElementById('canvas');
var context = canvas.getContext('2d');
// ...draw to the canvas...
var imageData = context.getImageData(0, 0, 19, 19);
chrome.browserAction.setIcon({
  imageData: imageData
});

Note that this would be possible even without this chrome-specific API, by instead using data URIs to set the image. There's a GMail labs plugin that does this to badge the favicon with the unread email count.

What makes this an interesting design domain is the limitation of the medium. Limited real estate (browser actions at 19x19 px, page actions at 16x16 px) adds significant constraints. Still, one can show numbers, colors, small icons and graphs in this context, or even small amounts of text.

Applications of Dynamic Icons

There are extensions that use dynamic icons already, such as the PageRank extension, which effectively shows the Google PageRank for the current page right inside the browser action.

Here are some other possibilities (not at all an exhaustive list):

Create badges for page actions (which don't implement setBadge* calls).
Icon of the weather forecast, click to toggle between days.
Bandwidth meter: how large was the download of this page?
A random profile pic of people that +1'ed the site.

The Smallest Music Visualizer

I wrote a sample extension to demonstrate the dynamic icon potential of Chrome extensions. This indispensible extension is a music visualizer that renders inside of a browser action button. I use the Web Audio API to playback a song and analyse the audio stream, render the visualized audio spectrum with a canvas element and then transfer the resulting image data to the browser action icon.

Try out the chrome extension here, but note that it requires the Web Audio API flag to be enabled under about:flags (and a browser restart afterward). Check out and fork the source on github.

Learnings

This music visualizer extension loads an mp3 file when the extension background page loads, which takes a certain amount of time. To provide a better user experience, I was hoping to change the icon to reflect that the file was being loaded, and ran into two issues.

The first issue was that when I tried to render a small string like "wait" in the icon, I wanted to use a custom @font-face embedded font, which is now well supported in CSS3. You can load CSS fonts

@font-face {
  font-family: "Silkscreen"
  src: url(slkscr.ttf);
}

and then use them in a canvas:

context.font = "8px Silkscreen";
context.fillText('load');

When using custom fonts from HTML, the browser waits for the font to load, and then does a relayout. When using it from canvas, things get a bit tricky since the browser of course doesn't do font relayout for you and there's unfortunately no DOM event that fires when all embedded fonts finished loading. More precisely, onload behaviors differ from browser to browser. Mozilla waits for all fonts to load before firing the event, WebKit doesn't. You can work around this problem by assigning the custom font to a div, and observing the div's width, which will change when the font loads (codified).

Although I ultimately didn't use this font to show loading state, I recommend checking out the silkscreen font for 8-bit style designs that lend themselves well to small resolution envirnoments. You can fit about 3x3 characters inside a 16x16 canvas:

To show that the extension is still loading, I went with a progress bar instead of a message. A second issue arose when I wanted to show the progress bar animating while the mp3 loads. Unfortunately the Web Audio API doesn't currently support asynchronous loading of files, so the UI thread gets blocked during the audioContext.createBuffer call of this code snippet:

var request = new XMLHttpRequest();
request.onload = function() {
  var audioBuffer = audioContext.createBuffer(request.response, false);
}

Async loading of audio buffers is now a tracked issue for you upvote in the webkit bug tracker. I thought of working around this with Web Workers, but gave up early because of difficulties with passing objects between worker threads, and no shared memory options that would let workers access the context of the main UI thread.

Another interesting observation is that requestAnimationFrame does not work in a background page. I initially tried to use it to animate the music visualizer, but it didn't work. This is of course the API is designed to only callback when the calling page is in the foreground, and since the background page is never foregrounded, the callback never fires.

That's it for me, now it's your turn! So, dearest reader, go forth and write some awesome Chrome extensions which tastefully use dynamic icons for page actions, browser actions, and favicons to make our browsing experience even better.

Crunching Numbers for the NHL Finals

2011-05-31T09:00:36Z

Crunching Numbers for the NHL Finals

Posted on May 31, 2011

Now that the Vancouver Canucks are in the NHL Finals, a special visit to Vancouver is in order. Unfortunately, the recently posted schedule really sucks. There’s just one weekend game, and the rest are spaced in such a way that I can only hope to see Game 5 (if it happens) and either Game 4 or Game 6 (if it happens) if I manage to work from home half of the week. The question is: which game to choose? I want to be in Vancouver when the Canucks take the cup!

Yay math!

Bring super nerdy, I decided to throw math at the problem. Brushing up on probability, I calculated the odds of a series ending at each game (4-7). For a series to end in game 4, one team has to win all four games. The odds of this, given a 50% chance for each team to win, is (1/2)^4. Since there are two teams, multiply by two, resulting in a 1/8 chance. You can do similar calcuations to get the following odds:

p(4)   p(5)   p(6)   p(7)
0.125  0.25   0.3125 0.3125

Based on this distribution, it seems that game 6 is more likely to see the end of the series than game 4, but then I would risk missing a game 4 victory. Arthur Benjamin would be proud of me (unless my numbers are wrong).

A bit of history

Historical data tells a different story. As it turns out, the NHL is really old, dating back to 1927, but the league switched to best-of-7 scoring in 1939. Wikipedia has a great chronology of NHL playoffs, complete with scores and brackets.

I analyzed series scores from all playoff games in the last 25 years (since 1985) by fetching raw wikipedia articles via wget and running them through a [python script]. Older records can also be found on wikipedia, but they are buried inside other pages, in harder to parse formats.

for year in {1985..2011} do
  wget -O ${year} "http://en.wikipedia.org/w/index.php?title=${year}_Stanley_Cup_playoffs&action=raw"
done

This yielded the following distribution for playoff games, based on 383 matches:

p(4)   p(5)   p(6)   p(7)
0.16   0.24   0.33   0.27

For finals games only, the distribution is more skewed, probably in part due to a shortage of data (just NN finals games from the wikipedia page), but finals game history is probably a better predictor for finals games than all playoffs games, so this is worthwhile:

p(4)   p(5)   p(6)   p(7)
0.28   0.24   0.27   0.21

Here are all three distributions plotted in a graph:

Analyzing...

The most striking thing about this graph is the huge difference between finals history and stats, especially in the 4 game series scenario. Perhaps some teams just buckle under the pressure, while their opponents remain steadfast. Or this is just a statistically insignificant fluke that can be attributed to having an insufficiently large sample of finals matches.

As expected, the historical distributions are skewed toward the series finishing in Game 4 due to unequal strength between teams. In other words, if one team is stronger than another, then the odds of that team winning each game would be greater, making the series shorter on average.

Decision!

Armed with numbers, I can decide whether to pick Game 4 or Game 6. Here are the odds of seeing the Canucks (yeah yeah, maybe the Bruins) win in games 4 or 5, compared to 5 or 6, as calculated from each of the three methods:

         Game 4 or 5            Game 5 or 6
Stats        38%                    58%
Playoffs     40%                    57%
Finals       52%                    51%

Interesting. Given how annoying it would be to miss a Game 4 victory and how favorable the finals history-based odds look, I’ll go for 4 and 5. Now to try to work from home, book flights and bask in Canuck glory!

Chrome Developer Tools Cheatsheet

2011-05-13T09:00:00Z

Chrome Developer Tools Cheatsheet

Posted on May 13, 2011

Chrome's Developer Tools have been getting much deserved love at this last Google I/O. Paul Irish and I started with an bootcamp lab, a hands-on walk through tweaking a web application using the developer tools. We handed out a cheatsheet to give developers an overview of available features.

The cheatsheet is available for download in PDF and PNG.

Thanks to the awesome I/O organizers, we also gave away free HTML5 cake (as promised) after the session ended:

The developer tools were also featured at the start of the Chrome keynote with a webkit-speech demo. Finally, Pavel Feldman (lead engineer for the dev tools) and Paul gave an great I/O talk.

One of the most promising aspects of the Chrome Developer Tools is that they are easy to extend with (experimental) chrome extension APIs. I'm stoked to see what people create to make these developer tools even better!

Multi-Touch for your Desktop Browser

2011-05-02T09:00:36Z

Multi-Touch for your Desktop Browser

Posted on May 2, 2011

In mobile development, it's often easier to start prototyping on the desktop and then tackle the mobile-specific parts on the devices you intend to support. Multi-touch is one of those features that's difficult to test on the desktop, since most desktops didn't have multi-touch hardware, and thus desktop browsers don't have touch event support. Things are different today (you hear every mother say). Most new Macs, for example, ship with multi-touch capable input of some sort. Unfortunately the browsers haven't really caught up yet.

Enter Fajran Iman Rusadi, who released a npTuioClient NPAPI plugin with a JavaScript wrapper. Unfortunately this library provides a non-standard API to multi-touch, which is not ideal for developers that want to write their multi-touch application on desktop and then run the same code on their mobile devices without modifications.

Browser Patches

As HTML5 grows up, browser vendors struggle to stay current up with the growing variety of specifications. The result is uneven feature support across browsers and a complex problem for web developers.

The web development community has rallied around shims and polyfills for the solution. These are bizarre terms that I find confusing and so will defer to Remy Sharp to define. The basic idea of both is to fill in functionality that's missing in the browser implementation.

Since we now have a well established touch events specification working group at the W3C, I wrote MagicTouch.js, a multi-touch polyfill thatlets you, the developer, write the same code, test it on your desktop browser and then, run it on your real device. Totally tubular!

MagicTouch.js still relies on the npTuioClient plugin, it just creates spec-compatible touch events. Incidentally, here's how you can trigger custom DOM events:

var event = document.createEvent('CustomEvent');
// Initialize the event, make it bubble up and possible to cancel
event.initEvent('touchstart', true, true);
// Assign properties to the event
event.touches = touchArray;
...
// Get the element associated with the event
var element = document.elementFromPoint(...);
// Assign the element
event.target = element;
// Finally, dispatch the event
element.dispatchEvent(event);

Note that this approach to create custom DOM events is not cross-browser compatible. I only tested in Chrome.

Installation

Here how to get multi-touch web events working in Chrome for Mac:

Download and install the npTuioClient NPAPI plugin into ~/Library/Internet Plug-Ins/.
Download the TongSeng TUIO tracker for Mac’s MagicPad and start the server
Download MagicTouch.js and include both the script and the plugin in your app.

The code for this is as follows:

<head>
  ...
  <script src="/path/to/magictouch.js"></script>
</head>
<body>
  ...
  <object id="tuio" type="application/x-tuio" style="width:0; height:0;">
    TUIO Plugin failed to load
  </object>
</body>

...and you're off to the races! Your multi-touch code will now work. Try out this finger tracking demo on either your multi-touch mobile device or your newly patched desktop browser.

Future Steps

As you saw, MagicTouch.js takes some effort to set up initially, requires you to use an <object> in the HTML, and also needs you to run a separate process for intercepting touch events. While we can't quite get away without having to run another process, we can eliminate the NPAPI plugin by using the WebSocket API to communicate to that process.

If you're interested in multi-touch mobile web development, check out this article on html5rocks.com.

jQuery Conference 2011

2011-04-26T09:00:36Z

jQuery Conference 2011

Posted on April 26, 2011

A few weekends ago I went to the jQuery Conference held at the MS campus in Mountain View. And I took notes!

Overall trends about the jQuery community:

People are writing more complex apps on top of jQuery and there is a widely understood need for MVC frameworks, such as Backbone.js, Knockout.js and JavaScript MVC.
Feature detection is important!
- Polyfill - replicates standard feature with a compatible API
- Shim - provides its own API for a future feature
Serious need for templating systems. Boris Moore showed a very performant demo of jQuery Templates. Many other templating systems exist as well, like one built into underscore.js and mustache.js.
Many new mobile performance tools: blaze.io -- a tool that gives a general overview of a site's performance, pcapperf -- a web performance analyzer that uses tcpdump output from mobile device activity, and jDrop -- a service that lets you capture large amounts of data on your mobile device and then analyze it on the desktop web browser.
People are rallying around JSHint, a fork of Crockford's JSLint project, but with more configurable JavaScript sanitation rules.
Haters gotta hate. Everybody seems to get a kick out of hating Douglas Crockford. Give the nice opinionated man a break and go write some JavaScript.

I went to a bunch of talks, and I took the most notes for during this talk:

State of jQuery

John Resig talked about a bunch of changes to the project structure, largely irrelevant to jQuery library consumers. He also covered some of many jQuery 1.6 improvements:

Rewrite of attr() and val(). For example, attr('val', false) removes the attribute
Separate prop() from attr(). Indeed!
$('input:focus') gets focused input box across platforms
Significant performance boosts:
- attr() performance ~85% faster, val() ~150% faster, data() ~115% faster
Integration with requestAnimationFrame for animations
$.map(Object, function) now works (as it does for Arrays)

Pro tip: jQuery automatically parses serialized JSON if it's included as the value of a HTML5 data attribute. Example: <header data-array="[0,1,2]"> then $('header').data('array')[1] == 1

State of jQuery Mobile

Mobile matters. 5.3 billion mobile subscriptions (cf. global population of 6.8 billion), 10 billion web-enabled mobile devices.

John Resig also touched on jQuery Mobile, and then Scott Jehl and Todd Parker went into a lot more detail.

Navigation model now uses the history API for hash-less URLs.
jQM minified and packed is ~18kb!
Nice gallery of goodness at jQuery Mobile Gallery
- Including Obama's mobile site!
Media queries
- Useful as a browser support cutoff heuristic.
- CSS classes added based on media queries, facilitating simpler styles
- Uses Respond.js, a polyfill for browsers that don't support media queries
Philosophy: easily brandable cross-device experience
All builtin views are ARIA-enabled

Pro tip: mouse events in some mobile browsers are on a 300ms delay to allow the browser to interpret user's gestures. jQuery Mobile includes a fix for this!

Prototyping Tools in jQuery

Super useful and informative set of tools!

MockJAX is a library that simulates a server.

Intercepts and simulates AJAX calls
- Define a URL structure and a response structure
Can define responses as a function.
Can simulate error responses.
Useful for unit testing as well!

MockJSON: create fake JSON on demand

A way to generate random-ish JSON
For example, {'age|0-99'} outputs {'age': randint_between_0_and_99}

Amplify: abstraction layer for all data

Abstracts away shifting server-side APIs
amplify.request.define can define a data store.

For example:

amplify.request.define("list", "ajax", {
  url: "/todo/",
  dataType: "json",
  type: "GET"
});

From Wordpress to Hyde

2011-04-20T09:00:36Z

From Wordpress to Hyde

Posted on April 20, 2011

I've used Wordpress for a couple of years now as my web publishing platform of choice. I customized it a little bit, and made a custom theme for it. But nothing is perfect, and neither is wordpress. As a result of relatively heavy use, I've collected a list of things I don't much like about it:

Can't upload source code since it restricts allowed file types.
Can't easily edit files with a regular text editor.
Painful theme customization.
PHP makes it unpleasant to extend.
Constant security upgrades required.

All of these things separately are perhaps small and insignificant, but together add up to create a certain barrier to blogging.

Static files

A blog is mostly static. Why should my article be stored in a database? It's hardly ever queried or searched. Comments are perhaps the exception here, but services like Disqus implement commenting with embeddable JavaScript. And in general, dynamic parts can be handled in client-side code.

So I looked to static site generators. There's about a million of these, the most prominent of which is Jekyll, written by the github co-founder. Ultimately I chose Hyde for it's use of Django templates and Python.

Migrating the old blog

The old blog was written in HTML, but I wanted to switch to a format that can be written faster, such as Markdown, Textile, or many others. To do this I used a tool called exitwp, which parses the Wordpress export and generates Markdown files appropriate for Jekyll. I forked exitwp and hacked it to generate files better suited to Hyde. Thanks Thomas Frössman for exitwp.

I did spent a fair amount of time tweaking the export, replacing Smart YouTube URLs with real YouTube embeds, and eliminating Syntax Highlighter markup.

Customizing hyde

Hyde is basically built on top of Django templates, so is customizable with template tags and filters, written in Python. It comes with a bunch of them, including ones that parse markdown and other structured text formats into HTML.

Surprisingly I didn't need to customize Hyde much, although I did fix a couple of bugs, which the author, Lakshmi Vyas accepted. Anyway, I feel much more future proof now than with Wordpress, since I can easily write extensions in Python, or in the worst case take my Markdown-formatted blog posts and easily export them to many different formats.

Dynamic stuff

The bit of custom JS for the site layout is written using jQuery.

I use the excellent highlight.js plugin to highlight code snippets. The jQuery timeago plugin does a great job of converting absolute dates (April 1, 2010) to relative dates (about a year ago). Finally, Disqus powers comments, which I migrated from the wordpress database.

New design

In my great wisdom, I decided that since there's so much flux in the blog, why not add some more chaos by switching to a new layout too? Here's the old layout:

I recently heard from sage advice from some experienced writers. Among things that stuck were:

Optimal posting time is 9am PST
Writing personal posts is OK sometimes
Show personal information in the sidebar

The last part is expressed in the new design. Overall the redesign was an exercise in typography, CSS3 features and responsive layout.

Google fonts is a nice collection of web fonts. I decided to switch from Myriad (used on borismus.com) to a serif for the main body for a change. I ultimately went with PT Serif, which seemed like a nice improvement to Georgia, which is probably my favorite serif web font.

I used a bunch (too many?) CSS3 features on the new site. Most of these are CSS box-shadows, gradients, transformations and transitions. I used SCSS.

Also, following the responsive layout philosophy, I wanted the site to scale well to a number of different resolutions. For example, the sidebar moves to the end of the articles if the window is too narrow. Also, in the projects page, the number of columns of projects is flexible. This is achieved through CSS media queries.

For posterity, here is the new layout at the time of writing:

New domain

I also recently bought smus.com from a fellow in Germany, and will be migrating my site there. I'll keep the borismus.com blog intact for a while, but will place an annoying banner there, and disable commenting. I guess I should also switch the feedburner feed to point to smus.com as well.

Not long ago, I wrote about how switching from webfaction and apache to slicehost and NGINX greatly enhanced page load time. Well, this seems to have happened again. Over the same time period, the average response time of my wordpress instance was 934ms:

Over the same time frame, my static blog, hosted on the same machine, responded on average in 371ms:

Oh, finally, the site is fully open source on github.

Thanks

Thanks to a bunch of people:

Sol, Kat and Jon for giving useful design feedback!
Noah Levin for awesome CSS3 tweaks
Steve Losh for his Hyde migration write-up

You, for reading this post and continuing to read this blog. Until next time!

Chrome Media Keyboard Shortcuts

2011-04-01T09:00:35Z

Chrome Media Keyboard Shortcuts

Posted on April 1, 2011

Few people want to synchronize their burgeoning music libraries across computers (SSDs are small), or even lift it into the clouds with something like Amazon's Music Cloud (uploading would take forever). As a result many are saying goodbye to iTunes and moving to web-based streaming music services like grooveshark, last.fm, thesixtyone, rd.io etc. This move has a significant UX drawback: many keyboards come with multimedia keys to control your music player, but these are useless if you use a web-based music player. This post addresses this inconvenience with Media Keys, a Chrome extension that lets you assign keyboard shortcuts in Chrome control web-based music players (currently thesixtyone only).

How it works

The extension relies on two injected content scripts: key.js, a keyboard event listener injected into all pages and player.js, a music player controller, injected into the player page.

Key.js intercepts keyboard commands (ex. the next song key), and if they match bound music player keyboard shortcuts (ex. next song key => change to the next song), it sends a message to player.js, which then does the right thing (ex. changes to the next song) by simulating mouse clicks. Sounds good on paper, but the snag here is that you can't easily do direct tab-to-tab communication in Chrome extensions, except possibly through a long-lived port connection. However using long lived connections doesn't make conceptual sense, since the lifespan of a tab is relatively short.

So we go the long way with the help of a background page.

On tab load, injected key.js messages the background page to retrieve media key bindings
On music player load, injected player.js messages the background page, reporting its tab ID. The background page opens a Port through which to communicate with the player page
Matching keyboard events on all tabs send messages to the background page. The background page then relays those messages through the port to the music player

Here's an abridged code sample from the background page:

chrome.extension.onRequest.addListener(
  function(request, sender, sendResponse) {
    switch(request.type) {
      case 'command':
        try {
          port.postMessage(request.command);
        } catch(error) {
          if (localStorage['autoload']) {
            chrome.tabs.create({url: SITE_URL});
          }
        }
        sendResponse({});
        break;
      case 'register':
        chrome.tabs.getSelected(null, function(tab) {
          port = chrome.tabs.connect(tab.id);
          sendResponse({});
        });
        break;
    }
  }
);

To summarize, this approach enables a client-side remote control for a specific web application from any other page. This is potent stuff!

Try it out

Want to try it out? Media Keys works across all Chrome platforms.

Install Function Flip (Mac only)
Check previous, pause/play and next buttons (F6, F7, F8 here) (Mac only)
Install the Media Keys extension
Open up the extension options, bind the keys and save
Open up a new page and press the key bound to play

I made a short screencast showing how to install configure and use the extension.

Wish list

Just to recap, the keyboard shortcut binding pattern described above injects a script into all tabs, which essentially listens to all key events. A malicious developer could write a key logger watches username and password fields, correlates to the current domain and sends harvested data to some private server.

There are also several limitations to this keyboard shortcut binding pattern. It simply won't work in the following cases:

Chrome is in the background
Focus inside chrome is not on the page (ex. location bar)
Chrome is on a special page (ex. new tab page) where content scripts don't get injected
The current page intercepts keyboard events and stops propagation (ex. Google Docs)

Keyboard shortcuts are super important to power users, and Chrome (OS) surely won't leave us in the dust, so I'm looking forward to helping address the security risks and practical limitations this approach as a Chromium project contributor.

Share and enjoy

One last thing. If you've read my previous post, you know that I'm a big fan of thesixtyone.com so my initial implementation works for this service only. Making it work for other music streaming services is just a matter of creating a customized player.js file for your favorite music app, and tweaking the manifest to inject the new player.js into the correct domain. Feel free to fork the project on github.

Chronos: Chrome Browsing Metrics

2011-03-21T09:00:36Z

Chronos: Chrome Browsing Metrics

Posted on March 21, 2011

Like most people, I'm slowly lifting most of my work into the cloud. This leads to a lot of time spent in the browser. Just how much, I'm not really sure. Enter Chronos, a Chrome extension to track how much time you spend on each domain you visit. Chronos gives a per-day breakdown of time spent actively browsing. In addition to showing a graphical summary of domain frequency, you also get a total time spent in Chrome, and how much time your Chrome spends idle.

Chronos, named after the Greek god of time, quietly sits and monitors keyboard and mouse events you generate. If you've been active recently, the domain you're visiting gets recorded. The data structure that stores this timing information persists on the client side using localStorage. This data is never sent to any servers, so your browsing privacy is preserved. Chronos' visualization is built out of HTML divs. Just for fun, if you become inactive in Chrome, the Chronos icon in the extensions toolbar fades out.

Some features that I would find useful to add to Chronos revolve around productivity and time management:

Chronos makes it really obvious which sites consume most of your time. It would make sense to be able to enforce time limits spent on sites, either by interfacing with an extension like StayFocusd, or by replicating that functionality.
Many people find it helpful to be reminded to take breaks from computing, either for RSI purposes or for general productivity. Chronos already tracks activity levels in Chrome, so it could be augmented to remind people to take breaks in a way similar to AntiRSI.

Have a great idea to add to Chronos? Let me know, or add it yourself! As usual, the source code is on github.

SXSW LEGO MINDSTORMS Hackathon

2011-03-17T13:19:00Z

SXSW LEGO MINDSTORMS Hackathon

Posted on March 17, 2011

I just returned to San Francisco after having a blast at my second SXSW. Last time I attended, two years ago, I spent most of my time attending sessions and networking. This year I was mostly on the other side: preparing talks, staffing booths and organizing events. This year's highlight was definitely the LEGO MINDSTORMS hackathon, held as part of The League of Extraordinary Hackers event put on by Google and others, that I helped Chris Messina organize. We modeled the event after Breadpig's Hack Club, but scaled it up. The event was so awesome that here I am blogging about it!

At 7pm, 12 teams of 3-6 people began building MINDSTORMS sumo robots which would then compete in a double elimination tournament. We had mixed skill levels, from complete novices to people working at National Instruments, the company that builds the LEGO MINDSTORMS software. Teams were given 4 hours to build their robots out of two complete NXT sets and additional LEGO parts. During battle, their autonomous robots were placed back to back, starting with a 180 degree turn before trying to push each other out of the ring. The battles started at 11:30pm, and went on for about an hour.

We had a great turnout of spectators and builders on the third floor of the Speakeasy. The crowd was engaged the whole time, thanks to the lively narration of MC @tielure, which is quite surprising, given the amount of party hopping that usually ensues during SXSW. Here's the final round (Tiger Blood vs. Beam Me Up):

At the end of the day, we made hundreds of geeks happy, and donated something on the order of $15,000 of LEGOs to charity. I got to work with tons of awesome people, including @chrismessina, @kcpike, @StevenCanvin, @Chrysaora and many others. The event even got some good press on mashable.

My only regret is not having my real camera with me, so if you have high quality footage of the event that I haven't included in this post, please let me know. Looking forward to next time!

Soundsourcing: The Sound of the Crowd

2011-02-20T10:39:05Z

Soundsourcing: The Sound of the Crowd

Posted on Feb. 20, 2011

After having dabbled in crowdsourcing research, I wanted to find a compelling MTurk application around which to build a real service that would simultaneously be useful, involve creative tasks, and be fun enough to capture my attention. Generating sound effects seemed to fit the bill. Creating sounds through imitation or collecting is creative, fun and (ahem, quality aside) easy due to the ubiquity of microphones. Enter experimental service soundsourcing.com.

First I had to verify some basic questions: Would turkers be willing to create and submit sound effects? Would their output be sufficiently good for someone to use in practice? Would it be possible to have some quality control process to weed out the blatantly bad submissions? Jumping into experimentation, I asked workers to imitate or record a cow mooing, paying ten cents for their efforts. After 3 days I had a set of moos:

Taking it up a notch, I requested the more obscure sound of a train leaving the station. There were many excellent submissions in the mix, although I suspect that many of them were not created by the workers but found online despite my empty threat to verify their originality and reject. My favorite is probably this train, improvised on harmonica:

As usual, the hard problem with crowdsourcing is quality control. I approach this problem with a rating step after the sound samples have been submitted, in which turkers select their top three samples from the sound set (shuffled in random order to avoid order bias). Each instance of such a task can be done cheaply (I tried $0.05), so ordering the whole set in quality becomes easy. In the future I'd like to write some JS to verify that the turker in fact listened to all of the samples. Below is the rating form presented to MTurk users:

When all of the pieces came together and some time remained, I built soundsourcing.com around this concept. I implemented the service using Django and Boto, reusing many concepts from the CrowdForge framework. The premise of the service is that people seeking very specific sound effects that they cannot find on existing sound effects sites can use soundsourcing to get their custom sounds built to order. The basic flow of HITs on MTurk looks like this:

Since the cow and train experiments, I have collected lion roars and sword slashes (submissions are in progress at the time of writing). Please check out the site, request a sound sample and submit some feedback below. Mechanical Turkers, let's make some noise!!

CrowdForge: Crowdsourcing Complex Tasks

2011-02-02T06:30:23Z

CrowdForge: Crowdsourcing Complex Tasks

Posted on Feb. 2, 2011

Work marketplaces like MTurk are great for accomplishing small, well defined nuggets of work, such as labeling images and transcribing audio, but terrible for many more complex and labor intensive real world tasks. Over the last year, Robert Kraut, Niki Kittur and I formalized a general process of solving such complex problems using MTurk. We proposed three basic types of tasks and explored them in two neat experimental applications. To facilitate these experiments, I implemented CrowdForge, a Django framework that takes output from MTurk HITs and uses it to create new MTurk HITs.

Solving complex problems on MTurk has always involved partitioning the complex task into simpler sub-tasks. CastingWords, one of the most popular MTurk requesters, transcribes long audio recordings by splitting them into overlapping segments, distributing the work among workers, performing quality control and then recombining the transcription fragments. CrowdForge formalizes this approach and takes it to the next level. CrowdForge proposes the following task breakdown, roughly inspired by the MapReduce programming paradigm:

partition tasks split a problem into sub-problems (one to many)
map tasks solve a small unit of work (one to one)
reduce tasks combine multiple results into one (many to one)

CastingWords uses human intelligence only for their map tasks, which consist of transcribing speech samples to text. Their partition task may involve an algorithm which seeks natural breaks in speech audio samples, while the reduce task may involve programmatic stitching of audio samples. For sufficiently complex cases, however, algorithms may be inadequate, and the partitioning and reduction require human intelligence. Here are some experiments that illustrate this idea:

Writing Articles

In the first experiment, turkers generated encyclopedia-style articles on a given subject. The approach I took was to first generate an article outline using an partition task, then for each heading in the outline, to collect facts on the topic, next to combine these facts into a paragraph, finally merging all of the paragraphs into a final article. The following diagram illustrates this process for an article on New York City:

In one incarnation of this experiment, we used this approach to create five articles about New York City. Articles cost an average of $3.26 to produce, required an average of 36 subtasks or HITs, included an average of 5.3 topics per article and consisted of an average of 658 words. As a comparison baseline, we created eight HITs which each requested one worker to write the full article, paying $3.26, the same amount required for the collaboratively written articles.

We then evaluated the quality of all articles by asking a new set of workers to each rate a single article based on use of facts, spelling and grammar, article structure, and personal preference. On average the articles produced by the group were of rated on par with the Simple Wikipedia article on the same topic, and higher than those produced individually. See the paper for gritty details on the stats.

Product Comparisons

We did another experiment to prove the generality of the approach. We used CrowdForge to create purchase decision matrices to assist consumers looking to buy a new car. Given a short description of a consumer need, we created two partition tasks: one to decide which cars might be appropriate to consider, and one to decide which features the consumer cares most about. This double partition resulted in an empty product comparison chart. Each cell in the chart then spawned a map task to collect related facts. Next, these facts are reduced into a sentence, resulting in a product comparison chart. Here's an excerpt:

The entire task was completed in 54 different HITs for a total cost of $3.70. When we tried to compare to the individual case, we had no success getting individuals to generate a similar product comparison chart, even when offering more money than we paid the entire group.

Sweet Applications

MTurk is a one of my favorite tools for doing creative and novel projects. As illustrated, applying human intelligence to reduce and partition tasks, we can solve a new set of interesting problems. But there is much more to explore! For example, imagine collaborative drawing assignments in which a worker sketches out a picture, and subsequent workers refine the original picture by drawing sub-pictures or specific objects. Imagine requesting a trip plan and getting a researched day-by-day itinerary of what to see and do. Imagine partitioning a Java class into methods, outsourcing the implementation and unit test implementations.

CrowdForge is written as a Django application that communicates with MTurk servers using the Boto interface, which is a Python framework that encapsulates the Amazon Web Services API. CrowdForge regularly polls MTurk and fires notifications whenever interesting things happen (a new result comes in for a HIT, all HITs of a certain type are finished, a HIT is expired or complete). CrowdForge can manage many flows, which encapsulate code that determines how to respond to these events. For example, the article writing flow knows to create fact collection HITs for each outline topic once the outline is submitted. So to solve a new kind of complex problem, you would extend CrowdForge with a custom flow, like the following:

"But Boris", you say, "how can I get my hands on this CrowdForge framework of yours?" Indeed, we just released it for non-commercial use under the Creative Commons license. Let me know if you have any questions, comments or interest in collaborating or maintaining the framework.

Update (February 2011): CrowdForge was covered in NewScientist and in a CMU article

Update (November 2011): The full paper was presented at UIST 2011 in Santa Barbara.

KML Tours in Google Earth

2010-12-25T16:17:42Z

KML Tours in Google Earth

Posted on Dec. 25, 2010

Over this last year and a half, I spent a lot of time in airports, flying nearly 150,000 km (queue environmental angst). I lived in Europe and both coasts of the US, somehow getting a masters from CMU in the end. For posterity, I wanted to digitally summarize this extensive globetrotting experience, but found no adequate applications. It was an experience worth celebrating, and how better to celebrate than by writing some code...? So I wrote an application demo that renders travel adventures as a tour in Google Earth.

First you fill in a pretty silly madlib describing your trip:

Then, you can view your trip in the browser using the [Google Earth plugin][], or download the tour and open it in the Google Earth application. Each trip gets a unique URL that can be shared with others. Here is a youtubified summary of my recent travels generated by this app:

How it works

The madlib provides a list of places and transportation modes, the places resolving to (lat, long) coordinates, and the transportation modes affecting the transition effects between places. Airplane transitions are shown with the camera oriented north, in a bird's eye view, while ground transport transitions are more complex, facing in the direction of travel and changing pitch angle. The application builds KML in JavaScript and eventually loads it all in a Google Earth plugin:

// Build the KML
var helper = new KMLHelper();
helper.processData(data.places, function() {
  // Generate KML tour for this
  data.kml = helper.kml();
  data.distance = helper.distance();

  var json = JSON.stringify(data);
  // Save the new data to the database
  $.ajax({...})
});

// Eventually render the KML (ge is a Google Earth instance)
$.getJSON('/trips/' + id + '/', function(data) {
  var kmlObject = ge.parseKml(data.kml);
  ge.getFeatures().appendChild(kmlObject);
});

Place-to-place transition effects are achieved by using KML Tours, which provide powerful camera control requiring some computation, like getting the midpoint between (lat, long) pairs. Math for midpoint and other common geodesy operations are well documented by Chris Veness, who also provides an implementation in the form of a handy JavaScript library.

In addition to destinations, travel paths and transitions, this demo uses the picasaweb API to get public picasaweb images as GroundOverlays near places you visited. While it's possible to get a user's photos tagged with a tag, there is, rather bizarrely, no way to get a specific user's images tagged with a given location.

As usual, all of the code is available on the github. It's running live on thattrip.appspot.com, so try it out.

Observations

The Google Earth plugin provides what is probably the richest map view available on the web, but it is not without drawbacks. The mere fact that it's a plugin immediately shrinks the possible user base of any app that relies it and adds to load time. There's no way to pre-cache the data that will be shown on a tour, causing a lot of chunky terrain during quick transitions. While KML is rich and makes it possible to embed images as ground and photo overlays, the web at large has far more options to handle images, through CSS transformations and animations or by embedding in a canvas or SVG element.

This demo is very geo-centric, telling a story almost exclusively about where you went, with the images added as an afterthought. In retrospect, a more compelling story is one about what you personally saw, shown in travel photos, with a geodesy cherry on top. Stay tuned for future work...

Tweetbacks in JavaScript

2010-12-16T12:58:04Z

Tweetbacks in JavaScript

Posted on Dec. 16, 2010

Every new post on this blog gets an unsolicited pingback from topsy, a service that tracks which users mentioned the post on twitter (known informally as tweetbacks). On one hand, topsy is a parasite, using sites like mine to rise in search engine rankings, but on the other, it satisfies this blogger's curiosity to learn who reads and enjoys my posts enough to tweet about them. In this experiment, I've exposed tweetbacks directly on this blog using topsy's convenient JSONP API.

The implementation is entirely in JavaScript, consisting of two scripts. The first is on the main post listing, showing the number of tweets for each post using http://otter.topsy.com/stats.js. The second is on the post page itself, showing the tweets posted in response to the post using http://otter.topsy.com/trackbacks.js. Both scripts execute after the page loads, so the only impact on page load time is the extra kilobyte of JavaScript code.

There are good reasons to use topsy over twitter search, even though at first glance, they both provide similar information:

Twitter search doesn't index old tweets
Topsy distinguishes influential tweets (more on this later)
JSONP API support

It's important not to overwhelm readers by showing too many tweets. Popular blog posts can have hundreds or even thousands of tweets, and showing all of them at once is a bad idea. Topsy associates an influence value with each twitter user, giving them a certain weight, making it easier to decide which tweets to show, and which to hide. My implementation shows at most N tweets; if there are over N total tweets, it shows at most N 'influential' tweets, as per topsy's definition.

var MAX_TWEETS = 10;
var BASE = 'http://otter.topsy.com/trackbacks.js?callback=?&perpage=' + 
  MAX_TWEETS;
var ALL = BASE + '&url=';
var INFL = BASE + '&infonly=1&url=';

function getTweets(url) {
  $.getJSON(ALL + url, function(data) {
    var response = data.response;
    if (response.total > MAX_TWEETS) {
      $.getJSON(INFL + url, function(infl) {
        processTweetList(infl.response.list);
        var count = (infl.response.total > MAX_TWEETS ? MAX_TWEETS : infl.response.total);
        updateTweetCount(count, response.total);
      });
    } else if (response.total > 0) {
      processTweetList(response.list);
      updateTweetCount(response.total);
    }
  });
}

I also urlify the tweet, converting all of the URLs and @mentions into

var URL_RE = /(\b(https?|ftp|file):\/\/[-A-Z0-9+&@#\/%?=~_|!:,.;]*[-A-Z0-9+&@#\/%=~_|])/ig;
var TWEET_RE = /@([A-Za-z0-9_]+)/g;

function urlify(text) {
  return text.replace(URL_RE,"<a href='$1'>$1</a>").
              replace(TWEET_RE, "<a href='http://twitter.com/$1'>@$1</a>");

The approach I took has some major advantages to over the tweetback plugin for wordpress:

It works... the wordpress plugin doesn't
No wordpress comment pollution since no wordpress comments are created
No server-side load since all comments are fetched in JS
No wordpress required. Other blog engines or static sites work just as well

A drawback of this approach is that it takes some time to run the JS, which changes the DOM after the initial page load, resulting in a jarring experience. By using a fade effect to make twitter information appear gradually, I try to mitigate this problem.

I conclude with a shameless plug: tweetbacks.js is running on this blog, so try it out by posting a tweet referring to this post's URL, and it should appear in the list below, or if there are more than 10 tweets, at least increment the count. If you find this concept interesting and would like to run it on your site, let me know and I'll pack tweetbacks.js up into a jQuery plugin or something. Thanks for reading!

Update: I no longer use this code, in favor of the official twitter button. For posterity, here are the topsy-twitterized files.

WebIntent, an Android PhoneGap Plugin

2010-11-25T18:42:26Z

WebIntent, an Android PhoneGap Plugin

Posted on Nov. 25, 2010

PhoneGap is a very useful cross-platform web application wrapper that lets developers package web applications into native apps. It also lets web apps access functionality traditionally only available in native apps. This how-to is about exposing additional Android functionality to the web with PhoneGap plugins. Intents are a fundamental part of the Android ecosystem, allowing a sort of message-passing mechanism between applications, but they are not exposed to web applications. The sample Android plugin I wrote is called WebIntent, which lets you create a first class Android applications in JavaScript.

WebIntent

Firstly, the WebIntent plugin is a means of creating Android activities via Intents. With just four parameters, action, url, type, extras, it's possible to get a lot of mileage from the Android OS. For example, you can send email:

Android.sendEmail = function(subject, body) { 
  var extras = {};
  extras[WebIntent.EXTRA_SUBJECT] = subject;
  extras[WebIntent.EXTRA_TEXT] = body;
  window.plugins.webintent.startActivity({ 
      action: WebIntent.ACTION_SEND,
      type: 'text/plain', 
      extras: extras 
    }, 
    function() {}, 
    function() {
      alert('Failed to send email via Android Intent');
    }
  ); 
};

Or, you can load Google Maps:

Android.showMap = function (address) {
  window.plugins.webintent.startActivity({
    action: WebIntent.ACTION_VIEW,
    url: 'geo:0,0?q=' + address,
  }, function () {}, function () {
    alert('Failed to open URL via Android Intent');
  });
};

Secondly, the plugin lets you react when your PhoneGap application gets invoked with certain intents. To do this, you need to setup correct intent-filters in the AndroidManifest.xml. For example, to respond to ACTION_SEND, something like the following should appear in the manifest:

<intent-filter> 
  <action android:name="android.intent.action.SEND" />
  <category android:name="android.intent.category.DEFAULT" />
  <data android:mimeType="text/plain" />
</intent-filter>

Then, in the JavaScript, you can check what extras were specified during invocation:

// deviceready is PhoneGap's init event
document.addEventListener('deviceready', function () {
  window.plugins.webintent.getExtra(WebIntent.EXTRA\_TEXT, function (url) {
    // url is the value of EXTRA_TEXT 
  }, function() {
    // There was no extra supplied.
  });
});

This lets you respond to Intents without writing native code. The plugin code is available for your use and/or perusal on my github. To install the plugin, move webintent.js to your project's www folder and include a reference to it in your html files. Create a folder called "borismus" within your project's src/com/ folder and move WebIntent.java into it.

Writing Android PhoneGap Plugins

The PhoneGap project provides a plugin architecture, which isn't very well documented. Perhaps as a result, there are very few Android plugins (see the github). There are two parts of an Android plugin: a native Java class that extends com.phonegap.api.Plugin, and a JavaScript wrapper for that Java class. The JavaScript wrapper registers the plugin with the PhoneGap plugin manager. A plugin API is defined in JavaScript, and exposed via window.plugins.myplugin. Calling the API is done as follows:

window.plugins.webintent.foo({ arg1: 'val1', arg2: 'val2', // etc });

The plugin API is defined in the MyPlugin JavaScript class:

var MyPlugin = function () {};
MyPlugin.prototype.foo = function (params, success, fail) {
  return PhoneGap.exec(success, fail, 'MyPlugin', 'startActivity', [params]);
};

The plugin needs to be registered with PhoneGap before it can be used.

PhoneGap.addConstructor(function () {
  // Creates window.plugins.myplugin, an instance of MyPlugin
  PhoneGap.addPlugin('myplugin', new MyPlugin()); 
  // Binds MyPlugin to the Java class com.example.MyPlugin 
  PluginManager.addService('MyPlugin', 'com.example.MyPlugin');
});

Finally, there needs to be a Java class that actually implements the desired behavior. The execute method takes an action parameter, which is the name of the function (foo in this example), and an array of arguments:

public class MyPlugin extends Plugin {
  public PluginResult execute(String
  action, JSONArray args, String callbackId) {
    if (action.equals("foo")) {
      // Implementation 
    }
  }
}

This architecture is the key to unlock any Android functionality to your PhoneGap-wrapped mobile web application. One of huge benefits of PhoneGap and mobile web is that it's a cross-platform solution. I admit that writing platform-specific plugins seems counter-productive to this end. However until the mobile web gets the love it deserves (via extra sweet mobile browsers), there will be a place for platform-specific PhoneGap plugins to make web applications fit better into native mobile platforms.

jQuery Mobile Hacker News

2010-11-17T08:00:25Z

jQuery Mobile Hacker News

Posted on Nov. 17, 2010

I'm about to embark on a 3-week heads-down coding spree to finish off the final project of my masters. I convinced my team and our clients that it's a good idea to use jQuery Mobile instead of native Android to implement our data-centric application, so I've been playing with it quite a bit over the last week. jQM is still in Alpha 2, so to get my feet wet, I decided to write a demo app before starting the final project. After stumbling over a few kinks and rough edges, a weekend of coding yielded a mobile Hacker News client.

Loading JavaScript

jQuery Mobile is firmly rooted in the progressive enhancement philosophy of web application development. In practice, this means that the simple markup that you write in it is processed by the framework's JavaScript and transformed into much more complex and JS-enhanced markup. In particular, jQM rewrites <a href="page.html"></a> into an element that loads the contents of the page via an AJAX request. Unfortunately, CSS and JavaScript referenced by page.html is not loaded.

To work around this, I've been using a makeshift JS loader which latches on to the jQM pageshow event so that whenever an HTML page loads, the corresponding JavaScript file also loads. To prevent the JS loader from loading already-loaded JavaScript whenever a page is reloaded, it employs a cache. Still, the JS needs to be triggered when the page is loaded, so there's a way to bind a page load to a function. Here's some sample usage of the ScriptCache:

// Create a new ScriptCache in the global scope
var scriptCache = new ScriptCache();
// Now when page.html loads, page.js will load

// Also, each page can register with the scriptCache
scriptCache.onPageLoad('page.html', pageInitFunction);

// Each page.js looks like this:
(function() {
  var init = function() {...};

  // Whenever this page is loaded, call init
  scriptCache.onPageLoad('page.html', init);
  // Call init the first time it's loaded too
  init();
})();

This approach was inspired by a discussion on the jQuery Forum, and I really hope that there will be a better answer from readers or the jQM developers. The ScriptCache loader is on github.

Passing Parameters

Another issue I came across was parameter passing between pages. The jQM Hacker News client needs to pass item IDs from the main page listing to item pages. Unfortunately jQuery Mobile seems to have no provision for doing this. A couple approaches come to mind. One way is through a global variable parameter passing convention (possible since jQuery pages aren't navigated to, but loaded with AJAX). The way I'm using is through GET parameters after the hash. Thus the URL to an item page looks like this: http://example.com/index.html#item.html?id=82831. However, the pageshow event fires just slightly before the window.location is updated with the requested URL. I'm currently working around this with a setTimeout, but this is clearly unacceptable. This issue is being discussed on github.

There's also an issue passing parameters via the jQM changePage call: $.mobile.changePage('page.html?key=value');. The parameters are simply ignored. Thus I was forced to use window.location.href += '#page.html?key=value';, which only works if your window.location.href is the main application page.

Hacker News

Using the workarounds outlined above, I wrote an application that lets you read and post to Hacker News, a startup-oriented online community run by Y Combinator. It uses the third party HN API provided by @ronnieroller, who has been very responsive on twitter (thanks!). For fetching posts and comments, the application relies on the JSONP methods of this API. It doesn't currently permit authenticated posting and voting on HN.

For commenting and submitting links, I embed actual Hacker News post pages in an iframe, working around the lack of support and avoiding trust issues. Unfortunately the Android browser has a bug which forces iframes to resize to the width of their contained textarea if one exists in the embedded page. This causes the width of the page to grow which forces jQuery Mobile to relayout. The reference implementation (iOS) does not suffer from this issue.

I'd be glad to get your feedback on this app. If you want to read the code or fork the project, it's available on github.

I've packed this application with PhoneGap, and currently use it as the HN reader on my phone. Existing HN applications on Android don't provide share intent handlers (to share from the browser or your RSS reader). This app provides share intent handling through WebIntents, an Android PhoneGap plugin that I've been working on. More on that next post, so stay tuned!

PhoneGap Games in the Android Market

2010-11-11T15:38:48Z

PhoneGap Games in the Android Market

Posted on Nov. 11, 2010

A little while ago I took an existing game called Onslaught! by Lost Decade Games and hacked it to use onscreen controls instead of the keyboard so that it could be enjoyed on mobile phones. This was intended as a proof of concept and not a polished product as I encountered many technical show-stoppers, described in the previous blog post. I've been itching to experiment with PhoneGap apps in the Android Market, so despite obvious issues, I decided to pack up my Onslaught! fork and upload it to the market.

I expected there to be many gameplay issues with varying screen sizes and form factors. I strongly suspected my use of the target-densityDpi would break on non-Nexus One handsets. Indeed, my suspicions were confirmed when, after a nominal $25 fee and a short setup, I uploaded the PhoneGap-created .apk to the Market.

Results

I've never published any application to any application store before. I was surprised to see Onslaught! available in the market within seconds of my submission, with a surge of new users and comments piling in nearly immediately. Apparently the experience is particularly horrible on the Galaxy S and it doesn't work at all on HTC Hero. The day I unleashed Onslaught! unto the Market, I was contacted by someone from Zeemote, who politely suggested that the game "could possibly benefit from the use of a joystick for improved control", and promptly sent me a Zeemote to experiment with. I haven't found the time yet, but thanks Zeemote, that was pretty sweet of you – nobody's ever sent me free stuff before!

Obviously this is a pretty embarrassing state of affairs, and I, like any slightly self respecting developer decided to remove the app from the market to hide the evidence! I also received a very friendly cease-and-desist style letter from the good folks at LDG (this is one of those times when the "ask for forgiveness, not permission" approach has its drawbacks). Needless to say, I gladly pulled the game and apologized profusely to the developers. For posterity, the APK is still available.

PhoneGap

Creating an Android PhoneGap application out of a web application is pretty simple. It's a matter of following a few steps outline on this documentation page. The process varies slightly for each platform and would surely become painful and tedious for developers supporting multiple platforms. Luckily the PhoneGap folks will soon have just the thing to address this issue. PhoneGap Build is a service that promises to package and compile web apps for a variety of platforms. Looking forward to trying it out!

Chrome Extension for thesixtyone

2010-10-31T08:09:10Z

Chrome Extension for thesixtyone

Posted on Oct. 31, 2010

It's a little late to join the Chrome Extension writing party, and unfortunately still too early for the Chrome Web Store launch. Timing issues aside, here's an extension that presents thesixtyone.com in a smaller, simpler interface, effectively saving a tab in Chrome and removing all of the (useless to me) social game mechanics from thesixtyone. This post walks through some technical details of the implementation, message passing between pages of an extension, JavaScript injection and DOM event generation.

Wrapping a web application in an extension may sound deceptively easy: just load the website in an iframe in the background (via background pages), and then control the website from the background page via commands from the popup page that appears when you click the extension icon in the toolbar. Not so fast! The key issue with this approach is the control part. For good reason, browser security disallows executing JavaScript in an embedded iframe if the src is another domain. The solution to this problem is two fold:

Chrome extensions can inject custom javascript into web pages.
It's possible to pass messages between multiple pages running in Chrome. This is an implementation of HTML5 postMessage.

With these two tools, we can inject javascript that implements a message listener into a target page. This lets you define a JavaScript API around the target page. In my case, the API around thesixtyone.com was the following set of simple commands: next, previous, play, pause and getSongInfo. Once we have injected JavaScript running in thesixtyone.com, however, it's impossible to call the site's native JavaScript for security reasons (from Chrome Extension Developer Guide):

Content scripts execute in a special environment called an isolated world. They have access to the DOM of the page they are injected into, but not to any JavaScript variables or functions created by the page.

Thus, I had to resort to some unfortunate hackery: generating fake mouse clicks. One of the benefits of Chrome Extension writing is that there's no cross-browser issues to deal with, which makes this sort of trick much more reliable:

function simulateClick(elementId) {
  var clickEvent = document.createEvent('MouseEvents');
  clickEvent.initMouseEvent('click', true, false,  document,
      0, 0, 0, 0, 0, false, false, false, false, 0, null);
  document.getElementById(elementId).dispatchEvent(clickEvent);
}

Overall, the interaction between the popup, background and injected code is rather complex and looks something like this:

One Chrome issue came up in the course of development: when you change an iframe's src attribute such that the only difference compared to the old one is the hash part of the URL, the iframe src page does not refresh. I haven't had the chance to test test this on other browsers yet.

Strictly speaking, what I described isn't really a mashup, since I only use one source. However, this approach is scalable to multiple sources, and could easily be used to mash multiple web applications up inside an extension. I hope this post is useful for people trying to wrap one or many cross-domain websites in their extension.

I added a sprinkle of design and out came a Chrome Extension called Radio 61! Let me know what you think if you try it out. Source code is available on my github.

The Best Question Answering Sites

2010-10-25T00:00:36Z

The Best Question Answering Sites

Posted on Oct. 25, 2010

There are hundreds of question answering sites on the Internet. Most of them focus on specific topics (Stack Overflow for software engineering, Hacker News for startups, etc). However, a number of Q&A sites exist for answering any kind of question, including the contentious "How is Babby formed?". Open domain Q&A sites have a distinct advantage over closed domain sites, since there is no need to know which closed domain site is most appropriate for the question at hand. Still, the asker needs to decide which open domain Q&A site to ask on! What follows is a self-referential approach to ranking popular open domain Q&A sites.

Who questions the Q&A sites?

I compiled a list of the top 7 most popular Q&A sites. Since alexa.com doesn't track subdomains (ex. answers.yahoo.com, and askville.amazon.com), I was forced to use compete.com. I then asked each popular site a "recursive" question:

What is the best question answering site (besides $SITENAME), and why?

Eventually, most Q&A sites responded with one or two answers, which I harvested and analyzed. Clearly this sample is too small to draw any scientifically valid conclusions, so my analysis is anecdotal at best. Still it paints an interesting picture of the open domain Q&A space. In addition to the responses themselves, I looked at other data, such as response time and response length. The following diagram summarizes the results.

The arrows in the above diagram represent recommendations from one community to another. The surveyed Q&A sites are grouped based on popularity into three buckets. The most popular bucket includes Yahoo! Answers and Answers.com, with 44 million and 31 million monthly uniques respectively. Despite having been around for the longest (Answers.com launched in 1996), these are the most useless Q&A websites. It took 4 days to get the answer "Another good answering site is answers.com" from a Y! Answers member, and no response ever came from Answers.com in 3 weeks.

The moderately popular bucket of Q&A sites includes Answerbag, Mahalo Answers and AskVille. These sites get about 4 million uniques a month, and generally gave reasonable answers. Mahalo answers yielded the longest response, a 120 word paragraph recommending Aardvark. The least visited sites and relative newcomers to the scene are the most promising and active communities. Aardvark's response speed is unparalleled, the first answer taking just 17 minutes, probably due to GTalk integration. While Quora didn't actually yield a Q&A site recommendation, the respondent produced an reasonable retort to my rather questionable question.

A Q&A "uber-site" that will do a good job being all things to all people is simply not going to happen - it is like "one size fits all" underwear, which never really fits anyone.

The Ranking

Overall, here's my anecdotal list of top 7 open domain Q&A sites (and links to the questions I posted):

Aardvark – Quick answers to your questions. Welcome to Aardvark!
Quora - Your question is flawed. Allow me to explain why.
Mahalo – I'll answer your question with quite a bit of detail, but it'll take a while.
Answerbag - heres a decent answer but forget punctuation
AskVille - Takes forever to give you a really bizarre answer.
Yahoo! Answers – This community is brain damaged. Don't bother.
Answers.com – This community is inactive. Don't bother.

With a lot more answers from the Q&A communities, and some notion of weight, you could imagine doing a PageRank-style calculation to determine a more meaningful ordering of the top Q&A sites. We leave the details of that approach to the reader :) PS. Thanks Sean, Jenn and Pat!

An Onslaught of Mobile HTML Games

2010-10-16T10:55:06Z

An Onslaught of Mobile HTML Games

Posted on Oct. 16, 2010

HTML5 games are really picking up. Casual Girl Gamer recently produced a nice list of impressive titles. The modern web platform (namely, fast javascript and canvas) is incredibly promising to a game developer. The advantages that it brings are huge: no installation required, and ubiquitous cross platform compatibility. I took a practical look at games in the HTML5 mobile space, taking Onslaught!, a particularly fun and well written game, and porting it to Android/iPhone.

Perhaps porting is too strong a word here. Onslaught! runs and performs reasonably well on my Nexus One running Android 2.2.1. The only problem is that the game uses keyboard input, making it completely unplayable on mobile devices. Luckily, the controls are quite simple, and only require a directional pad and two buttons. So I decided to build an on-screen virtual game controller, not unlike those found in many native iPhone games. At first I was inclined to build the controls by extending the game itself (using the canvas element), but then decided that an HTML-based approach is better (since it saves the trouble of hit detection), and might even work as a generic controller for other games.

Hoping to create a general game controller for Android/iPhone, I sought to generate keyboard events from JavaScript, based on touch input. While this is possible if your key event handlers are written in a particular framework (say jQuery), it seems to be generally impossible for security reasons. I wrote an Onslaught!-specific controller that should be easy to port to any other game. The controller simply embeds the Onslaught! game in it's original 640x480 resolution, which is then scaled to the device size using the meta viewport element. The result is a mobile game that's playable on Nexus One, and not-quite-playable on iPod Touch (2nd gen) due to slower JavaScript execution. These are the only two devices I currently have access to, so I would appreciate it if you could [try it][porting it to Android/iPhone] on your Android or iOS device (in landscape mode) and let me know how it goes!

I ran into a few stumbling blocks as I was developing and testing this port, most of which involve the Android browser on Nexus One running Android 2.2.1.

Very immature touch event (ontouchstart, ontouchend, etc) support. In fact, the browser doesn't seem to support multi-touch at all (ie. only one touch can be registered at a time). In contrast, Safari for iOS supports multitouch events quite well. For complete details, see this quirksmode writeup and this bug report.
The Android browser completely ignores many properties in the meta viewport element's content attribute. Specifically, the browser doesn't react to initial-scale, minimum-scale, maximum-scale and width. As a result, I had to hack around this issue by abusing an Android-only property called target-densityDpi. I suspect that I may be doing something wrong here, since it's a pretty fundamental issue. Still I logged a bug.
Less significant but still noteworthy, the CSS :active selector does not activate on the Android browser (at least for div elements). The reference implementation is iOS, where a touchstart event on a div element causes it to become :active until a touchend event.

In the short term, HTML as a gaming platform is emerging as a real Flash killer. In the long term, I wouldn't be surprised if HTML games will be widely played on Mac, PC, TV console, and mobile phone platforms. Whatever happens, browsers will continue to be pushed to conform to modern specifications and perform ever better, making mobile rich web applications more and more feasible.

In closing, many thanks to Lost Decade Games team for writing such a sweet game and not obfuscating the JavaScript! Oh, and a reminder that if you're working on a HTML game, be sure to submit it to Mozilla's Game On contest, then add some touch controls and submit it to this mobile HTML game contest. Let the games begin!

Instant Search in 60 Lines

2010-10-09T02:00:04Z

Instant Search in 60 Lines

Posted on Oct. 9, 2010

Google recently unveiled Instant, a search enhancement which show results as you type. The real technical challenge here is scaling the backend, which now needs to handle a lot more load. The frontend implementation, however, is quite simple. Yearning for some web development, I decided to get my hands dirty. Here's a minimal implementation in under 60 lines of jQuery code.

Instant Search relies on two separate data sources: a suggestions API, and a web search API. Roughly speaking, it works as follows: as you type in the search box, AJAX requests are made to the suggestion API. The top suggestion is then used as the query to search. In this case, I used the following two services, both of which support JSONP:

Suggest: http://suggestqueries.google.com/complete/search
Search: http://ajax.googleapis.com/ajax/services/search/web

To make your own search instant, all you need are your own suggest and search feeds, and some tweaks to my code. Just make sure your server can handle the load!

Stairs or Elevator? Use Bluetooth!

2010-09-30T12:17:31Z

Stairs or Elevator? Use Bluetooth!

Posted on Sept. 30, 2010

Most people carry mobile phones, and many of those phones have integrated bluetooth functionality. Some of these phones are in discoverable mode, making them detectable by other bluetooth devices (Note: many new phones disable discoverable mode on a timer). By building and deploying such scanners in a bluetooth sensor network, we can collect a lot of interesting information about people's behavior in public places. This post is about a minimal compelling application of this technology.

Unfortunately we had just 3 makeshift internet-enabled bluetooth sensor nodes at our disposal. Under this constraint, we brainstormed several interesting scenarios where a bluetooth sensor network can be used to collect data, for example:

People's movement patterns in a bar district
Tourists following informational signs to points of interest
Passengers riding a public bus system
Travel between floors in a building

We decided to study patterns of travel (ie. climbing stairs vs. taking the elevator) between two floors of a university building. We installed three sensors: two on separate floors, and the third on the stairwell connecting them (blue circles represent bluetooth scanner nodes in the bluetooth sensor network).

This is a specific example of a more general point-to-point problem with two possible paths. From this simple setup, It's possible to compute whenever a trip was made from floor A to floor B, and whether or not the traveler took the stairs or the elevator. Suppose that there was a trip from A to B. If the trip was also made through C, it must have been made via stairs, otherwise, it must have been made via elevator. Based on this basic data, we can infer a number of features, some of which are:

Popularity of each route (ex. stairs more popular in the morning)
Habit (ex. people that take the elevator always take the elevator)
Temporal patterns (ex. some people consistently take the elevator in the morning)
Duration of trips (ex. stairs down is faster, elevator up is faster)
Waiting time at each terminal (ex. waiting times higher for elevator)
Direction preferences (ex. many people prefer to take elevator up, stairs down)

Unfortunately this project finished before any data was actually collected! Luckily, the implementation is ready for use if you want to take over! The following describes a simple implementation using EeePCs or MacMinis:

Implementing the Sensor Network

The sensor network necessary for the inter-floor experiment consists of several components:

Each scanner is a computer with a bluetooth modem and a wireless connection. We used two Mac Minis running OS X and one ASUS EeePC running Linux. Every computer executed a bluetooth scanner program, which scanned for bluetooth devices in the vicinity every 10 seconds, and updated its list of nearby device IDs. Whenever the list of nearby devices changed, it reported an ENTER or EXIT notification to the central server via an HTTP POST.

The central server runs a MySQL database to store all notifications from the scanners. Periodically, this analyzer program crawls the database to extract trips from the data (ie. a device EXITs at one sensor and subsequently ENTERs at another one). It also discerns between trips taken by stairs and by elevator. Once trips are extracted, they are appended to a Google Spreadsheet. This happens in "delayed realtime", since a trip can only be inferred once the start and endpoint are known.

The final component is a custom visualization written using Raphael and the Google Visualization framework. This JavaScript program polls the spreadsheet every 10 seconds to see if any new trips were added. If new trips were found, the visualization would update itself, representing each trip as a circle moving from one terminal to another along the stairs or elevator path. In addition, this visualization stores a frequency chart over time to show how popular each of the possible paths are.

For more information on related projects, check out Vassilis' work at UMa.

Skip to Skip: a Running Gesture

2010-08-06T00:03:47Z

Skip to Skip: a Running Gesture

Posted on Aug. 6, 2010

Like many other runners, I like to listen to my music while training. Even with a playlist of running music, I often want to change the currently playing track. There are currently two popular options of doing this: using the device itself, or using a headphone remote. My ubiquitous computing project from last Spring explores a third option: imagine if your shoes had built-in accelerometers that allowed you to skip mid-stride to change tracks.

The most obvious way to change tracks is directly through the music player, but operating a touch screen while running is pretty annoying. A much better way of changing tracks is by using a button attached to a pair of headphones, but finding it, and then double clicking it is often a frustrating experience. Some clicks don't get registered, so one often ends up triple-clicking, which skips back a track. My project explores a novel way of changing tracks: mid-stride skip. This gesture is detected by having an accelerometer in each shoe which tracks your running patterns and detects when you perform the skip gesture. In addition to being useful, this running gesture makes an otherwise monotonous activity more varied and enjoyable.

I prototyped the skip-to-skip system with a wii remote attached to the runner's lower leg, as pictured above. A computer is paired to the wiimote, collecting accelerometer data (especially in the axis corresponding to the runner's vertical movement). This communication is established using a wii library from a previous post. The naive algorithm I use for detecting skips works as follows:

Find peaks by looking at the 1st derivative (positive slope, negative slope pairs)
Discard insignificant peak values (under a threshold)
Compute distances between peaks
Look at the last 5 distances, and compute the mode. That's the pace.
Look for declinations from the pace characteristic of a skip.

Once my gesture recognition code worked reasonably well, I ran a user study to see how people liked this method compared to the touch screen and headphone remote. After a few hitches (including getting kicked out of the Madeira Tecnopolo), I managed to test the prototype with many of my Madeiran classmates (thanks guys!). Most people preferred the skip-to-skip method over both the direct smartphone and headphone remote methods, which is promising.

For more details on this project, check out the paper that Vassilis Kostakos and I will present at Ubicomp 2010 in Copenhagen. If you're interested in the source code, it's located on github.

Update: here's the poster. The font-size is scary huge on A0!

Minimal Business Card Design

2010-07-21T00:11:36Z

Minimal Business Card Design

Posted on July 21, 2010

Business card design is a tricky art. It's a fairly constrained space, but that's what design is all about, right? I'm ordering a personal set of moo mini cards. These are small, two sided prints. One side contains an image, and the other contains contact information. On the image side, I'm putting snippets of travel photography. The other side is by default a conventional list of contact information, but moo conveniently allows it to be replaced by a custom image.

Rather than having each field separately labeled, I tried to uncover as much information as possible within my email address. Hidden inside are my first name, last name, website, and twitter account! Here's a minimal design concept that tries to break it down.

In my case, my email address contains all relevant info except the phone number which I don't want to include anyway. I hope that this simple idea can inspire you to come up with something more polished. Looking forward to what you come up with! P.S. Thanks for the help, Jenn!

Android-powered Mindstorms

2010-06-27T23:27:53Z

Android-powered Mindstorms

Posted on June 27, 2010

A few projects around the internet use an Android phone to control the LEGO Mindstorms NXT brick. Most involve an ugly hack in which the phone communicates with a computer over WiFi, and the computer (paired to the NXT through bluetooth) submits the command to the brick. These projects typically use Android as a remote control for the NXT robot, and not as part of the robot itself. Here is a missed opportunity: the NXT could be augmented by an impressive list of sensors, GPS and Internet access provided by an Android phone.

This project does just that, while eliminating the need for a computer in the loop, so that the Android directly communicates to the NXT. This allows for more powerful Android-powered NXT robots. As an example, I made a fully autonomous twitter-controlled robot. The NXT uses two motors to spin in place or move forward, and a third motor to control the tilt of a Android phone cradle. The Android phone keeps track of its orientation (compass heading and tilt), polls twitter search for new commands and sends commands to the NXT brick. After each command completes, the Android phone takes a picture and sends it to twitter. Any twitter user can look at the last few photos, decide which command makes sense to perform next, and issue it. This approach can be summarized succinctly as "crowdsourced teleoperation".

Here's a demonstration video of the robot in action:

I think that this marriage of Android and NXT can fuel a very interesting set of robots impossible to build with the NXT alone. @mindstorms is offline for now to save some battery life, but code is available at my new github. If you make use of my code or have some feedback, please reply below!

Prototyping with Wii Remotes in Python

2010-05-28T12:00:40Z

Prototyping with Wii Remotes in Python

Posted on May 28, 2010

I've been working on a couple of researchy projects involving gait recognition and running foot strike analysis. For my proof of concept, I turned to the wiimote, everyone's favorite physical interaction prototyping input device. Wiimotes are portable and rugged, and thus well suited to high-intensity activities like running. They attach easily to legs with a physio band, although the elastic tension tends to cut off circulation. No big deal, though... it's For Science!

This article is not about attaching Wiis to legs (more on that at a later date!), but about communicating with the Wii remote using python. I started out by writing a Cocoa application to harvest accelerometer data using the WiiRemote framework provided by the DarwiinRemote project. After some objective-c iterations of my initial gait recognizer algorithm, I decided to port to python, an environment better suited for light prototyping. There's a few packages explicitly developed to integrate with the wiimote. Pywiiuse provides a lightweight wrapper around the wiiuse library and does not work on OS X. An alternative, pywiimote claims to be multiplatform but pointedly isn't. Here's the start of their code:

from ctypes import *
kernel = windll.kernel32

Having found no existing wiimote-specific python libraries that would work on my platform, I had no choice but to dig a little into the bluetooth-based protocol that the wiimote uses. I found all the details in all their gory glory on the wiibrew wiki. The communication protocol involves two open L2CAP sockets between the host and wiimote: one for reading and one for writing. After an initialization string is sent over the write socket, the wiimote springs into life and sends a stream of data on the read socket. In this data are accelerometer values and button presses. Here's a simple python snippet using the lightblue library:

import sys, lightblue, hexbyte

WIIMOTE_DEVICE_NAME = 'Nintendo RVL-CNT-01'

# auto-discover nearby bluetooth devices
devs = lightblue.finddevices(getnames=True, length=5)
# find the one with the correct name
wiimote = [d for d in devs if d[1] == WIIMOTE_DEVICE_NAME] and d[0] or None
if not wiimote:
    print "No wiimotes found!"
    sys.exit(1)

# create a socket for writing control data
write_socket = lightblue.socket(lightblue.L2CAP)
write_socket.connect((wiimote, 0x11))

# create a socket for reading accelerometer data
read_socket = lightblue.socket(lightblue.L2CAP)
read_socket.connect((wiimote, 0x13))

# initialize the socket to the right mode
write_socket.send(hexbyte.HexToByte('52 12 00 33'))

# start reading data from it
while 1:
    byte = read_socket.recv(256 * 7)
    data = hexbyte.ByteToHex(byte)
    # do something interesting with the data
    print data

You'll need hexbyte.py to run the above snippet. I hope you (the wii remote wielding python fan) find this snippet useful. As a side note, if you've figured how to pair a wiimote with an android phone and released the code into the public domain, please let me know. Since Android 2.2 still doesn't ship with L2CAP APIs, I hit the wall.

Nike+ Hacking with Python

2010-05-18T05:27:56Z

Nike+ Hacking with Python

Posted on May 18, 2010

Nike+ is a clever little system designed by Apple and Nike to infer the runner's speed and augment the running experience. The runner places a small chip in his shoe which transmits data to the iPod using a proprietary RF-based protocol. The chip contains a piezoelectric cell which measures how long the foot exerted pressure on the ground. According to Apple's FAQ, this contact time is directly related to your pace. In this post I provide a snippet of python code for collecting data through Sparkfun's adapter.

Sparkfun dissected the transmitter and receiver and currently sell a Nike+ Serial to USB adapter. There's been a number of notable projects since, and quite a lot of interest in the system for general hackery.

I wanted to use Nike+ for my own project (a running bib that would automatically display the runner's speed on the back). Unfortunately, the only available implementations were in Visual Basic and perl, neither of which work on Mac. Here's a small script for Python on Mac OS X to collect Nike+ data using SparkFun's adapter.

#!/usr/bin/env python
import serial
from hexbyte import *

def readbytes(number):
    buf = ''
    for i in range(number):
        byte = ser.read()
        buf += byte

    return buf

# open the appropriate serial port
ser = serial.Serial('/dev/tty.usbserial-A6007uDh', 57600, bytesize=serial.EIGHTBITS)

# send the following init string to the Nike+ device:
init1 = 'FF 55 04 09 07 00 25 C7'
ser.write(HexToByte(init1))

# listen for the response string: FF 55 04 09 00 00 07 EC
response1 = ByteToHex(readbytes(8))
assert response1 == 'FF 55 04 09 00 00 07 EC'

# send the second init string
init2 = 'FF 55 02 09 05 F0'
ser.write(HexToByte(init2))

# listen for the response string: FF 55 04 09 06 00 25 C8
response2 = ByteToHex(readbytes(8))
assert response2 == 'FF 55 04 09 06 00 25 C8'

# now we're ready to listen for actual data
print "nike+ initialized. listening for data"

while True:
    byte = ser.read()
    # if a byte is coming down the port,
    if byte:
        # get the rest of the message (34 chars)
        message = byte + readbytes(33)
        # and decipher it
        data = {
            'number': ByteToHex(message[11]),
            'uid': ByteToHex(message[7:11]),
            'data': ByteToHex(message[12:]),
        }
        print data

You'll also need hexbyte.py, which contains convenience conversion methods between binary and hex.

I'm still unable to fully make sense of this data. Firstly, each step seems to inexplicably generate 8 packets instead of one. Second, there are 22 bytes in the Nike+ data with an unknown structure, probably containing pressure duration data. If someone figures out how to make sense of this please let me know!

Jam Hunt: Friendly Jam Sessions

2010-04-24T16:18:32Z

Jam Hunt: Friendly Jam Sessions

Posted on April 24, 2010

Ever wanted to join a band? I bet you have! Why? Because collaborative music making is an incredibly enjoyable and rewarding experience. But the barriers to entry are high: not only do you need to have baseline musical skills, you also need considerable managerial talent to find and bring together disorganized musicians. To find partners to jam with, people use craigslist and band matching sites to try to establish relationships with randoms. Why not leverage our social networks for this purpose?

Okay, now that you're fully convinced that there's a huge opportunity to tap into this friend-jam space, let me introduce Jam Hunt. The idea behind Jam Hunt is to allow you to manage your musical profile by specifying a list of instruments you are skilled at and a list of songs you know how to play. If your friends also maintain such profiles, Jam Hunt can look across the social graph and discover friends to try jamming with. Thus, the application enables spontaneous flash bands (in the spirit of flash mobs) to form brief, friendly jam sessions.

I developed a Jam Hunt prototype for SAUI class while pleasantly stranded in Pittsburgh as a result of Eyjafjallajökull's eruption. The assignment stipulated that I implement a Facebook application, which was initially distressing to me, due to the walled-garden nature of the platform. I was slightly mollified when I discovered three things:

that there is a way to use django to develop Facebook apps.
that the average Facebook user has a whopping 130 friends.
that there is a potentially bright future ahead for Facebook

My prospects for having fun while developing something useful, and potentially viral, and not entirely evil, were on the rise.

As it turns out, developing a django application for Facebook is no cakewalk. Firstly, python is not an officially supported language for Facebook development. As a result, there are a number of semi-abandoned open source projects to bridge that gap. Coupled with Facebook's outright disregard for API stability, calling Notifications.send and Stream.write were next to impossible from python. But surely writing PHP applications must be a breeze, right? Well, during the week that I was developing Jam Hunt, forum.developers.facebook.com, one of the most indexed resources on Facebook API questions, was consistently down. The sorry state of their hybrid documentation-wiki system was just icing on the cake.

Anyway, enough bitching! If you have some spare cycles and a Facebook account, please try Jam Hunt. Whether you find it interesting, appealing, pointless, ugly, or just outright broken, let me know.

CHI 2010 Poster

2010-03-21T12:40:33Z

CHI 2010 Poster

Posted on March 21, 2010

I'm confirmed to go to CHI2010 in Atlanta, so I spent some time making a poster for the Ubiquitous Drums project that was miraculously accepted as a WIP. It's nice to pretend to be a visual designer sometimes. Thanks to Mark, Jenn and Rebeca for your input.

I would really appreciate additional suggestions on how to improve the poster, or your thoughts on academic posters in general. Why are they usually so ugly?

Lightweight Wordpress on Slicehost

2010-02-26T05:18:28Z

Lightweight Wordpress on Slicehost

Posted on Feb. 26, 2010

I recently switched from shared hosting to a VPS, expecting to get an immediate and automatic performance boost. I was overly optimistic and ran into memory trouble right away. After endlessly struggling with Apache and mod_php configuration, I was ready to give up. Then on a whim, I switched to nginx/fastcgi to see the average response time drop from 1500ms to 300ms:

WebFaction is a great shared hosting environment, providing tons of functionality and impeccable customer support. However, shared hosting means that multiple users share resources on a single physical machine, resulting in wildly fluctuating site performance and uptime at the whim of other clients. Also, I prefer to do system administration from the command line, but WebFaction provides a powerful but clunky web-based administrative interface. All in all, despite my respect for WebFaction, I said goodbye and switched to Slicehost.

I ordered my shiny new slice and plunged into configuration. I migrated wordpress databases and set up Apache with modphp. Everything seemed to work reasonably well until Apache ran for a few hours and began consuming my memory allowance. The slice started thrashing and performance fell to a crawl. I found the culprit to be in my mpmprefork_module MaxClients and MaxRequestsPerChild configuration, but even after tweaking those, my slice was hitting the wall pretty quickly. I nearly returned to cushy WebFaction, where I had marginal performance without the headache, but decided to experiment more.

Nginx is a minimalist HTTP server written by Igor Sysoev for Rambler. It's now being used by Wordpress and other high profile sites. Seeking help from the internet, I eventually came across Thomasz Sterna's php-fastcgi init script and adapted it for my slice, using

PHP_FCGI_CHILDREN=5
PHP_FCGI_MAX_REQUESTS=100

for the low-memory environment. In addition, I installed the WP Super Cache plugin.

For my django site, I wrote an init script for launching the django fcgi server via django's manage.py. Here's the important part:

MAXCHILDREN=5
MAXSPARE=5
MINSPARE=2 
# ... 
start-stop-daemon --quiet --start \
  --pidfile $PIDFILE --chuid "$USER" \
  --exec /usr/bin/env -- python $SITEPATH/$SITENAME/manage.py runfcgi \
  --settings=settings \
  host=$HOST port=$PORT pidfile=$PIDFILE \
  maxchildren=$MAXCHILDREN maxspare=$MAXSPARE minspare=$MINSPARE

I'm happy with the performance both for django and php-powered sites. However, I'm just guessing when it comes to values of MAXCHILDREN, MAXSPARE and MINSPARE, PHP_FCGI_CHILDREN and PHP_FCGI_MAX_REQUESTS.

Do you have insight on how to tweak these parameters? Is your VPS hosting configuration better?

Visualizing MTurk Requesters

2010-02-06T12:39:41Z

Visualizing MTurk Requesters

Posted on Feb. 6, 2010

I signed up to do one month of paid research at CMU|Portugal before spring classes start. My task boils down to creating interesting visualizations. The bad news is that I have no experience visualizing data and the dataset I'm to visualize hasn't yet been collected. Fortunately, I've always been theoretically interested in data visualization, so I was happy to have a solid excuse to explore the subject. All I needed was a sufficiently rich data set, mad skills and a bit of inspiration.

Lately, I've been pretty excited about squeezing some new potential out of Mechanical Turk. Part of my research involves finding patterns in Mechanical Turk requester strategies. A few weeks ago, I began gathering data with a python program that extracts all scrape-able information about every HIT group and stores it in a sqlite3 database. This is quite an interesting data set, so I pounced on the opportunity to visualize it, killing two birds with one stone.

Due to lack of time, I decided to skip most of the visualization literature. Instead, I found a great visualization project database and started writing simple examples using some popular visualization languages and frameworks. I began with Processing and Processing.js but quickly tired of the raster-based drawing model. I turned to SVG with Raphaël and jQuery and ended up building a simple bubble chart. I had no specific visualization in mind, so my vague goal was to be able to visualize data in Hans Rosling's favorite five-dimensional (x, y, size, color, time) graph. Making this graph animate, however, proved to be quite difficult.

I turned to the internet for help, and Itai Raz explained to me in his thick Israeli accent that the Google Visualization API is pretty sweet. Moreover, it comes with the exact visualization I wanted to implement, apparently called the Motion Chart. Happily, Google Spreadsheets export data in the format that the visualization framework consumes. Thus my problem was greatly reduced to one of analyzing and uploading data to a Google Spreadsheet. Turns out that this too is quite easy using the python GData framework.

I decided to visualize differences between requester strategies on Mechanical Turk. Every time I scrape, I generate a Average Reward (x), Average Allotted Time (y), Total Number of Hits (size) graph for the top 50 requesters, and then upload it to a google spreadsheet. Here is the python program I wrote for this purpose. The results are interesting and fun to play with. Google's Motion Chart visualization is incredibly powerful and flexible. I won't go in detail into findings from the data since it's irrelevant to this largely technical discussion about visualization technologies. I'll soon get my hands on the data I'm expecting and create a custom visualization for it with the Google Visualization API. Stay chooned!

[]: /x/turk-visualizer/

Crowdsourcing Code

2010-01-16T14:46:01Z

Crowdsourcing Code

Posted on Jan. 16, 2010

As a follow up to my last post, I posted a HIT on Mechanical Turk asking 20 turkers if they know Java. I paid them 5 cents to answer the question. Surprisingly, 9 of 20 claimed to know. I was amazed at how strong selection bias was in this case, since surely not 50% of turkers know how to program! I then asked those turkers who know Java to complete the following trivial Java method. If they wrote it correctly, I paid them a 45 cent bonus.

public static String reverse(String source) {
  // your code here 
}

Here are the results:

4 turkers used StringBuffer.reverse
3 turkers created a new string by iterating through the original string in reverse
1 used recursion
1 used Collections.sort(l). I'm not sure what was intended

I was hoping that people would fill in the empty reverse method with their code, but many of them implemented their own methods and helpers. One person implemented their own class with extensive comments. This data as a nice existence proof, indicating that turkers can be harnessed for programming-related crowdsourcing.

I'd like to turn Mechanical Turkers into Mechanical Coders. Given a set of unit tests and a method to implement, their work could be automatically verified based on passing the unit tests. Furthermore, some turkers could be tasked to write additional unit tests for this method. Through this technique, it's conceivable to harness the power of The Turk to implement whole classes. Code quality aside, what sort of software quality could be achieved by following this approach?

Crowdsourcing Articles with Mechanical Turk

2010-01-14T07:07:08Z

Crowdsourcing Articles with Mechanical Turk

Posted on Jan. 14, 2010

Last semester at CMU, I was involved in a research project involving Mechanical Turk. The goal was to get Mechanical Turk users (turkers) to collaborate on creating online wikipedia-style articles. Prior to my team's involvement, an undergraduate created a mediawiki-based platform to allow turkers to collaborate on articles. Despite a high compensation, few turkers completed the task. My team tackled the problem and came up with some interesting videos on the way.

We began by conducting contextual interviews with turkers living in Pittsburgh, all of whom rather unexpectedly, were female. The general takeaway was clear: turkers are used to very short and repetitive tasks, but article creation requires a prolonged period of concentration. Our solution was to significantly tweak the task, making it seem less arduous. In addition to simplifying the HIT's flow, we switched from mediawiki to etherpad as the article editing and collaboration platform. As a result of these changes, we were able to churn out turker-created articles on a given topic for under ten dollars. Here's a video of turkers collaborating on an article about Halloween:

We started out by creating an etherpad instance with a simple paragraph about the topic, as well as some article quality guidelines. Next, we created a series of Mechanical Turk HITs referencing the etherpad instance's URL. We paid our turkers a quarter up front for accepting the task, and provided a nickel (up to one dollar) every time they returned to edit the pad. We had no good way to verify the bonus mechanism, so we generally gave out the maximum bonus to every active collaborator. Here's the evolution of an article on Windows 7:

Watching the etherpad explode in color as multiple turkers simultaneously edit the same article is still mesmerizing. Though the quality of the articles was quite low, it generally increased with each turker's successive pass. Also it's worth noting that errors that we deliberately inserted in the starting paragraph as well as in real time were swiftly edited out. Not much quantitative analysis of this collaboration data has been done yet, though there are plans to conduct more scientific experiments in the future.

Several other researchers have been conducting interesting studies on mturk. Greg Little's work at MIT generated an interesting project called TurKit, intended to simplify setting up experiments such as the one outlined above. Panos Ipeirotis at NYU runs a variety of turk experiments as well as an mturk statistics monitor, which continually scrapes Mechanical Turk and generates summary data. Most recently, Jennifer Boriss surveyed the Turk community about their browser preferences projecting a growing interest in Chrome.

These varied Mechanical Turk projects represent only a small fraction of the potential of crowd-sourced marketplaces. It's especially interesting to take complex tasks, break them down into turk-sized morsels, and recombine them again. To improve the article collaboration scenario discussed here, one could provide an outline of an article and task turkers to elaborate on each point. This seems to be exactly what Greg's group is doing in an collaborative essay writing experiment. Such an approach may also be effectively applicable to crowd-sourced software development, which I hope to explore in the near future. Do you know other interesting projects and resources related to Mechanical Turk? If so, please respond below!

Ubiquitous Drums

2009-11-18T14:22:15Z

Ubiquitous Drums

Posted on Nov. 18, 2009

Look at the riders of any city bus. Many of them are plugged into their music players, tapping away to the beat. I propose to augment our natural love of rhythm into a ubiquitous wearable drum system. The target user of this system isn’t only the typical rhythm loving bus rider, but also an amateur drummer. Drum kits are heavy and unwieldy, making them difficult to transport to a jam session. The proposed system can also act as a stand-in for a full drum-kit for quick, impromptu jamming.

I took a pair of jeans and imbued them with two force-sensitive resistors, one on each knee. The left pocket houses a sparkfun box containing an arduino and a breadboard. Wires run through the pant legs to connect the pads to the box. Wiring the pants was surprisingly easy, since as I discovered, electric tape easily adheres to denim.

The two FSRs are hooked into pull-down switches which connect to analog ports of the Arduino. Every time a pad is hit, Arduino sends the pad ID and the force of the impact through the serial port. A python program running on my machine listens on the serial port and synthesizes sounds corresponding to the data using pyserial and pygame respectively.

import pygame, sys, serial

# initialize the pygame mixer
pygame.mixer.init(frequency=22050, size=-16, channels=2, buffer=256)

# map positions to sounds
samples = {
    'lknee': pygame.mixer.Sound('media/hihat-open.aif'),    
    #...
}

# initialize serial port
s = serial.Serial('/dev/tty.usbserial-A6008ea9', 9600)

while 1:
    # read from the serial port
    line = s.readline().strip()
    # ...
    sample = samples[pad_id]
    sample.play()

if __name__ == '__main__':
    pass

This first prototype of Drum Pants is intentionally crude. Aside from increasing this system’s production value, there are a number of limitations that should be addressed. The current prototype requires a computer to synthesize sounds, which greatly hinders portability. By retrofitting the Arduino with a wifi shield, the system could communicate with any wifi-capable synthesizer, such as an Android phone.

Another issue with this system is that it’s built entirely into a pair of pants. This makes putting drum pads into other items of clothing impossible. To address this problem, the pads could wirelessly communicate to the Arduino device. In this case, the pads would be self-contained transmitters that could be placed anywhere. This opens up a wide variety of applications, such as placing the pad onto a pair of shoes to simulate a kick or hi-hat pedal. Do you like this idea? Please give me feedback below!

Update: Do you want to build your own drum pants? Check out this instructable!

Update 2: Accepted as a CHI 2010 WIP. Many thanks to Mark Gross! Extended abstract now available.

Arduino-Nerf Mashup

2009-10-20T20:48:54Z

Arduino-Nerf Mashup

Posted on Oct. 20, 2009

My favorite class at CMU is probably Making Things Interactive. For me it is an opportunity to take my thus far casual electronics hacking to to the next level. In this article, I'll briefly outline my submission for the "Motion" assignment. I used a servo motor to control a Nerf gun. I built it and installed it in my MHCI lab, which has a handful of Nerf pistols floating around. The idea was to have the gun automatically fire at unsuspecting visitors as they entered the room.

To arm it, one manually cocks the gun, loads a dart and resets the program by pushing the button on the Arduino board. The program then allows 10 seconds for the door to be open before it arms the system. When the system is armed, the servo activates and shoots the gun as soon as the door is opened.

As you can see, I generously used rubber bands and binder clips in this project. I used them to fasten the servo motor to the Nerf gun. I also used them to harness a telephone cable by wrapping a rubber band around the RJ11 connector, carefully inserting jumpers, and applying additional pressure (to ensure contact) with binder clips. This hacked telephone cable stretched from the gun to the door sensor.

The circuit was dead simple. The door acted as a switch for the pull down resistor circuit.

New Design

2009-09-28T10:31:29Z

New Design

Posted on Sept. 28, 2009

Since starting this site, I've been using a lightly tweaked version of the ClockWorkAir theme. The old design featured a prominent quote in prime screen real estate and a tag cloud in the upper sidebar. A blue title bar, reminiscent of the default wordpress theme, graced the blog header. The main text was small and there was hardly enough room to place images in the left margin. Here was my old blog design:

As I had hoped, the design class I'm taking at CMU gave me some good ideas. Here's my new blog design. I'm much happier with it than with the former look, but still have a backlog of things to tweak. My design was inspired by several people's web logs, which bring together form and function in a way which appeals to me. Please give me constructive feedback on the new design if you have time. I will release the wordpress theme if anyone wants it — I just haven't had time to bundle it yet. Here is my new blog design:

As of this month it's been a year since my first blog post. Surprisingly, I've been writing one to two blog entries per month and hope to keep my musings flowing at approximately the same rate in the future. It's very rewarding as a writer to have a steady flow of visitors, some recurring readership, and even the odd discussion here or there. Thank you all.

Offline Web Apps on the iPhone

2009-09-06T15:21:19Z

Offline Web Apps on the iPhone

Posted on Sept. 6, 2009

In the midst of my graduate studies, I somehow found the time to write a simple prototype for a mobile Guitar Unleashed client. It's more of a proof-of-concept for some cool new technologies that I've been meaning to play with. Two things led me down this path:

Since I'm no longer bound by corporate affiliation, I feel compelled to finally develop an interesting application for iPhone.
I'm very bad at remembering guitar chords and lyrics, but never bother making a cheat sheet to take to the campfire. I nearly always have my phone in my pocket, though.

The GU mobile prototype is very simple. Users can manage a list of favorite songs, and view their lyrics with chords. At first, I set out to create a native iPhone application to do this, but several considerations made me change my mind:

I don't want to deal with a potential App Store rejection, keeping in mind the dubious legality of lyrics
The simplicity and data-centric nature of the app lends itself well to a web implementation.

There are a few interesting technical challenges to overcome in implementing this application. Firstly, the app needs to blend visually with the iPhone look and feel. I chose the iUI web framework to develop my application. In retrospect, jQTouch would have probably been a better choice, since I ended up using jQuery. In addition, iUI is very much not bug free, and I don't like iUI's anchor-based navigation. Apple provides several iPhone-specific meta tags to enhance the web experience. For example, <link rel="apple-touch-icon" href="logo-touch-icon.png" /> allows to specify the icon that will appear in the iPhone springboard. Setting <meta name="apple-touch-fullscreen" content="YES" /> will remove the lower hud of MobileSafari. The full list of options is available from Apple.

The second broad challenge for mine and iPhone web applications in general is that they need to remain functional without an internet connection. For my mobile client, data would be provided by two calls returning JSON: /song/search?query=myQuery returning a list of songs and /song/get/myID returning specific song info. My first instinct was to implement local persistence via cookies, as is customary in web development. There's a 4K limit on the size of each cookie, so storing all songs in one cookie was out of the question. A simple alternative was to store a cookie with an array of song IDs, and a cookie for each song. This strategy worked quite well on WebKit and in the iPhone simulator. Unfortunately, cookie persistence works differently on the actual iPhone, likely for security reasons. Every time the phone reboots, MobileSafari's cookie jar is emptied.

The alternative to this is a much more modern approach: HTML5 databases. WebKit now allows you to store structured data locally in an SQLite database. I found this approach to work very well on iPhone. The database backend persists through reboots, as expected. I implemented both storage schemes in separate files, available for your scrutiny: the class SongJar for cookies in jar.js and SongDatabase for HTML5 storage database.js.

The last piece of the puzzle is how to force the web application's source files to get cached on the iPhone, so that the app remains accessible even when the phone is offline. This is done with a cache manifest, also new in HTML5. I ran into several problems trying to set this up, and would have benefited from these tips:

Reference the manifest from the HTML with <html manifest="cache-manifest">
Serve the manifest with the text/cache-manifest mime type
Ensure that all paths in the manifest are accessible

Though my app is a mere prototype, I'm quite happy with the result. After adding the application to the springboard, it almost feels like a first class iPhone application. You can try it out at http://www.guitarunleashed.com/m/, and please bear in mind that it's a proof of concept.

Never Delete Your Gmail Account

2009-08-28T20:36:29Z

Never Delete Your Gmail Account

Posted on Aug. 28, 2009

Since moving to the states, I have had nothing but grief from the .ca at the end of my all-purpose email address. Even in Canada, people would constantly confuse z3.ca for z3.com, resulting in email bounces. To resolve this problem once and for all, I decided to switch to Gmail like all the cool kids. I registered boris.smus long ago out but never used it. The first thing I did was try to link my z3.ca to the Gmail. When I couldn't figure out how to do that, I deleted my Gmail account in order to re-create a new pre-linked account with the same name. Sounds innocent enough, right?

Wrong! Google has an uncharacteristically evil account deletion policy which is not at all clearly communicated. The deletion page simply says:

You're trying to delete your Google Account that provides access to the Google products listed below. Please select each checkbox to confirm you fully understand that you'll no longer be able to use any of these products and all information associated with them, and that your account will be lost.

Meanwhile, the F.A.Q. reads:

If you use Gmail with your account, you'll no longer be able to access that email. You'll also be unable to reuse your Gmail username.

Long story short, I tried everything in my power to recover the old username. I found handfuls of frustrated users in the same position as me; some having deleted their account by accident, others victims of pranks and identity theft. I asked on official and unofficial Gmail forums, and even consulted with my Googler friends, all to no avail. After mourning the loss of boris.smus@gmail.com, it was time to take a critical look at Google's email offerings.

I enjoy Gmail's webmail client very much. It's a fast, intuitive, search and tag based model with virtually unlimited mailbox storage. In terms of usability, however, I much prefer Mail on Mac. On the iPhone, the native Mail client is far superior to the mobile web Gmail client. Fortunately, Google provides SMTP and IMAP services to fill this need. Sadly, both of these services are plagued with issues.

I recently sent an email which had roughly 50 bcc recipients. Google's SMTP server thought I was a spammer and banned me, despite my having authenticated via SSL. Perhaps sending email to 50 people is slightly unusual. Still, I would expect my mail gateway to be capable of performing such a 'feat'.

The way the Google IMAP maps directories is fundamentally incompatible with Mail.app expectations. Further, the IMAP server has a limit of 10 simultaneous connections, which often causes the connection threshold to be reached with just two connected clients. Additionally, the IMAP server is often down; I wish I had some data to support that, but I don't.

When I finally settled on boris@borismus.com, I had a decision to make: do I use Google Apps or Webfaction for email? I quite like Webfaction, and, perhaps irrationally, trust them more than Google with my private data. With the above limitations of Gmail in mind, I did not hesitate to choose Webfaction. I still miss having a first.last@gmail.com though. What's with the oddly draconian account deletion rules?

Plotting Something Radial

2009-08-07T13:00:17Z

Plotting Something Radial

Posted on Aug. 7, 2009

In anticipation of 48-739: Making Things Interactive, I've been itching to build something interesting. I decided to create a printing plotter out of my brother's Mindstorms set. There are already many excellent plotter designs floating around in the Mindstorms community, so I decided to try something new. Plotters typically draw straight, edge-aligned lines, since they have a caret motor which travels along the x-axis and a feed motor which aligns along the y-axis. There are many variations on this theme.

What if, instead of using the Cartesian coordinate system, a plotter was built against radial coordinates. That is, one motor would control the rotation of an arm (theta in radian coordinates), and one would drive a caret along the arm, controlling the distance from the origin (r in radian coordinates). I began building, and despite the ardent help of my young brother, we failed to create a reasonable construction. The problem we ran into was that as r increased to the maximum length of the arm, the engine driving theta would not have enough strength to rotate due to the increased torque. I'm ashamed to admit that we gave up.

A month later, I picked up the project again, and due to limited time, decided to build a Cartesian plotter after all. I did not consult state of the art of NXT plotters, and ended up with a plotter of unconventional design. Instead of a paper feed, a caret travels along the y-axis, and another caret travels along the first caret on the x-axis. I borrowed wheels from the RCX set, but the rest is stock NXT.

I named the robot "Malevich 2" in honor of Kazimir Malevich, an avant-guard Russian painter, famous for pioneering geometric abstract art, especially a series of paintings of squares. I prefer Malevich 2's rendition of the square, but maybe that's just me.

Extending Google Wave Terminology

2009-07-22T23:24:12Z

Extending Google Wave Terminology

Posted on July 22, 2009

For the last few days, I've had the chance to get my feet wet in the developer sandbox of Google Wave. My first impressions are very positive. I am as awed now by the scope and potential impact of Wave as I was after watching the hour long video from Google I/O.

As consolation to those itching to try it already, bear in mind that the sandbox is in a very raw state right now. Performance issues aside, informative text such as:

This wave is experiencing some slight turbulence, and may explode. If you don't wanna explode, please re-open the wave. Some recent changes may not be saved.

and

"Everything's shiny, Cap'n. Not to fret!" Unfortunately, you'll need to refresh. Wanna tell Dr. Wave what happened?

appear quite often. Of course, this is totally expected for pre-alpha software -- I'm merely pointing out the humorous messages!

For a more intimate look into Wave, especially some of the deeper, more social issues, here is part of an ongoing ha-ha-only-serious joke from the internal wave-discuss group about extending existing Wave terminology. The list is really telling, since indeed, I have read many ridiculous Drips and most of my fresh waves Surge only to get Borked by rickrolley.

Wavejack -- To hijack the contents of a wave to where it no longer resembles the original idea
* Drip* -- Stupid question no one answers
* Bork* -- To add useless, noisy or destructive bots to a wave. (aka The Swedish Chef or rickrolley or the borkforceone bot)
* Surge* -- The effect of a fresh wave that elicits the chaos of mass editing
* Drown* -- To have so many waves to follow that one can't keep up with them
Sea Sick -- A state of dizziness induced by an overly active wave
Tuna/Noise/Herring -- Wave speak for Spam
Fishnet -- Spam filter

Despite instabilities and widespread antisocial behavior, the wave sandbox is alive with a flurry of activity. People are busy creating annoying bots, forming role playing communities and writing collaborative books. So Google has a lot of issues to work out, not in the least how to stop us all from drowning :)

So far, I haven't had a chance to write any interesting wave extensions, but it's in my things to do!

Generating Guitar Chord Diagrams

2009-06-14T16:42:57Z

Generating Guitar Chord Diagrams

Posted on June 14, 2009

One day I wanted to add a feature to Guitar Unleashed which exists in some of the better guitar tab sites. When a user hovers over a chord, they are shown a diagram representing the guitar fret with overlaid finger positions required to produce this chord. Many of the most popular sites do this by showing a crude, plain-text representation of the chord.

For example, a C chord is shown as follows:

e ---|---|---|---|---|
B -x-|---|---|---|---|
G ---|---|---|---|---|
D ---|-x-|---|---|---|
A ---|---|-x-|---|---|
E ---|---|-o-|---|---|

Typeset guitar chord representation looks very different. Two most popular variants of the C chord would appear above the staff with small symbols that look something like this:

Such images have many benefits as compared to the plain text version:

Convention: many guitar players are used to learning from sheet music, so this chord notation is familiar to them.
Readability: the image representation is more compact and more pleasant to read.
Completeness: the image version has an easy way to display barre chords and specify fingerings (not pictured)

True to Guitar Unleashed's mission of being a truly usable guitar chords site, I decided to display typeset fret diagrams. For the implementation, a program would pre-generate images on the server based on the encoded chord shape and then serve these images dynamically via JavaScript. Unfortunately, I found no software to generate such images for all popular chords, so I ended up developing my own.

From some brief research, I decided that lilypond fret diagrams are the most elegant way of creating such images. The alternative was to use GCHORDS whose output I liked less, and which required depending on TeX. Typically lilypond typesets an entire piece of sheet music on a staff, with clefs, key signatures, etc. After discovering a really great lilypond bundle for textmate, I managed to get rid of these unnecessary features and display just a fret diagram. The following python-parametrized lilypond template does the trick:

\include "lilypond-book-preamble.ly"
\version "2.10.0"
\markup
\fret-diagram-terse #"%(chord_markup)s"

Then, using lilypond's fret diagram encoding, a python script replaces %(chord_markup)s with the desired markup. Lilypond's markup is very terse but mostly manageable. Sample markup for a C chord looks like this: x;3;2;o;1;o;, while the barre version looks a little bit more bizarre, like this: 3-(;3;5;5;5;3-);.

My python program contains a list of chord shapes for the most popular chords, encoded in the manner above, then outputs images for each chord and variant into a directory. I've made it available for download, in case your next project needs generated fret diagrams with a highly customizable look. You can also see the diagrams live at guitarunleashed.com.

Carnegie Mellon University!?

2009-04-18T12:47:07Z

Carnegie Mellon University!?

Posted on April 18, 2009

In late March, I was completely thrilled to hear that the [Carnegie Mellon University][] Human Computer Interaction Institute accepted me into their Master's program! In addition to admission, they offered a very juicy scholarship to spend part of the time on the beautiful island of Madeira. CMU was my most far fetched reach school, so I attribute this wonderful fortune to a clerical error made by the admission committee.

In addition to studying at one of the best universities in the world, traveling to Europe, and relaxing on the beach, here are a couple of reasons why my inner geek can't contain his excitement, try as he might. These incredible courses are offered at CMU in September 2009:

Making Things Interactive:

In this hands-on design-build class you will learn the skills to embed sensors and actuators (light, sound, touch, motion, etc.) into everyday things (and places etc.) and to program their interactive behavior using a microcontroller. Through weekly exercises and a term project the class will introduce basic analog electronics and microcontroller programming, as well as exploration into using kinetics and materials to make the things you design perform. Emphasis will be on creating innovative experiences using simple robotic technologies. The graduate edition of this course will require additional work including a paper that can be submitted to a peer-reviewed interaction design conference such as CHI, UIST, or TEI.Students from all disciplines are welcome: but please note that the class demands that you master technical material. Experience in at least one of: programming, electronics, or physical fabrication is strongly recommended.

Principles of Human-Robot Interaction:

This course focuses on the emerging field of human-robot interaction, bringing together research and application of methodology from robotics, human factors, human-computer interaction, interaction design, cognitive psychology, education and other fields to enable robots to have more natural and more rewarding interactions with humans throughout their spheres of functioning. This course is a combination of state-of-art reading and discussions, focused team exercises and problem-solving sessions in human-robot interaction, and a special team project resulting in the implementation of a human-robot interaction system.

Guitar Unleashed

2009-03-23T23:01:24Z

Guitar Unleashed

Posted on March 23, 2009

I've been collaborating with my dad on an experimental web-based guitar chord editing service. It's still a work in progress, but we are ready to launch a beta version. Please visit http://www.guitarunleashed.com/ to check it out and provide feedback.

We hope that the major features of the service are easily discoverable, but in case they are not, here is a list:

Add chords to your lyrics by clicking on any character
Move chords via drag and drop
Search for songs created by other users or from external sources
Save your work and share it with others
Add songs that you like to your favorites list
Already have lyrics with chords? Create a new song, paste lyrics with chords in and see the chords come to life.

Please note that the application relies heavily on JavaScript, so it needs to be enabled in your browser.

Known issues

Some lyrics may display incorrectly. This is an issue with LyricWiki.org
Long chords placed at the beginning or the end of lyrics line may extend outside of the editing area
Long lines in lyrics are not handled elegantly

Future features

Editing lyric text
Commenting on songs
Searching sources of lyrics and chords other than LyricWiki
Song books

Please give us feedback! Use the feedback button on guitarunleashed.com or contact us directly at feedback@guitarunleashed.com

Prettifying Embedded YouTube

2009-02-27T22:16:44Z

Prettifying Embedded YouTube

Posted on Feb. 27, 2009

In late January 2009, YouTube decided to change the default look of their embedded videos. They silently added an informative header which includes the video title and rating. Some time before then, a default search bar appeared at the top of the video. Thanks to these changes, most haphazardly embedded YouTube videos on the internet sport a repulsive new look.

What, you might ask, can be done about this excessive ugliness? Well, you could switch to Vimeo, which has a much nicer set of defaults, but that would mean bidding the YouTube community farewell -- and what a tragedy that would be! A better alternative is to learn the YouTube Embedding API, but there are still problems:

Embedding code for YouTube videos is ugly.
The YouTube embedding scheme often silently changes.
If you embed multiple videos, there is no way to specify a default set of embedding preferences.

If you are a wordpress user, all of these issues are resolved by the most excellent Smart YouTube plugin. With it you can embed videos by simply inserting the URL to any YouTube video into your wordpress page or post and replacing http with httpv. Through a settings page, you can modify the way all of your embedded videos look with one fell swoop.

Clean drag and drop upload in Safari

2009-02-15T00:25:50Z

Clean drag and drop upload in Safari

Posted on Feb. 15, 2009

Somehow I often find myself arguing in defense of the web browser as a viable platform for developing rich applications. In many such discussions, the issue of interoperability with the desktop arises. Someone will astutely observe that they can't even drag and drop from their OS file manager into their browser, and all hell will break loose.

Happily, this is changing! Since version 3, Safari on Mac OS X has had support for dragging and dropping files from the finder into file input boxes. In various kludgy ways, Firefox and IE are now following suite.

Unfortunately, even in Safari, the default look of the <input type="file"> box is quite ugly and the element itself is difficult to style. In addition, clicking anywhere in the file input element causes the default open file dialog to appear. I wanted to provide drag-and-drop uploading without ugly boxes or browser dialogs. The solution I came up with involves hiding the file upload box entirely by setting its opacity to 0, and then preventing the default action on click via event.preventDefault(). Here's a sample of what I mean, with the entire browser window converted into a drag area.

Note that the drag area must be the first DOM element to receive the drop event for this approach to work. Unfortunately I ran into a bug where the file dialog refuses to bubble click events to other elements below it. This is baffling to me, since event.preventDefault() should not stop event propagation, but only prevent the default browser handler from being called. You can see what I mean by trying to click the link in the sample HTML file. If this is not a bug, and someone has an answer, I would really appreciate it.

Note also that there are java applet-based drag and drop solutions, but they are reserved for developers who have nothing but disdain for their users.

iPhone car kit: roll your own

2009-02-05T22:23:31Z

iPhone car kit: roll your own

Posted on Feb. 5, 2009

Shortly after buying my first iPhone over a year ago, I found the right, low-budget solution to my in-car music needs. The ingredients involved are the following commonly found household items:

One Compact Disk
Some Duct Tape

The idea is to take the iPhone dock and stick it onto the CD with duct tape. Next, jam the CD into a tight slot in the dash of your car (easy to find for a Del Sol owner). Finally, attach power cables and/or AUX audio jacks to the dock. The result is a flexible, shock-absorbent mount for your everyone's favorite phone. I've used this hack for well over a year now with no problems. Here's how it looks in the interior of my car:

In a word: awesome.

Performance of Canvas versus SVG

2009-01-19T09:07:47Z

Performance of Canvas versus SVG

Posted on Jan. 19, 2009

At the core of the traditional HTML/CSS developer's toolkit is a set of nested boxes describing offset, margin, border and padding, known as the box model. Variations on the box theme are sufficient to describe most page layouts, but in some complex applications, it's necessary to render something more interesting, like diagonal lines, or polygons. There are currently two relatively well-supported web graphics technologies: SVG and Canvas. There are significant performance differences, however, which I would like to discuss in this article.

Scalable Vector Graphics (SVG) is by far the oldest of the two. It is a declarative, graphical language used to describe geometrical primitives via DOM elements. SVG was drafted in the late 90s, and the latest version of SVG, version 1.1 was finalized in 2003. It took 3 more years for it to be incorporated into shipping versions of Mozilla Firefox and Safari. Unfortunately, the length of the SVG adoption process caused the web development community to seek other options. The HTML5 Canvas element was introduced as a much simpler alternative to graphics on the web. It provides an image-like graphics context which can be accessed via a set of javascript calls, similar to a 2D subset of OpenGL. It was originally introduced by Apple in WebKit builds, but is now supported in Mozilla Firefox as well.

I produced some metrics to compare the two technologies in terms of performance by writing a Javascript program for collecting performance data. This program draws rows of circles onto a fixed-size drawing area in SVG and in Canvas, and then compares how long various operations take. It also records the duration to create the initial drawing context, to render the scene and to clear the scene. A test runner invokes the benchmark with the following variables: number of circles, drawing area dimensions and circle size. Each of these variables are varied independantly and automatically resulting in the following observations:

Here are the results of the first fruitful experiment, which clearly shows that SVG performance degrades quickly (exponentially on Safari?) in the number of objects, but Canvas performance remains at a near-constant low. This makes sense, since Canvas is just a bitmap buffer, while SVG has to maintain additional references to each object that it renders. Also, though not pictured, note that performance in clearing an SVG element also decreases in the number of drawn objects.

When varying the size of the drawing area, canvas performance degrades significantly, while SVG performance is completely unaffected. Canvas rendering performance seems to degrade linearly in the number of pixels in the canvas area. Not pictured on the graph is clear performance for large canvases, which also suffers linearly in pixel count.

I did not include the graph resulting from varying circle size as it had no significant impact on render time. Another interesting observation is that creating the canvas element takes a mysterious 10 ms on Firefox, but not on Safari. This is not significant unless you are dealing with large numbers of canvas elements.

Whether or not you use Canvas or SVG mostly depends on your specific application. A graphics-intensive game, where many objects are redrawn all the time is probably best implemented in Canvas. On the other hand, applications like map viewers may involve large rendering areas and might lend themselves better to SVG.

As always, please comment if something is unclear, inconsistent, boring or omitted. Thanks!

iWork.com Feedback

2009-01-07T19:47:31Z

iWork.com Feedback

Posted on Jan. 7, 2009

After spending a fair bit of time monitoring twitter feeds, I was pleasantly surprised by the world's response to the recently announced iwork.com. There were some premature flames, but mostly as a result of a fundamental misunderstanding of iwork.com's purpose. Someone even gave a technical breakdown of the application, which is especially endearing to me as an insider.

I See Dead CSS

2008-12-11T01:43:07Z

I See Dead CSS

Posted on Dec. 11, 2008

I wanted a tool that would analyze a complete web site, and report what CSS selectors and IDs are dead. By dead, I mean one of two things -- either the ID or selector is referenced from the HTML and undefined in the CSS or it is defined in the CSS but never referenced in the HTML.

A search for some CSS finding utilities proved somewhat fruitful. I found a brief survey of related utilities for this finding or cleaning dead CSS, the most promising of which was a Firefox plug-in called Dust-Me Selectors. But I wanted something that could be integrated into an automatic build process, easily invokable from the command line without requiring a browser, so I started thinking about a custom solution.

The problem can be solved as follows:

Find all referenced IDs and classes, called R
Find all defined IDs and classes, called D
Take a difference between the sets so that the list of undefined IDs and classes is (R - D), and the list of unreferenced IDs and classes is (D - R)

I've already had the pleasure of using BeautifulSoup python library to parse all sorts of HTML documents, and I quickly found cssutils to be a very handy CSS parser. In a matter of hours I was able to whip up a basic dead CSS finder in 100 lines of code using these great tools. I named it 7sense, after the movie. To run it, you just need to invoke ./7sense.py <list of files and directories> and the specified files will be parsed as if they were all part of the same webpage. Directories will be walked recursively, with all encountered CSS and HTML files assumed to be part of the web page.

But things are not as simple as I had hoped, and my program has several notable limitations.

Due to lack of time, 7sense does not look at what stylesheets are referenced by an HTML page. Instead, you have to tell it explicitly what stylesheets are used by passing them as arguments.
7sense does not properly decipher heirarchical CSS selectors like '#myContainer .aboutBox'. Instead, it splits the selector into whitespace separated tokens, ignoring their structure. I skimped on this feature also due to lack of time.
More fundamentally, 7sense is not aware of any Javascript modifications to the DOM. This could be remedied on a case-by-case basis. For example, one could write a parser to seek jQuery.setClass invocations, and extract additional classes from there.

At any rate, here's 7sense so far. Though flawed, it's a useful start. Ideas and code improvements, especially addressing the above limitations are very much appreciated!

Musical Mashups in Pure JavaScript

2008-12-04T21:18:37Z

Musical Mashups in Pure JavaScript

Posted on Dec. 4, 2008

When I want to learn a new song on guitar, I often search for chords online. There are many sites that provide chords and tabs, and Google indexes them nicely. But the quality of chords is often poor, and there's no way to submit corrections. When I ran a MoinMoin wiki, I kept my fixed versions of songs there. Even making modifications to existing chords was painful though, since it involved hand-editing a plain text file and ensuring that the chords were properly aligned with the lyrics. My preferred solution to this problem is to write a web application to facilitate easy collaborative editing of simple folk/rock/pop songs.

I started working on a proof-of-concept prototype. To begin with, I wanted an easy way of finding song lyrics, which provide the skeleton for most guitar pieces that I'm interested in. Later, I planned to work on annotating those lyrics. As I prepared to whip up a light django application to scrape lyrics sites, I realized that there may be a simpler way: if I found a friendly lyrics API, there would be no need to write any server side code. Could the entire service be written in Javascript?

LyricWiki.org had exactly what I need: a simple way to access lyrics in JSON format. Of course, you can't just do an XmlHttpRequest to lyricswiki.org because of XSS security restrictions. Instead, you can work around this cross-domain scripting issue by writing out JSONP. This technique makes it easy to simply evaluate the script tag, and wait for the song variable to change. Now If only there was a safe way of doing this sort of cross-domain scripting without introducing a host of XSS vulnerabilities, writing mashups would be a walk in the park!

After only two hours of hacking around, including learning jQuery, I came up with a little demo application, written in pure Javascript. Hooray for jQuery, humanmsg, and the LyricsWiki API!

Moving Day

2008-11-24T01:08:28Z

Moving Day

Posted on Nov. 24, 2008

Fear (of Shaw downtime) is the mind killer. So as of today, this blog is hosted on WebFaction.

Robotic Piano Playback

2008-11-15T02:56:39Z

Robotic Piano Playback

Posted on Nov. 15, 2008

After several weeks of casual spare-time research and implementation, I've finally built a fully working piano playback robot. The usage is simple: someone plays or sings an arbitrary monophonic melody, and the robot, parked on a piano bench, will play it back.

The physical construction of the robot is very simple: it consists of a car with a crane-like arm mounted on it. The arm is used to push down and release a single piano key. On either end of the car, there are sensors which detect if the robot has come too close to the edges of the piano bench. The simplicity of the robot comes at the price of significant limitations, such as only being able to play back melodies on the white keys.

Software is the challenging part of the project. The Mindstorms sound sensor is too primitive to use for pitch analysis. Without hacking it, you can only extract the amplitude of the sound signal, not any frequency details. Instead of the NXT sound sensor, I use a macbook pro and it's built-in microphone. A computer separate from the NXT is involved, so an additional set of communication problems arose.

Here's a rough outline of happens to make the playback work, from capturing the melody line to playing the melody back.

On the mac, using a Python AppleScript bridge, the QuickTime Player is invoked and starts capturing audio.
Once the audio is captured into an AIFF file, a very useful pitch detection library called aubio processes the audio file and extracts raw frequency-to-time data, sampled at some tick rate. Compiling this library on OS X was quite a feat!
Next, the raw data is processed by throwing out extraneous values and extracting a melody
Once we have the melody line, we inject it into an NXC program, and compile it with the nbc compiler
This program is then sent via bluetooth to the robot via nxtcom
Using NXT_Python and Lightblue Glue, the robot is told to execute the program.

Please let me know if you have any questions or suggestions!

The (Sorry) State of HTML Mail

2008-10-31T12:03:12Z

The (Sorry) State of HTML Mail

Posted on Oct. 31, 2008

Web design used to be a black art. Ten years ago, browser differences used to be so dramatic that the only viable solution for an HTML designer was to fall back to the least common denominator for page layout, which was HTML tables. In today's web design community, using table layouts is considered to be a heinous crime, since most popular modern rendering engines (IE, Gecko and WebKit) are converging to some shared interpretation of web standards. Unlike layout engines on the web, though, rich mail interpreters have remained stagnant, and in some cases have regressed. ~~Without pointing any fingers...~~

Much of the blame for this difference lies in Microsoft's decision to use the same engine for composing and viewing email in Outlook 2007. They wanted to make the life of email designers easier, or so the story goes. But since Internet Explorer doesn't have editing functionality, and Front Page is too heavy to embed, the remaining choice was Word '07. Unfortunately, Word's rendering engine is extremely limited, in the following notable ways:

No positioning or floating elements, so CSS-based layouts are out
No CSS backgrounds, combined with (1) means that there's no way to have an image background at all.

So, if you care a significant population of email readers (7% use Outlook '07), and want to deliver a rich media email, you are condemned to designing with table layouts, or to skip HTML altogether, and simply use images.

For a long time, there was no good way of determining a breakdown of email client usage. Recently, the good people at fingerprint have come up with a solution, and now use a large sample of people to determine global email client usage statistics. According to them, Hotmail, Yahoo Mail and Gmail add up to roughly 50% of all usage.

Since the UI of a web-based email client is written in HTML, so it is critical for webmail developers to ensure that the CSS and HTML found in the email does not interfere with the global look and feel. The canonical solutions are to decorate all ids and classes in the email with some kind of prefix to ensure that there are no name collisions, and use a white list approach to CSS styles. These white lists are usually quite long, but lack some important and oft-used properties. For example, CSS image backgrounds are disallowed, except on Yahoo. And of course, Microsoft had to leave its bizarre mark too: Hotmail strips the margin-top property, but not the margin-bottom property.

Historically, webmail clients used to ignore anything outside of the <body\> tag, which meant that all CSS had to be written inline, leading to unmaintainable layouts. In recent tests, however, popular webmail clients no longer ignore <style\> elements in the <head\>, and instead, apply the CSS sparingly inline. Among modern native mail clients, there is a positive trend as well. Microsoft Live Mail uses the IE7 rendering engine, Thunderbird uses Gecko and Mail.app uses WebKit. So it looks like there is light at the end of the tunnel for downtrodden HTML email designers. In the meantime though, I send them my deepest condolences.

Top 5 Sessions of Web 2.0 Expo NY

2008-09-23T00:22:16Z

Top 5 Sessions of Web 2.0 Expo NY

Posted on Sept. 23, 2008

Yesterday I got back from the Web 2.0 Expo in New York. There were many interesting sessions, and I, for the most part, took notes! Here is a short list of my five favorite speakers, in no particular order:

Cal Henderson's whirlwind introduction to video on the web. Cal really likes speak really fast, to make typos in his slides and to cuss, all of which makes for a very informative and entertaining hour.
John Resig's talk about Processing.js, his port of Processing to canvas. I'm constantly dealing with graphics on the web, and it's awesome to have such a powerful library available. Pretty demos, too!
Jason Fried's short but sweet keynote talk, focusing on minimalism in product management, and proper (read: very narrow) scoping of features. The philosophical question of "what would your software be like if it was physical?" struck me as a very useful thing to think about.
A browser panel including Chris Wilson from MS, Brendan Eich from Mozilla and an unnamed developer from Chrome responded to a nice set of questions about future directions of browsers. Poor Chris got a beating IE7's flaws, lack of canvas/SVG support, barriers to plugin development for IE.
Geir Magnusson Jr delivered an excellent introduction to scaling data in the cloud.

I also had the chance to talk to and bounce some ideas off of Pete Koomen, a Google App Engine product manager. He told me that there are plans both for process scheduling, and for support of django-1.0 down the road, but of course gave no time frame for either. Overall, the conference was interesting - there were other good sessions and keynotes which I simply haven't bothered to write up. The crowd wasn't very technical though, comprising in large part designers, marketers and managers.