Recently, I gave a talk about the development of a typographic portrait generation engine we built for Bell Social Portrait. It’s a story that explores some of the challenges of building a complex web application meant to be experienced on many different screen sizes and platforms. The project taught us valuable lessons that will surely influence future work.

Bell Social Portrait is a site that pulls a users social feeds and programatically generates artwork from an associated profile or user-submitted image. Traditional typographic portraits are normally constructed in graphic design software like Illustrator or Photoshop. We knew it’d be a challenge to simulate the process programatically – never mind deploying it to as many browsers and devices as possible (a self-imposed requirement).

Early research and prototyping was all done using the Canvas tag as a drawing surface – probably the most ubiquitous and well supported API’s in HTML5. Our intent was to generate as much reusable code as we could for the finished implementation. This meant that all image segmentation routines were written in Javascript – a task made easy by some groundwork I’d laid a few months earlier.

ASCII art (specifically image to text conversion) was a major source of inspiration for this project, and its basic implementation served as a good starting point. Typical ASCII art uses a quadrant (one region evenly split into four) based color weighted lookup to match character glyphs. The process is actually quite simple. Take, for example, the letter “L” – it could be mapped to a region with bright color (lots of white) in the top-left/bottom-right/bottom-left quadrants. If you’re interested in the full process (and more than just the most basic implementation), I highly recommend reading The Art and Science of Text Mode Conversion by Jari Komppa.

We used this same quadrant splitting practice in order to segment images into a cell grid of varying sizes. This type of structure is commonly known as a quadtree, and involves recursively splitting an image with one simple condition: stop splitting if the region contains only one color. You can see the results illustrated below at different iteration (number of times to recursively split) values.

Next, we merged all adjacent cells of the same dimensions. This produced a useful grid for mapping words, but we weren’t pleased with them aesthetically. Cells appeared too organized and symmetrically positioned. We wanted an element of chaos, and this meant a new approach was required.

We began investigating various box fitting strategies, and quickly began building our own algorithm. The approach was simple:

1. Pick a random point from an array of active pixels (not white).
2. Move that point horizontally left or right until a null pixel (edge of the image, white pixel, or another cell) is hit.
3. Move vertically up or down until the same condition is true.
4. Expand a rectangle, with a width/height ratio chosen randomly from a predetermined set, until any of its pixels hits a null pixel. Mark that rectangle as a cell.

Aesthetically, the resulting grids had the right amount of randomness and produced visually appealing portraits. They looked great, but a couple problems arose – average computation time of 30 seconds in Chrome, and huge performance variability across difference platforms. iOS (pre 5.1) performed particularly poorly, with the worst cases fully locking up devices. We needed to find a workaround, and that meant moving image segmentation to the cloud.

For a number of unrelated reasons which included tight application and web serving integration, an easy interface to mongodb, and the need for an event based architecture, we had already decided to implement the server portion of Social Portrait in Node.js. And since Node is basically “Javascript for networking”, the task of porting the image segmentation routines written for the browser was relatively easy. Thanks to Learnboost’s node-canvas package, we were able to simply copy and paste most of our code.

Recreating the portrait generation process on the server meant we could free the browser from any application level computation and have it narrowly focus on UI. This was a huge deal. The browser experience became light and responsive to different devices and platforms. Also, since we prototyped portrait generation in the browser, we could comfortably pass off a small JSON UI properties object along with the unaltered original image to the server. Bandwidth usage became very light – an important factor when accounting for mobile devices. Finally, we knew exactly what hardware the code was running on, which established a uniform performance metric. The performance variability problem had been solved.

Our decision to move image segmentation to the server also meant we had some new powerful tools at our disposal. One of these was imagemagick – a very mature, fast, and well supported graphics software suite that we could access through a child process. It allowed us to preprocess our input images with very little overhead. One technique we leveraged with imagemagick was down-up-sizing images before grid generation inorder to produce a blocky canvas. This resulted in an easier image to fit rectangles on, which reduced noise and improved segmentation speed.

The real fix to our speed problem came from having the ability to write our own native programs in C++ – specifically the ability to use data structures that do not exist in Javascript. When iterating over an array of active pixels (not white), we had no way of skipping over elements that were already assigned to cells. We tried many tricks in Javascript including overly complicated dynamic lookup tables, and splicing arrays. Nothing worked well. Then we found dynamic_bitset from Boost C++ – a data structure that only contains bits (on/off) and more importantly does not iterate over “off” elements. This means that as cells get progressively marked unavailable, iterating over the active pixels array gets faster. We rewrote our grid generation code as a C++ program and wrapped it in a Node child process. The result was a 10x speed increase – an average of 3 seconds!

It may seem like this story is a glowing endorsement of Node.js, but that’s not necessarily the case. Node was a very valuable tool in this project for a number of reasons – the ability to prototype directly on our drawing canvas and seamlessly port code over to the server, the access to a native environment that solved huge performance problems, and the access to powerful tools like imagemagick for preprocessing. But had we done the project over again, I’m not sure we would have implemented our entire server environment in Node. Static web serving is better handled in tried-and-tested environments like Apache or nginx, and worker management for multiple CPU cores can be daunting.

Nonetheless, the most important lesson we learned is that cloud technology and the new breed of server-side environments are necessary tools to utilize in modern-day web applications. With the variability of devices and platforms users move between, we can no longer solely rely on client-side environments to deliver rich experiences. Frame your UI’s in the browser – let the cloud do the work.

Yesterday we (Labs) ran a half-day hardware hacking workshop internally for all of the Teehan+Lax employees (everyone from designers and developers to the project managers and the “HR” people). It was meant as a light introduction to physical prototyping. Here are some of the creative, fantastic (and horrifying) results. We were very excited to see everyone’s results at the end of the day. The great thing was that everyone had successfully created something by the end of the workshop from scratch.

“The perception of system output must be greater than the perceived user input, and within that range there is a sweet spot where [user experience] is delightful and magical.”
- August de los Reyes

I can still recall the first time I saw an Augmented Reality demo. There was a sense of wonderment from the illusion of 3D models living within the video feed. Of course, the real magic was the fact that the application was not only viewing its surrounding environment, but also understanding it. AR has proven to be an incredible tool for enhancing perception of the real world. Despite this, I’ve always felt that the technology was somewhat limited in its application. It is typically implemented as output in the form of visual overlays or filters. But could it also be used for user input? We decided to explore that question by pairing the principles of AR (like real-time marker detection and tracking) with a natural user interface (specifically, touch on a mobile phone) to create an entirely new interactive experience.

The outcome of this examination is something we’re calling Touch Vision Interface – a tool that enables touch interaction on many different connected surfaces through a mobile phones’ camera view. The translation of touch input coordinates to the captured video feed creates the illusion of being able to directly manipulate a distant surface. As a result, the interaction feels natural and almost invisible.

Future applications of this technology are more than compelling. The barrier of cross device communication is lessened, whether in the living room or in large open spaces. Brands could crowd-source easier with billboard polls. Group participation on large installations could feel more natural. The possibilities become even more exciting when considering the most compelling aspect of the tool – the ability to interact with multiple surfaces without interruption. No need to switch devices through a secondary UI – simply touch your target. You could imagine a wall of digital billboards that users seamlessly paint across with a single gesture.

Other applications could enhance the collaborative creative process. Imagine a music creation experience where each screen becomes an instrument. Pitch and tone could be changed by moving your finger across the instruments surface allowing for natural gesture controls. Rather than being paired with an instrument, a band member simply plays what they’re “looking” at.

There are flaws with this technology, surface discovery and pairing being the most obvious. But along with the problems, it’s easy to see how this idea could be extended – moving past simple planar interaction and deeper into the world of real-world object manipulation. We’re excited to see how far we can take it.

“It’s clear that simplifying and physically tailoring input to output with tech like NFC is a good way to enhance user experience and can lead to some unique interactions. We plan on taking this idea further by experimenting with other platforms and input devices.”
—Physical and Frictionless Input. 7 June 2011.

It should be about the experience. Not the technology.

Continuing on with physical interaction we’ve been experimenting with Arduinos, RFID/NFC, social network APIs, and dreaming up various other end uses that we will discuss in a minute and end off with a really quick (but verbose) demo. When dealing with technology for novel interaction we really want to make sure that the interaction is put before the technology. A tech demo for tech sake isn’t always the best approach. Inventing solutions to problems that don’t exist doesn’t make much sense. Products (specifically tools — and digital or physical) are meant to service something and then get out of the way. If we take this approach to using RFID/NFC then whatever that tool ends up being should show up, do the job, then completely disappear so you can get on with living life. Without you needing to do anything. You don’t give the plumber step by step instructions do you? You just call and tell them to come fix something.

Imagine being at a Jay’s game and being able to swipe your ticket on your arm rest to +1 or like a big swing by José Bautista (or rather to +1 his childish temper tantrum over the weekend). Or maybe this winter we’ll finally be able to tweet that we witnessed the first Leafs Stanley Cup win since 1966/67 just by having your ticket in your pocket and your chair knowing it’s you. Because, obviously, there’s no way any of us are gonna stop cheering to get our phones out of our pockets to tweet about it.

Let’s face it though, that won’t happen anytime soon so maybe you could check in on Foursquare at a conference by swiping your conference pass at the entry gate. That could be extended into checking in anywhere. Check in at your work, the various rooms in your house, or check in at your toaster or fridge or at your computer for an all night Minecraft marathon.

We can even microchip ourselves — the same as microchipping your pet — and use that to send tweets for various appliances and pieces of furniture we end up using. Or more practically that microchip could be used to engage a locking mechanism on the front door or store personal identity/health info. It would be pretty rad if you could keep your health records stored in the fleshy bit between your thumb and index finger. The use cases for RFID/NFC are quite varied from really practical to kinda shady, whimsical, and fun.

The idea of this kind of passive interaction; and passive data logging, tracking, and publishing is both frightening and exciting. It could go down the scary big brother road where every action we take is recorded and posted somewhere publicly and used for ad streams and evil. Remember when your significant other could see the lovely purchases you made in secret for them on Amazon showing up on their Facebook wall, or when robbers used Foursquare to track good targets because they checked in somewhere that wasn’t their own home? Yeah that wasn’t cool.

I don’t see that happening…much. I see the nice green pastures of passive interaction for the betterment of humanity. There have already been many projects from around the world that prove this true many times over. More and more of these projects are moving away from tracking offline activities on social networking sites and are moving towards (actually a bit backwards — think remote controls and animal microtagging) friendly tangible interfaces and controllers for a variety of different products.

Before we get into the demo proper, a quick technical breakdown. An RFID is hooked up to an Arduino (or any microcontroller, we picked Arduino). That Arduino is in turn hooked up to a wifi or ethernet shield (a custom breadboard that adds additional functionality to the Arduino, in this case, the whole Internet) which sends information (a string containing an RFID Tag serial number, or in the case it’s rewritable, whatever we can squeeze into 1K of data) to a server (actually it sends a Serial communication across TCP/IP). The server is running a Python script that is listening for data coming from the Arduino and uses that serial number which is synced to a person’s OAuth access tokens to validate them and post some action to their account on a social network.

This setup can very quickly bring the offline world into the online world.

The part where we get to make something.

So, we’ve put together this simple system together that can send messages to a twitter account whenever you swipe a certain RFID tag over the Arduino. For the sake of keeping the demo as cost efficient as possible and simple, we won’t be doing any TCP stuff. Save that for next time. Plus, adding internet to your Arduino starts at about $50. We’ll just send serial from the Arduino to the laptop via the USB cable. Just pretend it’s an ethernet cable.

The prequel to the steps is to acquire a bunch of Arduino parts. You can get all this stuff from Sparkfun or Makershed and it’ll take maybe a week to ship.

1 x Arduino
1 x RFID Reader (breakout board or usb reader, we used the usb version)
1 x ID-12 or ID-20 RFID Reader
1 x RFID 125kHz Tag

Other things you’ll need are a bundle of Breadboard Jumper Wires and a Breadboard. You’ll probably also need some 6-pin and 8-pin headers (for the breakout board). If you need the headers you’ll need a soldering iron and solder. If you’ve never soldered before¹ you may also want a Solder Sucker and a Solder Wick for clean ups and fix ups. If you drink a lot of coffee I would also recommend a Helping Hand Vice.

Now that that is out of the way let’s build something. The first step is to assemble the Arduino. We’re using an Arduino Uno, Sparkfun’s RFID USB Reader board, and an ID-20² RFID Reader. If you’re using an RFID breakboard the PIN mapping is a bit different but will be explained as well.

For the USB Board we soldered a 6-pin header and stuck it on a solderless breadboard. From the Arduino attach digital pin 6 (marked ~6 on the Uno) to TX-0 on the USB Reader, 5V to VCC, and GND to GND and GNDT.

—Made with Fritzing. Showing wiring for RFID USB Reader.

With a breakout board you attach ~6 to D0 (pin 9), 5V to RES (pin 2) and 5V (pin 11), and GND to GNDT (pin 1) and +/- (pin 7).

—Made with Fritzing. Showing wiring for RFID Breakout Board.

The wiring is really easy if you plug the 5V and the GND from the Arduino into the + and – rails down the edge of your breadboard. You can then plug the various powers and grounds into the RFID reader from there. That completes setting up the hardware. It’s not too complex a set up.

There are two different pieces of code. One is Arduino³ and the other is some very simple Python⁴. We’ll set up the Arduino software first so that we can send a message out over serial.

For Arduino we also need the third party library called NewSoftSerial.

First we’re going to define a bunch of things that we’ll need later on and import the NewSoftSerial library.

#include <NewSoftSerial.h>

NewSoftSerial rfidSerial(6, 7); // (RX - pin 7 on usb rfid board, TX - pin 6 on usb rfid board)
String rfidTag, rfidTagClean;

int i = -1;

// Henry V Tweets
char* tweetStrings[] = {
  "O, pardon! since a crooked figure may Attest in little place a million;",
  "And let us, ciphers to this great accompt, on your imaginary forces work.",
  "Turning the accomplishment of many years Into an hour-glass:",
  "for the which supply, Admit me Chorus to this history;",
  "Who prologue-like your humble patience pray, Gently to hear, kindly to judge, our play."
};

The next thing we need to do is set up the communication over serial. void setup() is a special function that runs once when the application starts.

void setup() {
  Serial.begin(9600);
  Serial.flush();
  rfidSerial.begin(9600);
}

So all we want to do is open some ports and make sure there that anything that is currently there is flushed out of the system.

Next we need to build the rest of the application. Another special function in Arduino is void loop(), this function constantly runs, looping back to the start of the code when it reaches the end.

void loop() {
  if (rfidSerial.available()) {

    char incoming = (char)rfidSerial.read();
    // Packet structure, see ID-12 datasheet.
    // STX A B C D E F G H I J Y Z CR LF ETX
    //   2 - - - - - - - - - - - - 13 10 3

    // If the last character of the packet is 3, move on.
    if(incoming == 3) {

      // Skip the first character and take the next 12
      rfidTagClean = rfidTag.substring(1, 13);
      rfidTag = "";         

      // Shakespeare's Henry V Prologue/Act 1. You'll need to change the tag number to one of your own tags. See the final else statement on how that happens.
      if(rfidTagClean == "4500B89EC1A2") {
        i++;
        Serial.println(vStrings[i]);
        delay(500);
        // Reset counter so we can start reusing the already used tweets. 0 - 4.
        if (i == 4) {
          i = -1;
        }
      }

    } else {

      rfidTag = String (rfidTag + incoming);
      Serial.print(incoming); // Prints the the 12 digit number we want to use for the rfidTagClean if condition.

    }

  }
}

Inside the loop the first thing we want to do is make sure that the code only runs if we can detect that the RFID reader is available. From there we say if the scanned RFID Tag matches this 12 digit number print one of the tweets, and every time this tag is read move to the next tweet. On the 5th tweet go back to the start of the list. If the RFID tag does not match print the tag number to the serial monitor so that we can track it. Either add it to a list of approved tags or not.

That’s it for the Arduino. You can upload that sketch to the board and run it. Open the serial monitor, swipe your RFID tag and see what happens. Don’t worry, we’re not tweeting yet, so have at it.

The final step is to write a super small Python script that monitors the serial port that Arduino is communicating on and if it gets a message send it to Twitter. Unfortunately, before we do that we have to register an application with Twitter so we can get OAuth out of the way. That is probably the pain point of the set up. OAuth is also why I’m not showing you how to like on Facebook, or check in on Foursquare. Dealing with OAuth and social network APIs is full of no documentation or worse… bad documentation. Okay, go log in to your Twitter account, hit up the developer section and register your applicaiton. Make sure to give your app Read & Write privledges and finally authenticate your account with your app. This way not only will you have the Consumer Token and Consumer Secret, you’ll also have the Access Token and Access Secret which is a unique pair for every account registered.

Okay, one more caveat. There are 3 additional python libraries. Tweepy, pySerial, and Time. I’m also using the Python 2.7 build not Python 3.x.

import tweepy
import serial
import time

# make sure this is your arduino. Open Arduino > Tools > Board
arduino = serial.Serial('/dev/tty.usbmodemfd121', 9600)
status = arduino.readline()

auth = tweepy.OAuthHandler("XXXXXXXXXXXXXXXXXXXXXX", "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")
auth.set_access_token("XXXXXXX-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")

api = tweepy.API(auth)

while 1:  						# Infinite Loop
	status = arduino.readline() 	# Wait for new line to come across Serial
	api.update_status(status) 		# Post message to Twitter with Tweepy
	time.sleep(20) 				# Wait 20 seconds
	del status					# Clear status so we can start over with a clean slate

Okay, what’s going on here. We import the libraries we need to handle everything we’re about to do. Next we set up the serial so Python is listening to the Arduino so that it can send each line as a new tweet. The println function in the Arduino code sends each string with a new line (\n) at the end. After that we set up the Twitter authentication and tell Twitter’s API to use those specific credentials which will post to that account. The while function continously monitors serial data and each new line that comes across it posts to Twitter.

Upload the Arduino sketch to the board, leave the Arduino plugged into your computer via USB and run the python sketch in Terminal:

cd /path/to/python/sketch/
python twitter.py

And you’re set. Swipe the card and impress your followers with your new powers.

Taking it a few steps further we could move this back into the ethernet/wifi world. But something that might be more interesting is to attach the unique access tokens/secrets to an RFID card so that if Card 1 is swiped use my unique credentials and post to my account OR if Card 2 is swiped use your unique credentials to post to your account. This way you could have one piece of hardware posting for everyone that has a valid RFID card. Which becomes more compelling. More soon.

–

1. A few notes on electronics. If you’re buying a soldering iron for the purposes of hobby electronics I don’t recommend the ‘cold heat’ style soldering tools. They work by producing an electrical current that could possibly short out your circuits. Stick to the regular soldering iron that just heats up the tip. Just be careful to not burn yourself. And possibly get adult supervision. ↑
2. The ID-2, ID-12, and ID-20 all have the same PIN mappings. However, the ID-2 does not have an internal antenna so you’ll have to build one. Which is painfully complex. The ID-12 and ID-20 both have pins exposed for adding an external antenna but this will cause all kinds of reception problems. ↑
3. Arduino is both a hardware platform and a programming language. Actually the language is based on Processing which is built on Java. Arduino is built on C/C++ but maintains most of the syntax styling of Processing. So bits are written like Java and other bits are written like C/C++. Don’t worry it’s not as confusing as that makes it sound. ↑
4. Python is like Ruby but better. ↑

In the last years, with the advent of new web enabled devices — smartphones, tablets, connected TVs and more — our digital landscape has been rapidly expanding beyond the borders of traditional desktop/laptop screens. As mobile devices are increasingly becoming the primary way we access online content, responsive design has emerged as a popular design strategy.

Designing for all devices: apps and mobile websites

Smartphones and tablets have complicated the task of digital strategy and design by adding new resolutions and technical constraints. With plenty of different mobile devices around, dealing with this complexity—in particular, reconsidering the traditional 960px fixed width viewport—has been approached in two ways:


1. Create complimentary (native) mobile apps (a trend fostered by manufacturers with their ad hoc SDKs and app stores);

2. Create multiple mobile versions of websites and try to cover as many variations as possible.

 
Both of these options aim to create experiences that are perfectly adapted to the context of the device they appear on. Although they might look ideal from a coverage and support perspective, the problems with both these options are that:

1. They require an additional financial, design and maintenance effort that may be overwhelming for clients.

2. They may well prove to be a Pandora’s box in the long run as an endless number of new devices will appear with their own specificities.

Designing for any device: responsive web design

In parallel, another strategy emerged last year in the shape of a seminal article written by Ethan Marcotte, entitled Responsive Web Design. To avoid these drawbacks and based on the assumption that a web experience should (and could) be the same no matter the device (“one web”), Marcotte defended the idea of a single fluid website rendering different layouts per device.
His point was that a website designed following responsive principles would be more “future proof” as its layout would adapt to any device, content being smoothly rearranged and transformed to fit the device in question:

“Rather than tailoring disconnected designs to each of an ever-increasing number of web devices, we can treat them as facets of the same experience. We can design for an optimal viewing experience, but embed standards-based technologies into our designs to make them not only more flexible, but more adaptive to the media that renders them. In short, we need to practice responsive web design.”

To achieve this new ideal of “responsive design”, Marcotte defined three integrated features:


Many websites (including this one) launched in the last few months embraced this new approach and jumped on the responsive bandwagon with more or less success. Responsive web design is certainly a step forward towards a better and more flexible web, but it should not be accepted as the only way to go.

Our experiments with responsive web design

Recently, we experimented with responsive design for the new T+L website and thought it could offer an appropriate solution to supporting multiple devices.

We decided to prolong the experimentation with an internal challenge: T+L designers and developers were asked to come up with a series of layouts that would respond to changes in screen size. Ultimately we analyzed the outcome of this challenge and spent time breaking down responsive frameworks to figure out what were the pros and cons of using the responsive approach.

We identified two methods to initiate the design process:


• Either as a “progressive enhancement”, with mobile layout first and desktop layout at last.

• Or as a “graceful degradation”, with the full desktop website as a basis to design the other layouts.

So, what’s good in this?

Responsive web design is a valid approach as it requires a single effort to support a multitude of devices. Content leads the way and can be increased, reduced, shrunk or removed on demand and depending on context.

Then where’s the issue?

Actually there are a few…
Responsive design is based on the assumption that the substantive features of online content should not vary across devices. When we consider how much the user’s context can depend on where and how they are accessing online content, we see that this assumption may not be true all the time. If we use the case of a contact page as an example, it makes sense to display only a map or driving directions for the mobile context (since users are likely to be on the road), while displaying office photography and detailed contact information for the desktop context (since users are stationary and can afford to browse longer). A responsive design approach is too undifferentiated for these situations.

Creating a responsive layout is not easy and requires a lot of preparation – it must accommodate a series of grids on both portrait and landscape modes, as opposed to a fixed width layout with a single vertical view that adheres to a single grid system. This effort could be substantial, and might be a waste if the intended targets are explicitly defined (like in our contact page example). Ultimately, it’s important to ask: should my design be responsive or adaptive?

Another consideration we identified with responsive designs was their difficulty to treat type contextually. We observed that they were often too deterministic in trying to replicate typographic elements on small screens that were of minor importance for mobile usage. Conversely, we noticed some responsive designs were so fluid that they became too loose (excessively wide text columns, margins, etc.).

There are many solutions you can consider when trying to tackle varying screen sizes, but picking the right one depends on your projects’ goals and device context. Trying to find a golden solution might not be the right approach. Responsive design is just one of many solutions that can help with that problem, but it’s not the miracle solution that many are hoping for.

UPDATE 07/20/11: Updated the demo to take better advantage of built-in Paper.js features.
UPDATE 07/27/11: Compacted the demo even further utilizing the active layer object.

Recently, we’ve been exploring various vector graphics libraries in an order to craft some more compelling data visualizations. I’ll admit, we weren’t too enthusiastic about the prospect of manipulating SVG or learning some strange custom syntax. Fortunately for us, Paper.js was released a few weeks back.

Paper.js is a brand new open source vector graphics scripting framework with two important distinctions. First, it’s actually nothing new. Paper.js is entirely based off Scriptographer, the ten year old scripting framework for Adobe Illustrator, complete with its own mature and battle-tested display hierarchy and Object Model. This is welcome news for any developer familiar with Illustrator or – more importantly – Flash. Second, it uses the Canvas element as opposed to SVG. Besides the speed boost this grants, it also means we can take advantage of the rasterization and bitmap manipulation features of Canvas.

We’ve put together a simple example that showcases some of the structure and features of Paper.js including debugging guidelines, smoothing, and path manipulation. Additionally, it demonstrates simple interactions and event handling.

Click to view our tutorial example.

Paperscript (the Paper.js extension of Javascript) is an important differentiation from other vector graphic libraries in that it provides a sand-boxed scope for Paper.js scripts. It treats objects such as Point and Size like native data structures with operator overloaders in full effect. For example, you can add two points by simply performing: point1 + point2. Contrast this to a similar framework like Raphael, which requires the cumbersome manipulation of SVG strings, and you can start seeing the benefits of paperscript.

The first step in our demo is to declare our paperscript and tie a canvas instance to its scope. We’ll also declare some built-in handlers for resize, keyboard, and mouse movement events. The handlers are automatically bound once declared – no need to attach them to an element.

<canvas id="canvas" resize keepalive="true"></canvas>
<script type="text/paperscript" canvas="canvas">
	function onResize() {
		//
	}

	function onMouseMove(event) {
		//
	}

	function onKeyUp(e) {
		//
	}
</script>

Let’s construct a simple fake terrain effect with depth being simulated by brightness. In this case we’re building a series of paths stacked vertically with an equal number of segments across each path resulting in a square grid (gridSize x gridSize). First we need to declare our grid dimensions.

var gridSize = 20;

Path is the core visual building block in Paper.js that is used to construct other vector graphics objects in its display hierarchy. We’ll be using them directly to build our grid. Every row in our grid represents a new Path object with columns being represented by that object’s segments. We’ll instantiate each path on every loop iteration. Next we’ll use the Path method add() to add the segment points.

for (var j = 0; j <= gridSize; j++) {
	var path = new Path();

	for (var i = 0; i <= gridSize; i++) {
		path.add(new Segment());
	}
}

Let’s change some of the rendering properties of each Path so we can visually differentiate them. As we loop over each row, we assign a stroke color to the path which becomes brighter as increments get higher. Paper.js has built in color objects for handling RGB, HSB, HSL, and Grayscale. We’ll use HSLColor to display a range of greens. The stroke width can be set globally and inherited by every newly instantiated path. Paper.js has a base Project object which acts as our “document”. We set the stroke width of our projects’ current style to a value proportional to the canvas dimensions.

project.currentStyle.strokeWidth = (view.size.height + view.size.width) / 0.8 / gridSize;

for (var j = 0; j <= gridSize; j++) {
	var path = new Path();
	var brightness = j / gridSize * 0.608;
	path.strokeColor = new HSLColor(155, 1, brightness);

	for (var i = 0; i <= gridSize; i++) {
		path.add(new Segment());
	}
}

It’s important to note that once a visual object is created, it will be rendered as part of the display hierarchy. There is no manual draw() or render() method to call. Paper.js has several structures for managing the display hierarchy including Project, Symbol, and Layer objects. In our example, when we want to iterate over each segment, we reference the child nodes (the path objects) of our project’s active layer.

var layer = project.activeLayer;

Now we can start doing the fun stuff – segment manipulation. Inside the onMouseMove event handler, we construct our reference loop. Here we use a nested for loop for simplicity. It should be noted that this is a very inefficient method for performance. Structures such as linked lists would be more appropriate if we were dealing with a large number of segments.

function onMouseMove(event) {
	for (var j = 0; j <= gridSize; j++) {
		var path = layer.children[j];

		for (var i = 0; i <= gridSize; i++) {
			var segment = path.segments[i];
		}
	}
}

Next, we move the segments’ point. For our demo, we apply some force vectors to each grid point. These vectors are generated by offsetting the segment coordinates (point) by a normalized vector (delta) pointing to our mouse position (event.point). Recall that paperscript has operator overloaders. This means that we can simply subtract the mouse point object from the grid point (event.point – point). Additionally, points have a built-in normalize method, which means we can write the entire vector operation in one line ((event.point – o).normalize()). The normals are amplified by force (force) to give us a smoother bump effect and then applied to the grid point. Finally, we use the built-in smooth() method on each path so we don’t have to worry about bezier handle adjustments.

function onMouseMove(event) {
	mousePoint = event.point;
	for (var j = 0; j <= gridSize; j++) {
		var path = layer.children[j];
		for (var i = 0; i <= gridSize; i++) {
			var segment = path.segments[i];

			var point = view.size * 1.2 * (new Point(i, j) / gridSize);
			var delta = (event.point - point).normalize();
			var force = ((view.size.height + view.size.width) / 2) * 0.2;
			segment.point = point + delta * [force / -2, force];
		}
		path.smooth();
	}
}

The last interaction we want to apply is a toggle for the debugger guidelines. These guidelines visually look like a selected path in Illustrator, complete with bezier handles (non-interactive at the moment). The fullySelected property on our project’s active layer will trigger guidelines an all its children – another example of the robust display hierarchy in Paper.js. Now we can see how the path handles change in real-time.

function onKeyUp(event) {
	layer.fullySelected = !layer.selected;
}

The finished demo can be viewed here. Make sure to view the source for better documentation and the complete flow. We took this demo a little bit farther and tied it to the Audio Data API (currently only available in FireFox 4+) and created an audio visualizer written purely in Paper.js.

We’re excited about the possibilities Paper.js creates – particularly when applied to data visualizations. Support and development of the library seems to be healthy, and the roadmap includes some very interesting future features. Paper.js + Node.js, anybody?

Ryan Challinor is a programmer at Harmonix Music Systems who is best known for his gestural menu system in Dance Central for Kinect. Immersing himself further into Kinect UI, he’s currently developing a synaesthetic experience using Kinect, Ableton Live, and Quartz Composer. I had the opportunity to chat with Ryan about his work on developing UI for the Kinect.

What was the timeline for the whole project?
Dance Central was in full production for about 14 months. The primary Kinect gestural menu work took place over the course of 6 months of prototyping.

When you got the Kinect, what was your reaction to the tech?
The first time you see your skeleton on the screen, it’s pretty jaw-dropping. After you get past the initial wow factor and discover the limitations, it can be pretty disheartening. But after a while of learning how to properly exploit the technology, I think it’s tough not to become enamored with it again.

When you started designing UI for the Kinect, was it hard to wrap your head around this concept? What was the biggest hurdle you faced?
The biggest hurdle was trying to overcome existing UI paradigms. The obvious approach is to emulate mouse input, with a cursor and click action.  A cursor is easy enough to create, but it becomes apparent that the skeletal input is not well suited to detecting an analog to a “click” action. Once we eventually ditched the cursor, it became possible for us to start thinking about new ways to control a UI, ways that are more suited to the data Kinect gives you.

Biggest challenge for designing a gestural interface?
Overcoming the mental block of how to select something without an equivalent of a mouse click or button press.

What was your first experiment with the Kinect?
Our first approach was a mouse-like interface, with giant buttons, and an arm extension.  It failed pretty miserably. I actually did a whole talk on the various prototypes that we progressed through in developing Dance Central’s UI. You’ll probably find it helpful, you can find the slides here and the video here (in their little video app, click “2011″, then scroll until you find my name)

With all the tests you did, what were the most common gestures that people gravitate to when using the Kinect?
In their first exposure to the technology, when people were completely unfamiliar with Kinect and before we were properly messaging how to use the UI, people tend to do a lot of pawing at the screen. There’s a lot of leaning forward and reaching out awkwardly, trying to figure out what the game is looking for. We even had some people trying to touch the TV screen.  Without proper guidance from the game, the first thing people tried to do is push their hand forward, and the second most frequent interaction tended to be swiping their hand around, which is what we’re actually looking for in Dance Central. This tended to make the gesture pretty discoverable for a large percentage of new players.

What was the best way you found to flush out ideas?
My approach was to lay out in my head “what do we need?”, and “what do we have?”. I would prototype how you could use the data coming from Kinect (“what we have”) to control the different elements of a UI (“what we need”). Then, when something failed, I’d ask “what doesn’t work?”, and use that to whittle down the “what we have” field until I hit upon how to properly use the data to accomplish our goals.

What do you think of the Kinect mod community? Have they shown anything that piqued your interest?
I am very interested in the stuff in the Kinect mod community, I’ve actually recently entered the fray with Synapse. I do have a bit of a chip on my shoulder about the reactions to the mod community, though, in that people see a cool mod and say “that’s so cool, why aren’t the games that cool?”. Kinect mods, for the most part, are just toys that are fun to play with for five minutes. It’s easy to make a toy that people have fun with for five minutes, it’s much more difficult to create a real compelling experience, and a product that works well out in the real world as opposed to a neat thing that works for one kid in his bedroom. The games will come, we just need to give developers time to figure out how to properly exploit the technology.

Any advice on what works well and what doesn’t work well when designing UI for the Kinect?
Don’t try to figured out how to cram Kinect into an existing UI paradigm, instead design a UI paradigm that’s from the ground-up intended to exploit the Kinect’s functionality.

Do you see Kinect being used in other types of applications?
Absolutely! One look at the Kinect mod community gives you just a taste of all of the different things you could do with Kinect. I’m excited all the time by the new application of Kinect that I discover on YouTube.

Keep up with Ryan’s work on Synapse on his blog.

Lately, we’ve been thinking about ways to improve online collaboration and group participation. One aspect of collaboration that we really want to explore is input – in particular, physically linking the input experience closer to output. Imagine manipulating a real-world replica of a virtual object and synchronising both, or in a more real example, being able to check-in to a location through Foursquare by physically touching it – an experience profoundly more unified than scrolling through a list of locations based on your GPS coordinates. For our initial exploration, we decided to build a simple, frictionless way of manipulating an object and broadcasting the input to multiple clients.

We wanted to do with something with 3D, and when you’re looking for something simple to start with, that only means one thing: cubes. A colour cube was particularly appealing because of its simple physical construction and interaction – rotation to reveal colours. For the sake of experimentation, we chose the cube to represent only the colour property of the virtual plus symbol. That’s how we came up with two separate modes for the demo.

Reading data off needed to be frictionless and easy. NFC provided the perfect bridge between object and device as it requires no handshaking, no pairing, and no user setup. With this method, we could simply tap the device with the cube to change colours. The bouncing balls you see in the demo were only meant to re-enforce the fact that you’re manipulating a complex environment with simple input.

The cube itself is internally tagged with six separate NFC stickers on each face. The stickers are read/writable Mifare Classic’s that can hold 1k of data. We wrote each tag with plain text JSON that contained a single colour value. In order to read the tags, we built an Android app for the Nexus S that listened for NFC tag discovery and established a socket connection to a node.js server on EC2 once the NDEF (the standard NFC data format) was parsed. It was straightforward to work with node-websocket-server (socket.io was problematic with Android) to connect to a standard HTML5 WebSocket in the client browser. Finally, we used the three.js to render the plus symbol and 3D environment in WebGL.

One interesting result from the experiment was that the passive mode (laying the device flat on the table and tapping the cube) felt like a much better interaction. The device became invisible and we were left solely manipulating the colour cube. The link between physical input and virtual output felt stronger. Handling the cube with both hands gave us illusion of full control – regardless of the fact that the device was doing all the work. It’s also worth noting that this type of approach is easier to implement on a large scale as the availability of NFC-enabled devices is generally low (though Google is trying hard to fix that).

It’s clear that simplifying and physically tailoring input to output with tech like NFC is a good way to enhance user experience and can lead to some unique interactions. We plan on taking this idea further by experimenting with other platforms and input devices.

BASIC EXAMPLE (green background with vignette)
UPDATE: SPINNING PLUS SYMBOLS EXAMPLE (by popular demand)

Here’s a simple snippet that can add some nice styling to your three.js projects. We’re big fans of camera vignetting, and the effect was very easy to spin into a WebGL shader.

Add a Plane to your scene with a MeshShaderMaterial. The only uniform necessary is your stage resolution (which you can change when resizing).

uniforms = {
	resolution: { type: "v2", value: new THREE.Vector2() }
	};

material = new THREE.MeshShaderMaterial( {
	uniforms: uniforms,
	vertexShader: document.getElementById( 'vignette-vs' ).textContent,
	fragmentShader: document.getElementById( 'vignette-fs' ).textContent
	} );

mesh = new THREE.Mesh( new THREE.Plane( 2, 2 ), material );
scene.addObject( mesh );

Add a basic vertex shader.

void main() {
  gl_Position = vec4( position, 1.0 );
}

The fragment shader does the actual darkness operation – that can be increased/decreased by changing a constant.

#ifdef GL_ES
precision highp float;
#endif

uniform vec2 resolution;

void main(void) {
  vec2 p = gl_FragCoord.xy / resolution.xy;

  float darkness = 1.45;
  vec2 textureCoords = p - 0.5;
  float vignette = 1.0 - (dot(textureCoords, textureCoords) * darkness);

  gl_FragColor= vec4(vignette, vignette, vignette, 1.0-vignette);
}

I knew coming to Portland that I’d love the city. It’s like a smaller and friendlier version of Toronto with superior seafood. Delicious sushi and oysters aside (do visit Jake’s Famous Crawfish if you get a chance), the real reason I made the trip was to attend NodeConf – the first conference dedicated entirely to Node.js, a platform we’ve recently been heavily exploring.

Node has been something of a sensation this last year. It’s often rare to leave a developer news site without seeing at least one mention of its name. In fact, try to remember the last time “node.js” wasn’t buried somewhere on the first few pages of Hacker News. I certainly cannot. Along with HTML5, it’s part of a shift in the development world that’s raising Javascript’s utility to new heights.

Of course comparing HTML5 and Node is grossly unfair. While HTML5’s biggest accomplishment is largely making plug-in functionality redundant and unnecessary – a massive feat in itself – Node is something of a paradigm shift. Some argue that it’s nothing new. EventMachine and Twisted, non-blocking I/O frameworks for Ruby and Python, have been around longer and do essentially the same thing. That’s true, but only focuses on the tech and not the tech’s implications. During his session Node @ Facebook, Robert Kieffer (an engineer at Facebook) cites the reason his company needs to increase adoption is that Node is breaking down the silos between front and back end developers. In our own experience, using the same scripting language on the client and server dramatically speeds up development time, and allows for modules to move easily between environments (like image manipulation libraries). That’s a significant advancement.

The core Node team was in full attendance at NodeConf, including creator and “belevalent dictator for life” Ryan Dahl who revealed the 0.5 and 0.6 roadmaps during his keynote. Windows compatibility, an expectantly contentious topic for the kind of crowd that node attracts, was either a win or fail depending on who you asked. Personally, I’m excited for it, particularly since it introduces a brand new abstraction layer in the form of libio to deal with both Windows I/O completion ports and Unix file descriptors, moving much of what lib.js does to C. More importantly, who can argue against greater adoption?

There were several sessions showcasing popular community projects built for Node, including Socket.io, node-canvas, dnode, haibu (which was made public during Charlie Robbins session) and jsdom. Of greater interest to me was a session by Marco Rogers of Yammer, who explained the process of writing your own C/C++ node add-ons – a valuable technique because, as he puts it, besides resulting in really fast modules, writing C/C++ just makes you feel like a badass. If you’re brave enough to play in C-land, I highly recommend reviewing his slide deck.

Node’s youth was also on full display. Many sessions touched on debugging and testing – a struggle for everyone using the platform. Matt Ranney showed a REPL demo for explicitly invoking functions and even adding new ones for testing during runtime. Felix Geisendörfer talked about his testing libraries: node-fake, node-microtest, and node-far. He also revealed that in Transloadit’s case, test code more than doubled production code in length. Several other presenters stressed the importance of clear and frequent logging. The advice was good, but it is clear that testing and debugged is still a huge pain point for developers, particularly with large clusters.

Tim Caswell, who delivered the last session of the day, did a good job framing the rapid growth of Node. He said that it enables developers to build networked servers easily. This is a fundamentally different approach than other languages/frameworks since it is a philosophy engrained in the project. I’m a believer in using the best technology available for a given task, and its clear to me that Node will be a very important tool for research teams like Labs. I will certainly be attending next years NodeConf – hungry appetite and all.

Reference: List of slide decks from NodeConf