Create a Hyperlapse.

Hyper-lapse photography – a technique combining time-lapse and sweeping camera movements typically focused on a point-of-interest – has been a growing trend on video sites. It’s not hard to find stunning examples on Vimeo. Creating them requires precision and many hours stitching together photos taken from carefully mapped locations. We aimed at making the process simpler by using Google Street View as an aid, but quickly discovered that it could be used as the source material. It worked so well, we decided to design a very usable UI around our engine and release Google Street View Hyperlapse.

The site settings are purposely low (like having a maximum of 60 frames per animation) for greater accessibility. However, all the source code is available on Github (including examples and documentation) so developers can play with higher frame rates, better image quality, and more complicated camera movements.

The idea for this project came from one of our motion designers, Jonas, as part of his Labs experiment. He wanted to explore a tool that could help him create hyper-lapse videos with the assistance of available data sets and emerging technology. We built the tool – he built a video with it. The results are pretty stunning.

“Simplicity is the ultimate sophistication.”
― Leonardo da Vinci

Let’s take a moment to think about connected appliances (or “Smart” appliances if you’re a marketer) – those expensive, futuristic home units with touchscreen LCD’s and an overabundance of software features. At least, that’s what we saw plenty of at this years CES. Should a connected fridge really have Evernote integration? Should it track expiry dates, or tell you what foods are in season? Should it tweet, sync with Google Apps, or play Star Wars Angry Birds in HD? Increasingly, those are the kind of innovations we’ve seen demoed by the large appliance manufacturers – and it’s exactly the wrong approach they should be taking when designing these products.

A refrigerator has a simple purpose – prolonged food storage. The data associated with storage is inventory. So naturally the core function of an internet-connected fridge is to provide us with a tally of its contents. That’s it. Luckily, we already have a good model for taking a quick inventory – opening the door and looking inside. Our eyes and memory do the rest. A brief glimpse immediately tells us if we forgot to buy eggs last week, how many bags of milk are left (a Canadian thing), and if the fruit looks spoiled.

The most basic connected fridge that fulfills its purpose is one that takes a picture of its contents and pushes them to your devices. It provides you the opportunity to take a glimpse inside the unit wherever you are (probably the grocery store), just like you were at home. This approach is so brain-dead simple and easy to implement with some basic components, and yet we haven’t seen it from manufacturers. Instead, we can tweet.

We hacked together our own connected fridge that snaps shots every time the door opens and uploads them to the cloud. It took barely any time to construct (since it was patched together with masking tape and rubber bands) and used some very basic components. In actuality, the site to host the images took longer. The result is very rudimentary and ugly, but it works and communicates exactly what it should – inventory. Take a peek inside.

Build your own connected fridge.

Necessary Parts:
2 HD webcams (we used Logitech c615)
1 Raspberry Pi
1 Reed Switch
1 Magnet
Bunch of wires (USB, jumper, ethernet)

Wire a reed switch (ground, vcc, and signal) to the Raspberry Pi. We used GPIO4 for signal. Attach the switch to the door hinge at the top of the fridge. Glue a magnet to the top of the door such that when the door is at an open position, it’s directly under the reed switch. This will send a signal to the Pi.

Connect the cameras to the Pi and embed them in the fridge. We were lucky enough to have an opposing wall so the cameras could be mounted outside, but that’s an unusual setup. Embedding inside would be a lot more time consuming and would involve drilling through door for wire passthrough, plus resealing so cold air does not escape.

Next, you need to program the Raspberry Pi to watch for the signal from the reed switch, take photos, and send them up to the cloud. There are many ways to do this, but ours was with fswebcam, and some basic node.js.

Install fswebcam:

$ sudo apt-get update
$ sudo apt-get install fswebcam

Install node.js:

$ wget http://nodejs.org/dist/v0.8.19/node-v0.8.19.tar.gz
$ tar -zxf node-v0.8.19.tar.gz
$ cd node-v0.8.19
$ ./configure
$ make
$ sudo make install

Install the GPIO package in your project directory:

$ npm install gpio

Write a simple node program that takes photos from the cameras when GPIO reads the value 1 from pin 4. After the photos are taken you can upload them wherever you want. We push ours to S3 using knox.

var gpio = require("gpio");

function takeShot(device) {
	var filename = device + '.jpg';

	// Spawn the webcam child process.
 	var spawn = require('child_process').spawn,
		fswebcam = spawn('fswebcam', [
			'--device', '/dev/' + device, 
			'--no-banner', 
			'--resolution', '800x480', 
			'--rotate', '90',
			'--jpeg', '95', 
			'--save', shots_path + filename
		]);

	// Log fswebcam output.
	fswebcam.stdout.on('data', function (data) {
		console.log('stdout: ' + data);
	});

	// Log any errors.
	fswebcam.stderr.on('data', function (data) {
		console.log('stderr: ' + data);
	});

	// continue processing after it has taken photos
	fswebcam.on('exit', function (code) {
		// Do stuff, like saving/hosting your images.
		// We used knox (https://github.com/LearnBoost/knox)
		// to push the photos to S3.
	});
}

var gpio4 = gpio.export(4, {
   direction: "in",
   ready: function() {

		gpio4.on("change", function(val) {
			if(val == 1) {
				takeShot('video0');
				takeShot('video1');
			}
		});

	}
});

That’s it! Our office fridge is now open for public scrutiny: fridge.teehanlax.com.

This is a simple project, but it opens some great questions for discussion – namely, how should we be designing our appliances? What value can we add to them? When is adding features actually a detriment? Let’s talk.

Lines of light ranged in the nonspace of the mind, clusters and constellations of data. Like city lights, receding.
- William Gibson, Neuromancer

It’s no secret that – in some eternal quest to palliate my nostalgia – I’ve developed a mild obsession with text-mode art. Years ago, I learned all the relavent algorithms, techniques, and code pages in order to best simulate text-mode in modern browsers – an effort that resulted in (long since abandoned) projects like ASCIImeo and Flashterm. Recent explorations in bridging the digital-physical divide (plus a sense of withdrawal) have caused me to revisit image-to-text conversion with a quick experiment.

For many of us that have grown up with computers, text-mode art represents something deeper than nostalgia. It is an artform manifested from technological constraints, inspired by the same hacker ethos that build the early machines used to produce and view it. Fundamentally, it is both an expression and prisoner of the system it inhabits. This latest experiment attempts to free ASCII art from the confines of the screen and enable it to exist in physical space – with light and paint.

I wanted to approach image-to-text conversion a bit differently this time around, focusing primarily on speed. In order to achieve this, all the real-time conversion work was moved to the GPU. The resulting program (available as an openFrameworks add-on) can scale to huge dimensions with little strain.

A nice side-effect of writing the new library in OpenGL was easy portability to WebGL, enabling a unique twist on exploring the physical world:

Ascii Streetview

Advances in physical computing are changing the way we interact with our environment. These changes can influence how we go about doing our day-to-day activities, such as food organization. To explore the connection between the physical world and the digital world, we focused on a single universal question to find a way to merge these two worlds more intuitively into everyday life.

We’ve all been there. You pour a bowl of cereal, or a cup of coffee. Go to the fridge. No milk. Maybe you’re already at the grocery store in the dairy aisle and wonder, “How much milk do I have left at home?” What if you could take a peek into your fridge anytime, anywhere?

To monitor the amount of the remaining milk, we attached a weight sensor to the bottom of a milk jug to send the weight values to your phone. When the milk reaches dangerously low levels, you get an alert on your phone reminding you to get milk while using your current location to map nearby grocery stores.

But sometimes you need milk for more than just a bowl of cereal. You can also use the app to check the remaining levels of milk before you receive the alert.

Done. Got milk.

Why did we use a milk jug? For our non-Canadian friends that may be unfamiliar with the phenomenon known as milk bags, a 4-litre package comes with three milk bags that are usually stored in either a jug or a pitcher.

With this in mind, we also prototyped a companion weight sensor to monitor a drawer in the fridge used to hold the additional milk bags to get the combined weight of the remaining milk.

It’s easy to imagine all of the different ways to implement this idea. The natural inclination would be to track multiple items, or incorporate other elements like a recipe database. There are already “smart fridges” on the market that have various food management features. The downside is the features are built into the fridge itself, which makes it expensive and sometimes complicated to set up. In addition, while the smart fridges do have some features that interact with the mobile phone, there isn’t much communication with the person and their environment.

By using phone notifications and maps in the Milk? app, it goes from being just an organizational tool to interacting with the person by “reminding” you to get more milk and using your physical location to show you where to get it.

Although this experiment revolved around milk, the idea has broader implications that extend beyond the milk itself. How can this idea impact the way we organize our lives and the way we interact with objects using the digital capabilities available to us? The possibilities are as endless as our imagination.

Recently, I gave a talk about the development of a typographic portrait generation engine we built for Bell Social Portrait. It’s a story that explores some of the challenges of building a complex web application meant to be experienced on many different screen sizes and platforms. The project taught us valuable lessons that will surely influence future work.

Bell Social Portrait is a site that pulls a users social feeds and programatically generates artwork from an associated profile or user-submitted image. Traditional typographic portraits are normally constructed in graphic design software like Illustrator or Photoshop. We knew it’d be a challenge to simulate the process programatically – never mind deploying it to as many browsers and devices as possible (a self-imposed requirement).

Early research and prototyping was all done using the Canvas tag as a drawing surface – probably the most ubiquitous and well supported API’s in HTML5. Our intent was to generate as much reusable code as we could for the finished implementation. This meant that all image segmentation routines were written in Javascript – a task made easy by some groundwork I’d laid a few months earlier.

ASCII art (specifically image to text conversion) was a major source of inspiration for this project, and its basic implementation served as a good starting point. Typical ASCII art uses a quadrant (one region evenly split into four) based color weighted lookup to match character glyphs. The process is actually quite simple. Take, for example, the letter “L” – it could be mapped to a region with bright color (lots of white) in the top-left/bottom-right/bottom-left quadrants. If you’re interested in the full process (and more than just the most basic implementation), I highly recommend reading The Art and Science of Text Mode Conversion by Jari Komppa.

We used this same quadrant splitting practice in order to segment images into a cell grid of varying sizes. This type of structure is commonly known as a quadtree, and involves recursively splitting an image with one simple condition: stop splitting if the region contains only one color. You can see the results illustrated below at different iteration (number of times to recursively split) values.

Next, we merged all adjacent cells of the same dimensions. This produced a useful grid for mapping words, but we weren’t pleased with them aesthetically. Cells appeared too organized and symmetrically positioned. We wanted an element of chaos, and this meant a new approach was required.

We began investigating various box fitting strategies, and quickly began building our own algorithm. The approach was simple:

1. Pick a random point from an array of active pixels (not white).
2. Move that point horizontally left or right until a null pixel (edge of the image, white pixel, or another cell) is hit.
3. Move vertically up or down until the same condition is true.
4. Expand a rectangle, with a width/height ratio chosen randomly from a predetermined set, until any of its pixels hits a null pixel. Mark that rectangle as a cell.

Aesthetically, the resulting grids had the right amount of randomness and produced visually appealing portraits. They looked great, but a couple problems arose – average computation time of 30 seconds in Chrome, and huge performance variability across difference platforms. iOS (pre 5.1) performed particularly poorly, with the worst cases fully locking up devices. We needed to find a workaround, and that meant moving image segmentation to the cloud.

For a number of unrelated reasons which included tight application and web serving integration, an easy interface to mongodb, and the need for an event based architecture, we had already decided to implement the server portion of Social Portrait in Node.js. And since Node is basically “Javascript for networking”, the task of porting the image segmentation routines written for the browser was relatively easy. Thanks to Learnboost’s node-canvas package, we were able to simply copy and paste most of our code.

Recreating the portrait generation process on the server meant we could free the browser from any application level computation and have it narrowly focus on UI. This was a huge deal. The browser experience became light and responsive to different devices and platforms. Also, since we prototyped portrait generation in the browser, we could comfortably pass off a small JSON UI properties object along with the unaltered original image to the server. Bandwidth usage became very light – an important factor when accounting for mobile devices. Finally, we knew exactly what hardware the code was running on, which established a uniform performance metric. The performance variability problem had been solved.

Our decision to move image segmentation to the server also meant we had some new powerful tools at our disposal. One of these was imagemagick – a very mature, fast, and well supported graphics software suite that we could access through a child process. It allowed us to preprocess our input images with very little overhead. One technique we leveraged with imagemagick was down-up-sizing images before grid generation inorder to produce a blocky canvas. This resulted in an easier image to fit rectangles on, which reduced noise and improved segmentation speed.

The real fix to our speed problem came from having the ability to write our own native programs in C++ – specifically the ability to use data structures that do not exist in Javascript. When iterating over an array of active pixels (not white), we had no way of skipping over elements that were already assigned to cells. We tried many tricks in Javascript including overly complicated dynamic lookup tables, and splicing arrays. Nothing worked well. Then we found dynamic_bitset from Boost C++ – a data structure that only contains bits (on/off) and more importantly does not iterate over “off” elements. This means that as cells get progressively marked unavailable, iterating over the active pixels array gets faster. We rewrote our grid generation code as a C++ program and wrapped it in a Node child process. The result was a 10x speed increase – an average of 3 seconds!

It may seem like this story is a glowing endorsement of Node.js, but that’s not necessarily the case. Node was a very valuable tool in this project for a number of reasons – the ability to prototype directly on our drawing canvas and seamlessly port code over to the server, the access to a native environment that solved huge performance problems, and the access to powerful tools like imagemagick for preprocessing. But had we done the project over again, I’m not sure we would have implemented our entire server environment in Node. Static web serving is better handled in tried-and-tested environments like Apache or nginx, and worker management for multiple CPU cores can be daunting.

Nonetheless, the most important lesson we learned is that cloud technology and the new breed of server-side environments are necessary tools to utilize in modern-day web applications. With the variability of devices and platforms users move between, we can no longer solely rely on client-side environments to deliver rich experiences. Frame your UI’s in the browser – let the cloud do the work.

Yesterday we (Labs) ran a half-day hardware hacking workshop internally for all of the Teehan+Lax employees (everyone from designers and developers to the project managers and the “HR” people). It was meant as a light introduction to physical prototyping. Here are some of the creative, fantastic (and horrifying) results. We were very excited to see everyone’s results at the end of the day. The great thing was that everyone had successfully created something by the end of the workshop from scratch.

“The perception of system output must be greater than the perceived user input, and within that range there is a sweet spot where [user experience] is delightful and magical.”
- August de los Reyes

I can still recall the first time I saw an Augmented Reality demo. There was a sense of wonderment from the illusion of 3D models living within the video feed. Of course, the real magic was the fact that the application was not only viewing its surrounding environment, but also understanding it. AR has proven to be an incredible tool for enhancing perception of the real world. Despite this, I’ve always felt that the technology was somewhat limited in its application. It is typically implemented as output in the form of visual overlays or filters. But could it also be used for user input? We decided to explore that question by pairing the principles of AR (like real-time marker detection and tracking) with a natural user interface (specifically, touch on a mobile phone) to create an entirely new interactive experience.

The outcome of this examination is something we’re calling Touch Vision Interface – a tool that enables touch interaction on many different connected surfaces through a mobile phones’ camera view. The translation of touch input coordinates to the captured video feed creates the illusion of being able to directly manipulate a distant surface. As a result, the interaction feels natural and almost invisible.

Future applications of this technology are more than compelling. The barrier of cross device communication is lessened, whether in the living room or in large open spaces. Brands could crowd-source easier with billboard polls. Group participation on large installations could feel more natural. The possibilities become even more exciting when considering the most compelling aspect of the tool – the ability to interact with multiple surfaces without interruption. No need to switch devices through a secondary UI – simply touch your target. You could imagine a wall of digital billboards that users seamlessly paint across with a single gesture.

Other applications could enhance the collaborative creative process. Imagine a music creation experience where each screen becomes an instrument. Pitch and tone could be changed by moving your finger across the instruments surface allowing for natural gesture controls. Rather than being paired with an instrument, a band member simply plays what they’re “looking” at.

There are flaws with this technology, surface discovery and pairing being the most obvious. But along with the problems, it’s easy to see how this idea could be extended – moving past simple planar interaction and deeper into the world of real-world object manipulation. We’re excited to see how far we can take it.

“It’s clear that simplifying and physically tailoring input to output with tech like NFC is a good way to enhance user experience and can lead to some unique interactions. We plan on taking this idea further by experimenting with other platforms and input devices.”
—Physical and Frictionless Input. 7 June 2011.

It should be about the experience. Not the technology.

Continuing on with physical interaction we’ve been experimenting with Arduinos, RFID/NFC, social network APIs, and dreaming up various other end uses that we will discuss in a minute and end off with a really quick (but verbose) demo. When dealing with technology for novel interaction we really want to make sure that the interaction is put before the technology. A tech demo for tech sake isn’t always the best approach. Inventing solutions to problems that don’t exist doesn’t make much sense. Products (specifically tools — and digital or physical) are meant to service something and then get out of the way. If we take this approach to using RFID/NFC then whatever that tool ends up being should show up, do the job, then completely disappear so you can get on with living life. Without you needing to do anything. You don’t give the plumber step by step instructions do you? You just call and tell them to come fix something.

Imagine being at a Jay’s game and being able to swipe your ticket on your arm rest to +1 or like a big swing by José Bautista (or rather to +1 his childish temper tantrum over the weekend). Or maybe this winter we’ll finally be able to tweet that we witnessed the first Leafs Stanley Cup win since 1966/67 just by having your ticket in your pocket and your chair knowing it’s you. Because, obviously, there’s no way any of us are gonna stop cheering to get our phones out of our pockets to tweet about it.

Let’s face it though, that won’t happen anytime soon so maybe you could check in on Foursquare at a conference by swiping your conference pass at the entry gate. That could be extended into checking in anywhere. Check in at your work, the various rooms in your house, or check in at your toaster or fridge or at your computer for an all night Minecraft marathon.

We can even microchip ourselves — the same as microchipping your pet — and use that to send tweets for various appliances and pieces of furniture we end up using. Or more practically that microchip could be used to engage a locking mechanism on the front door or store personal identity/health info. It would be pretty rad if you could keep your health records stored in the fleshy bit between your thumb and index finger. The use cases for RFID/NFC are quite varied from really practical to kinda shady, whimsical, and fun.

The idea of this kind of passive interaction; and passive data logging, tracking, and publishing is both frightening and exciting. It could go down the scary big brother road where every action we take is recorded and posted somewhere publicly and used for ad streams and evil. Remember when your significant other could see the lovely purchases you made in secret for them on Amazon showing up on their Facebook wall, or when robbers used Foursquare to track good targets because they checked in somewhere that wasn’t their own home? Yeah that wasn’t cool.

I don’t see that happening…much. I see the nice green pastures of passive interaction for the betterment of humanity. There have already been many projects from around the world that prove this true many times over. More and more of these projects are moving away from tracking offline activities on social networking sites and are moving towards (actually a bit backwards — think remote controls and animal microtagging) friendly tangible interfaces and controllers for a variety of different products.

Before we get into the demo proper, a quick technical breakdown. An RFID is hooked up to an Arduino (or any microcontroller, we picked Arduino). That Arduino is in turn hooked up to a wifi or ethernet shield (a custom breadboard that adds additional functionality to the Arduino, in this case, the whole Internet) which sends information (a string containing an RFID Tag serial number, or in the case it’s rewritable, whatever we can squeeze into 1K of data) to a server (actually it sends a Serial communication across TCP/IP). The server is running a Python script that is listening for data coming from the Arduino and uses that serial number which is synced to a person’s OAuth access tokens to validate them and post some action to their account on a social network.

This setup can very quickly bring the offline world into the online world.

The part where we get to make something.

So, we’ve put together this simple system together that can send messages to a twitter account whenever you swipe a certain RFID tag over the Arduino. For the sake of keeping the demo as cost efficient as possible and simple, we won’t be doing any TCP stuff. Save that for next time. Plus, adding internet to your Arduino starts at about $50. We’ll just send serial from the Arduino to the laptop via the USB cable. Just pretend it’s an ethernet cable.

The prequel to the steps is to acquire a bunch of Arduino parts. You can get all this stuff from Sparkfun or Makershed and it’ll take maybe a week to ship.

1 x Arduino
1 x RFID Reader (breakout board or usb reader, we used the usb version)
1 x ID-12 or ID-20 RFID Reader
1 x RFID 125kHz Tag

Other things you’ll need are a bundle of Breadboard Jumper Wires and a Breadboard. You’ll probably also need some 6-pin and 8-pin headers (for the breakout board). If you need the headers you’ll need a soldering iron and solder. If you’ve never soldered before¹ you may also want a Solder Sucker and a Solder Wick for clean ups and fix ups. If you drink a lot of coffee I would also recommend a Helping Hand Vice.

Now that that is out of the way let’s build something. The first step is to assemble the Arduino. We’re using an Arduino Uno, Sparkfun’s RFID USB Reader board, and an ID-20² RFID Reader. If you’re using an RFID breakboard the PIN mapping is a bit different but will be explained as well.

For the USB Board we soldered a 6-pin header and stuck it on a solderless breadboard. From the Arduino attach digital pin 6 (marked ~6 on the Uno) to TX-0 on the USB Reader, 5V to VCC, and GND to GND and GNDT.

—Made with Fritzing. Showing wiring for RFID USB Reader.

With a breakout board you attach ~6 to D0 (pin 9), 5V to RES (pin 2) and 5V (pin 11), and GND to GNDT (pin 1) and +/- (pin 7).

—Made with Fritzing. Showing wiring for RFID Breakout Board.

The wiring is really easy if you plug the 5V and the GND from the Arduino into the + and – rails down the edge of your breadboard. You can then plug the various powers and grounds into the RFID reader from there. That completes setting up the hardware. It’s not too complex a set up.

There are two different pieces of code. One is Arduino³ and the other is some very simple Python⁴. We’ll set up the Arduino software first so that we can send a message out over serial.

For Arduino we also need the third party library called NewSoftSerial.

First we’re going to define a bunch of things that we’ll need later on and import the NewSoftSerial library.

#include <NewSoftSerial.h>

NewSoftSerial rfidSerial(6, 7); // (RX - pin 7 on usb rfid board, TX - pin 6 on usb rfid board)
String rfidTag, rfidTagClean;

int i = -1;

// Henry V Tweets
char* tweetStrings[] = {
  "O, pardon! since a crooked figure may Attest in little place a million;",
  "And let us, ciphers to this great accompt, on your imaginary forces work.",
  "Turning the accomplishment of many years Into an hour-glass:",
  "for the which supply, Admit me Chorus to this history;",
  "Who prologue-like your humble patience pray, Gently to hear, kindly to judge, our play."
};

The next thing we need to do is set up the communication over serial. void setup() is a special function that runs once when the application starts.

void setup() {
  Serial.begin(9600);
  Serial.flush();
  rfidSerial.begin(9600);
}

So all we want to do is open some ports and make sure there that anything that is currently there is flushed out of the system.

Next we need to build the rest of the application. Another special function in Arduino is void loop(), this function constantly runs, looping back to the start of the code when it reaches the end.

void loop() {
  if (rfidSerial.available()) {
    
    char incoming = (char)rfidSerial.read(); 
    // Packet structure, see ID-12 datasheet.
    // STX A B C D E F G H I J Y Z CR LF ETX
    //   2 - - - - - - - - - - - - 13 10 3
    
    // If the last character of the packet is 3, move on.
    if(incoming == 3) {
      
      // Skip the first character and take the next 12
      rfidTagClean = rfidTag.substring(1, 13);
      rfidTag = "";         
      
      // Shakespeare's Henry V Prologue/Act 1. You'll need to change the tag number to one of your own tags. See the final else statement on how that happens.
      if(rfidTagClean == "4500B89EC1A2") {
        i++;
        Serial.println(vStrings[i]);
        delay(500);
        // Reset counter so we can start reusing the already used tweets. 0 - 4.
        if (i == 4) {
          i = -1;
        }  
      }
      
    } else {
      
      rfidTag = String (rfidTag + incoming);
      Serial.print(incoming); // Prints the the 12 digit number we want to use for the rfidTagClean if condition.
      
    }
 
  }
}

Inside the loop the first thing we want to do is make sure that the code only runs if we can detect that the RFID reader is available. From there we say if the scanned RFID Tag matches this 12 digit number print one of the tweets, and every time this tag is read move to the next tweet. On the 5th tweet go back to the start of the list. If the RFID tag does not match print the tag number to the serial monitor so that we can track it. Either add it to a list of approved tags or not.

That’s it for the Arduino. You can upload that sketch to the board and run it. Open the serial monitor, swipe your RFID tag and see what happens. Don’t worry, we’re not tweeting yet, so have at it.

The final step is to write a super small Python script that monitors the serial port that Arduino is communicating on and if it gets a message send it to Twitter. Unfortunately, before we do that we have to register an application with Twitter so we can get OAuth out of the way. That is probably the pain point of the set up. OAuth is also why I’m not showing you how to like on Facebook, or check in on Foursquare. Dealing with OAuth and social network APIs is full of no documentation or worse… bad documentation. Okay, go log in to your Twitter account, hit up the developer section and register your applicaiton. Make sure to give your app Read & Write privledges and finally authenticate your account with your app. This way not only will you have the Consumer Token and Consumer Secret, you’ll also have the Access Token and Access Secret which is a unique pair for every account registered.

Okay, one more caveat. There are 3 additional python libraries. Tweepy, pySerial, and Time. I’m also using the Python 2.7 build not Python 3.x.

import tweepy
import serial
import time

# make sure this is your arduino. Open Arduino > Tools > Board
arduino = serial.Serial('/dev/tty.usbmodemfd121', 9600) 
status = arduino.readline()

auth = tweepy.OAuthHandler("XXXXXXXXXXXXXXXXXXXXXX", "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")
auth.set_access_token("XXXXXXX-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")

api = tweepy.API(auth)

while 1:  						# Infinite Loop
	status = arduino.readline() 	# Wait for new line to come across Serial
	api.update_status(status) 		# Post message to Twitter with Tweepy
	time.sleep(20) 				# Wait 20 seconds
	del status					# Clear status so we can start over with a clean slate

Okay, what’s going on here. We import the libraries we need to handle everything we’re about to do. Next we set up the serial so Python is listening to the Arduino so that it can send each line as a new tweet. The println function in the Arduino code sends each string with a new line (\n) at the end. After that we set up the Twitter authentication and tell Twitter’s API to use those specific credentials which will post to that account. The while function continously monitors serial data and each new line that comes across it posts to Twitter.

Upload the Arduino sketch to the board, leave the Arduino plugged into your computer via USB and run the python sketch in Terminal:

cd /path/to/python/sketch/
python twitter.py

[download id="14"]

And you’re set. Swipe the card and impress your followers with your new powers.

Taking it a few steps further we could move this back into the ethernet/wifi world. But something that might be more interesting is to attach the unique access tokens/secrets to an RFID card so that if Card 1 is swiped use my unique credentials and post to my account OR if Card 2 is swiped use your unique credentials to post to your account. This way you could have one piece of hardware posting for everyone that has a valid RFID card. Which becomes more compelling. More soon.

–

1. A few notes on electronics. If you’re buying a soldering iron for the purposes of hobby electronics I don’t recommend the ‘cold heat’ style soldering tools. They work by producing an electrical current that could possibly short out your circuits. Stick to the regular soldering iron that just heats up the tip. Just be careful to not burn yourself. And possibly get adult supervision. ↑
2. The ID-2, ID-12, and ID-20 all have the same PIN mappings. However, the ID-2 does not have an internal antenna so you’ll have to build one. Which is painfully complex. The ID-12 and ID-20 both have pins exposed for adding an external antenna but this will cause all kinds of reception problems. ↑
3. Arduino is both a hardware platform and a programming language. Actually the language is based on Processing which is built on Java. Arduino is built on C/C++ but maintains most of the syntax styling of Processing. So bits are written like Java and other bits are written like C/C++. Don’t worry it’s not as confusing as that makes it sound. ↑
4. Python is like Ruby but better. ↑

In the last years, with the advent of new web enabled devices — smartphones, tablets, connected TVs and more — our digital landscape has been rapidly expanding beyond the borders of traditional desktop/laptop screens. As mobile devices are increasingly becoming the primary way we access online content, responsive design has emerged as a popular design strategy.

Designing for all devices: apps and mobile websites

Smartphones and tablets have complicated the task of digital strategy and design by adding new resolutions and technical constraints. With plenty of different mobile devices around, dealing with this complexity—in particular, reconsidering the traditional 960px fixed width viewport—has been approached in two ways:


1. Create complimentary (native) mobile apps (a trend fostered by manufacturers with their ad hoc SDKs and app stores);

2. Create multiple mobile versions of websites and try to cover as many variations as possible.

 
Both of these options aim to create experiences that are perfectly adapted to the context of the device they appear on. Although they might look ideal from a coverage and support perspective, the problems with both these options are that:

1. They require an additional financial, design and maintenance effort that may be overwhelming for clients.

2. They may well prove to be a Pandora’s box in the long run as an endless number of new devices will appear with their own specificities.

Designing for any device: responsive web design

In parallel, another strategy emerged last year in the shape of a seminal article written by Ethan Marcotte, entitled Responsive Web Design. To avoid these drawbacks and based on the assumption that a web experience should (and could) be the same no matter the device (“one web”), Marcotte defended the idea of a single fluid website rendering different layouts per device.
His point was that a website designed following responsive principles would be more “future proof” as its layout would adapt to any device, content being smoothly rearranged and transformed to fit the device in question:

“Rather than tailoring disconnected designs to each of an ever-increasing number of web devices, we can treat them as facets of the same experience. We can design for an optimal viewing experience, but embed standards-based technologies into our designs to make them not only more flexible, but more adaptive to the media that renders them. In short, we need to practice responsive web design.”

To achieve this new ideal of “responsive design”, Marcotte defined three integrated features:


Many websites (including this one) launched in the last few months embraced this new approach and jumped on the responsive bandwagon with more or less success. Responsive web design is certainly a step forward towards a better and more flexible web, but it should not be accepted as the only way to go.

Our experiments with responsive web design

Recently, we experimented with responsive design for the new T+L website and thought it could offer an appropriate solution to supporting multiple devices.

We decided to prolong the experimentation with an internal challenge: T+L designers and developers were asked to come up with a series of layouts that would respond to changes in screen size. Ultimately we analyzed the outcome of this challenge and spent time breaking down responsive frameworks to figure out what were the pros and cons of using the responsive approach.

We identified two methods to initiate the design process:


• Either as a “progressive enhancement”, with mobile layout first and desktop layout at last.

• Or as a “graceful degradation”, with the full desktop website as a basis to design the other layouts.

So, what’s good in this?

Responsive web design is a valid approach as it requires a single effort to support a multitude of devices. Content leads the way and can be increased, reduced, shrunk or removed on demand and depending on context.

Then where’s the issue?

Actually there are a few…
Responsive design is based on the assumption that the substantive features of online content should not vary across devices. When we consider how much the user’s context can depend on where and how they are accessing online content, we see that this assumption may not be true all the time. If we use the case of a contact page as an example, it makes sense to display only a map or driving directions for the mobile context (since users are likely to be on the road), while displaying office photography and detailed contact information for the desktop context (since users are stationary and can afford to browse longer). A responsive design approach is too undifferentiated for these situations.

Creating a responsive layout is not easy and requires a lot of preparation – it must accommodate a series of grids on both portrait and landscape modes, as opposed to a fixed width layout with a single vertical view that adheres to a single grid system. This effort could be substantial, and might be a waste if the intended targets are explicitly defined (like in our contact page example). Ultimately, it’s important to ask: should my design be responsive or adaptive?

Another consideration we identified with responsive designs was their difficulty to treat type contextually. We observed that they were often too deterministic in trying to replicate typographic elements on small screens that were of minor importance for mobile usage. Conversely, we noticed some responsive designs were so fluid that they became too loose (excessively wide text columns, margins, etc.).

There are many solutions you can consider when trying to tackle varying screen sizes, but picking the right one depends on your projects’ goals and device context. Trying to find a golden solution might not be the right approach. Responsive design is just one of many solutions that can help with that problem, but it’s not the miracle solution that many are hoping for.

UPDATE 07/20/11: Updated the demo to take better advantage of built-in Paper.js features.
UPDATE 07/27/11: Compacted the demo even further utilizing the active layer object.

Recently, we’ve been exploring various vector graphics libraries in an order to craft some more compelling data visualizations. I’ll admit, we weren’t too enthusiastic about the prospect of manipulating SVG or learning some strange custom syntax. Fortunately for us, Paper.js was released a few weeks back.

Paper.js is a brand new open source vector graphics scripting framework with two important distinctions. First, it’s actually nothing new. Paper.js is entirely based off Scriptographer, the ten year old scripting framework for Adobe Illustrator, complete with its own mature and battle-tested display hierarchy and Object Model. This is welcome news for any developer familiar with Illustrator or – more importantly – Flash. Second, it uses the Canvas element as opposed to SVG. Besides the speed boost this grants, it also means we can take advantage of the rasterization and bitmap manipulation features of Canvas.

We’ve put together a simple example that showcases some of the structure and features of Paper.js including debugging guidelines, smoothing, and path manipulation. Additionally, it demonstrates simple interactions and event handling.

Click to view our tutorial example.

Paperscript (the Paper.js extension of Javascript) is an important differentiation from other vector graphic libraries in that it provides a sand-boxed scope for Paper.js scripts. It treats objects such as Point and Size like native data structures with operator overloaders in full effect. For example, you can add two points by simply performing: point1 + point2. Contrast this to a similar framework like Raphael, which requires the cumbersome manipulation of SVG strings, and you can start seeing the benefits of paperscript.

The first step in our demo is to declare our paperscript and tie a canvas instance to its scope. We’ll also declare some built-in handlers for resize, keyboard, and mouse movement events. The handlers are automatically bound once declared – no need to attach them to an element.

<canvas id="canvas" resize keepalive="true"></canvas>
<script type="text/paperscript" canvas="canvas">
	function onResize() {
		//
	}
	
	function onMouseMove(event) {
		//
	}

	function onKeyUp(e) {
		//
	}
</script> 

Let’s construct a simple fake terrain effect with depth being simulated by brightness. In this case we’re building a series of paths stacked vertically with an equal number of segments across each path resulting in a square grid (gridSize x gridSize). First we need to declare our grid dimensions.

var gridSize = 20;

Path is the core visual building block in Paper.js that is used to construct other vector graphics objects in its display hierarchy. We’ll be using them directly to build our grid. Every row in our grid represents a new Path object with columns being represented by that object’s segments. We’ll instantiate each path on every loop iteration. Next we’ll use the Path method add() to add the segment points.

for (var j = 0; j <= gridSize; j++) {
	var path = new Path();
	
	for (var i = 0; i <= gridSize; i++) {  
		path.add(new Segment());
	}
}

Let’s change some of the rendering properties of each Path so we can visually differentiate them. As we loop over each row, we assign a stroke color to the path which becomes brighter as increments get higher. Paper.js has built in color objects for handling RGB, HSB, HSL, and Grayscale. We’ll use HSLColor to display a range of greens. The stroke width can be set globally and inherited by every newly instantiated path. Paper.js has a base Project object which acts as our “document”. We set the stroke width of our projects’ current style to a value proportional to the canvas dimensions.

project.currentStyle.strokeWidth = (view.size.height + view.size.width) / 0.8 / gridSize;

for (var j = 0; j <= gridSize; j++) {
	var path = new Path();
	var brightness = j / gridSize * 0.608;
	path.strokeColor = new HSLColor(155, 1, brightness);
	
	for (var i = 0; i <= gridSize; i++) {  
		path.add(new Segment());
	}
}

It’s important to note that once a visual object is created, it will be rendered as part of the display hierarchy. There is no manual draw() or render() method to call. Paper.js has several structures for managing the display hierarchy including Project, Symbol, and Layer objects. In our example, when we want to iterate over each segment, we reference the child nodes (the path objects) of our project’s active layer.

var layer = project.activeLayer;

Now we can start doing the fun stuff – segment manipulation. Inside the onMouseMove event handler, we construct our reference loop. Here we use a nested for loop for simplicity. It should be noted that this is a very inefficient method for performance. Structures such as linked lists would be more appropriate if we were dealing with a large number of segments.

function onMouseMove(event) {
	for (var j = 0; j <= gridSize; j++) {
		var path = layer.children[j];

		for (var i = 0; i <= gridSize; i++) {
			var segment = path.segments[i];
		}
	}
}

Next, we move the segments’ point. For our demo, we apply some force vectors to each grid point. These vectors are generated by offsetting the segment coordinates (point) by a normalized vector (delta) pointing to our mouse position (event.point). Recall that paperscript has operator overloaders. This means that we can simply subtract the mouse point object from the grid point (event.point – point). Additionally, points have a built-in normalize method, which means we can write the entire vector operation in one line ((event.point – o).normalize()). The normals are amplified by force (force) to give us a smoother bump effect and then applied to the grid point. Finally, we use the built-in smooth() method on each path so we don’t have to worry about bezier handle adjustments.

function onMouseMove(event) {
	mousePoint = event.point;
	for (var j = 0; j <= gridSize; j++) {
		var path = layer.children[j];
		for (var i = 0; i <= gridSize; i++) {
			var segment = path.segments[i];
			
			var point = view.size * 1.2 * (new Point(i, j) / gridSize);
			var delta = (event.point - point).normalize();
			var force = ((view.size.height + view.size.width) / 2) * 0.2;
			segment.point = point + delta * [force / -2, force];
		}
		path.smooth();
	}
}

The last interaction we want to apply is a toggle for the debugger guidelines. These guidelines visually look like a selected path in Illustrator, complete with bezier handles (non-interactive at the moment). The fullySelected property on our project’s active layer will trigger guidelines an all its children – another example of the robust display hierarchy in Paper.js. Now we can see how the path handles change in real-time.

function onKeyUp(event) {
	layer.fullySelected = !layer.selected;
}

The finished demo can be viewed here. Make sure to view the source for better documentation and the complete flow. We took this demo a little bit farther and tied it to the Audio Data API (currently only available in FireFox 4+) and created an audio visualizer written purely in Paper.js.

We’re excited about the possibilities Paper.js creates – particularly when applied to data visualizations. Support and development of the library seems to be healthy, and the roadmap includes some very interesting future features. Paper.js + Node.js, anybody?