Background: coincidences and a bit of luck

A few years ago I was attracted by the great interface of “The Turn” website. Digging into the credits, I found out that Guille Lopez was responsible for the development. Funny enough, Guille was from Barcelona and he studied exactly the same and in the same place as I did.

Encouraged by the parallelism, I decided to write him and send my congratulations for the great work. By that time, I was still based in Berlin and Guille told me that he just moved to London where he started working at B-Reel.

About a year ago, it was me who moved to London and I had the chance to personally meet Guille right before he moved from London to B-Reel’s New York office.

Ten months later, Guille contacted me again and asked me if I was interested in taking part of a very exciting HTML5 music video project in collaboration with Google and B-Reel.

As soon as I got the briefing and we started talking about it, I got very excited. The video was going to be directed by Chris Milk, Aaron Koblin was behind it and Mr.Doob himself (who I casually met just a few days before all this happened) was going to be actively involved.

We were really lucky to have someone like Jaime Sánchez joining the dev team, which was completed by George Michael Bower and Michael Chang from the Google Creative Lab, and Eric Haeton who was in charge of the backend.

My role in the project

As you may already know, The Wilderness Downtown is composed by several pop up windows showcasing different contents during the interactive video-clip experience. Each content pops in and out at a certain time during track’s playback.

We had an amazing team of developers taking care of the production of all window’s contents: bird flocks, canvas 360° street-view, tree explosions, 3D optimisation, Google maps integration and routes calculation, Harmony drawing tool’s adaptation and so on…

B-Reel wanted me to create the mainframe of the application while making sure that everything came nicely together. There were a bunch of feature requests in order to make the concept work: content pre-loading, sharing information in-between windows, sound spectrum support, liquid pop-up window layout, a main controller, tracking …

The sequencer – aka the engine –

Luckily enough, Mr.Doob created this magic little Sequencer JS object that provides sequencing control of what we call Effects. Effects can be added to the sequence as follows:

seqeuncer.add ( effect, start_time, duration );

Effects are objects with show / hide functions which are used to update sequencer’s state at a given time.

So the Sequencer object provides the core functionality of the engine but this needed to be extended to fully support the content / pop up window system we were aiming for.

One of the biggest constraints was obviously the fact of using pop-up windows since these need to be launched by a user click in order to avoid the default popup-blocker behaviour. This actually means that we need such a work-flow:

1) Create all content objects and start loading them
2) When loaded, tell the user to proceed by clicking a launch button
3) Create all popup windows at once and “hide” non-used windows
4) Define content targets (tell every content on which window should go)
5) Start audio+sequencer playback
6) Append contents to target windows as soon as they need to be in the sequence

So we needed to create two different kinds of Effects for the sequencer: Popup Window Effects and Content Effects.
Popup Window Effects can be understood as cue-points that define the size and position of a popup window at a given time.
Content Effects are mostly Canvas experiments that are appended into a popup window’s body at “show” (start_time) and removed at “hide” (end_time = start_time + duration). Content Effects may need to preload images or video and need to notify when they are successfully preloaded.

The complexity of making adjustments

We had initially a 8-window setup, and only during the first act of the video, we had already generated about 30 window cue-points. One cue-point looks like something like this:

sequencer.add(new PopupEffect(popup3, {x: 10, y: 1024, width: 100, height: 60}, false), 132.36700439453125, 24.432998657226562);

Notice that the latest two numbers refer to “start_time” and “duration”.
Besides the window cue-points, we need to assign every content to a specific window as follows:

addContentToSequence(1, popup2, 31.117, 28.383);

So, only during the first act of the video, we had already about 50 lines of code with floating point numbers defining times and durations.
The full clip is defined by 4 acts, so we could expect about 200 lines of code like the ones above.

I suddenly realised how painful would it be when Chris and Ben, creative director at B-Reel, would like to make content and time adjustments to the sequence, as well as change content / window assignments, pushing a part of the sequence a few seconds forwards, resizing windows, and so on.
Doing such changes manually could become a mess and would keep me too busy right at the end of the project, which could be quite inconvenient.

We like tools to automate boring processes

I thought it would be useful for Ben and Chris to be able to do changes in the sequence by themselves, so I decided to build an interface for generating our sequencer code.

Since the project was about Chrome and HTML5, I thought it would make sense to use the same technologies, so the sequencer tool was created with CSS/HTML + Javascript and a little bit of PHP to dynamically generate JS files on the server. To minimise production time, the tool was uniquely targeting Chrome / Webkit since was for internal use only.

The soundtrack timeline

The tool works time-based around a main timeline reproducing the same “We Used To Wait” soundtrack as a reference. On besides, the original animatic sketch from Chris Milk is added into it, so we have a constant visual reference of the current act / sequence.

It is possible to navigate through the timeline either by dragging the progress marker or by entering a time code manually. Hitting the spacebar key will toggle play/pause.

The cue-point editor: controlling windows

I have created a light JS window manager to control windows. The “fake” windows can be resized and moved around which updates / creates automatically a window cue-point.

History
The cue-points are stored and sorted by time values into a history drop-down which allows to directly jump to the particular cue-point.

Tweens
We have a tween option that allows the windows to be animated when a cue-point is reached (We didn’t use this feature at the end although is still implemented and ready to be used).

Playback
If the sequence is played, it is possible to see on-the-fly the window animatic sequence as it will behave later within the “real experience”.

Copy & Paste
Ben wanted to be able to copy & paste cue-points since the “hidden” status of a window happens to be always the same width, height, x and y settings. This saved Ben some time

Content cue-points: assigning contents to windows

We need to distinguish in between two types of content. In one side we have pure video contents, and in the other hand the interactive / JS Canvas experiments.

The tool has a built-in video library that can be browsed and which displays video information such as size in pixels and duration. When assigning a video content into a window, we can automatically let the system calculate the duration of the content based on the original video duration.

If we are assigning other contents rather than video, we’ll only be accessing the “standard” parameters: width, height and duration.

All content objects support dynamic size (width and height). When selecting a target window for a content, the width and height are automatically adjusted to fit window’s size, so everything makes sense.

Dynamic canvas

We had a minimum target resolution of 1024 x 768 but loads of users have big screens with higher resolutions.

The tool displays a cyan 1024 x 768 canvas where to position and resize the windows. Later on, the real browser application is dynamically adjusting the positions of the windows and contents according to user’s resolution.

I need a break, please save my progress

There is a save functionality built-in, so the sequence can be finished another time.

Finally, let me preview it and see how it looks like

Current settings can be saved as Javascript sequencer code and be previewed in a version of the real application within a click.

Considerations and what could have been much better

The sequencer tool was a very useful toy for playing around and generating the final animatic but the interface and UX could have been much better.

You have to take into account that I couldn’t spend too much time on it since I needed to be working on all the other stuff; but would have been great to have a graphic drag-and-drop timeline interface to visualise and move around the cue-points.

Another cool feature would have been previewing content directly built in the fake windows of the sequencing tool.

Some problems faced during the progress and things worth sharing

Preloading
The contents were generated before being appended to the DOM. Some contents needed to fetch data from Google and preload some images but I faced a problem with the videos since media elements (video and audio) won’t load if they aren’t appended to the DOM; so we had to rely on cache for the videos.

It was only possible to preload a video by appending it into a DIV located outside the main window or invisible and remove it again when the video reached the “ready” state (no need to actually preload the whole video).

Safari is very, very picky
Safari is very strict on which audio and video codecs should be in the web. It would be a nice move if Apple would add support for Vorbis/Theora but they clearly want H264 and AAC to be the standard HTML5 codecs.

I recommend to any Safari user in Mac to install this great Quicktime component that will allow your Safari as well as your Quicktime player to support OGG and OGV files: http://www.xiph.org/quicktime/download.html

We obviously needed to have OGG / OGV with MP3 and MP4 fallbacks, but the most ridiculous episode was when Safari wasn’t working with MP3 files encoded with LAME. We had issues with different versions of Safari and at the end we had to encode the MP3 file using iTunes. Surprised?

A lovely python script to generate sound spectrum values at a perfect framerate
We needed some audio spectrum support and you can’t do this in real-time using Javascript. Well, Firefox is able to do this (at least the 4 beta), but not Safari nor Chrome. Mr.Doob pointed me at a great python script that lets you generate an array of spectrum values at a perfect framerate, given a WAVE file:

import math
import struct
import wave
import sys

w = wave.open(sys.argv[1], 'rb');
# We assume 44.1k @ 16-bit, can test with getframerate() and getsampwidth().
sum = 0;
value = 0;
delta = 0;
amps = [ ];

for i in xrange(0, w.getnframes()):
	# Assume stereo, mix the channels.
	data = struct.unpack('<hh', w.readframes(1))
	sum += (data[0]*data[0] + data[1]*data[1]) / 2
	# 44100 / 60 = 735 (60 fps)
	if (i != 0 and (i % 735) == 0):
		value = int(math.sqrt(sum / 735.0) / 10)
		amps.append(value) # absolute value no delta
		delta = value
		sum = 0

print amps

More information in his post here.

The web trend loop

When I started working in Barcelona back in 1999 there was a hype thanks to DoubleYou. It was the Netscape vs Explorer times and the hype was called DHTML (Dynamic HTML). DHTML isn’t far of the technology used for “The Wilderness Downtown”: HTML + Javascript.

Flash was by then stuck at version 4 and AS1 and it was becoming popular thanks to people like Joshua Davis and Gmunk.

I remember one of my first projects. We created a website for the spanish pavilion of an art fair that I barely remember. We used DHTML and we recreated a Mac OS 9 OS using Javascript and with a JS window manager, quite a coincidence
I remember one of the “plus” features being an animated (DIV tween) desktop background. It was 2000 and it was running in the two major browsers: Netscape and Explorer. The site won’t work in any of the modern browsers today.

I have been working with AS2 and AS3 for many years now and I really wanted to work in a HTML / Javascript interface again, and this project was as challenging as ironically bringing me in a way back to where I started. Of course many things have changed: AJAX, Canvas, Video, Audio, minimised cross-browser pain and WebGL around the corner; but Javascript feels quite the same as it did back then.

Oh, and don’t get me wrong. I am not trying to participate in any “Flash VS HTML5″ bullshit. I just enjoy using different technologies and keep learning things out of challenging projects. In fact, I think I should step out of the web soon and invest a few months in other types and formats of work.

I would like to know more about the graphics / content part

If you would like to know more about the 3D / Graphics, optimisation and the Google API, you should read this great post by Mr. Doob.

There is also some more information about the project here.

I have been interested in using MIDI devices for different specific interfaces lately. I have been using mainly Openframeworks to play a little bit around with it and understand the possibilities of such hardware interfaces.

If you have followed my previous MIDI / OF tutorial, you’ll see how easy is to get your MIDI hardware up and running in OF, but what about using MIDI hardware in Flash? I was really interested on being able to use a MIDI controller within a Flash application, so I researched a little bit and found this helpful post.

Apparently, you can send data from OF (C++) to Flash via TCP using binary sockets in as3 and ofxTCPServer in OF.

The approach is simple:

1) Create a C++ TCP Server application using OF to send encapsulated MIDI messages.

2) Create an as3 client application that reads the MIDI messages and do whatever you want to do with them

TCP Server (OF)

Let’s start with the OF server side: you’ll need to use the ofxMidi and ofxTCPServer addons.

The first thing I did was to create a simple C++ class to encapsulate the necessary information of my MIDI messages to be sent:

/*
 *  SocketMessage.h
 *  MIDISocket
 *
 *  Created by Eduard Prats Molner on 04/10/2010.
 *  Copyright 2010 Jocabola. All rights reserved.
 *
 */

#ifndef SOCKET_MESSAGE
#define SOCKET_MESSAGE

class SocketMessage{

public:

	SocketMessage();
	~SocketMessage(void);

	int MIDI_val;
	int MIDI_ID;
	int MIDI_port;

};

#endif

I used the ofxTCPServer OF sample combined with my MIDI test class. This is how the TestApp header looks like:

#ifndef _TEST_APP
#define _TEST_APP

#include "ofMain.h"
#include "ofxMidi.h"
#include "ofxNetwork.h"
#include "SocketMessage.h"

class testApp : public ofBaseApp{

	public:
		void setup();
		void update();
		void draw();

		void keyPressed  (int key);
		void keyReleased(int key);
		void mouseMoved(int x, int y );
		void mouseDragged(int x, int y, int button);
		void mousePressed(int x, int y, int button);
		void mouseReleased(int x, int y, int button);
		void windowResized(int w, int h);

	void newMessage(ofxMidiEventArgs &args);

	ofxMidiIn midiIn;

	ofxTCPServer TCP;

	ofTrueTypeFont  mono;

	vector <string> storeText;

};

#endif

In the setup method we’ll need to initialise the MIDI and TCP Server as follows (setting up the TCP server at port 11999):

void testApp::setup(){
	midiIn.openPort();
	ofAddListener(midiIn.newMessageEvent, this, &testApp::newMessage);

	//load our type
	mono.loadFont("type/mono.ttf", 8);

	//setup the server to listen on 11999
	TCP.setup(11999);

	ofBackground(0, 0, 0);
}

Now, I’ll update the MIDI “new message” listener to be sending the desired MIDI arguments using the TCP server and encapsulating the MIDI port, id (which knob, button, toggle) and value into a SocketMessage class instance:

void testApp::newMessage(ofxMidiEventArgs &args){
	SocketMessage* sm = new SocketMessage();
	sm->MIDI_port = args.port;
	sm->MIDI_ID = args.byteOne;
	sm->MIDI_val = args.byteTwo;
	TCP.sendRawBytesToAll((char*) sm, sizeof(SocketMessage));
	storeText.push_back("Sending message: <port: " + ofToString(sm->MIDI_port) + "; id: " + ofToString(sm->MIDI_ID) + " value: " + ofToString(sm->MIDI_val) + ">");
}

Notice that I stored the messages into a storeText vector which I use for printing messages in the little server app. That’s how my update and draw methods look like:

void testApp::update(){
	if(storeText.size() > 28){
		storeText.clear();
	}
}

//--------------------------------------------------------------
void testApp::draw(){
	mono.drawString("TCP SERVER launched on port: " + ofToString(TCP.getPort()), 10, 20);

	for(int i=0;i<storeText.size();i++){
		mono.drawString(storeText[i], 10, 30 + 10*i);
	}
}

Client application (AS3)

Is time for grabbing the MIDI messages from Flash!

My Flash application is called MidiSocket and has a Socket class instance that acts as a client application:

//--------------------------------------
//  CONSTANTS & VARIABLES
//--------------------------------------

private var _client:Socket;

public function MidiSocket(){
	_client = new Socket();
	_client.addEventListener("ioError", retry);
	_client.addEventListener("connect", connectionEstablished);
	_client.addEventListener("socketData", socketData);

	_client.connect("localhost", 11999);
}

Now, before getting into the actual socket data parsing / fetching. Let’s make some memory about the two variables of the MIDI messages that are really important to us: byteOne and byteTwo.

As their names already tell us, each of these variables is represented by one byte (8 bits). byteOne is an unsigned integer that represents the ID of the MIDI parameter being changed (identifies a particular knob, a fader, a button, …). byteTwo is the actual value of it which is represented by an unsigned integer that goes from 0 to 127 (128 possible values).

When fetching the values in Flash, I faced the problem that the values being read where really long weird nonsense numbers. I suspected that had to do with the bit representation of the integers since Socket.readInt() returns a 32-bit integer, so I had to refresh “bit shifting”: Great explanation here.

Basically, I needed to shift right 24 bits to convert the values back to the original 8 bit representation. In other words, to be able to have nice integers from 0 to 128 (much easier to handle) once again.

private function socketData(evt:ProgressEvent):void{
	var buffer:ByteArray = new ByteArray();
	_client.readBytes(buffer, 0, _client.bytesAvailable);

	var val:uint = buffer.readInt() >> 24;
	var id:uint = buffer.readInt() >>  24;
	var port:uint = buffer.readInt();	

	trace("MESSAGE: <port: " + port + ", id: " + id + ", value: " + val + ">");
}

There are also two more important things to take into account. Make sure to read the whole buffer; otherwise you’ll be accumulating some weird data that will end with wrong values being read. So best practice is to always read the whole buffer: _client.readBytes(buffer, 0, _client.bytesAvailable);

Once you grab the bytes, remember to fetch the arguments in the same exact order as they are defined in the C++ SocketMessage class: For instance, first int will be the value, second the id and last one the port.

That’s pretty much it! Get the sources here and feel free to drop any questions as comments.

Considerations

1) I think it should be better to create a pure C++ command line tool for such a server app.

2) Notice that MIDI was just an example. Using this technique you can send any data from OF to Flash, and works very fast (complete real-time feeling with the MIDI controller).

3) I haven’t tried the buffer.readObject in depth. Would be great to serialise objects; Something like AFM / JSON and work with objects encapsulating all variables instead.

Basically, once recorded, the main animation is constantly looping with a frame offset of +1 in each consecutive tile.

Imagine the recorded animation having 250 frames. We set up a counter that goes from 1 to 250 and then starts over again.
Each tile renders the current counter frame + an offset defined by its position in the matrix as shown below (n represents current frame):

[n] [n+1] [n +2] [n +3] [n + 4] …

The results of a few test animations turned into this:

Visit the full-browser version if you want to see it bigger.

To store the frames I used the BitmapData Object and its draw function which allows to easily capture any DisplayObject as pixels. I created a base Sprite subclass called “BitmapRenderer”. The class contains a little “renderer” sprite which is masked by a square defined by the bitmap size (tile size) that we want:

//
//  BitmapRenderer
//
//  Created by Eduard Prats Molner on 2010-06-06.
//  Copyright (c) 2010 Jocabola. All rights reserved.
//

package render{

	import flash.display.Sprite;
	import flash.display.BitmapData;
	import render.BitmapRendererEvent;

	/**
 	* Class.
 	*
	* @langversion ActionScript 3.0
 	* @playerversion Flash 9.0
 	*
 	* @author Eduard Prats Molner
 	* @since  2010-06-06
 	*/
	public class BitmapRenderer extends Sprite{

		//--------------------------------------
		//  CONSTANTS & VARIABLES
		//--------------------------------------

		public static const BITMAP_WIDTH:uint = 80;
		public static const BITMAP_HEIGHT:uint = 60;

		protected var __bitmap:Array;

		protected var __renderer:Sprite;
		protected var __rendererMask:Sprite;

		/**
	 	* @constructor
	 	*/
		public function BitmapRenderer(){
			super();
			__bitmap = new Array();

			addChild(__renderer = new Sprite());
			addChild(__rendererMask = new Sprite());

			__renderer.mask = __rendererMask;

			__rendererMask.graphics.beginFill(0x000000, 1);
			__rendererMask.graphics.drawRect(0, 0, BITMAP_WIDTH, BITMAP_HEIGHT);
		}

		//--------------------------------------
		//  PUBLIC METHODS
		//--------------------------------------

		public function begin():void{// to be extended - here we add the actual animated objects into __renderer}

		//--------------------------------------
		//  PRIVATE & PROTECTED INSTANCE METHODS
		//--------------------------------------

		protected function dispatchBitmapEvent():void{
			dispatchEvent(new BitmapRendererEvent("renderComplete", __bitmap));
		}

		protected function addBitmapData():void{
			var bmpd:BitmapData = new BitmapData(BITMAP_WIDTH, BITMAP_HEIGHT, false, 0x000000);

			bmpd.draw(this);
			__bitmap.push(bmpd);
		}
	}

}

Once our animation is ready, we’ll fire a “BitmapEvent” which contains our array of BitmapData objects (the actual frames of the animation):

//
//  BitmapRendererEvent
//
//  Created by Eduard Prats Molner on 2010-06-06.
//  Copyright (c) 2010 Jocabola. All rights reserved.
//

package render{

	import flash.events.Event;

	/**
 	* Class.
 	*
	* @langversion ActionScript 3.0
 	* @playerversion Flash 9.0
 	*
 	* @author Eduard Prats Molner
 	* @since  2010-06-06
 	*/
	public class BitmapRendererEvent extends Event{

		//--------------------------------------
		//  CONSTANTS & VARIABLES
		//--------------------------------------

		private var __data:Array; // BitmapData array

		/**
	 	* @constructor
	 	*/
		public function BitmapRendererEvent($type:String, $data:Array, $bubbles:Boolean = false, $cancelable:Boolean = false){
			super($type, $bubbles, $cancelable);
			__data = $data;
		}

		//--------------------------------------
		//  GETTER/SETTERS
		//--------------------------------------

		public function get data():Array{
			return __data;
		}

	}

}

So, each of the 5 simple animations are basically child classes of “BitmapRenderer”. Here is one sample:

//
//  CircleScaleRenderer
//
//  Created by Eduard Prats Molner on 2010-06-06.
//  Copyright (c) 2010 Jocabola. All rights reserved.
//

package render{

	import render.BitmapRenderer;
	import flash.display.Sprite;

	import com.greensock.*;
	import com.greensock.easing.*;

	/**
 	* Class.
 	*
	* @langversion ActionScript 3.0
 	* @playerversion Flash 9.0
 	*
 	* @author Eduard Prats Molner
 	* @since  2010-06-06
 	*/
	public class CircleScaleRenderer extends BitmapRenderer{

		//--------------------------------------
		//  CONSTANTS & VARIABLES
		//--------------------------------------

		private var __circle:Sprite;

		/**
	 	* @constructor
	 	*/
		public function CircleScaleRenderer(){
			super();

			__renderer.addChild(__circle = new Sprite());
			__circle.graphics.beginFill(0xFFFFFF);
			__circle.graphics.drawCircle(0, 0, 24);

			__circle.x = BitmapRenderer.BITMAP_WIDTH/2;
			__circle.y = BitmapRenderer.BITMAP_HEIGHT/2;
		}

		//--------------------------------------
		//  PUBLIC METHODS
		//--------------------------------------

		override public function begin():void{
			TweenLite.to(__circle, 1, {scaleX: 0.1, scaleY: 0.1, ease: Strong.easeIn, onUpdate: addBitmapData, onComplete: reverseAnimation});
		}

		///--------------------------------------
		//  PRIVATE & PROTECTED INSTANCE METHODS
		//--------------------------------------

		private function reverseAnimation():void{
			TweenLite.to(__circle, 1, {scaleX: 1, scaleY: 1, ease: Strong.easeOut, onUpdate: addBitmapData, onComplete: dispatchBitmapEvent});
		}

	}

}

Notice that in this sample I am using Greensock’s TweenLite engine. Luckily, and thanks to Mr.Doob, I have recently switched to eaze tween engine for my Flash work.

I hope this would be helpful to someone. Feel free to drop a comment if you have any questions.

I recently realised that some London crosswalk lights were updated with a countdown (or at least never realised before). This is a neat feature, giving visual feedback to the annoying “long” waiting period with both, traffic lights and pedestrian lights, in red.

Apparently, the pixel data was somehow corrupted which generated beautiful unexpected results. It looks quite similar to a broken graphic card’s driver or so. The graphics looked even cooler in motion

Update:
These accidents where due to a bug in current OF for iPhone: Here is a fix by Theo.

The mini splash page of Cell Flot has been updated with a video showcase of current and upcoming work. We’ll be adding new videos until we launch a portfolio site.

The application has a set of variables: colour (r, g, b), size, extent and opacity which can be randomized with a click. My first preset is super basic, just using circles and mapping their resolution to one of the parameters, so circles eventually become hexagons, quads, triangles or even lines (2 segments).

The second preset is driven by short-living particles which are drawing their growth before dying.

Presets can be combined which, together with randomizing the settings, creates infinite output. The application features other presets by Andreas Müller, Kim Asendorf, Martin Fuchs, Philip Whitfield and Mike Tucker. For more information visit http://sktchapp.com/

It was a fun experience and an open door to creative application development for the iPhone using OpenFrameworks.

You can buy the app here and visit its Flickr group page!

Siemens SIMATIC S7-1200 is a desktop application which I have developed in Action Script 3 and Zinc.

Jared shared a little source code of a particle which he called the “firefly”. The code of a basic particle class in Processing would be basically something like this:

// Particle class
class Particle{
  float x;
  float y;
  float vx;
  float vy;

  Particle(float _x, float _y, float _vx, float _vy){
    x = _x;
    y = _y;
    vx = _vx;
    vy = _vy;
  }

  void update(){
    x = x + vx;
    y = y + vy;

    // change direction
    vx += random(-1,1);
    vy += random(-1,1);

    //apply friction
    vx *= 0.9;
    vy *= 0.9;
  }
}

I started to experiment a little bit while learning Processing syntax, the main idea was to use the particles to actually draw their routes instead of rendering them as moving objects.

The different results where obtained by applying different basic rules in the system.