Nicolas Garcia Belmonte

Behind the scenes: Visualizing the conversation around the #WorldCup

2014-07-10T00:00:00+00:00

Yesterday we released an article showing Twitter data about the global conversation around the World Cup. The main piece consisted on a data visualization showing how each country mentioned the national teams.

This is a behind the scenes on this work. I will show how we got to design the visualizations and interactions we came up with for the project and also I’ll cover some interesting insights taken from the data visualization.

So first things first, if you haven’t yet please go on, have a look!

The visualization has two main parts. The top one lets you select a country that participated in the World Cup and for that country one can see things such as:

What were the top countries mentioning the selected country, also called mentions of
What other countries were mentioned by the selected country, also called mentions by

This was done by analyzing hashflag mentions in Tweets.

We went through a few iterations for the top visualization:

First iteration

The first take we had on it merged both the menu and the graph in one single graphic:

The above row shows “mentions by”. The bottom row shows “mentions of”. The above image shows the results for Brazil. This image shows for example that people in Brazil also mention Argentina and Germany (cropped). It also shows that most of the other countries in the World Cup mention Brazil constantly.

To enforce the notion of directionality we implemented the tapered representation recommended in Holten and van Wijk paper.

However, after some user testing we found out that although the visualization combined both the menu interaction and the mentions data in a compact way, it was confusing for people. The directionality was not clear and also it wasn’t clear that the labels were also used as controls to select a country.

Second iteration

The second iteration then focused on simplifying the graph. The simplest encoding we came up with to represent the information were two sets of bar charts, one showing a list of top countries mentioning the selected country or mentions of; and another bar chart mentioning the list of top hashflags mentioned by the selected country or mentions by:

These two bar charts convey almost the same information than the first design. The main differences are that we only select the top 4 countries on each side of the list, and also that we decide to add percentages to show how often mentions of and mentions by happened.

This design also forced us to separate the interaction part from the visualization. We created a separate menu listing all hashflags:

Paradoxically we found that this design was in a way worse: although it looked simple, it now consisted of two disconnected parts and it was hard to visualize the flow of mentions by/of.

Third iteration

After more user testing and gathering feedback from our team we came up with the third design. This one still focused on “low entropy” but also included the notion of directionality we had on the first design. It shows almost the same information than the first one, but needed less text and less explanation time to be understood by our test users.

A few interesting insights emerge from this visualization:

For Brazil, its team was widely discussed among football fans in Chile and Mexico, while Brazilians were paying a lot of attention to semifinal opponent Germany, archrivals Argentina, and fellow semifinalists Netherlands:

For France it’s interesting to see some of the immigration patterns. The most mentioned hashflag from France is #ALG. Top countries mentioning #FRA are french speaking countries and ex-colonies:

For the US the most mentioned hashflag is #MEX (for Mexico).

Some other connections can be geographical, like Bosnia and Herzegovina and Croatia mentioning each other:

For Argentina, it’s interesting to see that besides Brazil the main mentioned country is Colombia. The Argentine coach Pekerman was the coach for the Colombian team this year.

Bottom visualization

The bottom visualization was actually the first visualization we worked on. It is mostly an exploratory visualization and shows with finer detail all the hashflag mentions for the selected country day by day.

The viz is a bit harder to read, but there is a rationale behind the design choices for it.

Each column represents a day, and the width of each column is sized according to the volume of Tweets with hashflags sent from that country on the given day. Inside each column we have the relative distribution of hashflag mentions during that day. Here’s an example for Argentina:

The most challenging feature to grasp might be the width encoding for each day. Why not just create a stacked bar chart and use the (cumulative) height of each bar to represent the amount of Tweets for each day? This would be the result for Argentina:

As you can see (or not see), some days are readable, but most of them aren’t. This happens because we’re encoding two things with height: the number of Tweets for each hashflag on a given day and the cumulative number of Tweets on a given day. Yes one adds up to the other, but these are different metrics and can be encoded differently.

To get our visualization the first step is to get rid of the height encoding for the Tweets per day value. We now have a percentage based area chart:

Second step is to encode total Tweets per day on the x-axis.

This is not a novel design: you can find this in slice and dice treemaps. As opposed to squarified treemaps, the slice and dice treemap algorithm maintains order and stability for nodes, but its nodes usually suffer from having high aspect ratio. You can explore squarified, slice and dice and strip treemap layout algorithms in this implementation I made with the JavaScript InfoVis Toolkit a long time ago.

The treemap layout also has some interactions. By clicking on a day one can zoom in and see the relative mentions for each hashflag with more detail:

All in all it proved to be a nice exploratory tool to infer some of the findings I mentioned for the top visualization.

Conclusion

Data visualization is an iterative process. Moreover, you cannot do data visualization by yourself and expect it to be optimal. Do not underestimate user testing and most of all, don’t underestimate peer feedback. With that I’d like to thank the communications and analytics team at Twitter for their feedback and testing of the visualization.

Multilevel Agglomerative Edge Bundling in JS

2013-12-16T00:00:00+00:00

Hello data visualizers! I just released a JS version of the MINGLE edge bundling technique: https://t.co/Gvm29H6gf2 pic.twitter.com/SJadXUK2nW
— nico (@philogb) 13 Décembre 2013

This is a JavaScript implementation of the paper Multilevel Agglomerative Edge Bundling for Visualizing Large Graphs (Emden R. Gansner, Yifan Hu, Stephen North, Carlos Scheidegger).

The edge bundling algorithm groups edges together to minimize the amount of ink used to render a graph. This particular paper introduces a fast technique to perform edge bundling.

Take for example this map connecting locations between the east coast in the US and western Europe:

The algorithm creates a proximity graph for the edges where each of the edges is represented by a node. Then the algorithm bundles edges as long as we’re saving some ink in the final rendering. Here’s an intermediate step on the bundling animation:

And here’s the final result:

This implementation is solely based on the paper. The license for the code is MIT.

Examples

This simple example shows links connecting locations in the Bay Area. The rendering uses 2D Canvas but could use any other rendering API.

You can see an example here.

DotJS

2013-12-04T00:00:00+00:00

Last Monday I was in Paris for DotJS, organized by Sylvain Zimmer and dotConferences, arguably the biggest JavaScript conference in Europe, with more than 900 JavaScript “savants”.

#dotjs from the front. Thems be a lot of people http://t.co/RqEXbrbJEl
— Remy Sharp (@rem) 3 Décembre 2013

My talk was titled “Growing a Language for Graphics”, which was about JavaScript as seen from the eyes of a data visualization / graphics person. By having a look at GLSL - a domain specific language (DSL) for graphics used in WebGL - I talk about what language features would help graphic developers on the Web and I show the long history of trying to standardize operator overloading into JavaScript.

I’ve been wanting to a give a talk about this topic for a long time, having witnessed most of the discussions around operator overloading in JavaScript and it’s different proposals since early 2007. Going from generic functions to a double-dispatch approach to multimethods, to value proxies and finally value objects, I must say that my interest in having this feature into JavaScript never faded.

My talk is inpired on a great talk by Guy Steele Jr. titled “Growing a Language” and that I definitely recommend you to watch:

In case you’re interested in knowing more about operator overloading proposals in JavaScript I’m pasting a dump of the bibliography here:

Why operator overloading

http://www.jroller.com/cpurdy/entry/the_seven_habits_of_highly1

ES4 overview

http://www.ecmascript.org/es4/spec/overview.pdf

Generic functions

https://mail.mozilla.org/pipermail/es-discuss/2007-November/005043.html

https://mail.mozilla.org/pipermail/es-discuss/2007-November/005049.html

https://mail.mozilla.org/pipermail/es-discuss/2007-November/005052.html

https://mail.mozilla.org/pipermail/es-discuss/2007-November/005053.html

2009 Operator Overloading proposal

https://mail.mozilla.org/pipermail/es-discuss/2009-January/008535.html

https://mail.mozilla.org/pipermail/es-discuss/2009-January/008541.html

http://wiki.ecmascript.org/doku.php?id=strawman:operator_overloading_with_double_dispatch

ExtendScript OO

http://forums.adobe.com/thread/646268

http://www.indiscripts.com/post/2010/05/operator-overloading-with-extendscript

Operator Overloading revisited

https://mail.mozilla.org/pipermail/es-discuss/2009-June/009603.html

https://mail.mozilla.org/pipermail/es-discuss/2009-June/009604.html

https://mail.mozilla.org/pipermail/es-discuss/2009-June/

Value types

http://wiki.ecmascript.org/doku.php?id=strawman:value_types

Decimal type:

http://intertwingly.net/stories/2008/09/20/estest.html

https://mail.mozilla.org/pipermail/es-discuss/2009-January/008646.html

Value types page started:

https://mail.mozilla.org/pipermail/es-discuss/2010-January/010677.html

Value objects

http://wiki.ecmascript.org/doku.php?id=strawman:value_objects

https://mail.mozilla.org/pipermail/es-discuss/2012-March/021411.html

http://www.slideshare.net/BrendanEich/value-objects

https://bugzilla.mozilla.org/show_bug.cgi?id=749786

Value proxies

http://wiki.ecmascript.org/doku.php?id=strawman:value_proxies

ES next

https://mail.mozilla.org/pipermail/es-discuss/2013-April/029949.html

Laika

2013-10-16T00:00:00+00:00

Last night I presented at the data visualization meetup at Twitter HQs. It was really awesome to meet the people there and nerd-out about visualization related things.

You can find the slids for my presentation here. Also the video is here:

The presentation was mostly about the process of releasing public-facing data visualizations at Twitter, and also some analysis on one project we’re currently working on called Laika.

#interactive

Some of the examples I’ve shown are interactives that complement a story. For example I presented about Twitter’s biggest moments

and also about our last Emmys visualization.

In the presentation I explained that these projects are generally done within the week (sometimes even in less time). The ratio of time invested vs. amount of engagement is high, so it’s a strategy that currently works for us.

Laika

Laika is a data visualization showing geolocated activity on Twitter around the world. The first iteration of the project was done during hackweek. We had great feedback from people in the office so per request the project turned into a live visual that is currently shown in the displays around the Twitter HQs.

The main view shows a globe view, with day and night textures, displaying geolocated Tweets and relevant Tweets around news (in the top part). Click play to watch the short video.

Within this globe view we can focus on a particular city and fetch a timeline of relevant Tweets happening there. Click below to watch the short transition.

Although people like the globe view, this view hides 50% of what’s happening in the world at each moment. We added an unfolding of the globe into a plane for this. Click play below to watch the short video.

Following the “Overview, zoom and filter” mantra you can focus in a city to get relevant Tweets of what’s happening there along with news of that city.

Mobility Flow of France

2013-06-09T00:00:00+00:00

Click here to access the map (A WebGL capable browser and computer is required).

A few months ago I spoke at Strata in Santa Clara, CA. My presentation was about Exploring Web Standards for Data Visualization and in the presentation I showed some examples of visualizations done at Twitter with HTML/SVG, 2d Canvas and WebGL.

I also showed a custom visualization of the mobility flow in France (i.e. how people moved from state to state in the past five years) and an attempt at visualizing this pretty rich dataset (> 45,000 entries) with different standards. You can find the result of the visualization here. Please make sure you have a WebGL capable browser (and machine) to view it.

Although this is a pretty classical map, the work was inpired by Jacques Bertin’s Semiology of Graphics. Since he was a french cartographer his book is filled with maps and cartograms of France for different datasets.

The visualization has two types of maps. The first one is an aggregated map by state and county, where states are reddish if the overall state population increased in the past five years, and a more blueish color if the population decreased. Clicking on a state will show this information at a county level.

The other map shows mobility flow for a selected state. It gets triggered when a state is hovered more than 2 seconds. In that map reddish states are the ones getting people from the current hovered state and blueish states are states where people moved to the hovered state.

Following the second map there are some interesting insights. For example, if you hover on Paris you’ll note that most people move from the city centre to the suburbs.

If you hover the suburbs, you’ll see people going even further away from Paris. If you follow this pattern you’ll find that a lot of people are moving from Paris to Bretagne:

Another interesting fact is that if you hover some state in the south of France you’ll see a lot of people from everywhere else going there, except for Paris: in that case there are lots of people going from the south of France to Paris.

There are other interesting patterns too. I think this does a better job than the US mobility map. The US map does not show information when there’s no interaction, and when there is the edges added to the viz occlude the actual color of the states and things become hard to read.

Standards Used

One of the challenges for the visualization was to display data at the county level:

I first tried to just simply load the SVG county map of France, but this just took too long:

I know it looks like a neat animation, but this is actually the result of loading the county svg map (without caching) in chrome. It takes more than 30 seconds to load!

Instead I tried a layered approach, in which different layers for maps of France where loaded. One for the contours, another for the data itself and a last one for the picking / interactions.

For picking we used a map in which each state has a different color, same for the counties.

When a user hovers a state or county, we can retrieve the color of the pixel under the cursor in the picking maps, and so we get an id for the state to retrieve information from. I explained this technique of color picking before in examples like the wind map and the flight patterns visualization.

The result is a smooth interactive visualization that fully uses the GPU to render.

Crazy Idea for Web Worker Extension

2012-11-04T00:00:00+00:00

Introduction

This article describes an extension to Web Workers by enabling the use of a subset of the Shading Language 1.0 – as defined by the WebGL and OpenGL ES specifications – on a Web Worker.

The Shading Language is already present in web applications that use WebGL or CSS Shaders. Web Workers are implemented and exposed by the major browsers Opera Software, Google Chrome, Apple Safari and Mozilla Firefox. The idea is to combine these two things to enable Web Workers to execute Shading Language code.

Motivation

This would enable (depending on the inner implementation) faster code execution, but more important graphics programmers or developers in need for more clever idioms for mathematics or computational geometry operations could use this language to perform these operations easily. Much of the Web today depends on graphics, as can be seen by the numerous languages and APIs designed to render content on the screen (HTML/CSS, WebGL, 2D Canvas and SVG) and although JavaScript is suitable for prototyping, it doesn’t provide features that are friendly to graphic developers such as operator overloading, static typing, advanced built-int math and geometry functions and matrix and vector classes.

How it works

The main idea of the api extension would be to define an extra option argument to the Web Worker constructor that would specify the type of the file to be executed. By default the file type would be JavaScript:

var worker = new Worker('myworker.glsl', { type: 'glsl' });

Then the worker api would remain the same, with onmessage and postMessage operations. If used with GLSL, the postMessage operation would be restricted to send transferable objects, in particular ArrayBuffers or TypedArrays. Continuing the example we would have:

//create array of floats
    var floats = new Float32Array([1, 2, 3]);
    //send this data over transferable objects to the worker
    worker.postMessage(floats);
    //add a listener to get back the data
    worker.onmessage = function(e) {
        //print the resulting typed array
        console.log(e.data);
    };

The shader file would require a main function which can have an in transferable object argument and out transferable object argument. For example, we could have inside myworker.glsl:

void main(in float numbersIn[3], out float numbersOut[3]) {
    numbersOut[0] = numbersIn[0] * numbersIn[0];
    numbersOut[1] = numbersIn[1] * numbersIn[1];
    numbersOut[2] = numbersIn[2] * numbersIn[2];
}

Once main returns, the out parameter will be set as the data property of the event object and sent in the onmessage callback.

Implementation notes

I don’t think that in this case GLSL would be run in the GPU (that would be crazy!), but I do think that GLSL is a language that could be safely executed or safely transpiled into any language in the ANSI C family. As opposed to the WebCL/OpenCL 1.0 language, GLSL does not have pointers and seems safer enough that it’s currently being used in CSS Shaders and WebGL.

Twitter's Political Engagement Map

2012-11-01T00:00:00+00:00

I worked on an exploratory data visualization at Twitter that tracks engagement in the US for tweets from the candidates. You can see the article on the Twitter blog here.

There are some interesting insights from the visualization summarized in the blog post too. For example, searching for tweets with the “coal” keyword shows more engagement on the states that are leading producers of coal:

Other interesting searches ares ones for Pell Grant or God.

You can also take a look at the topics that interest more a given state by clicking on it. For example, here’s what we get about New York:

You can find more insights on the Twitter blog.

Group Theory for Fun and Profit

2012-09-20T00:00:00+00:00

Sometime earlier this year I started reading (for the third time) Conway’s book on quaternions and octonions. I never finished reading this book the previous times because I would get hooked with a specific topic mentioned there, do more research on that and never actually finish the book itself.

So that happened to me this time also. I started reading on symmetry groups, or wallpaper groups. Wallpaper groups are two dimensional repetitive patterns obtained by translations, rotations, reflections and glide reflections. As shown in Conway’s book, there are 17 different possible groups:

With the help of my collegue Bei we built a WebGL application that lets you create your own motifs and then apply the transformations of these 17 symmetry groups on your drawing. You can access the app here!.

Here’s a quick video that shows how to use it:

Neil Armstrong Video

2012-08-31T00:00:00+00:00

I created a data visualization that shows the reaction on Twitter after NASA announced that Neil Armstrong passed away on August 25th 2012.

There were more than 1.6 million tweets mentioning Neil Armstrong on a 48 hour period. Each tweet is represented by a yellow dot in this video:

The video team worked on this idea to create a more polished video for the visualization that you can find in Twitter’s blog.

Line integral convolutions

2012-08-14T00:00:00+00:00

TL;DR: I implemented a nice technique to animate fluids in WebGL with floating point textures that you can try here. The technique is based on a very interesting paper that I describe below.

When I was a kid my physics teacher showed us this amazing experiment where he would put a magnetic dipole inside of a box with some sort of “iron sand” and one could see the sand taking the shape of the vector field:

which I always thought was similar to the curves in the sand that one may find in a Japanese garden:

There’s an interesting technique in which a vector field can be rendered to look just like the above pictures. This technique is called line integral convolution. I’ve been working on this and implemented a couple of papers related to this technique in JavaScript, 2D Canvas and WebGL.

LICs in the CPU

The first implementation I worked on is from the paper Imaging Vector Fields Using Line Integral Convolution by Brian Cabral and Leith (Casey) Leedom. This technique takes a vector field and a white noise image, and convolves them in order to get something very similar to the images above. For example, here’s a rendering of a (-y, x) field with this technique.

One drawback is that the algorithm’s complexity is relative to the number of pixels in the image, since for a given subset of those pixels a line integral convolution has to be calculated on the streamline where this pixel is included.

While the rendering shows the streamlines of the vector field, it doesn’t really show the direction of the vectors themselves. I worked on another technique based on the paper Animating Flowfields: Rendering of Oriented Line Integral Convolution that uses a filter to create oriented line integral convolutions. This is combined with other type of textures (not white noise) and creates something like this:

Both techniques use the CPU and due to their complexity they are pretty slow to render. You can find my 2D Canvas implementations here and here.

While IMO these techniques don’t make much sense on the CPU (a few particles and a blend operation would suffice to create the OLIC effect), there are really interesting variations of these techniques that can be used on the GPU and allow for many more interesting effects and animations.

LICs on the GPU

While looking for more techniques to render LICs I came across this mind blowing paper called Lagrangian-Eulerian Advection of Noise and Dye Textures for Unsteady Flow Visualization.

Instead of calculating for each pixel the line integral convolution, this technique takes the background image (white noise image) and advects it with some control points. This advected image is then blended with the original image to create a third image that has a very similar result to the techniques mentioned above.

There are many considerations to take into account when implementing LICs via texture advection like coordinate and noise initialization, coordinate integration, noise advection, edge treatment, noise injection, coordinate reinitialization, noise blending, post-processing, etc. but the algorithm is performed in the GPU and thus is much much faster than the previous ones. I created a small example of an animated rendering of the same vector field with PhiloGL that you can find here. The demo uses WebGL and floating point textures. I add a video below in case your browser doesn’t support this:

Another good thing is that I was finally able to simulate the magnetic field experiment I mentioned at the beginning of this article. You can find the example here. I also add the video in case your browser doesn’t support the features:

Fluid simulation

The paper also describes a technique to do dye texture advection. This is pretty neat to simulate fluids. I created a cool demo showing this. You can find the demo here. I also add a video of the demo below:

All three papers have been implemented in 2D Canvas and WebGL and you can find the code here. These techniques are pretty neat to render and animate vector fields. I hope you find them useful!

Euro 2012 Streamgraph B-sides

2012-07-02T00:00:00+00:00

I worked on a data visualization for the 2012 European football champioship at Twitter that you can find here.

The visualization shows day-by-day data aggregated by country but also shows tweets per minute for each match of the tournament.

I made some tweaks to the JavaScript InfoVis Toolkit to create the StreamGraph visualization. During the process I made some mistakes that ended up in curious pseudo-visualizations like this ScribbleGraph:

Or like this ScatteredFeatherGraph:

Or even this Windows solitaire tribute graph:

I’m thinking on writing a paper about these (not).

Next Talks

2012-04-16T00:00:00+00:00

I’ll be giving a couple of talks related to data visualization, JavaScript and Web standards in the next couple of weeks:

Apr. 20 - Tech Talk on Data Visualization and Web Standards at Stanford ACM. It’s free and open to the public so if you’re around please come and say hi!
May 19 - I’ll be at JSConf Argentina with a more WebGL / JavaScript focused talk. I’m really excited to be presenting in my country again, last time I presented it was at Microsoft for BarCamp BA like six years ago!

I hope to see you there!

Wind Motion Patterns

2012-02-27T00:00:00+00:00

I’ve been working on a WebGL data visualization of wind motion patterns for a 72 hour period in the United States.

Gathering the data from the National Weather Service was pretty interesting, I didn’t know USA had so many weather stations! The visualization shows wind direction encoded in line angles, wind speed encoded in line lengths and disk radius, and temperature encoded in hue. All this for about 1200 weather stations across the country.

You can switch between different visual markers from the top menu, also play the wind motion for the 72 hours or select a specific time from the timeline below the graphic.

One interesting thing I found with this visualization is what appeared to be an outlier in the data, around mount Washington:

The data is seems actually correct after I took a look at the Wikipedia article for Mount Washington:

For 76 years, a weather observatory on the summit held the record for the highest wind gust directly measured at the Earth’s surface, 231 mph (372 km/h) (or 103 m/s)

With this example I also released a new version of the framework.

WebGL Camp

2011-12-09T00:00:00+00:00

I generally don’t do write-ups of events I go to (now that I actually check my blog I’ve never made one) but I had such a great time at WebGL Camp today and learnt so many new things that I think that (especially for the sake of my bad long term memory) I’ll do a write-up about my experience at WebGL Camp #4 at Mozilla HQs.

This is not going to be a chronological review but more like stuff I learnt about state-of-the-art WebGL, and the incredible community that we have here in the Bay Area around this technology.

On the things I learnt there’s that WebGLCamp this year gathered people that make great products and apps using WebGL; it also gathers people who like to write great WebGL frameworks either for games, art or data visualization and it also gathers people who define and participate directly on the making of the actual WebGL API. All of these people are together in the same place, discussing about these three things, and since this is a small group of very knowledgeable people, the outcome of this event is extremely valuable for the people there.

Just take a look at the agenda. In one day you will see how a profitable business like My Robot Nation uses WebGL as a core component of their product, you will also find out the technical aspects of implementing MapsGL in one of the most visited applications on the Web; you will also hear about how WebGL is an integral part for modeling and simulation in medical training applications and finally how the Department of Defense uses WebGL in their real time graphic simulation applications. Oh and yes, Autodesk also showcased their new WebGL applications.

So much changed since last year at WebGL Camp. While WebGL was still “alpha” and people built ideas on top of the thought that in the future it’ll become a solid and robust foundation, WebGL is now massively used in products/businesses that span to any possible field you could imagine!

Another very interesting aspect on this is that not only cool ideas around businesses show up, there’s a lot of discussion on technical stuff, on three different levels I would say: the standards body, the framework API designer/developer group and finally the consumers of these frameworks/APIs that create compelling demos, products and apps.

Just to name a few frameworks, Paul Lewis presented about A3 and Bartek Drozdz presented about J3D. Oh, and the Department of Defense said they were using GLGE for their applications. There was also a very interesting presentation of an Audio/Kinect/WebGL mashup that used CubicVR.js.

GLSLUnit was also presented by Google devs and a very interesting idea of using it to feature detect that a visitor to your site has an adecuate graphics card to run your WebGL application was suggested.

Another very interesting idea from the J3D framework was to design a special templating language that would render to proper glsl code to reuse shaders properly. Although J3D’s glsl language is pretty simple (only has a few special annotations), I would love to see something more like the Django templates in which inheritance, filters and include statements could be added in an easy way to reuse shaders.

Also, I really enjoyed Brandon Jones talk on texture (up and down)loading performance.

But what about me? I presented about PhiloGL last year at Google HQ’s for WebGLCamp #3, so this time it was all about meeting the people and not having any pressure on presenting anything :)

I also met today a couple of developers for OLPC that have been using and contributing code back to V8-GL for OpenGL/WebGL integration in the OLPC project. Can you believe it? V8-GL helps OLPC devs!

Finally there’s also the people who will talk to you about the future of the APIs you’re using. People working on the standard, who will actually listen to your experiences using the WebGL API and probably at the next iteration of WebGL Camp they will present some standard improvement that will include at least part of your ideas in it. And I’m not lying about this. You can witness that in the WebGLCamp #3 video (right towards the end) I was complaining about the fact that simple JSON was not enough as an interprocess/worker format, that it would be nice to be able to move typed arrays from one place to the other. This change was being added as I spoke and this year happened the exact same thing again. I talked two minutes to Ken Russell from Google about the structured cloning algorithm used for worker communication and how typed arrays still had to be cloned from here to there and he told me that transferable objects were just added to Chrome. Yay!

I really enjoyed this event and hope to see the community again for the next WebGL Camp. I had a great time and I think that there aren’t so many ocasions in which you can gather a group of people like this that have so many different backgrounds and ideas on such different topics related to WebGL like standards, products and frameworks.

A big props to Mozilla and Henrik Bennetsen from Katalabs for organising the event! Until the next WebGL Camp!

PhiloGL 1.4.0

2011-10-14T00:00:00+00:00

I’m very excited to announce that PhiloGL version 1.4.0 has been released!

This release focuses on two main things: a better picking algorithm that caches scene rendering and a simple image post-processing chain to add some nice fragment shader effects when rendering a scene.

Both these features are implemented in a revamped World Flights example I just released, so let’s take a look at how these things are implemented in the example.

World Flights (revisited)

World Flights is a data visualization that shows flying routes information for various airlines around the world. For 200 airlines, more than 57.000 flights and 6.000 destinations are displayed. You can find more information on the example here.

You can access the example here.

Image post processing

There’s a nice neon effect in the airline routes as well as in the cities displayed in this example. This is done by rendering two different scenes; one with a “black” planet:

…and another one with the current texture for the planet:

Here’s the code that renders them to a world and world2 FBOs and their associated world-texture and world2-texture textures:

app.setFrameBuffer('world', true);
  program.earth.use();
  gl.clear(clearOpt);
  gl.viewport(0, 0, 1024, 1024);
  //do not use a texture but paint it black
  program.earth.setUniform('renderType',  0);
  scene.renderToTexture('world');
  app.setFrameBuffer('world', false);

  app.setFrameBuffer('world2', true);
  program.earth.use();
  gl.clear(clearOpt);
  gl.viewport(0, 0, 1024, 1024);
  //use a regular planet texture
  program.earth.setUniform('renderType',  -1);
  scene.renderToTexture('world2');
  app.setFrameBuffer('world2', false);

These two textures are then combined to generate the screen image:

Media.Image.postProcess({
    fromTexture: ['world-texture', 'world2-texture'],
    toScreen: true,
    program: 'glow',
    width: 1024,
    height: 1024
  });

The glow fragment shader will take these two generated textures, create a bloom filter for the one with the black planet, and add it to the one that has the regular textured planet and provide a result for that:

#ifdef GL_ES
precision highp float;
#endif

#define BLUR_LIMIT 4 
#define BLUR_LIMIT_SQ 64.0

varying vec2 vTexCoord1;

uniform bool hasTexture1;
uniform sampler2D sampler1;

uniform bool hasTexture2;
uniform sampler2D sampler2;

void main(void) {
  vec4 fragmentColor = vec4(0.0, 0.0, 0.0, 0.0);
  float dx;
  float dy;

  if (hasTexture1 && hasTexture2) {
    //Add glow
    for (int i = - BLUR_LIMIT; i < BLUR_LIMIT; i++) {
      dx = float(i) / 512.0;
      for (int j = - BLUR_LIMIT; j < BLUR_LIMIT; j++) {
        dy = float(j) / 512.0;
        fragmentColor += texture2D(sampler1, vec2(vTexCoord1.s + dx, vTexCoord1.t + dy)) / BLUR_LIMIT_SQ;
      }
    }
    //Add real image
    fragmentColor += texture2D(sampler2, vec2(vTexCoord1.s, vTexCoord1.t));
  }
  gl_FragColor = vec4(fragmentColor.rgb, 1.0);
}

This is not the best way to make a bloom filter since it’s O(N * N), but it could be easily transformed into a two-pass bloom filter with the postProcess chaining API:

//Make a horizontal pass
  Media.Image.postProcess({
    fromTexture: 'world2-texture',
    toFrameBuffer: 'bloom1',
    program: 'bloom',
    width: 1024,
    height: 1024,
    uniforms: {
      'orientation': 1
    }
  //Make a vertical pass on the result
  }).postProcess({
    fromTexture: 'bloom1-texture',
    toFrameBuffer: 'bloom2',
    program: 'bloom',
    width: 1024,
    height: 1024,
    uniforms: {
      'orientation': 0
    }
  //Combine the result to the original texture
  }).postProcess({
    fromTexture: ['world-texture', 'bloom2-texture'],
    toScreen: true,
    program: 'glow',
    width: 1024,
    height: 1024
  });

The result is:

Lazy Picking

The World Flights example displays around 6.000 cities served by the airlines. If you hover those cities a tooltip will display showing information about the city:

As mentioned in older posts, I implemented the color picking algorithm which assigns a different “picking color” to each city, then renders all 6000 cities to a texture with those picking colors and retrieves the color of the pixel pointed by the mouse. From that color it’s easy to identify which city has been hovered.

The current algorithm though renders this alternate scene each time the mouse moves over the canvas. A better approach would be to only take a snapshot of the scene when it’s updated in some way (the planet is dragged and dropped, a new airline is selected and the earth moves, etc.).

This can now be done by setting lazyPicking to true and then using a scene.resetPicking() method when the time is right to take a new snapshot. For all other cases a cache of the capture stored in a typed array will be used improving the performance of the algorithm:

events: {
  //enable picking and lazy picking
  picking: true,
  lazyPicking: true,
  //remove the center origin option for picking
  centerOrigin: false,
  onDragStart: function(e) {
    //do stuff...
  },
  onDragMove: function(e) {
    //do stuff to rotate the earth...
  },
  onDragEnd: function(e) {
    //reset the picking by removing the cached image capture
    this.scene.resetPicking();
  },
//more code here...

You can find more information on both these features in the Media and Scene APIs.

Thanks

Thanks to Luz Caballero for helping me out with the design of World Flights!

Upcoming Talks

2011-09-06T00:00:00+00:00

I thought I’d put my schedule for the upcoming talks here so I can convince whoever is reading this blog to come and meet me at some of these events! :)

I’ll be speaking at:

Data Visualization Group in Bay Area on New Tools for Visualization in JavaScript.
Visual.ly Data Visualization Meetup on The Ingredients of a Great Data Visualization.
Tech Talks at Mozilla on PhiloGL and its API.
SenchaCon 2011 in Austin, TX on WebGL fundamentals.

I hope to see you there!

PhiloGL 1.3.0 released

2011-08-07T00:00:00+00:00

Just a quick post to announce that PhiloGL version 1.3.0 has been released!

I’ve been working hard on this new release, which is mainly focused on many performance improvements and the ability to have reflection and refraction available in the default shaders.

Performance Improvements

Performance improvements happened on three different places of the framework:

Math functions

Math functions for Vector and Matrix operations are faster than ever. I changed the inner implementation of the math methods to use typed arrays whenever possible. The result is an amazing performance improvement that can be seen on the WebGL Matrix Benchmarks for Safari, Firefox and Chrome (these tests were made on a 13-inch MacBook Pro Core 2 Duo).

The API for the math module has changed a little bit. I explain most of the API changes in the documentation (look for new from version 1.3).

Deep typed array integration in O3D.Models

I changed how attributes where handled by models in the O3D module, integrating typed arrays very deep and reusing them whenever possible. In order to do this I created getters and setters for most of the model attributes like vertices, normals, color, etc. so you can feed to those properties either a typed array or a simple array of values and they will be converted internally into a typed array. The typed array will be reused whenever possible. When accessing some of these properties, you will always get a typed array too; even if you didn’t assign one to that property.

Micro Optimizations

Did you know that using forEach instead of a for-loop can be up to 50% slower in most browsers?. Some micro-optimizations take care of changing forEach calls into regular for-loops in the main rendering loop code.

Reflection / Refraction

I added the ability to have reflection and refraction in O3D. There are two new properties in O3D.Model: reflection and refraction which indicate the reflection and refraction index correspondingly.

I created two simple demos to demonstrate some of the new features in the framework.

The first one uses the marching cubes algorithm and web-workers as a divide and conquer technique to render reflective metaballs. I also use cube map textures and the new reflection index. You can access the example here.

The second example displays an interesting technique to render quaternion julia sets. Quaternion julia sets are four dimensional fractals that are created by using the same formula than the one used with two dimensional fractals, Zn+1 = Zn + C, but instead of using complex numbers as Z we use hyper-complex numbers called quaternions. Once we obtain the four-dimensional shape, what we do is project it into a three dimensional space to get “the shadow” of the four dimensional object. A very interesting article into how this works and also how this can be fully calculated via a fragment shader can be found here. I really enjoyed reading that article and recommend it to anyone interested in math :)

Anyway, you can access the example here.

SIGGRAPH 2011 WebGL BOF

Last but not least, I’ll be making a presentation on PhiloGL at SIGGRAPH 2011 in Vancouver on August 10th. If you’re there, come say hello!

PhiloGL 1.2.0 released

2011-06-06T00:00:00+00:00

I’m very excited to announce that PhiloGL version 1.2.0 has been released!

This release adds a lot of goodies to the Framework, like a much better design for multiple program management, a better design for picking enabling us to have finer grain picking, a brand new Quat class for quaternions and a couple of new primitives and improvements in O3D. All this with some updated documentation and a new demo!

Multiple program management

We always had the possibility to create multiple programs and use them in a PhiloGL application (see lesson 13), but in a more object-oriented model of applications we might want to assign programs to objects themselves than manually managing programs here and there.

You can define multiple programs right in the main PhiloGL constructor function as an array of program configurations with a specified id:

PhiloGL({
  program: [{
    id: 'program1',
    from: 'uris',
    vs: './vertex1.vs.glsl',
    fs: './fragment1.fs.glsl'
  }, {
    id: 'program2',
    from: 'uris',
    vs: './vertex2.vs.glsl',
    fs: './fragment2.fs.glsl'
  }]
  //other options here...
});

Then you can set a program id to any model you like by instanciating a model with:

var myObject = new O3D.Model({
  program: 'program1'
  //other options here...
});

and then each model will have a program assigned and used before being rendered. Easy!

Finer grain picking

The picking algorithm used in PhiloGL is a color picking algorithm. Each model is assigned a different color and the scene is rendered to a texture. Then, the pixel pointed by the mouse position is retrieved from the texture and the color of that pixel is used to identify the model.

Sometimes we want to know more than just which object has been picked. For example, we might want to know which face of that object has been picked. In that case the [O3D]((http://senchalabs.github.com/philogl/doc/o3d.html) constructor options pickingColors and pick are useful. By defining your own set of per vertex colors and a method that given a pixel returns special information on what part of the object has been retrieved, then it is possible to have finer grain picking. For more information about how to use this you can take a look at the World Flights example or go to the Google group of the framework.

More stuff

A [Quat]((http://senchalabs.github.com/philogl/doc/math.html#Quat) class has been added and also other interesting options in O3D.Model constructors have been included, like an attributes object to define buffers and attributes per model.

World Flights

World Flights is a data visualization powered by PhiloGL that displays the routes of all major airlines in the globe. Red markers indicate cities served by airports, and blue curves show the routes each airline makes and their destinations. You can drag and drop the globe, and also select new airlines from the controls on the right. The link to the application is here.

WebGL camp #3 at Google HQ

Last but not least, I’ll be making a presentation on PhiloGL at WebGL camp this Friday. Regirstration is still open, so I hope to see you there!

PhiloGL 1.1.1 and JavaScript InfoVis Toolkit 2.0.1 released

2011-05-05T00:00:00+00:00

Just a quick post to let you know about two patch releases for the JavaScript InfoVis Toolkit and PhiloGL respectively.

JavaScript InfoVis Toolkit 2.0.1

The new patch release for the JavaScript InfoVis Toolkit includes many bug fixes (that you can find here) and new configuration options for Area, Pie and Bar charts.

As I mentioned before I’m very excited that the project has been selected for the Google Summer of Code 2011 program, and that two student applications have been already accepted. The selected students will work on making the toolkit better by implementing new visualizations like Voronoi TreeMaps, Scatter Charts, Line Charts and other work related to code design and refactoring. Expect some interesting updates on the project in the next few months!

PhiloGL 1.1.1

PhiloGL version 1.1.1 has been released with a bunch of bug fixes and improvements. Multiple point lights (up to 50) are supported now. You can check a complete changelog here.

Our entry for Google's Data Visualization Challenge

2011-03-27T00:00:00+00:00

Launch the Visualization by clicking here! (browsers supported are the latest of Google Chrome, Firefox, Opera and Safari -not IE).

Luz Caballero and I have been working together for a couple of weeks to create an interesting visualization for Google’s Data Visualization Challenge.

Every year, Americans fill out income tax forms and make a payment to the IRS. Where does it all go? Using data provided by WhatWePayFor.com, we created a data visualization that will make it easier to understand how the government spends our tax money.

The Visualization

We want to help people grasp what these millions and billions mean… 43.03M on Federal aid to highways… is that too much or too little? The Federal Budget is traditionally divided in 20 “functions” that, under names like “National Defense” or “Energy,” show the purposes to which money is allocated. Our goal is to show not only how much money is spent for each function on a given year, but also

How does that compare to other functions? What are our priorities? Would cutting this give us money enough to fund that?
How has the amount spent on each function changed over the years? Hopefully this should give us all an idea on what has traditionally been considered normal highway spending ;)
Where does the money allocated to each function finally go? (Because we would never have guessed that we would find a USD 150M bailout under “Other advancement of commerce”!)

This visualization is made of two components. First, a tree layout shows, for each given year, how the budget (blue) is divided on each function (green), subfunction (yellow), and “account” (the most granular subdivision accounted for in the budget, which gives an idea of the use to which the money is put) (red):

The hegiht of each node/section in the tree is proportional to the amount of the budget spent on that function, subfunction or “account”. You can click on the nodes to navigate and filter functions and subfunctions. To go back you can right-click or click in the back arrow on the left of the visualization.

The second component of the application is a timeline that puts the amounts you are visualizing in the tree in a temporal context, showing the evolution of that particular spending category through time. Click on the bars representing the different years to navigate through time.

We wanted to allow our users to compare today’s spending to the 80s’ without artifacts caused by inflation, and changes in population and GDP. This is why our visualization shows inflation-adjusted USD, gives you the the option to see spending in USD, in USD per capita, or as a percentage of the time’s GDP. Plus, we wanted to give our users a good shot at pinpointing if/when spending changes are a consequence of government policy or of the economic climate. This is why you can overlay indicators like debt, GDP, inflation and yr/yr population change, to give you an idea of how they affect and are influenced by spending trends.

Interaction

What makes this application most interesting is the interaction between the components, which are tightly coupled.

When a function is selected, the timeline will adapt to show the budget spending trend for that particular function over the years.

When clicking on a timeline element, the tree will animate to show how the budget is spent on each function, subfunction and account, for the selected year.

If you are interested in comparing how the budget changed between two different moments in time, there is an option to transition between years using colors, red and green, to highlight a decrease or increase of total budget spending for each item.

Demo Video

We created a video that shows how to navigate and filter data and some interesting finds we made using our visualization. Click here to see the video in HD.

Acknowledgements

We would like to thank the people who gave us feedback on this visualization and helped us to improve it. Thanks for your help!

Click here to access the visualization!

Google Summer of Code and NASA

2011-03-22T00:00:00+00:00

The JavaScript InfoVis Toolkit selected for Google SoC 2011

The JavaScript InfoVis Toolkit has been accepted for the Google Summer of Code 2011!

An ideas page with some interesting tasks to complete for the project has been posted, and also we already got some awesome feedback from highly motivated students wanting to participate. The feedback has been great so far!

Being part of the Google Summer of Code 2011 is great not only for the JavaScript InfoVis Toolkit as a project, but also for our mentoring organization, Sencha Labs, to connect with students to promote Open Source activities and help build a new generation of JavaScript developers.

If you like interesting challenges with JavaScript and visualization don’t hesitate to contact me at nicolas at sencha dot com.

NASA

I have more exciting news: I’ve been invited by NASA to the first NASA Open Source Summit!

The summit will bring together 60 leaders from the open source community, NASA developers, and NASA policymakers to lay the foundation for improving how NASA develops, releases, and uses open source software.

I’m really looking forward to meet everyone there!

PhiloGL version 1.1.0

2011-03-11T00:00:00+00:00

PhiloGL version 1.1.0 has been released! This version includes a couple of new features like improved WebGL detection, linear fog, picking, added primitives like Cylinder and Cone and some bug fixes with directional lighting.

Better hasWebGL

Until now WebGL was currently detected via the existence of the WebGLRenderingContext object in the window object. This check is good to see if the browser we’re using provides access to the WebGL API, but doesn’t really check whether the hardware is capable to render WebGL - or at least create a WebGL rendering context.

In other words, we could have Firefox 4 installed on a ventilation system and get true for the check of !!window.WebGLRenderingContext.

In order to go a little further on the check I transformed hasWebGL into a static method that checks whether the WebGLRenderingContext exists but also checks whether the 3D context can be created. This is a noticeable step forward to check for WebGL capable machines. It’s worth a small API change (from static property to static method). You can check the documentation for this method here.

Picking

Picking is the ability to interact with 3D objects in the scene via events like mouseenter or click. Picking can be used by setting pick to true in the events configuration for the PhiloGL constructor. Internally the Event object uses the Scene pick method.

For example:

//Create App
    PhiloGL('surface-explorer-canvas', {
      events: {
        //enable picking
        picking: true,
        //now the model is returned as the target of
        //the event.
        onMouseEnter: function(e, model) {
          model.uniforms.colorUfm = [1, 1, 1, 1];
        },
        onMouseLeave: function(e, model) {
          model.uniforms.colorUfm = [0.5, 0.5, 0.5, 1];
        }
      },
      onError: function() {
        alert("There was an error while creating the WebGL application");
      },
      onLoad: function(app) {
        /* do stuff with app */
      }
    });

Fog

Linear fog was added in version 1.0.2, but only now it’s official. Fog can be selected as a scene effect in the configuration object. You can find more information about how to configure linear fog here.

Primitives

I added Cone and Cylinder as new primitives. You can find documentation about them here.

Mini Demo

I made a video featuring fog, picking and our new primitives. In this particular example I only set cones and spheres as pickable objects.

I hope you enjoy the new features :) You can download the library here.

The White House uses the JavaScript InfoVis Toolkit

2011-03-03T00:00:00+00:00

I was surprised when I entered the main White House website the other day to find out that this logo was showing a very familiar treemap.

Clicking on the logo took me to another page where this treemap was featured. The application / visualization showing the 2012 budget is made with a TreeMap, in JavaScript, using the JavaScript InfoVis Toolkit. Click on this link to go to the White House page that features the visualization.

PhiloGL

2011-02-22T00:00:00+00:00

For some time now I’ve been working on a new project built around WebGL that today I’m very proud to release: PhiloGL.

PhiloGL is a WebGL Framework for advanced Data Visualization, Creative Coding and Game Development.

PhiloGL uses cutting edge technology and JavaScript idioms and good practices to deliver elegantly coded WebGL applications that are focused on performance. PhiloGL also provides a rich module system covering Program and Shader management, IO, XHR, JSONP, Web Worker management, Effects and Tweening, and much more.

PhiloGL is Open Source Software, licensed under the MIT license and owned by the SenchaLabs foundation.

PhiloGL has a complete API documentation with a detailed description of all the modules and class methods. Another very cool thing is that all Learning WebGL lessons have been ported into PhiloGL code so that you can easily learn PhiloGL, WebGL or both at the same time. You can find a list of the ported Learning WebGL lessons here.

In order to show how PhiloGL applications are made I created three featured projects that deal with Data Visualization in different aspects. In order to see these examples you’ll need the latest Google Chrome and/or Firefox 4. If you don’t have a WebGL capable browser then you’ll see a video showcasing the application.

Visualizing Temperature Anomalies

Access the example here.

NASA collects year by year data about temperature changes around the globe. This information has been collected since 1880 and tracks temperature anomalies (changes) in different points of the earth as a 2D heatmap. By loading these images into textures and then mapping them into a 3D histogram we are able to track the temperature changes around the globe interactively. Smooth animations between date ranges enable us to spot the overall differences in temperature across the years. Can you guess what the temperature changes will be for the next decade? Access the example here.

Real time 3D color Histogram Analysis

Access the example here.

Color decomposition on RGB or other color schemes takes a 3D form. In order to show the color decomposition of an image each dot on a 3D color histogram is assigned a different diameter. The following example renders fine grain and precise color decomposition schemes of each frame of a video in real time. Notice how color varies on each take, and feel free to change the color scheme or pause the video to have a closer look at the color decomposition. You can interact with the visualization by using drag and drop and zooming in/out with your mouse wheel. Access the example here.

Explore 3D Parametric Surfaces

Access the example here.

Exploring 3D surfaces is a fun way to learn about geometry, physical phenomena or even terrain modeling. Parametric surfaces are surfaces that also change their shape with an extra variable: time. Explore 3D surfaces by using drag and drop and the mouse wheel to zoom in/out. You can enable the t parameter (by clicking on the checkbox) and hit play to see it move. There are three predefined surfaces for you to explore or modify. Access the example here.

Go get it!

I hope these examples show what you can do with the framework and interest you enough to explore the code. I’ve had a lot of fun building and using PhiloGL and I can’t wait to see what you’re capable of doing with it, you can download it here!

Our entry for the Mozilla Open Data Visualization Challenge

2010-12-14T00:00:00+00:00

Launch the Visualization by clicking here!

Maria Luz Caballero and me have been working together for a couple of weeks to create an interesting visualization for the Mozilla Open Data Visualization Challenge.

Mozilla is releasing a new version of Firefox, version 4. This new version introduces a new UI and they have been monitoring how users interact with the different UI components of Firefox 4. They gathered some interesting information about how UI components are accessed and other elements taken by a survey to the Firefox users describe their expertise level, age, place where Firefox is used, etc.

We decided that this information could be visualized as a TreeMap in which the area of each node is proportional to the number of times a UI component has been accessed. For example in this image we can see that around 20% of UI interactions happen in the Navigation Toolbar, and it’s possible to compare the area of this rectangle to others to give us a clue of the overall interactions:

Each node also has a color that spans from red to blue, indicating an average of the expertise level of the user interacting with this component.

It’s interesting to note how some components are more often accessed by expert users than by beginners and viceversa. However since we provide an average we lack the information about the real distribution of user skills accessing the component. In order to avoid misleading results, when drilling down the TreeMap PieCharts indicate the real user skill distribution for each UI component interaction.

What I believe is by far the most interesting aspect of this visualization is the fact of using the ordinal and nominal information provided by the user survey to provide dynamic filtering for the TreeMap. Dynamic filtering (or morphing) enables an easy comparison between different categories of users on how they interact with these components. The filter panel we provide is shown below:

If you’re interested in taking a look at the visualization and playing with it please go to our entry. Browsers supported are Firefox 4, Opera 10.6+ and the latest of the Webkit family.

Animating Isosurfaces with WebGL and Workers

2010-12-10T00:00:00+00:00

If you just want to take a look at the JavaScript demos with WebGL and Workers you can click here or just go towards the bottom of the page

For some time now I’ve been considering rendering isosurfaces with JavaScript. Isosurfaces are 3D surfaces that can be defined with implicit 3D equations.

While “regular” 3D surfaces are defined as:

z = f(x, y);

isosurfaces can also be represented as:

c = f(x, y, z);

Some shapes that can be represented as implicit equations and that can’t be rendered just as “regular” 3D surfaces include the Sphere. While with regular surfaces we can have either the top or bottom part of a Sphere:

z = f(x, y); // ->

  z = (+|-)Math.sqrt(1 - x * x - y * y); //either + or -

Implicit 3D equations can define the whole shape:

c = f(x, y, z); // ->

  1 = x * x + y * y + z * z;

Implicit 3D equations can be used to model organs after taking a computed tomography, or to model other physical phenomena (like fluids), or even to do old-style OpenGL animated demos like Metaballs :P

How do I render an Isosurface?

There’s this very interesting Marching Cubes algorithm that describes how to create primitives that can be later sent to an OpenGL renderer in order to plot Isosurfaces (the primitives are triangles in this case). The main principle of the algorithm is to create some sort of differentials by dividing the space in which the surface is defined into “very small” cubes within the main volume where the visualization is contained and study how the surface intersects these small cubes (when it does). We can identify in this way 256 different cases in which the surface can “cut” these small cubes:

For each of these cases we can obtain the group of triangles corresponding to the part of the surface that intersected the small cube. That way we can recreate the surface to be rendered.

Approaching the Problem with WebGL

The power of WebGL lies in being able to compute these algorithms in some shader language. In the WebGL programable pipeline we can write shader code at two different stages: when processing a vertex (vertex shader) or when doing per pixel manipulation (fragment shader). Unfortunately this means that we are forced to send to the vertex shader an array of predefined vertices to render the surface before computing anything on the GPU. There is no way to send any other type of information through the shader that once arrived to the vertex shader would allow it to “emit” new vertices.

For example, one approach I would have enjoyed doing is to send the grid details to the shaders and then calculate the triangles and emit new vertices to render the scene from the shader. However I haven’t found a way to do this with WebGL because there’s no geometry shader.

So we’re stuck on the client-side to calculate all vertices and then send the information to the vertex shader with WebGL. This proves to be a little bit slow.

Web Workers to the rescue!

The divide & conquer nature of the algorithm made me think that I could divide the computing space into an octree and parallelize each octant with a Worker. A worker is a JavaScript API that provides asynchronous threads of execution. One can post messages to the workers and add onmessage listeners to them to do something in particular when they finish the computation or they send us some message. Workers can call other workers and also one can create multiple workers out of the same code. For more information about Workers read the link above ;).

So now what I needed is something like a WorkerGroup that would create a certain number of workers and map different configurations for each of them, and then reduce all workers responses into a single “merged” result of vertices to send to the shaders.

A way to implement the WorkerGroup would be:

//Create an array of workers
function WorkerGroup(fileName, n) {
  var workers = this.workers = [];
  while (n--) {
    workers.push(new Worker(fileName));
  }
}

WorkerGroup.prototype = {
  //create a configuration array to post to each worker
  //when reduce is called
  map: function(callback) {
    var workers = this.workers;
    var configs = this.configs = [];

    for (var i = 0, l = workers.length; i < l; i++) {
      configs.push(callback(i));
    }
  },
  //run all workers with their proper configuration and
  //define a group-callback to merge the results
  reduce: function(opt) {
    var fn = opt.reduceFn,
        workers = this.workers,
        configs = this.configs,
        l = workers.length,
        acum = opt.initialValue,
        message = function (e) {
         l--;
         if (acum === undefined) {
          acum = e.data;
         } else {
          acum = fn(acum, e.data);
         }
         if (l == 0) {
          opt.onComplete(acum);
         }
        };
    for (var i = 0, ln = l; i < ln; i++) {
      var w = workers[i];
      w.onmessage = message;
      w.postMessage(configs[i]);
    }
  }
};

The performance improvements of using Workers are very noticeable. The algorithm remains on the client-side though, so if you want fast performance I’d recommend you to use some webkit nightly.

The Result

I made two demos, one is a Metaballs demo and the other one is an Isosurface visualizer. I really like the second example because you can specify your own formulas. There are predefined formulas like a Torus or Gyroid. To see these demos you’ll need a WebGL enabled browser, I do encourage you to install a webkit nightly for this.

Click here to take a look at the Metaballs demo, or you can watch a video here:

Marching Cubes + WebGL + Workers = Fun from Nicolas on Vimeo.

For the Isosurface generator you can click here or take a look at the video below:

Awesome Equations from Nicolas on Vimeo.

GitHub

I’m pushing all experiments from now on to a Playground repo for you to check the code or improve. There’s also some 2D Canvas, WebGL and Scala code from Three ways to make 3D there.

My Talk at YOW! Developer Conferences

2010-12-09T00:00:00+00:00

Last week I was in Melbourne and Brisbane, Australia, to speak about interactive data visualizations with JavaScript at the YOW! Developer Conferences. My presentation was a one hour talk in which I covered some InfoVis Theory basics (some ideas from Jacques Bertin’s Semiology of Graphics); also I spoke about the main things the JavaScript InfoVis Toolkit has to offer and finally I presented some future work involving WebGL, web workers and some cool algorithms I’ll be presenting in a future post.

I want to thank Dave Thomas and the entire team for giving me the opportunity to speak in Brisbane and Melbourne. I had the honour to speak right before Ben Fry’s Computational Design talk and Dan Ingalls 40 years of fun with computers - both amazing presentations.

You can find the slides for my presentation here, the audience went wild with my presentation in Brisbane!

Creating Interactive Data Visualizations for the Web - YOW! Developer Conferences Australia

View more presentations from philogb.

One of the highlights of YOW! for me was to see Dan Ingalls, father of five generations of Smalltalk, winner of two ACM awards and now working with cutting-edge web technology as SAP fellow introduce bindings for the JavaScript InfoVis Toolkit in the lively-kernel:

Being at YOW! Developer Conferences was a great experience.

Sencha Acquires the JavaScript InfoVis Toolkit

2010-11-06T00:00:00+00:00

Sencha Inc., a company that focuses on client-side technologies acquired this Friday the JavaScript InfoVis Toolkit!

What does it mean?

Sencha now owns the toolkit. Following the Sencha Labs philosophy, the project will remain with an Open Source license (BSD), and will be part of the Sencha Labs foundation. This means that someone from Sencha will take some time to maintain and still develop the toolkit from now on. That person can be me, but I also encourage the Sencha Team to take a closer look at the project, there are many cool concepts in there that even trascend the fact that it’s done in JavaScript, and I would love to get some help on the development of the toolkit. Oh, and yes I’m looking at you too reader, the project has been up for more than two years and you still haven’t contributed? Now is the time!

What does it mean… for me?

I started this project all by myself three years ago. I still remember I was playing around with Hyperbolic Geometry when the experiment was featured for the first time in Ajaxian. My intention at first was far from building an Open Source project or gathering a small community of people interested in it, it was just about learning Data Visualization. After taking a look at Tamara Munzner’s Google Tech Talk, I was amazed by this field and tried to learn all I could about state of the art infovis.

Then I moved to France to work for Exalead. I was primarily working with the Java stack on enterprise products (not related to infovis). Then I would go home and spend more time learning infovis and putting into practice the things I learnt by hacking.

One of the things I found interesting about infovis papers is that they generally provide an overview of some visualization technique, they provide a user study to show how the visual encodings presented are better in some situations, but there is little concrete talk about the implementation itself. Of course, the papers are about Data Visualization, not Computer Graphics. We’re not supposed to be reading pseudo-code, but instead how a visualization provides new insights on data. That “lack of information” about a concrete implementation was very interesting to me at first: each visualization became a technical challenge. With the time I got to learn something very interesting though. The real challenge lies in the creativity required to invent a visualization that actually helps the user and increases his/her overall productivity when browsing some dataset.

At first the toolkit was… WYSIWYG: I was just experimenting with infovis and since I was building everything from scratch design errors showed up. There were some early adopters though, like OpenCRX that included the toolkit as part of their mobile application. With time the toolkit got much better, and people from Google, Mozilla, The Guardian, CERN, UBS, Platform Computing and more started using it.

Along with Protovis, I believe that the JavaScript InfoVis Toolkit became the most robust solution to provide Web based interactive Data Visualizations.

Thanks to the toolkit I got the chance to meet the InfoVis community in Paris by speaking at the Parisian Seminar on InfoVis and HCI, and to present the toolkit at JSConf in Berlin. I also began working on building visualizations at Exalead Labs, and the toolkit was also one of the reasons I moved from France to California to work for Sencha. Also I’ll be presenting the toolkit at the YOW! Developer Conferences in Australia this year.

Within this three years I also learnt how to create, license, develop and maintain an Open Source project and its community by hosting the project, accepting collaborations through GitHub, answering questions at the Google Group, etc. This might seem silly to some of you, but maintaining an Open Source project requires some hard work and dedication.

The project has reached a maturity I’m very proud of. And there are still much things to work on. Exploring new visualization techniques and robust WebGL support are just a few things that come to my mind.

Analyzing JavaScript Class Patterns Performance

2010-10-10T00:00:00+00:00

For a while now I’ve been considering different JavaScript class patterns and their performance when initializing and calling methods from them. Depending on the use-case some class patterns can show considerable performance improvements over the typical pseudo-classical patterns. In some cases, the use of closures and private member initializers in function constructors, as opposed to what’s stated by the MDC, can also show performance improvements over classical approaches.

JavaScript class performance can be analyzed by altering two variables: the JavaScript class can have public/private properties and/or public/private methods. On the other hand, code performance can be measured on class definition, class constructors, or instance methods.

I’ll just state the conclusion before making the analysis: there’s not a unique class pattern that will make your code faster in all cases. Most of the time it just depends on your use-case. In most cases people try to delegate all computations at class definition, since this generally happens only once. When this is not possible there’s a tradeoff between having computations in the class constructor or performing these computations when a method is called.

The premise for this analysis is that the extra-computations can’t be avoided, and their complexity can’t be simplified. In some sense, they’re just like a black box.

In the next section I’ll show how you can use closures, variable scope and a class prototype to delegate computations at class definition, class constructor or to just keep the computations when calling a method.

Class.prototype 101

Any JavaScript object, (String, Object, RegExp, etc), generally has two type of properties: the own properties and the properties found in the prototype chain. Each object has a special property proto that actually points to another object (or is null).

When we try to access a property in an object:

var propertyValue = object.myProperty;

it will first try to see if it’s an own property, something that can be checked by doing:

Object.prototype.hasOwnProperty.call(object, 'myProperty');

and use that property if it exists. If it doesn’t exist then it will try to find that property inside the object pointed by the proto property:

object.__proto__.myProperty;

If the property isn’t found in the object pointed by proto then it will try to find it recursively in the prototype chain (object.proto.proto, etc) until it gets to the top of it.

When we create a Class with a prototype:

function Person() {
    //constructor
 }

 Person.prototype.sayHi = function() {
    alert("hi");
 };

 var p = new Person();
 p.sayHi(); //"hi"

Internally JavaScript will assign to any Person instance proto property the Person.prototype object we defined above. This means that we actually define an object with methods only once, and any instance of Person will have a proto property pointing to those methods. Since all Person instances point to the same Person.prototype people usually say they share those methods, and so they’re shared properties across instances.

var p = new Person();
p.__proto__ === Person.prototype //true
p.sayHi === p.__proto__.sayHi //true

The convenience about this is that we don’t have to execute an entire code that actually re-creates methods and assigns them to a newly created object in the function constructor. Since each time new Person is called the Person function is executed, it would be better to just define the methods in the prototype object for once and not execute code for augmenting an object with new methods each time we call Person.

Let’s see how this could be used to increase the performance of certain class patterns.

Public Properties

In the public properties class definition we just move all properties definitions from the class constructor to the prototype object. By moving these computations we can see that there are some interesting performance improvements on constructor calls.

The code preparations are:

function ConstructorPublicProperties() {
  this.publicProperty1 = 1;
  this.publicProperty2 = 2;
  this.publicProperty3 = 3;
  this.publicProperty4 = 4;
  this.publicProperty5 = 5;
  this.publicProperty6 = 6;
  this.publicProperty7 = 7;
  this.publicProperty8 = 8;
  this.publicProperty9 = 9;
  this.publicProperty10 = (function() {
   var i = 100;
   while (i--);
   return 10;
  })();
}

function PrototypePublicProperties() {}

PrototypePublicProperties.prototype = {
  publicProperty1: 1,
  publicProperty2: 2,
  publicProperty3: 3,
  publicProperty4: 4,
  publicProperty5: 5,
  publicProperty6: 6,
  publicProperty7: 7,
  publicProperty8: 8,
  publicProperty9: 9,
  publicProperty10: (function() {
   var i = 100;
   while (i--);
   return 10;
  })()
};

The code we actually compare is:

for (var i = 0; i < 1000; i++) {
 new ConstructorPublicProperties();
}

vs.

for (var i = 0; i < 1000; i++) {
 new PrototypePublicProperties();
}

As you can see by the test case that you can run yourself moving computations to the prototype object can lead to considerable performance improvements in class constructors.

Public Methods

For public methods the case is just the same than with public properties. Moving all definitions and initializations to the prototype object leads to performance improvements in class constructors.

The preparation code for this is:

function ConstructorPublicMethods() {
  this.publicMethod1 = function(n) {
   return 2 * n;
  };

  this.publicMethod2 = function(n) {
   return 3 * n;
  };
 }

 function PrototypePublicMethods() {}

 PrototypePublicMethods.prototype = {
  publicMethod1: function(n) {
   return 2 * n;
  },

  publicMethod2: function(n) {
   return 3 * n;
  }
 };

The test code is:

for (var i = 0; i < 1000; i++) {
 new ConstructorPublicMethods();
}

vs.

for (var i = 0; i < 1000; i++) {
 new PrototypePublicMethods();
}

As you can see constructors are also faster when public methods are defined inside the prototype object.

Private Methods

Private methods can be defined in JavaScript by using the class constructor as encapsulation for other function definitions. This is a pattern I’ve seen done by Douglas Crockford and is used in many other places:

function ConstructorPrivateMethods() {
  function privateMethod1(n) {
   return 2 * n;
  }

  function privateMethod2(n) {
   return 3 * n;
  }

  this.publicMethod1 = function(n) {
   return privateMethod1(privateMethod2(n));
  };
 }

If you’re only defining private methods (and not private properties) you can also augment the prototype object with public methods and use encapsulation for defining private methods around the prototype object. I’ve seen this clever technique done by WebReflection some time ago:

function PrototypePrivateMethods() {}

 (function() {

  function privateMethod1(n) {
   return 2 * n;
  }

  function privateMethod2(n) {
   return 3 * n;
  }

  PrototypePrivateMethods.prototype = {
   publicMethod1: function(n) {
    return privateMethod1(privateMethod2(n));
   }
  };

 })();

The code tested is the same as in the previous sections:

for (var i = 0; i < 1000; i++) {
 new ConstructorPrivateMethods();
}

vs.

for (var i = 0; i < 1000; i++) {
 new PrototypePrivateMethods();
}

As you can see for yourself the prototype version has the fastest constructor. The performance improvements are considerable with this pattern which is interesting in use-cases such as when defining some helper functions to be used with a class.

Private Properties

The examples above are kind of simple to analyze: moving the computations from the class constructor to the prototype object can lead to faster constructors, and more generally can increase performance since the code is computed once at the class definition and not once per each constructor call.

With private properties there’s a tradeoff. Since private properties are generally accessed and/or modified by public methods, defining a private property inside the constructor forces the definition of the public method into the class constructor too, decreasing the overall performance of the class constructor. Another interesting technique could be to just use an underscore prefix for private members:

function ConstructorPrivateProperties() {
  var privateProperty1 = 1,
      privateProperty2 = 2,
      privateProperty3 = 3,
      privateProperty4 = 4,
      privateProperty5 = 5,
      privateProperty6 = 6,
      privateProperty7 = 7,
      privateProperty8 = 8,
      privateProperty9 = 9,
      privateProperty10 = 10;

  this.publicMethod1 = function() {
   return privateProperty1
        + privateProperty2
        + privateProperty3
        + privateProperty4
        + privateProperty5
        + privateProperty6
        + privateProperty7
        + privateProperty8
        + privateProperty9
        + privateProperty10;
  };
 }

 function PrototypePrivateProperties() {}

 PrototypePrivateProperties.prototype = {
  _privateProperty1: 1,
  _privateProperty2: 2,
  _privateProperty3: 3,
  _privateProperty4: 4,
  _privateProperty5: 5,
  _privateProperty6: 6,
  _privateProperty7: 7,
  _privateProperty8: 8,
  _privateProperty9: 9,
  _privateProperty10: 10,

  publicMethod1: function() {
   return this._privateProperty1
        + this._privateProperty2
        + this._privateProperty3
        + this._privateProperty4
        + this._privateProperty5
        + this._privateProperty6
        + this._privateProperty7
        + this._privateProperty8
        + this._privateProperty9
        + this._privateProperty10;
  }
 };

The code we’re testing is:

for (var i = 0; i < 1000; i++) {
 new ConstructorPrivateProperties();
}

vs.

for (var i = 0; i < 1000; i++) {
 new PrototypePrivateProperties();
}

As you can see for yourself using the prototype object yields faster constructors again.

NOTE: The results for the next section seem to have been reverted around Chrome 25. What this means is that now the execution of methods accessing properties in the prototype object is faster than methods using private variables in a closure. You can see this for yourself by testing this code:

var instance = new ConstructorPrivateProperties();

for (var i = 0; i < 1000; i++) {
 instance.publicMethod1();
}

vs.

var instance = new PrototypePrivateProperties();

for (var i = 0; i < 1000; i++) {
 instance.publicMethod1();
}

The augmented prototype object version is the fastest now. There’s an interesting jump in Chrome 25. You can follow the thread here to understand exactly how this works.

So are you targeting faster constructors, class definitions or methods? The fact is that any of these three stages can be optimized, but generally with a tradeoff.

JSConf

2010-09-28T00:00:00+00:00

JSConf just finished. It was a great experience: I got to meet a lot of interesting people and attend awesome talks, and I even got the chance to speak about the toolkit!

The slides from my talk are here:

JavaScript InfoVis Toolkit - Create interactive data visualizations for the web

View more presentations from philogb .

All talks were great, some of the talks I really enjoyed were:

…and more.

You can follow JSConf for updates about when the videos will be released.

Finally I want to thank janl, cramforce and hblank for gathering all javascripters in Berlin, making an excellent event, and giving me the chance to introduce the toolkit to the JS crowd. You guys are awesome!

Jekyll

2010-09-20T00:00:00+00:00

So I just moved from WordPress to Jekyll, a Ruby based static site generator. Moving from WordPress wasn’t as pleasant as I thought, and the site might have some “bugs” (let me know if you find one!) but I’m much more happy to be writing this post in my favorite editor, and not having to rely on WordPress, PHP or MySQL to run the blog.

While moving stuff from WordPress I found a few interesting old posts, I hope you find them interesting too:

…oh well, there are more interesting posts, so I recommend you to go to the main page to see a list of them.

If you find some bugs please let me know!

JavaScript InfoVis Toolkit + WebGL

2010-09-17T00:00:00+00:00