Jim Vallandingham

Animating Uncertainty

2018-05-28T00:00:00+00:00

Ages and ages ago as part of the Bocoup Data Visualization Team, I was fortunate to work on a project with Irene Ros and Peter Beshai where we were asked to prototype some experiments around visualizing uncertainty.

As the idea and importance of displaying uncertainty is not going away any time soon, I wanted to document my small contributions to this area with the hopes that it could spark inspiration for future (uncertain) work down the road.

Axis of Uncertainty

For any topic, it can be useful to define a loose taxonomy around it - to facilitate the exploration of its ‘space’. The space won’t be perfect, but it helps expand from existing work to possible future work (plus it makes you seem more of an expert!).

For uncertainty visualization, we started conjuring a space along two axes: the possible methods that could be used to visualize the uncertainty, and the data type of the “uncertain” data we are trying to visualize.

Visualization Methods

Now, visualization possibilities are what I typically think of first when I think about visualizing uncertainty. Nathan Yau has a great post on visualizing uncertainty where he outlines a number of different options, all of which (in my opinion) deal directly with encoding choices.

If I were to rename his headers a bit, the possible options I see him suggesting include:

Modification of glyphs (Obscurity)
Additional glyphs (error bars and other overlays)
Standalone visualizations (e.g. showing distributions)
Animation (Simulation)

These categories are a bit generic, and are probably not comprehensive, but provide one way to slice up the uncertainty visualization space.

Data Types

The other dimension we might split on is around different types of data that exist and thus can have uncertainty associated with them. Here I’m talking about statistical data types as opposed to data types of a particular programming language. Briefly, the main data types I like to think about are:

Categorical (sometimes called nominal)
Ordinal (where the values are discrete, but order matters)
Quantitative (aka continuous or real-valued)

To this basic list, we can add a few more types that I believe warrant separating:

Temporal (dates, hours, minutes - time is special!)
Spatial (or geographical, e.g. lat & lon values)

An Uncertain Taxonomy

With the possible values set for these two axes, we get a grid of possibilities to explore. And everyone knows that when there’s a grid, there’s a taxonomy! This table provides a tiny framework by which we can start to categorize existing uncertainty visualizations. It also provides a bit of inspiration for where one might explore to create novel uncertainty visuals.

This grid is super coarse in its capabilities to catalog. It’s my thought that each of the above “visualization methods” could be expanded further, either based on particular encoding options or perhaps by which visual metaphors are used, which would form a sort of taxonomy cube!

Our ultimate hope with our experimentation was to explore this entire cube, but for now I will just provide some examples from a tiny corner of the taxonomy.

(Also, I think there is a lot more to say about visual metaphors in data vis, but that is a topic for another time).

Uncertainty and Simulation

The area I really wanted to explore more was around how animation was used to visualize the notion of simulating possible outcomes from an uncertain future. As the reactions around the NYT Election Needle indicated, animation can be a very motivating when displaying uncertainty.

Fortunately, there is some prior work in this area. Jessica Hulman et al have developed HOP Plots which I see fitting very nicely in grid for using animation to visualize uncertainty around quantitative values. You can find a nice write up on the UW Interactive Data Lab Blog.

Here are some basic recreations of the HOP plot idea where I’ve added a stop/start button, but left out any scale or annotation.

A slow HOP plot:

A fast HOP plot:

As an exercise, we experimented with showing a history of simulations using onion skinning, which I deemed the “Onion HOP Plot”:

A slow Onion HOP plot:

A fast Onion HOP plot:

So, using this existing work as guidance, we started thinking about how animation could be used to display uncertainty for these other data types. I’m excited to start to fill in two additional little boxes in the grid for animation and categorical uncertainty and geographic uncertainty.

Morph Icons

Let’s start with the simpler of the two: using animation to visualize uncertainty around categorical data.

Consider some categorical data that can each be one of a few possible values. Say also that these possible values can be represented as icons. Here is our universe of possible iconic values:

The kicker is that for any particular data point, we aren’t 100% certain which value it should be classified as. And so for each data point we have probabilities associated with a subset of these categories.

The specifics of what this data is, is hopefully somewhat generic - and not all that important to the visualization method. Perhaps this is an automatic labelling of a set of images by a computer algorithm, or some other classification problem with a finite number of possible categories.

Our Morphing Icons idea is a pretty simple one: take those probabilities associated with each possible category and show each icon for a length of time based on that probability. So, if we are 90% certain an item is of type A and 10% it is of type B, then show type A’s icon 90% of the time, and type B’s 10%.

Here’s examples using 2 and 3 possible icons:

Morphing Icon with 2 types:

Morphing Icon with 3 types:

The morphing icon on the left is showing 50% A and 50% C probabilities. The one on the right is showing 70% A, 20% D, and 10% E.

Here are a few other examples, just for fun:

Of course there are all sorts of caveats and limitations to this approach to displaying categorical uncertainty. It breaks down if there are too many possible categorical values, or if there are no acceptable icon representations. It’s also unclear how long each loop of the animation should take.

Also, while this visual certainly uses animation, it is not really a simulation of the possible data. A real simulation would draw results based on the associated probabilities and thus flash back and forth between categories rapidly, based on which one was selected each iteration.

All that being said, I still enjoyed this representation. One could imagine a toggleable ‘detail’ view that showed the probabilities explicitly when needed.

Wandering Dots

The other prototype I wanted to show also uses animation to show uncertainty, but this time for spatial data.

Here we have a very similar problem. Say we have events that happen somewhere in a particular region of the world. We want to display them, but don’t know exactly where they happened. For whatever reason, we only know the location of each event at some resolution.

For example, the data might have a group of events as happening at a particular city, but another group we might only know the county or state they occured in. These resolution levels thus indicate the amount of uncertainty associated with each event location.

One of the approaches we experimented with tackling this type of uncertainty with animation was what we called Wandering Dots. The idea again, is nothing mind-blowing, and has plenty of drawbacks, but is still interesting (at least to me).

With Wandering Dots, each event is representing as a dot (genius!). Each dot is animated to indicate the resolution of the data. Dots wander far (have a larger radius) for events that we are more uncertain about.

Wandering Dots with a large radius:

Wandering Dots with a small radius:

The wandering is random within the constraints of their assigned radius. But they maintain a fluid trajectory, instead of hoping all over the place, to all the following of an individual event, if need be.

Wandering Dots with many points:

Wandering Dots with few points:

Wandering Hulls

Briefly, we also explored the idea of a using a solid plane to represent this uncertainty, called Wandering Hulls.

Wandering Hull with large radius:

Wandering Hull with small radius:

The idea was that we could perhaps use a choropleth approach with these hulls. However, the lack of access to individual points, as well as worries around occlusion, made us abandon this idea from further exploration.

Bounded Wandering Dots

Back with our dots, often times the level of resolution for an uncertain spatial data point corresponds with a particular geographic region - like a county or state.

Peter Beshai made a clever variant, the Bounded Wandering Dot which constrains the wandering to a specific shape.

Wandering Dots bounded by a square:

Wandering Dots bounded by a map region:

Pretty cool right?

An Uncomfortable Analysis

As part of our research, some of these prototypes were cleaned up and incorporated with other visuals into a dashboard-style tool for a small group of analysts to try out for a set of decision making activities.

Unfortunately we didn’t perform a controlled academic study around these representations, but we did get some initial impressions and feedback from these users.

One of the main pieces of feedback that has stuck with me is that the shifting visual representation made them feel a bit uncomfortable. But uncomfortable in a good way!

With more traditional uncertainty visualizations it can be easy to see the level of uncertainty, but ultimately ignore it, choosing the average over the error-bars, for example. While simpler aggregates are easy additions to these visuals, the animation provides perhaps a more visceral experience of the uncertainty that is harder to ignore.

Filling in the Cube

These quick examples are almost certainly not the “best” ways to represent uncertainty in these data types, but they are a start.

With more research in this area, I’m excited about the expansion and improvement to our informal taxonomy, as well as how other areas of the cube will be filled in with more ways to show the unknown. There are a lot of visual encodings and associated visual metaphors that can, and should be experimented with in this area. I’m looking forward to experimenting more in the future - and seeing how others tackle this uncertain issue!

An Introduction to Altair

2018-03-23T00:00:00+00:00

Recently I attended a wonderful talk by Jake VanderPlas (and hosted by the always astute Eli Bressert) about a wonderful package he has been working on to make visualization in python easier.

As a long time ggplot2 user, I’m quick to bemoan the challenges of succinctly visualizing data in python. While there are many visualization packages out there, none of them seem to approach the intuitiveness, completeness, and consistency found in R’s go-to tooling.

I’m happy to say that this new python package Altair might put an end to my grumblings (and yours) once and for all.

I’d like to take you through the basics of experimenting with Altair to show a bit of what it can do. None of my ramblings is really more than what you will find in the documentation, but I will describe some of my experiences in starting to learn how to use this tool.

What is Altair?

Altair provides a way to write declarative data visualization code in python by harnessing the power of Vega and Vega-Lite. Ok, so what is Vega?

Vega is a visualization grammer (think Grammar of Graphics the concepts that ggplot2 is built around) that can be written as a JSON specification. Vega-lite provides most of the power of Vega in a much more compact form by relying on smart defaults and simpler encodings.

Vega-lite provides a format to specify data, data encodings, and even interactions, all in a relatively simple and intuitive specification.

These packages and many more have been created out of the blood, sweat, and tears of the UW Interactive Data Lab. Arvind, Ham, Dom, Jeff, and all the rest: we can’t thank you enough for this vision and these amazing tools!

Anyway, back to Altair.

Altair provides a way to generate these Vega-lite specifications using an intuitive and pythonic API. So we can write python, but get out Vega-Lite. Intrigued? Then let’s get started!

JupyterLab Experimentation

All my experimentation has been done using the newly-ready-for-use JupyterLab, the successor to Jupyter Notebooks.

JupyterLab provides an updated and richer IDE-like experience for building interactive notebooks. While the new features, like an integrated text editor and terminal console, are interesting, the real exciting stuff is under the hood!

It has been built from the ground up using a plugin-based architecture providing consistent and powerful ways to integrate new features through different types of plugins. One type of plugin, Mime Renderer Extensions allows for custom Javascript-based views of different filetypes.

It is exactly this new feature that Altair leverages. It generates the Vega-Lite spec and passes it off to the built-in Vega renderer to be visualized.

Installation and Setup

These instructions are based on:

jupyterlab v0.31.8
altair v2.0.0rc1

And might need a bit of updating as Altair is developed. First, I created a new python 3 conda environment to work in. Then installed jupyterlab via pip:

pip install jupyterlab

To install the latest Altair, you need to supply the specific version at the moment:

pip install altair==2.0.0rc1

Finally, I had to add the vega3 JupyterLab extension, but this step won’t be necessary for future JupyterLab versions:

jupyter labextension install @jupyterlab/vega3-extension

Now open your lab with jupyter lab and start a new Python 3 notebook to get started!

Our First Altair

We can start with the ‘Hello World’ of Altair. To do this we first need to import the package.

import altair as alt

And then load some data. Handily, Altair provides some default datasets. We will start with the classic cars dataset.

cars = alt.load_dataset('cars')

Now let’s visualize this dataset.

alt.Chart(cars).mark_point().encode(
    x='Horsepower',
    y='Miles_per_Gallon',
    color='Origin',
)

Note: The result is a Vega-Lite specification that is rendered automatically inside of JupyterLab. To show the result in this post, I will use the handy-dandy Vega Embed to display the spec below.

Pretty impressive, right? Let’s break this code down a bit.

First, we are creating a Chart instance and passing it the data that will be visualized. Next, we indicate what kind of Mark to use in the visualization. For a dot plot, we want the ‘point’ mark, but as we will see, there is a suite of marks to choose from. Finally, each mark has a number of possible ‘channels’ to encode data with. We set these in the encoding() method, passing in attributes (columns) of our data to use to encode x, y, and color.

Again, if you are a ggplot2 user, you can start to see why I was so excited to try Altair! Marks are just another word for Geoms, and encodings look almost exactly like your typical aes() mapping! If you’re not a ggplot2 user, ignore all that - and just marvel at the intuitive reading of the Altair code!

Same Chart, Different Data

Now I know these test datasets are boring and dumb and boring and useless and boring. Robert Kosara is one of many voices I have heard in the recent past driving this point home.

If there is one rational to possibly hide my laziness behind, it is that when using a boring dataset for an introductory tutorial we can focus more on the tool used to visualize it, rather than spend time looking for insights in the data. A pitiful rational, I know, so before we move on, I wanted to show how easy it can be to load up your own data and visualize with Altair.

So we will use the wonderful FiveThirtyEight data repository to pick out a dataset about people’s favorite candy.

Here’s how you would load and visualize this data using pandas and Altair:

import pandas as pd

candy = pd.read_csv('candy-data.csv')

alt.Chart(candy).mark_point().encode(
    x='sugarpercent',
    y='winpercent',
    color='hard:N',
)

Nearly the same command, but completely different data (we will explain that :N in a second).

Most of the rest of this tutorial will be using the cars dataset because of my slovenliness, but keep in mind the ideas can be applied to any data.

Marks Marks Everywhere

This marks and encodings pattern starts to really shine when you want to visualize the same data differently, using different marks. Let’s make a bad visualization by replacing our points with bars:

alt.Chart(cars).mark_bar().encode(
    x=alt.value(0),
    x2='Horsepower',
    y='Miles_per_Gallon',
    color='Origin',
)

We can see the only things that have changed are the use of mark_bar() instead of mark_point(), and the additional x2 encoding. And with those small changes, we have a whole visualization!

Each chart can be further customized with constants provided at the mark or encoding level. For example, the opacity for each bar can be reduced by just passing in the opacity option to the mark_bar().

alt.Chart(cars).mark_bar(opacity=0.2).encode(
    x=alt.value(0),
    x2='Horsepower',
    y='Miles_per_Gallon',
    color='Origin',
)

Faceting

One feature I always look for in a new visualization tool or package is faceting based on a categorical variable. When dealing with matplotlib, this kind of visualization typically requires loops to create, which I feel takes you out of the ‘what’ you are trying to visualize and keeps you stuck in the ‘how’. Fortunately, Altair doesn’t require this cognitive break.

You can just use the facet() method!

alt.Chart(cars).mark_point().encode(
    x='Horsepower',
    y='Miles_per_Gallon',
    color='Origin',
).facet(column='Origin:N')

Notice the strange :Q trailing our data attribute. This is a special shorthand to indicate to Vega-Lite what type of data value Origin is. While you might think that the tool should be able to figure this out automatically, remember we are ultimately just building up a JSON specification - and so some of these nuances need to be supplied explicitly.

A Grammar of Interaction

Ok. Grammar of Graphics, big deal right? You could imagine any number of tools and specifications to implement these concepts, so what makes Vega, and thus Altair any different?

Recently, through some amazing work by the Vega team, Vega and Vega-lite now include a sort of “Grammar of Interaction”.

This means our specifications can encode not just our data visualizations, but what happens when people interact with our visualizations!

We can get a simple pan-and-zoom interaction on any chart in Altair by simply adding the interactive() method:

alt.Chart(cars).mark_point().encode(
    x='Horsepower',
    y='Miles_per_Gallon',
    color='Origin',
).facet(column='Origin:N').interactive()

But this is only the beginning! With Altair, we can build up much more complex interactions using selections. As the Vega-Lite documentation puts it:

They map user input (e.g., mouse moves and clicks, touch presses, etc.) into data queries, which can subsequently be used to drive conditional encoding rules, filter data points, or determine scale domains.

Altair comes with 3 basic selection types:

selection_single() - for interacting with a single element at a time.
selection_multi() - for selecting multiple items at once through clicking or mouseover.
selection_interval() - for selecting multiple items through a brushing interaction.

Here’s an example of using a selection_interval() to make a brush interaction that works across our previously faceted chart.

brush = alt.selection_interval()

alt.Chart(cars).mark_point().encode(
    x='Horsepower',
    y='Miles_per_Gallon',
    color=alt.condition(brush, 'Origin', alt.value('lightgray')),
).properties(
    selection=brush
).facet(column='Origin:N')

A few interesting pieces come out of this example. First, we can see we pass this brush into our chart as its selection property. Next, we use it as part of a condition - which allows us to use an if-else concept to encode the color channel. If the point is included in the current brush, it will be colored based on its Origin, else it will be lightgray. Try it out below by clicking and dragging the mouse.

We will play a bit more with selections later on in the tutorial.

Summarizing Your Data

Often times we are interested in visualizing an aggregation of the data, rather then just raw values. Altair comes equipped with a number of aggregation and binning functions that can be applied to specific encoding.

For example, we can use the count() aggregation to turn our faceted scatterplot into a histogram.

alt.Chart(cars).mark_bar().encode(
    alt.X('Horsepower', bin=True),
    y='count(*):Q',
    color='Origin',
).facet(column='Origin:N')

In this example we also see an alternative way to specify encodings. Using alt.X() we can provide options to the encoding, like the bin flag.

And, as you might expect, this default binning can be customized. Altair provides the BinParams object to manage binning specifics.

Here’s the same chart, but with 40 bins per facet.

alt.Chart(cars).mark_bar().encode(
    alt.X('Horsepower', bin=alt.BinParams(maxbins=40)),
    y='count(*):Q',
    color='Origin',
).facet(column='Origin:N')

Layering Marks

We will wrap up this tutorial with a look at how to combine charts to make much more sophisticated visualizations.

First, we can layer charts on top of one another to make Marks that are available by default.

Altair provides the layer() method to do this, as well as a shorthand version using the + operator.

Here is an example of using this capability to create a box-and-whisker plot (derived from one of the newly created examples). It uses layer() so that the data doesn’t have to be repeated over and over in the chart. Also note the use of a base chart that each piece of the box plot reuse.

def boxplot(data, x, y, ytype='Q', xtype='N'):
    # make all the aggregation strings up front
    min_agg = f'min({y}):{ytype}'
    max_agg = f'max({y}):{ytype}'
    median_agg = f'median({y}):{ytype}'
    q1_agg = f'q1({y}):{ytype}'
    q3_agg = f'q3({y}):{ytype}'
    x_val = f'{x}:{xtype}'

    # create a single base chart
    # which the other layers will
    # be augmented from
    base = alt.Chart().encode(
        x=x_val
    ).properties(
        width=400
    )

    # now we only need to specify what is unique
    # to each layer!
    whisker_low = base.mark_rule().encode(
        y=alt.Y(min_agg, axis=alt.Axis(title=y)),
        y2=q1_agg
    )

    box = base.mark_bar().encode(
        y=q1_agg,
        y2=q3_agg
    )

    midline = base.mark_tick(
        color='white',
    ).encode(
        y=median_agg,
        y2=median_agg
    )

    whisker_high = base.mark_rule().encode(
        y=max_agg,
        y2=q3_agg
    )

    # combine with layer()
    return alt.layer(whisker_low, box, whisker_high, midline, data=data)

# now we can use this function
boxplot(cars, x='Origin', y='Horsepower')

Pretty cool, right?

Layers and Interactions

We can use interaction with layered charts too!

Here is a quick attempt at an scatterplot that shows more details when hovering over each point. The text is a separate layer that is conditionally displayed.

We get to see selection_single() in action, which allows us to select one thing at a time. We can customize it to make it work on mouseover.

# the 'empty' setting makes all text hidden before any mouseover occurs.
pointer = alt.selection_single(on='mouseover', nearest=True, empty='none')

base = alt.Chart().encode(
    x='Miles_per_Gallon', y='Horsepower'
)

chart = alt.layer(
    base.mark_point().properties(selection=pointer).encode(color='Origin'),
    base.mark_text(dx=8, dy=3, align='left').encode(text=alt.condition(pointer, 'Name', alt.value(''))),
    data=cars
)

chart

As for real tooltips, the relevant parties are still discussing the details, but I’m sure something will be figured out soon.

Concatenating Charts

Another way charts can be combined in Altair is through concatenation. We can concatenate charts either vertically or horizontally.

Thealt.vconcat() method or the & operand is used to vertically concat.
Thealt.hconcat() method or the | operand is used to horizontally concat.

Let’s finish with an interactive dashboard made from two charts. Brushing on the top chart filters the bottom one. This chart uses Altair’s data transformation to filter the bottom chart based on the selection. We won’t discuss this feature more, but definitely worth exploring!

brush = alt.selection(type='interval')

# the top scatterplot
points = alt.Chart().mark_point().encode(
    x='Horsepower:Q',
    y='Miles_per_Gallon:Q',
    color=alt.condition(brush, 'Origin:N', alt.value('lightgray'))
).properties(
    selection=brush,
    width=800
)

# the bottom bar plot
bars = alt.Chart().mark_bar().encode(
    y='Origin:N',
    color='Origin:N',
    x='count(Origin):Q'
).transform_filter(
    brush.ref() # the filter transform uses the selection
                # to filter the input data to this chart
)

chart = alt.vconcat(points, bars, data=cars)
chart

Make sure to brush around on the top chart!

Limitations and Looking Forward

Hopefully this long drawn out demo of Altair at least gets people excited about the potential for this tool and the promise of declarative chart making in python.

That being said, like any great thing, there are some caveats. Here are a few:

The API is still pretty new. Some of this could break!
The documentation is still incomplete. Sometimes you need to look at Altair and Vega-Lite docs to find an answer.
The number of data points Altair can handle is currently pretty low. Right now it is capped at 5,000, but that is modifiable.
As noted above, tooltips are still on the way.
It would be cool to have the ability to extend Altair to make new Mark/encoding types. The boxplot function above would then be created inline as alt.mark_boxplot() or something similar.

But even with these slight short-comings, I am excited about this great new package, and can’t wait to try it on more interesting datasets in the future. I hope you are excited too!

Multivariate Map Collection

2017-12-10T00:00:00+00:00

There are many types of maps that are used to display data. Choropleths and Cartograms provide two great examples. I gave a talk, long long ago, about some of these map varieties.

Most of these more common map types focus on a particular variable that is displayed. But what if you have multiple variables that you would like to present on a map at the same time?

Here is my attempt to collect examples of multivariate maps I’ve found and organize them into a loose categorization. Follow along, or dive into the references, to spur on your own investigations and inspirations!

Before we begin, certainly you’ve heard by now that, even for geo-related data, a map is not always the right answer. With this collection, I am just trying to enumerate the various methods that have been attempted, without too much judgement as to whether it is a ‘good’ or ‘bad’ encoding. Ok? Ok!

3D

With the interactivity available to the modern map maker, it is not surprising that extending into the third dimension is a popular way to encode data.

Chicago Energy Map

Source: Datascope Analysis Chicago Energy Data Map

The above uses color and 3D height to encode natural gas and electric efficiencies of various neighborhoods in Chicago. It doesn’t provide freeform rotation, but does allow you to rotate to different cardinal directions, which helps with the occlusion. This tool also provides a detailed census block view of the data after clicking a neighborhood.

Median Household Income

Source: Visualizing America’s Middle Class Squeeze

Created by Max Galka, this map duel encodes median household income for various cities using both color and tract height. This makes it not strictly multivariate, but it uses the same ideas.

Choropleth Hillshade

Source: Choropleth Hillshade

This tool is provided as an ArcGIS add-on for creating hillshaded versions of choropleth maps. I don’t use ArcGIS, but its interesting to see a generic tool to create these kinds of maps.

Topographical Crime Maps

Source: Topographical Crime Maps

This was created by Doug McCune. It is not really multivariate, but I always really loved the style where he retains the basemap visual but uses hillshading to show geo-data in a very organic way. It seems like a technique that should have caught on more.

Data Mountains

Source: CartoDB

Taking the idea from exact shapes toward less precise icons are CartoDB’s Data Mountains. These maps use color and “mountain” size to encode multiple variables. The idea reminds me very much of geo-based Joyplots, like this great “Joymap” from Andrew Mollica showing the population density of Wisconsin:

Color

The idea of using color alone to represent multiple pieces of data may seem strange, but it can happen! Let’s take a look at a few examples

Bivariate Choropleth Maps

Source: Quartz: Where Medicaid Cuts Hit Hardest

This map of Trump voters vs Medicaid coverage is just one example of a somewhat popular technique. Joshua Stevens has a great article with everything you ever wanted to know about bivariate choropleths, so make sure you check that out.

Trivariate Choropleth Map

Source: Good: Reading, Writing, and Earning Money

If you thought two colors were hard, wait till you see three! This map generated a lot of musings back in 2011, so be sure to check out all the heat it garnered before trying to emulate it.

Trivariate Dot Map

Source: Stamen: Trees, Cabs & Crime

Originally created in 2009 by Shawn Allen while he was at Stamen, this artistic piece no doubt influenced the trivariate choropleth we just looked at. With the city-level data in the dot map, you can see more interesting patterns (if you are familiar with San Francisco).

Small Multiples

If one of the variables you are visualizing is categorical in nature, it is possible to show a multitude of maps, one for each category. That is what we find in the map below.

Source: Andrew Gelman: Estimates of support for School Vouchers

They also have a modified version with a different color scheme:

Source: Andrew Gelman

Embedded Charts and Symbols

Now we get to the interesting stuff! When you have multiple values to display specific locations on your map, why not layer in other chart types to display these values?

It seems in some ways obvious, but as we will see below, this just doesn’t always work out.

Circles

Source Carto: Madrid subway complaints by station

Source: Bill Scherkenbach

Pies

Source: A Map Analysis of US Airline Competition (found in a tweet from Tim Wallace)

Source: The Eddington Transport Study (pdf)

Radars

Source: Moral Statistics of France (pdf)

Lines

Source: Atlas of American Agriculture

Probably my favorite of the bunch, but that’s just cause I like old maps.

Histograms to View Oodles of Doodles from Google

2017-08-31T00:00:00+00:00

I was honored to be asked to talk at Uber’s Visualization Nights in Seattle, August 31st, 2017. Here’s the Meetup Link.

In May 2018, I also was fortunate to present a version of this talk at the Mini-Conference on Data Visualization and AI currated by Lynn Cherny in Lyon, France.

As you can guess by the amazingly clever title, it was a very serious topic. I used my recent How Long Does it Take to (Quick) Draw a Dog? visualization to talk about histograms and other fun aggregate views of data.

Slides

I used keynote to make the slides, and included a few animations / videos. Unfortunately, these don’t transfer well to the internet. But here’s a speaker deck to give you most of the content of the talk.

References and Links

Here is a list of references I used during the talk. Thanks to all the prior work and tools listed!

Quick Draw Links and Prior Art

Quick Draw Game

Quick Draw Data

Deborah Schmidt: Letter Collages

Forma Fluens

Quartz: How do you draw a circle?

Yannick Assogba: PCA Quick Draw Exploration

Histogram Examples

Flowing Data: How Histograms Work

Flowing Data: How to Read Histograms

FiveThirtyEight: Bob Ross Bar Chart

NYT Upshot: Chipotle Calories

Karl Pearson

NYT Upshot: Runners Time

The Pudding: Song Lyric Repetitiveness

NYT: Summers Getting Hotter

Pianogram

Via Velox

LA Times: Disneyland Wait Times

The Pudding: Free Willy by the Numbers

Amelia McNamara: What’s so Hard About Histograms

Dark Horse Analytics: Visualizing Distributions

James Cheshire: Population Joyplot

ggjoy

An Intro to regl for Data Visualization

2017-05-30T00:00:00+00:00

regl is a technology meant to simplify building awesome things with WebGL. Recently, Mikola Lysenko, one of the creators of regl, gave a great talk at OpenVis Conf that got me wanting to spend more time with WebGL and regl in particular - to see how it could be applied to my data visualization work.

With this tutorial, I hope to share my brief learnings on this wonderfully mystical technology and remove some of that magic. Specifically, how do we make the jump from the interesting but not so applicable intro triangle example:

To using regl for some sort of data visualization? At the end of this tutorial, hopefully you (and I) will find out!

We will start with the triangle. Try to understand WebGL and regl in the context of this example, then work to modify it to expand our understanding. We will cover some basics of WebGL and shaders, setting up your development environment to work with regl, and then explore a subset of regl’s functionality and capabilities.

The final result will be a data stream visualization:

This falls short of the amazing regl visualizations Peter Beshai has recently produced and discussed, but hopefully this tutorial can serve as a stepping stone towards this greatness.

And if you want to just skip to the results, I’ve made a bunch of blocks to illustrate the concepts covered:

What is WebGL?

First, it might be useful to step back a bit and talk at a high level about WebGL and regl and what they are good for. If you know all this, feel free to skip ahead.

As the wonderful Mozilla Developer Network so succinctly puts it, WebGL - or the Web Graphics Library - is a way to create interactive 2D and 3D visuals in a web browser with just Javascript. It is a low level API that utilizes the canvas tag, and can be used to make some pretty impressive stuff.

WebGL has a lot going for it. Its supported in many browsers, it can be crazy fast as the code is executed on the GPU, and there are lots of impressive examples to get you started.

But it is also a crazy confusing API for folks just getting started who are not familiar with these types of systems. WebGl is a graphics pipeline that includes a number of steps that are used to get an image rendered to the screen. The API is very low-level and it can take a lot of work just to get something on the screen.

What is regl?

The regl framework simplifies the task of developing WebGL code by providing a more functional interface. It takes cues from the React world, providing a consistent and familiar way to pass state and properties into the WebGL code.

So, you still have to write some WebGL, but the parts that you are responsible for writing are simpler, isolated, and have a defined signature.

Throwing Shade with Shaders

The WebGL code you still need to write are known as shaders. Shaders are functions written in a special C-like graphics language called GLSL or OpenGL Shading Language (OpenGL being the standard WebGL is based on).

There are different types of shaders, specifically two types:

Vertex shaders
Fragment shaders

Each type is responsible for configuring a different portion of the WebGL rendering pipeline. You need to implement both to make a complete

A Vertex shader is given each point (vertex) of the thing that is being rendered (triangle, sphere, rabbit, dataset) and its job is to determine the position of each of these vertices in WebGL space. If we ponder this idea from a data visualization / D3 perspective, this is kind of like implementing the most specific d3.scale ever. Each vertex is a data point, and the shader is a scale that maps each input point to a location in space and time for your specific visual.

A Fragment shader deals with the visual display of the things rendered in the WebGL pipeline to the screen. Specifically, they need to set the color for each pixel that is being displayed. (Why is it called a fragment shader and not a pixel shader? Good Question!).

As an aside, shaders are called shaders because fragment shaders are used to control lighting and other special effects when using GLSL in game development.

We won’t go into the details of GLSL in this tutorial, but hopefully the simple shaders we use aren’t too confusing. I’d suggest reading a bit of The Book of Shaders if you haven’t seen GLSL at all before - it provides a nice smooth introduction (though it focuses soley on fragment shaders).

As an aside, The Book of Shaders also has a great shader editor you should check out that includes all sorts of nice features. You can learn more about it here.

Here are some other resources that I shamelessly borrowed from, and might cover these concepts more elegantly than me:

Setting up an Exploration Environment

Before we put our pedals to the metals in implementing these concepts in our very own regl program, let’s take a moment to setup an environment that helps facilitate an exploration of these new technologies in a way that doesn’t incite us to throw our computers into the ocean out of frustration.

My solution to reducing frustration with new technologies is typically to supplement existing tools I like with new features. To this end, I’m going to suggest some Atom plugins to use that could make working with GLSL code easier for you.

But there are many other approaches! Feel free to ignore these recommendations and skip ahead, if you have a different methodology for WebGL development.

Also, I’ve included Blocks for each of the steps in the tutorial - which work without any additional setup - so if you don’t want to setup your coding environment now, you could just start forking those!

Atom Packages for GLSL Fun

Here are the Atom packages I would recommend using as we get started. Each can be installed via Atom’s “Preferences” menu.

First, grab the language-glsl package for some nice syntax highlighting of our GLSL code. Initially our GLSL code will be written inline as strings, but eventually we will write this code in .glsl files, so this package will come in handy then.

Next, you might be interested in the autocomplete-glsl which gives you handy autocompleting of GLSL types and functions. It even provides links to the documentation for each function!

Finally, I never leave home without a linter - and linter-glsl provides nice inline warnings and errors in your code so you can catch them early and (hopefully) avoid hours of glaring angrily at your screen just because you forgot a ‘.’ somewhere (it might still happen though!).

To get the linter working, you need to install glslangValidator - which if you are on a Mac and use homebrew you can do easily:

brew install glslang

A Baseline for good regling

Ok, after far too much yawning, let’s get to some code. Here we will add the necessary JS packages to our development environment - so again skip ahead if you are just working with the Blocks for now. Most of this section is a rehash of the lovely from nothing to something in WebGL blog post by Ricky Reusser - so feel free to use the original source.

First, let’s create a new Javascript project using npm init:

mkdir regl-start
cd regl-start
npm init

You can just hit enter to select the defaults for the project if you like - or tweak them as necessary. This command adds a package.json file to our new regl-start directory. We will use this file to manage the npm packages we will use.

Some Nice-to-have Packages

We will install a few packages to get things up and running quickly. The commands to run inside your project are:

npm install --save regl
npm install --save-dev browserify budo

Here’s a short description of what the development packages provide us:

browserify is a package that allows us to require other javascript modules in the browser (the require function comes from the node world and browserify brings it to the web).
budo is a development web server that works well with browserify and provides auto-reloading your browser every time you make a change to your code. Handy!

The --save-dev flag puts them in the devDependencies of the package.json file. These packages are needed only to develop and build the project.

Starting up with `start`

Next we need to create our Javascript starting point. Let’s name it index.js:

touch index.js

touch is a unix command that creates an empty file if no file exists. Inside our index.js, let’s write a simple test to ensure it is getting loaded (regl-less for now)

console.log("hello!");

Now we need to make browserify and budo work for us to build and serve this script.

Back in the package.json, we have a "scripts" section. Here we will remove the test script for now and add a start script that looks like this:

"scripts": {
  "start": "budo index.js --open --live"
},

This tells budo to browserify our index.js file and open a web browser to view it. The --live indicates to repeat this process any time the code changes.

Let’s run our start script from the command line using npm!

npm start

If everything went correctly, a new browser window should open and we should see our hello in the Console.

Hello Triangle!

Ok, so far that was just a how to get budo and browserify on your machine tutorial - which is fine but not what we promised. So let’s add some regl!

We will start by displaying a triangle which is the first example provided on the examples page - and serves as something like the ‘hello world’ of regl.

Let’s put this in our index.js file, and then go over the details:

const regl = require("regl")();

const drawTriangle = regl({
  // fragment shader
  frag: `
  precision mediump float;
  uniform vec4 color;
  void main () {
    gl_FragColor = color;
  }`,

  // vertex shader
  vert: `
  precision mediump float;
  attribute vec2 position;
  void main () {
    gl_Position = vec4(position, 0, 1);
  }`,

  // attributes
  attributes: {
    position: [[-1, 0], [0, -1], [1, 1]]
  },

  // uniforms
  uniforms: {
    color: [1, 0, 0, 1]
  },

  // vertex count
  count: 3
});

drawTriangle();

(Note the ` around the shader code. They are actually strings passed in to frag and vert).

This draws a triangle:

Block Link

Hurrah! But how?

Ok, first step is pretty simple. We require regl (remember we are using browserify - so we have that power). On the same line, we are calling the returned function which creates a full screen canvas element. This sets up our canvas and WebGLRenderingContext automatically.

Next we create a regl draw command by calling regl(). As the docs state, draw commands are the main point of the library. They allow for packaging shader code, as well as state and input parameters into a single reusable command.

Minimally, a regl draw command takes an object with a few parameters:

frag which specifies the fragment shader code (written in GLSL).
vert which specifies the vertex shader code (again in GLSL).
count which indicates the number of vertices to draw.

In this example, both shaders are written as one big string (we will see how to improve this setup later). This draw command also provides more parameters: attributes and uniforms which we will look at below.

Now let’s recap the purpose of these two shaders, and look a bit at how they work.

The vertex shader needs to accomplish its goal of positioning each vertex. It is called once for each vertex and needs to set the gl_Position global during each call.

The fragment shader needs to set the color for each fragment. It does this by setting the gl_FragColor global each time it is called.

No matter what else happens in these shaders, these two variables (gl_Position and gl_FragColor) are what need to be set.

Also note the general structure of a shader. You start with the declaration of variables and then use these variables in your shader’s main() function. The precision mediump float; line sets the precision of floating point values.

We run the draw command by calling it on the last line:

drawTriangle();

And a triangle is born!

We can see some of the interactions between regl and shader code - but not everything is immediately clear. For example, we see color listed in uniforms section of the regl command, and then we see uniform vec4 color specified and used in the fragment shader, but what is a uniform?

Let’s talk more about the different variable types in shaders, then come back to see how we work with these in regl.

The Many Shader Variable Types

In shader land, there are three types of variables you can declare. They are all confusing, but I like the explanation provided by html5rocks, so I’ll try to summarize here:

Uniforms are accessible in both the vertex and fragment shader. They are ‘uniform’ because they are constant during each frame being rendered. But (as we will see below), their value can change between frames.

Attributes are only provided to the vertex shader. There is a value for each vertex being displayed. The individual value is provided to the vertex shader when dealing with that specific vertex.

Varyings are variables declared in the vertex shader that can be shared with the fragment shader. We won’t use this in the rest of the tutorial, but its good to know they exist!

regl and Variables

So finally we are getting to what makes regl interesting and worth trying out. How it organizes what you pass in to the fragment shader functions.

Now that we know that uniform and attribute are variable types, I bet you can guess what the uniforms and attributes parameters of the draw command object are for, right? They allow us to specify the values of variables that are accessible in our shaders!

Let’s break it down a bit more.

Our uniforms parameter looks like this:

uniforms: {
  color: [1, 0, 0, 1]
}

This indicates that there will be a color uniform available to the vert and frag shaders. This color variable is a vec4 - a vector of length 4 - as seen in the declaration in the fragment shader, so it is declared here as an array with 4 values.

(Check out the Book of Shaders Color Chapter to learn more about how colors are defined in GLSL).

Our attributes parameter looks like this:

attributes: {
  position: [
    [-1, 0],
    [0, -1],
    [1, 1]
  ]
}

We can see we define one attribute, position, that is an array with 3 values. As we have our count parameter set to 3, the vertex shader code in vert will be run 3 times, each time its corresponding position attribute will be set to the correct value of the position array inside our attributes.

Note that the coordinate system is a bit different then what you might expect. The x and y scales go from -1.0 to 1.0 as shown in this handy diagram from MDN’s WebGL Concepts.

So thats how this triangle gets drawn. Kind of cool, right?

regl Inputs

But wait there’s more!

Currently in this example, the input uniforms and attributes are all static values. But it doesn’t have to be this way!

A big feature of regl is that you can supply inputs to regl through a number of different ways.

To see some of these benefits, let’s convert our triangle from a frame that is displayed just once, to a visual that is displaying over and over through time.

A Triangle In Time

Currently our triangle is rendered in one shot. It is displayed and then it is done. In order to see the benefits of dynamic data, we want to render this triangle over and over again so we can pass in different value each render.

Looping can be done in many ways, but regl provides regl.frame for just such a purpose. This allows us to call our drawTriangle draw command inside a requestAnimationFrame().

Let’s modify our code to use it!

Change our call to drawTriangle() to this:

regl.frame(function(context) {
  drawTriangle();
});

Now, our browser should reload, and we should see our triangle again. In fact, it should look exactly the same.

The difference is that drawTriangle() is now being called over and over again in a loop. We can verify this by console logging a value in context. Try tweaking the regl.frame call to this:

regl.frame(function(context) {
  console.log(context.tick);
  drawTriangle();
});

In the Console, you should see our tick count. And it should be increasing!

(I would remove that console.log after verifying that it is indeed looping - cause it clogs up the Console and slows things down).

The context variable is something that regl populates with a few values. Let’s return back to regl and talk about the ways it allows inputs to the draw commands.

regl Inputs: Context, Props, and this

So we have learned that a core feature of regl is handling inputs to our shaders. We know that shader variables come in 3 flavors: uniforms, attributes, and varyings. And while the triangle example we have been working with deals with static versions of these inputs, I indicated that dynamic inputs were also possible. So lets’ find out how!

regl allows for inputs to its draw commands to come from 3 possible sources:

Context: Context variables are state accessible between child draw commands (which we won’t look at here). It is also the place where regl keeps a number of variables you might find useful - including tick and time.

Props: Props are the most common way to pass in data to a regl draw command. You can pass them in as an input object when you call the draw command.

this: The this variable can be used to save state within a regl command. We won’t look more into this here - but something to keep in mind.

If you are familiar with React, as the regl documentation states, you can use this knowledge to better understand these input options.

regl’s Context is analogous to React’s Context
regl’s Props is similar to React’s Component Props.
regl’s this parallels a Component’s State in React.

But how do you use these regl inputs in your draw command?

Well it turns out that, similar to D3 attr, the values of uniforms and attributes properties in regl can be specified using functions! And these functions get access to the context and props of the draw command.

Pretty cool! Let’s try this out.

Flying Triangle

Let’s use our knowledge of the context.tick variable and base our vertex position attribute values on its current value. To do this, we will change position to use a callback function. This function is called inside regl to set the ultimate value of the attribute on each render. This callback function receives the context and props for the draw command (not unlike having access to the bound data for an element when setting an attr in D3).

Modify the attributes portion of your draw command to look like this:

attributes: {
  position: function(context, props) {
    return [
      [-1 * Math.cos(context.tick / 100), 0],
      [Math.sin(context.tick / 100), -1],
      [Math.sin(context.tick / 100), 1]
    ];
  }
},

And our triangle should start flying around on the screen!

Block

Its interesting to note that we made no changes to our shader code to make things dynamic. Instead, we simply put our draw command in a loop (using regl.frame) and then took advantage of regl’s input capabilities to make it dynamic.

Note that while I passed in props to the callback function, we didn’t actually use it - but I wanted to show that this object is also accessible. Let’s look at using this next!

Flying Triangle with Props

The regl documentation states that Props are the most common way to pass data into draw commands, but it doesn’t exactly say why.

With a bit of that React knowledge, and some sample code examination, we can see that a draw commands Props are passed in to it when calling the draw command function.

So, for example, if instead of using a static value for color we wanted to pass in a color Prop to our drawTriangle, we could do so by changing the call to the function like so:

drawTriangle({
  color: [0.208, 0.304, 1.0, 1.0]
});

(I used the Book of Shaders Editor to pick out a nice blue).

And then we can use this color prop in our uniforms:

uniforms: {
  color: function(context, props) {
    return props.color;
  }
},

Now your triangle should be moving, and blue!

Block

(Click on the Block link to see the animation).

If you like shortcuts, regl also provides a handy shorthand notation for accessing props: regl.prop(). Rewriting the above uniforms to use this shorthand would look like:

uniforms: {
  color: regl.prop('color')
},

From Triangles to Dots

Hopefully now we know a bit more about how shaders and regl fit together. We’ve learned about context and props which point to ways to get data into WebGL via regl to visualize.

Let’s move past the triangle and start using this wisdom on some dot data vis.

Displaying Points with regl

The first thing I want to cover is the minimum set of changes we need to convert the drawTriangle draw command into a drawDots draw command.

Here is the full code to get us started (which should look very similar to our triangle code for now). Below we will go over the changes.

const regl = require("regl")();

const drawDots = regl({
  frag: `
  precision mediump float;
  uniform vec4 color;
  void main () {
    gl_FragColor = color;
  }`,

  vert: `
  precision mediump float;
  attribute vec2 position;
  // @change acquire the pointWidth uniform
  //  this is set by the uniforms section below
  uniform float pointWidth;

  void main () {
    // @change Set gl_PointSize global to
    //  configure point size
    gl_PointSize = pointWidth;
    gl_Position = vec4(position, 0, 1);
  }`,

  attributes: {
    position: function(context, props) {
      // @change I tweaked the constants here so
      //  the dots are not off the screen
      return [
        [-1 * Math.cos(context.tick / 100), 0.2],
        [Math.sin(context.tick / 100), -0.8],
        [Math.sin(context.tick / 100), 0.8]
      ];
    }
  },

  uniforms: {
    color: function(context, props) {
      return props.color;
    },
    // @change: Add a pointWidth uniform -
    //  set by a prop
    pointWidth: regl.prop("pointWidth")
  },

  count: 3,
  // @change: Set our primitive to points
  primitive: "points"
});

regl.frame(function(context) {
  drawDots({
    color: [0.208, 0.304, 1.0, 1.0],
    // @change: Pass in the pointWidth prop
    pointWidth: 10.0
  });
});

Load this up and you should have 3 dots moving about the screen.

Block

(The points are hard to see - click on the Block link to see the animation).

So what changed?

The smallest but most important change is near the bottom of the drawDots draw command. We need to switch the primitive type to "points" in our regl setup:

primitive: "points";

This tells WebGL which primitive type to use in the display.

The other main change is that we need to specify a value for gl_PointSize in our vertex shader. We could hard code this value, but its more fun - and more flexible - to use our regl props knowledge to pass it in as a prop. Hopefully you can follow along with the @change comments to see this input setup in the code.

Dots Representing Data

OK! we are coming to the end of the road. We now have dynamic dots, but what we really want is data-driven dots, right?

So using our regl knowledge, we are going to pass in some dynamically generated data and get regl displaying it!

Pretend, if you will, we have a dynamic dataset that represents some collection of values over time. Maybe its car speeds on a length of road, or download speeds for a set of computers, or something more exotic. Probably not the most common of datasets - but what can you do.

Pretend some more that you might want to create an ambient streaming “data vis” to represent this data. Different nodes will move at different speeds and have varying sizes. I write data vis in quotes as this is still a bit of a stretch in terms of something useful - but at least its a start in mapping data to visual attributes with regl.

So, hopefully you have some inklings on how you might accomplish this, given what you know about regl so far. Essentially, we are going to take our drawDots draw command and expand it to use more then three vertices. In fact, our vertices will be controlled by data passed in as Props to the draw command.

Here is the final code block:

const regl = require("regl")();

// Helper function to create a random float between
// some defined range. This is used to create some
// fake data. In a real setting, you would probably
// use D3 to map data to display coordinates.
function randomFromInterval(min, max) {
  return Math.random() * (max - min + 1) + min;
}

// Helper function to create a random integer between
// some defined range. Again, you would want to use
// D3 for mapping real data to display coordinates.
function randomIntFromInterval(min, max) {
  return Math.floor(randomFromInterval(min, max));
}

// Some constants to use
var MAX_WIDTH = 2000;
var MAX_HEIGHT = 800;
var MAX_SPEED = 25;
var POINT_SIZE = 10;
var POINT_COUNT = 400;

// Helper function to generate some fake data.
// Each data point has an x and y and a 'speed'
// value that indicates how fast it travels
function createData(dataCount) {
  var data = [];
  for (var i = 0; i < dataCount; i++) {
    var datum = {
      id: i,
      speed: randomFromInterval(1, MAX_SPEED),
      y: randomIntFromInterval(POINT_SIZE, MAX_HEIGHT),
      x: 0,
      size: randomIntFromInterval(POINT_SIZE, POINT_SIZE * 3)
    };

    data.push(datum);
  }
  return data;
}

// Helper function, goes through each
// element in the fake data and updates
// its x position.
function updateData(data) {
  data.forEach(function(datum) {
    datum.x += datum.speed;
    // reset x if its gone past max width
    datum.x = datum.x > MAX_WIDTH ? 0 : datum.x;
  });
}

const drawDots = regl({
  frag: `
  precision mediump float;
  uniform vec4 color;
  void main () {
    gl_FragColor = color;
  }`,

  vert: `
  precision mediump float;
  attribute vec2 position;
  attribute float pointWidth;

  uniform float stageWidth;
  uniform float stageHeight;

  // helper function to transform from pixel space to normalized
  // device coordinates (NDC). In NDC (0,0) is the middle,
  // (-1, 1) is the top left and (1, -1) is the bottom right.
  // Stolen from Peter Beshai's great blog post:
  // http://peterbeshai.com/beautifully-animate-points-with-webgl-and-regl.html
  vec2 normalizeCoords(vec2 position) {
    // read in the positions into x and y vars
    float x = position[0];
    float y = position[1];

    return vec2(
      2.0 * ((x / stageWidth) - 0.5),
      // invert y to treat [0,0] as bottom left in pixel space
      -(2.0 * ((y / stageHeight) - 0.5)));
  }

  void main () {
    gl_PointSize = pointWidth;
    gl_Position = vec4(normalizeCoords(position), 0, 1);
  }`,

  attributes: {
    // There will be a position value for each point
    // we pass in
    position: function(context, props) {
      return props.points.map(function(point) {
        return [point.x, point.y];
      });
    },
    // Now pointWidth is an attribute, as each
    // point will have a different size.
    pointWidth: function(context, props) {
      return props.points.map(function(point) {
        return point.size;
      });
    }
  },

  uniforms: {
    color: function(context, props) {
      // just to be a bit strange, oscillate the color a bit.
      return [Math.cos(context.tick / 100), 0.304, 1.0, 1.0];
    },
    // FYI: there is a helper method for grabbing
    // values out of the context as well.
    // These uniforms are used in our fragment shader to
    // convert our x / y values to WebGL coordinate space.
    stageWidth: regl.context("drawingBufferWidth"),
    stageHeight: regl.context("drawingBufferHeight")
  },

  count: function(context, props) {
    // set the count based on the number of points we have
    return props.points.length;
  },
  primitive: "points"
});

var points = createData(POINT_COUNT);

regl.frame(function(context) {
  // Each loop, update the data
  updateData(points);

  // And draw it!
  drawDots({
    pointWidth: POINT_SIZE,
    points: points
  });
});

And this makes a data stream!

Block

Take a look and see if it makes any sense (and let me know if it doesn’t!). I’ve added some helper functions to generate fake data. Each datum is an object with x, y, speed, and size attributes.

The main difference between our previous three-dotted visual and this one is that we are passing in the points data as a Prop, and using this array to derive uniforms and attributes.

As a quick example, here is how the position attribute is now created:

position: function(context, props) {
  return props.points.map(function(point) {
    return [point.x, point.y]
  });
},

So, this makes an array of arrays, with each inner array containing the x and y value from our points. As before, the vertex shader will be called once for each point in the points data (thanks to our modification to count) and the position for that particular vertex will be set by its corresponding position entry.

The other trick in this code is that the vertex shader has a helper function to convert our data x and y values into WebGL coordinate space. This was stolen right out of Beshai’s tutorial, so you will have to read that to find out more.

You’ll notice we are still using the CPU (code outside of a shader) to handle positioning the data values. Wouldn’t it be great if we could do that on the GPU as well? Turn in next week when we attack this critical issue!

GLSLify Your Life

One last thing! So far we haven’t gotten a chance to use all those Atom packages or that linter I made you install. What gives?

Well, to simplify the generation of the blocks, I decided to keep all the GLSL shader code as strings - like you see in most of the regl examples.

But do you like coding complex C-like languages in strings?? Me neither.

So with one additional package, we can throw off the chains of shader strings and feel the warm embrace of better development tools. Trust me? Ok, let’s do it.

Return, if you will, to your command line and install the glslify package:

npm install --save-dev glslify

What is glslify? Two things:

A way to import glsl code from a .glsl file so you don’t have to write shaders as giant strings.
A mechanism that allows you to import glsl code that other people have written - so you can use them in your own shaders. This provides a sort of NPM for shader code (which is a pretty cool idea).

We will focus on the first of glslify’s abilities, but I hope to get to experiment with existing glsl packages soon.

In order to enable glslify in our development environment, we need to tell browserfiy to use it as a transform when processing our scripts. We can do this in our package.json by adding a new browserify section:

"browserify": {
  "transform": [
    "glslify"
  ]
}

Now we are ready to extract our GLSL code into their own files.

To do this you can create a sub-directory in your project (I called it shaders) and here create two files:

shaders/draw_dots.fs.glsl
shaders/draw_dots.vs.glsl

The fs will store our fragment shader, and the vs will store our vertex shader.

We can pretty much copy and paste over the shader code out from the strings in index.js and into these new files (don’t copy over the string ticks - just the code inside them).

If you installed all those packages I told you about, you should see syntax highlighting and linting in these files. Pretty cool right?

To import these shader files back into index.js, use glslify:

// import glslify at the top
var glslify = require('glslify');

// ...

// use glslify to import shader code!
regl({
  frag: glslify('./shaders/draw_dots.fs.glsl'),
  vert: glslify('./shaders/draw_dots.vs.glsl'),
  // ...
});

I’ve made a little starter repo that has this all setup in it, if you’d like to start with it for your explorations:

regl explorations repo

Go Forth you regl Beings

That’s all I have on my experiences with regl. If this piqued your interests and you want to go further - or if this was too boring an intro - be sure to check out Peter Beshai’s Beautiful regl Point Animations. Hopefully, though this tutorial does repeat a bit of his (and hey its good to repeat things sometimes), they work together to provide an intro and exploration of point visualizations in regl.

There is plenty more to explore! As always, be sure to share your regl experiences or questions in a regal manner in the comments or @vlandham on Twitter.

Bonus: Circle Points

Just in case your next comment is “great, but how do I make these points circles?”, I’ve got you covered!

Check out this block.

Block

A Data Driven Exploration of Kung Fu Films

2017-01-24T00:00:00+00:00

Recently, I’ve been a bit caught up in old Kung Fu movies. Shorting any technical explorations, I have instead been diving head-first into any and all Netflix accessible martial arts masterpieces from the 70’s and 80’s.

While I’ve definitely been enjoying the films, I realized recently that I had little context for the movies I was watching. I wondered if some films, like our latest favorite, Executioners from Shaolin, could be enjoyed even more, with better understanding of the context in which these films exist in the Kung Fu universe.

So, I began a data driven quest for truth and understanding (or at least a semi-interesting dataset to explore) of all Shaw Brothers Kung Fu movies ever made!

For those not dedicating some portion of their finite lives to these retro wonders, the Shaw Brothers Studio is the most famous (to me) Kung Fu film producer of all time. Their memorable title screen is almost always a part of my Kung Fu watching experience.

I figured this company’s entire martial arts collection would provide for a consistent and thorough look at the genre. Fortunately, after a bit of searching, I stumbled on what appears to be a comprehensive list of Shaw Brothers Films. I decided to pull down details for each of these movies from the amazingly useful Letterboxd movie-list-creation site to explore them in a data driven way to see what patterns could be discovered and what context I could learn from those patterns.

So here is a bit of data exploration fun. The analysis is in R, using tips and tricks from Hadley Wickham’s wonderful new Data Science in R book.

The full analysis code can be found in this R Notebook, which includes the code and graphs in an integrated format. And (spoilers!), the end Actor Collaboration Network and the rest of the code can be found on github.

Come for the Kung Fu, stay for the word embedding and interactive networks!

Shaw Brothers, Through The Ages

To get started, here is a look at the count of Shaw Brothers films by year.

I’m using the wonderful theme_fivethirtyeight for these charts. Someday, I’ll make my own.

That’s 260 films over 22 years.

The first Kung Fu Shaw Brothers film in this data set is Temple of the Red Lotus from 1965. From the reviews, it sounds like it was a bit rough around the edges - but that’s about what you would expect from this burgeoning genre.

Looks pretty sweet to me, I’ll have to check it out

The studio hits its stride in the early 70’s, with a lull in the mid 70’s and another spike in the late 70’s / early 80’s. Keep in mind that even during the lull, most years the studio is still putting out 10 or more Kung Fu movies.

To create this graph, I first loaded my raw JSON file into R using the tidyjson package like this:

# load the library
library(tidyjson)

# read the raw json as text
filename = '../out/shaw.json'
shaw_json <- paste(readLines(filename), collapse="")

# parse the json into a table, pulling out
# the variables we want to explore.
films <- shaw_json %>% as.tbl_json %>% gather_array %>%
  spread_values(
    title = jstring("title"),
    director = jstring('director'),
    year = jstring('year'),
    watches = jnumber("watches"),
    likes = jnumber("likes"),
    time = jnumber("time")
  )

I then graphed count by year using ggplot:

films %>% ggplot(aes(x = year)) +
  geom_bar() +
  labs(title = 'Shaw Bros Films by Year')

Not too shabby.

Which Shaw Brothers Film should I watch?

If you are just getting started with the Kung Fu classics, 260 movies can be difficult to wade through. How do you get to the best of the best to make your initial experience in this genre a pleasant one?

Well, we can use the Letterboxd “watches” and “likes” metrics to help winnow down to the films that are the best bang-for-your-buck.

As you might expect, these two metrics are highly correlated:

Basically, anything with more than 400 watches or 100 likes seems like a good place to start. The standout, with over 800 watches is 1984’s Eight Diagram Pole Fighter. Not the catchiest title, but as one reviewer puts it:

Some of the raddest fights from any Shaw Bros films I’ve seen (specially that last one).

I haven’t seen this one yet, so I can’t comment - but it’s definitely on my list!

Prolific Directors

We have the director for each movie in our dataset, let’s look to see if there are any popular standouts.

I’d say! Chang Cheh directed 67 or roughly 26% of all Shaw Brothers Kung Fu!

According to his Wikipedia page, he was known as the “The Godfather of Hong Kong cinema”, and rightly so - at least in terms of quantity.

Let’s pull out the top 5 directors, in terms of movie count, and see when they were most active.

Here’s the R code:

# pull out just the top 5 directors
top_directors <- by_director %>% head(n = 5)
# filter films to those directed by these titans of Kung Fu
films_top_director <- films %>% filter(director %in% top_directors$director)

# add a label to distinguish top directors from everyone else
films_top_director_all <- films %>% mutate(director_label = ifelse(director %in% top_directors$director, director, 'Other'))

# graph
films_top_director_all %>%
  ggplot(aes(x = year)) +
  geom_bar(aes(fill = director_label)) +
  labs(title = 'Shaw Bros Director Count by Year', fill = '') +
  theme_fivethirtyeight()

and plot:

We can kind of see that Chang Cheh’s reign is towards the beginning of the Shaw Brothers timeline and tapers towards the end. Let’s view the same data as a percentage of the total movies made each year:

This shows how dominate Chang Cheh was in directing nearly half of Shaw Brothers films in some years. In the mid and later years, Chor Yuen came in to direct many films as well.

Title Showdown: Shaolin vs Swordsman

We will get to the actors in these films in a second, but first I wanted to explore the words used in the titles of these movies.

These classic Kung Fu films typically have thrilling and mysterious names, like The Thundering Sword and The Invincible Fist. I wondered if there were any terms used frequently in titles, common threads or themes that became popular in the genre.

As I really wanted a chance to play a bit with the exciting new tidytext package, I decided to use it for this very simple text analysis.

This package works well to extract text-based data into a format ready to be analyzed and parsed with dplyr, ggplot, and other packages from the tidyverse.

Here is how to split the titles column of our dataset so that each word is a separate row in a new data frame.

# load the library
library(tidytext)

# saves entire title in `title_all` column,
# then splits up title column creating the `word` column -
# with a row for every token (word).
titles <- films %>% mutate(title_all = title) %>% unnest_tokens(word, title)

Not too difficult right?

I also wanted to remove stop words, and that can be easily done with the included stop_words data:

# load stop_words into R environment
data("stop_words")

# filter stopwords
titles_filter <- titles %>% anti_join(stop_words, by = "word")

anti_join is part of the dplyr package.

Let’s take a look at the most frequently used words in these Shaw Brothers films.

Interesting! Our latest two films from my Netflix history were the afore mentioned Executioners of Shaolin and Shaolin Martial Arts, but I hadn’t realized “Shaolin” movies were so prevalent in the Shaw Brothers cannon.

Also interesting is the number of “Swordsman” movies - a term I don’t remember seeing yet.

Were these terms associated with different eras? Let’s graph their usage over time:

It would appear as if “Swordsman” movies were the hot stuff during early Shaw Brothers years, but they switched to mostly Shaolin’s in the mid-seventies. Here are the two terms overlaid:

Now the count-per-year isn’t mind-blowing or anything, but it does seem interesting that there was at least one movie with “Swordsman” in the title for six straight years, before an explosion of “Shaolin”.

I wanted to investigate a bit further. One of the first Swordsman movies they created was The One Armed Swordsman, and according to Wikipedia it was a big hit.

It was the first of the new style of wuxia films emphasizing male anti-heroes, violent swordplay and heavy bloodletting. It was the first Hong Kong film to make HK$1 million at the local box office, propelling its star Jimmy Wang to super stardom.

I must confess, being a casual enjoyer of Kung Fu, I was not familiar with the term “Wuxia”, so this needed a bit more investigation.

Almost immediately, I found a useful description from This essential guide to Wuxia:

The Chinese martial arts movie is generally split into two primary subgeneres: the kung fu film and the wuxia film. The kung fu film is newer and focuses primarily on hand-to-hand combat, it’s steeped in traditional fighting forms and there’s a general emphasis on the physical skill of the performer: special effects are generally disdained.

Wuxia is a much older form, based ultimately in the long tradition of Chinese adventure literature … Its heroes follow a very specific code of honor as they navigate the jianghu, an underworld of outlaws and bandits outside the normal streams of civilization.

Ah Ha! So the Swordsman / Shaolin dichotomy could be representative of the switch from this older style wuxia, to the new hip pure Kung Fu. The reappearance of both terms in the eighties could indicate a new found enjoyment for both styles. Apparently more famous modern films like Crouching Tiger, Hidden Dragon fall into the wuxia category.

I say “modern” here, but can you believe this movie is 17 years old? Time flies!

Or, it could be that the creators of these movies were lazy and just wanted to rely on the success of The One Armed Swordsman to make some money. Either way, its an interesting split.

Not knowing much about Chinese culture from that time period, I was also interested in learning more as to why “Shaolin” in particular was such a buzzword for Kung Fu movies. I’m still learning, but I found a great essay on history in the Shaw Brothers that points to an answer:

These films, focused on the Shaolin Temple as a center for anti-Qing resistance, provide a dizzying metaphorical potential, with the Qing variously standing in for Western imperialists, the Japanese, the Nationalist Kuomingtang, the Communists, or even simply the Manchurians themselves, while the Buddhism of the monks allows for examining of the contradictions at the heart of traditional Chinese belief systems, between the imperatives of social justice and withdrawal from worldly concerns.

Silly Kung Fu Titles with word2vec

Inspired by the ever wonderful Lynn Cherny’s word2vec experimentation, I wanted to use this opportunity to experiment just a bit with word embeddings and “word arithmetic”.

So I came up with an idea too silly to be useful, but fun for me to play with. For each noun detected in a title, I used Gensim’s word2vec implementation to find the nearest word to that noun, after subtracting “Chinese”, and adding another country.

Again, don’t take this seriously, I was just wanting to see what would happen.

So, cherry-picking a comical example:

buddha - Chinese + American = God Fearin'

See? kind of funny at least in theory. I did this for every title, for a number of different countries. In practice, the results aren’t as hilarious as I had hopped for, probably because country vectors don’t impact the trajectory of most of these words much. Also, I used a pre-trained model which obviously impacts the embeddings.

But, I made a little toy for exploring these new titles anyways, check it out!

The raw results and the code I used to generate these are also available.

Actor Troupes, Groups, and Clusters

Let’s end this exploration with a few insights into the actors in these films.

Even with my novice-level consumption of Shaw Brothers films, one thing you notice early on is a lot of familiar faces show up over and over in many of the movies.

We can see the extent of actor-over-use with another simple chart counting the number of movies frequently seen actors are found in.

Wow! Ku Feng apparently appeared in 82 Kung Fu movies. That’s a lot of Kung Fu!

His Wikipedia page isn’t as impressed with this feat as I am, providing little information on this Martial Arts Maniac. Apparently his real name is Chan Sze-man, and his first film was in 1959, and he is still acting today! The HKMDB, or Hong Kong Movie Database, provides just a bit more info:

In 1965, Ku formally signed an acting contract with Shaw Brothers where he made around 100 films for them and became most notably known as one of their top character actors. He has worked with just about every top Hong Kong director in a variety of films.

Ok then, well props to you Ku.

Did most of the top actors’ careers span multiple decades, or did actors come and go quickly?

We can graph the number of years an actor was featured in a movie over the total number of years in our dataset:

For the top actors, we see most were active more than half of the entire time Shaw Brothers Studios was making Kung Fu movies.

Here’s another quick graph showing the beginning and ending of these actors’ tenures:

Finding a Mob of Venoms

One phrase that came up when researching these Shaw Brothers films, related to actor-reuse, is the Venom Mob, a group of actors that did indeed appear in a lot of Shaw Brothers films together. They became well known after the success of The Five Venoms, hence the catchy name.

So, can we find a Mob of Venoms in our data?

Inspired by David Robinson’s network analysis of Love Actually, I decided to try out the igraph package for a bit of network exploration.

After a lot of filtering and frustration, I ended up with a basic, but still fairly hairball-y network:

In this network, the nodes are actors who have appeared in many films. The edges are co-occurrences of actors in the same movies, with the width of the edges proportional to the number of movies they were found together in.

You can see that there are a lot of actors appear together.

The Venom Mob includes Chiang Sheng, so here, I’ve highlighted in red everyone he is connected with. In this sub-cluster, you can see Philip Kwok, Lu Feng, and many of the other Venoms.

igraph is great for digging into properties of nodes, edges, and networks - and there is plenty more that could be done in this tool just for this simple dataset. I however, was wanting a bit more of an interactive exploratory tool that I could use to browse Kung Fu actor connections.

You were too? Great! That’s why I created the amazing Shaw Brothers Actors Network Visualization.

With it, you can clearly pick out the Venom Mob on the right, in the screenshot. But that’s not all, you can also browse all the movies actors appeared together in, and modify the network in lots of fun ways.

The code is based on my interactive network Flowing Data tutorial, which recently has been updated to use plain old Javascript and D3v4.

That wraps up my little data-driven exploration of Shaw Brothers films. As with any analysis, there’s plenty more to explore - but hopefully this was fun for you too, and inspires some data-driven exploration of your own.

Creating Bubble Charts with D3v4

2016-11-13T00:00:00+00:00

Well, as you probably have seen, heard, and experienced, our beloved D3 has shed its skin, and metamorphosed into a shiny new version. D3v4 is out and ready for engaging with by interactive data visualizationers world wide. Its full of fun new features and optimizations for us all to appreciate and enjoy.

But as you would expect from a major version increment, there are some breaking changes as you move from v3 to v4. Most of these, like the renaming of scales and using .merge because selections are now immutable are fairly straightforward to find and modify in existing code.

The force layout, however, has received a significant facelift, which may cause it to not be all that recognizable anymore, even to close friends. I believe John Burn-Murdoch puts it well, when he says:

Transitioning to #d3js version 4 was going so well, and then ... force layouts pic.twitter.com/EAWcRpsspX
— John Burn-Murdoch (@jburnmurdoch) October 27, 2016

As many of my previous tutorials and talks use the force layout to some effect, I wanted to share what I’ve learned from converting some of them to use the new v4 API and stymie just a bit the inevitable bit rot and decay that accompanies the never-ending march of technology.

To that end, I’ve reimplemented my old bubble chart tutorial and example to use D3v4. I’d like to share what I’ve learned along the way and how you can use the new force simulation in your own projects.

Check out the demo and source code to see the new cool version, then come back here to see how its done.

The bubble chart makes use of the force simulation capabilities of D3 so its useful to get an overview of what the new force API looks like. Then we will dive into the code to make this fun, splittable bubble chart.

Force Fundamentals

Fundamentally, the general idea of the force hasn’t changed. It works as a little physics simulation, ran multiple times until it runs out of energy and stabilizes. Each iteration or “tick” of the simulation, forces are “applied” to nodes and edges - meaning they push and pull on them in various manners.

These forces (and other parameters of the simulation) determine where the nodes should be each tick, then it is up to us to visually represent the nodes and edges as we see fit and move them to their specified positions.

With this general framework in mind, let’s look at how this is implemented in D3v4 and how it differs from the previous versions of D3.

Starting up a Simulation

In the old world of D3 version 3, the force layout was an amalgamation of parameters and functions that configured a good baseline of effects applied to the nodes and edges of your graph.

The new D3v4 force more cleanly separates the setup of the force simulation from the forces applied to entities in the simulation. Instead of the old all-in-one approach, this abstraction provides an easy way to compose multiple simple forces together in a simulation to make something magical.

Let’s take a look at this new API in comparison with the old one.

Previously, as mentioned, we would create a new force layout with a bunch of parameters, like so:

// old v3 layout creation
var force = d3.layout.force()
  .size([width, height])
  .gravity(-0.01)
  .friction(0.9);

With v4, we create a new instance of the forceSimulation to house our nodes and edges as well as the forces applied to them. Most of the configuration of the old API is gone. Instead, as we will see, we configure the individual forces, instead of the simulation as a whole.

// new v4 force simulation creation
var simulation = forceSimulation()
  .velocityDecay(0.2);

We don’t even specify a size anymore! There are still some things you can configure at the simulation level. Here we adjust the velocityDecay which as stated in the documentation, works like friction, adjusting the velocity of the nodes by multiplying by 1 - velocityDecay each tick.

This points to some changes in what attributes are stored in each node, which we will look more closely at in a bit.

The simulation also controls the alpha or “energy” of the simulation (this alpha has nothing to do with opacity - as it is sometimes indicates when talking about colors). As the simulation progresses, the alpha decreases, and the simulation loses power and slows down. Eventually, when alpha reaches a threshold, the simulation stops. The rate of decay of the simulation can be adjusted via the alphaDecay function.

Note that in v4, the simulation starts automatically, without having a .start() method. If you want it to hold off, say until you have added some nodes to the simulation, then you want to follow the creation of the simulation with:

simulation.stop();

To keep that from happening.

Noodling on the Nodes

Ok, I’ve said “nodes” about a hundred times already, but what are they?

Nodes are the input data to your simulation. You can pass in an array of nodes when you initially call forceSimulation, or you can add them more explicitly via the nodes method:

var myNodes = [{name:'a'}, {name:'b'}, {name:'c'}];
simulation.nodes(myNodes);

As you can see from the above example, each node should be an object. This object can contain attributes about the data you are representing, but it also is where the force simulation puts node positioning information.

With D3v4, each node object you provide will get the following attributes added to it:

x - the current x position of the node.
y - the current y position of the node.
vx - the node’s velocity in the x direction.
vy - the node’s velocity in the y direction.
index - the index of the node in the nodes array.

The main differences between old and new force with respect to nodes, are those velocity bits.

In D3v3, there was no vx or vy. Instead, the simulation kept track of the previous position that the node was in using dx and dy. This doesn’t matter too much for our Bubble Chart implementation, but affects things like how collision prevention works.

Forces Galore

Finally we get to the coolest part of the D3v4 force simulation - and also the most different and confusing part (at least to me): forces!

That collision prevention I just mentioned above? Thats powered by a force! In fact most of the details of positioning nodes in the force simulation is now left in the capable hands of one or more forces that get added to the simulation.

Each force is a function that gets called during each tick of the simulation. Its job is the modify the position of some or all nodes in the simulation, using those node attributes discussed above.

And while creating our own force functions should be fairly easy, thanks to Mike Bostock’s hard work, we get a great force starter pack built right in to D3! Check out the forces API docs to find forces made for x and y positioning, centering, and preventing collisions.

This distinction between the simulation and the forces applied to the simulation allows each force to care only about configuring itself. This also allows the users of these forces (us) to combine them in interesting ways to solve the unique problems of our force layout needs.

Adding a force to the simulation is straightforward. Here is an example of adding the centering force which keeps nodes centered on a particular point:

simulation.force('center', d3.forceCenter(width / 2, height / 2));

Adding a force using the force method takes a name for the force and then the instance of the force itself. We can configure the force as it is constructed. For the centering force, this means telling it where you want the nodes centered.

Each force has its own quirks and configurations, so go check out the documentation for the details.

Bigger, Stronger, Faster, Bubble Charts

Now that we have seen all the great new stuff about forces in D3v4, let’s finish off with how to implement a bubble chart with it.

As pointed out a few times now, the code is on Github and here is a demo.

Most of the code is fairly generic D3 - so let’s focus on the interesting force simulation bits.

We initialize the simulation with a few forces in place:

// constants used in the simulation
var center = {x: width / 2, y: height / 2};
var forceStrength = 0.03;

var simulation = d3.forceSimulation()
  .velocityDecay(0.2)
  .force('x', d3.forceX().strength(forceStrength).x(center.x))
  .force('y', d3.forceY().strength(forceStrength).y(center.y))
  .force('charge', d3.forceManyBody().strength(charge))
  .on('tick', ticked);

Here we create a new simulation and add some forces to it. The forces named x and y use positioning forces to push our nodes (which haven’t been added yet) towards the center of the visualization. We use these positioning forces instead of the forceCenter because, as the documentation states:

This force helps keeps nodes in the center of the viewport, and unlike the positioning force, it does not distort their relative positions.

But I want to distort their relative positions. I want them to all crowd around and jostle for the center.

Then we add a forceManyBody force to make the nodes repulse away from each other. In D3v3, we had access to the charge method, which assigned a positive or negative charge for each node. The many body force provides a similar capability, but is a heck of a lot faster then before.

And as you would expect, the strength of this force is a function that gets called for every node so we can adjust the charge based on some data attribute of our nodes.

Here is what the charge function looks like that is passed in as the force’s strength:

function charge(d) {
  return -forceStrength * Math.pow(d.radius, 2.0);
}

We based the charge on each node’s radius. The same forceStrength as what is used in the positioning forces is used here, so that all the forces work with the same amount of strength (which took some experimentation to make look “right”). The value returned by charge is negative, so the nodes repulse one another.

The end result of this many body force is that each node will avoid overlapping other nodes. We could have also used the Collision force, to prevent overlap, but with the many body force, the effect is more “organic” looking.

Check out Michael Currie’s awesome version of the bubble chart to see how to use collision instead of many body.

Finally we provide a tick callback method that will draw the nodes after each iteration of the simulation.

The ticked function for this is super simple, as it just moves the SVG circles used to make the bubbles to their new positions:

function ticked() {
  bubbles
    .attr('cx', function (d) { return d.x; })
    .attr('cy', function (d) { return d.y; });
}

Awesome. So now we have our bubbles all coalescing together in a big bubble pile.

Note that we aren’t using any transitions here. The “animation” is taken care of by the rapid repeated execution of the ticked function.

But how do we get them to split apart and into their categorical groupings? Well, that might be the one interesting trick this post can reveal.

It turns out if we add a force to a force simulation using force() and use the name of an existing force, it will first remove that existing force and then initialize our new force. You can see this in the force function’s implementation.

This means we can clobber our x force with a new one, then restart the simulation, and it will start using the new force with no issues. Great stuff!

For our bubble chart, we can use the following new forceX that will replace the “move to the center” force with a “move to your year” force.

Here’s what it looks like:

// This function is called once for each
//  node and provides the appropriate
//  x position for that node
function nodeYearPos(d) {
  return yearCenters[d.year].x;
}

// Reset the x force of the simulation
simulation.force('x', d3.forceX().strength(forceStrength).x(nodeYearPos));

And that’s it! Because of the beautiful composability of forces, the many body and y forces will work together with this new x force to create the node groupings.

There is one additional detail in that we need to restart the simulation to get the nodes to move to their new locations. We also need to “recharge” the simulation so the nodes have enough energy to get to their new locations.

We can do that simply by resetting the alpha of the simulation to 1 and restarting:

simulation.alpha(1).restart();

This is pretty much it! There are a few additional tweaks to the rest of the code, for scale renaming and such, but rest of the logic is the same. In the source code for the bubble chart I’ve used @v4 in the comments for sections that were modified in the switch to D3v4.

Hopefully this helps kick start your experimentation with the new force simulation awesomeness. Now go out there, build something cool, and let me know what you make!

Becoming a Data Visualization Practitioner

2016-08-11T00:00:00+00:00

If I can do it - so can you!

Here are some notes on an anecdotal path towards becoming a data visualization practitioner - Someone who does data vis. These steps are in rough sequential order, with of course overlap and repeating expected. All resources are tiny subsets of what could or should be included. Such is the nature of resource lists.

Step 1 - Ponder

Ponder what type of practitioner you would like to be.

Some options:

Technically focused, typically with computer science background

Examples:

Design focused, with entry points into data vis

Examples:

Domain focused, providing expertise to a narrow

Examples:

Biology - Bang Wong
Music - Alexander Chen

General Advocate, preaching to the masses.

Examples:

Jon Schwabish (also is a domain expert on vis in government)
Noah Illinsky

Academic focused, do data vis in a research setting (often overlaps with domain expert)

Examples:

Christopher Collins

Journalism focused, some of the best vis out there comes from newspapers

Examples:

Of course many people expand, overlap, and break out of these categories.

Moritz Stefaner is a example of someone who has combined technical, design, and academic focuses.

Also, you don’t have to use any of these generalized “templates” or even decide which type to become now, its just useful to keep these broad categories in your head.

Step 2: Read

You need to get a base line of existing data visualization examples, practices, and research. You will start to develop methodologies of how to talk about data vis, and what has been done in data vis. This comes in the form of books, blogs, and research papers.

Step 3: Monitor

Get a sense of what has been happening in the world of data visualization recently.

The readings address the history of data vis, but you also want to see what is going on RIGHT NOW.

Blogs to scroll through:

Additional Twitter People to follow:

Lynn Cherney - vis, data science & python
Nadieh Bremer - d3, style & interactives
Elijah Meeks - networks & opinions
Kim Rees - do good with data
Martin Wattenberg - google and famous
Jonathan Corum - science and news
Hadley Wickham - R, data science
Mike Bostock - D3

Audio:

Data Stories - podcasts
Policy Vis - podcasts

Step 4: Learn

Should be done concurrently with Step 3 and Step 5.

In order to create visualizations or talk about the creation of visualizations, you are going to need to learn how to use some tools.

These tools could be programming languages, code libraries or toolkits, or even GUI-based desktop applications.

Some suggestions. The important thing is to not try to learn them all at once, but pick one or two to explore based on your current strengths:

Programming Languages

R - if you want to do static images and data exploration, its hard to beat R. The ‘tidyverse’(ggplot2, dplyr, etc) of tools originally built by Hadley Wickham is really a full stack for researching and visualizing data. The R language is weird and not the easiest - but with a few concepts under your belt, you become very powerful.

Python - if you want a more standard scripting language to work with, try python. Pandas, Matplotlib, and Seaborn: go a long way to munging and visualizing data.

Processing - for more design / educational tool, processing is great. Many of the tutorials are moving to Processing.js - a javascript port of the tool. This allows for the creation of web-based interactive visualizations as well. But standard Processing is still a powerful and interesting option.

Code Libraries & Toolkits

D3 - Javascript library, still the Czar of interactive data visualizations. If you want to make interactives, and want them on the web - you should probably take a look at D3.

Processing.js - as mentioned, Processing.js is a great tool for ‘painting with data’. it takes a different paradigm then D3 - which some might find easier to reason about.

Desktop Applications

Excel - Still the workhorse of the non-programmer data world, its good to have some basic knowledge of Excel just to be able to share spreadsheets and quick analysis with others. And you can make some pretty interesting visualizations with it (as long as you stay away from the defaults).

Tableau - Success story of the Business Intelligence (BI) world, Tableau has matured into a robust and fast data exploration and visualization suite. I would pick this over Excel for non-coding exploration. Their flagship product is expensive, but they also have Tableau Public which is free and the way to start.

Step 5: Create

Should be done concurrently with Step 3 and Step 4

Creations can be in the form of

a visualization (static or interactive)
an analysis of a topic or dataset
an opinion of a hot data visualization topic
an aggregation or collection of visualizations around a central theme
a visualization book review
a code tutorial or process

Start by creating a website to hold your creations.

I would hands down recommend using Github Pages with a custom domain name. Jekyll (the tool powering these sites) is powerful and useful tool to learn and use.

You can put javascript interactive as separate repos. its awesome. And, it only costs ~15$ a year (for the domain name).

Hopefully you have seen visualizations that resonate with you, or visual forms you might want to explore more. Alternatively, you might have seen or heard or thought about interesting topics with data you might want to visualize.

Follow one of these routes down a path to capitalize on your learning and start the creation process. Creations can be simple, but well done - the best you can at this point in time. And always cite inspirations and data sources in an ‘about’ section!

Data to explore:

Data is Plural - hand curated and always interesting.
Open data portals: New York, Seattle, and more!
My Pinboard of data

Text Analysis and Visualization Workshop

2016-04-24T00:00:00+00:00

For OpenVis Conf 2016, we had the wonderful opportunity of offering workshops to conference goers.

In collaboration with my amazingly talented coworker, Yannick Assogba, we created a brand new, full day, text analysis and visualization workshop. It was a lot of hard work, but it was all worth it. For one magical day, we overcame internet difficulties and hurdled bad IT policy to provide an experience that impacted deeply my teaching approach and was universally enjoyed by our 30+ attendees.

Overview

The workshop content is available here.

We started with A series of Jupyter notebooks that familiarized everyone with the basics of working in a iterative notebook environment. We covered the basics of notebook use and of python - in case folks were unfamiliar.

Then we dove into text analysis, starting at the sentence level and working our way up to corpora. We looked at how to turn raw text into data, and a number of different metrics useful for understanding and visualizing the text.

Attendees each copied these notebooks to their machines, and used them as starting points for hands-on exercises throughout the morning.

After learning ‘enough to be dangerous’ about text analysis methods, we jumped over to the visualization side of things. Yannick and I gave a short intro to D3 and Processing.js before diving into an exploration of text visualization examples. The content for this presentation was organized based on visual encoding strategies of the text visualization methods.

The day ended with a short collaborative hackathon. Small groups were formed based on common interests and we all got to work building our own text visualizations. The results were impressive! Even after a full day of workshopping, the amount of interaction and the progress made by the teams was far more than I expected.

It was a great day, and I believe from the reviews and the interactions I’ve had since then that folks really enjoyed it.

Talk

After the conference, I truncated this workshop into a talk around text analysis and visualization methods and presented it to a visualization class at the University of Washington iSchool, hosted by Mike Freeman.

Those slides can be viewed on my site

The source is available on github.

The material is probably more suited to the workshop experience - but at least the visualization examples are interesting!

Creating Bubble Charts in JavaScript

2016-04-11T00:00:00+00:00

This post is very old and no longer represents the current state of how to use D3 properly. You should check out my updated Creating Bubble Charts with D3v4 instead!

This tutorial is a remake of my original Gates bubble chart tutorial. The content, data, and visualization are all the same. The difference is this one is remade with absolutely no CoffeeScript. JavaScript has moved on. I’ve moved on. I hope this tutorial continues to be useful for folks who don’t want to learn or deal with CoffeeScript. I’ve also stripped out jQuery, Bootstrap, and other cruft - allowing us to focus just on the bubbles.

You can find the wonderfully annotated sourcecode on GitHub and see it does indeed work as expected on the live demo.

There are JS bubble chart implementations from other folks too! If you are interested, check out ducky’s gist translation and the great Gravity Bubbles from Leonardo Flores.

Below, I’ve updated the original tutorial to match the content of the new implementation. It is a light touch, so if you’ve read it before, you probably won’t miss anything by skipping it now.

Bubbles!

A now classic New York Times piece featured a bubble chart of the proposed budget for 2013 by Shan Carter. It features some nice organic animations, and smooth transitions that add a lot of visual appeal to the graphic. This was all done using D3.js.

As FlowingData commenters point out , the use of bubbles may or may not be the best way to display this dataset. Still, the way this visualization draws you in and gets you to interact makes it a nice piece and one that makes you wonder how they did it.

In this post, we attempt to tease out some of the details of how this graphic works.

An Animated Bubble Chart

In order to better understand the budget visualization, I’ve created a similar bubble chart that displays information about what education-based donations the Gates Foundation has made.

You can see the full visualization here

And the visualization code is on github

The bubbles are color coded based on grant amount (I know the double encoding using size and color isn’t very helpful - but this data set didn’t have any quick and easy categorical fields besides year).

The data for this visualization comes from the Washington Post’s DataPost. I’ve added a categorization to the amount for each grant (low, medium, high) and pulled out the start year, but otherwise have left the data alone.

The rest of this tutorial will walk through the functionality behind this Gates Foundation visualization. This visualization is just a sketch of the functionality in the New York Times graphic - so we can see the relevant parts clearly.

D3 Force Layout Quick Reference

The Force Layout component of D3.js is used to great effect to provide most of the functionality behind the transitions, animations, and collisions in the visualization.

This layout essentially starts up a little physics simulation in your visualization. Components push and pull at one another, eventually settling into their final positions. Typically, this layout is used for graph visualization . Here, however, we forgo the use of edges and instead use it just to move around the nodes of a graph.

This means we don’t really need to know much about graph theory to understand how this graphic works. However, we do need to know the components that make up a force layout, and how to use them.

nodes

nodes is an array of objects that will be used as the nodes (i.e. bubbles in this graph) in the layout. Each node in the array needs a few attributes like x and y.

In this visualization, we set some of the required attributes manually - so we are in full control of them. The rest we let D3 set and handle internally.

gravity

In D3’s force layout, gravity isn’t really a force pushing downwards. Rather, it is a force that can push nodes towards the center of the layout.

The closer to the center a node is, the less of an impact the gravity parameter has on it.

Setting gravity to 0 disables it. gravity can also be negative. This gives the nodes a push away from the center.

friction

As the documentation states, perhaps a more accurate term for this force attribute would be velocity decay.

At each step of the physics simulation (or tick as it is called in D3), node movement is scaled by this friction. The recommended range of friction is 0 to 1, with 0 being no movement and 1 being no friction.

charge

The charge in a force layout refers to how nodes in the environment push away from one another or attract one another. Kind of like magnets, nodes have a charge that can be positive (attraction force) or negative (repelling force).

alpha

The alpha is the layout is described as the simulation’s cooling factor . I don’t know if that makes sense to me or not.

What I do know is that alpha starts at 0.1. After a few hundred ticks, alpha is decreased some amount. This continues until alpha is really small (for example 0.005), and then the simulation ends.

What this means is that alpha can be used to scale the movement of nodes in the simulation. So at the start, nodes will move relatively quickly. When the simulation is almost over, the nodes will just barely be tweaking their positions.

This allows you to damper the effects of the forces in your visualization over time. Without taking alpha into account, things like gravity would exert a consistent force on your nodes and pull them all into a clump.

How This Bubble Chart Works

Ok, we didn’t cover everything in the force layout, but we got through all the relevant bits for this visualization. Let me know if I messed anything up.

Now on to the interesting part!

There are two features of this bubble chart I want to talk about in more detail: collision detection and transitions.

Bubble Collision Detection

When I first saw the NYT chart, I was intrigued by how they could use different sized bubbles but not have them overlap or sit on top of one another.

The charge of a force layout specifies node-node repulsions, so it could be used to push nodes away from one another, creating this effect. But how can this work with different sizes if charge is just a single parameter?

The trick is that along with a static value, charge can also take a function, which is evaluated for each node in the layout, passing in that node’s data. Here is the charge function for this visualization:

// Charge Function
function charge(d) {
  return -Math.pow(d.radius, 2.0) / 8;
}

// Creation of d3 force layout
var force = d3.layout
  .force()
  .size([width, height])
  .charge(charge) // <- Using the charge function in the force layout
  .gravity(-0.01)
  .friction(0.9);

Pretty simple right? The charge documentation provides us with the understanding that this parameter can be a constant, but can also be a function. Sweet!

The charge function takes in an input parameter, d, which is the data associated with a node. The method returns the negative of the radius squared of the node, divided by 8.

The negative is for pushing away nodes. Dividing by 8 scales the repulsion to an appropriate scale for the size of the visualization.

A couple of other factors are at work that also contribute to the nice looking placement of these nodes.

We set its gravity to -0.01 and its friction value to 0.9 (which is the default). This says that nodes should be ever so slightly pushed away from the center of the layout, but there should be just enough friction to prevent them from scattering away.

So, nodes push away from one another based on their diameter - providing nice looking collision detection. The gravity and friction settings work along with this pushing to ensure nodes are sucked together or pushed away too far. It is a nice combo.

Here is the code that is used to configure and startup the force directed simulation:

Animation and Transitions

The capstone of the original graphic is the nice transitions between views of the data, where bubbles are pulled apart into separate groups. I’ve replicated this somewhat by having a view that divides up Gate’s grants by year.

How is this done? Well, let’s start with the “single group” view first. The position of each node is determined by the function called for each tick of the simulation. This function gets passed in the alpha for this iteration of the simulation.

// Group circles into a single blob.
function groupBubbles() {
  force.on("tick", function(e) {
    bubbles
      .each(moveToCenter(e.alpha))
      .attr("cx", function(d) {
        return d.x;
      })
      .attr("cy", function(d) {
        return d.y;
      });
  });

  force.start();
}

function moveToCenter(alpha) {
  return function(d) {
    d.x = d.x + (center.x - d.x) * damper * alpha;
    d.y = d.y + (center.y - d.y) * damper * alpha;
  };
}

The bubbles variable holds the svg circles that represent each node. So for every tick event, for each circle in bubbles, the moveToCenter function is called, with the current alpha value passed in. Then, The cx and cy of each circle is set based on it’s data’s x and y values.

moveToCenter is charged with modifying the circle’s x and y values based on the current alpha and the circle’s position in the previous iteration of tick.

To reiterate, moveToCenter returns a function that is called for each circle, passing in its data. Inside this function, the x and y values of the data are pushed towards the center point (which is set to the center of the visualization). This push towards the center is dampened by a constant, damper and alpha.

The alpha dampening allows the push towards the center to be reduced over the course of the simulation, giving other forces like gravity and charge the opportunity to push back.

The variable damper is set to 0.102. This probably took some time to find a good value for, and I just copied that from the original. Thanks NYT!

Ok, we’ve now seen how the nodes in the simulation move towards one point, what about multiple locations? The code is just about the same:

// Split circles based on year.
function splitBubbles() {
  force.on("tick", function(e) {
    bubbles
      .each(moveToYears(e.alpha))
      .attr("cx", function(d) {
        return d.x;
      })
      .attr("cy", function(d) {
        return d.y;
      });
  });

  force.start();
}

function moveToYears(alpha) {
  return function(d) {
    var target = yearCenters[d.year];
    d.x = d.x + (target.x - d.x) * damper * alpha * 1.1;
    d.y = d.y + (target.y - d.y) * damper * alpha * 1.1;
  };
}

The switch to displaying by year is done by restarting the force simulation using force.start(). This time the tick function calls moveToYears.

moveToYears is almost the same as moveToCenter. The difference being that first the correct year point is extracted from yearCenters. Here’s what that object looks like:

var yearCenters = {
  2008: { x: width / 3, y: height / 2 },
  2009: { x: width / 2, y: height / 2 },
  2010: { x: (2 * width) / 3, y: height / 2 }
};

You can see each year has its own centroid position mapped to it.

moveToYears also multiplied by 1.1 to speed up the transition a bit. Again, these numbers take some tweaking and experimentation to find. The code could be simplified a bit if you wanted to pass in the unique multipliers for each transition.

So we’ve seen a general pattern for both of these views: setup the tick method to iterate over all the nodes to change their locations. This was done using the each method.

In fact you can chain multiple each methods that call different move functions. This would allow you to push and pull your nodes in various ways depending on their features. The NYT graphic uses at least two move functions for these transitions. One to move nodes towards a group point (like above), and one to move nodes towards their respective color band.

Hope this tutorial shows you some interesting ways to use D3’s force layout. I thought the original NYT visualization was interesting enough to find out more about it. Again, the full visualization code can be found on github. It is about 400 lines long, but includes copious comments for your perusing pleasure.

Force layouts: they are not just for force-directed graphs anymore!

What is Data Visualization?

2016-03-01T00:00:00+00:00

In 2016 I got the opportunity to speak in front of a small group of highschoolers who were part of the Girls Who Code group at Ingraham High School. Though we were interrupted by a power outage, I did my best to talk through my (always expanding) ideas about the point and power of data visualization.

Thanks very much to Erin Shellman for the invite!

Slides

Resources

Here are a bunch of links of things referenced in my talk. Thanks to all those who make great visualizations that I can show off!

Data visualization intersection

Visual encoding accuracy

Visualization memorability paper

Connected scatter plot paper

Scott Murray talks

Wind map

NYT - where we came from

Selfiecity

NYT - connecting the dots

Clubs that connect the world

How much is a limb worth

Visualizing Shakespeare

NYT - gun sales soar

Bloomberg - tracking pace of social change

NYT - jet debris

Americas prolific Wallpunchers

Dear Data

Fox best pie chart ever

Fox bar chart

Visualizing algorithms

Visualizing machine learning

Bloomberg hottest year

Gun deaths

RAW by Design Density

Quartz guide to bad data

Learn JS data

Storytelling with data

The right chart for your data

Everything is a remix video

Everything is a remix visual

Flowing Data

Lynn Cherny pintrests

Women in Data Vis

NYT chart iterations

Trends in Data Visualization

2016-02-22T00:00:00+00:00

In 2016 I was honored to speak at a visualization symposium held at Stanford University. The title of the talk was Exciting Trends in Data Visualization where I attempted to map things we were working on at Bocoup to larger trends seen in the industry, and then project where these trends might lead in the future.

Slides

Stereotropes

2015-08-29T00:00:00+00:00

A project I was fortunate to work on, in collaboration with the other members of the Bocoup Data Vis Team, Stereotropes looks at the descriptions made for TV and film tropes through the lens of gender.

Also check out our detailed Stereotropes Design Post for more on how and why it was made.

A Git Workflow Walkthrough Series

2015-08-28T00:00:00+00:00

I wrote a bunch of words about how you can use Git in a simplistic, sane manner when collaborating with others on code. Check out the whole series!

Implementing a Vintage Bump Chart

2015-04-29T00:00:00+00:00

A quick shoutout for another vintage visualization I implemented in code:

The entire thing was implementing using JavaScript and D3.js. It’s about 700 lines of code with 200 lines of data and configuration.

Learn more about this classic visualization, and the code that was used to build it, in my Bocoup blog post.

So You Think You Can Scroll Talk

2015-04-06T00:00:00+00:00

This April, I had the privilege of coming back to OpenVis Conf for a second time as a presenter. This is doublely special for me as I recently became an Open Web Engineer at Bocoup - the organizers of this great conference (though it was after they had already accepted my talk).

Video

Check out the video on youtube:

or on the OpenVis Conf Site.

Slides

The slides for the talk are online - but because of all the videos used, this can talk quite a while to load remotely.

Alternatively, you can clone the talk repo, and get all the content on your own machine!

Examples

Perhaps, more usefully, I’ve collected the examples used in the talk, and some that didn’t make it in but are still awesome, as an examples page.

Code

Update: I’ve updated this code to use D3v4. Please Enjoy!

The demo application and associated tutorial can also be found on my site.

And of course, make your own scrolling art with embroiden - my fun little bookmarklet.

Hope these resources are useful to you!

So You Want to Build A Scroller

2015-04-06T00:00:00+00:00

Update: This code has been updated to use D3v4. Please Enjoy!

A critical component of successful interactive visualizations is orienting the users to what they are looking at. In a multi-faceted data story, it can be useful to explicitly move a user through the different views of the data.

Previously, we’ve looked at steppers as a way to implement this guidance system.

Here, I’d like to look at how the simple act of scrolling can be harnessed to direct a user through a complex visualization combined intuitively with a guiding narration.

Data Scrolling

There have been many interesting uses of scrolling in data visualization and interactive storytelling (which I will talk a lot more about in my upcoming presentation). In fact, Mike Bostock recently published a wonderful set of best practices on How To Scroll to maximize the use of this tool.

For this tutorial we will look at some of the implementation details for scrolling in data visualizations. I’ll focus on a common pattern that moves the user through discrete sections of content, while modifying the visualizations accompanying the text to create a unified data driven story.

This technique has been used in many places including the New York Times with Gregor Aisch’s The Clubs That Connect The World Cup:

And at Bloomberg with Adam Pearce’s American Truck Addiction and Measles Outbreak visualizations.

You can find a lot more examples in my talk and associated resources page.

I’ve created a demo that attempts to explain a lot of how to make these types of interactives, but covering a much less serious topic then these actual news reporters. As this tutorial will contribute to my upcoming OpenVis Conf talk in April, I decided to take a look at all the mistakes I made in my 2013 talk.

Specifically, it is a story around the frequency of filler words - “ah”, “um”, and “uh” - said during the talk. Take a look, and then, lets get started with looking behind the curtain at how to build this.

As usual, the code is available on github.

HTML and CSS Structure

Like all great web-based masterpieces, we must start with the basic, but critical task of getting our HTML framework for this visualization in place.

Let’s create a #sections div that will be used to hold the text portion of our interactive. Each section of text will live in a separate section element. The visualization component go in #vis which is put after .sections:

<div id='graphic'>
  <div id='sections'>
    <section class="step">
      <div class="title">OpenVis Conf 2013</div>
        I did what no presenter should ever do: I watched my own talk...
    </section>
    <section class="step">
      <!-- another section -->
    </section>
  </div>
  <div id='vis'>
  </div>
</div>

Now the CSS needs to get the scrolling text next to the visual display. Here is the gist of it:

#sections {
position: relative;
display: inline-block;
width: 250px;
top: 0px;
z-index: 90;
}

.step {
margin-bottom: 200px;
}

#vis {
display: inline-block;
position: fixed;
top: 60px;
z-index: 1;
margin-left: 0;

height: 600px;
width: 600px;
background-color: #ddd;
}

The inline-block display type allows us to position the #sections and #vis elements side-by-side while still providing a width and height. We also make the #vis position fixed, which makes it stay in the same place, even as we scroll. The margin on the .step elements ensures we have some space to scroll in for each section.

I’ve added a height, width, and background to the #vis - just so we can see where it will be. Even without any JavaScript, we can start to see the structure. Pretty cool!

A Reusable Scroller

Now, let’s turn to the details of figuring out where the page will be scrolled to. With a bit of work, we can encapsulate most of this code into a stand-alone function that can be reused in other projects. Most of this is based on the Bloomberg Visual’s excellent work - so thanks again to Adam Pearce for all his help!

The code for this scroll detection capability is in scroller.js.

The basic idea of what the code does is fairly straightforward:

Given a set of sections, figure out where these elements are located vertically down the page. When the page is scrolled, figure out which of these elements is currently front-and-center in the browser’s viewport. If this element is different than the last ‘active’ element, then switch to this new section and tell the visualization.

Easy right? Well, plenty of details to get right.

First, our main function takes a D3 selection that indicates the scrollable elements. In the demo, we call it by passing in d3.selectAll('.step'), but it is generic enough to accept any selection.

We want to get the vertical coordinates of where these sections start on the page so that we can find the nearest one as we scroll. There are probably a dozen ways to do this, but here is one:

sectionPositions = [];
var startPos;
sections.each(function(d,i) {
var top = this.getBoundingClientRect().top;

if(i === 0) {
startPos = top;
}
sectionPositions.push(top - startPos);
});

Here sections is again, the text steps that our users will scroll through. We use getBoundingClientRect to get the position of each element. The trick is that this position is relative to the viewport. If the user has scrolled partway down the page and then reloads, the top value for all the sections they have already passed will be negative.

To accommodate this, we make all section positions relative to the first section. So the section at index 0 has a section position of 0 and all other sections follow from that.

sectionPositions will be accessible to the code that figures out which section the user has scrolled to, which we will look at next.

Detecting the Active Section

The next step of our little scrolling algorithm is to detect which section the user is scrolled to. This position function will be called initially on load, and then whenever the window is scrolled by binding to the scroll event like this:

d3.select(window)
.on("scroll.scroller", position);

The .scroller suffix is how event namespaces are created in D3, so we don’t accidentally clobber event binding in other parts of our code.

Here is the position function. We will walk through it in detail below:

function position() {
var pos = window.pageYOffset - 10;
var sectionIndex = d3.bisect(sectionPositions, pos);
sectionIndex = Math.min(sections.size() - 1, sectionIndex);

if (currentIndex !== sectionIndex) {
dispatch.active(sectionIndex);
currentIndex = sectionIndex;
}
}

The first step is to find pos - the users current position on the page. Here, we use pageYOffset which is an alias to scrollY. This gives the vertical position on the page that the browser is currently scrolled to. We offset it just a bit with the - 10.

Next, we turn to the power of d3.bisect to pull out the index of the current section that has been scrolled to. I discussed d3.bisect in my linked small multiples tutorial, but it merits another look.

Given a sorted array of data and a new data element, d3.bisect will find the right spot for inserting that new element in the array to maintain its sorted-ness. It bisects the current array in the right spot to insert the new element. Its does this by performing a binary search on the array to find the insertion position.

The default bisect provides the index to the right of the insertion point. Using it here, we are asking for the index of the section after the section that was just scrolled passed.

It took me awhile to wrap my head around this, so perhaps an example will help. Here is an array of sorted values:

var nums = [0,100,200,300];

d3.bisect returns the index to the right of a match:

d3.bisect(nums,0);
=> 1

And also:

d3.bisect(nums,99);
=> 1

d3.bisect(nums,100);
=> 2

This is why we subtract 10 from the current position, so we start at index 0:

d3.bisect(nums,-10);
=> 0

The index can be one past the length of the sectionPositions array, so we make sure to keep it within bounds using:

sectionIndex = Math.min(sections.size() - 1, sectionIndex);

This index serves as the key to the visualization component of our site to know what it should be displaying. If the currently scrolled to section index doesn’t match the currentIndex, indicate the new index using d3.event.

Event Dispatching and Binding

Events are a great way to keep your code de-coupled and free of explicit dependencies, so let’s deviate a bit from the code flow to talk about them a bit more.

Like a lot of frameworks, D3 has its event generation and dispatching capabilities. Though, this works a bit differently than other eventing libraries I’ve used.

First, you need to create a dispatcher, using d3.dispatch. On creation, you indicate the event types the dispatcher will be able to dispatch. For our scroller, our dispatcher code looks like this:

var dispatch = d3.dispatch("active", "progress");

This means we can dispatch the active event and the progress event, which we will get to in a bit.

Now these event types are methods on our dispatch object. So, as we saw in the position function, we can send a new active event like so:

dispatch.active(sectionIndex);

The sectionIndex is passed as an input parameter to the handler of this event type.

In our visualization code, we will handle this event by adding a listener to the dispatcher’s event using dispatch.on. The code to listen to active events will look something like this:

scroll.on('active', function(index){
// ...
}

One hiccup is that .on is a method of the dispatcher, but our dispatch object is internal to our scroller code. We don’t really want to expose this internal detail to users of scroller. A cool trick to solve this issue is to use d3.rebind to copy our dispatch .on method to our scroller instance:

d3.rebind(scroll, dispatch, "on");

Check out the details in the example but this gives the scroll object the power of .on, which acts like a pass through - sending it on to the dispatcher to handle things.

Progressing Through a Section

So the code we have now will let us get notified when the user scrolls to a different section of text. This is great. It means we will be able to trigger visualization changes on these section changes.

But there are times when you might want to hook up animations within a section as the user scrolls. Tony Chu provides a great example of this kind of interaction with his Let’s Free Congress project. Scrolling within sections modifies the visuals in interesting and fun ways.

Inspired by this functionality, we can modify our code slightly to also allow for these kinds of within-section interactions.

Back in our position function, I’ve added this little snippet to the end of the function:

var prevIndex = Math.max(sectionIndex - 1, 0);
var prevTop = sectionPositions[prevIndex];
var progress = (pos - prevTop) / (sectionPositions[sectionIndex] - prevTop);
dispatch.progress(currentIndex, progress);

Here we just find the previous section’s top value, and compute how far into the current section we are. The number will range from 0.0 to 1.0, and we send that value along with the current index in the progress event.

With this little change, we can now get updates as to where the user has scrolled within a section.

Warning : there is some inconsistencies with how far different scrolling mechanisms move the page. If you want to get real specific, you might checkout a tool like jQuery Mouse Wheel that attempts to normalize this scrolling across devices.

Now, let’s finally use our scroll events to make some visualizations move!

Visual Transition Tips

With the scrolling events in place, most of the visualization work becomes standard D3 functionality. The code for this part of the demo should be well commented. Its long, but not all that exciting, from a cool-new-code perspective.

But there are still a few details worth talking through. Here are the steps, and perhaps a set of suggested best practices that I’ve compiled while examining example code and building up this visualization.

Setup All Visual Elements on Start

Sometimes I enjoy getting the enter, exit, and update functions of D3 working together in the same place. But with so many visual elements, I found it easier to just append all of them at one time during the initialization of the visualization.

setupVis is the function that performs this initialization. The elements for each section are added, and then hidden. Here is an example:

// square grid
var squares = g.selectAll(".square").data(wordData);
squares.enter()
.append("rect")
.attr("width", squareSize)
.attr("height", squareSize)
.attr("fill", "#fff")
.classed("square", true)
.classed("fill-square", function(d) { return d.filler; })
.attr("x", function(d) { return d.x;})
.attr("y", function(d) { return d.y;})
.attr("opacity", 0);

Here, I’m setting up all my .square’s with width, height, and position. Then hiding them by setting their opacity to 0.

If you visualization was even more complicated, you could have a separate function for setting up each sections elements to keep things more module.

Each Section Executes a Function

When the user scrolls to a new section, we want to show the new visuals for this section. Each of these visual changes is encapsulated in its own function

As we will be passed in the index of the current section, we can organize our section functions in an array:

activateFunctions[0] = showTitle;
activateFunctions[1] = showFillerTitle;
activateFunctions[2] = showGrid;
activateFunctions[3] = highlightGrid;
activateFunctions[4] = showBar;
activateFunctions[5] = showHistPart;
activateFunctions[6] = showHistAll;
activateFunctions[7] = showCough;
activateFunctions[8] = showHistAll;

Then bind to the active event to activate the right function

scroll.on('active', function(index){
plot.activate(index);
}

Here, the activate function will call the current section’s function using this activateFunctions array:

activateFunctions[index]();

It does some other work as well, so we will look at it in more detail in a minute.

Our progress event is processed in a similar fashion.

Section Functions Show Their Section and Hide Their Neighbor Sections

Technically, each section function could comprehensively deal with every single element of the visualization. With a few sections, this might not be terrible, but the complexity would quickly spiral out of control with more sections added.

Fortunately, this is not necessary. If you think about it, a section function really only needs to worry about showing the elements used for that section, and hiding the elements that were used in the section before it. But since a user can scroll both ways, we need to ensure the section that comes after is dealt with as well.

A simple example from the code is the first title section function. As nothing comes before it, it only needs to be concerned with showing its text elements and hiding elements of the title that comes after it in the second section.

function showTitle() {
g.selectAll(".count-title")
.transition()
.duration(0)
.attr("opacity", 0);

g.selectAll(".openvis-title")
.transition()
.duration(600)
.attr("opacity", 1.0);
}

Track Last Section Seen to Deal with Fast Scrolling

Unfortunately, what I’ve just told you about these simplistic section functions was a lie. A lie I wanted to believe myself. It’s true that under normal scrolling conditions, dealing with neighboring sections is all that needs to be done. With speedy scrolling, there is a chance that our scrolling event won’t fire for sections that are passed through too quickly.

So there is a chance that these intermediate step functions won’t get executed, leaving your visualization in an invalid state.

We could deal with this issue in the scroller code, but that seems a bit too brittle and specific to this particular use case. Instead, we can perform a little extra work in the activate function to ensure that all functions that should be are executed in order.

Here is the full code for activate:

chart.activate = function(index) {
activeIndex = index;
var sign = (activeIndex - lastIndex) < 0 ? -1 : 1;
var scrolledSections = d3.range(lastIndex + sign, activeIndex + sign, sign);
scrolledSections.forEach(function(i) {
activateFunctions[i]();
});
lastIndex = activeIndex;
};

So, we track the last activeIndex we have seen. When activate is called, we create an array of indexes using d3.range. With normal scrolling behavior, this range will just go from the current activeIndex to the new one. But with faster scrolling, the array will contain all the in-between steps, so we can call them in the right sequence.

Make Everything a Transition

Inside these section functions we are moving around elements and modifying lots of their attributes. The important thing to remember is to make all these modifications d3.transitions, even if you want them to happen immediately. This way, if the user is scrolling from one section to another during a transition, the transition will be interrupted and the transition for the current section will come into effect.

Here are some examples to help clarify the issue. Each example shows a circle being animated back and forth across the screen using transitions.

First we have the general case. The left-to-right transition finishes, and then the right-to-left one begins. No problems here. Everything looks dandy.

What happens if a transition isn’t completed before the next transition starts? For example, the user scrolls to a different section before the current sections animations are complete. Here is what we want to have happen. The left-to-right transition takes too long, and the right-to-left transition starts before it has a chance to complete. As long as they are both transitions, it doesn’t matter. The first transition is interrupted and the second one starts, as expected.

But, if we had not been using transitions for both movements? Say, we had tried to set the circle’s position directly with something like:

vis.selectAll(".circle")
.attr("cx", x);

Well, that would be bad! Instead of interrupting the transition, the animation would just continue unexpectedly - putting your visualization into an invalid state, until another transition corrected things:

If you want an effect to be immediate, use a duration of 0 for that transition.

g.selectAll(".bar-text")
.transition()
.duration(0)
.attr("opacity", 0);

If you try to mix transitions with straight attribute modifications, you most likely will end up with visualizations in strange states.

Use Named Transitions for Transitions that Shouldn’t Be Interrupted

For a few of my transitions, like where the little filler squares leave their spot on the grid, they can be interrupted multiple times with fast scrolling and not quite make it back to their spot. The results are less than flattering.

This was happening because transitions focused on changing these square’s fill value were interrupting this movement transition. I could double check the square’s locations in multiple different section functions, but this would complicate these unrelated functions unnecessarily.

Instead, I used the new-ish D3 feature of named transitions to ensure other transitions wouldn’t clobber this one.

g.selectAll(".fill-square")
.transition("move-fills")
.duration(800)
.attr("x", function(d,i) {
return d.x;
})
.attr("y", function(d,i) {
return d.y;
});

The name allows it to run concurrently with other transitions, ensuring the squares will always make it back to their homes!

Here is an example of using named transitions to change the color and position of the circle. The two transitions don’t interrupt one another, as they have two different names.

I haven’t used this feature much, but I’m excited to try it in other complex transition scenarios.

Go Forth and Scroll

Hopefully with this post I’ve dispelled some of the dark art of scrolling for you, and for myself.

Scrolling should still be used carefully and with the end user always in mind. And as pros like the New York Times continue to show us, sometimes a stepper is exactly the right tool for the job.

But when scrolling is what you want, check out the details of this post and the demo code. And if you want to hear me blather even more about scrolling, then you’ll be excited about my upcoming OpenVis Conf talk on this very subject!

As always, the code for this demo is on github Have suggestions or comments or complaints? Let me know!

An aside: The title comes from the epic Sierra Adventures Game Quest for Glory I - which I try to bring up in almost every conversation. If you haven’t tried it yet, go download Boxer, grab a copy and enjoy. Scrolling can wait for a bit.

Making Learn JS Data

2015-03-23T00:00:00+00:00

Recently Bocoup released a new guide: LearnJSData.com.

LearnJSData.com attempts to be a starting point for data processing and analysis using JavaScript. In it, Irene Ros and I, and a number of other collaborators, try to provide examples of common processing techniques using native JavasScript and a minimal set of libraries, namely D3.js and lodash.

The idea behind this project is to help those who may have some JavaScript experience, but have not really dealt much with data. Also, the guide can help those who have done data processing in other scripting languages, like python or R, and now have a need for similar data manipulation tools in JavaScript.

The guide serves as a starting point, and it is by no means comprehensive, but it suggests a way to get started and contains some handy tips that I, Irene, and others have learned along the way.

For a while, I have been wanting to create a tutorial around all the uses of d3.nest in terms of restructuring and summarizing data. Its a powerful tool, but one that isn’t often considered when processing data. It took a small tweet from Lynn Cherney regarding the lack of comprehensive data munging guides for JavaScript to suggest that there might be a need for this kind of introductory material.

Recently, the Bocoup family participated in their extra-special week-long Bocoupfest extravaganza. This is a special time for Bocoupers where we are encouraged to connect with others on collaborative projects. As a new member of the Bocoup Data Visualization team, my teammates, Irene Ros and Yannick Assogba suggested that this would be a good time to focus on this fledgling guide idea, and turn it into something real.

The process was incredible. During the week, Irene productionized the build system, wrote the difficult tasks of string manipulation and regular expressions, provided expert guidance, and even made an awesome logo for the project! Yannick and other Bocoupers, Earle Castledine, and Mike Pennisi helped with content suggestions and ways to make the guide more interactive in the future. Even Adam Sontag got involved by giving it a name and a place to live. It was a real team effort! Given their previous work with Learn CSS Layout, I guess that should come as no surprise.

At the end of the week, this scrap of an idea had been transformed into a real live thing!

It’s only been up for a few days, but so far the response from the community has really been fun to watch and be a part of. Even as we released it to the public via Twitter at the end of Bocoupfest, I was dubious of how others would perceive an introduction to a topic in a language not really known for its data processing functionality. I expected a few tweets and then perhaps some snarky comments of this being a bad idea, and that would be that.

But in just 7 days, Learn JS Data has had nearly 4,000 visitors. And people seem to be actually reading some of it! With an average of 2.5 pageviews per session. According to Twitter analytics, my tweet about the project was retweeted 95 times and has been seen by nearly 20,000 pairs of eyeballs (if you believe the measure of impressions is legitimate which I have my doubts about).

And the community isn’t just consuming. Folks are actively improving and adding content!

Already, we have a completely new task around combining data contributed completely by Timo Grossenbacher. And Davo Galavotti and friends are already working on a Spanish translation of the entire guide!

The experience has really driven home the power of collaboration for me. This project wouldn’t have been nearly as successful without the selfless contributions of those mentioned above.

This process also allowed me to get a bit more insight into the specialness of Bocoup. Because of the responsibility, dedication, and care Bocoup has shown in the Open Source and Open Web worlds, people seem to feel comfortable with contributing to Bocoup projects. They know that the same care and responsibility will be taken with their contributions. Its an interesting facet of these communities that I hadn’t experienced, or even considered before.

So, go checkout LearnJSData.com, and give us some feedback on how to improve it. And who knows, you just might find it useful yourself!

Interactive Small Multiples

2015-02-28T00:00:00+00:00

Based around the findings of my linked small multiples tutorial - I gave a talk about different types of interactions you can use with small multiples and how you might build one yourself.

Check out the slides here

I gave this talk at Bocoup (to silence-is-golden rave reviews).

I love talking about awesome visualization techniques like small multiples. If I can present this to your group - let me know!

Kansas City Streets Map Remix

2014-12-31T00:00:00+00:00

Technologies change with the times. Nothing stays the same. Awhile back, I experimented with TileMill to create a “streets only” view of Kansas City - with hopefully enough detail in the instructions that you could do it yourself for your own hometown.

Recently, Mapbox has released Mapbox Studio - which looks to be the successor to TileMill for all your map design needs. A revolutionary feature is the use of vector tiles to power the maps, instead of traditional image based solutions. Mapbox has been kind enough to include lots of data sources built right into Mapbox Studio .

This means, to make street maps, we might not need to import any additional data. Wow!

I wanted to try my hand at revamping the simple streets only Kansas City map using these new technologies. Join me, won’t you, on this adventure!

Start Me Up

After registering for a Mapbox account and downloading Mapbox Studio, we are introduced to Mapbox Studio by a nice little default map.

While we could probably start styling right now, I’d like to explore the project creation part of Mapbox Studio. Clicking on the “Projects” button in the lower-left portion of the window, opens the project menu.

It took me a second to figure out that there were two types of projects - style projects that deal with the look and feel of the map, and source projects that are used to define data sources. Guess I should have read a bit more of the documentation !

So, I want a new style project, which is toggled by default.

Clicking on “New Project” provides a plethora of built-in styles to start from.

We could start with a blank slate, but I was intrigued by the “High Contrast” style - which looked a bit like a supped up version of the streets map. Selecting this style, we are transported back to the map panel, but we a new view and a new starting location.

Style Me Up

Already, its looking pretty awesome!

On the left, we can see the CartoCSS that is used to make the map look the way it does. There are multiple style files, each one focusing on styling different portions of the map. The files provided with this style are style for general map features, road for styling roads at different zoom levels, and labels for (you guessed it) labels.

Real quick, let’s switch to our region of choice - Kansas City, using the nifty built in search feature.

No panning and dragging necessary!

Ok, now lets start styling.

First, we can ditch the labels style file, as we don’t want any labels at all. Click on the labels tab and then the little minus sign next to it. After confirming to remove, the style file is gone - but the labels remain. What?

Turns out we have to save to get the update to show up (as was the case with TileMill).

Clicking on the “Save As” button allows you to find a place to save the style project. After saving, we can now see the labels are gone.

Now we want to style some roads. The plan is to copy most of the style from my previous streets style . This means I’ll end up deleting most of the built in roads styles too - but I wanted to keep it around to get a feel for how to work with this new data source.

Looking at the styles in roads, we can see that #road entries have a class associated with their OpenStreetMap tags . Here is a sample:

To confirm that these classes are in the data, we can switch to the data explore view to see the metadata behind all these little lines on our map.

On the top bar, click the little “layers with magnifying glass” to switch views. Now you can view an x-ray of the data on the map. Click around a bit to see some of what is available.

Pretty darn impressive, if you ask me.

We can see that indeed each road has a property called class which seem to match the OSM names. So, I need to modify my original stylings to use these classes.

(Also, the class names are listed in the description section of the roads layer data)

I deleted the roads stylesheet now (even though it was really cool and useful to learn from), and augmented my old CartoCSS a bit. Here is the styling I came up with.

I’m sure I’m missing out on a ton of cool CartoCSS features, but I just wanted to get something out quick.

It looks like you have to zoom in to see all the road types (not sure how to override these default zoom levels for all roads).

Extract Me Out

So now we have a basic road map, we want to get it out of Mapbox Studio to print.

Unfortunately, unlike TileMill, Studio doesn’t (yet?) have the ability to export to SVG or another vector format (which seems a bit odd since the input data is vector now…).

However, you can export high resolution pngs, so let’s try that out.

Currently, there are two great tutorials from Mapbox about printing from Mapbox Studio:

Both might be more useful then my own ramblings, so refer to them if I’m not doing it for you.

First, we should go into settings and switch to 24-bit png to maintain top quality.

Next, we open the “Export Image” tool, which is located in the upper-right hand corner of the window.

Exporting still sounds a little tricky.

First, you draw a bounding box around the portion of the map you would like to export.

Then, you need to zoom in until you are at the desired zoom (which determines the size of the image) you want to create. In this test, I’ve zoomed to level 13.

Make sure the output format is as you like. You want png as the image format, and I’m exporting to 300ppi.

Then hit Download!

According to the web to canvas , if your bounding box is too big then you will need to export slices of the map and join them back together in the end.

I expect some of that might be automated in the future.

For 300ppi, at 13 zoom, I could download the entire KC export in one piece.

Put Me Up

The end result is a 40MB png file of Kansas City - ready for printing.

Still fun to explore and find the little hints of the city that you remember.

Linked Small Multiples (FlowingData)

2014-10-15T00:00:00+00:00

This primer and tutorial on FlowingData focuses on the use of interaction to link small multiples together to create a richer experience for the reader. We look at a number of examples before diving in to how to create your own sortable and scrubbable small multiples.

Experiments!

2014-03-18T00:00:00+00:00

Want some more data visualization attempts, studies, and undertakings? Check out my new Experiments Section

Here you will find an ever growing list of half-baked projects and hair-brained ideas.

As a bonus, I’ve whipped up a little tag-based navigational tool for finding what you are interested in, faster.

Most of the code for these little projects is easily accessible - just in case anyone wants to explore an idea further.

Check back often to discover what I have been experimenting with lately.

Let's Make a Bar Chart with Lyra

2014-02-28T00:00:00+00:00

WARNING: Lyra is labelled as alpha. Its interface might have changed since this posting.

A few days ago Arvind Satyanarayan from the UW Interactive Data Lab released an alpha version of Lyra that promises to be a way to create complex data visualizations without code. Think of it like an open source Adobe Illustrator, for data visualizations. Sounds pretty awesome, right? But does it work?!

Let’s find out.

There are a number of very interesting examples to learn from on the Lyra main page. These examples recreate some well known complex visuals.

I wanted to see what would be required to create a really basic bar chart - as a way to start to learn more about this interesting tool.

Here’s the final product:

Pretty simple right? But sometimes simple ways are best . So follow along with me for a bit of Lyra fun!

Your Visualization Development Environment

Here’s what it looks like when you first open Lyra :

I’ve annotated the app with my own terminology to provide a bit of orientation.

The far left panel provides capabilities to add data to the visualization and manipulate this data.

The top bar allows you to drag new visual components or marks into the visualization.

These marks are organized into layers (just like Illustrator!). Properties of the marks in a visualization can be manipulated in the far right panel. This panel is also used to modify the size of the visual, its axes, etc.

The middle gives you a preview of the visualization. You can also move around and resize the layers of your work here.

We will start with the data input and manipulation panel first. Far left.

A Simple Data Pipeline

Let’s pull in some data. Adding and manipulating data in Lyra is called a Data Pipeline. To avoid complications, I used medals, one of the sample data sets provided by Lyra. This set gives data about who won what at the Olympics (I’m guessing it is the 2014 Winter Olympics, but I don’t really know. I don’t really know much Olympics).

Click on the From Source drop down and select medals.

The original data includes gold, silver, and bronze medal winnings. Let’s just go for gold!

Filter Transform

To get this done inside Lyra, we can use the Transforms feature to modify the input data.

Click on the New Transform button.

Then click on the Filter button

We get a little Filter input field. We are going to perform our first data binding in Lyra now. This involves dragging column names from the Data section to a little circle data binding visual.

In this instance we are going to drag the column Medal_Type over to the Filter section.

Here is a little animated gif of the whole process:

I’ll break it down step-by-step for the anti-gif folks out there. Here’s the Medal_Type move.

Once you do that, Medal_Type should show up in your filter input.

Now we are going to complete the filter by checking this column against the string “Gold”.

This appears to be JavaScript, so you are going to want to use the double equals: Medal_Type == "Gold". In fact, a single equals sign in that filter seems to modify your data on the fly… which could be useful in other circumstances.

Once you have the filter setup, click Add to Pipeline. This will add the filter step to your pipeline (as you might guess by the name of the button).

If you look at the values shown in the Data panel, you should see that only those with Medal_Type as Gold are shown. Pretty cool right?

Sort those Medals

Now I want to sort the data in order of medal counts, so the countries with the most medals appear at the top. You might have guessed it by now, but we are going to use the sort Transform.

Click on the New Transform button again, but this time select Sort.

Now we need to pull Count down to the Sort By field:

Then select Descending for the Order. Once you add this to the pipeline, we should have all our data transformations done, and be ready for some visualizing.

The Visualization

Now let’s move our attention over to the far right panel.

We see Visualization at the top, and Layer 1 which is clickable to show more options.

Give Yourself Some Room

One thing that took me a while to find was how to change the width and height of the entire visualization. The width and height of a layer (even the first layer) isn’t the same as the width and height of the entire visual.

To change the visualizations size, click on the Visualization header. You should see Dimensions and Padding options.

As you can see in the image above, I gave my visualization some space by changing the Width to 700 and Height to 500. I am assuming this is in pixels. For now, let’s leave the Padding setting alone.

Your First Mark

Now is the moment we’ve all been waiting for: the start of the actual visualization.

Visual elements, like rectangles, lines, and even text are known as marks. We are going to add a rectangle mark for our bar chart.

To do this, drag the rectangle mark from Marks Menu on top down to the visualization preview section:

And release. You’ll be left with a little blue rectangle in your visualization preview.

Animated gif version:

The visualization panel should reflect this addition as well. You should see a Rect 1 mark under Layer 1. This is where we are going to associate data elements with the attributes of the rectangle.

Binding the Data

Ok, so far this just looks like a drawing app. That square is nice and all, but isn’t what you would call a “data visualization”. Let’s really start making the vis happen by binding some data to the visual properties of the rectangle mark.

The process is similar to how we used data elements in our data transforms. It works by drag-and-drop.

Because we are making a vertical bar chart, I want to associate the width of one or more rectangles with the count of Olympic medals for a country. You should see Width listed as a property of Rect 1.

Right now it is set to a constant number. We are going to drag over the Count data element from our data pipeline and connect it to the little circle visual next to Width. You really have to drop it right on the circle to make it count.

Once you release the data element, you should see your visualization preview change to show a bunch of rectangles on top of one another - instead of what looked like just one.

Not a bad start, but a bit crowded. Let’s move them off one another. I think you could do this a number of ways, but try binding the Country_Code data element to the mark’s Y Position property. Since the country code is unique for each bar (as we have already filtered to only include gold medals), this make each bar move to a unique position. Try it. Drag the Country_Code data element to Y Position and release.

My Lyra looks something like this:

We can start to see a little bar chart starting to form. Exciting!

Getting the Look

Now that we have bars, let’s make them a bit more attractive. First, modify the Height property to be 15 - so they aren’t sitting on top of one another.

Also, set the X Position to 0. We will move it around in a second.

Now we can gild our blue bars. Modify the Fill color property. You can type in a specific hex or other css color name. Feel free to pick one less terrible than mine. #e68b02 was as gold as I wanted to get.

Now I’m going to make room for my title and bar annotations by moving the layer over a bit. You could do this in the right panel by clicking on Layer 1 and setting the x/y position as well as the width and height. But, you can also do this in a more visual manner by moving the layer around in the Visualization Preview panel.

If you move your mouse inside the Visualization Preview, you should see the layer parameter highlighted with a little dotted line. Click inside that dotted line (but not on a bar) to select Layer 1. The little rectangles around the parameter should indicate it is selected.

Now click on these parameter rectangles and resize each side of the layer until its in the location you want. This will take some jostling - and can always be modified in the future.

You can check the settings in the Layer 1 properties. I ended up with the following:

X Start : 105
Width : 580
Y Start : 50
Height : 400

I don’t know if the visual manipulation is easier or more work than just trying out different values in the input boxes, but its nice to have options.

Again, the general idea is that I know I want to make room for a title and some labels on the side.

Label Those Bars

Ok, we make room for them, now let’s add some labels. As you might suspect, we are going to use the Text mark. Drag one of them down to your visualization preview - just like we did with the rectangle.

You should see a little blue text that reads “text”. Cute, but useless.

To get the country names on the screen, bind the Country_Name data element to the Text property of the text mark. This is the same drag-and-drop as before.

Now you have a big jumble of little blue text.

Bind the Y Position property to Country_Code, like you did for the rectangles. For some reason, I was having some problems with this drag-and-drop. Make sure the right circle changes color before you let go with your mouse.

Now the labels are starting to look like labels. To clean them up a bit, I set their Align Horizontal to Right Align.

Next, I played around with the x/y position and offset. I ended up with a -7 X Position and a 10 Y Offset. Also, I changed the color to #777777. Ta Da!

Always Add an Axis

Things are coming along now!

But we need to give our visual some perspective. What numbers are these bars actually representing? 50? 500? Let’s add an axis to find out.

In Lyra, it looks like Axes are associated with a particular layer. If you scroll back to the top of the right panel, you should see a little Axes section with a plus button to add a new axis. Click that button now.

You should see a little new axis panel appear:

For Type we are going to use X. For orient, we are going to use bottom. Now we need to associate a Scale with the axis. In the bottom left-hand corner of the screen, you should see your Scales menu. Drag up your X Scale and drop it in the Axis Scale property.

Once its dropped in, you should see your scale appear at the bottom of the preview panel. Pretty slick!

I changed all the colors of each element of the axis to #777777. I did try messing with the other properties of the axis mark and ended up screwing everything up. This will probably not be an issue in the future - but be advised - right now it seems a little picky.

Finish with a Title

Ok, my fingers are cramping up. We need to end this tutorial quick, so let’s add a title.

To do this, drop another Text mark into your visualization preview. Instead of associating its text with a data element, we will just type out a title. I choose “Gold Medals by Country”. You can use that too, if you want. A better title would be “Olympic Gold Medals by Country”. An even better title would include a date.

Next position and style. This should all be familiar, as we just did it with the labels. To mix things up a bit, I switched the Font Family to Georgia, and set the color to #595959. Font Size is 24.

Right now this text mark is part of the same layer as everything else. This might be the wrong way to do things - but it works at least in this simple example. A consequence of this association is that the Y Position will need to be negative to move the title above the chart. I set the Y Position as -24 and the X Position as 130 to get my particular styling. You can do better.

Down at the bottom of the screen are a few utility buttons:

You can export your work of art as SVG or PNG for static showing. You can also export as a Vega spec to allow for more interactive end goals. I haven’t done much with Vega yet, but this is a particularly exciting possibility that will require future exploring.

You should also be able to save the Lyra settings in your browser history and load up old ones from this button bar.

Your Homework

Hopefully you enjoyed this little introduction to Lyra. I think it might turn out to be a very useful tool in the future for creating data visualizations in a systematic, but non-programmatic way.

If you want to build on this example, what about filtering out countries with 0 gold medals?

Right now India, Armenia, etc are shown, but their bar values are really 0. Try filtering out these countries and re-situating the visual elements to take up this extra space.

For the more adventurous, try getting a visual display for all 3 types of medals. This might involve using the Group By Transform to add some hierarchy to the data.

And that is a simple introduction to Lyra. Hopefully starting with the basics will encourage us all to start making more advanced graphics with these tools and allow us to better understand the more complex starting points created by the author.

My first thoughts on Lyra is that it is certainly rough around the edges, but could provide real value to the visualization community.

It has some overlap with Design Density’s wonderful RAW app. While RAW is more polished, I think Lyra might ultimately provide a more powerful feature set for those interested in learning the tool.

Lyra’s binding capabilities allow for comparisons to some of what is featured in Bret Victor’s Drawing Dynamic Visualizations tool. If his tool is released, it will be interesting to compare further.

D3’s success in the data visualization world has been in part because of its large thriving community . And this large community was built up quickly in part because of the many examples provided by the authors as well as other individuals.

Let’s hope Lyra, if it proves useful to others, is also able to foster an active community - capable of taking it to the next level.

Visualizing Bird Songs with Web Audio

2014-01-27T00:00:00+00:00

UPDATE (07/04/2018): The amazingly talented Eren Cakmak has contributed a significantly updated version of spectrogramJS. This new version uses D3v5, has zoom capabilities, and allows you to specify a color scheme for your spectrogram. Thanks so much to Eren for these contributions!

Bird-based Visualizations

Recently, I rediscovered a wonderful book on my shelf, which ended up sparking a bit of visualization exploration.

The book was a 1966 copy of Birds of North America . I must of borrowed, bought, or stolen it at some point in life, only to forget of its existence.

Get yourself a copy, if you get the chance. Even Tufte agrees : its good. (In fact, I have started casually collecting as many Golden Guide and Golden Field Guide books as I can find. They are all amazing.)

Laced throughout the pages of this great resource, I discovered an interesting visual approach to depicting bird calls and songs on paper. Labelled a “sonagram” in the book, this method is now known as a spectrogram (probably to reduce confusion with modern day baby detecting and depicting technologies).

You can see them below, right above the range maps for each species.

From the notes in the front of the book, this is the first time this type of visual representation was used on the general public. And I think it works pretty darn well.

Its not a complex format. Really, a sort of specialized scatter plot. Time is on the x axis, frequency on the y. As you might guess, low notes have lower frequencies and high notes have higher frequencies. Cells within the graph are shaded by the intensity or loudness of a particular frequency at a particular time.

This simplistic framework allows for the complexity of natural sounds to be revealed.

Sounds from bird calls to cat calls show patterns that span the spectrum. Once you are able to pair a sound with its spectrogram, you can start hearing all the little complexities that are present in the visual. Furthermore, you can develop an intuition about what something will sound like - just from the spectrogram itself.

Of course, when I saw how beautiful this visualization was, I wanted to create some of my own. So I made a little tool to do just that!

Its called SpectrogramJS and you can try it out here and use it in your own projects!

Click analyze to begin processing the provided sound. Once done, you can hit play again to follow along with the spectrogram as the sound plays.

Simple, but I think pretty true to the original - while being more useful then a static image.

Check out the SpectrogramJS github to start visualizing your own bird calls today.

More on how SpectrogramJS was built below.

Reinventing the Wheel

It turns out others before me have heard the spectrogram’s siren call, and have developed software to generate similar charts. Audacity appears to have a spectrogram view . In fact, the Raven product suite, which includes the free Raven Lite, looks to be a powerful tool specifically for the purposes of analyzing bird songs and other animal calls using spectrograms.

Ok, so the spectrogram ground is pretty well covered. Implementing my own version wouldn’t necessarily be practical - or even useful - in light of these other much more robust systems. Or at least, these were my initial thoughts.

However, these tools are heavyweights in their field. And with all that power and bulk can be limiting in their own way.

What about a lightweight tool for quick and dirty spectrogram creations? One that could provide interactive capabilities outside of traditional applications and instead provide a web-based experience.

HTML5 Web Audio API to the Rescue!

So, I turned to the still developing Web Audio API to see if such a web-based experience could be had. It turns out that this API has audio analysis capabilities that make it suited for all sorts of audio visualizations.

In fact, cool spectrogram projects have already been implemented using this API. Google created a 3d sonogram to demo the use of the Web Audio API. Jos Dirksen also has a great moving spectrogram as part of his tutorial on web audio.

So, for my project, I focus on just replicating the look and feel of my original inspiration - the bird book. Clean and simple, providing insight and context.

I am just dipping my toes in the Audio API stream right now, but most of my knowledge comes from the wonderful Web Audio API book by Boris Smus (which is currently free to read on O’Reilly). Some of this content is also in the tutorials on HTML5 Rocks - which is also a great starting point.

From what I understand, the Web Audio API is used by creating and connecting nodes together in a directed graph, called the audio context . These nodes can be audio sources, audio outputs, or nodes to analyze and manipulate.

It is these analyzing nodes that we take advantage of to allow for the analysis of the temporal and frequency aspects of a sound. My audio context looks pretty simple for this visualization:

Specifically, createAnalyser() will create an instance of a Web Audio Analyzer that can provide frequency data through a Fourier transform . Its getByteFrequencyData() function dumps an array of frequency values for the portion of the audio stream currently being analyzed.

To sample frequency data throughout the sound clip, createScriptProcessor() is used to create a JavaScript processing node that will execute a callback at a particular sample rate.

The sampling callback is pretty simple. After some setup, it really just appends frequency data to an array of hashes - one for each sample point:

this.count += 1;
this.curTime = (this.sampleRate * this.count) / this.buffer.sampleRate;
this.analyser.getByteFrequencyData(this.freqs);

var d = { key: this.curTime, values: new Uint8Array(this.freqs) };
this.data.push(d);

Once the entire audio input has been analyzed, this data array forms the basis of the visualization. D3 is used to display this data. Due to the density of the dataset, I opted to use canvas to handle the main display, with SVG used for the axes and other visuals.

So, if nothing else, this visualization might be a good example of combining SVG and canvas for a nice one-two punch!

The visualization code is pretty simple. After some cleanup, the data array is drawn in the canvas using a loop:

Spectrogram.prototype.draw = function() {
  // ...

  this.svg.select(".x.axis").call(this.xAxis);
  this.svg.select(".y.axis").call(this.yAxis);

  var visContext = d3
    .select(this.selector)
    .select(".vis_canvas")[0][0]
    .getContext("2d");

  // clear canvas to prepare for new draw
  visContext.clearRect(0, 0, this.width + this.margin.left, this.height);

  this.data.forEach(function(d) {
    for (var i = 0; i < d.values.length - 1; i++) {
      var v = d.values[i];
      var x = that.xScale(d.key);
      var y = that.yScale(that.getBinFrequency(i));

      // draw to canvas
      visContext.fillStyle = that.zScale(v);
      visContext.fillRect(x, y, that.dotWidth, that.dotHeight);
    }
  });
};

As shown in the previous code snippet, d.key is the time point being sampled. d.values is the actual frequency data. Each value in d.values, v is plotted on the graph - its numerical value dictating its fill color.

For a 4 second audio clip (with a sample rate of 44.1 kHz), processing each 512 frames, you get around 358,000 data points to visualize. You can bump the processing down to every 256 frames to get 716,800 if you want - but things get a bit lethargic when updating the visualization. Still, that’s a lot of points to look at!

Future Work

Right now SpectrogramJS is a bit of a proof of concept. A lot more could go into making it more interactive, add features, and clean up the horrendous code.

If you think you might want to try using SpectrogramJS out - and have suggestions or need improvements, let me know!

So get out there and start visualizing sounds in your browser today!

Mapping Seattle - Streets

2013-10-18T00:00:00+00:00

Recently, I’ve up-rooted my family and transplanted from Kansas City, MO to Seattle, WA. This represents the first major venture out of the Midwest for my wife, dauther, and myself. Its exciting and scary and fun and difficult.

To begin to learn the city, I’ve started making basic maps - with the attempt to highlight different aspects or views of how things are arranged and where things are.

My first set attempts to take a very zoomed-out look at the streets.

Here are the streets of Seattle.

Not too exciting, but this provides a base line for those unacquainted with the lands up here.

Now on to my actual maps!

First, we have all the bus routes and other public transportation.

Click on the image to get to a super-big png version. I’ll try to work on vector-based versions as well, but my machine was having problems with that.

From this, we can guess at where the downtown is located - the dense ball of paths there near the sound. We can also see that the routes tend to emanate from this downtown hub and move North and South from the city. The little loops at the ends of the routes indicate where the bus turns around - the edges of the transportation network.

As a bonus, we get to see some of the ferry routes that arc away from the city’s center over to the islands of the Olympic Peninsula.

The rest of these maps focus on interesting features of how streets are named here. First, as you drive around, you notice almost all the streets have a direction as part of their name. The direction is in front of the name for streets that run East / West ( West Thurman Street ) and at the end of the name for Streets that run North / South ( 13th Avenue West ).

I wanted to find out where the boundaries of these directions were. Are all the West streets in one location, or scattered around? What section(s) of Seattle are ‘North’ or ‘Northwest’? These are some of the questions I had. So, (instead of researching this question) I decided to make a map! Here, the color of the streets based on their direction.

I am still pondering the color choices, but right now I am trying to use the color wheel to mirror the compass. Red for North, blue for south, and so on.

In any case, you can see that the directional streets of Seattle are highly compartmentalized. There are hard cut-offs that look to be fairly well maintained. Also, all of downtown is exempt from this directional practice (gray means no direction on the name).

The other odd thing about Seattle streets (at least odd for me - in my limited exposure to the world) is that there are numbered streets that run both North / South as well as East / West. This is one reason why that front / back direction is important.

What wasn’t clear to me was where the numbered streets start and in what direction they increase. So, here are some maps that try to answer these questions.

First, East/West streets. The gray streets are numbered streets with a direction in their name. Lower numbered streets are less dark. Higher numbered streets are more dark. Orange streets are those East/West streets that have a direction, but are not number streets.

The image itself needs a legend, but hopefully the pattern is clear from the explanation above. First, the numbers don’t start till you are fairly far away from the city center (only orange in the middle). And, the first numbers up North start at 50 or so - and not 1.

Secondly, numbers increment in both directions. Strange indeed!

Here is the same for North/South streets.

Same pattern. They increase in both directions.

With these maps, I’m at least getting an overall picture of how streets are arranged in my new town. Hopefully they might be interesting to you too!

Bonus: parks and greenspaces of Seattle

Apparently, all those boxes on the upper Western coast are Discovery Park tidelands .

Maps were created using the ever wonderful TileMill . I was exporting super large png files, so the street widths were jacked way up, which makes some areas look a little messy. Data comes from the infinitely useful Metro Extracts . Park data comes from the city of Seattle.

Some very basic processing of the road shapefiles was done using python with Fiona to help with the parsing. I definitely want to take advantage of shapely in the near future.

I also want to start layering data from other sources onto these base road maps. Foursquare and Yelp data are on my radar - as well as Google Places .

Abusing The Force Talk

2013-06-07T00:00:00+00:00

I was very fortunate to be able to speak at the first OpenVis Conf in Boston in April 2013.

The speakers were wonderful. The organizers were graceful. The venue was delightful.

Video

The talk was recorded, and can be found on Youtube:

or the archived OpenVis 2013 Site.

Slides

Here are my slides for Abusing the Force

Use the arrow keys on your keyboard to navigate. Each slide is its own html page. The source is all on github

I also put the slides on speaker deck for easy viewing:

It was a great time, and again, I feel very honored and humbled to have been part of OpenVis Conf.

Abstract Maps Talk at CityCampKC 2013

2013-04-21T00:00:00+00:00

I was fortunate to be part of the 2013 CityCampKC conference that occured 4/20 at the lovely Kauffman Foundation Conference Center

This one day unconference focuses on bringing together people from all walks of life under the banner of local community and government improvement. While a majority of the attendees appeared to be connected to the tech and startup scenes in Kansas City, I also got a chance to meet government officials, librarians, and artists from the community. The variety of people was great, with the caveat that trying to create a talk for this kind of group was difficult.

I wanted to include a bit of visualization into the mix, so decided to give a talk in the afternoon sessions. The title of my talk was “Abstract Maps for a Powerful Impact”. The premise of this talk is that while maps are an extremely potent way to tell a story with data, the ubiquitous use of Google Maps, what I would call the ‘default presets’ of online mapping, can dilute your data driven story.

So, I attempt to provide an overview of some alternatives to Google Maps, and to traditional maps in general, with applications to local data. I cover reductionist data/art techniques, Stamen Maps, cartograms, time-scale maps, and my own racial divide map.

I tried to show remixable examples of what could be used at a local level, as well as some iterations on failure. For example, my failed attempts to use dorling cartograms to display census tract data.

You can find the original talk here:

Abstract Maps for Powerful Impact

And I put it on SpeakerDeck - just for the fun of it:

It was a great chance to meet some new people. And fun to attempt to create some material around a topic that I really enjoy: Maps! I felt I went a little too lecturey, not quite hitting the right entertain/educate ratio. But you might enjoy!

Thanks to the CityCampKC organizers, Neighbor.ly, SingleWing, and everyone else involved in this great event.

VisKC for KC based Visualizations

2013-04-10T00:00:00+00:00

As I have been busy creating tutorials for FlowingData, I haven’t updated this site in a while.

I wanted to point out a new fledgling project I’ve started: VisKC.com - an aggregation site for Kansas City centric data visualizations.

With this site I’d like to highlight interesting maps, visualizations, as well as data that focus on the Kansas City Metro area. Right now its mostly maps (as expected) and some data. I hope to update about once a week.

In the future, I’d like to use it as a way to showcase some of my own KC based visualizations.

So, if you get a hankering for some local Data Vis, come on down to VisKC.com !

Animate Transitions Between Charts (FlowingData)

2013-01-17T00:00:00+00:00

Check out my tutorial on FlowingData and learn how you can use D3.js to transition between different chart types with nifty cool animations!

Small Multiples with Details on Demand

2012-12-17T00:00:00+00:00

Small multiples are a great way to show changes in data. By lining up multiple visualizations, they effectively allow for direct comparisons to be made with little effort. This is in contrast to potential alternatives such as video or animation, which force the observer to remember previous shots of the visualization to perform the same comparisons.

But some times the small in small multiples can be limiting. It can be difficult to cram in relevant annotations, scales or other data that should be in your data visualization to help explain your point. You want to use small multiples because Tufte says they’re the best , but you also don’t want your visualization to end up on Junk Charts because you splattered clutter everywhere. Is there any hope?

A Manifest Solution

Recently, Michael Porath provided an interesting graphic that I think includes an elegant solution to address the needs of the small but detailed visualization. I’ll call this method: Small Multiples with Details on Demand.

In Manifest Destiny , we see the story of how the United States came to be in a series of small multiples. Its a great piece, and the feature that I wanted to learn more about was the smooth looking transitions that occur when a user clicks on a map. You get a nice zoom effect that lets you know where the map came from, then details and more exploration in this single map. Returning back to the small multiples provides, again, a nice zoom to re-orient the user and allow them to continue to explore.

In this blog post we will look at a way to implement this effect using D3.js and jQuery.

C02 Emitted Over the Years

To isolate and explain this functionality, I’ve created a small multiple visualization of CO2 emissions that incorporates much of this functionality. Check it out by clicking the image below.

The code is available on github for downloading and remixing - so you can start on your own awesome interactive small multiple today!

C02 emissions for the current leading CO2 producers are displayed for each year between 1961 and 2008. But the top polluters now weren’t always the top polluters back then. We see the steady increase in emissions in China. We see Japan struggle to maintain lower emissions in the 80’s and 90’s. We see the decadent United States, boldly and consistently dioxiding the world - especially when compared to all of the European Union.

The details on demand element of the visualization is demonstrated when you click on a plot. Each bar chart gets a zoomed in detail view, providing more details about that year. In the example, the detail view is pretty simple, but the idea is that more complex annotations and interactions can occur here to provide more insights into the data without compromising the small multiples view.

Ready to learn how to create one on your own? Let’s get started!

Above the Smog: An Overview of the Implementation

The general idea for implementing this method isn’t too tricky:

Each chart in the small multiples view is contained in its own div and its own SVG element. 48 years means 48 SVGs on the page.

The detail view is created in a separate SVG element that spans the entire display area. The elements making up the detail view start out hidden.

When a chart is clicked, the small multiple version is drawn in the detail view so that it covers up the original below it. Additional annotations are then added to the detail view version of the graphic.

Finally, D3 transitions are used to zoom the detailed chart front and center while CSS transitions are used to obscure the small multiples in the background. Clicking on the detail view essentially undos the process: a transition pulls the detailed chart back to its origin ending with the detail view being re-hidden.

Now that we have the stratospheric perspective, let’s hit a few of the more tricky details. We’ll start with some details on the underlying html and CSS - but don’t worry! We will take a hard look at the visualization implementation too.

Building our Smog Factory: Proper Positioning of DOM Elements

To make this look good, it is important to get all the elements on the page positioned correctly. We have a few things to worry about: the positioning of the small multiple graphs, the detail view overlay covering the current viewing area, and a clean transition between small multiple and detail view.

Here is the overall structure of the html that will hold both the small multiple SVGs as well as the detail view:

<div id="main" role="main">
<div id="vis">
<div id="previews"></div>
<div id="detail" class="hidden">
<div id="detail_panel">
<svg width="800" height="800">
<g id="detail_view"></g>
</svg>
</div> <!-- end #detail_panel -->
</div> <!-- end #detail -->
</div> <!-- end #vis -->
</div> <!-- end #main -->

The #previews div is where each of the charts in the small multiple view will be created. It is currently empty as will generate these divs dynamically using D3.

The detail view is a bit more complicated. The top #detail div will serve to overlay the entire view. It starts hidden so the entire detail view is not visible initially.

The #detail_panel div will be used to center the detail view SVG so that it sits on top of the small multiples view.

Inside the #detail_panel we prepare a svg element with a #detail_view group. This g node is where the detail view visualization will be drawn.

On the CSS side, there are two topics I want to hit. First, we need to get the position of the detail view figured out.

In our example, as in the original visualization, the header and footer are fixed so that they stay visible on screen as the rest of the page is scrolled. This means we also want our #detail fixed and set below our header. Here’s the CSS for the most-outer #detail div:

#detail {
  position: fixed;
  top: 128px;
  width: 100%;
  background-color: rgba(255, 255, 255, 0.5);
}

You can see I’m also using rgba CSS3 to make the background semi-transparent.

Check out the rest of the stylesheet for details on positioning of other components.

Now, let’s look at the nice CSS transition used to transition between the two views.

The MDN has a nice tutorial on CSS transistions - so I won’t go into too much detail, but I wanted to look at the how this technology is used for the subtle background effects here.

The transition is controlled by two classes .hidden and .visible. Here’s what they look like:

.hidden {
opacity: 0;
transition: opacity 500ms ease-in;
-webkit-transition: opacity 500ms ease-in;
-moz-transition: opacity 500ms ease-in;
z-index: 0 !important;
}

.visible {
opacity: 1;
transition: opacity 500ms ease-in;
-webkit-transition: opacity 500ms ease-in;
-moz-transition: opacity 500ms ease-in;
}

We will apply the .hidden class to the small multiple view when the detail view comes up. Instead of immediately disappearing, the CSS transition allows us to apply our opacity over time. When this class is applied to a div, it will gradually disappear over half a second.

Likewise, the .visible class will make the elements reappear gradually over time. A nice subtle effect with just a bit of CSS.

Year by Year: Building the Small Multiple View

Ok, enough with the layout, let’s get to the visualization! The full coffeescript code is here - and below we will walk through some of the important parts.

Coffeescript Implementation

This visualization is written in CoffeeScript . I just enjoy it so much more than Javascript, I couldn’t bear to change. I invite you to take a 20 minute look at CoffeeScript - if you aren’t already familiar. It is a really simple language and really allows us to focus on the implementation details - rather then syntax.

First up, we need to create a separate div and SVG element for each of the charts in the small multiple. We will do this by using D3’s data method to bind the data to an empty selection and then appending the needed elements.

To better understand what the code is doing, here is what the data looks like:

[
{
"year": 1961,
"values": [
{
"name": "China",
"value": 552066.85,
"percent_world": 0.059
},
{
"name": "USA",
...
}
]
},
{
"year": 1962,
...
}
...
]

So data is an array of objects. Each object represents a single year’s data. The values array of the object stores the CO2 values for each country for that year.

And here’s the code:

 # bind data to svg elements so there will be a svg for # each year
pre = d3.select(this).select("#previews")
.selectAll(".preview").data(data)

      # create the svg elements
      pre.enter()
        .append("div")
        .attr("class", "preview")
        .attr("width", width)
        .attr("height", height)

      svgs = pre.append("svg")
        .attr("width", width)
        .attr("height", height)

      # create a group for displaying the bar chart in
      previews = svgs.append("g")

There are no div’s with class .preview initially, so when we use enter() this will create a div for each year. Inside each div, we append a SVG element. Both elements share the same width and height. We will use the position information of the preview div’s to figure out where to start the detail view transition. Inside the SVG, we create a group element to draw the chart in.

Next let’s draw the actual charts. I wanted to use the same code to draw the basic chart in both the small multiples view and the detail view. To make this happen, I encapsulated the code in a function called drawChart() and then execute this function to draw all the small multiple charts by calling it through D3’s each() function:

 # draw the graphs for each data element.
previews.each(drawChart)

The each() function will execute drawChart() for each of the SVG elements .

This drawing function will be passed to arguments: the data associated with the element, and its index in the data array. Additionally, inside drawChart(), this will be the current DOM element - meaning for us the g inside each SVG element that we want to draw the chart in.

So what does drawChart() look like?

drawChart = (d,i) -> # the 'this' element is the group # element which the bar chart will # live in
base = d3.select(this)
base.append("rect")
.attr("width", graphWidth)
.attr("height", graphHeight)
.attr("class", "background")

    # create the bars
    graph = base.append("g")
    graph.selectAll(".bar")
      .data((d) -> d.values)
      .enter().append("rect")
      .attr("x", (d) -> xScale(d.name))
      .attr("y", (d) -> (graphHeight - yScale(d.value) - yPadding))
      .attr("width", xScale.rangeBand())
      .attr("height", (d) ->  yScale(d.value))
      .attr("fill", (d) -> colorScale(d.name))
      .on("mouseover", showAnnotation)
      .on("mouseout", hideAnnotation)

    # add the year title
    graph.append("text")
      .text((d) -> d.year)
      .attr("class", "title")
      .attr("text-anchor", "middle")
      .attr("x", graphWidth / 2)
      .attr("dy", "1.3em")

This is pretty standard D3 for chart creation, based on existing bar chart examples .

Again, this is the g to draw in, so we select it first and then append elements to this element.

Our d input argument is the data bound to this element - meaning the object containing data from a particular year. For each year, we want to create a bar for each Object in the values array.

As described in the documentation , we can pass a function into data to return an array inside the parent data. This is why we use .data((d) -> d.values) when creating .bar rects - to bind a new rectangle for each of the country objects in values.

As part of the bar creation, we also assign mouseover and mouseout functions. But if you interact with the demo , you will see that nothing happens when you mouse over bars while you are in the small multiples view.

So what’s going on?

After executing drawChart(), we perform one more step with our small multiple bar charts:

previews.append("rect")
.attr("width", graphWidth)
.attr("height", graphHeight)
.attr("class", "mouse_preview")
.on("click", showDetail)

This overlays a transparent rectangle on top of each small multiple. When this rect is clicked, it will execute the showDetail function. It also prevents mouse interactions from reaching the bars underneath.

With relatively little code, we can make a nice small multiples display without duplicating code.

The Dirty Details: Building the Detail View

But we’re not done yet! Let’s investigate what happens when a small multiple is clicked - by looking at showDetail(). First, the creation and drawing code:

showDetail = (d,i) -> # switch the css on which divs are hidden
toggleHidden(true)

    detailView = d3.select("#detail_view")
    detailView.selectAll('.main').remove()

    # bind the single element to be detailed to the
    # detail view's group
    detailG = detailView.selectAll('g')
      .data([d]).enter()

    # create a new group to display the graph in
    main = detailG.append("g")
      .attr("class", "main")

    # draw graph just like in the initial creation
    # of the small multiples
    main.each(drawChart)

    # add details specific to the detail view
    main.each(drawDetails)

First, toggleHidden(true) swaps the .hidden and .visible CSS classes we looked at earlier, starting the transition to fade out the background and fade in the detail view.

Then we create a new g element inside our detail view’s SVG with the class .main. We use data([d]) to bind the data as d in this function refers to a single year’s data object. We want to use all the D3 functions - even when dealing with a single object - so binding to an array containing just this element is a good way to do that.

We use the same drawChart() function to create a bar chart in the detail view SVG. This chart will be initially identical to one of the charts in the small multiple view. The drawDetails() function adds annotations to the detail view chart to make it, well… more detailed.

drawDetails = (d,i) -> # like in 'drawChart', 'this' # is the group element to draw # the details in
graph = d3.select(this)

    # add names under bars
    graph.selectAll(".name")
      .data(d.values).enter()
      .append("text")
      .attr("class", "name")
      .text((d) -> d.name)
      .attr("text-anchor", "middle")
      .attr("y", graphHeight - yPadding)
      .attr("dy", "1.3em")
      .attr("x", (d) -> xScale(d.name) + xScale.rangeBand() / 2)
      .attr("font-size", 8)

    # add values above bars
    graph.selectAll(".amount")
      .data(d.values).enter()
      .append("text")
      .attr("class", "amount")
      .text((d) -> if d.value == 0 then "No Data" else shortenNumber(d.value))
      .attr("text-anchor", "middle")
      .attr("y", (d) -> (graphHeight - yScale(d.value) - yPadding))
      .attr("dy", (d) -> if yScale(d.value) < 10 then "-0.3em" else "1.1em")
      .attr("x", (d) -> xScale(d.name) + xScale.rangeBand() / 2)
      .attr("font-size", 5)

We can see that the d, i, and this variables all behave the same as in drawChart().

Here we add text for displaying the country name and values associated with each bar.

In this simple example, we are able to re-use most of the drawChart() code and simply append more details with drawDetails(). If you were making a more extreme detail view, you might not be able create it in such an elegant manner.

An important thing to remember is that initially the detail view is the same small size as the bar charts in the small multiples.

But now its time for the big finish - like a beautify butterfly the detail view will be transformed!

I’ve left most of my comments in the code snippet - even though it takes away some of the suspense.

 # getPosition is a helper function to # return the relative location of the graph # to be viewed in the detail view
pos = getPosition(i)

    # scrollTop returns the number of pixels
    # hidden on the top of the window because of
    # the window being scrolled down
    scrollTop = $(window).scrollTop()

    # first we move our (small) detail chart to be positioned over
    # its preview version
    main.attr('transform', "translate(#{pos.left},#{pos.top - scrollTop})")

    # then we use a transition to center the detailed graph and scale it
    # up to be bigger
    main.transition()
      .delay(500)
      .duration(500)
      .attr('transform', "translate(#{40},#{0}) scale(#{scaleFactor})")

First we figure out where the small multiple that was clicked is currently in the browser’s window. Then we transition the detail view’s chart to this location immediately.

Finally, we transition the entire SVG group to create the zoom effect.

We provide both a translation position and a scaling factor. D3 does all the rest of the work. It is able to recognize numbers inside strings and automatically tween between the starting location and the final destination.

The effect is simple, but I think very effective. The intuitive and meaningful nature of the animation maintain constancy and look good to boot.

Again, the code is available on github . Please let me know what small multiples and/or details on demand you create!

And thanks again to Michael Porath for the great and inspiring Manifest Destiny visualization.

Building a Bubble Cloud

2012-09-10T00:00:00+00:00

This post is very old and no longer represents the current state of how to use D3 properly. You should check out my updated Creating Bubble Charts with D3v4 instead!

For the 2012 Republican and Democratic national conventions, Mike Bostock , Shan Carter , and Matthew Ericson have created a series of visualizations highlighting the words being used in the speeches of both gatherings. These word-cloud-like word bubble clouds (what I’ll call bubble clouds, unless you can think of a better name) serve as a great interface for looking at the differences in the two conventions and for browsing through quotes from the talks.

Check them out here:

While there is a lot that could be discussed about all the little things that contribute to the quality and polish of these visualizations, in this tutorial we will look at some of the implementation details that make them tick.

Bubble Cloud Demo

I’ve created a basic bubble cloud visualization that tries to replicate some of the functionality of the NYT version. Click on the image below to see the demo .

Try clicking on the bubbles, then try dragging them around. Change the source text with the drop down in the upper-left corner.

The source code is available for download and use in your own projects.

This visualization uses D3’s force-directed layout, so if you aren’t familiar with that, you might check out my post on creating animated bubble charts , or the designing interactive network visualizations tutorial on Flowing Data.

While I won’t be going over the basics of the force layout, hopefully there is still enough in this implementation to keep things interesting. The main topics I want to cover here are:

The use of SVG and plain html components in the same visualization
Saving the state of the visualization using links
Creating a custom gravity effect
Creating a custom collision detection mechanism

Here is all the CoffeeScript that makes this visualization in case you want to follow along in the actual code. Sorry in advanced to the CoffeeScript haters.

Ready to get started? Let’s go!

You got HTML in my SVG!

Typically when using D3 , the more advanced visualizations are made with SVG. In a previous tutorial, we dabbled in a SVG-less D3 world , but the trade-offs were steep.

These bubble cloud visualizations actually combine SVG and Html elements elegantly! Specifically, the bubble elements themselves are circle’s inside of a SVG element, but the text on top of them is actually maintained in regular div ’s. As a bonus, both sets of visual elements are backed by the data, so there is very little code duplication or overhead using this structure.

Why would you want this duality? My initial thought when I looked at the implementation was that this was for increased browser compatibility. I figured that in IE8 users might just see the html elements, but miss out on the bubble backgrounds.

This turned out not to be the case. Navigating to the site with an old browser just gets you a static picture. The interactive versions still require browsers with SVG support.

My current guess for why this implementation route was chosen is because of how SVG deals with text wrapping.

Surprisingly, the 1.1 version of the SVG specification does not deal with word wrapping . There are some proposed workarounds , and the 1.2 spec includes the textArea element, which is supposed to solve this deficiency. But none of the solutions look particularly clean, dynamic, or without speed costs.

So the words and phrases the visualizations that will be showing might be on multiple lines. SVG doesn’t support this natively, and it looks like there are some trade-offs with the SVG only workarounds. What should we do?

Why not just implement this part in HTML? This looks to be the path Mike et al. at the New York Times took.

And this combo plater actually works out pretty well. To get started, here’s how we setup the elements that will hold the nodes and labels:

      # node will be used to group the bubbles
      node = svgEnter.append("g").attr("id", "bubble-nodes")
        .attr("transform", "translate(#{margin.left},#{margin.top})")

      # label is the container div for all the labels that sit on top of
      # the bubbles
      # - remember that we are keeping the labels in plain html and
      #  the bubbles in svg
      label = d3.select(this).selectAll("#bubble-labels").data([data])
        .enter()
        .append("div")
        .attr("id", "bubble-labels")

Here this refers to the #vis div. For the labels, we add a container div called #bubble-labels. Adding the circles that make up the nodes for the visualization is stuff we have seen before, so lets focus on the labels. First we bind the same data we use to build up the nodes to .bubble-label elements in our label selection:

  # ---
  # updateLabels is more involved as we need to deal with getting the sizing
  # to work well with the font size
  # ---
  updateLabels = () ->
    # as in updateNodes, we use idValue to define what the unique id for each data
    # point is
    label = label.selectAll(".bubble-label").data(data, (d) -> idValue(d))

So we can still use data bindings, just like in any other D3 visualization!

Now we build up our .bubble-label ’s by entering our (currently empty) selection and appending an anchor and some divs:

    # labels are anchors with div's inside them
    # labelEnter holds our enter selection so it
    # is easier to append multiple elements to this selection
    labelEnter = label.enter().append("a")
      .attr("class", "bubble-label")
      .attr("href", (d) -> "##{encodeURIComponent(idValue(d))}")
      .call(force.drag)
      .call(connectEvents)

    labelEnter.append("div")
      .attr("class", "bubble-label-name")
      .text((d) -> textValue(d))

    labelEnter.append("div")
      .attr("class", "bubble-label-value")
      .text((d) -> rValue(d))

Here we can see that each label is an a with two div elements inside it. One to hold the word/phrase name, the other to hold the count. We will look at how these anchors work in the next section.

The last thing we need to do when creating these labels is determine the size of the elements holding this text. We want the text to be able to spill out a bit on either size of the bubble, but also to wrap around if it gets too long. Let’s look at one way this can be done. Here’s the code used to style and position the text:

    # first set the font-size and cap the maximum width to enable word wrapping
    label
      .style("font-size", (d) -> Math.max(8, rScale(rValue(d) / 2)) + "px")
      .style("width", (d) -> 2.5 * rScale(rValue(d)) + "px")

    # now we need to adjust if the width set is too big for the text
    # do this using a temporary span
    label.append("span")
      .text((d) -> textValue(d))
      .each((d) -> d.dx = Math.max(2.5 * rScale(rValue(d)), this.getBoundingClientRect().width))
      .remove()

    # reset the width of the label to the actual width
    label
      .style("width", (d) -> d.dx + "px")

    # compute 'dy' - the value to shift the text from the top
    label.each((d) -> d.dy = this.getBoundingClientRect().height)

Let’s analyze this in more detail. As the comments say, we set our font size based on the same scale we are using to scale the bubbles. Good. Bigger bubbles will have bigger text. We also get word wrapping for free by setting the width to an appropriate value.

But if the text is actually too small for the bubble, it will skew to the left. The width will pad it out and make it not line up correctly with the bubble. So we need to find out if the bubble or the word-wrapped text is bigger.

We can do this by adding the text to a temporary span element and grabbing its actual width using getBoundingClientRect . each is used here so we can also set the dx property of our labels, which will be used when positioning them.

With this corrected width known, we can reset our label’s width to account for smaller words.

Finally, we use getBoundingClientRect again to get the amount to shift the label down.

Its a bit confusing, but I think its a great solution to this issue - and now that its been worked out, we can all use it.

To close off this label issue, here is how the dx and dy properties are used in the tick function to position the labels:

    # As the labels are created in raw html and not svg, we need
    # to ensure we specify the 'px' for moving based on pixels
    label
      .style("left", (d) -> ((margin.left + d.x) - d.dx / 2) + "px")
      .style("top", (d) -> ((margin.top + d.y) - d.dy / 2) + "px")

And there you have it. A nice way to use SVG and Html together, keeping with the D3 paradigm of binding data and using the same data backend for both.

A Way to Save State

By saving state, I mean being able to return to or share a particular view of a visualization. The way to do this in web-based interactive visualizations is by modifying the url so as to encode the current state of the visualization in the url itself.

It’s an important detail, and one that is commonly missing from even professionally created visualizations. I liked Bryan Conner’s remark on this subject when discussing the Money Ball WSJ political donor visualization :

Having the ability to share a specific view as a link on Twitter or Facebook is definitely important and, as far as I’m concerned, should be standard for almost any contemporary visualization.

With this bubble cloud implementation, we really only have to keep track of two variables: the text being viewed and the current word selected. I’ll focus on the word selected tracking, which we try to manage in a simple and elegant manner.

Clicking a bubble causes an immediate change to the url. And it is this url change that serves as a signal to update the visualization. This keeps the code clean and neat . Let’s see how it works.

The functionality depends on modifying the url’s hash section which is the text that follows the hash symbol (#). Here is the click callback function, which is executed each time a bubble is clicked:

  # ---
  # changes clicked bubble by modifying url
  # ---
  click = (d) ->
    location.replace("#" + encodeURIComponent(idValue(d)))
    d3.event.preventDefault()

We use location.replace to kill the previous selection (if anything was selected before) and switch to the newly highlighted bubble. The use of the replace function also means the history won’t be polluted with a bunch of versions of the same visualization.

The hash component of the url is accessible via location.hash. What’s more, we can register to be notified when this component of the url changes by using the hashchange event. Combining these insights, the bubble cloud first hooks into hashchange in the visualizations initialization function:

      # automatically call hashchange when the url has changed
      d3.select(window)
        .on("hashchange", hashchange)

And then uses this event to trigger a change in the active bubble.

  # ---
  # called when url after the # changes
  # ---
  hashchange = () ->
    id = decodeURIComponent(location.hash.substring(1)).trim()
    updateActive(id)

  # ---
  # activates new node
  # ---
  updateActive = (id) ->
    node.classed("bubble-selected", (d) -> id == idValue(d))
    # if no node is selected, id will be empty
    if id.length > 0
      d3.select("#status").html("<h3>The word <span class=\"active\">#{id}</span> is now active</h3>")
    else
      d3.select("#status").html("<h3>No word is active</h3>")

This decouples the event handling implementation from the visualization and would allow us to add more callbacks for hashchange to modify other parts of the page if we wanted to. Here, we are just modifying a div, but think of the possibilities!

Of course you can use your own custom event to perform the same decoupling, but I think it is nice having state saving via url modification and user interaction changes wrapped up in this nice little package.

Spreading Gravity Thin

In D3, the gravity component of a force actually draws nodes towards the center of the force layout. It is useful to ensure your nodes don’t fly off the screen. But the default gravity implementation is symmetrical, meaning it pulls on a node’s vertical and horizontal positions equally.

But what if you want a non-symmetrical gravity? Say your visualization is wider then it is tall (like in this example) and you want nodes to spread out along the x axis a bit more? Well, then you can implement your own ‘gravity’ function to push your nodes around how you like!

To get your own forces working on the nodes, we will call them each iteration of the force simulation by binding the force’s tick event to a callback function:

  # The force variable is the force layout controlling the bubbles
  # here we disable gravity and charge as we implement custom versions
  # of gravity and collisions for this visualization
  force = d3.layout.force()
    .gravity(0)
    .charge(0)
    .size([width, height])
    .on("tick", tick)

Every time the tick event is triggered, it will call our function, which happens to be called tick. We also remove the built in gravity and charge forces by setting them to 0. We will implement both of these forces ourselves.

Here is the tick function:

  # ---
  # tick callback function will be executed for every
  # iteration of the force simulation
  # - moves force nodes towards their destinations
  # - deals with collisions of force nodes
  # - updates visual bubbles to reflect new force node locations
  # ---
  tick = (e) ->
    dampenedAlpha = e.alpha * 0.1

    # Most of the work is done by the gravity and collide
    # functions.
    node
      .each(gravity(dampenedAlpha))
      .each(collide(jitter))
      .attr("transform", (d) -> "translate(#{d.x},#{d.y})")

    # As the labels are created in raw html and not svg, we need
    # to ensure we specify the 'px' for moving based on pixels
    label
      .style("left", (d) -> ((margin.left + d.x) - d.dx / 2) + "px")
      .style("top", (d) -> ((margin.top + d.y) - d.dy / 2) + "px")

You can see for each node we are calling gravity and collide which perform all the work on our nodes. After which we update our node and label positions to their new x and y coordinates. collide will be in the next section, so let’s focus on gravity now:

  # ---
  # custom gravity to skew the bubble placement
  # ---
  gravity = (alpha) ->
    # start with the center of the display
    cx = width / 2
    cy = height / 2
    # use alpha to affect how much to push
    # towards the horizontal or vertical
    ax = alpha / 8
    ay = alpha

    # return a function that will modify the
    # node's x and y values
    (d) ->
      d.x += (cx - d.x) * ax
      d.y += (cy - d.y) * ay

You can see how the only thing we really tweak is how much alpha affects the x and y components of the gravitational pull. The alpha parameter comes from D3’s force layout and is the cooling temperature for the layout simulation . alpha starts at 0.1 and decreases as the force simulation continues, getting to around 0.005 before stopping.

Here we are reducing the alpha value to be applied to the x movement by 8. When we multiple the movement towards the center by ax and ay, this makes the y movement stronger. Thus, while the nodes should remain centered vertically, they will be allowed to drift along the x axis.

Redistributing nodes like this allows more content below it to be ‘above the fold’ without running the risk of cutting off the bottom of the bubbles.

Don’t Burst Your Bubbles

As we saw in the bubble chart tutorial , a form of collision avoidance can be implemented in D3 by making the charge associated with each node a function of the size of the node. This provides a visually interesting, organic, experience, where bubbles push on one another in an particle-like way.

The bubbles in these word visualizations act in a subtly but significantly different manner. Here the nodes bounce off one another, maintaining a rigid parameter space around themselves. So how is this affect achieved in D3? By implementing a custom collision detection and avoidance algorithm!

Its actually not as complicated as it might sound. Here is the code:

  # ---
  # custom collision function to prevent
  # nodes from touching.
  # ---
  collide = (jitter) ->
    # return a function that modifies
    # the x and y of a node
    (d) ->
      data.forEach (d2) ->
        # check that we aren't comparing a node
        # with itself
        if d != d2
          # use distance formula to find distance
          # between two nodes
          x = d.x - d2.x
          y = d.y - d2.y
          distance = Math.sqrt(x * x + y * y)
          # find current minimum space between two nodes
          # using the forceR that was set to match the
          # visible radius of the nodes
          minDistance = d.forceR + d2.forceR + collisionPadding

          # if the current distance is less then the minimum
          # allowed then we need to push both nodes away from one another
          if distance < minDistance
            # scale the distance based on the jitter variable
            distance = (distance - minDistance) / distance * jitter
            # move our two nodes
            moveX = x * distance
            moveY = y * distance
            d.x -= moveX
            d.y -= moveY
            d2.x += moveX
            d2.y += moveY

We compute the distance between each node pair and see if it is less then the minimum distance allowed by our visualization. If so, then we move the nodes away from one another. The fact that D3 stores the current x and y position of each node as part of the data associated with that node makes it easy to get the values required to perform these calculations.

Notice the input variable jitter. What’s it do? We use it to help scale the distance that will be used to move the colliding nodes. Higher values will make this distance larger, smaller values will reduce it.

I’ve connected the value of jitter to a range control underneath the visualization. This lets you explore what the collisions look like when the nodes really start pushing each other hard. The default value, 0.5 visually seems to make the interactions look natural.

The Quad Connection

The implementation as shown above is a brute-force strategy. All nodes are compared with all other nodes, regardless of how close they are in the visualization. While this works decently for a small number of nodes, it will start bogging down as the node count increases.

If we had some rough-grained insight as to which nodes were nearest one another, we might be able to speed things up significantly by only performing these calculations for these nearby nodes.

This is actually how the NYT version of the collision algorithm is implemented. How do they know which nodes are nearby? By using a Quadtree .

A Quadtree is a tree-like data structure in which each internal node has four children (hence the name). You can use a Quadtree to partition up a two-dimensional space such that areas with more nodes will have more partitions. You can then visit the nodes in an efficient manner and stopped traversing once you have made it out of the quadrants that could affect the node in question.

D3 has a sparsely documented Quadtree implementation which is used in the force layout in this manner to make it fast. While this Quadtree implementation is a great option to have, and should be considered for collisions between many nodes, I think the brute-force version provides the same basic idea, without more technical overhead.

Your Own Bubble Cloud in the Sky

That does it for this tutorial. Hopefully this provides a bit more insight into these great pieces from the New York Times (and hopefully they don’t mind me continuing to exploit their great pieces).

Again, the code is on github , so grab it and let your bubble clouds accumulate!

Taking this to the next level would involve splitting the bubbles based on some variable, like the dual convention version .

For that, check out Mike’s great demonstration of how they split the bubbles to get a sense of how to add this kind of visualization.

Enjoy!

Interactive Networks Visualizations (FlowingData)

2012-08-02T00:00:00+00:00

Check out my tutorial on FlowingData and learn how you can use D3.js to make some awesome interactive networks.

Steps for Building a Stepper Visualization

2012-06-27T00:00:00+00:00

As the NYT’s Amanda Cox has said in some of her talks , annotating a visualization is one of the most important, but also one of the hardest things to do. In this post, we won’t tackle the ‘what’ to annotate, but an example of the ‘how’ to implement an annotation layer that needs to change as a user progresses through a visualization. Here’s what we will be building

The code for this example is on github . Each build of this process is broken up into its own folder.

We will look at implementing this using jQuery, and then as a bonus, a D3.js only implementation.

According to Amanda’s Eyeo 2012 talk (which I saw streaming, but don’t know if it’ll be up anywhere), the New York Times graphics department calls this type of interactive element a Stepper. A great example of a stepper can be found in the previously discussed Obama 2013 Budget Proposal visualization.

As the user clicks on each step of the stepper, the visualization changes - along with its annotations. This allows the visualization to focus on different points of the data presented at each step, and bring the viewer through the entire “story” of the data.

Before we get started, let me say that everything presented in this demo is pretty basic html and javascript. I just wanted to have a concise example of one way to implement a stepper for web-based. That being said, if you aren’t too bored already, then let’s get started!

Overview

The method I will present here is largely based on the implementation shown in the Obama Budget visualization. There are dozens of ways this kind of stepper could be implemented. Here are some pros to this process:

Keeps the annotations as html

This allows for easier editing and styling via css.

Uses jQuery to switch between annotation steps

Odds are you have some familiarity with this framework.

Pretty simple implementation

Nothing too crazy going on, just a few commonly used html/javascript features.

So the basic steps we will follow will be to:

Create the step annotations in html
Style them using css
Transition between them using jQuery

The Annotation Html

Each step will have its own div, with a unique id, and a class identifying it as a step div. Inside this div, each chunk of annotation text can be separated in its own div for positioning.

Importantly, all the step divs should be wrapped in a container div. The reason we will see in a minute.

The stepper nav links can be built out of an unsorted list, like any other navigation control.

A simple example would look something like this:

Our vis-container div wraps the entire visualization. The annotation sets for each step are contained in their own div with an .annotation-step class, and all annotation steps are children of #annotation-steps.

The #vis-canvas div would be where the actual visualization would be created.

If we were to stop now, here is what it would look like

The CSS

So far our annotation layer isn’t very impressive. The text for all the steps are visible at the same time, and they go down the page instead of being stacked on top of one another. Let’s look at the minium amount of css that is required to get our annotation layer looking right.

The main trick used to get positioning how we want it is absolute positioning inside a relative div . Like the link states, when absolute-positioned divs are inside a relative-positioned div, we can specify the exact location of these absolute divs within their parent div. Meaning, we don’t have to worry about positioning our annotations relative to the entire body of the page (which would get pretty annoying every time a little change was made).

For our .annotation-step divs, we want them all to stack on top of one another. Thus, using just position: absolute; should do the trick. It will absolutely position the div at the top of its relative parent.

For our specific blocks of text within the annotation, we can use absolute positioning, along with the left/@top@ styles to define exactly where each should go inside its annotation-step. You wouldn’t want to write an entire site using absolute positioning, but this method provides the kind of precision you want for where your annotation goes. Also, if you have better ways to position elements like this, let me know!

We finish off with some z-index to ensure the annotation layer and stepper controls are on top of the visualization.

Our basic stylings now look like this:

The comments in the code should let you follow what is going on.

Note that we have hidden all the .annotation-step divs, but we would like to start by showing the annotations for the first step. This can be done by adding style="display:block;" to the html of the first step’s div. This overrides the display:none; in our css file.

</code> While we are at it, lets make the stepper look half-way decent. The Obama Budget visualization uses a nice looking multipart button, complete with fancy rounded corners and shadows. You can find a close approximation to this style from [Twitter Bootstrap’s Button Group](http://twitter.github.com/bootstrap/components.html#buttonGroups) . Twitter Bootstrap makes it easy to make your UI elements look good. For this demo, however, I didn’t want to pull in their css and add any complexity. So instead, we will just style them as simple boxes: As an aside, I am using a [reset css file](https://github.com/vlandham/stepper_example/blob/gh-pages/final/css/reset.css) in addition to the css above. This is from the [HTML5 Boilerplate template](http://html5boilerplate.com/) . ## The Javascript Our javascript needs to switch the annotations to a new step when a step link is pressed, and look classy doing it. To do this, we will rely on the consistent naming of the id’s of the `.step-link` anchors and the id’s of the `annotation-step` divs. First, lets hook up a click callback function for the step links: This code is using [jQuery 1.7.2](http://jquery.com/) , but shouldn’t really be too version specific. Basically, we just need to get the new step’s id somehow (this uses jQuery’s `attr` method), and use it to switch to the next step. I’ve broken up the switching of the step into two functions: `switchStep` changes which step is active in the stepper links. `switchAnnotations` will change which `.step-annotation` is being displayed. Here is the entire js for this demo: So in `switchStep`, we use jQuery’s `.toggleClass` method to de-activate all steps and then activate the new step. In `switchAnnotation`, we do much the same thing with the annotation step divs. First we hide all the annotation steps, then use the `newStep` to find the step that should be turned on. To make things a little special, we use the `fadeIn` method to have the annotation appear after a bit of a delay. Fancy! ## Bonus: D3 Only Version Since as of late, I’ve been excited to use [D3.js](http://d3js.org/) , I was interested in how difficult it would be do implement this functionality just D3 instead of jQuery. It turns out, its not too hard at all. Here’s the [example just using D3](http://vallandingham.me/stepper_example/final_d3/) . The javascript code is below: You can see it might be a bit more verbose, but it gets the job done. A few things to point out with this D3 example: ##### Be mindful of when to use `.select` or `.selectAll` With D3, you need to use the `.select` method when accessing a single element on the page (like when we activate one of the stepper nav links), and `.selectAll` when you are going to be modifying the attributes of multiple elements (like hiding all the stepper annotations). ##### Difference between `.attr` and `.style` I always start using `.attr` to attempt to change CSS styles of an element when I need to use `.style`. Don’t be like me. Also, its important to remember that `.attr` (and `.style`) are accessors as well as setters. This feature of `.attr` is used to get the `id` of the stepper nav link clicked. ##### Implementing your own fade in D3 doesn’t have short-cuts for fancy transitions. Instead, you implement your own using the `d3.transition()` selection. This usually isn’t too hard - and means you can make very complicated transitions very quickly. In this example, I set all the `.annotation-step` div’s opacities to 0, then transition to an opacity of 1. `.delay` and `.duration` ensure that it looks the same as our jQuery example. ##### Move js code inclusion to end of html file. To simulate the functionality of `$(document).ready()`, I’ve found [recommendations](http://stackoverflow.com/questions/7169370/d3-js-and-document-onready) that say putting the javascript code at the end of your html file simulates the desired behavior of waiting for the document to load. This is what I did for the [index.html of this D3 example](https://github.com/vlandham/stepper_example/blob/gh-pages/final_d3/index.html) . Again, all this [code is on github](https://github.com/vlandham/stepper_example) as a separate project, so have fun building some steppers!

D3 Without SVG

2012-05-09T00:00:00+00:00

Typically, when creating interactive visualizations with d3.js, you are building your graphics up in SVG. While much of the D3 library is platform-independent, and there is even a small example using just html in the D3 tutorials, SVG is the typically used for all but the simplest D3 experiments.

Recently, the New York Times published an interactive visualization on the electorial map for the up-coming American presidential election. It features a bubble chart inspired interactive ‘game’ in which users can make their own predictions about which way undecided states will swing to see how the election might turn out.

I think its a great piece. One detail that isn’t immediately apparent until you start looking at the code is that the entire visualization is built with no SVG. This means that the entire visualization works great in both modern browsers and old ones that don’t support SVG like IE8 (I tested it and its just a little less smooth then in Chrome, but otherwise fine).

Here I’d like to look at just this one facet of how this visualization was made: how they use D3 without SVG.

SVG-less D3 Demo

To explore how the NYT group created their visualization, I’ve created a small demo using the same methods they did. The result is below:

Move

Overview

Here is the code for this visualization on github

And the corresponding CSS

Before looking at the implementation details, let’s get a high-level overview of how this works:

First we create a class that represents a bubble in the vis. This class is called SimpleBubble. It stores the data as well as the visual representation for that data. Here, the visual component is a series of divs that are styled using css.

By class I really mean that it is a constructor function with additional methods added to it’s prototype object . In javascript, there are no classes, but an object inherits properties from it’s prototype object. And functions in javascript are really Function objects . So a function has a prototype object to inherit functionality from. By defining ‘methods’ at the prototype level, we can instantiate instances of the function object (in this case, the bubbles in the graph), and they retain these methods. There are lots of posts online that explain javascript prototypes more. Here is one about OO in JS .

Next we create a class that represents the entire visualization. This SimpleVis class is handed the data we want to visualize, and the name of the DOM element to build the visualization at. It then instantiates SimpleBubble objects to store and visualize this data. The D3 force layout is used to position/move the bubbles and is kept at the SimpleVis level.

Sound good? Ok, lets look at the nitty-gritty of how this works.

Useful jQuery Methods

This visualization depends a lot on jQuery to make things easier. Before we look at the demo code, it might be useful to explain the jQuery methods used.

jQuery.proxy()

The proxy method wraps a function in a particular context . This is very similar to the fat arrow => in coffeescript . In fact, the class-based approach used in this demo would make it a good candidate for implementation in coffeescript. However, the NYT visualization used the prototype approach, so I wanted to try it out as well, for this demo.

proxy used in the demo and the original visualization to keep a consistent context inside of mouse-related callback functions. This allows us to get access to the location and data of the specific SimpleBubble that is being moused over to display in the tooltip.

jQuery.append()

append is used to combine the various divs used to make up the SimpleBubble visual representation. It is similar to d3.append, but works on jQuery objects. append is also used to add the bubbles to the main visualization div.

jQuery.remove()

The compliment to append, remove removes an element from the DOM. We use it here as a way to get rid of the tooltip box. Probably not the most efficient method, but it works.

Now that the jQuery introduction is over, lets look in more detail at the two classes that make up this demo, starting with our data container/view class SimpleBubble.

SimpleBubble

As mentioned above, instances of SimpleBubble store the data and DOM elements used to represent a bubble. Below is its init function which gets called in the constructor when we say something like var b = new SimpleBubble(data, id, canvas); :

SimpleBubble.prototype.init = function() {
/_ Elements that make up the bubbles display_/
this.el = $("<div class='bubble' id='bubble-" + this.id + "'></div>");
  this.elFill = $("<div class='bubbleFill'></div>");
this.el.append(this.elFill);

/_ Attach mouse interaction to root element _/
/_ Note use of \$.proxy to maintain context _/
this.el.on('mouseover', $.proxy(this.showToolTip, this));
  this.el.on('mouseout', $.proxy(this.hideToolTip, this));

/_ Set CSS of Elements _/
this.radius = this.data;
this.boxSize = this.data \* 2;

this.elFill.css({
width: this.boxSize,
height: this.boxSize,
left: -this.boxSize / 2,
top: -this.boxSize / 2,
"background-color": colors(this.data)});
};

So first we create some free standing divs that will make up the visual representation of the bubble. We store them as part of the SimpleBubble instance. the jQuery.append() method is used to attach the sub-elements to the root this.el element.

This demo only uses one sub-element, this.elFill. The NYT Visualization uses a number of child elements to hold text, background images, etc.

The jQuery.proxy() method is used to allow us to use tooltip functions defined in SimpleBubble to show and hide the tooltip when mousing.

Finally, CSS is applied using the jQuery object’s css method for the child elFill element. This is done to position it correctly inside the parent element.

Getting the positioning correct requires just the right css position incantations. While this all could be specified in the javascript, I decided to put them in a stand-alone CSS file, since they are not dependent on any data associated with the bubble. Here is the CSS applied to the bubble elements:

.bubble {
display: block;
position: absolute;
}

.bubbleFill {
position: absolute;
display: block;
border: solid 1px white;
-webkit-border-radius: 70px;
-moz-border-radius: 70px;
border-radius: 70px;
}

You can see that the main .bubble elements and their .bubbleFill child elements have their positions set to absolute. This means .bubble element’s position is based on its parent element (the main visualization div) and .bubbleFill element’s position is based on .bubble. So modifying the left and top stylings of .bubble will be done relative to the main visualization element (and not the entire page) and will in turn modify its child element’s positions.

The other thing to note is that we use the border-radius property to make the divs into circles. This leads to a complication with IE, as will be talked about more later.

Speaking of moving the bubbles, let’s look really quickly at the move method:

SimpleBubble.prototype.move = function() {
this.el.css({top: this.y, left:this.x});
};

Pretty simple. The bubble’s x and y are expected to be modified by SimpleVis. The move function just modifies the css of the bubble to shift its position. This isn’t the most abstracted way to implement this functionality, but it works.

The data given to the bubble instance in this demo is a simple integer used to determine its size. In a real visualization, you would probably use a javascript object to pass more information to the SimpleBubble for it to store.

The tooltip labels are lazily done, because I got lazy, so lets not focus on those, ok?

SimpleVis

SimpleVis stores the force layout, creates new SimpleBubble instances for each data point, and moves these bubbles based on the force layout. Most of this functionality is in it’s init method, so lets look at that:

SimpleVis.prototype.init = function() {
/_ Store reference to original this _/
var me = this;

/_ Initialize root visualization element _/
this.canvas.css({
width: this.width,
height: this.height,
"background-color": "#eee",
position: "relative"});

/_ Create Bubbles _/
for(var i=0; i< this.data.length; i++) {
var b = new SimpleBubble(this.data[i], i, this.canvas);
/_ Define Starting locations _/
b.x = b.boxSize + (20 _ (i+1));
b.y = b.boxSize + (10 _ (i+1));
this.bubbles.push(b);
/_ Add root bubble element to visualization _/
this.canvas.append(b.el);
};

/_ Setup force layout _/
this.force = d3.layout.force()
.nodes(this.bubbles)
.gravity(0)
.charge(this.bubbleCharge)
.friction(0.87)
.size([this.width, this.height])
.on("tick", function(e) {
me.bubbles.forEach( function(b) {
me.setBubbleLocation(b, e.alpha, me.centers);
b.move();
});
});

this.force.start();
};

Ok, so first we setup the div that will be holding our visualization. this.canvas is a jQuery object wrapped around a div present in the page. This div gets passed into the SimpleVis constructor, as we will see. Here the position is set to relative using the css method instead of in a static style-sheet. This is just to show another way it could be done.

Then we iterate over our data and add create new bubbles for each data point. Initial positions are just a guess on my part. They could be set to anything - depending on how you wanted the initial state of your visualization to look.

Finally, the d3.layout.force is initialized. This process is very similar to how my previous bubble chart example worked. Note that a method in SimpleVis determines the next location for each bubble on each tick. And then each bubble is moved to that location.

We use me so that we can refer to the original SimpleVis inside of the tick callback function.

The force’s charge parameter is defined by a function that will be passed each node. Lets look at this function really quickly:

this.bubbleCharge = function(d) {
return -Math.pow(d.radius,1) \* 8;
};

So, just like before, it uses the node’s radius to determine a relative charge to push away other nodes with. Quick and easy collision ‘detection’.

Lastly, lets checkout the setBubbleLocation method:

SimpleVis.prototype.setBubbleLocation = function(bubble, alpha, centers) {
var center = centers[this.bin(bubble.id)];
bubble.y = bubble.y + (center.y - bubble.y) _ (0.115) _ alpha;
bubble.x = bubble.x + (center.x - bubble.x) _ (0.115) _ alpha;
};

We can see that we get a center location based on the bubble’s id, then modify the x and y variables to move the bubble closer to that position. This movement is tempered by the force’s alpha, which will decay as time goes on.

Caveats and Trade Offs

The Binding Question

So just because we can use D3 without SVG doesn’t mean its always a great solution. We’ve gained access to D3’s great force directed layout. Using some shims (described below), we can also use D3’s scales and some of its array manipulation features.

We didn’t use one of D3’s most powerful constructs - binding data to the data’s representation using the built in selectAll and data functions. While we certainly can use D3 to bind data to any DOM element, the choice to use a class-based approach to encapsulate the data and representation in SimpleBubble replaced this D3 binding functionality.

More investigation might bring out a way to cleanly use D3’s binding capabilities with more complicated object based representations.

It Works in IE?

The original NYT graphic works great in IE8. This demo has some problems.

Specifically, the round appearance of the bubbles is created using the CSS border-radius property - but IE8 doesn’t support this! So how does the NYT graphic still look good? If the browser is found not to support border-radius, then a static circular image is used instead.

Generating a bunch of .gif images for this demo isn’t something I’m prepared to do, so I’ll have to be satisfied with boxes in IE. But otherwise it is impressive how cross-platform the NYT visualization is. They include a number of tweaks to optimize the experience for the iPad as well. Something that I might look at more later.

Other Tips

es5-shim

This demo also uses es5-shim to add missing functionality to Arrays in IE. The NYT visualization doesn’t use this particular library, but d3 uses some of these functions in places like scales, so its useful to have.

html5

As this bug report points out, loading d3 in IE won’t work unless the page is html5. So keep this in mind.

Here is a link to the code again. Thanks to the NYT group for creating this great graphic to learn from. In a future post, we may explore the drag and drop capabilities that make this visualization so interactive.

Kansas City Streets Visualized

2012-04-05T00:00:00+00:00

Using Tilemill and OpenStreetMaps for Printable Map-Art

Awhile ago, Gregor Aisch, creator of the awesome up-and-coming Kartograph posted his Berlin Streets Visualized , a map of just Berlin streets inspired by Ben Fry’s All Streets Map .

When I saw how nice and simple it was to create, I wanted to make one for myself, of my hometown - Kansas City. Here’s a scaled down version of the end result:

Gregor used street data in Adobe Illustrator form from the wonderful Cloudmade OSM downloads . Alas, Cloudemade’s downloads don’t include city level exports for North American cities (at least I didn’t see it anywhere).

So I decided to try a slight variation on the theme: use Shapefiles from OSM and style them using TileMill . Here I’ll spell out some of the specifics on how this works.

Metro Extracts

While it would be possible to use Osmosis to extract my local cities OSM data, fortunately for me, this was not necessary.

The wonderful Metro Extracts provided as a service by Michal Migurski (who’s blog you should subscribe to right now) gives you nice OSM extracts for a large list of metro areas world-wide.

The one gotcha with this service is that, unlike Cloudemade’s downloads, these extracts have not been converted to Illustrator format. I’m sure that getting it in that format wouldn’t be too hard, but honestly I’m pretty aweful at Illustrator, so wanted to try out a different way to style this data.

To this end, I grabbed the imposm shapefiles to work with in TileMill.

TileMill

TileMill is a great tool from the MapBox team. Its really meant create and style map tiles to be hosted on a service like their map publishing platform . They have some great looking map styles and you can upload your own custom tiles or data, like the NYPL Lab’s 1940s Census Tiles used on their Direct Me NYC site.

In addition to all this map tile stying, since version 0.6 TileMill has had the ability to export to SVG. This means we can style our map with TileMill, and get it in a vector format suitable for printing!

Here’s the basic process I used to import, style and export my street data:

Extract the shapefile archive.

Now you have a folder with lots of shapefile files inside.

Open TileMill, create a new Project called “KC Print Map”

I included the ‘Default data’, but then hid it in my final map styling.

Open your new Project

You should see a nice view of the world.

Select the Layers tab and load your shapefile

With the layers tab open, select Add layer. Navigate to where you extracted your shapefiles. Select the kansas-city-lawrence-topeka.osm-roads.shp file for your Datasource and use roads for your ID.

Note: Some of the road types have their own shapefiles, so you could import these layers separately and style them separately. This is the approach I started with, but ran into issues with missing road types and layering. So lets import all the roads, style the types separately (as we will see below).

Select “Style & Save” and then zoom in to your road data.

You should have a nice view of all the roads in KC (or whatever metro you choose).

Now lets style these roads!

Carto

Carto is the styling syntax used by TileMill. It looks a lot like CSS, but is based on less so has support for variables and conditional effects, which we will make use of here.

Road Styles

The full style.mss file used to style the map can be found in this gist

Here, I’ll provide a shortened version of the code, to talk about:

Lets go over the details.

First we create two variables, @minor_color and @major_color, to store the two colors used to style the different road types.

The first style, for #roads sets the basis of what all the roads will look like. It makes all the roads skinny and sets them to the minor road color.

The stylings after that override this default setting for certain types of roads. We use conditional styles, (the [key='value'] component) to select which road type we are talking about.

First we hide a number of types that aren’t roads. The above example hides railways. I used QGIS to find all the different types of roads in the shapefile, but there are probably lots of ways to do that.

Then, for major roads and highways, we increase the line width and use the darker @major_color. These major road stylings use the New symbolizer syntax (the ::newsymbol) which creates a ordering of overlays. A poor-mans version of separate layers. This works because we start with small roads and overlay larger ones. So the order of the stylings matter. Something to keep in mind.

The nice thing about this approach is that we can try out all sorts of different stylings really easily. Want to make major roads green and highways really fat? No problem. A two line change and you have:

TileMill makes it really easy to explore different combinations of styles.

Export

Once you have something you like, its time to get it into a format suitable for printing.

Select Export at the top-right of the screen. Then select SVG. You might need to zoom into your region of interest again and crop using shift-drag as necessary. When you have the cropping how you want it, select Export.

This will generate a file in your TileMill’s export directory. I believe, by default, it is in ~/Documents/MapBox/export.

Finishing Touches

Ok, so I would be misleading if I didn’t mention that I finished up by importing the SVG into Illustrator to add a white border and to resize it to be 20x24in. Theoretically, you could do this part in any editor capable of handling SVGs. However, none of the actual styling of the map was done in Illustrator.

I enjoy how interesting landmarks jump out at you, even when presented with just the streets of a city you are familiar with. For example, here is our lowly airport:

So go out and create some interesting map art with TileMill and Open Street Map data!

And thanks to Gregor for the nice tutorial to copy!

Creating Animated Bubble Charts in D3

2012-02-21T00:00:00+00:00

This post is very old and no longer represents the current state of how to use D3 properly. You should check out my updated Creating Bubble Charts with D3v4 instead!

Update: I moved the code to its own github repo - to make it easier to consume and maintain.

Update #2 I’ve rewritten this tutorial in straight JavaScript. So if you aren’t that in to CoffeeScript, check the new one out!

Recently, the New York Times featured a bubble chart of the proposed budget for 2013 by Shan Carter . It features some nice, organic, animations, and smooth transitions that add a lot of visual appeal to the graphic. This was all done using D3.js .

In this post, we attempt to tease out some of the details of how this graphic works.

Simple Animated Bubble Chart

In order to better understand the budget visualization, I’ve created a similar bubble chart that displays information about what education-based donations the Gates Foundation has made.

You can see the full visualization here

And the visualization code is on github

**Warning Coffeescript** The example is written in [CoffeeScript](http://coffeescript.org/) as I find it much easier to read and write than javascript. Everything is packaged up in a nice `BubbleChart` class. Hopefully this does not get in the way of you understanding what is going on. The original NYT visualization is in javascript. Let me know if something is confusing.

The data for this visualization comes from the Washington Posts DataPost . I’ve added a categorization to the amount for each grant (low, medium, high) and pulled out the start year, but otherwise have left the data alone.

D3’s Force Layout

The Force Layout component of D3.js is used to great effect to provide most of the functionality behind the transitions, animations, and collisions in the visualization.

This layout essentially starts up a little physics simulation in your visualization. Components push and pull at one another, eventually settling in to their final positions. Typically, this layout is used for graph visualization . Here, however, we forgo the use of edges and instead use it just to move around the nodes of a graph.

This means we don’t really need to know much about graph theory to understand how this graphic works. However, we do need to know the components that make up a force layout, and how to use them.

Force Layout Quick Reference

Here are some of the concepts this visualization will use.

nodes

nodes is an array of objects that will be used as the nodes (i.e. bubbles in this graph) in the layout. Each node in the array needs a few attributes like x and y.

In this visualization, we set some of the required attributes manually - so we are in full control of them. The rest we let D3 set and handle internally.

gravity

In D3’s force layout, gravity isn’t really a force pushing downwards. Rather, it is a force that can push nodes towards the center of the layout.

The closer to the center a node is, the less of an impact the gravity parameter has on it.

Setting gravity to 0 disables it. gravity can also be negative. This gives the nodes a push away from the center.

friction

As the documentation states , perhaps a more accurate term for this force attribute would be velocity decay.

charge

alpha

The alpha is the layout is described as the simulation’s cooling factor . I don’t know if that makes sense to me or not.

How This Bubble Chart Works

Ok, we didn’t cover everything in the force layout, but we got through all the relevant bits for this visualization. Let me know if I messed anything up.

Now on to the interesting part!

There are two features of this bubble chart I want to talk about in more detail: collision detection and transitions.

Bubble Collision Detection

When I first saw the NYT chart, I was intrigued by how they could use different sized bubbles but not have them overlap or sit on top of one another.

charge: (d) ->
-Math.pow(d.radius, 2.0) / 8

Pretty simple right? But, for me at least, it was unclear from the charge documentation , that this function based approach was possible.

Lets break it down. Again this is coffeescript, so what we are seeing is a method called charge in my BubbleChart class. This method takes in an input parameter, d, which is the data associated with a node. The method returns the negative of the diameter of the node, divided by 8.

The negative is for pushing away nodes. Dividing by 8 scales the repulsion to an appropriate scale for the size of the visualization.

A couple of other factors are at work that also contribute to the nice looking placement of these nodes.

Here is the code that is used to configure and startup the force directed simulation:

display_group_all: () =>
@force.gravity(-0.01)
.charge(this.charge)
.friction(0.9)
.on "tick", (e) =>
@circles.each(this.move_towards_center(e.alpha))
.attr("cx", (d) -> d.x)
.attr("cy", (d) -> d.y)

@force.start()

So @force is an instance variable of BubbleChart holding the force layout for the visualization. You can see the use of the charge method for the layout’s charge input. We set its gravity to -0.01 and its friction value to 0.9 (which is the default). This says that nodes should be ever so slightly pushed away from the center of the layout, but there should be just enough friction to prevent them from scattering away.

We will get to the tick function in the next section.

Animation and Transitions

The original graphic has some nice transitions between views of the data, where bubbles are pulled apart into separate groups. I’ve replicated this somewhat by having a view that divides up Gate’s grants by year.

How is this done? Well, lets start with the all-together view first. The position of each node is determined by the function called for each tick of the simulation. This function gets passed in the tick event, which includes the alpha for this iteration of the simulation.

We can see this function in the code segment above. I’ll highlight it again here:

# ...

.on "tick", (e) =>
@circles.each(this.move_towards_center(e.alpha))
.attr("cx", (d) -> d.x)
.attr("cy", (d) -> d.y)

The @circles instance variable holds the svg circles that represent each node. So what this code does is for every tick event, for each circle in @circles, the move_towards_center method is called, with the current alpha value passed in. Then, The cx and cy of each circle is set based on it’s data’s x and y values.

So, move_towards_center must be doing something with the data’s x and y values to get things to move. And indeed it is:

move*towards_center: (alpha) =>
(d) =>
d.x = d.x + (@center.x - d.x) * (@damper + 0.02) _ alpha
d.y = d.y + (@center.y - d.y) _ (@damper + 0.02) \_ alpha

So move_towards_center returns a function that is called for each circle, passing in its data. Inside this function, the x and y values of the data are pushed towards the @center point (which is set to the center of the visualization). This push towards the center is dampened by a constant, 0.02 + @damper and alpha.

The alpha dampening allows the push towards the center to be reduced over the course of the simulation, giving other forces like gravity and charge the opportunity to push back.

The variable @damper is set to 0.1. This probably took some time to find a good value for.

Ok, we’ve now seen how the nodes in the simulation move towards one point, what about multiple locations? The code is just about the same:

display_by_year: () =>
@force.gravity(@layout_gravity)
.charge(this.charge)
.friction(0.9)
.on "tick", (e) =>
@circles.each(this.move_towards_year(e.alpha))
.attr("cx", (d) -> d.x)
.attr("cy", (d) -> d.y)
@force.start()

move*towards_year: (alpha) =>
(d) =>
target = @year_centers[d.year]
d.x = d.x + (target.x - d.x) * (@damper + 0.02) _ alpha _ 1.1
d.y = d.y + (target.y - d.y) _ (@damper + 0.02) _ alpha \_ 1.1

The switch to displaying by year is done by restarting the force simulation. This time the tick function calls move_towards_year. Otherwise it’s about the same as display_group_all.

move_towards_year is almost the same as move_towards_center. The difference being that first the correct year point is extracted from @year_centers. Here’s what that variable looks like:

@year_centers = {
"2008": {x: @width / 3, y: @height / 2},
"2009": {x: @width / 2, y: @height / 2},
"2010": {x: 2 \* @width / 3, y: @height / 2}
}

You can see this is just an associative array where each year has its own location to move towards.

move_towards_year also multiplies by 1.1 to speed up the transition a bit. Again, these numbers take some tweaking and experimentation to find. The code could be simplified a bit if you wanted to pass in the unique multipliers for each transition.

So we’ve seen a general pattern for both of these views: setup the tick method to iterate over all the nodes to change their locations. This was done using the each method.

Hope this tutorial shows you some interesting ways to use D3’s force layout. I thought the original NYT visualization was interesting enough to find out more about it. Again, the full visualization code can be found on github. It is just over 200 lines of code - including comments.

Force layouts: they are not just for force-directed graphs anymore!

iBook Widgets with D3.js

2012-02-16T00:00:00+00:00

With the release of iBooks Author , the idea of using powerful javascript libraries inside using their HTML Module functionality probably came to a lot of people. I have certainly been thinking about the potential power of something like d3.js or leaflet.js embedded in an interactive text-book.

When iBooks Author was announced, I spent 5 minutes entertaining this idea, but soon was frustrated with having to come up to speed with Dashcode and other Apple developer tools - when all I really wanted to do was make a fun, interactive visualization.

Recently, I stumbled upon ibooksauthor.es a blog by Miguel Sanchez Molina that describes how to create an interactive visualization using Raphael without the need for Dashcode.

Consider this write-up a slight spin on the same idea, but using d3.js, with a hint of automation.

It turns out, a widget is really just a specially named folder with the .wdgt extension.

You can right-click on a widget, select ‘Show Package Contents’ and see the HTML and javascript files that go into it.

If I look at the default ‘Countdown’ widget from Dashcode, without any modifications, here are the files and directories it uses:

* attributes.js
* Default.png
* en.lproj
* iCal.js
* Icon.png
* Images/
* Info.plist
* main.css
* main.html
* main.js
* Parts/

Where the Parts/ directory stores Dashcode javascript libraries. The required components to a widget are just a subset of these files:

Info.plist - the plist file that tells a user of the widget how to run it.
Default.png - a preview image of the widget when it is not running.
main.html - or some html file to give it some functionality.

And most likely you’ll want one or more javascript files - as we are trying to make an interactive visualization.

So again, our goal is to create an interactive widget using d3.js, without the need for using Dashcode directly.

To this end, I’ve created a vis widget starting point that includes the necessary files to kick start just such a project.

Here’s what it’ll look like. Click for the actual interactive visualization. You can click and drag the circles around:

You can see, its not much to look at - but its something to start with.

This vis_widget repo includes d3.js as well as Twitter bootstrap css, jQuery, and some other handy files. The html and file structure is based on HTML5 Boilerplate

If you are familiar at all with Ruby and Bundler , you should be able to run:

bundle install
thin start

And then open up http://0.0.0.0:3000 in your web browser to view the visualization. This lets you develop your interactive graphic in the comforts of your favorite web-browser and only move it to iBook Author once you are ready to publish.

Looking at the js/vis.js code, you can see this is a pretty basic (i.e. useless) visualization. However it does prove out some important functionality:

Loading of a bundled data file using d3.csv works without any issues.
D3’s drag behavior works well in this environment.
Custom css and javascript libraries are possible inside an iBook Widget.

To create your own vis widget, simply fork or clone this repository, then modify index.html and js/vis.js to visualize what you want.

When you have a visualization you like ready, follow these steps:

Create your own Default.png by taking a screen-shot, or otherwise exporting a preview image of your visualization. ** Make sure to get the dimensions right (see below).
Tweak your Info.plist as needed (see below).
Run make in your vis_widget directory
Drag your newly created .wdgt file to iBooks Author

The Makefile provided is super simple. It just copies the vis_widget directory and moves it the vis.wdgt without modifying it. It should create a .wdgt package suitable for transferring to iBooks Author.

Once in iBooks Author, you can tweak the positioning and styling of the initial display of the widget - removing its title or description as necessary.

Info.plist

The provided Info.plist will get you pretty far into developing a widget for iBooks, but there might be a few things you want to tweak:

The CFBundleIdentifier you might want to modify to use your own name.

The Height and Width integers might need to be modified if you don’t want use the same size visualization as me.

Size Issues

When rendering the widget in an iBook, the sizing appears to be dependent on many factors. Not only does the width and height entries in Info.plist need to be correct, but the dimensions of your Default.png must match up with these dimensions.

When determining what size to display your widget in, it appears the system uses the smallest dimensions of either the Info.plist or the Default.png. That’s why its important that these both match - and they really are the total size of the visualization you want to use.

Interactive Textbooks with D3

Hopefully this is enough to at least get you interested in the potential behind interactive visualizations in iBooks.

You can download a copy of my iBook file that includes this D3 visualization.

Installing Galaxy Tools

2012-01-03T00:00:00+00:00

If you followed along with my galaxy server install guide you should have a basic but functional Galaxy server on your machine.

The next step is to get the tools that Galaxy depends on installed and working on this box. Galaxy’s wiki has an (incomplete) listing of tool dependencies . However, they don’t have a lot of information about exactly how Galaxy expects these dependencies to be installed / configured.

Galaxy Documentation Deficiencies

From discussions on the mailing list, and poking around in the xml files in galaxy-dist/tools, it looks like the most straight forward way to satisfy these requirements is to have executables accessible in your galaxy user’s PATH.

However, with tools that are packaged as jars (like picard ), the jar files must be present in galaxy-dist/tool-data/shared/jars (or a sub-directory of this directory - depending on the package). This caveat is not mentioned anywhere that I can find. It is discussed a bit on the Galaxy development forum , but it really is unclear to me how people installing Galaxy are supposed to find out about these critical but obscure details.

Also, in the Galaxy news from November 2010, there is mention of using an external <tool_dependency_dir> to hold env.sh scripts that will be sourced prior to running the command. However, there no documentation as to the best way to take advantage of these sourced files.

Because of the lack of documentation on how best to use this feature, I decided to take the simple route and get executables accessible from the PATH and symlink jar’s into their required directories.

Bio.brew

I think my fork version of bio.brew is one of the best ways to get started with the installation of these tools.

Bio.brew is a simple package manager, originally created by David Rio Deiros with a focus on bioinformatic packages. It is similar to homebrew - but without many bells or whistles, and designed to work on both Mac and Linux.

My branch of bio.brew simplifies some of the recipe creation process, and adds additional commands that I found useful (like ‘fake’ to pretend to install a prerequisite package). It also moves towards the idea of maintaining multiple versions of a particular tool, though currently this functionality is pretty basic.

So the idea is to get bio.brew setup, use it to install tools, and when necessary supplement the install with setup specific to Galaxy.

In the future, it might be nice to add Galaxy specific functionality to bio.brew to simplify the process even more.

Bio.brew setup

I think the setup for bio.brew is pretty easy - largely due to it being just bash scripts. As the readme states, simply clone bio.brew and add BB_INSTALL and BB_PATH variables to your environment and PATH.

Below are the commands I used to get bio.brew installed and configured. Tools will be installed in /usr/local/galaxy/tools (along with bio.brew itself).

# as galaxy user

cd /usr/local/galaxy
mkdir tools
cd tools
git clone git://github.com/vlandham/bio.brew.git

Then edit your .bashrc file to include:

# ...

export BB_INSTALL=~/tools
export BB_PATH=~/tools/bio.brew
PATH="$BB_INSTALL/bin:$BB_PATH/bin:\$PATH"
export PATH

This tells bio.brew to install packages into ~/tools and adds the bin directory to your PATH.

Bio.brew installing an executable

So, lets try out installing a required tool. samtools is required by many Galaxy tools - so it provides a good test case for the bio.brew installation process. With bio.brew, you should be able to install samtools with two commands:

bb install samtools
bb activate samtools

bb install pulls down samtools and compiles it, getting everything ready. bb activate modifies the bio.brew symlinks used to point to the current version of a tool to point to this new version. The reason it is split up into two commands is to allow for testing of a tool prior to adding it to your path.

Bio.brew installing jar’s

As mentioned above, we need to symlink jar files to a specific directory for Galaxy to know about them. Here we will install picard using bio.brew and add the .jar files to galaxy-dist/tool-data/shared/jars/picard so Galaxy knows about them. We will use the fake command to fake install java, as usually java is already installed on your machine and you don’t want to manage it through bio.brew.

bb fake java
bb install picard
bb activate picard

# now create symlinks

cd /usr/local/galaxy/galaxy-dist/tool-data/shared/jars
mkdir picard
cd picard
ln -s /usr/local/galaxy/tools/stage/picard/current/\*.jar ./

Restart Galaxy and Test

Now its time to see if this actually worked. First restart Galaxy using the init script we setup before:

sudo /etc/init.d/galaxy restart

Now try out the NGS: Picard -> FASTQ to BAM tool in your local Galaxy server on a test fastq file. If all goes well, you should see green all through your history. Enjoy!

Limitations

bio.brew currently doesn’t have all the Galaxy prerequisites in it - but that is something that can be fixed easily. Simply create a new install recipe for the missing tool and contribute it back to bio.brew!

Galaxy Server Install and Setup

2011-12-06T00:00:00+00:00

Installing Galaxy on Scientific Linux 6

Galaxy is a platform for managing, running, and sharing computational biology data and pipelines. Its a big idea, and a good idea, but a bit of a task to get working exactly how you might want it.

What follows is a brain dump of the initial setup and configuration of this tool. Sorry about the length.

Goals for sever configuration

Follow all recommended settings for production level Galaxy server
Use Nginx proxy front-end
- Enable as much of proxy components as possible
Use local postgreSQL server
Try to make these instructions as complete as possible

Future goals for follow up configuration include

Enable and configure as many of the tools as possible
Provide easy access to local directories of sequencing data inside Galaxy
Experiment with with Galaxy Toolshed
Experiment with cluster configurations

About the System

Scientific Linux release 6.1 (Carbon)

Will use separate galaxy user to run galaxy

Galaxy user’s home directory will be located: /usr/local/galaxy

Resources

Most information comes from apapow.net

And the galaxy wikis Production Server page

Check base install

Ensure python is installed and at 2.6

which python

# /usr/bin/python

python --version

# Python 2.6.6

Ensure PostgreSQL is installed

sudo yum install postgresql postgresql-server

Modify PostgreSQL config file

Tricky part to getting postgreSQL working is the pg_hba.conf file. Edit it to allow local connections.

sudo vim /var/lib/pgsql/data/pg_hba.conf

A blog post explaining the syntax of this file . It should look something like:

local all all trust
host all all 127.0.0.1/32 trust
host all all ::1/128 trust
host all all 0.0.0.0/0 md5

Startup PostgreSQL

sudo service postgresql initdb
sudo chkconfig postgresql on
sudo service postgresql start

Add galaxy user

Note

The galaxy user created here is not capable of using sudo. Every time sudo is used in this document, it is done from a sudo capable user.

Galaxy user’s home directory is at: /usr/local/galaxy

This was done because /home is a remotely mounted disk.

sudo /usr/sbin/useradd galaxy --home /usr/local/galaxy
passwd galaxy

Install dependency packages

# install git just to have

sudo yum install git

# install mercurial to download galaxy

sudo yum install mercurial

Install Galaxy

Switch to galaxy user

su galaxy
cd ~

Download galaxy

hg clone https://bitbucket.org/galaxy/galaxy-dist

Download virtualenv

wget https://raw.github.com/pypa/virtualenv/master/virtualenv.py

Create sand-boxed Python using virtualenv

python ./virtualenv.py --no-site-packages galaxy_env
. ./galaxy_env/bin/activate
which python

# ~/galaxy_env/bin/python

Configure galaxy user

Edit ~/.bashrc to define TEMP and to add virtualenv source

source ~/galaxy_env/bin/activate

TEMP=\$HOME/galaxy-dist/database/tmp
export TEMP

Ensure that ~/.bash_profile sources ~/.bashrc

# this should be in ~/.bash_profile

if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

Setup PostgreSQL database for Galaxy

sudo su - postgres

Use createdb to create new database for galaxy

createdb galaxy_prod

Connect to database using psql

psql galaxy_prod

Create galaxy user for PostgreSQL database

CREATE USER galaxy WITH PASSWORD 'password';
GRANT ALL PRIVILEGES ON DATABASE galaxy_prod to galaxy;
\q

Test galaxy PostgreSQL user.

Exit out of postgres user. Return to galaxy user and then attempt to connect to database.

Visualizing the Racial Divide

2011-10-20T00:00:00+00:00

Using d3 and force-directed maps to see the impact of segregation in cities

I wanted to create a visualization to highlight the impact of the deep segregation that still exists in many US cities today. I think the Choropleth maps like those in Salon’s most segregated urban areas slide-show are great in terms of showing the reality of this problem. I wanted to come up with a more visual and perhaps more visceral way to show these racial divides and how they serve as real boundaries in and around communities.

Force-Directed Maps

Originally, I thought about using elevation to show these breaks, similar to these topographical crime rate maps . This still might be interesting to experiment with, but I decided to try to stick to a 2D method.

Force-directed maps is a name you could give to a force-directed graph made up of neighboring components of a map. Mike Bostock has demonstrated them a few times using a map of the United States: both in his Trulia Talk and his SVG Open Keynote . Here’s his example:

I decided to use this concept to show how segregation in a city can ‘break it apart’. Here’s an example from Kansas City:

See animated visualizations for a number of cities here

The visualization uses d3.js and works best with Google Chrome.

Cities were picked from the list of most segregated cities, ordered from most to least.

About the Visualization

Data is from the 2010 Census, at the tract level. The links are hidden, but each tract is connected to each of its neighbors. The lengths of these connections encode the disparity between racial make-up between neighboring tracts. So, if a ‘mostly white’ tract is connected to another ‘mostly white’ tract, then the connection is short. If a city had uniform proportions of races in each tract, the map would not move much. However, longer connections occur where there is a sharp change in the proportions of white and black populations between neighboring tracts. These longer connections create rifts in the map and force areas apart, in some ways mimicking the real-world effects of these racial lines.

I’d love to hear your thoughts on these visualization style. Certainly it is not really meant to provide exact numbers. These maps only encode differences between white and black populations. It would be interesting to include other racial divides as well. The line length encoding also has some issues when there are few neighbors around with short connections. This can be seen in the smaller towns such as Wichita. Large cities like Chicago have too many tracts to keep the animation smooth.

Designing Data Visualizations Review

2011-10-07T00:00:00+00:00

Having just finished reading Designing Data Visualization by Noah Iliinsky and Julie Steele, I wanted to provide a short review of this book.

DDV’s Focus

DDV was made for people like me in mind: novices in the world of data visualization. From the preface, we read that the book’s purpose is to provide a high-level overview of the visualization landscape:

It will give you the general lay of the land. It is a set of steps and rules to follow that will get you 80% of the way to turning out great work.

I think this is a worthy goal - and one that this book is successful at.

At first, I was surprised a bit by the physical size of the book. In the same Amazon purchase, I also had grabbed Visualize This and Show Me the Numbers . Compared to these two, Designing Data Visualizations is tiny: roughly 85 pages of content, with another 10 pages of appendix. However, given the goal of this work - I believe the size is appropriate. The appendix includes a checklist to go over when designing visualizations. Much of this book then serves to elaborate, explain, and justify this checklist as a way of creating solid visualizations. I think when attempting to explain an entire framework for how visualizations should be created, succinctness is important, less the reader gets lost in the details.

DDV’s Content

Designing Data Visualizations is broken up into two parts. Part 1 is short, only two chapters, and serves to provide a vocabulary of visualization types and approaches to categorizing these types. The preface also includes a section entitled “What We Mean When We Say…” which is very reminiscent of the recent The Many Words for Visualization from Flowing Data . I found these definitions to be very useful in defining this vocabulary, and though some definitions are repeated in chapters 1 and 2, I wish they had fleshed out this list as a third chapter. It would have provided a more comprehensive ‘language of visualization’. As it stands, its still a good list.

Part 2 is where the meat of the book is, as stated in the Preface:

The goal is to help you think in a linear way about how to select and apply appropriate encodings for your data.

The encoding of data dimensions into visual components is something I have been really looking for more info on - and where Designing Data Visualizations shines. DDV provides a clean, systematic way of understanding, prioritizing, and applying the myriad of the possible encodings available to visualization designers.

Encoding of data is certainly a main topic of other data visualization books and papers. The Classic Paper Reading List from Fell in Love with Data , for example, includes Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods, which is mainly on encoding. And as Enrico summarizes:

[V]isual encoding is hard stuff and you shouldn’t take it too lightly. And that visual primitives do have a ranking that you have to take into account if you want to design effective data visualizations.

DDV takes these various encoding discussions, combines them, and presents them as part of a larger design framework. This framework for how to think and design visualizations is great and a process that can be immediately followed by beginners and customized or altered by more experienced data encoders. The framework encompasses a lot of topics. Some highlights include: using position as a primary encoding, the proper use of colors, and some pitfalls to avoid - especially when dealing with circle-based graphs.

DDV doesn’t spend a lot of ink talking about the specifics of different types of graphs, like bar charts, steamgraphs, etc (although a nice organized list is given). I think this is a good thing. Visualize This as well as texts like Ben Fry’s PhD Thesis already provide through introductions to chart types. Duplication of this effort of “chart enumeration” in DDV would have made many readers of this book glaze over a bit.

DDV’s Weaknesses

I have two small issues with the text. The first of which is the format that I purchased it in.

This is one of the first books I’ve read in a physical form where I thought it would have been better experienced in the digital version instead. I like reading on paper still, underlining, making notes and what-not (perhaps I should finally buy a Kindle). This book is printed in gray-scale only, but many of the example charts and graphs discussed in it don’t really make a lot of sense unless you are looking at the color versions. This is mentioned explicitly at the end of the prefix, with a link providing access to the full-color versions. However, jumping back and forth between a book and a screen is distracting and sometimes not possible (another reason I like books is to get away from my computer screen for awhile). This is why I would recommend getting the ebook version of this text - or even better - get the bundled version and have the digital copy at the ready for viewing graphics.

Secondly, I think they could have structured the reading list in the appendix a bit to provide more of a guided tour of where to go next in developing a strong data visualization design process. The goal of this book is to “start you down the path” of data visualization - which it does nicely. Now that you are on the path, however, what direction do you go at the crossroads?

That said, the reading list is still very valuable - and includes many more texts I plan on picking up. I just wish they had ended it with a nice big sign that said “Now Go Here for another 10%”. Perhaps the authors are planning a sequel…

Summary

All in all, Designing Data Visualizations is a great resource for individuals just getting started in the visualization world, and I would guess it will be valuable for experienced designers as well, as a way to organize their processes in the context of data design framework. The book is relatively short and to the point. No words are extraneous. The ink is maximized for content. I would recommend getting the digital copy, if possible, for the smoothest reading experience.

Thanks very much to Julie and Noah for this great resource!

From Shapefile to GeoJSON

2011-09-30T00:00:00+00:00

UPDATE - Feb 11, 2015 This is an old post, and needed some love. I’ve updated it a bit with the following changes:

New github repo - for the D3 visualization part at the end
Automatically determines the projection’s scale and translation (from a Bostock Example ) - so this should work easier for any geojson file generated.
JavaScript as well as CoffeeScript implementations for the D3 part.

Hope this helps keep the content relevant. Let me know if you see improvements that could be made!

A method for editing, merging, simplifying, and converting Shapefiles to GeoJSON

FYI - You might also want to check out Mike Bostock’s Let’s Make a Map post which covers a lot of the same ground.

D3.js supports cartographic visualizations by being able to display lines, polygons, and other geometry objects . It uses GeoJSON as the storage format for this type of visualization.

Likewise, map tiling libraries like Leaflet can use GeoJSON to create map layers .

You can find some GeoJSON files around, like in D3’s choropleth example , but sooner or later you will want to visualize a more specific portion of the world.

Specifically, I am interested in visualizing census data for my hometown: Kansas City. Kansas City is perhaps a bit unusual, from a census point of view, as it is actually two cities in two different states: Kansas , and Missouri . Also, I’m not really interested in the rest of these states - just the Kansas City metro area. So what I really want to do is merge multiple Shapefiles, cut out the interesting section, and convert this to GeoJSON. Making it as small as possible (in terms of filesize) in the process.

Below is how I created a small GeoJSON file of just the KC metro from census shapefiles.

WARNING: I am not an expert in GIS or cartography in general. This may not be the best way to get a custom GeoJSON. In fact, cutting up shapefiles into custom GeoJSON files might not even be a good idea in the first place. However, if you want to look at small piece of the world in D3 or other GeoJSON capable tools, this might be a way to start.

Tools

I’m using a mix of different applications to perform this conversion:

Quantum GIS is used to deal with shapefiles.
- NOTE: I ran into a few issues trying to get QGIS installed on my Mac - though a Mac version is available. So instead of fighting this, I just used the Windows version on an XP machine. This all should be possible using just the Mac version, but the screenshots will be of Windows.
MapShaper a free online tool is used to simplify the Shapefile and reduce its filesize.
GDAL is used to convert shapefiles to GeoJSON. I was able to install this easily on my Mac using homebrew:

brew install gdal

Initially I did run into an issue with homebrew when it was installing the geos prerequisite. Following the instructions on this github issue fixed this. GDAL would also work on a Linux box, and there looks to be a Windows port as well.

Getting the Data

Census.ire.org provides a way to download census 2010 data, including shapefiles of the tracts / counties the data is for. I downloaded Shapefiles for Kansas and Missouri at the Census Tract Summary Level. The files together are over 10MB in size, and transforming them into GeoJSON at this stage creates over 30MB of data - too much to render in the browser all at once. Certainly there are more sophisticated ways to deal with this issue, but for the time, I just want to explore some KC data quickly.

Merging Shapefiles

Open up Quantum GIS and select Add Vector Layer as shown below:

Click Browse under Source and select your Shapefile. Repeat if you have multiple shape files. Here’s Mine with both Kansas and Missouri open:

Merging these layers was tricky (for me). I ended up needing the MMQGIS plugin for merging. To install this, first select Plugins -> Fetch Python Plugins. In the Repositories tab, I selected Add 3rd party repositories. When it adds the plugin repositories, it seems to get stuck on one of them. Clicking Abort will let you continue without this repo. In the plugins tab, search for MMQGIS, select it, and then click Install plugin. Then you should be good to go.

With this plugin installed, you will have a new menu item under Plugins called mmqgis. Navigate here, then select Merge Layers:

Select the Shapefile layers you want to merge then select a location to save the merged Shapefile. After this, you should have an additional Shapefile with all the geometry from your multiple Shapefiles in one file.

Selecting a Portion of Shapefile

Ok, maybe your region doesn’t exist in multiple Shapefiles, but some times you don’t want to deal with an entire state’s worth of data with visualizing. Use the “Select features by rectangle” tool to highlight the portion of the merged Shapefile you are interested in. Holding down Ctrl allows you to add / remove individual sections as well. Once you are satisfied with your selection, right click on the merged Shapefile in the left-hand Layers table, and select Save selection as :

And presto! You should now have a Shapefile of the smaller selection saved on your system. You can open it back up with Quantum GIS, but for the rest of the tutorial, we will be done with this application.

Simplifying a Shapefile

To reduce size further, you might want to simplify your Shapefile. This looks to be possible in Quantum GIS, but I wanted to try out MapShaper which is recommended in the D3 documentation. Click launch, and upload the .shp file from the output of your Quantum GIS work.

I don’t have a good idea of how simplified you can go, but for Census Tract Shapefiles, I didn’t see much degradation even at 50% simplification level. When happy, click export in the upper right-hand corner and choose Shapefile - polygons. Download the two resulting files.

NOTE For the next step, we want to maintain the ‘properties’ section populated by data from the Shapefiles. Do ensure this section isn’t empty, replace your .shp and .shx files with these simplified versions, and don’t remove the rest of the files. This is to say delete your original .shp / .shx files and rename these simplified versions to match your old ones. Let me know if that doesn’t make sense.

Converting to GeoJSON using GDAL

Transfer these files to your Mac / Linux machine. You should have a folder with all the Shapefile files. Now we want to create a GeoJSON file for D3 to use from these files. Using the ogr2ogr command to do this is pretty straightforward. It would look something like:

ogr2ogr -f geoJSON kc.json kc.shp

Note that the output file name comes before the input file name.

You should now have a GeoJSON file of your custom Census Tract section, ready to be used in D3.

Here is my KC Metro GeoJSON file . You can see (if click on a tract) that the properties object maintains the information necessary to map the GeoJSON feature back to a Census Tract.

Display in D3

Update Feb 11, 2015 Now this code has its own github repo: d3 geojson example

As a test, I wanted to display my new GeoJSON file in D3. Make sure to include d3.geo.min.js in addition to d3.min.js for the path functions. Below is the minimum amount of code I used to display this GeoJSON file using D3 and coffeescript (and I now have a JavaScript version implemented in the repo ). This assumes that there is a div with the id of vis in your html .

This is mostly the same code as the D3’s choropleth example . This now figures out the translation and scale automatically, thanks to Mike once again.

I also use the “STATEFP10” property of each tract to determine the fill color. This results in different colors for Kansas and Missouri, to show that these properties are present in the simplified GeoJSON. Here’s an image of the result:

This is certainly just a start to the possibilities of using D3 and geojson. As mentioned at the top, Mike Bostock also has an amazing map making tutorial that focuses on the benefits of topojson - so check that out!

Recreating Old Visualizations with New Technology

2011-09-26T00:00:00+00:00

Using d3 and coffeescript to reproduce old charts

Recently, I stumbled upon a set of statistical charts from the 1800-1900’s in the David Rumsey Map Collection . While perhaps not the best forms of data visualization by today’s standards (pie charts, stacked bar charts, etc), they serve as beautiful inspirations visualization techniques due to their organization, attention to detail, color scheme, and presentation.

Here’s an example from the collection:

Based on the mantra’s of Enrico Bertini on Fell in Love with Data and perhaps also the guidelines of creativity from Everything is a Remix, I wanted to try to recreate (copy) a small selection of these charts using modern tools.

I choose to try out d3.js for this experiment. This is a javascript framework that is the official successor to the popular Protovis visualization toolkit. d3 allows you to represent your data with native SVG elements, which provides a lot of power in terms of layout and style. Furthermore, to make writing javascript more fun, I wrote the visualizations in coffeescript - a language that compiles down to javascript.

The process of recreating these graphs was a lot of fun and results are in my visualization section . Here’s an image of my d3 version of the example above (click for the actual d3 version):

Though I only have a few examples (3 right now, but I hope to have time to make more), I believe it shows some of the potential power of d3 in terms of customizing exactly how you want your visualizations to look. In the future, I might look at adding functionality like interactions and transformations of the raw data - all of which should be pretty easy with d3.

RNA-seq: Comparing Similar Libraries

2011-08-10T00:00:00+00:00

This task involves comparing RNA-seq data generated from 3 similar libraries. The difference between the libraries is the number of cells collected to be used in the construction of the libraries. This dataset contains 2 samples each from libraries constructed from 1,000, 10,000 and 30,000 mouse bone marrow derived hematopoietic stem cells.

The idea, as it was explained to me, is to see if researchers can get away with small cell numbers when constructing libraries for use with RNA-seq. Specifically, what do you loose in your data as you use lower and lower cell numbers.

Not a very concrete question, but perhaps a good one all the same.

Analysis Goals

Discussing with other analysts about what to do, Here are some questions I’d like to answer during analysis:

What is the percent FPKM change as we move from higher number of cells to lower number?
How does gene coverage change from high to low?
- Does coverage overall go down with fewer cell numbers?
- Are there spots in the exome that are lacking in the low cell number samples that have coverage in the higher cell number samples?
How do the numbers of multi-reads change?
- Multi-reads are those reads that map to multiple places on the genome. Does the percentage of the total number of aligned reads that are considered multi-reads increase in low cell number samples?
Does the absolute number of genes covered at all by the data change?
- Here we would like to see if more cells provide more information in terms of the number of genes found in the data. We might expect that the lower cell numbers would have more genes missing from the RNA-seq data, due to lower coverage.
- Information could be graphed with cell numbers on x-axis, and on the y-axis, show the percentage of the total number of genes of the organism that were detected in the data.

Tools Used

It is my attempt to get a cohesive picture of each of these datasets, so as to better understand what is different about them. So I used a number of tools that can provide various details about the dataset, along with the normal alignment and RPKM tools.

TopHat v1.3.1
Cufflinks v1.0.3
Samtools v0.1.16
picard v1.49
BEDTools v2.12
fastqc v0.9.3
FASTX-Toolkit

Pipeline

The initial process is similar to my previous RNA-seq analysis. However after alignment and RPKM values are determined, we need to do a lot more analysis to see patterns between datasets

fastqc for initial read quality
Align with TopHat
Cufflinks for RPKM values
picard for alignment metrics
BEDTools to discover gene coverage

I’ll avoid explaining all the parameters again for the tools used, but go over how they were used in the analysis.

Removing Duplicates

As before, no duplication removal was performed. It seems that this would remove information that could be valuable to this particular analysis.

Trimming

The initial look at the data using fastqc indicated serious degradation of the quality of some of the samples’ reads after 25bp. This is unfortunate, and perhaps indicates that the analysis with not be as fruitful as one would like. However, the analysis must go on! FastX was used to trim the samples from all libraries to 25bp. This deals with the bad quality of some of the reads, and prevents biases that would be present for longer reads, if all of the samples were not trimmed.

Here’s how Fastx was called:

zcat sequence.fastq.gz | fastx_trimmer -f 1 -l 25 -z -o sequence.trim.fastq.gz -Q33

As we are using CASAVA 1.8, everything comes compressed. So zcat is used to decompress on the fly. The -z option re-compresses the trimmed reads, so we can keep using .gz files. The -Q33 option is required to indicate the type of quality score used. CASAVA 1.8 switches to using the standard Sanger 33-offset-based quality scores. At the time FastX couldn’t figure this out, so you have to tell it. This is a undocumented parameter that I found out about from this SEQAnswers post .

TopHat

Most of the defaults were used. Because of the short reads, Tophat was complaining that the --segment-length parameter should be made to be equal to roughly half the read length. So that’s what I did. Also, The gene model annotation file (.gtf) was provided so that it wouldn’t try to figure out junctions de-novo.

tophat -G /n/data1/genomes/bowtie-index/mm9/Ens_63/mm9.Ens_63.gtf -p 4 \
--segment-length 12  -o tophat /n/data1/genomes/bowtie-index/mm9/mm9 sequence.trim.fastq.gz

Where the -G flag provides the gene model annotation, and the second to last parameter is the path to the bowtie index files.

Cufflinks

Similar to TopHat, most of the defaults were used:

cufflinks -p 4 -o cufflinks -G /path/to/GTF/file.gtf accepted_hits.bam

Where the input BAM file for each sample was the output produced by TopHat.

picard

I wanted to see what information picard’s CollectAlignmentSummaryMetrics tool would provide, so I ran this on all the bam files. Picard and GATK can be particular about the order of the bam files. So, first I used picard’s AddOrReplaceReadGroups and it’s ReorderSam to get the bam files in a state where they would be taken by picard without dying:

java -jar picard/AddOrReplaceReadGroups.jar INPUT=accepted_hits.bam OUTPUT=accepted_hits.group.bam \
VALIDATION_STRINGENCY=LENIENT SORT_ORDER=coordinate RGLB=1 RGPL=illumina RGPU=1 RGSM=name

java -jar picard/ReorderSam.jar INPUT=accepted_hits.group.bam OUTPUT=accepted_hits.group.reorder.bam \
VALIDATION_STRINGENCY=LENIENT REFERENCE=mm9.fa CREATE_INDEX=true

java -jar picard/CollectAlignmentSummaryMetrics.jar INPUT=accepted_hits.group.reorder.bam \
OUTPUT=alignment_summary.out.txt REFERENCE_SEQUENCE=mm9.fa

All that work, and what did I get? Not much. This tool seems like it could be more useful for paired-end data with reads that don’t pass filter in it. It does provide the total number of reads in the bam file, but this would be easy to acquire by other means.

BEDTools

BEDTools has a coverageBed program that I thought could help shed light on how well the various libraries covered each gene. I used a bed file we have containing exons to provide it the locations of the exons.

**Note on Exon Bed File** There are some particulars about this bed file that other analysts helped me understand and generate. First, There are a number of exons that overlap one another. Keeping them all would inflate the perceived coverage of those areas where a lot of overlapping exons existed. So, BEDTools mergeBed tool can be used to squash these overlapping exons. However, then you hit another problem where overlapping exons that belong to separate genes will get squashed together. We don’t want this to happen, as we are also trying to show the number of genes hit by the data. So, the solution we have here is to break up the bedfile based on genes, then squash exons at a per-gene level, and then merging the exons back together. Not sure if there is an easier way…

coverageBed -s -split -abam ./accepted_hits.bam -b ./mm9.Ens_63.exons.bed > ./coverageBed.out.txt

Analysis

Coverage

As mentioned above, I used BEDTools to get coverage information of the exons covered for each sample.

The command executed should provide an output file that contains a line for each line in the input .bed file. Each output line starts with the corresponding bed file line and then ends with a few more tab-delimited fields:

Number of reads in A (the bam file) that overlap the B (the bed file) line’s interval by at least one base pair.
Number of bases in B that have some coverage from A.
The length of the interval of B
The fraction of bases in B that have some coverage from A.

From this run, an example would be:

chr1    59768115        59768968        exon281764      1       +       172     342     853     0.4009379

So, for this particular exon, we can see that of the 853 bases in the exon, 342 of them were overlapped by one or more of 172 reads from the bam file. This means about 40% of the exon was covered by the reads, with a maximum coverage of 172… right?

Further analysis of this output was performed with R. Eventually, I extracted from these bed files:

Total number of exons that were covered by some amount of reads from the sample.
Number of exons per chromosome that were covered.
Percentage of all exons that were covered by sample (both for the whole sample and by chromosome).
Total number of unique genes that were covered by some reads.
- Found by getting the unique gene names from the exons list.
Percentage of unique genes covered by reads.

I’m unsure as to what a meaningful criteria is for the “some amount of reads” coverage stipulation. Should it be at least X number of bases covered by reads, Y number of reads on the exon, or Z percentage of the exon covered?

With the coverageBed output, I have access to all three options. The trends look to stay the same, regardless of the selection - but the specific numbers change as you would expect.

I analyzed each sample separately, even though there was two samples per library. The differences were enough to be potentially interesting.

I also then combined the coverage datasets per library to show a more aggregated form of the data.

Expression Values

Next, I turned to the FPKM values generated from Cufflinks to see if differences could be seen in the various cell numbers.

First I looked at the covariance between the samples to see if samples from the same library were similar, and if the expression values from different cell numbers looked different. I borrowed code from another analyst that displays covariances between samples as a matrix of color squares, where the color indicates the correlation.

RNA-seq: Finding Differentially Expressed Genes

2011-05-27T00:00:00+00:00

Here are some notes on processing RNA-seq data to find differentially expressed genes.

A much abbreviated version of this pipeline comes from Cufflink’s Tutorial Page

The extended version given below follows the more exploratory pattern of someone who hasn’t done much (any) NGS data analysis before.

Tools Used

TopHat v1.1.2
Cufflinks v1.0.1
Samtools v0.1.16

Requirements

Sequence Files

We are starting from Illumina generated Fastq sequence files. The reads contained in these files are paired-end and 40bp long. We are using mouse data. The naming convention follows the format: s_LANE_PAIRED_sequence.txt. So s_1_1_sequence.txt is the “left” side of the paired-end data on lane 1 and s_1_2_sequence.txt is the “right” side.

Pre-built Index

There are a number of pre-built indexes that TopHat makers provide . For the mouse genome, they have “m_musculus_ncbi37”, “mm8”, and “mm9” currently listed in their ftp directory.

We also have our own indexes, in our genomes directory. The index used in the initial run was labelled Mus_musculus.NCBIM37.52.

Gene Model Annotations

We also want to supply TopHat with a list of gene model annotations. This is in GTF v2.2 format, and the chromosome names must match those in the index. For the initial run *Mus_musculus.NCBIM37.52.gtf was used.

Pipeline Run

The pipeline looks to be pretty simple:

Sequences ~~> TopHat~~> Manual Check ~~> Cufflinks~~> Analysis

Removing Duplicates

No duplicate removal step was performed. Pros and Cons that this seqanswers post seems to address well. Bottom line is that it is impossible to tell if a duplicate read really is from pcr bias. Removing them causes a loss of information and location based removal methods don’t work with RNA-seq data.

TopHat

TopHat Configuration

There are some settings to get right for TopHat. Here are the flags and the initial values I used.

Average Mate-Pair Inner Distance

`-r/--mate-inner-dist`

This is a required setting for paired-end data with no default value. For this kind of setting, I would expect them to have a table or something that provides some suggestions on how to get this right.

Also, when speaking to associates, it sounds like their formula incorrect - or at least misleading. Here, I’ll try to explain it a bit more clearly as how we see it.

We are trying to provide TopHat the average distance between the ends of the paired-end reads. So, we have our library which is composed of fragmented cDNA (the DNA insert), ligated to adapter sequences on both sides. This library is size selected after the adapters are ligated to the DNA insert. So, a 300bp library would have 300bp - Adapter_Length size inserts.

The trick is that during sequencing, sequencing starts at the DNA insert - not at the beginning of the adaptor. You can see that this is the case from Illumina documentation. For example in the Cluster Generation step shown here - it ends with the sequencing primer being hybridized to the DNA-templates. So, in order to figure out the distance between pair-end sequences, we need to subtract the Adapter_Length from the library size. There are adapter sequences on both sides of the DNA insert, so we need to subtract from both sides as we are reading from both sides. The total length of the Left_Adapter + Right_Adapter should suffice - this is what I’m calling Adapter_Length.

Mate-pair Distance Diagram

p=.

In total, to determine a good value for -r we need:

The size-selection target for the final library (usually around 300bp)
The read length (here 40bp)
The adapter length (for TruSeq adapters this looks to be 120bp or 121bp. For other paired-end adaptors, it is 98bp.).

The formula then becomes:

Inner_Distance = Library_Size - (2 \* Read_Length) - Adapter_Length

Or, for our specific run:

315 - (2 \* 40) - 121 = 114

315 was the reported average size of the library for this particular sample.

I didn’t fully understand all this on my initial run of the TopHat. So, my -r was set to 122 - which is close. I re-ran TopHat using the same parameters, but with an -r of 114, to see if there are any noticeable differences in the mapping quality.

Gene Model Annotations

`-G/--GTF`

Full path to the .gtf file for mouse that matches the index file you will be using (see above). In my case, Mus_musculus.NCBIM37.52.gtf. This is on a separate file system, but that does not seem to be a big deal.

Threads

`-p/--num-threads`

You would think that these software packages would settle on a single unified flag to indicate the number of processes to use - but you would be wrong. I’ve tried running with 4 and 8. Details on run times for these choices below.

Output

`-o/--output-dir`

As all the files generated by TopHat have the same name from run to run, it is best to use this option to put them into their own directory.

TopHat Run

The total TopHat execution string looks something like:

tophat -r 122 -G /full/gtf/path/Mus_musculus.NCBIM37.52.gtf \
-p 4 -o unique_output_dir/tophat/ \
/full/index/path/Mus_musculus.NCBIM37.52 s_1_1_sequence.txt s_1_2_sequence.txt

I would also ensure that your output directory exists prior to running. I do not know if TopHat creates this directory or exits out if it is not present.

With -p 4 this took around 13 hours to complete. With -p 8 it took ll hours.

TopHat Run Analysis

I wanted to have some level of confidence that the output produced by this run was good. The log files produced by TopHat are undocumented and less than helpful.

samtools flagstat

Online, I found some posts using samtools flagstat to look at sam/bam files.

I don’t understand why a tool that everyone seems to know about is not mentioned at all in the samtools manual page

Running samtools on the command line with no arguments provides a short help section. Here flagstat is listed with the description “simple stats”. Very helpful.

Here is the output for the first TopHat run that used 122bp for the -r parameter:

\$ samtools flagstat accepted_hits.bam

102925213 + 0 in total (QC-passed reads + QC-failed reads)
0 + 0 duplicates
102925213 + 0 mapped (100.00%:nan%)
102925213 + 0 paired in sequencing
51854282 + 0 read1
51070931 + 0 read2
65032742 + 0 properly paired (63.18%:nan%)
96061406 + 0 with itself and mate mapped
6863807 + 0 singletons (6.67%:nan%)
0 + 0 with mate mapped to a different chr
0 + 0 with mate mapped to a different chr (mapQ>=5)

Results from the second run with -r 114 were similar:

102918381 + 0 in total (QC-passed reads + QC-failed reads)
0 + 0 duplicates
102918381 + 0 mapped (100.00%:nan%)
102918381 + 0 paired in sequencing
51850866 + 0 read1
51067515 + 0 read2
63594438 + 0 properly paired (61.79%:nan%)
96054574 + 0 with itself and mate mapped
6863807 + 0 singletons (6.67%:nan%)
0 + 0 with mate mapped to a different chr
0 + 0 with mate mapped to a different chr (mapQ>=5)

Since -r 122 performed a bit better, I used it for the rest of the samples

I am not sure as to how useful this output really is. The 100% mapped sequences would indicate that it is not really looking at the whole picture.

From a flagstat biostar post - we see this information about the “properly paired” section:

“Properly paired” is from the flags set by whatever aligner you’ve used. For instance, if you use bwa’s ‘sampe’ alingment, you can specify the maximum insert length with -a. For a pair to be “properly paired” it would need to have both reads mapped to the same sequence within this distance.

So the properly paired section is (perhaps) an indication of how well our choice of the mean inner distance worked for this data (63% not being that great?).

To clarify, the flag that it looks to be counting is 0x2 in the sam/bam file. From the SAM format description this flag is described as “each fragment properly aligned according to the aligner”

samtools view -c -f 0x2 accepted_hits.bam

# 65032742

# same number as above in properly paired

According to an unanswered SEQAnswers question , TopHat might not be setting this flag correctly or perhaps this flag is used differently in TopHat. I haven’t found any answers from the TopHat developers, and I haven’t looked in the source code yet.

samtools view

The main question I would like to answer about this data is how many of the raw sequences were aligned and made it into the accepted_hits.bam file. Fastq files have 4 lines per sequence. So, to find the total number of sequences, one can use wc -l and divide by 4:

wc -l s_1_1_sequence.txt

# 188366840

# 188366840 / 4 = 47091710

# multiply by 2 to get

# 94183420 reads from both pairs

Then we need to know the number of unique alignments in the accepted_hits.bam file. It is possible for one sequence from the fastq file to align multiple times.

The accepted_hits.bam have collapsed together the two pairs, so the last component of the sequence names - the /1 and /2 are missing. (Look at head s_1_1_sequence.txt, then look at the first few values of field 1 of samtools view accepted_hits.bam and you will see the name difference). Hence, the total number will be 2 times the number of unique names found in the .bam file. Here is a piping of commands that should pull out the name of the read and then count unique names to determine how many unique read were aligned:

samtools view accepted_hits.bam | cut -f 1 | sort | uniq | wc -l

# 45153786

# 45153786 \* 2 = 90307572

So, from all this, we find that the total number of reads from the original sequence files that made it to the accepted_hits.bam file is:

90307572 / 94183420 = 0.958 = 96%

fastqc

It is also possible to run fastqc on .bam files. I did that for the accepted_hits.bam file, but didn’t get too much from the resulting output.

Increased `-r` parameter

As a final experiment, I used -r 200 to see what difference it would make (as a decrease lowered the % properly paired). The flagstat results did show an increase in the number of properly paired sequences:

103022448 + 0 in total (QC-passed reads + QC-failed reads)
0 + 0 duplicates
103022448 + 0 mapped (100.00%:nan%)
103022448 + 0 paired in sequencing
51902899 + 0 read1
51119549 + 0 read2
72500578 + 0 properly paired (70.37%:nan%)
96158642 + 0 with itself and mate mapped
6863806 + 0 singletons (6.66%:nan%)
0 + 0 with mate mapped to a different chr
0 + 0 with mate mapped to a different chr (mapQ>=5)

However, I get the same number of unique mappings from the samtools view:

samtools view accepted_hits.bam | cut -f 1 | sort | uniq | wc -l

# 45153786

So we will stick with -r 122

Cufflinks

Initially at least, I am only looking for differentially expressed genes, and not attempting to discover novel genes and transcripts, so I didn’t think I would need Cufflinks. However, it does provide raw FPKM values - which I found useful when trying to compare the expression values of the sample replicates.

I ended up running Cufflinks after I did the analysis with Cuffdiff to go back and show that the replicates were similar. The parameters I used for Cufflinks were pretty much the same as the ones used for cuffdiff.

Cuffdiff

From the cufflinks manual

Cufflinks includes a program, “Cuffdiff”, that you can use to find significant changes in transcript expression, splicing, and promoter use.

Cuffdiff Configuration

Cuffdiff requires just a few inputs

GTF file

`<transcripts.gtf>`

This is the same gene annotation file that is provided as input with the -G flag to TopHat.

Output directory

`-o`

Same as TopHat, quarantine all the output to a uniquely named directory.

Number of Threads

`-p`

Same as TopHat, number of processes to use.

Labels

`-L/--labels`

Comma separated input to name each sample with.

Sample Bam files

`<sample1.sam>`

Bam files produced by TopHat runs

Cuffdiff Run

For this experiment, we have two wild type sample replicates and two mutant sample replicates. Thus, we call Cuffdiff like:

cuffdiff -p 4 -o unique_output_dir/tophat/ -L wt,mut \
/full/gtf/path/Mus_musculus.NCBIM37.52.gtf wt_1.bam,wt_2.bam mut_1.bam,mut_2.bam

And that was it. It produces a set of files - including gene_exp.diff which was the main output that I used for analyzing gene expression differences.

Analysis

Some quick notes on the limited analysis I performed on this data. Most analysis was done using R:

Replicate Analysis

First I wanted to show that the data was good. This included comparing the gene expression profiles of the two replicates for each sample. I normalized the FPKM data by taking the log2 of these values. I also removed values that were marked as having problems from TopHat (one of the columns in the gene_exp.diff has an “OK” if things were ok, and other values if things went wrong. I used only “OK” entries - which proved important when trying to compare replicates), and genes that were not expressed in both replicates.

I produced a lattice of scatterplots that showed correlation between each sample. I also produced a correlation matrix and the resulting correlation values that showed the replicates to be highly correlated.

Differential Expression Analysis

First I plotted the log2 values and highlighted those that cuffdiff thought were statistically significant (Up-regulated in red. Down-regulated in blue). Cuffdiff listed over 8,000 genes as noteworthy, so most of the rest of the analysis here was spent cutting that down to smaller chunks.

I used the log2 values and took only those genes with a 2-fold or more change. And then cut this down further by specifying to only look in the region we were interested in.

The End

Hopefully this can help others just getting started with RNA-seq data analysis. We hope to evolve this process and get good at finding biologically interesting/relevant changes for the researchers we support.

Really Quick Git

2011-05-17T00:00:00+00:00

An attempt to quickly demonstrate the usefulness of Git for the single-developer environment

What does Git give you as an individual developer?

History of changes you made to your files.
- Allows you to see when particular changes / features were implemented.
- Also can provide an explanation to why you changed something the way you did.
‘Universal undo’
- Revert changes to any file you are tracking in Git.
- Completely external to the particular editor you are using at the time.
- Useful when a rm goes bad, or when you get crazy with Vim (“:d20j :wq”).
Test out new changes / ideas without fear.
- Branching capabilities in Git make it easy to try out large scale modifications to your code that you can always revert if things don’t work out.
- Also could be useful if you need to have a number of versions of your code with small tweaks in them.
Tagging allows you to know exactly how your code looked at a particular revision
- Useful for when you need to know if a bug / feature was present in the code at version x

What a central Git repository (like Github) gives you

Easily share code in a consistent, familiar manner.
Provide a backup of your source code for when your harddrive crashes.
A quick place to share code with yourself on multiple computers.
- Keep working (and maintaining a history of your work) no matter where you are.

Worth it?

I think it can be hard to prove that the added benefit you get from a version control system is worth the effort it takes to use it. Hopefully I can show that even with just a minimal amount of overhead, Git can be an valuable tool.

Assumptions

You are on a *nix based machine
Git is already installed on your system

Initial Setup

There are a few steps that are good to take care of up front, to avoid frustrations later on. We will just execute these commands without too much explanation as to the details of what they do. Commands are executed in the terminal.

Set your get name and email

git config --global user.name "Jim Somebody"
git config --global user.email "jims@domain.com"

Turn on color for your git output

git config --global color.ui true

Add global gitignore file for temporary files

# create .gitignore in home directory

touch ~/.gitignore

# add some files patterns to ignore

echo "\*.swp" >> ~/.gitignore
echo ".DS_Store" >> ~/.gitignore

# tell git about our global ignore

git config --global core.excludesfile ~/.gitignore

# you can add to this .gitignore file later

# and each git project also can have its own

# specific .gitignore

Ok, that takes care of most of the setup stuff. Lets get to using Git.

How Git works - the basics

Here’s a quick rundown of the basic points on how Git operates.

You Commit Changesets to a Git repository

Git works with groups of changes to your files, which we will call changesets. A changeset can include adding / removing text inside a file and / or adding / removing entire files. A changeset is commited to the git repository to track the changes. The idea here is that each changeset should optimally contain changes for a single logical modification to your code.

So, for example, if you codebase is a command line tool, a possible changeset would be adding the ability to specify an output directory. You would commit all the file modifications to complete this change as a single changeset. Another changeset could be the creation of a new class to represent ‘xyz’. That would go into its own changeset and that changeset would be committed to the repository.

You can work at a lower or higher scope than this, but I belive working with individual logical changes as individual changes allows you to get the most from your repository.

Git has a Staging area to Prepare your Commits

As changesets can and often do include multiple files, you need a way to group your changes before committing. That’s where the staging area comes in. The staging area is a holding place for changes in a changeset that will be committed.

This makes committing a change a 2 step process:

First, a change is added to the staging area
Then, the set of changes in the staging area (your changeset) is committed.

This makes it easy to have lots of modified files, but pick and choose which changes go into individual commits. Without a staging area, you would have to either only work on a single changeset at a time, or be forced to combine modifications from different logical modifications into one large commit. Both of these options are no good - hence the staging area.

Working with Git

Enough of the theoretical, lets start using it.

Initialize Git Repo

The first step is to create a Git repository. You can do this in a directory that already has files in it that you now want to track, or you can do this in a clean directory as a first step to starting a project.

cd /path/to/directory
git init

You now have a git repository in your current directory capable of tracking changes to all files in this directory and any sub-directory of this directory. It current is not tracking any files.

Preparing for the First Commit: Excluding Files

Usually, you will git init in a directory that already has files in it. Lets assume that is the situation for the rest of this article.

In this situation, it is beneficial to add all current files at once to git, and use this as your first commit.

But before we do that, lets make sure that there aren’t any files we want to not have tracked in git. Usually, files that are automatically generated and temporary files are good candidates for files we don’t want to track.

Lets say that there is a directory called temp that we don’t want to track changes to. To tell git about this directory, we add it to the repositories .gitignore file. So create this file and add temp to ignore:

touch ./.gitignore
echo "temp/\*" >> ./.gitignore

Notice that we actually give a file matching pattern that says “ignore all files and folders inside of temp”. Git automatically ignores empty directories, and so Git will ignore the temp directory as well as we have made it ‘empty’ by ignoring all the files inside it.

First Commit

Ok, now we are ready to commit our current set of files to git.

First, its nice to see the status of git using git status

git status

# output with all files listed in 'untracked files' section

git add

Like I mentioned above, committing in git is really a 2 step process: add and then commit. This allows you to be very particular about which files go into a specific commit.

As this is our first commit, we want to add all the files to our repository. This can be done a number of ways, but I like the following method:

git add .

And that’s it. See what you just did with git status

git status

# output with all files in the 'Changes to be committed' section

git commit

Now we are ready to commit these changes to our repository. Each commit in git is given a “commit message” which describes what these changes are about. You can make these messages as long as you want, but for individual developers, its usually easiest to provide short but meaningful comments on what you changed. You can provide this short style comment using the -m flag when committing.

git commit -m "initial commit"

And that’s it! Your new files have been added to git to track. Check git status and you will see that there are no new changes.

Git Workflow

With just a few commands, you can quickly utilize git to track changes, try out new experimental changes, and figure out why or when things changed.

Adding changesets to git

By far the most common procedure is checking in new commits to git. Here is the general workflow for this process:

git status

# see what changes have been made

git add file1
git add file2

# add particular files that have been modified as part

# of the current change you are working on

git commit -m "added xyz feature for abc reason"

# commit with comment

So just 3 commands: git status, git add, and git commit. If you aren’t sure what you did to a particular modified file use git diff file_name to get a quick diff of the uncommitted changes.

Bioinformatic File Cheatsheet

2011-05-13T00:00:00+00:00

Some quick notes on common file formats used in the bioinformatic world I am now part of.

GFFv2

Apparently there are 3 different versions of GFF from 2 different sources. GFFv2 and GFFv3 are the most interesting currently.

Name

General Feature Format (v1,v2)

Format

Nine required fields that must be tab-separated.

seqname - The name of the sequence. Must be a chromosome or scaffold.
source - The program or database that generated this feature.
feature - The name of this type of feature. Some examples of standard feature types are “CDS”, “start_codon”, “stop_codon”, and “exon”.
start - The starting position of the feature in the sequence. The first base is numbered 1.
end - The ending position of the feature (inclusive).
score - A score between 0 and 1000. If no score, then ‘.’ is used
strand - Valid entries include ‘+’, ‘-’, or ‘.’ (for don’t know/don’t care).
frame - If the feature is a coding exon, frame should be a number between 0-2 that represents the reading frame of the first base. If the feature is not a coding exon, the value should be ‘.’. Field should be called “Phase”
attribute - not-well defined field with tag value ; structure

Application Support

UCSC Genome Browser

Source

GFF v1 & GFF v2 come from The Sanger Institute

GFFv3

Name

Generic Feature Format (v3)

Format

9 fields separated by tab characters

Application Support

SnpEff - for new genome creation

Source

GFF v3 comes from The Sequence Ontology Project

Example

##gff-version   3
##sequence-region   ctg123 1 1497228
ctg123 . gene            1000  9000  .  +  .  ID=gene00001;Name=EDEN

ctg123 . TF_binding_site 1000  1012  .  +  .  ID=tfbs00001;Parent=gene00001

ctg123 . mRNA            1050  9000  .  +  .  ID=mRNA00001;Parent=gene00001;Name=EDEN.1
ctg123 . mRNA            1050  9000  .  +  .  ID=mRNA00002;Parent=gene00001;Name=EDEN.2
ctg123 . mRNA            1300  9000  .  +  .  ID=mRNA00003;Parent=gene00001;Name=EDEN.3

ctg123 . exon            1300  1500  .  +  .  ID=exon00001;Parent=mRNA00003
ctg123 . exon            1050  1500  .  +  .  ID=exon00002;Parent=mRNA00001,mRNA00002
ctg123 . exon            3000  3902  .  +  .  ID=exon00003;Parent=mRNA00001,mRNA00003
ctg123 . exon            5000  5500  .  +  .  ID=exon00004;Parent=mRNA00001,mRNA00002,mRNA00003
ctg123 . exon            7000  9000  .  +  .  ID=exon00005;Parent=mRNA00001,mRNA00002,mRNA00003

BED

Name

BED

Format

3 initial required fields and 9 additional optional fields. Fields are separated by a tab.

Required

chrom - name of chromosome
chromStart - Starting position in chromosome. First base in chromosome is numbered 0. This 0 based indexing is different from other formats that use a 1 based index.
chromEnd - End position of feature in chromosome.

Optional

name - Name of BED line.
score - Score with range of (0..1000). Can denote display indcations
strand - ‘+’ or ‘-’
thickStart - The starting position at which feature is drawn thickly
thickEnd - The ending position at which feature is drawn thickly
itemRgb - RGB value example: 255,0,0 which can denote color to display data in BED line. Should limit to 8 colors or less in file.
blockCount - Number of exons (blocks) in the BED line
blockSizes - comma separated list of block sizes. Number of items in list should be equal to blockCount
blockStarts - comma separated list of block starts. Values are relative to chromStart. Number of items should be equal to blockCount

Application Support

USCS Genome Browser

Source

USCS?

Example

chr1  213941196  213942363
chr1  213942363  213943530
chr1  213943530  213944697
chr2  158364697  158365864

chr7  127473530  127474697  Pos3  0  +  127473530  127474697  255,0,0
chr7  127474697  127475864  Pos4  0  +  127474697  127475864  255,0,0
chr7  127475864  127477031  Neg1  0  -  127475864  127477031  0,0,255

track name=pairedReads description="Clone Paired Reads" useScore=1
chr22 1000 5000 cloneA 960 + 1000 5000 0 2 567,488, 0,3512
chr22 2000 6000 cloneB 900 - 2000 6000 0 2 433,399, 0,3601

Jim Vallandingham

Animating Uncertainty

Axis of Uncertainty

Visualization Methods

Data Types

An Uncertain Taxonomy

Uncertainty and Simulation

Morph Icons

Wandering Dots

Wandering Hulls

Bounded Wandering Dots

An Uncomfortable Analysis

Filling in the Cube

An Introduction to Altair

What is Altair?

JupyterLab Experimentation

Installation and Setup

Our First Altair

Same Chart, Different Data

Marks Marks Everywhere

Faceting

A Grammar of Interaction

Summarizing Your Data

Layering Marks

Layers and Interactions

Concatenating Charts

Limitations and Looking Forward

Multivariate Map Collection

3D

Color

Small Multiples

Embedded Charts and Symbols

See Also

Histograms to View Oodles of Doodles from Google

Slides

References and Links

Quick Draw Links and Prior Art

Histogram Examples

An Intro to regl for Data Visualization

What is WebGL?

What is regl?

Throwing Shade with Shaders

Setting up an Exploration Environment

Atom Packages for GLSL Fun

A Baseline for good regling

Some Nice-to-have Packages

Starting up with start

Hello Triangle!

The Many Shader Variable Types

regl and Variables

regl Inputs

A Triangle In Time

regl Inputs: Context, Props, and this

Flying Triangle

Flying Triangle with Props

From Triangles to Dots

Displaying Points with regl

Dots Representing Data

GLSLify Your Life

Go Forth you regl Beings

Bonus: Circle Points

A Data Driven Exploration of Kung Fu Films

Shaw Brothers, Through The Ages

Which Shaw Brothers Film should I watch?

Prolific Directors

Title Showdown: Shaolin vs Swordsman

Silly Kung Fu Titles with word2vec

Actor Troupes, Groups, and Clusters

Finding a Mob of Venoms

Creating Bubble Charts with D3v4

Force Fundamentals

Starting up a Simulation

Noodling on the Nodes

Forces Galore

Bigger, Stronger, Faster, Bubble Charts

Becoming a Data Visualization Practitioner

Step 1 - Ponder

Step 2: Read

Step 3: Monitor

Step 4: Learn

Starting up with `start`