What is an API? API stands for Application Programming Interface. APIs serve as interfaces between different software programs and facilitate their interaction, similar to the way user interfaces facilitate interaction between humans and computers. The key thing to understand is that APIs don’t have to be an interface between your application and the outside world. An API can be incredibly useful for applications running under the same roof.

At Data Realization, we strongly encourage creating simple, flexible APIs around data. What does this mean? Let’s think about charts and graphs. You can grab a free charting widget online, and there are typically two ways of delivering data to it: in the HTML where the widget is instantiated, or available at a specific URL. Either way, the chart will typically grab the data from where you point it, display it, and sometimes allow you to interact with it (e.g. mouse-over to show a number). This can be sufficient if all you need to do with your data is plot it using a free chart widget, but not for more advanced analytics.

With advanced analytics and data visualization, you’re creating an application that runs on a client computer (often in a browser), and has a conversation with a server. The conversation takes place via an API, where the client tells the server what data it wants, and the server responds with the data. The server becomes a simple data provider, focusing on security/authentication and scaling/caching, while ignoring presentation.

Let’s say you have an interactive chart that allows you to view a year of data or zoom in to view an hour of data. You don’t want to load a year’s worth of data in a one-minute resolution. Instead, the front-end application can say to the server, “I want a year’s worth of data in one-week resolution.” If the user zooms in, the application makes a new request, saying, “I want a month’s worth of data in one-day resolution.” On the back-end, preparing for this kind of use-case is trivial, where you would create roll-up tables in the database that house data at several resolutions. The API would then take the requested resolution and find the most sensible table to pull from.

The end user experience is an application in which they can put their hands on the data, and zoom in with animations to help give additional context to the data.

APIs for data are easy (in most cases).

Database records are easily serialized (JSON, XML, etc). Most data is immutable and easily cache-able and pre-cache-able. Taking the conversational approach is easier than it sounds, as request parameters are validated and translated directly into database queries. Providing data through an API encourages modularity and separates data processing from presentation.

One of the most common visualization tools is the line graph. It’s simple, intuitive, and helps answer the question, “How did this value change over time?” Typically, the value is pulled straight from the database or logs.

Plotting a value over time on a line graph is taking the engineering approach: examine the input format and find a matching output format. When you start from a user’s perspective, you’ll hear phrases like “the relationship between,” and “the rate of,” and “how does X affect Y?” These are not as easy to drag and drop into a graphing library, but the results are much more profound.

The best answer isn’t always a number. The answer can be a shape, a gradient, or a trend.

It all depends on the question.

Heat Map rendered in Flash

Three years ago, I rendered a large heat map representing almost 100,000 Digg users and the 300,000 friendships between them. I used PHP/GD2 to render the image and it took quite a while to render. Due to the difficulty of redrawing it, the heat map was never updated. It would have been great if the heat map reflected current information, but instead it got to live on as a snapshot taken in 2007.

Since then, I’ve wondered about ways of rendering heat maps on the fly, ideally using the visitor’s CPU. Finally, in November, I was able to hack together a highly optimized Flash application that produces pretty awesome-looking heat maps (if I do say so myself) within a second or two.

Check it out

Dopplr Travel Data Visualizations

These visualizations are fun to look at and share with your friends, but are they useful? The answer is debatable, but for most of the widgets, the answer is probably no.

Effective visualizations focus on answering questions. You shouldn’t visualize for the sake of visualizing. You shouldn’t take data and dump it onto a graph, chart, map, or table without purpose. One Dopplr widget shows your personal velocity by averaging the distances you’ve traveled over time. The result is an interesting number, and to have your personal velocity compared to a glacier, duck, or butterfly is fun, but what’s the point?

Dopplr should be commended for taking a step into this space. It’s fascinating data and it’s wonderful to see this personal data visualized in fun and interesting ways. But can we take it a step further?

Where Dopplr falls short is in creating visualizations that effectively lead to a better understanding of the data. In order to make people understand, you must first figure out what needs to be understood. What facts can be extracted from this data using visualization? It’s best to formulate this as a fact being an answer to a question. So what are the questions here?

After exporting my personal travel data from Dopplr as a JSON object, I explored some ways of learning more from my data. For example, I was curious how much time I spent at home in Phoenix, Arizona — as a fraction or percent of the year. Since it’s such a simple question and answer, I didn’t bother representing it in a very visual way.

This week, I came up with a more complicated question. With the weekend approaching, I was excited to spend an entire weekend in Phoenix. I felt like I hadn’t have very many full weekends at home. That lead me to want to justify this feeling. How many weekends had I spent at home during the first 3.5 months of 2010? That’s a question that can be answered effectively through visualization.

Visualizing my travel data

Surprisingly, Dopplr doesn’t have a calendar view (or maybe I just don’t know where it is). It looks like you can subscribe to an iCal feed for your trips, but it is likely only your upcoming trips. Displaying my travel travel on a calendar made it easy for me to figure out how much time I’ve spent on the road and how many weekends I’ve spent at home.

The result made me question how I would define a weekend at home.

If a “full” weekend required that my Friday night and early Monday morning not be infringed by travel plans, then I’ve only had one “full” weekend before this weekend (early April). If you don’t include the Monday morning restriction, I had a full weekend in February, too. If you don’t restrict based on Friday night travel, I had two weekends at home in February.

This example was fairly basic and trivial until I added the legend, showing which cities each color on the calendar represented. Using the 1 box = 1 day metaphor already present on the left, the legend was also able to double as a sort of bar chart, expressing the number of days I’ve spent in each city.

It’s worth noting that even though I set out to visualize my geographic distribution, I didn’t jump straight to putting the data on a map. Focus on the questions you are trying to answer, and let that dictate the type of visualization you use.

How many people are watching now? And how about now? And now?

The number changes over time, meaning we can easily express how the number changes over time with a time-series line graph. You can see if you are getting an influx of new viewers or if people are tuning out.

There’s a critical flaw in this viewer count number, however. If your broadcast is hovering around 1,000 viewers and most people are captivated and sticking around, the line graph will look very similar to a graph representing the same quantity of viewers with new viewers tuning in at the same rate that people are leaving. Those are two very different scenarios that could look exactly alike on a graph.

Live Viewership with Churn Rate

In broadcast television, the rate at which viewers come and go is called “churn rate”. This variable can be added to the viewership line graph, for example by plotting new connections and disconnections relative to the viewer count. The wider the line, the higher the churn rate.

Churn Rate Sparkline

This can also be applied to sparklines, where small, unlabeled graphs can show not only the trend in viewership count, but also the churn rate (as illustrated by the thickness of the colors.