What follows is a blog-ified and amended version of that talk, originally entitled “Secrets of Successful Data Scientists.”

1. Choose The Right-Sized Tool

Or, as I like to say, you don’t need a chainsaw to cut butter.

If you’ve got 600 lines of CSV data that you need to work with on a one-time basis, paste it into Excel or Emacs and just do it (yes, curse the Flying Spaghetti Monster, I’ve just endorsed that dull knife called Excel).

In fact, Excel’s and Emacs’ program-by-example keyboard macros can be fantastic tool for quick and dirty data clean-up.

Alternatively, if you’ve got 600 million lines of data and you need something simple, piping together a several Unix tools (cut, uniq, sort) with a dash of Perl one-liner foo may get you there.

But don’t confuse this kind of data exploration, where the goal is to size up the data, with building proper data plumbing, where you want robustness and maintainability. Perl and bash scripts are nice for the former, but can be a nightmare for building data pipelines.

When you’re data gets very large, so big it can’t fit reasonably on your laptop (in 2010, that’s north of a terabyte), then you’re in Hadoop, parallelized database , or overpriced Big Iron territory.

So, when it comes to choosing tools: scale them up as you need, and focus on getting results first.

2. Compress Everything

We live in an IO-bound world, where the dominant bottlenecks to data flow are disk read-speed and network bandwidth.

As I was writing this, I was downloading an uncompressed CSV file via a web API. Uncompressed, it was 257MB, ZIP-compressed: 9MB.

Compression gives you a 6-8x bump out of the gate. When moving or crunching data of a certain heft, compress everything, always: it will save you time and money.

That said, because compression can render data difficult to introspect, I don’t recommend compressing TBs of data into a single tarball, but rather splitting it up, as I discuss next.

3. Split Up Your Data

“Monolithic” is a bad word in software development.

It’s also, in my experience, a bad word when it comes to data.

The real world is partitioned – whether as zip codes, states, hours, or top-level web domains – and your data should be too. Respect the grain of your data, because eventually you’ll need to use it to shard your database or distribute it across your file system.

Even more, it’s this splitting up of data that enables the parallel execution in Hadoop and commercial data platforms (such as Greenplum, Aster, and Netezza).

Splitting is part of a larger design pattern succinctly identified in a paper by Hadley Wickham as:     split, apply, combine .

This is, in my mind, a more lucid formulation of “map, reduce” to include key selection (”split”) as a distinct step before any map/apply.

4. Sample Your Data

Let’s say hypothetically you’ve got 200 GBs of data from your portmanteau of a start-up, FaceLink. Someone wants to know if more people visit on Mondays or Fridays, what do you do?

Before you wonder “if only I had 64 GB of RAM on my MacBook Pro”, or fire up a Hadoop streaming job, try this: look at a 10k sample of data.

It’s easy to visually inspect, or pull into R and plot.

Sampling allows you to quickly iterate your approach, and work around edge cases (say, pesky unescaped line terminators), before running a many-hour job on the full monty.

That said, sampling can bite you if you’re not careful: when data is skewed, which it always is, it can be hard to estimate joint-distributions – comparing the means of California vs Alaska, for example, if your sample is dominated by Californians (an issue that statistics, that sexy skill, can address).

5. Smart Borrows, But Genius Uses Open Source

Before you create something new out of whole cloth, pause and consider that someone else may have already seen it, solved it, and open-sourced it.

A Google Code Search may find turn up a regular expression for that obscure data format.

The open source community allows you, if not to stand on the shoulders of giants, to at least rely on the gruntwork of fellow geeks.

6. Keep Your Head in the Cloud

This past week, an engineer friend was just thinking about buying a dream desktop: a high RAM, multi-core box to run machine learning code over TBs of data.

I told him it was a terrible idea.

Why? Because the data he wants to work on isn’t local, it’s on an Amazon EC2 cluster. It’d take hours to download those TBs over a cable connection.

If you want to compute locally, pull down a sample. But if your data is in the cloud, that’s where your tools and code should be.

7. Don’t Be Clever

I once heard Brewster Kahle discuss managing the Internet Archive’s many-petabyte data platform: “everytime one of our engineers comes to me with a new, ingenious and clever idea for managing our data, I have a response: ‘You’re fired.’”

Hyperbole aside, his point is well-taken: cleverness doesn’t scale.

When dealing with big data, embrace standards and use commonly available tools. Most of all, keep it simple, because simplicity scales.

I know of a firm that, several years ago, decided to fork one part of Hadoop because they had a more clever approach. Today, they are several versions behind the latest release, and devoting time & energy to back-porting changes.

Cleverness rarely pays off. Focus your precious programmer-hours on the problems that are unsolved, not simply unoptimized.

“There are no more promising or important targets for basic scientific research than understanding how human minds… solve problems and make decisions effectively.” - Herbert Simon

In my previous post , I discussed the forces behind what I’m calling The Data Singularity. My basic thesis is that as information generating processes become more frictionless — as humans have been excised from information read-write loops — the velocity and volume of data in the world is increasing, and at an exponential rate.

But where we go from here? What are the consequences of living in an age where every datum is stored? Where are the bottlenecks, pain points, and opportunities? Which technologies are addressing these?

The upshot is this: a new class of tools are evolving for Big Data because traditional approaches can’t scale up. But these tools share a common goal: scaling down data, and making it human-sized. That’s the “reduce” part of MapReduce, the single statistic from analysis, or the hundred pixel line from one hundred million events.

What’s happening today isn’t entirely new, though. There were echoes of it decades ago, when surveillance satellites first began scanning the globe.

VI. How Satellite Data Paralyzed the CIA

Beginning in the early 1970s the CIA began relying more on global satellite reconnaissance imagery for its intelligence operations. But according to one history, this massive, rich data didn’t accelerate the pace of US intelligence: it slowed it down.

Why? Because confronted with this firehose, CIA leaders attempted to analyze every image, chase every half-formed hypothesis, simply because it was possible. The few good leads were washed out by the many mediocre. The CIA didn’t adjust their decision-making to this new scale, and they were drowned by it.

Many organizations are at a similar inflection point now, with access to massive, rich data about their customers or products. And, like like the CIA in the 1970s, they find themselves paralyzed by the possibilities.

VII. People Still Pull the Big Levers

That Big Data paralyzes human decision-makers matters, because humans still make the big decisions. When someone praises a company as being “data-driven”, I’d like to imagine that this is literally true: that the company is nothing more than a few server racks blinking & humming away, slinging bits and earning money.

But no such company exists. What “data-driven” really means is that the executives & employees use data as inputs for making decisions. Companies may be data-fueled, but they’re people-driven.

VIII. Human-sizing Big Data: Filter & Crunch

All of the analytics in the world won’t matter if it remains inaccessible to the people driving an organization — the human decision-makers.

We have processes all around us acting as data amplifiers, recording events at a pace & scale that we can’t comprehend. But this has created a disequilibrium: our capacity to create data is vastly outstripping our ability to consume it. Analytics is the act of taking Big Data streams and human-sizing them for our small data brains.

We can reduce data by either filtering it, which sifts through but does not alter data, or by crunching it, reducing many data points to a few.

Google and Facebook are Filters . Many consumer web technologies might be viewed as powerful filters. Google is a relevance filter for 20 billion web pages. Facebook is a social filter for baby photos. FourSquare is a geo-social filter for hipster bars. Amazon is a filter for retail products, combining search with a powerful recommendation engine.

Wikipedia is a Natural Language Cruncher . Crunching data is harder than filtering it. Perhaps the toughest nut to crack involves processing natural language: if you read a thousand web pages about the Gutenberg Bible, how would you describe it in a few paragraphs? Wikipedia is a human-powered natural language cruncher, powered by its army of mechanical turks, whose collective actions even reveal news trends.

Crunch the Past to Predict the Future . Crunching of quantitative data is at the heart of many prediction tasks: the National Weather Service aggregates weather station measurements into forecasts, Fair Isaac calculates a score of credit-worthiness by examining your credit history, and a sports contest might be construed as an algorithm — operating on a sequence of individually played points — to predict the best team or athlete.

Number crunching has its more banal forms, as well, in the kind of sums and averages found in your phone or utility bill. These are necessary, but predictive algorithms — the kind involved in weather forecasting — will continue to grow in importance. For at a certain scale of data, exact reporting become an insurmountable task: we can only hope to have probabilistic answers.

IX. Business Intelligence is Dead: New Tools for a New Era

That our traditional tools don’t operate at scale was highlighted by Tim O’Reilly recently, when he declared “Business intelligence as we knew it is dead.”

A new class of tools is emerging along the Big Data stack, in three areas: (1) storage & computation, (2) analytics, and (3) dashboards & visualization.

These tools will disrupt and attack many of the traditional Business Intelligence firms, ranging from tool-makers like SAS and SPSS, to relational database vendors like Oracle, to custom hardware providers.

• 1. Storage & Computation: Mixed Platforms, not Monolithic Databases . At the lowest level of storage & computation, Big Data is driving the success of cloud computing platforms like Amazon’s Elastic Compute Cloud — a massive, virtualized commodity-hardware grid — as an alternative to the Big Iron sold by hardware makers.Big Data has also catalyzed widespread adoption of the distributed, fault-tolerant Hadoop platform — an open-source implementation of Google’s BigTable that was developed by Yahoo, and is now commercially supported by Cloudera.

  A bit further up the stack, relational databases are suffering: newer commercial entrants in this space — such as Greenplum, Aster Data, Vertica, and Netezza — offer parallelized relational systems that operate at greater scale and lower cost than Oracle and Teradata.Many open-source, non-relational data stores — with a colorful constellation of names such as HBase, MongoDB, CouchDB, Cassandra, and Voldemort — have gained traction for high-traffic, content-driven web sites.

  SQL & NoSQL are Complementary, Not Antagonistic. While some may view storage technologies as antagonistic, either-or choices, the truth is that most Big Data-driven companies use a mixture of tools in complementary ways. Hadoop is often used for batch-processing and transformation of log data that is fed to more structured data stores, such as a distributed RDBMS, in backend systems. Non-relational data stores are in turn ideal for front-facing, high-performance web applications, where queries return a bolus of data related to a single key — often a product, user, or page identifier. All of these pieces working together form an information platform: an ecosystem of APIs working together.

• 2. Analytics: There Are No Turnkey Solutions. Imagine if any piece of data you ever wanted was within a query’s reach: what would you do with it? We’re fast approaching this scenario, and making data meaningful is the bottleneck. But unlike storing data — where use cases & technologies are common and becoming commoditized — the ways that firms filter and crunch their data varies widely.

  This reflects the range of analytics needs that firms have: for example, a financial firm may need low-latency, continuous analysis of data streams, while an online retailer or pharmaceutical firm can tolerate 24-hour delays for analysis.

  Scaling Up Analytics is Hard. R, my favorite analytics tool, is fantastic for modeling either aggregated data sets or samples of data that can fit in memory, but methods for deploying R in a large-scale data environment are still nascent.One promising approach is Saptarshi Guha’s RHIPE , which combines R with Hadoop ( slides ) from his March presentation at the Bay Area R Users Group . Another MapReduce-based framework for large-scale data analysis include the Apache Mahout project.

  Learn, Then Apply: But Stay Close to the Data. In general, there are two pieces in any analytics pipeline: (i) learning, or the training of a model with historical data, and (ii) prediction, or the application of a model to new data. On the the learning side, it’s been said that more data beats better algorithms , and this is certainly true for many classification problems. In general, training a model is a computationally intensive task, and the development of methods that can train on massive data sets is an area of active research.

  On the application/prediction side of modeling, the challenges often revolve around deployment, or How do we get the model to the data? (Since the reverse, pushing data to the model, is more expensive). To address the desire of porting models across different environments PMML (Predictive Modeling Markup Language) has been developed, which is supported by a range of database vendors.

  The meme of “in-database analytics” is resonating because given data’s increasing heft, efficient analytics will follow the pattern of having the training & execution of models stay close to where the data lives.

  As it will be several years before either open-source or commercial analytics tools are mature here, the most successful Big Data modelers will be those data scientists who can build and glue together their own methods, tailored for individual environments and needs.

• 3. Dashboards & Visualization: Why “I See” is a Synonym for “I Understand” . The most visible way in which Big Data is disrupting old tools is by changing the way we look at data. The ultimate end-point for most data analysis is a human decision-maker, whose highest bandwidth channel is his or her eyeballs. To take optimal advantage of the human visual system, dashboards and data visualization must be well-designed, and until recently, tools that achieved even a minimal standard of competence were rare.

  Visual Literacy is on the Rise. But a new set of visualization tools and packages, as well as growing popular interest in data visualization — catalyzed by the books of Edward Tufte, blogs like Nathan Yau’s FlowingData and talks at T.E.D. conferences – are changing this.As I’ve written about before, there are two distinct kinds of data visualization pathways: (i) exploratory, a highly interactive path whereby a data scientist may permute through dozens or even hundreds of views of a data set to understand its shape or fit to a hypothesized model, and (ii) narrative, a more constrained path whereby only one or several views of the data are presented.

  Exploring Data Requires Fast, Frequent Feedback . For the exploratory path, desktop tools are ideal. The open-source language R has several outstanding visualization packages, including ggplot2 and lattice (based on William Cleveland’s trellis). Two solid commercial products for exploratory visualization are SpotFire and Tableau (the latter of which has been praised by the hard-to-please Stephen Few).

  Sharing Visualizations: Web Dashboards Are Ideal. Ultimately, however, visualizations need to be shared beyond a single user, to an audience. Web-driven dashboards are an ideal form for sharing narrative visualizations, by allowing navigation along defined axes of the data.The challenge is moving visualizations from the desktop to the web. Tableau has this capacity, but with R the process is less straightforward. One promising route is via Jeff Horner’s RApache tool , which embeds R inside an Apache server (which I’ve used for my MLB Pitch F/X tool, and which Jeroen Ooms’ uses to power his ggplot2 web app ).

  The major limitation of R-driven web graphics is that achieving some interactivity within the graphic itself is difficult, as R’s graphics model is focused on static graphics. There are, however, several routes for achieving highly interactive, web-based data visualizations, whether by using Javascript, HTML5’s Canvas, or Flash. Two in particular are: (i) Ben Fry’s Processing , an expressive language for vector animation, which recently added Javascript as one of its implementations, and (ii) the Protovis framework out of Stanford: a Javascript graphing toolkits whose conceptual integrity and expressive flexibility was inspired (like ggplot2) by Wilkinson’s grammar of graphics.

X.  Collaborating with Big Data: Analytics is a Social Process

In the same talk that Tim O’Reilly proclaimed the death of BI “as we knew it”, he also highlighted a new initiative by Greenplum called Chorus (Greenplum is a Dataspora client, but I confess I’ve only seen a limited preview).

The animating spirit of Chorus is that analytics is not only about data, models, and visualizations — it’s also about the people who work on these various pieces. One of the reasons I love Box.net is the layer of social information that’s overlayed onto my files: appended notes, access statistics from collaborators, automatic notifications when a change is made.

Chorus is a vision to do this with Big Data; it allows, for instance, an analyst to link a data visualization to an underyling data source, include the R code that created the visualization, and append a note about a recent change to it.

As the Big Data stack matures, tools that help manage the workflow from data to analytics to visualizations, and ultimately to decisions, will be critical. Someday, creating and sharing a data analysis through a web dashboard should be as easy as writing a blog post. Until that day, there’s plenty of work to keep us data scientists well-employed.

If crunching terabytes of data is the kind of thing you’d like to do for breakfast, please send me a note at med @ dataspora.com.  I’m looking to hire technologists & analytics experts for a new venture… more on that soon.

In a nutshell, the Data Singularity is this: humans are being spliced out of the data-driven processes around us, and frequently we aren’t even at the terminal node of action. International cargo shipments, high-frequency stock trades, and genetic diagnoses are all made without us.

Absent humans, these data and decision loops have far less friction; they become constrained only by the costs of bandwidth, computation, and storage– all of which are dropping exponentially.

The result is an explosion of data thrown off from these machine-mediated pipelines, along with data about those flows (and data about that data, and so on). The machines all around us — our smart phones, smart cars, and fee-happy bank accounts — are talking, and increasingly we’re being left out of the conversation.

So whether or not the Singularity is Near, the Data Singularity is here, and its consequences are being felt.

But before I discuss these consequences, I’d like to expand on the premise. The world wasn’t always drowning in this data deluge, so how did we get here?

I. Data at the Speed of Speech

For most of human history, information traveled no faster than the sound of the human voice. The origin of human language was the original singularity: it marked the birth of a non-biological information channel, distinct from our DNA.

But despite this achievement , the production of information — whether farmers’ almanacs or merchants’ ledgers — was still constrained the by costs of ink and parchment and the write-speed of the human hand.

All 70,000 volumes of the Library of Alexandria, the collected body of human knowledge in antiquity, could fit on two thumb drives today.

Thus the transmission and production of data, when it was done at all, was painstaking in form, small in scale, and occurred between people.

People --> People

II. Data at the Speed of Light

With the telegraph, for the first time, data flowed at the speed of light.

In the late 18th century, the first substantive telegraph line connected Paris to a suburb 210 kilometers to its north, using optical semaphores rather than electrical currents to communicate. Yet while data hopped between stations at light speed, it had to be routed by human operators at each station.

Centuries earlier, the printing press dramatically reduced the production costs of information. Still, human authors transmitted their hand drafted manuscripts to type setters, who set type with fonts optimally designed for human eyes.

III. Programmable Looms and Reading Machines

Punch cards represented the movement of data away from human-readable, anthropocentric substrates, onto a medium designed principally for consumption by machines.

Punch cards were developed in the early 18th century to control industrial looms , in France.

Now, machines were the final terminus of data transmission. This act of communicating with our machines, programming them, was at the heart of Charles Babbage’s Analytical Engine, which came more than a century later.

People --> Machines

IV. Phonographs and Recording Machines

Developing on the other side of the communication spectrum were machines that excelled at writing and storing data.

The modern rotating disk drive feels less inspired by punch cards, but by Thomas Edison’s cylinder machines, better known as phonographs.

The human voice was a natural data format, and if early pioneers had a vision for the modern human-machine interface, I imagine it would have been to program machines by voice. It’s a vision that still eludes us.

By the middle of the 20th century, a slew of semiconductor technologies emerged to close the loop of data generation: we had machines that produced digital data, and machines that continuously consumed it, without human intervention.

Machines --> Machines

These technologies also sparked the beginning of a less-celebrated, but equally important exponential curve: the falling cost of data storage.

V. Listening to the Pulse of the Planet

The exponential drop in data storage costs has meant that logging historical data about a process, or billions of processes, is economically feasible.

I conjecture that the largest share of data on the planet sits in log files; these are the EKGs of the server farms that manage our cell phones, our e-mail accounts, and every other facet of our online existence — and which consume 3% of the US energy budget .

Ubiquitous networking and cheap bandwidth has meant these pools of storage are no longer isolated on individual sensors, phones, or servers, but form the tributaries feeding an ocean of data in the Cloud.

And yet, funneling these massive volumes of data creates enormous technological pressures, against which companies struggle. So why keep the data?

Because inside these log files, amidst the myriad conversations recorded between machines, lies the pulse of their customers.

Collectively, these logs reveal the pulse of the planet — flight delays, package shipments, job losses, and human sentiments.

And as I’ll discuss in my next post, those who can extract a meaningful signal from this thunderous cacophony — the analysts, statisticians, and data scientists — are uniquely positioned to change the world.

“The adoption of a relational model of data, as described above, permits the development of a universal data sub-language.”– E.F. Codd, 1969

“Database research has produced a number of good results, but the relational database is not one of them.” – Henry Baker, 1991

Outside of programming language flame wars, few questions raise the hackles of hackers more than: “how should I store my data?”

I will argue here, like many such debates , the answer is:  it depends on what you’re doing.

While the rise of non-relational data stores serves a much-needed niche, the death of SQL and relational databases has been much exaggerated.  E.F. Codd may be dead, but SQL is alive and well as a simple yet powerful data query language.

3NF Crusaders vs NoSQL Rebels

While the current critique relational databases shares features of earlier debates (such as in the 1990s, when object-oriented databases were heralded as the next big thing), it has some new twists.  Thus to review the players and their positions:

On our right are the relational curmudgeons, the kind of folks who pen manifestos and crusade against NULL values. They have converted nearly all of big business to their ministry, and have billions of dollars in their coffers to show for it. They insist that data should be stored in terms of its relations, to protect its integrity and facilitate its analysis. Ideally that means third-normal form, but more liberal branches of the church exist.

On our left are the folks from the misnomered NoSQL movement, shaggy kids from the likes of Facebook and Twitter . They’ve rebelled against the shackles of relational tables (and bear the scars of MySQL scaling struggles). They believe that data should be persisted as it’s programmed: in objects. And they’ve spawned a constellation of colorfully named open-source projects – Cassandra, Voldemort, CouchDB, MongoDB, and Dynamo – to consummate their cause.

A Three-Pronged Attack on SQL:  Syntax, Schemas, and Performance

At the heart of the NoSQL movement are three distinct critiques:

1. A dislike for SQL’s syntax, which is ill-fitted to programming patterns.  It’s painful to write select statements to grab the data spread out across many tables, when all you want is a record.  Within web frameworks, the interface problem has been solved to a large degree by object-relational-mappers, such as Ruby’s ActiveRecord.
2. A rejection of the strong typing of relational schemas, which make it painfully difficult to alter one’s data model.  It also makes writing to the data store a complex process.
3. A critique of performance, which in turn relates to how concurrency and partitioning of computation is handled.  Most relational databases maintain a shared state, which strives for perfect concurrency, but complicates distributed computation over many nodes.  NoSQL architectures are built on languages and tools, like Erlang and Hadoop, that favor distributed processes which (to use two favorite catch phrases) “share nothing” but are “eventually consistent.”  The NoSQL philosophy also weighs heavily against joins.

These critical threads are mirrored in the movement and their associated projects.  One the one hand you have developers who prefer the programmatic ease of interacting with NoSQL data stores, such as Cassandra and CouchDB.  They also don’t suffer the performance penalties of scale:  unlike with relational tables, the performance of look-ups does not degrade as the stored number of objects rises.

On the other, you have Big Data analysts (like myself), who love Hadoop because it allows easy distributed computation over massive, loosely typed data sets.

Analytics:  MapReduce for Munging, SQL for Set Operations

With regard to analytics, the Hadoop ecosystem makes it easy to dump several billion records of varying formats into a data store and process them – without having to conform them to a common data model.   Thus NoSQL framework is great for massive data munging.

But if I had to access an already structured massive data set, I prefer SQL’s declarative syntax to MapReduce constructs.

I recently sat down at an SQL terminal with several hundred billion call records behind it. With a simple SQL query, I determined how many distinct people the average American telephones more than once in a given month (answer: five). In a few hundred seconds, I’d generated a report on the global state of the customer calling network.

Contrary to what the NoSQL may inveigh, it’s not that relational databases can’t scale – in fact, they can scale to petabytes, as those who know Fortune 500 enterprise computing can attest . The problem is that relational databases require lots of ETL cruft to munge fluid blobs of data into strongly typed tables.

I can’t imagine the programmer pain and suffering that went into building one, unified, global database.  But once it’s there, I’d much prefer to access it with SQL statements than MapReduce code .

And I’m not alone in feeling this way:  Jeff Hammerbacher of Cloudera recently told me that, for an enterprise deployment, usage jumped 10x when an SQL interface – HIVE (which I mention below) – was placed on the cluster.

NoSQL is a Misnomer: SQL is Innocent!

Which brings me to my defense of SQL.  I agree with two of three above critiques that embody the NoSQL philosophy, namely the need for schema-less storage and distributed architectures.  But when they go after SQL, and name the movement in opposition to it, they’ve named the wrong villain. (Your honor,) SQL is just an innocent query language!

As evidence of innocence, look no further than Google’s GQL and Hadoop’s HIVE, two SQL-style query languages for NoSQL data stores.

Why SQL in a NoSQL data store?   For one, it’s a language that both business analysts and developers already know; so the zero-th order adoption step is shorter.

But SQL lives on for a deeper reason: it is a simple yet powerful language for set operations. SQL captures the essential patterns of data manipulation, such as:

1. intersections (JOINs)
2. filters (WHEREs)
3. reductions or aggregations (GROUP BYs)

I suspect that many developers who profess a disdain for SQL have been deceived by its simplicity.  One of my favorite packages in R is sqldf, which allows SQL queries on R data frames.  SQL’s declarative expressions are frequently more readable and compact than their R programmatic equivalents.

MapReduce is Possible in SQL

Until very recently one of the more difficult operations to perform in SQL was a top-K query, for example, finding the five highest priced items in for every store in a retail database. But so-called window functions, which make such queries easy to express, have become part of the SQL standard and are now natively supported in Postgres.

Window functions are powerful because they provide a “split-apply” functionality, otherwise known as a map function. Combine these with SQL’s GROUP BY operations, which is a reduce function, and you have achieved – voila! – map-reduce in SQL. And as with all map functions, window operations are massively parallelizable (something that has not gone unnoticed by some commercial vendors.)

Verdict:  Don’t Use a Chainsaw to Cut Butter (Use the Right Tool)

Both NoSQL and SQL have their place in an analytics ecosystem.   In the Big Data workflow that I’ve advocated in the past, I view SQL as a pipe feeding data into more sophisticated modeling and visualization tools, such as R.  But it is an easy-to-use pipe, and it allows analysts to quickly pull out a subset of data — and start asking questions of that data.

The verdict in the great NoSQL debate is:  know your tools and know your goals.  In the Big Data space today, there can be an undue focus on formats or mechanics, but these are just a means to one end:  products.  Remember, Paul Graham and his team wrote Viaweb in Lisp, and it just worked.

Confessions from a Massive, Nightmarish Data Project

Back in 2000, I went to France to build a genomics platform. A biotech hired me to combine their in-house genome data with that of public repositories like Genbank. The problem was the repositories, all with millions of records, each had their own format. It sounded like a massive, nightmarish data interoperability project. And an ideal fit for a hot new technology : XML.

So I dove in, spending my days designing DTDs, writing parsers, tweaking tags (”taxon” or “species”? attribute or element?). At night I dreamt in ontologies. It was perfect.

Then reality struck. The pipeline was slow: Oracle loaded XML at a crawl. And it was a memory hog, since XSLT required putting full document trees in RAM.

We had a deadline to meet (and, mon dieu, a 35 hour work-week). So we changed course. We hacked our Perl scripts to emit a flat tab-delimited format — “TabML” — which was bulk loaded into Oracle. It wasn’t elegant, but it was fast and it worked.

Yet looking back, I realize that XML was the wrong format from the start. And as I’ll argue here, our unhealthy obsession with XML formats threatens to slow or impede many open data projects, including initiatives like Data.gov.

In the next sections, I discuss how XML fails for Big Data because of its unnatural form, bulk, and complexity. Finally, I generalize to three rules that advocate a more liberal approach to data.

Three Reasons Why XML Fails for Big Data

I. XML Spawns Data Bureaucracy

In its natural habitat, data lives in relational databases or as data structures in programs. The common import and export formats of these environments do not resemble XML, so much effort is dedicated to making XML fit. When more time is spent on inter-converting data — serializing, parsing,translating — than in using it, you’ve created a data bureaucracy.

Indeed, it was what Doug Crockford called “impedance mismatch inefficiencies” that sparked him to create JSON - standardizing Javascript’s object notation as a portable data container.

II. Yes, Size Matters for Data

Size matters for data in a way it does not for documents. Documents are intended for human consumption and have human-sized upper bounds (a lifetime’s worth of reading fits on a thumb drive). Data designed for machine consumption is bounded only by bandwidth and storage.

XML’s expansiveness — for even when compressed, the genie must be let out the bottle at some point — imposes memory, storage, and CPU costs.

III. Complexity Carries a Cost

I never fail to sigh when I open a data file and discover an army of tags, several ranks deep, surrounding the data I need. XML’s complexity imposes costs without commensurate benefits, specifically:

• In-line, element-by-element tagging is redundant. Far preferable is stating the data model separately, and using a lightweight delimiter (such as a comma or a tab).
• Text tags are purported to be self-documenting, but textual meaning is a slippery thing: it’s rare that one can be sure of a tag’s data type without consulting its DTD (in a separate document).
• End-tags support nested structures (such as an aside (within (an aside)). But to facilitate data exchange, flattened out structures are preferable, and arbitrary levels of nesting are best using sparingly.

XML’s complexity inflicts misery on both sides of the data divide: on the publishing side, developers struggle to comply with the latest edicts of a fussy standards group. While data suitors labor to quickly unravel that XML format into something they can use.

Three Rules for XML Rebels

I. Stop Inventing New Formats (as Tim Bray said in 2006)

Before you call for “an XML format for X”, let me tell you a story about LaTeX and MathML. (And while these are document formats, there’s a lesson here for data).

The LaTeX typesetting system is the lingua franca for composing scientific documents. As the one-million plus LaTeX-formatted articles on arXiv.org attest, it is spoken by scientists worldwide.

MathML, on the other hand, is a markup language for mathematics recommended by the W3C. If you’re a scientist looking to use MathML, you have two choices: (i) find a program to convert LaTeX, which you already know, to MathML 3.0 or (ii) familiarize yourself with this handy 354-page spec and code it yourself.

Two years ago, Mike Adams thought of a third way: why not just let people use LaTeX directly in WordPress? So he wrote a plug-in that did it. The applause was deafening.

Spoken languages are strengthened by usage, not by imperial fiat, and data formats are no different. Far better to evolve and adapt the standards we already have (as JSON and SQLite’s file format do), than to fabricate new ones from whole cloth. As John Udell says, “good-enough solutions [that are] here now, and familiar to people, often trump great solutions that aren’t here and wouldn’t be familiar if they were.”

II. Obey The Fifteen Minute Rule

Interviewed several years ago, James Clark stated “If a technology is too complicated, no matter how wonderful it is and how easy it makes a user’s life, it won’t be adopted on a wide scale.”

Accordingly, if you absolutely must develop a new API, language, or format, it should satisfy a simple rule: a person of reasonable ability should be able to get from zero to ‘Hello World’ in fifteen minutes. (This does not preclude complex languages or formats, per se: it does require that additional complexity not be sui generis, but built on some existing foundation, for example.)

Despite a noble vision for the semantic web , the barriers for adopting the W3C’s proposals for linked data are too high. The beauty of original HTML standard was that it was dead simple. The flaw of RDF is that it is too hard.

III. Embrace Lazy Data Modeling

To keep data bureaucracy to a minimum, several Big Data thinkers have advocated a more catholic approach to data: building data stores that accommodate a broad range of data types and formats.

Lazy data modeling is similar to lazy evaluation. The right schema for data depends on future use cases, in as-yet-undeveloped applications. Instead of trying to guess the future, we can store the data “as-is” — and deal with its transformation when (and if) a necessary use case arises. As Michael Franklin and colleagues note: ”the most scarce resource available for semantic integration is human attention.”

This liberal view also reduces barriers for data sharing, barriers which threaten initiatives like Data.gov. The US Census Bureau shouldn’t expend resources to publish in XML if they have a good-enough format available right now.

For the data geeks in the trenches, who are building the next generation of data services, the laws of economics hold fast: there are unlimited opportunities in the face of one limited resource, time. (Which also explains why data geeks seem to get no sleep).

XML’s unfulfilled promise for data testifies that formats can create friction. The easier it is for data to be shared and consumed, the more quickly we’ll realize our visions for smarter businesses and better governments.

(25-Aug-2009 Update: Read a response from open gov advocates at Sunlight Labs).

As others have noted, we’ve reached a tipping point in history: more data is being manufactured by machines — servers, cell phones, GPS-enabled cars — than by people. The early, document-centric web was populated by hand-coded hypertext files; today, a hand-coded web page is as rare as hand-woven clothing.

Through web frameworks, wikis, and blogs, we have industrialized the creation of hypertext. Similarly, we’ve also industrialized the collection of data, and spliced out the human steps in many data flows, such that data entry clerks may soon be as rare as typesetters.

The web we experience will continue to be dominated by documents — e-mail, blogs, and news. And while many sites are data-centric — Google maps, Weather.com, and Yahoo finance — it’s the web that we can’t see that surging with data. It’s not about us, it’s about servers in the cloud mediating entire pipelines of data, only occasionally surfacing in a browser.

But the web’s data architecture is fractious and in flux: many competing standards exist for serializing, parsing, and describing data. As we build out the data web, we ought to embrace standards that mirror data’s form in its natural habitats — as programmatic data structures, relational tables, or key-value pairs — while taking advantage of data’s stream-like nature. Mark-up languages like HTML and XML are ideal for documents, but they are poor containers for data, especially Big Data.

Sacred “Words & Enthusiasm” vs Meaningless Utterances

Documents and data are different.  The table below reflects my thin grasp of the fissure lines, as a step towards arguing why we ought to design around them.

Documents are made of “words and enthusiasm”: sonnets, cake recipes, blog posts, Supreme Court rulings, and dictionary definitions. Their core stuffing is text. Their structure is unpredictable and irregular — even fractal.

Data are not created but collected (something given, not something made): city temperatures, stock prices, web visitors, and home runs. They are observations in time and space, with periodic and predictable structure. Data are reorderable and divisible: you can relay city temperatures in any order, but you can’t rearrange a Shakespearian sonnet without muddling its meaning. Some documents are so meaningful as to be considered sacred.

Data are, in this regard, meaningless on their own; they do not signify, they simply are. These data are the utterances of the spimes that surround us.

Documents as Trees, Data as Streams

The argument for shifting away from markup languages as data formats is not just practical, it’s philosophical: it’s about pivoting our conception away from the dominant metaphor of documents — trees — towards one far more suitable for data — streams.

Trees are rooted and finite: you can’t chop up a tree and easily put it back together again (while XML has made concessions to document fragments, it is not a natural fit).

Streams can be split, sampled, and filtered. The divisibility of data streams lends itself to parallelism in a way that document trees do not. The stream paradigm conceives of data as extending infinitely forward in time. The Twitter data stream has no end: it ought have no end tag.

Conceiving of data as streams moves us out of the realm of static objects and into the realm of signal processing. This is the domain of the living: where the web is not an archive but an organism, reacting in real-time.

XML Considered Harmful for Data

XML is a poor language for data because it solves the wrong problems — those of documents — while leaving many of data’s unique issues unaddressed.   But many promising alternatives exist — microformats like JSON, Thrift, and even SQLite’s file format – as I will detail in a my next post.

In prepping for tonite’s talk at the Google IO Ignite event, this quote inspired me to muse about how sex appeal and statistics might go together: so I chose to mash up a few scatter plots with Andy Warhol’s Marilyn Monroe.

Statisticians’ sex appeal has little to do with their lascivious leanings (ahem, BedPost), and more with the scarcity of their skills.  I believe that the folks to whom Hal Varian is referring are not statisticians in the narrow sense, but rather people who possess skills in three key, yet independent areas:  statistics, data munging, and data visualization.  (In parentheses next to each, I’ve put the salient character trait needed to acquire it).

Skill #1: Statistics (Studying). Statistics is perhaps the most important skill and the hardest to learn. It’s a deep and rigorous discipline, and one that is actively progressing (the widely used method of Least Angle Regression was only recently developed in 2004). I expect to be on its learning curve my entire life. This being the case, people who possess a solid grasp of modern statistics are rare.   And yet problems that require its application continue to multiply.  The text that I was exposed to in graduate school and find to be an unparalleled survey is Hastie, Tibshirani, and Friedman’s Elements of Statistical Learning.

Skill #2: Data Munging (Suffering). The second critical skill mentioned above is “data munging.” Among data geek circles (you can find us with a Twitter search for #rstats), this refers to the painful process of cleaning, parsing, and proofing one’s data before it’s suitable for analysis. Real world data is messy. At best it’s inconsistently delimited or packed into an unnecessarily complex XML schema. At worst, it’s a series of scraped HTML pages or a thoroughly undocumented fixed-width format.

A good data munger excels at turning coffee into regular expressions and parsers, implemented in a high-level scripting language of choice (often Perl, Python, even Javascript). This is problem solving with programming, and quite different from statistics. An aspiration towards elegance — in the form of a perfect XSLT filter, for example — is rarely rewarded, and often punished. A decade ago, I thought that the world’s data would soon be well-structured, and my talent for syntactical incantations of regular expressions would be a moot skill. I was wrong. (Perhaps there’s an analogy with the paper industry: the growing volume of data means we’ll likely need more regular expressions before we need less).

Related to munging but certainly far less painful is the ability to retrieve, slice, and dice well-structured data from persistent data stores, using a combination of SQL, scripting languages (especially Python and its SciPy and NumPy libraries), and even several oldie-but-goodie Unix utilities (cut, join).

And when data sets grow too large to manage on a single desktop, the samurai of data geeks are capable of parallelizing storage and computation with tools like 96-nodes of Postgres, snow and RMPI, Hadoop and Mapreduce, and on Amazon EC2 to boot.

Skill #3: Visualization (Storytelling). This third and last skill that Professor Varian refers to is the easiest to believe one has.  Most of us have had exposure to basic chart-making widgets of Excel (and to date myself, tools like Harvard Graphics). But a little knowledge is a dangerous thing: these software tools are often insufficient when faced with the visualization of large, multivariate data sets.

Here it’s worth making a distinction between two breeds of data visualizations, which differ in their audience and their goals. The first are exploratory data visualizations (as named by John Tukey), intended to faciliate a data analyst’s understanding of the data. These may consist of scatter plot matrices and histograms, where labels and colors are minimally set by default. Their goal is to help develop a hypothesis about the data, and their audience typically numbers one or a small team.

A second kind of data visualization are those intended to communicate to a wider audience, whose goal is to visually advocate for a hypothesis. While most data geeks are facile with exploratory graphics, the ability to create this second kind of visualization, these visual narratives, is again a separate skill — with separate tools.  (R is excellent for static visualizations, but cannot compete with the kinds of rich interactive visualizations that tools like Processing and Flare make possible). Luckily, successful collaboration often occurs between data analysts and designers, the occasional fracas notwithstanding.

The ability to visualize and communicate data is critical, because even with good data and rigorous statistical techniques, if the results of an analysis are poorly visualized, they will not convince: whether it’s an academic discovery or a business proposal.

Put All Three Skills Together: Sexy. Thus with the Age of Data upon us, those who can model, munge, and visually communicate data — call us statisticians or data geeks — are a hot commodity.  I grew up before the age of geek chic, when the computer wizzes were social pariahs, and feature-length movies were dedicated to nerds seeking revenge.  But in the last decade, Steve Jobs became an icon, the Internet became cool, and an entire generation of tech kids grew up well adjusted.  They even built the social web to prove it.   I believe the same could happen to statistics and data geeks too.

• Stamen: Reaching through Maps
• Protovis: A Declarative, Open Source Graphical Toolkit
• A Mathematician’s View: A Visualization is a Hypothesis
• UUorld: Multidimensional Extrusion Maps

Stamen: Reaching through Maps

Eric Rodenbeck (Stamen) started by highlighting several mapping visualizations that Stamen has been hacking on recently and in the past, including Cabspotting in San Francisco , Crimespotting in Oakland, and Olympic Stadium spotting in London.

Eric showed how Stamen has attempted to move away from what Schuyler Erle has dubbed “red dot fever”, whereby the overlayed data can overwhelm our visual attention, and toward allowing various data layers to “reach through” the maps.

For example, the London Olympic maps provide a mixture of schematic, satellite, and webcam images.  These various drill-downs of detail are not all exposed, but rather collaged.  Even more interesting was a movable ‘lens’ that, as it is moved over regions of a map, reveals another layer (reminiscent of a polarized-light based mural at Boston’s MoS).  In these ways, additional layers of data are only selectively brought into focus (echoing a design pattern in Japanese gardening, mie gakure, meaning “seen and unseen”).

One practical gem that Mike Migurski shared regarding the Oakland Crimespotting site was, “the design of a comments section is a huge part of how its perceived and used.”  Nota bene, social web developers.

Protovis: A Declarative, Open Source Graphical Toolkit

Mike Bostock (Stanford CS) introduced Protovis, an extensible visualization toolkit implemented using Javascript’s canvas element. Protovis draws inspiration from Leland Wilkinson’s Grammar of Graphics, which argues for moving away from the prevailing method of building visualizations, where data are simply poured into one of several chart types — pie, stacked bar, or scatter.

Wilkinson argues that visualizations should not be cast from chart typologies, but rather composed of graphical primitives. In Protovis, these primitives include dots, areas, lines, and labels (called “marks”).

Among Protovis’s strengths are:

A More Declarative Syntax for Creating Graphics

One disadvantage of directly using Javascript’s canvas is its imperative style. To draw a diagonal line, the code must manipulate and move a pen using x,y coordinates. With Protovis, however, the code declares (roughly) “add a bar to this graph” (example). Thus Protovis provides a grammar for statements about graphical marks, rather than statements about graphical mechanics.

Visible Open Source

With Protovis, the source code is not just open and available, it’s viewable from within the browser. I have an admittedly personal bias for open source data visualization, but lowering the barriers to sharing source code ultimately drive faster adoption and iteration of visualization techniques.

Mike has used Protovis to recreate classic data visualizations by Will Burtin, Florence Nightingale, William Playfair, and others. You can find these at the Protovis site and in their InfoVis ‘09 paper.

(For those interested in a Wilkinson-inspired approach for graphics in R, check out Hadley Wickham’s ggplot).

A Mathematician’s View: A Visualization is a Hypothesis

Jason Morton (Stanford Mathematics) made the argument that a data visualization is not merely a descriptive vessel, it is a predictive model.

A visualization is a model is because, especially with large data sets, not every dimension of every observation can be shown. Quite simply, a (compressed) 100k data visualization cannot losslessly describe a (compressed) 10 Mb data set: information must be discarded. What remains is a model of the original data, albeit a visual model.

Moreover, a data visualization’s model is predictive: it presents a hypothesis about how observable data points were generated, and implies predictions about future, as-yet-unobserved data.

Seen from this perspective, Stamen’s Crimespotting maps are powerful precisely because they make compelling hypotheses about when and where crime occurs in Oakland. Their London Olympic maps, which integrate time series photographs of the stadium site, take a position about the pace of construction and how it is impacting the landscape.

“Form Ever Follows Function”

And if the function of a data visualization is to make hypotheses, then its form should follow this function. The arbitrary use of color, position, shape, and ornament — only adds noise.

The ever popular Wordle provides a visual model for word distribution in a text: more frequent words are larger. However, a word’s color, position, and font are arbitrarily chosen - they carry no meaning, and model nothing. Indeed, the “randomize” button is an admission of as much (for it does not randomize size).

Adding arbitrary marks or dimensions to a visualization carries two related risks: first, it can obscure the true model that’s trying to be conveyed (what do same-colored have in common?); second, this added complexity, beyond polluting the information channel, has a cost: the visualization is larger. Bar graphs with iPhone ads in the background cannot be succinctly rendered.

The parallels to the modernist movement in architecture are obvious. Adolf Loos wrote in 1908 that “the evolution of culture marches with the elimination of ornament from useful objects.” The American modernist Louis Sullivan proclaimed that “form ever follows function.”

But the truth is that stripping visualizations down to their bare models can be counterproductive. Call it noise or ornamentation, but even visual marks that do not advance a hypothesis can act to support it,  by guiding the eye, providing context, or otherwise speeding the absorption of a pattern by the human brain. At the very least, this functionalist perspective can help data visualizers use ornamentation intentionally, not inadvertently.

UUorld: Multidimensional Extrusion Maps

Zach Wilson (UUorld) showcased his company’s software that simplifies creating and exploring extrusion maps. Among the several interesting applications of his software, Zach showed off a temporal visualization of the spread of swine flu in the United States over the past several weeks.

In response to the critique that layering data dimensions on two-dimensional maps could be done more effectively by use other indicators such as color — instead of the simulation of a third dimension of height — Zach indicated that research has shown that physical dimensions (or their simulation) possess greater visual saliency to the human eye.

Zach also mentioned UUorld’s data portal which contains thousands of downloadable statistics from a variety of public sources; some of which have been used to generate UUorld visualizations.

“Avoiding catastrophe becomes the first principle in bringing color to information: Above all, do no harm.”  — Envisioning Information, Edward Tufte, Graphics Press, 1990   

Color is one of the most abused and neglected tools in data visualization. It is abused when we make poor color choices; it is neglected when we rely on poor software defaults. Yet despite its historically poor treatment at the hands of engineers and end-users alike, if used wisely, color is unrivaled as a visualization tool.

Most of us think twice before walking outside in fluorescent red underoos. If only we were as cautious in choosing colors for infographics. The difference is that few of us design our own clothes. But until good palettes (like ColorBrewer) are commonplace, to get colors that fit our purposes, we must be our own tailors.

While obsessing about how to implement color on the Dataspora Labs’ PitchFX viewer I began with a basic motivating question:

Why use color in data graphics?

If our data are simple, a single color is sufficient, even preferable. For example, below is a scatter plot of 287 pitches thrown by the major league pitcher Oscar Villarreal in 2008. With just two dimensions of data to describe — the x and y location in the strike zone — black and white is sufficient. In fact, this scatter plot is a perfectly lossless representation of the data set (assuming no data points perfectly overlap).

Fig 1. Location of Pitches (Villarreal, HOU, 2008)

But what if we’d like to know more: for instance, what kinds of pitches (curveballs, fastballs) landed where? Or their speed?  Visualizations live in two dimensions, but the world they describe is rarely so confined.

The defining challenge of data visualization is projecting high dimensional data onto a low dimensional canvas. (As a rule, one should never do the reverse: visualize more dimensions than what already exist in the data).

Getting back to our pitching example, if we want to layer another dimension of data — pitch type — into our plot, we have several methods at our disposal:

1. plotting symbols - vary the glyphs that we use (circles, triangles, etc.),
2. small multiples - vary extra dimensions in space, creating a series of smaller plots
3. color - we can color our data, encoding extra dimensions inside a color space

Which techniques you employ depend on the nature of the data and the media of your canvas. I will describe all three by way of example.

Multivariate Method I:  Vary Your Plotting Symbols

Fig 2. Location and Pitch Type (Villarreal, HOU, 2008)

In this plot, I’ve layered the categorical dimension of pitch type into our plot by using four different plotting symbols.

I consider this visualization an abject failure.  In fact, the prize for my most despised graphs in graduate school goes to bacterial growth curves rendered this way . The reason these graphs make our heads hurt is because (i) distinguishing glyphs demands extra attention (versus what academics call ‘pre-attentively processed‘ cues like color), (ii) even after we visually decode the symbols, we have yet another step: mapping symbols to their semantic categories. (Admittedly this can be improved with Chernoff faces or other iconic symbols, where the categorical mapping is self-evident).

Multivariate Method II:  Small Multiples on a Canvas

Folding additional dimensions into a partitioned canvas has a distinguished pedigree in information graphics. It has been employed everywhere from Galileo sunspot illustrations to William Cleveland’s trellis plots. And as Scott Mccloud’s unexpected tour de force on comics makes clear, panels of pictures possess a narrative power that a single, undivided canvas lacks.

In this plot below, the four types of pitches that Oscar throws are splintered horizontally.   By reducing our plot sizes, we’ve given up some resolution in positional information. But in return, patterns that were invisible in our first plot, and obscured in our second (by varied symbols) are now made clear (Oscar throws his fastballs low, but his sliders high).

Fig 3:  Location and Pitch Type (Villarreal, HOU, 2008)

Multiplying plots in space works especially well on printed media, which can hold more than ten times as many dots per square inch as a screen. Both columns and rows can be used to lattice over additional dimensions, the result being a matrix of scatter plots (in R, see the ‘splom‘ function).

Multivariate Method III: Color Your Data

So why bother with color?

First, as compared to most print media, computer displays have fewer units of space, but a broader color gamut. So color is a compensatory strength.

For multi-dimensional data, color can convey additional dimensions inside a unit of space — and can do so instantly. Color differences can be detected within 200 ms, before you’re even conscious of paying attention (the ‘pre-attentive’ concept I mentioned earlier).

But the most important reason to use color in multivariate graphics is that color is itself multidimensional. Our perceptual color space — however you slice it — is three-dimensioned.

In the example below, I’ve used color as a means of encoding a fourth dimension of our pitching data: the speed of pitches thrown. The palette I’ve chosen is a divergent palette that moves along one dimension (think of it as the ‘redness-blueness’ dimension) in the CIELUV color space, while maintaining a constant level of luminosity.

Fig 4. Location, Pitch Type, and Velocity (Villarreal, HOU, 2008)

Holding luminosity constant is important, because luminosity (similar to brightness) determines a color’s visual impact. Bright colors pop, and dark colors recede. A color ramp that varies luminosity along with hue will highlight data points as an artifact of color choice.

I chose only seven gradations of color, so I’m downsampling (in a lossy way) our speed data - but further segmentation of our color ramp is not likely to be perceptible.

I’ve also chosen to use filled circles as my plotting symbol, as opposed to the open circles in all my previous plots. This is done to improve the perception of each pitch’s speed via its color: small patches of color are less perceptible. But a consequence of this choice — compounded by our choice to work with a series of smaller plots — is that more points overlap. We’ve further degraded some of our positional information. However, in our last step, we attempt to recover some of this.

Now I’ve finally brought color to bear on this visualization, but I’ve only encoded a single dimension — speed. Which leads to another question:

If color is three-dimensional, can I encode three dimensions with it?

In theory, yes. Colin Ware researched this exact question. In practice, it’s difficult. It turns out that asking observers to assess the amount of ‘redness’, ‘blueness’, and ‘greenness’ of points is possible, but not intuitive (I suspect it’s somewhat like parsing symbols).

Another complicating factor is that a nontrivial fraction of the population has some form of color blindness. This effectively reduces their color perception to two dimensions.

And finally, the truth is that our sensation of color is not equal along all dimensions; it’s thought the closely related ‘red’ and ‘green’ receptors emerged via duplication of the single long wavelength receptor (useful for detecting ripe from unripe fruits, according to one just-so story).

Because the high level of dichromacy in the population, and because of the challenge of encoding three dimensions in color, I  feel color is best used to encode no more than two dimensions of data.

So, for my last example of our pitching plot data, I will introduce luminosity as a means of encoding the local density of points (using a kernel density estimator). This allows us to recover some of the data lost by increasing the sizes of our plotting symbols.

Fig 5. Location, Pitch Type, Velocity, and Density (Villarreal, HOU, 2008)

Here we have effectively employed a two-dimensional color palette, with blueness-redness varying along one axis for speed, and luminosity varying in the other to denote local density.

One final point about using luminosity. Observing colors in a data visualization involves overloading, in the programming sense. We rely on cognitive functions that were developed for one purpose (perceiving lions) and use them for another (perceiving lines).

Since we can overload color any way we want, whenever possible, we should choose mappings that are natural. Mapping pitch density to luminosity feels right because the darker shadows in our pitch plots imply depth. Likewise, when sampling from the color space, we might as well choose colors found in nature. These are the palettes our eyes were gazing at for the millions of years before #FF0000 showed up.

Color, used thoughtfully and responsibly, can be an incredibly valuable tool in visualizing high dimensional data.

FutureMan Asks: What about Animation?

This discussion has focused on using static graphics in general, and color in particular, as a means of visualizing multivariate data. I’ve purposely neglected one very powerful tool:  motion. The ability to animate graphics multiplies by several orders of magnitude the amount of information that can be packed into a visualization.  But packing  information into a time-varying data structure has to be done by someone (you or me) and from my view, this remains a significant challenge.  Canonical forms of animated visualizations (equivalent to the histograms, box plots, and scatterplots of the static world) are still a ways off, but frameworks like Processing and Prefuse are a promising start towards their development.

Methods

The final product of these five-dimensional pitch plots — for all available data for the 2008 season — can be explored via the PitchFX Django-driven web tool at Dataspora labs.

All of the visualizations here were developed using R and the Lattice graphics package.  (Of note, Hadley Wickham is developing ggplot2, a bold re-write of the R graphics system based on a grammar of graphics).

References for Further Reading

• Ross Ihaka - Lectures on Information Visualization, Lectures 12-14

• Colin Ware - Information Visualization, Ch. 4

• Edward Tufte, Envisioning Information, Ch. 4.

• Deepayan Sarkar - Lattice: Multivariate Data Visualization with R (web site with code)

• Maureen Stone - StoneSoup Consulting (color consultant to Tableau Software)

• Stephen Few, Information Dashboard Design, Ch. 4

I was going to blog all about it — but Tom Carden of Stamen Design already has a great write-up.

… Dataspora invited a few people to a Dataviz Salon yesterday evening. Mike and I went along and huddled in a brick-built basement in SoMa to listen to the following:

.