Really, the whole politicized debacle offends my sensibilities as a citizen and as a data professional.

Furthermore, the recent zeal for budget cuts has resulted in budget cuts that affect our ability to make smart budget cuts. Specifically, I’m talking about attacks on government-created statistical data—data that’s* used by lawmakers, social service organizations, and businesses to make decisions and allocate increasingly-scarce resources.

Two examples I’ve written about recently:

• Is Federal Spending Transparency on the Decline?: a guest post for the Sunlight Foundation’s blog about the demise of the Consolidated Federal Funds Report and why that makes it harder to understand federal spending.
• American Community Survey Under Attack: the House recently passed a spending bill that prohibits the Department of Commerce from funding the American Community Survey (ACS). The yearly ACS replaced the decennial census long-form questionnaire, and its data helps* state and local governments determine how to distribute funds, among other things. See here, here, and here for more information about the widespread usefulness of the ACS.

Of course, order and logic sometimes need to be tempered with a dose of pragmatism. But when our governing body is governed almost entirely by short-term thinking, we should think about not electing them again.

*Language evolves!

The biggest a-ha moment, and one that I’ll be writing about in the future, came from Ben Goldacre’s keynote, when he compares big data practitioners to drunks looking for car keys only where the light shines. We focus on the data that’s available without asking, “what’s missing?” Plus, it’s fun to hear someone with a British accent say “blobogram.”

Getting Started With Hadoop

This was a three hour overview of what Hadoop is, how its two components—Hadoop Distributed Filesystem (HDFS) and MapReduce—work, and the tools that round out its ecosystem. Sara from Cloudera explained the details clearly and in plain English, using language that could be understood by technologists of all stripes, not just big data geeks.

• Hadoop gives you scalability and the ability to handle failure
• In return, you need to rethink your data processing algorithms
• Writing MapReduce jobs directly is like coding assembly. Hadoop ecosystem has tools that help you create, schedule, and run jobs, query HDFS, move data between HDFS and other types of storage, and aggregate files for Hadoop input

The Craft of Data Journalism

This three hour sessions was run by two folks from The Guardian (one is Simon Rogers), who demonstrated some of their data-driven projects and shared some of their tools. The Guardian’s data stories are blogged here: http://www.guardian.co.uk/news/datablog.

• Google spreadsheets: crucial because they’re free, easy, can be used as a backend, and are easy to mashup
• Google Fusion tables: used to create maps from location data. Pop-ups that appear for each data point on the map are customizable
• Google refine: for data cleanup
• Scraperwiki: to pull data from websites
• Bit.ly://fusionborderlinks: Guardian-created list of links to different boundaries (can be combined with Fusion table location data to add boundaries to maps)
• Presenters reported that ~80% of their time is getting, cleaning, and verifying the data—not creating the visualizations. This is consistent with my experience and also verifies that pre-cleaned & formatted data could be useful to journalists

Wednesday Keynotes

Keynotes were a series of short presentations. Favorite quotes:

• Systems thinking is the science in data science
• Store raw data and project schemas onto it dynamically – Doug Cutting, Cloudera
• More, messy data has greater value than smaller, cleaner data. – Dave Campbell, Microsoft
• Problems confronting humanity: this is where big data needs to go – Mike Olson, Cloudera
• Big data is most valuable when it tells us what we don’t know we don’t know (HT Donald Rumsfeld). Much more valuable than data puking onto dashboards – Avinash Kaushik , Market Motive
• Big data can and should be used to find out what’s missing – Ben Goldacre, Bad Science
• What data do we have that we aren’t using? What are we throwing away—especially things that could better help us quantify impact (web logs, eblast data, social media stats)

The Science of Visualization

• When creating visualizations exploit the power of human visual processing
• Humans are slow are mental math (e.g., adding things up in tables) but fast visually
• Therefore, use visual pop-outs when possible (e.g., coloring numbers that should stand out)
• Data view: exploit bi-directional processing
• Data stories: show people where to look
• Purpose of visualizations is to speed up the cycle of analysis
• Some best practices (get the slides for the complete overview):
• If there are 3 variables, code the 3rd with size, not with color
• People see area, not height
• Size is ordinal; don’t use it to encode non-ordinal data
• Color is okay of ordinal data but works even better for non-ordinal data (as long as there aren’t too many values, up to 20 or so)…takes advantage of the visual pop-out
• Nominal: if there are over 20 items, colors become less effective, but shapes can work (good example is the animal graph in Tufte’s Beautiful Evidence)
• Small sizes make colors and shapes much less effective
• Although heatmaps are currently popular, they are often not the best choice for quantitative data
• If heatmap is being used to identify outliers, that’s not really quantitative data, so it’s okay
• But otherwise, it’s often best to use size instead of heatmaps (especially filled maps)
• Stephen Frew’s Show me the Numbers: accessible and practical advice for getting started

Effective Data Visualization

Although this session yielded some useful information, it was too focused on the presenter’s company, with not enough information about general best practices. That said, the company itself (DataMarket) is interesting—they are trying to be the Google of datasets. You find a dataset on their site, choose it, and are immediately presented with a visualization. Furthermore, people can now upload their own data to the site.

In other words, they have solved the problem of creating generic visualization code that will work with any dataset.

• DataMarket uses a Python/Django/Postgres stack
• In 2010, they reviewed 100+ libraries to see which met their 5 requirements for the site (compatibility for all major browsers, server-side rendering, vector output, iOS compatibility, full control). Protovis was the choice, although it lacked server-side rendering and doesn’t work with IE 7/8 (due to SVG output)
• DataMarket developed a server-side solution using node.js, svglib, and some other stuff to transform SVG to PDF for rendering
• They created a shim for IE 7/8 that captures Protovis SVG output and transforms it to VML (IE 7/8 = ~20% of their users)
• Although Protovis is no longer being actively developed, there is still an active community. DataMarket’s strategy is to stick with Protovis until more IE users are on 9 and then switch to D3

Data Visualization for a Better World

This was a Bay-area meetup group and not part of Strata. Really great event—Jake Porway, the founder of Data Without Borders, is smart, humble, and passionate about matching data scientists with social and non-profit orgs.

• NYT R&D Twitter sharing visualization front-end is JS and processing. Python to get data and MongoDB to store
• 2nd viz demoed uses OpenPaths.cc and webgl
• On their own, hackers will develop apps that solve their own programs. Therefore, hackathons need to include those who will use the apps to change the world (i.e., social sector orgs)
• “Unglamorous” things that are easy for data scientists to do can be transformative for social service orgs.

Video Graphics: Engaging and Informing

• Video lets you bring a human persona to the data
• On articulating data into stories:
• It goes beyond finding the x & y coordinates that match. Must engage people in the insights
• The story is the concept, not the characters
• Information doesn’t want to be free…it mostly wants to lie around. Don’t let your data lie around!
• Don’t confuse insight with story
• Find the “red thread” of story
• On showing multi-modal data:
• Words and pics need to be brought together
• Cross the streams! Show AND tell. Pictures AND words. Data AND interpretation.
• Don’t be afraid of entertainment and gloss
• Use the visual strengths of the medium
• Not everything needs to look like a diagram
• Why bother with flat video graphics (e.g, a bunch of words & numbers with narration)? People make these b/c they’re cheap to make, but a PowerPoint or PDF would be just as good.
• Example of non-flat words: intro text to Star Wars movies.
• Make things real
• Spread data socially
• Humor
• Use a known expert
• Video must be relevant to NOW (e.g., Crisis of Credit video)
• Videos work well as an introduction to a subject

From Predictive Modeling to Optimization

• Predictive models don’t cause change!
• We don’t pay enough attention to the process of using predictive model output to optimize
• Traditional process (using insurance biz as an example)
1. Objective: more money!
2. Levers: what variables you can tweak (policy prices, etc.)
3. Data
4. PhD-quality algorithm (actuaries)
• Need to rethink this process!
• New process uses:
1. Modeler/simulator to test what happens when you pull the levers
2. Optimizer: pulls levers over and over again until you reach the desired objective (replaces the PhD-quality algorithm)
• Think of an area where the optimization process hasn’t yet been done. Good example is marketing:
1. Objective: maximize lifetime value of customer
2. Levers: recommendation system, offers, discounts, customer care calls
3. Data. What data can you collect? That is the right question, not “what data do you have already.” In insurance example, presenter implemented random pricing for 6 months to get enough data to create the model.
• You need a causal dataset to create a good model. For example, Amazon doesn’t have one, so their recommendations tend to tell you things you already know about.
• Look to the engineers! They have modeling/simulating/optimizing for decades, solving real-world problems

Visualizing Geo Data

• Just Plot It from Aaron Koblin is a helpful resource for map ideas
• Processing creates beautiful data visualizations
• TileMill is great for mapsslippy map tiles
• Unfolding: glues together Processing and TileMill
• Cartograms (not covered in the session) are useful for displaying quantities on maps
• Unless you have training, do not choose your own colors for maps! Use Color Brewer.
• Stats of the Union is an excellent example of an iPad mapping app
• You don’t always need to use the traditional boundaries on a map. A good example is the United States of Craigslist, which displays data according to Craigslist regions.
• Get Shapefiles at census.gov/geo/www/tiger
• Pyshape: Python library for shapes
• Add a spatial tree to your bad of tricks—helps maps render much faster. Rtree is one tool. Quick way to query and associate data with counties (on Github)
• Choice of bin sizes for map colors change the story. See slides for a great example of this!
• Equal intervals are bad! Doesn’t take into account the way your data is structured.
• Using quantiles is a reasonable first pass.
• Presenter (Jason Sundram of Where, Inc) has tweaked D3 choropleth code to make it interactive! See Github…
• WebGL used to make the spinning globe thing
• Very cool, but probably not useful to an org that specializes in U.S. data only
• Need < 250,000 points
• Cheat by binning (e.g., 72.123456 = 72.1)
• Scale values
• Viz.runningwithdata.com/vgd
• Bit.ly/Strata_vgd

Big Data for the Common Good

• Jake Porway and Virginia Carlson
• There are two kinds of government data:
• Data generated as a result of the government going about its business (e.g., logs of White House visitors). When Gov 2.0 folks talk about data, they are mostly talking about this.
• “Designed” data. Created for the public (e.g., Census Bureau reports such as CFFR).
• The Designed Data is what social orgs often rely on for their business, but it is going away. For example, food banks can’t get good data about their local economic conditions and know where to provide services.
• Foundations are asking for metrics. Organizations are sitting on tons of data. Result = funnel effect…more data coming in all the time, but the outgoing pipe remains the same size.
• There is a gap between civic coders and community groups/leaders.
• Something that may seem easy/trivial to a data scientist might be transformative to a non-profit
• You can’t plan social programs without data any more than you could do it without a budget.
• Virginia is pitching commercial data for the common good.
• Break down silos between data (e.g., gov), talent (e.g. Google), and social good (e.g., Unicef).
• Example: Could Google tell the foodbank where searches for their services are coming from?

Exploring the Stories Behind the Data

• Cheryl Phillips, Seattle Times
• Data’s increasing involvement in news stories doesn’t change the basics: you still need to tell who, what, when, why, how. Good example of this is NYT’s Faces of the Dead (casualties of war project)
• Nutgraph…journalism term. 3rd or 4th paragraph in a story…the “why” that follows the lead
• The nutgraph shows the reader why he or she should care, so don’t put it too late in the story
• Data in the story should support the nutgraph. Don’t plug in too many numbers.
• Visualizations also need a good lead-in and nutgraph
• Narrative techniques apply. Are there patterns (e.g., chronologic, geographic)?
• Images and visualizations allow you to use details without overwhelming the reader with the details in text.
• However, too many details in a viz can be as boring as too many details in text.
• Cheryl thinks visualizations should allow readers to annotate and share and encourages developers to get busy creating that feature.

The very next day, Hjalmar Gislasun of DataMarket gave a talk at O’Reilly’s Strata Conference. DataMarket faced the same problem back in 2010 after reviewing over 100 visualization libraries and choosing Protovis (a predecessor of D3). Not wanting to exclude the 20% of the world still using IE 7/8, they developed protovis-msie, a tool to convert Protovis SVG output to VML, a vector format understood by older browsers.

And…they open sourced it. So Protovis is now on the table for use at National Priorities Project. Thank you, DataMarket!

Like Flare, Protovis is no longer under active development. That said, it still has an active user community (unlike Flare). And the output won’t be Flash, so iOS is back on the table.

DataMarket’s strategy is to continue using Protovis until most IE users are on version 9 (which supports SVG) and then switch over to D3. It was refreshing to hear browser support strategies from people developing visualizations for commercial use; they don’t have the luxury of ignoring IE 8, which is tempting to do but not viable in the real world.

Flare was ideal because it includes sample code for a stacked area chart with tooltips–exactly what we wanted. I had some concerns about the Flash output, but many of our website visitors use browsers that don’t support SVG (IE8), so tools like D3 aren’t an option just yet.

Here’s a preview of what we’ll include (not the final version).  The first example is built with normalized data:

For the second example (total federal spending by category), we wanted to convey the overall size of the budget over time, so we didn’t normalize the data. As a result, the huge numbers caused some formatting issues, but it’s still an interesting story–especially the 2009 spike. Also note the rise in healthcare spending over time: 7% of the budget in 1976 and 25% in 2013.

Flare makes it easy to lay out the data and create the animated transitions, and after making a few tweaks to the Flare library and the stacked area sample code, I’m happy with the way these turned out.

That said, I’d be reluctant to use Flare again. It isn’t being actively developed, and there’s nowhere to turn for help when you get stuck. Visualizations are evolving, and the tools to create them–no matter how good they are–need to evolve too.

Excel remains the de-facto graphing tool at National Priorities Project. A simple chart is often the best way to convey information about federal spending and budgeting, and Excel is the common language among our researchers and IT team.

Using Excel, however, is no excuse for ignoring style and the best practices of information display. So many organizations put out amazing, well-researched publications and then tack on default Excel graphs as an afterthought. But graphs are often what people look at first, and they deserve to be first-class citizens in the editing process.

I created some Excel chart templates for NPP, drawing on two sources for inspiration and practical advice: the classic Visual Display of Quantitative Information by Edward Tufte and The Wall Street Journal Guide to Information Graphics by Dona Wong.

Tufte is big on eliminating “unnecessary ink” that distracts from the information, and Wong advocates requiring the least amount of work on the reader’s part. With their advice in mind, I modified Excel’s default bar chart from this:

Bar chart: Excel default

To this:

Bar chart: new template

• Smaller gap between bars
• Don’t make readers guess the numbers; if possible, label the bars directly
• Direct labeling means you don’t need the noisy gridlines or even the x-axis
• Remove the y-axis tick marks for even more noise reduction
• Get rid of those zeros by showing data in millions or billions
• Make sure the entire length of the bars is shown (in this case, by setting the horizontal axis minimum to zero). HT Jon Peltier.

The pie chart got a similar treatment. The Excel default:

Pie chart: Excel default

The new template:

Pie chart: new template

• Label the pie slices directly—don’t make people use a legend to decode
• Avoid the default Office color palette and develop your own (ours is based on colors from our website)
• A white line between pie slices emphasizes the boundaries

Excel isn’t perfect, but it’s out there in the world, and you can’t ignore it. Luckily, a little extra effort goes a long way.

So when writing a brief paragraph to introduce a new dataset in the Federal Priorities Database, I thought I nailed it:

Federal tax collections represent the amount of federal taxes paid in a year.  The IRS publishes this information by state, breaking it into ten categories.  Because part of our job is to make government-published data more useful, we condensed these ten categories into three: taxes paid by businesses, taxes paid by individuals, and total taxes collected. We believe that these broad categories better reflect what most people want to know about tax revenue and are less confusing than the original data. In fact, the individual taxes calculation played an integral part in a recent publication, Federal Spending Keeps Iowa, New Hampshire Afloat.

I have a few trusted readers in the office, so I sent the text around for some quick feedback. About five minutes later, one of them appeared at my desk, saying that my language was non-accessible and “made her eyes roll back in her head.” Wow. Hard to hear, but she followed up with some specific suggestions that resulted in a much improved, non-eye-rolling version:

We recently took some data from the IRS and made it even better. We call it Federal Tax Collections, and it shows federal taxes collected in a year. Although the IRS helpfully provides these numbers for each state, they break the amounts into ten categories, not all of which are useful. For example, do any but the wonkiest people need to know how much railroad retirement tax was collected in 2010? To make things easier, we condensed these categories into taxes paid by businesses, taxes paid by individuals, and total taxes paid. We think these broader groupings are less confusing and more practical. Our recent publication, Federal Spending Keeps Iowa, New Hampshire Afloat, shows the individual taxes category in use.

Thank goodness for honest feedback! It seems obvious, but the lessons here are easy to overlook when you’re cranking through a to-do list.

• No matter what kind of deadline you think you have, it’s always worth the time to get other eyes on something your organization will publish, even if it’s just a small blurb, and even if your colleagues are busy. After all, it’s in everyone’s best interest to show the world your best stuff.
• Make sure your trusted readers include, as my friend calls herself, the dummy demographic, a term of affection for readers who aren’t experts in your subject matter.

Overall, however, being a technologist is a lot more exciting in 2012 than it was in the Y2K remediation days. That said, it’s time to stop worrying about every shiny new toy that comes along and focus on a few things that appeal to my interests and career aspirations.

NoSQL
I know my way around an RDBMS, but 2012 will be the year I stop reading about NoSQL and start doing something with it. Flavor TBD.

D3.js
Data visualization is more than a fad. Not those disjointed, meaningless infographics floating around, but numbers rendered as pictures that tell a story. I’m not a graphic designer, but I am a programmer, so it makes sense to learn a tool that can programmatically create pictures of the federal data I work with every day.

The administrator of a blog I follow recently asked the question, “should I force our blog’s authors to use Markdown?”

I then read a completely unrelated post on another blog that advised WordPress users to get rid of the visual text editor because “it makes you lazy” and you should “force yourself to learn some basic HTML.”

When I got into a heated discussion with another developer about the above two items, he used the same word to describe non-Markdown/HTML-writing, WYSIWYG-dependent online content authors: lazy.

This isn’t the first argument I’ve had with a technologist who likes to play the lazy card, and it won’t be the last. I’m not disputing that everyone in today’s workforce should always be learning; the days of doing the same job the same way for thirty years are over. But to imply that a blogging co-worker isn’t holding up her end of the learning bargain because she doesn’t want to learn HTML or Markdown is arrogant.

As technologists, we explore and experiment with new technology. Our content-creating colleagues presumably explore and experiment in their respective areas of expertise. Who are we to dictate that they should increase their cognitive overhead by worrying about valid XHTML markup?

Yes, WYSIWYG editors are pretty terrible. They puke out Microsoft Word detritus and let you change the font color to cyan. Forcing people to write in Markdown, however, shifts a technical problem from the technologist to the user, which is the opposite of ideal.

This rant isn’t about blog authors or Markdown. It’s about acknowledging that technology isn’t everyone’s primary concern, nor should it be.

Of course there are appropriate times to force technology changes. But even in those situations the resistors aren’t lazy–they just have a different job than you. In today’s hectic and stressed workplace, shouldn’t we give colleagues the benefit of the doubt and help them succeed with technology rather than slapping a label on them?

I achieved the goal, but not without a lot pain. If you’re a Windows user getting started with Python/Django, you might have an easier time installing a virtual Linux machine.

Here’s a re-cap of the Windows-specific instructions for installing Python, Django, MySQL, and a few necessary packages and tools.

• Part 1: installing Python, Pip, and iPython
• Part 2: virtual environments
• Part 3: making iPython recognize virtual environments
• Part 4: installing Django
• Part 5: installing MySQL

• Part 1: getting started
• Part 2: virtual environments
• Part 3: making iPython recognize virtual environments
• Part 4: installing Django
• Part 5: installing MySQL (you are here)

By now, you should have Django installed into a virtual environment.  These tutorials aren’t meant to cover building a django app, just to point out the quirks involved with getting a project up and running on Windows.  These tutorials also assume you want to construct real applications using a real development environment.

To that end, you’ll want a heftier database than sqlite.  We use MySQL at the office, so these instructions cover installing it and using it with Django.

Install MySQL

1. Download and install MySQL.
2. Once MySQL is installed, proceed through the configuration wizard. Check Include Bin Directory in Windows PATH box.
3. When prompted, set a password for the MySQL root account.
4. Once the installation wizard is done, open a command window and log in to MySQL with the root account: mysql -uroot -p (you’ll be prompted for the password).
5. After logging in, run the following commands to create a database, create a user for your Django project, and grant the user database access.

Install MySQL-python

You’ll need the MySQL-python package, a Python interface to MySQL.

1. Download the windows MySQL-python distribution here.  The author has some instructions about the appropriate version; assuming a 32-bit version of Python 2.7, you’d download this package (.exe).
2. After downloading, do not run the Windows installer. Doing so will install MySQL-python to your root python, which virtual environments created via –no-site-packages won’t be able to see.
3. Instead, install the downloaded package to your virtual environment by using easy_install, which can install from Windows binary installers:
   easy_install file://c:/users/you/downloads/mysql-python-1.2.3.win32-py2.7.exe (modify to reflect the location of the downloaded installer and its name).

Configure Django

Next, you’ll need to update the database-related settings of your Django project.

1. From the directory of your Django project, open settings.py using your favorite editor.
2. Update the default key in the DATABASES dictionary.  Set ENGINE to django.db.backends.mysql and set NAME, USER, and PASSWORD to the database name, username, and password you chose when installing MySQL.  See Part I of the Django tutorial for more information about database settings.
3. Open a command window, activate your virtual environment, and change to the directory of your Django project.
4. Type python manage.py syncdb. This command creates the underlying tables required for your Django project.
   
5. If the syncdb worked, you have Python, Django, and MySQL communicating in harmony.  Congratulations!  You can now proceed through the Django tutorial and create your first application.