This comic was really amusing, although it might be because I'm a big nerd entertained by all things from PHD Comics...

It's my blog, and I can laugh if I want to.

Have a nice weekend, everyone.

[Thanks, Stephen]

Check out what they have to offer. They make your data useful:

Xcelsius Present — Transform spreadsheets into professional, interactive presentations.

NetCharts — Build business dashboards that turn data into actionable information with dynamic charts and graphs.

InstantAtlas — Enables information analysts to create interactive maps to improve data visualization and enhance communication.

Tableau Software — Data exploration and visual analytics for understanding databases and spreadsheets that makes data analysis easy and fun.

Email me at nathan [at] flowingdata [dot] com if you'd like to sponsor FlowingData, and I'll get back to you with the details.

Interaction designer Christian Swinehart takes a careful look at the popular Choose Your Own Adventure books from the 1980s. We saw something like this before, but Swinehart takes it a step further.

For those unfamiliar, these CYOA books let you make the protagonists' main decisions, and your choices would have an impact on the outcome of the adventure. Some endings were bad and some were good.

With this in mind, Swineheart begins by color-coding pages. Either it's a decision page or one of five possible types of endings: great (you win), favorable, mediocre, disappointing, and catastrophic. The analysis goes from there.

Check out Swinehart's report for the full analysis.

[Thanks James & Peter & Jodi]

UPDATE: I missed out on the best part. Check out the animated versions and watch the stories unfold. Sexy. [Thanks, gregvb]

Imagine a world where data becomes the everyday, simply embedded in what you normally do. It's really not far off if you think about it. We use charts, graphs, and viz to make important decisions with investments, businesses, and to stay informed on the news, so why not use it in our own lives?

This infographic video, The Ambient Life, is a vision of that future by Freeband Communication and IIPIC. Watch for the dramatic twist in the middle. Just when you thought it was safe to go for a jog...

We've seen this type of projected future before (see here, here, and here). Still though, I never get tired of this stuff. It's going to happen... eventually.

[via datavisualization.ch via Cool Infographics]

Last week I posted some parallel coordinate plots that related SAT scores and class size. Now it's your turn to take a crack.

You can find the data that I used (and more) from the National Center of Education Statistics. There's a link to download the data as an Excel file. You can find pupil/teacher ratios here.

The two best entries each win a copy of David McCandless' The Visual Miscellaneum: A Colorful Guide to the World's Most Consequential Trivia as well as eternal glory on FlowingData. Yes. Eternal.

Different Views and Other Factors

There were a lot of great comments on alternative ways to view the dataset and other factors that play into average SAT scores and education quality. Obviously class size isn't the only thing.

For example, the percentage of graduates who take the SAT in each state plays a big part in the story that wasn't in my original graphic. Eighty-seven percent of Maine graduates took the test while only 3% in Iowa. There's going to be a higher average if the only people who take the test in your state are the ones who want to go to college.

Some also suggested a scatterplot with some line-fitting to tell a more complete story while others wanted some actual numbers to see what was up. Both good ideas.

How to Enter

For those new to Visualize This, here's how it works.

1. Take your best crack at visualizing the data
2. Submit your entry to this thread in the FlowingData forums

If you haven't registered in the forums yet (it's free), you'll want to do that first.

A few of you posted alternatives in the comments of the original post. You're welcome to submit those as your entry. Just make sure you post it to the forum thread.

Finally, you're welcome (and encouraged) to bring in other data if you feel it tells a better story.

Get your entries in by November 24, 2009, 11:59pm EST. One entry per person please.

The Prize

Like I said, the two best entries will each win a copy of The Visual Miscellaneum. If you're international, you'll just have to cover shipping. Entries will be judged based on appropriate interpretation of the data, design, aesthetics, and how well you tell the story of class size and quality of education (not just SAT).

Good luck, everyone. I'm looking forward to what you all come up with.

David McCandless' The Visual Miscellaneum: A Colorful Guide to the World's Most Consequential Trivia hit the shelves last week (in the US). As I flipped through 320 pages of original graphics during my flight from New York to California, I thought to myself, "FlowingData readers are going to love this."

What It's About

I'm sure you've seen some of David's work by now. He's been posting graphics at Information is Beautiful during the past few months, leading up to the book release and recently started doing a weekly graphic for The Guardian Datablog. The book is like that, but much more of it.

The Visual Miscellaneum has about 100 original graphics designed specifically for the book, and yes, it is very miscellaneous. On one page is a comparison between scientific consensus and skeptics of global warming, and on the very next is a chart on dictators' wives.

Not Just Charts and Graphs

If you haven't figured it out by now, many of the graphics in this book are very tongue in cheek, but at the same time, they're quite informative.

I really just had fun flipping through the pages. I felt like I was learning something new and interesting with each graphic, and while random, they seemed to be linked somehow. It was kind of like when you look something up on Wikipedia and one hour later, you find yourself reading about types of beards (yeah, there's a graphic for that). Speaking of which, a lot of the data come from Wikipedia and other open data sources, so after you see David's graphics, you can go and try things out on your own.

With that said, not every graphic is perfect. There were a few where I wasn't sure what I was looking at or it took a little while for me to process, but that's what I was expecting. Many of the graphics are probably experiments in design more than they belong in a Tufte book, but I'm all for experimentation, as I'm sure all of you know.

Bottom Line

I liked The Visual Miscellaneum a lot. It's fun, it's interesting, it's informative, and it's enjoyable.

P.S. One quick note. It says "hardcover" on Amazon, but really it's more of a thick paperback. Just so you know.

Needless to say, my old high school and middle school now each have a copy of the series. My mom hand-delivered them (thanks, Mom). I've also been sending prints to schools, libraries, and education departments across the country near those who have already bought prints for themselves (thanks, all).

But I need more help.

From here on out, until all the prints are gone out of my garage, for every print you buy, I'll send one to local education. If you simply don't have any wall space, how about sending the series to a high school near you or your local library? I'll send another to your local education board.

Get the warm fuzzies, and spread awareness today. For the cost of a few lattes, you'll be supporting education, your community, and data. Plus, you'll be getting some beautifully designed prints.

I'm not sure how I feel about this one. I mean, it's basically population density, but it's separating population and area and then comparing the two. Then the population circles are sized based on 1,000 people per square mile which seems like it was more out of convenience than anything else.

What do you think - does this graphic work? Leave your thoughts in the comments below.

[via FlowingData Flickr Group]

The Result

Here's what we're after. It's the most recent unemployment map from last week.

Step 0. System requirements

Just as a heads up, you'll need Python installed on your computer. Python comes pre-installed on the Mac. I'm not sure about Windows. If you're on Linux, well, I'm sure you're a big enough nerd to already be fluent in Python.

We're going to make good use of the Python library Beautiful Soup, so you'll need that too. It's a super easy, super useful HTML/XML parser that you should come to know and love.

Step 1. Prepare county-specific data

The first step of every visualization is to get the data. You can't do anything without it. In this example we're going to use county-level unemployment data from the Bureau of Labor Statistics. However, you have to go through FTP to get the most recent numbers, so to save some time, download the comma-separated (CSV) file here.

It was originally an Excel file. All I did was remove the headers and save as a CSV.

Step 2. Get the blank map

Luckily, we don't have to start from scratch. We can get a blank USA counties map from Wikimedia Commons. The page links to the map in four sizes in PNG format and then one as SVG (USA_Counties_with_FIPS_and_names.svg‎). We want the SVG one. Download the SVG file onto your computer and save it as counties.svg.

The important thing here, if you're not familiar with SVG (which stands for scalable vector graphics), is that it's actually an XML file. It's text with tags, and you can edit it in a text editor like you would a HTML file. The browser or image viewer reads the XML. The XML tells the browser what to show.

Anyways, we've downloaded our SVG map. Let's move on to the next step.

Step 3. Open the SVG file in a text editor

I want to make sure we're clear on what we're editing. Like I said in Step 2, our SVG map is simply an XML file. We're not doing any photoshop or image-editing. We're editing an XML file. Open up the SVG file in a text editor so that we can see what we're dealing with.

You should see something like this:

We don't care so much about the beginning of the SVG file, other than the width and height variables, but we'll get back to that later.

Scroll down some more, and we'll get into the meat of the map:

Each path is a county. The long run of numbers are the coordinates for the county's boundary lines. We're not going to fuss with those numbers.

We only care about the beginning and very end of the path tag. We're going to change the style attribute, namely fill color. We want the darkness of fill to correspond to the unemployment rate in each given county.

We could change each one manually, but there are over 3,000 counties. That would take too long. Instead we'll use Beautiful Soup, an XML parsing Python library, to change colors accordingly.

Each path also has an id, which is actually something called a FIPS code. FIPS stands for Federal Information Processing Standard. Every county has a unique FIPS code, and it's how we are going to associate each path with our unemployment data.

Step 4. Create Python script

Open a blank file in the same directory as the SVG map and unemployment data. Save it as colorize_svg.py.

Step 5. Import necessary modules

Our script is going to do a few things. The first is read in our CSV file of unemployment data. So we'll import the csv module in Python. We're also going to use Beautiful Soup later, so let's import that too.

import csv
from BeautifulSoup import BeautifulSoup
 

Step 6. Read in unemployment data with Python

Now let's read in the data.

# Read in unemployment rates
unemployment = {}
reader = csv.reader(open('unemployment09.csv'), delimiter=",")
for row in reader:
    try:
        full_fips = row[1] + row[2]
        rate = float( row[8].strip() )
        unemployment[full_fips] = rate
    except:
        pass
 

We read in the data with csv.reader() and then iterate through each row in the CSV file. The FIPS code is split up in the CSV by state code (second column) and then county code (third column). We put the two together for the full FIPS county code, making a five digit number.

Rate is the ninth column. We convert it to a float since it's initially a string when we read it from the CSV.

The rate is then stored in the unemployment dictionary with the full_fips as key.

Cool. The data is in. Now let's load the SVG map, which remember, is an XML file.

Step 7. Load county map

Loading the map is straightforward. It's just one line of code.

# Load the SVG map
svg = open('counties.svg', 'r').read()
 

The entire string is stored in svg.

Step 8. Parse it with Beautiful Soup

Loading svg into Beautiful Soup is also straightforward.

# Load into Beautiful Soup
soup = BeautifulSoup(svg)
 

Step 9. Find all the counties in the SVG

Beautiful Soup has a nifty findAll() function that we can use to find all the counties in our SVG file.

# Find counties
paths = soup.findAll('path')
 

All paths are stored in the paths array.

Step 10. Decide what colors to use for map

There are plenty of color schemes to choose from, but if you don't want to think about it, give the ColorBrewer a whirl. It's a tool to help you decide your colors. For this particular map, I chose the PurpleRed scheme with six data classes.

In the bottom, left-hand corner, are our color codes. Select the hexadecimal option (HEX), and then create an array of those hexadecimal colors.

# Map colors
colors = ["#F1EEF6", "#D4B9DA", "#C994C7", "#DF65B0", "#DD1C77", "#980043"]
 

Step 11. Prepare style for paths

We're getting close to the climax. Like I said earlier, we're going to change the style attribute for each path in the SVG. We're just interested in fill color, but to make things easier we're going to replace the entire style instead of parsing to replace only the color.

# County style
'font-size:12px;fill-rule:nonzero;stroke:#FFFFFF;stroke-opacity:1;
stroke-width:0.1;stroke-miterlimit:4;stroke-dasharray:none;stroke-linecap:butt;
marker-start:none;stroke-linejoin:bevel;fill:'
 

Everything is the same as the previous style except we moved fill to the end and left the value blank. We're going to fill that in just a second. We also changed stroke to #FFFFFF to make county borders white. We didn't have to leave that value blank, because we want all borders to be white while fill depends on unemployment rate.

Step 12. Change color of counties

We're ready to change colors now! Loop through all the paths, find the unemployment rate from the unemployment dictionary, and then select color class accordingly. Here's the code:

# Color the counties based on unemployment rate
for p in paths:
 
    if p['id'] not in ["State_Lines", "separator"]:
        # pass
        try:
            rate = unemployment[p['id']]
        except:
            continue
 
        if rate > 10:
            color_class = 5
        elif rate > 8:
            color_class = 4
        elif rate > 6:
            color_class = 3
        elif rate > 4:
            color_class = 2
        elif rate > 2:
            color_class = 1
        else:
            color_class = 0
 
        color = colors[color_class]
        p['style'] = path_style + color
 

Notice the if statement. I don't want to change the style of state lines or the line that separates Hawaii and Alaska from the rest of the states.

I also hard-coded the conditions to decide the color class because I knew beforehand what the distribution is like. If however, you didn't know the distribution, you could use something like this: float(len(colors)-1) * float(rate - min_value) / float(max_value - min_value).

Step 13. Output modified map

Almost done. We just need to output the newly colored SVG map.

# Output map
print soup.prettify()
 

Save your Python script. For the sake of completeness, here's what your Python script should now look like:

### colorize_svg.py
 
import csv
from BeautifulSoup import BeautifulSoup
 
# Read in unemployment rates
unemployment = {}
min_value = 100; max_value = 0
reader = csv.reader(open('unemployment09.csv'), delimiter=",")
for row in reader:
    try:
        full_fips = row[1] + row[2]
        rate = float( row[8].strip() )
        unemployment[full_fips] = rate
    except:
        pass
 
# Load the SVG map
svg = open('counties.svg', 'r').read()
 
# Load into Beautiful Soup
soup = BeautifulSoup(svg)
 
# Find counties
paths = soup.findAll('path')
 
# Map colors
colors = ["#F1EEF6", "#D4B9DA", "#C994C7", "#DF65B0", "#DD1C77", "#980043"]
 
# County style
path_style = 'font-size:12px;fill-rule:nonzero;stroke:#FFFFFF;stroke-opacity:1;
stroke-width:0.1;stroke-miterlimit:4;stroke-dasharray:none;stroke-linecap:butt;
marker-start:none;stroke-linejoin:bevel;fill:'
 
# Color the counties based on unemployment rate
for p in paths:
 
    if p['id'] not in ["State_Lines", "separator"]:
        # pass
        try:
            rate = unemployment[p['id']]
        except:
            continue
 
        if rate > 10:
            color_class = 5
        elif rate > 8:
            color_class = 4
        elif rate > 6:
            color_class = 3
        elif rate > 4:
            color_class = 2
        elif rate > 2:
            color_class = 1
        else:
            color_class = 0
 
        color = colors[color_class]
        p['style'] = path_style + color
 
print soup.prettify()
 

Step 14. Run script and save new map

Now all we have to do is run our script and save the output.

We're done! Open your SVG in FIrefox or Safari, and you should see a nicely colored map similar to the one above. Oh wait, there's one teeny little thing. The state borders are still dark grey. We can make those white by editing the the SVG file manually.

We open our new SVG in a text editor, and change the stroke to #FFFFFF from #221e1f around line 15780. Do something similar on line 15785 for the separator. Okay. Now we're done.

Where to Go From Here

While this tutorial was focused on unemployment data, I tried to keep it general enough so that you could apply it to other datasets. All you need are data with FIPS codes, and it should be fairly straightforward to hack the above script.

You can also load the SVG into Adobe Illustrator or your favorite open source vector art software and edit the map from there, which is what I did for the final graphic.

So go ahead. Give it a try. Have fun.

Sunlight Labs, one of my new favorite data groups, has partnered with Google, RedHat, Mozilla, and several others to get the open source community involved in open government projects. They're calling it the Great American Hackathon and it's happening December 12-13.

For those unfamiliar with Sunlight, it's an organization that promotes open government data and transparency, and they fund technology projects (mainly online apps) that move this idea forward.

Organize an Event

If you're a developer or designer and want to help out, organize an event in your local area for December 12-13, and get as many people involved as you can. The more we make use of open government data, the more people that will see its usefulness, and the more people that care, the more the government will put into data. Get involved now.