From the Hands of David Yanofsky

quartzthings: We’ve just open-sourced Chartbuilder, the tool...

Tue, 30 Jul 2013 15:14:44 -0400

We’ve just open-sourced Chartbuilder, the tool that all reporters use at Quartz to quickly make simple charts at graphics-desk quality. Read more about how Chartbuilder came to be and how we use it in David Yanofsky’s piece for the Nieman Journalism Lab.

theannotationlayer: A couple of hours ago, I was telling a...

Fri, 29 Mar 2013 10:06:13 -0400

theannotationlayer:

A couple of hours ago, I was telling a colleague of mine how good this Periscopic graphic about gun deaths is over beer. If you haven’t seen it, you should check it out right now. The way that the dots (people) just drop off of their potential lifespans, and how, once the animation gets up to full speed, the whole thing looks like a machine gun firing…it’s super affecting.

But I’m starting to question the editorial judgement a little bit. I took another look at the graphic tonight after finding it to share the link with my colleague. I hadn’t actually realized that you can click on any one of those lines—which, of course, represent real individuals—and be taken to the news story about the corresponding person’s death.

After filtering out all but the deaths in the past seven days, I found and clicked on one that had taken place in my own borough of Brooklyn. Apparently, the victim had stabbed somebody, and then lunged with his knife at the cops who arrived on the scene. The cops ended up shooting and killing him.

I’m not sure that including gun deaths like this one in the graphic was a sound decision. Clearly the graphic was intended to inform the debate about gun regulation in the US. It was published when Sandy Hook was very fresh in everyone’s mind and Wayne LaPierre was on TV almost every day.

So, in addition to tacitly arguing for tighter gun control, is it also arguing that police officers shouldn’t have guns? And, is it really fair to say that someone who gets shot after threatening a group of cops with a knife has had his life stolen from him? He played some role in his demise, no?

Obviously, I have no idea what actually happened that night. The cops could have been trigger-happy or bigoted or just a bunch of dumbasses. Maybe they did fire without cause and maybe they did steal a life. I’m not sure.

But the point is that Periscopic isn’t either. They made the decision to include all gun deaths and to declare the consequent lost years of the victim “stolen” regardless of who fired the gun and whether or not it was self-defense.

And I understand—there are a lot of gun deaths in this country, unfortunately, and going through every individual death probably isn’t all that feasible for the Periscopic team.

But if you’re going to take on a project this ambitious and important, I think that you should do your best not to be misleading. A simple way to do that would be to not include cases where a cop was the shooter. Surely, police officers have caused a slew of unnecessary gun deaths. But save that injustice for a different graphic.

A final fiscal cliff wall story

Sat, 19 Jan 2013 14:27:35 -0500

A friend of mine was at a party last night. The hosts pull her aside and say,

“ Lauren, we need to talk. ”

They take her into the bathroom.

"For the last month we’ve looked out this window in our shower and saw this big ‘No’ taped on that wall over there…we couldn’t figure out what it was or why it would be there. We were obsessed. We would try to figure it out all the time. But now it’s gone.

“ The other day we were on Facebook and we saw that you liked a picture of the wall!

“WHAT IS THIS WALL?”

http://www.hastheusgoneoffthefiscalcliff.com/

How we built hastheusgoneoffthefiscalcliff.com

Thu, 10 Jan 2013 17:48:00 -0500

quartzthings:

Today we put hastheusgoneoffthefiscalcliff.com to sleep after a month of service, so we wanted to explain how it came to be.

How did the site work?

We had a DSLR plugged into a AC power supply, on a tripod, hooked up to a Mac Mini with a USB cable:

The Mac Mini ran a bash script every 5 minutes through the crontab

the script triggered a camera capture through the USB cable and downloaded the image

the script created two smaller resized copies of the image (one for the site one for social media use)

the script uploaded those images to our web server, replacing the previous captures

the script put a timestamp in the full sized image’s filename and moved it to an archive on the Mac Mini (for posterity)

The webpage was hard coded to the location of the image file and had some javascript that would update the image every 2.5 minutes (faster than images are taken to reduce the lag between what one user may see and what is actually in the office). That script used jQuery and looked like this:
setInterval(function() {
    $("#camimg").attr(
        "src",
        "http://hastheusgoneoffthefiscalcliff.com/imagecapture_1000.jpg?timestamp=" + (new Date()).getTime()
    )
},1000*60*2.5);
This redefined the path to the image every 2.5 minutes; appended with the date stamp as a parameter to make sure we dont get a cached version of the image.

The clickable areas were defined in a Google spreadsheet that was loaded in every time the page loaded and on each subsequent image replacement. We updated this document by hand every time we changed the wall.

How did the wall work?

The hard way: Every morning we got up, printed out some headlines, tweets, quotes and pictures, tiled them together and taped them to the wall.

Why??!

The idea for the single serving site from the beginning was Zach’s. Sometime in October he noticed that that the hastheusgoneoffthefiscalcliff.com domain was available to register, and he brought it up as something maybe worth pursuing. The first idea he sketched out was this:

A man who slowly inches towards the edge of a cliff paired with links to stories around the web about the topic and an explanation of what the fiscal cliff was. That conversation devolved into the merits of different depictions of cliffs:

After seeing Brian Rea’s coverage of the US Presidential Election Night on Instagram and remembering the website for Sagmeister and Walsh, I thought about making a webcam of a wall of stuff. I made this as a proof of concept:

Everyone got on board, and this is what resulted over the next month:

- David Yanofsky

View the code on github

Development time: 2 days

GIFing the 1040 and other notes on hacking the IRS website

Mon, 17 Dec 2012 10:30:26 -0500

quartzthings:

I published a story on Thursday about the complexity of the US tax code over time and used the length of the IRS Form 1040 over time as a proxy. It lead to responses like this one on Facebook:

Imagine the human being who took the time to make this. We must deeply honor the focus of that person.

Apparently it seemed like a crazy task to filter through almost 100 years of documents and tabulate information about them. Let’s asume they thought I was doing this by hand.

I wasn’t. Not even the GIF. Heres how:

Making the GIF

There’s a command line tool called ImageMagick that will both turn a PDF into a series of images and then turn that series of images into a GIF. These are the two commands using imagemagick I used to accomplish this:
$ for infile in *.pdf; do convert -density 400 -resize 400 -trim -extent 500x700 -gravity north $infile jpeg/$infile.jpg;  done
$ convert -delay 25 -loop 0 jpeg/f1040__*-0.jpg animated1040.gif
The first line tells ImageMagick to look at every PDF in the current directory, convert it to a 400px-wide PNG using a resolution of 300ppi for vector data, extend the edges of the image to 500px by 700px (anchoring the image to the top center of the new bounds), save it in the folder named jpeg. The second line tells ImageMagick to merge every .jpg file in the jpeg folder (i.e., every file I just created) with a file name ending in “-0.jpg” (this is the first page of the former PDF) into a GIF called “animated1040.gif” that flips through each image at 25 hundredths of a second and loops continuously.

After cleaning and optimizing it in Photoshop I had this.

Finding the files

All of this was dependent on having all of these 1040s. When I started looking for them, I was hoping some think tank or library would have an archive of the documents.

I decided to start simple. The current form is easy to find. A web search for “1040” revealed the IRS served PDF as the top result. Now what about the old forms? A web search for “2010 Form 1040” also returned a PDF on the IRS website but it had a slightly different URL: www.irs.gov/pub/irs-prior/f1040—2010.pdf. “irs-prior” — I like the look of that — “f1040—2010.pdf” Could all of the filenames be systematized?

Yes! A couple minutes of URL manipulation in my browser allowed me to find that there were files at this URL dating back to 1913 (though there were no forms for 1914 and 1915, since those years used the same 1913 form).

Downloading the docs

The next step was to download all of the files. Should I change the year in each URL and save as from my browser? TERRIBLE IDEA. I opened up my command line used the interactive prompt of python to download all the files super quick. It went something like this:
$ python
>>> import urllib
>>> years = range(1916,2012)
>>> for y in years:
. . .    urllib..urlretrieve("http://www.irs.gov/pub/irs-prior/f1040--%s.pdf" % (y,), "f1040--%s.pdf" % (y,))
. . .  
Here’s what that means:

start python

load the library I need to download files

create a list of years that I want to download: start in 1916 end in 2011 (one year before 2012) call it “years”

cycle through every year in that list calling the current year “y”

download the file using the url and naming system I figured out before, save the file using the same system

Two minutes later there were 97 PDFs in my folder for this project. I opened up the 1913 form in my browser and downloaded it. BOOM. Every 1040 ever.

Counting pixels

So I now we had all these files and we had to quantify exactly how much more complex they got over time. My first idea was to use the amount of ink used on each document as a proxy for complexity. I wanted to count the number of black pixels in each document. I used ImageMagick to convert all the PDFs to images and could start counting pixels.

Using a Python library called PIL, I opened up each file with Python, converted it to grayscale and counted the number of black pixels, calculated the ratio of black-to-total pixels, associated that JPEG with the appropriate year and save that information as a JSON blob and CSV spreadsheet. I’ll save you from that code here, but you can see it here on github.

Using the CSV I got out of that, I made this chart showing the amount of “ink” on the form over time:

It was antithetical to what we knew was true. If the amount of printing was a proxy to complexity, this chart would show that the tax code is less complex than the first years of the system.

Were the older documents just bigger? Use larger type? I charted the same information but as a ratio of amount of black per page. Same story. Then I realized that the older documents have more instructions on them! (More recent 1040s include instructions in a separate appendix.) What if we just looked at the tabulation page. No luck. Apparently today’s documents are more ink-efficient than those of yesteryear.

Counting lines

I crafted a new strategy: count the number of line items on the form. (Our methodology for what we counted is recounted in the piece.) The slow way to do this would be to double-click each file in a document viewer, count how many lines were on each page and input that into a spreadsheet, hoping I dont miss anything or make a typo. (It was beer o’clock in the office.)

The fast way is to write more code. I created another Python script that would open up each page of every document individually and prompt me to enter how many lines were on that page and whether I should overwrite the current number of lines I recorded or add these to the number of lines already recorded. Once complete, the script saved a spreadsheet of the recorded information.

I made this chart.

Thinking about the transfer of instructions from the form to a separate document, I decided to take a look at the instructions booklet and see how those have changed over time. I used the same exact scripts as above to download all of the instruction files by changing the URL slightly to “i1040” from “f1040.” (This naming convention was also revealed by a web search for “1992 form 1040 instructions.”)

Counting pages

The recent documents were long: 2011 is nearly 200 pages. I used some more code to count the number of pages, and it looked like this:
$ python
>>> from pyPdf import PdfFileReader
>>> years = range(1939,2012)
>>> for y in years:
. . .    print y, PdfFileReader(file("i1040--%s.pdf"%(y,),"rb")).getNumPages()
. . .  
I copied the data from the output (I didn’t save this to into a file for speed’s sake), pasted it into Excel, and made this chart:

Fifteen years ago, tax instructions were half the size! More striking, the booklets from the ’80s have smaller pages, but were still significantly shorter than today’s.

So now I had a GIF, three charts, and a whole bunch data. All that was left was words.

Read them all here: Line for line, US income taxes are more complex than ever

-David Yanofsky

View the code on github

Development Time: 1 day

My latest interactive is a comparison tool for the Center for...

Mon, 15 Oct 2012 18:10:00 -0400

My latest interactive is a comparison tool for the Center for Global Development’s Commitment to Development Index, a barometer of developed countries dedication to supporting poorer nations enhance their standing.

The piece is responsive and accompanied by a bunch of words by Tim Fernholz.

It is running the visualization library d3 making this my second published work to leverage it. (The first was a dashboard for the release of the jobs report that I will write about soon) This would have not been possible without Scott Murray’s tutorial.

Inspiration also came from the recently launched State-by-State interactive from the Bloomberg Visual Data group, seeing the HTML and CSS markup in their drop downs was very instructive.

That feeling when you weren’t expecting a byline.

Thu, 26 Jul 2012 18:46:38 -0400

That feeling when you weren’t expecting a byline.

"Unfortunately, we have not selected your project to move forward"

Sun, 22 Jul 2012 20:53:00 -0400

“"Unfortunately, we have not selected your project to move forward"”

- Now is when I try to come up with a way to do TableTent with no money. I’ve been thinking about using ScraperWiki.

Knight News Challenge Round 2: Table Tent: A Destination and API about the People Testifying to Congress

Tue, 12 Jun 2012 21:01:31 -0400

Knight News Challenge Round 2: Table Tent: A Destination and API about the People Testifying to Congress:

I submitted this the other day and it has a great response so far. Head over to the original post and heart it up!

newschallenge:

1. What do you propose to do?

We wish to build an aggregator, exploratory tool, and API revealing in-depth information about who is testifying in front Congress, how often they’re doing it, what they’re saying, and who they work for.

2. How will your project make data more useful?

By…

Pyglet Image Upside-down Example Fix

Tue, 01 May 2012 17:52:00 -0400

I was doing some video processing in python today and I couldn’t find any good answers on how to save rightside-up images using pyglet. Apparently pyglet saves them upside-down by default. Here’s my solution:

#load the video
video = pyglet.media.load(“myVideo.wmv”)

#get the first frame
frame = video.get_next_video_frame()

#get the image data of the first frame
imageData = frame.get_image_data()

#get the pixels of the frame but invert the pitch
pixels = imageData.get_data(imageData.format,imageData.pitch *-1)

#set inverted pixels to the image
dataimageData.set_data(imageData.format,imageData.pitch,pixels)

#save the image
imageData.save(“myVideo_capture.png”)

What are the best undergrad B-Schools in the U.S.? My latest...

Thu, 22 Mar 2012 18:27:15 -0400

What are the best undergrad B-Schools in the U.S.?

My latest work is an exploration tool of Bloomberg Businessweek’s 2012 Undergraduate Business Schools Ranking.

Some things you can do with it include…
☞ see the top ranked schools
☞ the schools rising and falling the most in the ranking
☞ search for a school
☞ find out which students say they’re workload is too heavy
☞ discover which students spend the most time preparing for class
☞ AND MORE

http://buswk.co/GGmChP

The final in the series of energy interactive I made last week...

Thu, 22 Mar 2012 18:19:00 -0400

The final in the series of energy interactive I made last week was an exploration of where new wind and solar energy installations were going around the world, how large the installation have been, and what the unit cost of the new equipment was.

It uses data from Bloomberg New Energy Finance from 1990 with their projection through 2030.

The Great Renewable Energy Race

Deja Vu? Kind of.Yes those are the same types of charts, in the...

Wed, 14 Mar 2012 21:47:48 -0400

Deja Vu? Kind of.

Yes those are the same types of charts, in the same order, on the same type of page, with similar headlines, and subject-matter, but this is about wind–TOTALLY DIFFERENT

Wind Innovations Drive Down Costs, Stock Prices

I just published the first in a series of energy related...

Tue, 13 Mar 2012 21:30:00 -0400

I just published the first in a series of energy related interactives. This one looks at solar-generation and when the costs associated with it come into line with the rest of the energy system, i.e. when it becomes profitable without subsidy. Some countries are already there, some are a still years away.

Read more about that and explore the interactive charts here http://bit.ly/zIvQX2

My Latest work is a map of heritages and ethnicities in the...

Wed, 07 Mar 2012 19:36:55 -0500

My Latest work is a map of heritages and ethnicities in the U.S.

It allows for the exploration of the results of the Census’s American Community Survey to see the distribution of any of the largest 20 ancestral groups across the country, an ancestry’s concentration county by county, or a heritage’s relative prominence to another around the nation.

Test your regional stereotypes here http://bit.ly/yWwGoB

If you’re wondering how it works, it’s loosely based on Jon Bruner’s method using Raphael.js.

For the latest quarterly financial reports filed by candidates...

Thu, 02 Feb 2012 19:39:05 -0500

For the latest quarterly financial reports filed by candidates we updated our visualization module to follow the money that is making the most difference in the campaign so far: Super-PACs.

See how much each candidate has raised, spent, and how much cash they have on hand in the interactive graphic here: http://bloom.bg/xz9WbV

Catching Up

Sun, 29 Jan 2012 16:41:13 -0500

I haven’t posted anything I’ve made in almost a year. Let’s recap.

I made two Pieces about natural disasters.

The Tōhoku earthquake and tsunami hit and the resulting crisis at the Fukushima reactor got everyone talking about nuclear power. Where are the plants in the U.S.? How old are they? Who runs them? How long do their permits last?

http://www.businessweek.com/magazine/nuclear-power/

A tornado hit Joplin, Missouri and we created a page to show the cost of extreme weather over the past 10 years.

http://www.businessweek.com/magazine/StormMap

I made ten pieces about Education topics.

What’s the return on investment after attending certain colleges?

http://www.businessweek.com/magazine/collegeroi/

Whats the return on investment after attending certain business schools?

http://www.businessweek.com/magazine/bschoolsroi/

Which companies do university students say they want to work for? What types of students want to work at which companies?

http://www.businessweek.com/magazine/undergraduate-employer-preferences/

How do countries’ education systems compare around the world?

http://www.businessweek.com/top-news/the-state-of-education-10132011-gfx.html

How well do students around the world score on standardized tests in math science and reading?

http://www.businessweek.com/top-news/head-of-the-class-10132011-gfx.html

China is giving a lot of money to higher education programs around the U.S. Who’s getting what, and what does this mean for academic independence on issues like human rights and Tibet?

http://www.bloomberg.com/news/2011-11-01/china-says-no-talking-tibet-as-confucius-funds-u-s-universities.html

The results of comprehensive surveys of business school students. How do they rate their own programs?

http://www.businessweek.com/business-schools/parttime-mba-programs-drilling-down-11102011-gfx.html

http://www.businessweek.com/business-schools/what-emba-students-really-think-11102011-gfx.html

http://www.businessweek.com/business-schools/the-complete-parttime-mba-ranking-11102011-gfx.html

http://www.businessweek.com/business-schools/the-complete-executive-mba-ranking-11102011-gfx.html

I made five pieces about energy

Where does the U.S. get its energy from? Where does it get used? How efficiently is it utilized?

http://www.bloomberg.com/data-visualization/americas-energy-where-it-comes-from-where-it-goes/

What effect would the increase or decrease in the use of different energy types have on the U.S. power mix?

http://www.businessweek.com/magazine/clean-energy-calculator-07272011-gfx.html

How much is each state using of the different types of energy? How reliant is each state on different types of energy?

http://www.businessweek.com/magazine/powering-america-07272011-gfx.html

With a new natural gas pipeline in the north sea, will politics have less control on the european gas market?

http://www.bloomberg.com/news/2011-10-28/russian-gas-takes-route-in-the-baltic-sea.html

The raw material for making solar panels has plummeted in price.

http://www.bloomberg.com/news/2011-11-08/solar-shakeout-could-soon-reach-china.html

I made 2 pieces about sports

Who has spent their money most wisely in professional sports? This is Meta-SABRmetrics.

http://www.businessweek.com/magazine/the-smartest-spenders-in-sports-10202011-gfx.html

Where does the money in college football come from? Where does it go?

http://www.bloomberg.com/data-visualization/texas-tops-college-football-with-75-operating-profit

I made seven pieces relating to Markets, Finance, Banks, or Trade

The Federal Reserve lent $1.2 Trillion to banks during the financial crisis and kept it secret. Bloomberg filed a FOIA request and sued them for the information about who got what and when. This is a comprehensive tool to explore the information that the U.S. Supreme Court instructed the Fed to hand over.

http://www.bloomberg.com/data-visualization/federal-reserve-emergency-lending

While much of the euro zone encounters inflation, Germany does not.

http://www.businessweek.com/global/euro-crisis/archives/2011/11/can_ecbs_draghi_solve_eu_inflation_disparity.html

Hear about that hot IPO? Buy some on the first day of trading? That might not have been the best idea.

http://www.businessweek.com/finance/occupy-wall-street/archives/2011/11/groupon_beware_of_20_hot_ipos_20_tanked_later.html

An overview of the economic indicators and credit ratings of euro-zone nations.

http://www.bloomberg.com/data-visualization/euro-zone-debt-risk/

Which countries have the best transportation infrastructure? How do they compare?

http://www.businessweek.com/technology/cars-trains-and-planes-12012011-gfx.html#

A follow up piece on the data released from the Fed about the emergency lending during the financial crisis and how much banks profited from the below market interest rates.

http://www.bloomberg.com/news/2011-11-28/secret-fed-loans-undisclosed-to-congress-gave-banks-13-billion-in-income.html

Even though it is certified as “fair trade” some cotton in Burkina Faso is picked by unpaid children. This is the story of one of them.

http://www.bloomberg.com/news-media/clarisse-kambire-victorias-secret-child-labor-cotton-picker/

What do the Global Business Elite think about markets and economies around the world?

http://www.bloomberg.com/data-visualization/bloomberg-global-poll/

I made seven or eight about Politics and the Election…But most were temporary and had no permanent location.

Where are defense dollars spent in Virginia, what is at risk of “across-the-board” cuts?

http://www.bloomberg.com/news/2011-11-17/military-cuts-threaten-defense-dependent-states.html

The difference between New York news and Boston news today

Thu, 23 Jun 2011 09:55:56 -0400

The difference between New York news and Boston news today

Oh! Also, Raj guilty on all counts? Time to update the graphic.

Sun, 15 May 2011 03:21:32 -0400

Oh! Also, Raj guilty on all counts? Time to update the graphic.

I’m not saying that this was the most straight forward way...

Sat, 14 May 2011 21:45:28 -0400

I’m not saying that this was the most straight forward way to display this information. I am saying I made it.

http://www.businessweek.com/magazine/undergraduate-employer-preferences/