Data Diversions

Join Me on Medium

Wed, 01 Jun 2016 09:26:05 -0400

Over the past few years, I’ve had a lot of fun writing on dfkoz.tumblr.com. Beginning in 2012, however, I started a new adventure with FirstMark, which left less and less time to write here.

While I intend to occasionally share a data-driven post here from time to time, I’ll be focusing a lot more of my writing time on sharing lessons from my work at FirstMark.

If you want to come along for the ride, you can join me at https://medium.com/@dfkoz/.

A 15% Tip Button Could Cost NYC Cabbies $60 Million

Thu, 01 Jan 2015 14:30:07 -0500

This post is a response to a question posed by Chris Whong on Twitter: “I am curious to know what the $ implications of adding a 15% button would be.” Chris is the source of the raw data for my previous post, Analysis of NYC Taxi Tip Data: 44% of Passengers Hit the 20% Button.

By my math, a 15% tip button reduces total taxi tips by between $35MM and $60MM. This works out to a range of $2,600 to $4,600 for each of the 13,437 medallions active in 2014 and $700 to $1,200 for each of the roughly 50,000 drivers in the city.

I calculated the impact four different ways. On the low end, I assumed that only those who default to the cheapest button (currently the 20% button) migrate to the 15% button. This reduces total tips by $26MM, but seems like unlikely behavior in practice.

The second approach reduces all of the exact 20%, 25%, or 30% tips by 5%, with the assumption that all “non-standard” tipping behavior remains consistent. This leads to a $35MM reduction in tips.

The third calculation lowers all credit card tips by 5% across the board, for an aggregate impact of $60MM.

Finally, it is possible that the introduction of a 15% button would completely change tipping norms. Cash tips go almost completely unreported, but I assume in the base case that cash tipping behavior matches credit card tipping behavior. If true, the total impact impact of the introduction of a 15% button would be $112MM.

Analysis of NYC Taxi Tip Data: 44% of Passengers Hit the 20% Button

Wed, 31 Dec 2014 11:37:00 -0500

Thanks to Ben Wellington, the man behind I Quant NY, I discovered that Chris Whong posted a year’s worth of NYC taxi fare data, which he obtained from the New York City Taxi and Limousine Commision by FOIA request. What follows is an analysis of 578,233 rides, or about 1/300th of the roughly 175 million yellow taxi rides taken in 2013.

Tipping Behavior

45% of taxi rides were paid for with cash. Of these 264,154 transactions, only 29 (0.00931%) included a tip. As a result, the following graphs depict only credit card transactions, totaling 311,623 rides.

The average tip was $2.53 with a standard deviation of $2.43. Excluding riders who did not tip at all, the average tip was $2.61 with a standard deviation of $2.43. As a fraction of the total fare, the average tip was 18.9% for all rides, and 19.5% for all rides that included a tip. 3.16% of riders did not tip at all. 1.75% of riders tip more than 30% and 0.21% tip more than 50%.

Unsurprisingly, there are some extreme data points in the raw data. In this sample of 1/300th of the records, there were about 20 negative fares and tips that (supposedly) went as high as 4067%.

As NYC residents know, the in-cab credit card interface offers three default tip levels: 20%, 25%, and 30%, once described as the “The $144,146,165 Button.” 44% of riders defaulted to the 20% button, 10% to the 25% button, and 4% to the 30% button.

Tipping Behavior by Time of Day

Riders apparently feel most generous shortly before lunch and through the end of the work day. That, or - more likely - employers are picking up tab, resulting in higher average tips during the day.

Here is how rides are concentrated across an average day, an average weekday (defined as 6 PM Sunday to 6 PM Friday), and an average weekend (6 PM Friday to 6 PM Sunday). One hour that stands out is 4-5 PM, which - as any resident will tell you - is when New York cab drivers change shifts, leading to fewer available cabs.

Tipping Behavior by Day of Week

There is also a relationship between day of week and tipping behavior: Monday (the slowest taxi day of the week) is the best day for tips, and Friday (the busiest day) is the worst.

Again, my theory is that this has more to do with the type of trip (professional vs. personal) than anything else. Both Sunday and Monday are lower-volume days, but the tipping behavior is different.

Tipping Behavior by Month

There also appears to be a relationship between time of year and tipping behavior. December is the most generous month for tips:

Distribution of Total Fares

The average taxi fare in 2013 was $14.80, with a standard deviation of $12.13. About 30% of all taxi fares are less than $10, and nearly 80% are less than $20. 3.79% of fares were greater than $50 and 0.15% were greater than $100. The most expensive trip recorded in my sample was a whopping $544, paid in cash. Here’s what the distribution of total fares, cash included, looks like:

Zoomed in a bit:

And the distribution of total fares:

That’s a wrap for me. But if you’re hungry for more taxi data visualization, you can check out Chris Whong’s NYC Taxis: A Day in the Life or I Quant NY’s analysis of Green Taxi tipping behavior by neighborhood (which is evidently the kickoff to a series of posts.)

Happy New Year to all of you! And remember to tip your cab driver an extra 1.25% tonight.

Generate $315MM of Economic Impact by Wishing People an “HT” Today

Wed, 26 Nov 2014 13:17:00 -0500

There are currently 319,151,000 people in the United States. As of October 31, 2014, 150,000,000 of them were employed in full-time positions. At this wonderful time of year, most or all of them are presumably wishing colleagues, friends, and family a “Happy Thanksgiving”, whether they’re doing it from their PCs or phones. What would happen, however, if we collectively agreed to wish each other a “Happy Turkey Day”?

If the average adult types at 40 WPM and average word length is 5 characters, then an adult produces 200 characters in a minute, or one character every 0.3 seconds. Assuming that an average employee produces $67.32 of GDP per hour worked, one extra character – repeated across, say, 25 holiday greetings – amounts to $0.14025 of GDP. With 150,000,000 workers, that one-character difference costs a whopping total of $21,037,500 of GDP.

So if we all wished each other a “Happy Turkey Day,” we would reclaim two characters and $42,075,000 of GDP. “Happy T Day,” which eliminates seven characters, represents a savings of $147,262,500. Taken to the extreme, we would all simply write “HT,” saving 15 characters and $315,562,500 of GDP.

I’ll do my part: wishing all of you an HT :)

The Best of TV: Analyzing IMDB Data on 90,000 TV Series

Sun, 15 Jun 2014 16:04:00 -0400

Founded in 1990, IMDB is a treasure trove of data with “…2,882,569 titles (includes episodes) and 5,909,020 personalities in its database.” Using a simple scraper written in Ruby, I was able to grab data on the >90,000 TV shows in the IMDB database. This is what I found.

There has been significant growth in the number of new television series created each year:

Here is the same chart from 2000 forward:

It is important to note that our collective record-keeping has improved since IMDB’s launch in 1990, and with the rise of internet usage generally (and IMDB specifically) over the past decade. IMDB’s Google Trends chart:

War and history series are the highest-rated genres on average. Among common genres, documentaries, animated series, and dramas are the highest-rated. Game shows and reality TV are rated significantly lower than other genres:

Certain genres have grown or faded significantly since 1950:

For example, westerns have almost completely disappeared:

The number of reality TV shows has had an astonishing rise starting in the late 90s, but has thankfully flattened out:

The prevalence of fantasy shows has been relatively steady over time, but has made a comeback in the past few years:

The overall quality of TV peaked in the 1960s, declined through 2004, and has since steadily improved (although I would take this chart with several grains of salt):

Here is the same chart from 1990 through today:

The top ten highest-rated television shows of all time are as follows:

1. Breaking Bad (2008): 9.6
2. Game of Thrones (2011): 9.5
3. Planet Earth (2006): 9.5
4. Cosmos: A SpaceTime Odyssey (2014): 9.5
5. The World at War (1973): 9.5
6. The Wire (2002): 9.4
7. True Detective (2014): 9.4
8. Sherlock (2010): 9.3
9. The Sopranos (1999): 9.3
10. Cosmos (1980): 9.3

The top TV series that have debuted since 2010 (note that I did *not* filter out shows with a small number of ratings):

1. Game of Thrones (2011): 9.5
2. Cosmos: A SpaceTime Odyssey (2014): 9.5
3. True Detective (2014): 9.4
4. Sherlock (2010): 9.3
5. Rick and Morty (2013): 9.3
6. Africa (2013): 9.3
7. Kardes Payi (2014): 9.3
8. Leyla and Mecnun (2011): 9.2
9. Fargo (2014): 9.2
10. 24: Live Another Day (2014): 9.2

Best of the 2000s:

1. Breaking Bad (2008): 9.6
2. Planet Earth (2006): 9.5
3. The Wire (2002): 9.4
4. The Life of Mammals (2002): 9.3
5. Al-Taghriba Al-Filistinia (2004): 9.2
6. Arrested Development (2003): 9.2
7. Firefly (2002): 9.2
8. Life in the Undergrowth (2005): 9.2
9. Life in Cold Blood (2008): 9.2
10. Avatar: The Last Airbender (2005): 9.1

Best of the 90s:

1. The Sopranos (1999): 9.3
2. The Private Life of Plants (1995): 9.2
3. The Trials of Life (1990): 9.2
4. The Life of Birds (1998): 9.1
5. Friends (1994): 9.0
6. Twin Peaks (1990): 9.0
7. Freaks and Geeks (1999): 9.0
8. Batman: The Animated Series (1992): 9.0
9. Cowboy Bebop (1998): 9.0
10. Ice TV (1996): 9.0

Best of the 80s:

1. Cosmos (1980): 9.3
2. The Living Planet (1984): 9.2
3. The Simpsons (1989): 9.0
4. Seinfeld (1989): 9.0
5. Dragon Ball Z (1989): 8.9
6. Only Fools and Horses…. (1981): 8.9
7. Blackadder Goes Forth (1989): 8.9
8. Fauji (1988): 8.9
9. Solunski patrdii (1985): 8.9
10. The Morning Program (1987): 8.9

Best of the 70s:

1. The World at War (1973): 9.5
2. Life on Earth (1979): 9.2
3. I, Claudius (1976): 9.1
4. La zulianita (1977): 9.0
5. Doña Bárbara (1975): 9.0
6. Fawlty Towers (1975): 8.9
7. Matador (1978): 8.9
8. El chavo del ocho (1972): 8.8
9. The Man and the City (1971): 8.8
10. The Julie Andrews Hour (1972): 8.8

Best of the 60s:

1. Monty Python’s Flying Circus (1969): 9.0
2. Fujimaru of the Wind (1964): 9.0
3. The Prisoner (1967): 8.8
4. Nu, pogodi! (1969): 8.8
5. Ferien in Lipizza (1966): 8.8
6. The Great Adventure (1963): 8.8
7. Alle Hunde lieben Theobald (1969): 8.8
8. Hafenpolizei (1963): 8.8
9. Landarzt Dr. Brock (1967): 8.8
10. Die Gäste des Felix Hechinger (1964): 8.8

Best of the 50s:

1. Twilight Zone (1959): 8.9
2. Walt Disney Presents: Annette (1958): 8.8
3. Corky and White Shadow (1956): 8.8
4. Mayor of the Town (1954): 8.8
5. The Spade Cooley Show (1958): 8.8
6. Alfred Hitchcock Presents (1955): 8.7
7. Mr. Squiggle and Friends (1959): 8.7
8. Studio 57 (1954): 8.7
9. J.P. Patches (1958): 8.7
10. The Honeymooners (1955): 8.6

Best of the 40s:

1. Mama (1949): 8.5
2. Mary Kay and Johnny (1947): 8.5
3. Kukla, Fran and Ollie (1947): 8.2
4. The Howdy Doody Show (1947): 8.1
5. Stump the Stars (1947): 8.1
6. Fireside Theatre (1949): 8.1
7. The Lone Ranger (1949): 8.0
8. The Goldbergs (1949): 8.0
9. Time for Beany (1949): 8.0
10. The Aldrich Family (1949): 8.0

Movie Scripts Ranked by Flesch-Kincaid Grade Level

Sat, 03 May 2014 12:50:00 -0400

Remember the Flesch Kincaid Grade Level? If you ever ran Microsoft Word's spell check in in the 90’s, you probably do:

The Flesch and Flesch-Kincaid tests are “readability tests designed to indicate comprehension difficulty when reading a passage of contemporary academic English.” The scores incorporate two primary factors: the average number of words per sentence, and the average number of syllables per word. A higher Flesch-Kincaid Grade Level is supposed to reflect more difficult text, with longer sentences and more multi-syllabic words.

The Ruby gem Lingua makes it quite easy to perform readability analysis against text. And with more than 950 scripts available on the Internet Movie Script Database, I had the raw ingredients necessary to run a fun and entirely inane analysis of the Flesch-Kincaid scores of movies. Here are the results.

Movies with the lowest Flesch-Kincaid grade level (simpler):

Glengarry Glen Gross: 0.11
Win Win: 0.13
Last Tango in Paris: 0.25
Silver Linings Playbook: 0.45 [Oscar winner, Best Adapted Screenplay]
Gran Torino: 0.54
Next Friday: 0.60
Hannibal: 0.64
Sling Blade: 0.65
Frozen: 0.66
Boogie Nights: 0.68 [Oscar nominee, Best Original Screenplay]

Movies with the highest Flesch-Kincaid grade level (more complex):

Fletch: 5.94
The Adventures of Buckaroo Banzai Across the Eighth Dimension: 5.48
The Ninth Gate: 5.07
L'Avventura (The Adventure): 4.74
Mulan: 4.31
Omega Man: 4.18
The Last Samurai: 4.17
Gattaca: 3.99
The Saint: 3.76
American Psycho: 3.76

If you’re interested in seeing the full list, click here. This list includes two additional scores: the Flesch Reading Ease score, which uses slightly different weights (with a lower score indicating more readable text), and the Gunning fog index, which incorporates average sentence length and the density of words with three or more syllables.

A few caveats: first, this list is far from complete. In fact, there were ~100 scripts on the IMSDb that have been taken down (e.g., most of the Batman and all of the Harry Potter movies).

Second, the analysis is unfairly favorable to scripts with detailed stage directions, which tend to be more complex than dialogue. Here is the first passage from the “most sophisticated” movie, Fletch:

Seagulls squawk, and the waves pound, but we’re not talking about Malibu Colony, here. This is a fairly rundown beach area, catering to lower-echelon surfers, vagrants, and strung out druggies of all ages, several of whom stand or sit on their haunches by a dilapidated old hamburger stand. Over the stand is a faded sign: “FAT SAM’S HAMBURGERS”.

In any case, enjoy. In the immortal words of John McClane:

Yippe-ki-yea

The Most Linked-To Wikipedia Articles on Stack Overflow

Sat, 26 Apr 2014 12:57:00 -0400

I discovered the Stack Overflow Data Explorer this morning, which allows you to query the entire catalogue of Stack Overflow posts. After playing with it, I decided to figure out which Wikipedia articles have been linked to most often by Stack users over the past two years.

Over a two-year period, there were 10,831,467 total Stack Overflow posts. 93,290 (0.86%) posts contained the text ‘en.wikipedia.org’, representing 115,591 links. These links pointed to about 25,000 unique articles. 14,600 articles (58.25%) appeared just once. 91.76% of the articles appeared fewer than ten times, accounting for 38.31% of all links.

The most common link, SQL Injection, was cited over 1,000 times. Another 42* links appeared more than 200 times, and 113 total links appeared more than 100 times. The top 10 most linked-to articles are below. You can see the full list here.

en.wikipedia.org/wiki/SQL_Injection (1001 links)
en.wikipedia.org/wiki/Same_origin_policy (909)
en.wikipedia.org/wiki/JSONP (501)
en.wikipedia.org/wiki/Cross-origin_resource_sharing (492)
en.wikipedia.org/wiki/Singleton_pattern (468)
en.wikipedia.org/wiki/Undefined_behavior (420)
en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller (406)
en.wikipedia.org/wiki/Single_responsibility_principle (402)
en.wikipedia.org/wiki/Don%27t_repeat_yourself (398)
en.wikipedia.org/wiki/Database_normalization (358)

[1] There were about 5 references to the answer to the ultimate question of Life, the Universe, and Everything appeared.

When Do Students Apply for Jobs?

Sun, 26 Jan 2014 13:22:00 -0500

Over at FirstMark, we’re running an internship program called FirstMark Elite that connects the best undergraduate and graduate computer science students with summer internships at some of the most exciting startups in the world.

We have received an incredible response from colleges all over the country. I know because I receive an email each and every time a student applies for the program. Over the past few months, I have noticed a distinct pattern in the days and time that we receive applications. So, as part of the “data exhaust” of running FirstMark Elite, here is an overview of “when students apply for jobs”. Presumably, these are also graphs of “when students aren’t eating or having fun.”

I created an interactive visualization of VC-backed Bitcoin...

Sun, 24 Nov 2013 22:04:00 -0500

I created an interactive visualization of VC-backed Bitcoin startups using Highcharts and Crunchbase data. See it live here: http://dfkoz.com/bitcoin/.

Real-Time Thermometer with SmartThings and PHP

Tue, 12 Nov 2013 20:35:00 -0500

Here is a real-time FirstMark temperature monitor, created with a SmartThings starter kit, Highcharts.js, a little bit of PHP, and MySQL. (Doesn’t look great on mobile yet.)

Now we’ll always know whether it's LDL or RH temperature! This is how crazy simple the SmartThings SmartApp code is:

https://gist.github.com/dfkoz/7442030

The Most Popular Classic Rock Songs

Mon, 14 Oct 2013 14:49:00 -0400

Using the yes.com API, I was able to load playlists for over 1,000 different radio stations. There is a lot of data to analyze here, but one of the first questions I sought to answer was “What are the most popular songs and artists on classic rock radio?”

Originally, I categorized any station that played Aerosmith and Led Zeppelin at least once per day as a classic rock station. An astute friend pointed out that I was “sampling on the dependent variable.” To address this, I filtered for stations that played (i) at least one song by Pink Floyd, Led Zeppelin, The Beatles, Fleetwood Mac, the Rolling Stones, and Aerosmith, and (ii) played at least one song by all of those artists. The source data do not have reliable genre tags, so this is the most appropriate test I could think of; I am open to alternative suggestions.

Of the 1,000 radio stations, 91 are considered ‘classic rock.' Across 1 week, 111,000 songs, 5,421 unique songs, and 1,448 unique artists, here are the twenty most popular songs:

Sweet Home Alabama, Lynyrd Skynyrd (0.55 plays/day)
Sweet Emotion, Aerosmith (0.53 plays/day)
Carry On Wayward Son, Kansas (0.51 plays/day)
More Than A Feeling, Boston (0.49 plays/day)
Dream On, Aerosmith (0.47 plays/day)
Another Brick In The Wall (Pt II), Pink Floyd (0.47 plays/day)
The Joker, Steve Miller (0.46 plays/day)
Don’t Stop Believin’, Journey (0.45 plays/day)
We Will Rock You, Queen (0.45 plays/day)
Another One Bites The Dust, Queen (0.45 plays/day)
You Shook Me All Night Long, AC/DC (0.45 plays/day)
Renegade, Styx (0.45 plays/day)
Wanted Dead Or Alive, Bon Jovi (0.44 plays/day)
Magic Man, Heart (0.44 plays/day)
I Want You To Want Me (Live), Cheap Trick (0.44 plays/day)
We Are The Champions, Queen (0.43 plays/day)
Barracuda, Heart (0.43 plays/day)
Centerfold, J. Geils Band (0.43 plays/day)
I Love Rock 'N Roll, Joan Jett (0.43 plays/day)
Free Fallin’, Tom Petty (0.42 plays/day)

These are the twenty most popular classic rock artists:

Led Zeppelin (5.30 plays/day)
Van Halen (4.36 plays/day)
Rolling Stones (4.18 plays/day)
Pink Floyd (3.92 plays/day)
Aerosmith (3.63 plays/day)
Queen (3.59 plays/day)
Tom Petty (3.55 plays/day)
AC/DC (3.14 plays/day)
Journey (3.09 plays/day)
Eagles (2.98 plays/day)
Boston (2.82 plays/day)
ZZ Top (2.74 plays/day)
Steve Miller (2.60 plays/day)
Foreigner (2.59 plays/day)
Def Leppard (2.57 plays/day)
Who (2.42 plays/day)
Beatles (2.42 plays/day)
U2 (2.26 plays/day)
Police (2.16 plays/day)
Lynyrd Skynyrd (2.15 plays/day)

Send a Random Vacation Photo Each Day with Honeymailer

Tue, 25 Jun 2013 14:21:00 -0400

I was backing up some photos the other day and spent 20 wonderful minutes flipping through pictures from our honeymoon. I thought it was a shame that we rarely took time to enjoy old photos. We snapped 1,700 on a month-long honeymoon in Spain, and have thousands more from other trips. Then, inspiration struck: why not get a regular email with a random photo from our personal archives? Thus was born Honeymailer, a completely passive and spontaneous way to enjoy some old memories.

Now, my wife and I get gems once a day, like this photo from Ainsa, Spain:

Here is the code for Honeymailer. It runs as a Google Apps Script. If you have any trouble setting this up, don’t hesitate to leave a comment.

https://gist.github.com/dfkoz/5860786

According to Twitter, Chris Bosh looks like...

Thu, 20 Jun 2013 14:20:00 -0400

Based on a sample of over 2,000 tweets with the text “Chris Bosh looks like”:

Two days in the Citi Bike program. Red areas indicate that more...

Fri, 07 Jun 2013 11:43:40 -0400

Two days in the Citi Bike program. Red areas indicate that more bikes are available relative to the green areas. Full heatmap at dfkoz.com/citibike/.

Heatmap of Citi Bike density over time

Sat, 01 Jun 2013 13:38:07 -0400

Heatmap of Citi Bike density over time

The Rock and Roll Hall of Fame: From Acid Rock to Zydeco

Tue, 28 May 2013 21:22:11 -0400

The Rock and Roll Hall of Fame completed its 26^th induction ceremony about a week ago, honoring Rush, Heart, Randy Newman, Donna Summer, Albert King, and Public Enemy. This year’s crop of inductees struck me as more appropriate for a “Popular Music Hall of Fame” than one dedicated specifically to rock. In fact, if I were a betting man, I would put money down on a name change in the next 10 years.

The first hip hop artist to be inducted into the Hall was Grandmaster Flash and The Furious Five in 2007. Since then, Run D.M.C. (2009) and the Beastie Boys (2012) have also been inducted. Artists are eligible for election beginning 25 years after their first record, so a large cohort of popular hip hop artists will become eligible over the next decade; the Grammy for Best Rap Album was first awarded (to Naughty by Nature) in 1996.

There are also 11 artists that have been inducted into both the Country Music Hall of Fame and the Rock and Roll Hall of Fame, including Elvis and Johnny Cash. Of course, there are at least three reasons this is not surprising: genres are fluid, artists’ styles evolve, and—perhaps most importantly—rock was born, in part, from country music.

Even Miles Davis, who “flirted with rock from the late 1960’s on” was inducted as a performer, which the New York Times described as “provocative for a second, and then a little meaningless.” In any case, this made me wonder about the breadth of genres represented across the Hall’s 279 inductees, 179 of whom are performers.

I wrote a short script to crawl Wikipedia pages for each inducted performer and tabulate the genres mentioned. Neil Young’s page, for example, lists “rock” in the lead left position followed by “folk rock, country rock, experimental rock, hard rock”. This surfaced 144 unique genres, 50% of which are attached to only one artist, including baroque pop (Bee Gees), bubblegum pop (The Jackson 5), drone (The Velvet Underground), glam metal (Alice Cooper), jangle pop (R.E.M.), krautrock (David Bowie), spoken word (Leonard Cohen), and zydeco (Dr. John).

There is nothing extraordinarily surprising about the overall distribution of genres. The Rock and Roll Hall of Fame has, for the most part, been true to its name – although we will see how much longer that remains true.

“Lead left” genre, all genres with at least 5 inductees

% of inductees tagged with most common genres

Peak social? Numbers show new social startups are declining

Tue, 26 Mar 2013 17:58:00 -0400

The following graph shows the total percentage of startups listed in Crunchbase that include the word “social” in either the company’s description or tags, by year founded.

In response to a comment regarding percentages versus count of startups, I cooked up the following graph. This shows the total number of “social” startups founded in a given year (blue) and the total number of startups founded according (blue + gray).

This chart shows an increasing number of startups founded between 2005 and 2011, with a 33% decline from 2011 to 2012. I am not sure whether entrepreneurial activity has increased or decreased over the past two years, but I am fairly certain that the startup activity did not drop by a third. Instead, my guess is that either (i) data quality is higher for startups that have been around longer, or (ii) that Crunchbase is becoming a less popular place to list one’s startup (see AngelList). This is why the original analysis looked at the data on a percentage basis.

sxBlock: A Chrome Extension That Hides All SXSW-Related Tweets

Sat, 09 Mar 2013 18:50:00 -0500

I have seen a lot of demand on Twitter for a way to filter out SXSW-related tweets from one’s timeline. Well, now you can do that with sxBlock. You can download it here.

Also, if you would like to help refine the extension, here is the source code.

CPM and Price of a Super Bowl Ad Over Time

Wed, 09 Jan 2013 10:20:30 -0500

Yesterday the New York Times reported that the average price for a Super Bowl ad was between $3.7 million and $3.8 million this year. Here is the average price of a Super Bowl spot over time, adjusted for inflation:

CPM represents the cost for every thousand viewers. The ratings for each game are from Zap2It. I found historical price data on Wikipedia, although I do not completely trust the numbers; I edited what appears to be an overly precipitous price drop after the dot com bubble burst. That said, others have been through this exercise before and the graphs look quite similar, so I believe the data are approximately correct.

Fewer Crimes Are Committed on Christmas

Thu, 27 Dec 2012 16:09:00 -0500

Based on data from the Philadelphia Police Department, 1,640 crimes are committed each day in Philadelphia. That number falls to 838, or about half of the average, on Christmas day.

The following chart shows the number of crimes reported on each calendar day between 2006 and 2012. Note that crime data for the last few days of December 2012 is projected based on 2006-2011 data.

The graph does not show a post-Christmas spike, so the drop in crime appears to be real (i.e., the PD does not defer the reporting and timestamping of a crime until the next working day). The next nine lowest-crime days are all in the cold of February, which is true of New York City, too.

Other holidays show significant drops in crime. For the following graph, 11/1 corresponds to the first Monday in November, 11/2 to the first Tuesday, 11/8 to the second Monday and so on. For simplicity, only 28 days are plotted for each month.

Here, the day with the fewest crimes is the first Wednesday in March, which shows a 38% drop from the average. I am not sure why. Thanksgiving is the sixth safest day with a 32% reduction. Labor Day, the first Monday in September, exhibits a 12% drop while the first Sunday in September (the day before Labor Day in six of the past seven years) shows an 11% drop. Crime is at or above average across other days in early September.

Wishing everyone a happy and safe holiday season! And in case you are wondering, New Year’s Eve and Day are in fact 17% and 11% safer than an average day.