design:analysis::datascope:analytics

Smelly Socks, Beards, and Streaks -- Pt. 1

Thu, 18 Apr 2013 00:00:00 -0500

If you pay any attention to hockey whatsoever, you know that the Chicago Blackhawks had a historically strong start this year, obliterating a particular NHL record as they opened the abbreviated 2013 season. The last three teams to start out this well took home the Stanely Cup.

The Blackhawks charge the record books

The Blackhawks set a new record for what’s called the “season-opening point streak.” A “point streak” is not quite a win streak, because it also includes games that are a tie after 3 periods, even if the team goes on to lose in overtime or a shootout.

Why is it called a “point” streak? For any game that does not result in a regulation loss, a team is rewarded at least one point in the standings: one point for tying in regulation but losing in overtime or a shootout, or two points for any win. The points are used to determine which teams go on to the playoffs. Hence, a “point streak,” and one starting at the beginning of the season is a “season-opening point streak.”

People started noticing the Blackhawks’ season-opening point streak after their February 15 win against the San Jose Sharks. This advanced their record to 11-0-3, for a point streak of 14 games, tying them for third place for best-ever season-opening point streak with the 1944 Montreal Canadiens. In the 1943-1944 season the Canadiens established this record and went on to win the Stanley Cup

Over the next three games, the Blackhawks surpassed the 1985 Edmonton Oilers, who broke the record with a 15-game season-opening point streak in the 1984-1985 season and won the 1985 Stanley Cup, and the 2007 Anaheim Ducks, who set the current record in the 2006-2007 season with 16, and again won the 2007 Stanley cup.

The Blackhawks continued to build their new record for eight more games, finally losing 6-2 against the Colorado Avalanche on March 8 after 24 games without a loss. The past three teams to set this record won the cup, and the Blackhawks didn’t just break the old record. They smashed it. Does this mean that the Blackhawks are going to bring the Stanley Cup home to Chicago again in 2013?

Now, any data junkie worth her datapoints knows that three is far too small a sample size to make a decent predictive claim about the season’s outcome. And anyone with their head screwed on correctly probably suspects that there is no magic threshhold for the season-opening point streak that will cosmically guarantee ultimate victory. But, using the full dataset of NHL wins and losses stretching back to 1942, we can see historically how teams have done depending on their opening season point streak.

take me to the visualization »

What we made

We took data on every hockey game from the start of the Original Six era through the last full season and calculated the season-opening point streak for every team in every season. If you’re curious, that comes out to 1,155 team-seasons and 45,011 hockey games played between 1942 and 2012.

We also took note of how each team fared at the end of the season and coded their playoff outcome in one of six categories - one for missing the playoffs, four for getting eliminated at each possible round of the playoffs, and one for going all the way and winning the Stanley Cup.

Finally, we put all of this information in the widget at the link so that you can explore the data and decide for yourself whether the season-opening win streak has a metaphysical effect on Stanley Cup victory.

Go check it out, here.

So what is the point of all this?

Have we just gone to a lot of trouble to show you that better teams are better? Not really. The Blackhawks have not merely put up a strong number for this statistic: their performance was the best ever. And our data show that there is, to date, a nonlinearity in the outcomes of teams with very high season opening point streaks. This is best illustrated in the line graph at the top right of the widget. As the point streak cutoff increases, the chance of winning the Stanley Cup increases, smoothly at first, and then rocketing straight upwards around 12. This graph changes slightly depending on the year range selected, but for all seasons, the probability of winning the cup at 12 games is 50%, at 13 games is 80%, and (as we already know), all teams achieving 14 or greater consecutive point-scoring games at the beginning of the season went on to win it all.

Why? Why? Why!

Thu, 04 Apr 2013 00:00:00 -0500

Lunch? Our Chicago food truck tracker goes mobile and infiltrates UChicago!

Mon, 01 Apr 2013 00:00:00 -0500

Photo credit: Jamie Manley of The Chicago Maroon

Last spring, when we were new to working in the Chicago Loop and wanted to try all the great eateries on wheels that concentrated downtown, we built Lunch?. Lunch? is a web app that tracks and maps Food Trucks based on what they tweet, filtering through all the Twitter banter to get straight to the point: finding food.

We’ve listened to the feedback that you guys have given us, and have worked to improve our app. Now, we present to you a new and improved version that now works on mobile devices and also includes Hyde Park at the University of Chicago!

Food Trucks are now allowed to cook on site, so try using Lunch? to find some fresh, tasty mobile dishes on your desktop, tablet, or smartphone!

Physical histograms of mail

Thu, 28 Mar 2013 00:00:00 -0500

Photo by Dean Malmgren

Four short months ago, before the recent frenzy over rural post office closings and 5-day delivery, we had a lunch conversation about the utility of the U.S. Postal Service. The Postal Service has an important role in American history, but I’ve been frustrated by the huge fraction of mail that is unsolicited catalogues and brochures (could this be the reason?). At lunch, I think I was even bold enough to claim that “close to 99% of the mail we received was junk.”

To test just how far-fetched that claim was, and intrigued by doing a non-digital project, I’ve been hoarding all incoming mail to DsA, including bills, payments, brochures, letters, holiday cards, take-out menus and catalogues. Every day, I’ve split the mail into two piles, creating a histogram that physically sits on a table. The first pile consists of “legitimate mail” that was requested by DsA in one form or another, such as invoices, checks, and other mail personally sent to DsA. The second pile consists of “junk mail” that was not specifically requested by DsA, such as brochures, catalogs, Comcast pamphlets, etc.

I concede that 99% was an aggressive estimate, but was still surprised to see that junk mail literally outweighs legitimate mail 2:1. This is admittedly a small sample (n=1 address), but seriously, I’m amazed at how much mail is unsolicited crap.

What is even more surprising is the fraction of junk mail that is just a glossy brochure or catalogue versus delivered in an envelope.

For those with a sense of humor, most of the “junk envelope” deliveries are from Comcast soliciting their high-speed internet service. This is pretty stupid because of they know we can’t even be customers: we requested Comcast service when we moved in and they wouldn’t wire our building. Seriously Comcast, can’t you save some paper and postage by doing a better job at targeting customers that have a non-zero probability of buying your shit?

More staggering still is the fact that, after buying some furniture from them, Crate & Barrel/CB2 has sent us a glossy page for every glossy page that all other senders send, combined. In fact, since Crate & Barrel sends us two copies of everything, they have sent us just as many pages of unsolicited crap as our entire business has received legitimately over the last few months [not pictured]. Wow.

In addition to being simple, this side-project was an enjoyable way to visualize data. We often set up complex databases and analysis for digitally recording physical things, but in this case all it took was a photo and, voilà, a histogram of the objects which we want to analyze. We’ve seen other examples of physical histograms, such as M&M colors in a bag, music collection aggregated by artist initial, and exam grades. Do you have any other examples of physical histograms?

The Cheating Commish

Thu, 21 Mar 2013 00:00:00 -0500

The time for NCAA bracket submission is upon us. As promised last year, we one-upped ourselves and present to you another DsA creation: a data-driven app that chooses the best possible bracket for your pool, which we call:

The Cheating Commish does exactly what it advertises: if you are the commissioner of your NCAA tournament pool, and you have few to no scruples, this tool allows you to enter your opponents’ brackets and scoring rules of your pool and will calculate the bracket that will maximize your likelihood of taking away your friends’ hard-earned money.

If you’re not the commissioner of your league, or if you’re just not really in a kill-or-be-killed, Rodney Ruxin kind of mood, we have a gentler option, too, which calculates the bracket with the greatest payoff likelihood for your pool. We model the competition based on the “market” of actual bracket picks from espn.com.

The whole process takes a little bit of time (up to ten minutes if you have a lot of brackets to enter), so if you are in a hurry, click now and come back to read about the method behind the madness.

How to use The Cheating Commish

Add a new bracket by clicking on the “+” at the top. Enter your opponents’ picks by clicking through the team names. Modify the points system and payout structure of your pool in the settings. If you don’t know anyone else’s brackets, you can still enter details about your pool (number of opponents; winnings for top finisher, top three, etc), and we’ll generate optimized picks for your pool structure based on mass-market picks.

When you’re done, just click “Submit all brackets.” We’ll tweet or email you when your best bracket is ready to view.

Work smart, not hard

Last year, we tried to create the ultimate predictive algorithm to foretell the winner of the NCAA tournament. Offensive ranking, defensive ranking, strength of schedule, home court advantage, boosts for win streaks -- like alchemists slaving away, ever tinkering, we produced a bracket-picker which was a burnished work of art... and which came up with pretty much the same picks as everyone else out there.

This year, in our more seasoned maturity, we realized that there is greater power not in joining the pack, but in observing it. Hundreds of dataheads just like us were using the same available statistics to make the same bracket picks, arguing over the eleventh decimal place like hyenas gnawing the last bits of meat from a carcass. If you want the sleekest, most sophisticated bracket based on statistical analysis of performance, we are sure you will be able to find it. The problem is, everyone else can find it, too, and eventually entire pools will be winning and losing the same points on the same games because their brackets were all based off of essentially the same model, and everything will end in a boring, 12-way tie. You’ll all go home with the same $5 you put in the pot, and worst of all, no one will get to taunt anyone else.

The smarter approach comes from looking at the actual people that you will be competing against and making your picks based on what they have picked. Maybe your pool includes three or four stats junkies, some superfans who can’t bring themselves to pick against their alma mater, and a handful of fairweather followers who make their picks based on the cutest mascots and jersey colors. For any given pool, there exists some bracket which, in light of the other brackets in the pool and the underlying performance-based probabilities of tournament outcome, will maximize the chance of walking away with a piece of the money. The Cheating Commish is the app that finds that “golden” bracket for you.

How it works

The Cheating Commish works by exploiting the differences between picks that are the most popular and picks that are most likely to advance in the tournament. These gaps are defined within the context of a “market” -- your competition. Since winning your bracket pool depends on your performance vs. the peers in your market, victory depends not only on choosing the winners, but on picking right when your competition picks wrong.

So what are people picking? We can get an idea of what most brackets look like by using espn.com’s Who Picked Whom. And to determine which picks are undervalued, we need a trusted source, which we cull from the demonstrated statistical prowess of Nate Silver. To illustrate how the Cheating Commish exploits differences between popular opinion and statistical likelihood of winning, take a look at Louisville and Florida. Louisville is a properly-valued pick. Nate Silver has them at a 23.8% chance of taking the tournament, and sure enough, 21.7% of espn.com brackets have Louisville in the top position. Now look at Florida. As of writing, only 2.9% of users have chosen Florida to win the tournament, but their odds of winning are much higher than that -- 13.8%. This discrepancy means that making a bet on Florida could help you to pull away from the pack.

This, of course, is only valid when “the pack” is everyone making picks on espn.com. But we can do a better job by tailoring our picks to a more specific “market” -- to the brackets in your particular pool, the ones that your bracket will actually be competing against. Maybe your bracket pool isn’t undervaluing Florida, because they read Nate Silver just like we do. But due to certain sentimentalities (have a bunch of Michigan alums in your pool?), shared readership of the same sports blogs, or simple fluctuations of chaos, there are other gaps that can be exploited.

This is when the Cheating Commish really shines: anyone with access to the brackets of everyone else in his or her pool can enter those brackets, and a genetic algorithm will find the bracket that is the most likely to result in cash winnings for that specific pool. In addition, we take into consideration other factors like the total number of people in the pool and the specifics of who walks away with money. The optimal bracket is different if you need to take bigger risks and shoot for first place or nothing, or if you can be a little more conservative because the top three places all take money. (Side note: if you are in a pool where multiple finishers win money, but you would rather follow a first-place-or-nothing strategy, feel free to lie about the pool structure when you enter your brackets. The Cheating Commish will not know the difference, and it certainly has no room to judge you on moral grounds.)

The Genetic Algorithm

Since there are 63 games played in the NCAA tournament finals, there are a total of 2⁶³ different possible brackets. If simulating one bracket takes a tenth of a second, it would take 2,500,000,000,000,000 (2.5 quintillion) years to run all possible brackets and find the one overall winner. In order to get a bracket selected by Thursday at noon, we have to take a more sophisticated approach.

We simulate the tournament 100 times, where each team’s chances of winning are based on Nate Silver’s predictions. For each set of tournament simulations, we calculate how each of your opponents fared based on their bracket picks, the points system, and payout structure of your pool. We also randomly generate 50 possible brackets for you, and calculate how they would fare.

This is where genetics come in. If we view each of these 50 possible brackets as a sequence of 63 games where each game has “Team A” winning or “Team B” winning, this is like viewing each bracket as a string of A’s and B’s 63 letters long, sort of like a DNA string!

We then calculate the “fitness” of your 50 brackets against your opponents for 100 tournament simulations, and the top 10% (i.e. the 5 brackets that made the most money) “survive” to the next round of the algorithm; the bottom 45 brackets become dinosaur food.

Next, we “mutate” most of the remaining brackets by randomly changing the outcome of a few games, and make some of them “have children.” Having children means first undergoing genetic recombination: taking two different brackets, switching their strings at a random point, and outputting a new string that has parts from each of its “parents.” We also introduce new brackets, again by randomly generating strings of length 63, as “immigration” to this population of brackets.

The outcome of all this genetic stuff is that we have 50 brackets to try in the next set of tournament simulations: some of these brackets are the top performers from last time, some are “related” to those top performers, and some are totally new.

After 200 generations, the bracket you take to your pool has already emerged a victor several thousand times over. We can’t guarantee that the result will be the same in your pool, but you can submit your picks with confidence that they have been polished by the finest algorithms science has to offer.

As always, we welcome your feedback, and we’re always looking to improve. Please tell us what you think!

Strata 2013 Santa Clara Tweeps

Wed, 27 Feb 2013 00:00:00 -0600

A couple of us (@deanmalmgren and @mstringer) just arrived in Santa Clara for the Strata Conference. We're looking forward to meeting people, staying aware of what's going on, and speaking about one of our projects.

To help navigate who's speaking and tweeting at the conference, we scraped up some data about Strata attendees and speakers that use Twitter. The following table shows anybody who has a valid Twitter handle linked on their Strata conference profile, along with whether they are speaking and the number of followers within the Strata conference.

Name	Twitter	Speaker?	Followers
Philipp Reiner	@phillyooo	No	0
Doug Willoughby	@dougwillobee	No	0
Edd Dumbill	@edd	Yes	84
Kavalpreet Kawatra	@kavalpreetganti	No	2
Sami Ahmed	@samiahmed	No	0
James Hendler	@jahendler	Yes	10
Michael Stringer	@mstringer	Yes	5
Brad Lynch	@blynchdata	No	0
Bill Carmody	@alterseekers	No	0
Marc Chipouras	@mchipouras	No	0
John Foreman	@john4man	Yes	34
Michael Taczak	@emergent_one	No	0
Sasha Issenberg	@sissenberg	Yes	2
John Faughnan	@jfaughnan	No	0
Amit Bendov	@banditmove	No	5
Keith Swafford	@heswaff	No	2
Shanley Kane	@shanley	No	16
Andrew Brown	@roughfalls	No	0
Wei Xia	@hotelpro	No	0
Felix Giguere Villegas	@felixgv	No	1
Alexander Gray	@skytreehq	Yes	3
Timo Smura	@timosmura	No	0
David Tishgart	@dtish	No	0
Ron Pecunia	@rpecunia	No	0
Luz Caballero	@gerbille	No	1
Eric Tschetter	@metamarkets	Yes	21
Jayant Shekhar	@jshekhar	Yes	2
Robyn Randell	@robynrandell	No	0
Stephanie Mcreynolds	@slangenfeld	No	0
John Akred	@bigdataanalysis	No	18
Rene Saroukhanoff	@saroukhanoff	No	0
Per Bjorke	@perbjorke	No	0
Hugo Evans	@techpascode	No	1
Amanda Laucher	@pandamonial	No	1
Kim Rees	@krees	No	28
Monica Rogati	@mrogati	Yes	60
Siyun Fan	@mind_wanderer	No	0
Peter Vanderhaak	@vanderhaak	No	0
Feilding Cage	@fcage	Yes	1
Michi Kaifu	@michikaifu	No	1
Dennis Duckworth	@dennisduckworth	No	5
Chris Chew	@chriskchew	No	1
Fred Trotter	@fredtrotter	Yes	6
Amrith Kumar	@amrithkumar	No	2
Robert Long	@roblong	No	0
Michael Bailey	@michaelcbailey	Yes	1
Martin Remy	@mremy	No	1
Jeff Bollinger	@jbollinger	No	0
Aaron Goodman	@aaronjgoodman	No	5
Andre Luckow	@drelu	No	2
Michael Bean	@ablemachine	Yes	0
Cata Milos	@catam	No	0
Jonathan Bryant	@watersofmemory	No	1
William Tully	@williamtully	No	0
Michael Kieser	@geoengis	No	1
Jim Mclean	@crosslinkcap	No	0
Kate Crawford	@katecrawford	Yes	3
Farah Giga	@amplifypartners	No	2
Khaled El Emam	@kelemam	Yes	0
Steve Paulsen	@stevenjpaulsen	No	0
Sheldon Monteiro	@sheldon_tm	No	0
Daniel Debate	@ddebate	No	0
Amina Abdulla	@inventiongirl	No	19
Mark Bonicillo	@markbonicillo	No	0
Nitin Purohit	@thisisnitin	No	0
Douglas Mason	@douglasmason10	No	3
Elisabeth Crawford	@liscrawford	Yes	2
Alex Gorbachev	@alexgorbachev	No	8
Anthony Gerrits	@anthony_gerrits	No	0
Kanghak Kim	@freefloating	No	0
Sreeni Iyer	@av0gadr0	No	0
Kathy Yu	@myid	No	0
Alexander Wilcox	@silentvisuals	No	0
Yue Cathy Chang	@yuec	No	2
Krish Krishnan	@datagenius	Yes	33
Robert Eisses	@simbatech	No	1
Michael Jones	@splitmind	No	0
Harlan Harris	@harlanh	No	16
Scott Murray	@alignedleft	Yes	15
Siva Narayanan	@k2_181	No	4
Leeann Dang	@dangtechnickel	No	1
Pauline Nist	@panist	No	1
Jeffrey Kelly	@jeffreyfkelly	No	46
Nick Gorlovski	@nivago	No	0
Brett Slatkin	@haxor	No	1
Brett Hardin	@miscsecurity	No	2
Patrick Pang	@kikhoon	No	0
Kiyoshi Mizumaru	@kmizumar	No	0
Aaron Heckmann	@aaronheckmann	No	1
Mark Madsen	@markmadsen	Yes	58
Arshak Navruzyan	@alpinedatalabs	No	4
Josh Breinlinger	@jbreinlinger	No	0
Susan Etlinger	@setlinger	Yes	5
Wes Mckinney	@wesmckinn	Yes	35
Bard Roh	@bardroh	No	0
Chanju Kim	@imseti	No	0
Marian Gallagher	@nubedehelado	No	0
Steve Mayzak	@springsource	No	9
Greg Rahn	@gregrahn	No	27
Nicolas Averseng	@naverseng	No	0
Jan Karstens	@tvittergevitter	Yes	1
Christian Beedgen	@raychaser	No	3
Monte Zweben	@mzweben	No	1
Paul Kent	@hornpolish	No	13
Vids Samanta	@vidssamanta	No	0
Mick Thompson	@dthompson	No	1
Dave Thau	@davethau	No	0
Ivan Provalov	@iprovalov	No	0
Kunal Agarwal	@unraveldata	No	0
Justin Kestelyn	@kestelyn	No	22
Robert Read	@robertread3	No	0
Sujay Kumar	@sujaykum	No	0
Dj Patil	@dpatil	No	96
Vic Hargrave	@vichargrave	No	0
Arfon Smith	@arfon	Yes	6
Don Howren	@donhowren	No	0
C. Aaron Cois	@aaroncois	Yes	0
Dr. Vijay Srinivas Agneeswaran	@a_vijaysrinivas	Yes	0
Brady Forrest	@brady	No	27
Richard Cave	@richardcave	No	1
Tim Gasper	@timgasper	No	10
Nina Stingo	@nstingo	No	0
Ben Werther	@bwerther	No	28
Eric Legrand	@elegrand	Yes	1
Wendy Shepperd	@wendyshepperd	No	1
Basudev Chaudhuri	@basudevc	No	0
Carlo Brooks	@citysearch	No	0
Shishir Garg	@gargs	No	0
Joe Savak	@joesavak	No	1
Darwin Schweitzer	@bizinfomall	No	0
Svet Ivantchev	@svetef	No	0
Andrew Caird	@acaird	No	0
Pk Gnanam	@pk_gnanam	No	0
Neil Kodner	@neilkod	No	33
Antoni Batchelli	@disclojure	No	6
Joe Markee	@jdm_100	No	2
Alex Miller	@puredanger	No	25
Jen Van Der Meer	@jenvandermeer	Yes	0
Joe Ray	@joeanalytics	No	0
Mohanganeesh Dorairaj	@mohanganeesh	No	0
Zoltan Toth	@zoltanctoth	No	0
Rich Miller	@rhm2k	No	6
Alec Shuldiner	@biggreenbrother	No	0
Shingo Omura	@everpeace	No	0
Abe Taha	@abetaha	No	8
Jeremy Howard	@jeremyphoward	Yes	37
Rajan Bhatt	@rajanbhatt	No	0
Jean-Thomas Rouzin	@webgeoservices	No	0
Eldad Farkash	@eldadfarkash	No	1
Jesse Hamlin	@jesse_hamlin	No	1
Christo Kutrovsky	@kutrovsky	No	1
Robert Towne	@binarymechanic	No	0
Tom Newton	@thomaswnewton	No	1
Greg Lindahl	@glindahl	Yes	4
Sriram Rangarajan	@smrangara	No	0
Carl Schmidt	@carl_schmidt	No	0
Jeff Richards	@jrichards	No	0
Timothy Mohn	@m3thdman	Yes	1
Naomi Robbins	@nbrgraphs	Yes	11
Karen Davis	@kroterd	No	0
David Andrzejewski	@davidandrzej	No	10
Vivek Chandran	@vivekchandran	No	1
Scott Feldman	@patentesquire	No	0
John Crupi	@johncrupi	No	4
Anna Smith	@omgannaks	Yes	7
Thomas Cabrol	@thomascabrol	No	3
Alan Gardner	@alanctgardner	No	1
Shawn Davis	@shawnrdavis	No	0
Jongsoo Lee	@everyevery	No	0
Joseph Cline	@joscline	No	0
Carl Steinbach	@cwsteinbach	Yes	17
Howard Green	@azulsystems	No	1
Justin Erickson	@justinerickson	Yes	7
Herain Oberoi	@herain	No	2
Tony Mignot	@tmignot	No	0
Joey Echeverria	@fwiffo	No	14
Luis Soto-Maldonado	@luisdans	No	4
Chris Nelson	@gopherkhan	No	0
Tamara Dull	@tamaradull	No	15
Jonathan Creasy	@johann8384	No	4
Donald Fischer	@dff	No	3
Julian Hyde	@julianhyde	No	15
Loren Larsen	@lorenlarsen	No	0
Sumanth Chinthagunta	@xmlking	No	0
Meghan Desai	@meghandesai	No	0
Nancy Grady	@nancygrady	No	0
Bukyoung Choi	@zeus7993	No	0
Tim Abraham	@timabe	No	1
Suzanne Axtell	@suzaxtell	No	14
Renee Chu	@reneighbor	No	0
Simon St. Laurent	@simonstl	No	9
Patrick Moroney	@patrickemoroney	No	0
Chak Kolli	@chakk	No	0
Alexey Kharlamov	@aih1013	No	0
Nadav Aharony	@nadavaha	Yes	1
Mario Brenes	@mariobrenes	No	0
Steve Gross	@sgross24	No	3
Kirk Gray	@kirkgsworld	No	0
Joe Hellerstein	@joe_hellerstein	Yes	53
Parviz Deyhim	@pdeyhim	Yes	1
Robin Bloor	@robinbloor	No	22
Renee Diresta	@noupside	No	11
Kaniska Mandal	@phi_symmetry	No	0
Jef Sharp	@ej2040	No	0
John Feland	@jmfeland	Yes	1
George Powlick	@gpowlick	No	0
Ian Moss	@teserasystems	No	1
Lisa Green	@boudicca	Yes	10
Vaibhav Nivargi	@vnivargi	No	1
Al Nevarez	@imusicmash	No	8
Steve Kaeser	@stevekaeser	No	0
Nathaniel Coon	@nathanielcoon	No	1
Alasdair Allan	@aallan	Yes	23
Peter Skomoroch	@peteskomoroch	Yes	104
Phillip Morris	@ecophilm	No	0
Justin Langseth	@justinlangseth	Yes	1
Joel Grus	@joelgrus	No	3
Dulcie Madden	@restdevices	No	0
Sebastian Pietrowski	@pedrowaty	No	0
Byron Ellis	@fdaapproved	Yes	5
Ian Murphy	@ibmurphyatdi	No	6
Erdem Agaoglu	@agaoglu	No	0
John Zimmerman	@datapop	No	0
James Todd	@jwtodd	No	2
Greg Kroleski	@gregkroleski	No	0
Aldo Cistrone	@cistronealdo	No	0
Sergey Boldyrev	@verdyr	No	0
Lengning Liu	@lengningliu	No	0
Dave Danielson	@dreasoning	No	11
Arjanna Van Der Plas	@arjannavdplas	No	1
Jed Crosby	@jedcrosby	No	0
Benjamin Tseng	@benjamintseng	No	1
Anish Jariwala	@anishj	No	0
Stewart Collis	@stewartcollis	Yes	0
Stefan Groschupf	@datameer	No	43
Mauricio Vacas	@mjvacas	Yes	2
Daniel Tunkelang	@dtunkelang	Yes	40
Suzanne Hoffman	@revenuemaven	No	3
John Rauser	@jrauser	No	21
Brandon Eller	@bl_eller	No	0
Michael Ludden	@michael_ludden	No	0
Christian Racca	@pendolare	No	1
Boyd Meier	@bwmeier	No	0
Bradley Gula	@notbrad	No	2
Jennifer Pahlka	@pahlkadot	Yes	34
Danese Cooper	@divadanese	No	4
Peter Grossman	@petergrossman	No	0
Peter Schlampp	@peteschlampp	No	6
Pashu Christensen	@hivedata	No	10
Anurag Shrivastava	@xebiaindia	No	0
Rich Dill	@richdill	No	1
Jameson Buffmire	@jtbuffmire	No	0
Evan Miller	@evmill	No	1
Rick Gordon	@rgordon0	No	0
Josh Dzielak	@dzello	No	0
Stan Van De Burgt	@stannie	No	2
Jason Rutherglen	@jasonrutherglen	Yes	5
Uri Laserson	@laserson	No	4
Rayid Ghani	@rayidghani	Yes	7
Bogdan State	@bogdanstate	No	0
Michael Kopp	@mikopp	No	4
Nathan Shetterley	@nateshetterley	No	1
John Myers	@johnlmyers44	No	26
Mubarak Seyed	@mubarakseyed	No	4
Chris Wensel	@cwensel	No	33
Michael Manoochehri	@ntangledmichael	Yes	3
Ted Dunning	@ted_dunning	Yes	43
Matthew Jacobs	@matt_jacobs_	No	7
Darcy Dittrich	@darcyd	No	1
Jacque Istok	@jstok	No	1
Raj Pai	@claritics	No	0
Vishwanath Ramarao	@impermium	Yes	0
Jonathan Hsieh	@jmhsieh	Yes	32
Sanjeev Kumar	@dataplumbers	No	3
Ed Ruth	@edruthless	No	0
Ken Krugler	@kkrugler	No	16
Chad Carson	@chadcc	No	1
James Kobielus	@jameskobielus	No	58
Marck Vaisman	@wahalulu	Yes	27
Robert Munro	@wwrob	Yes	7
Jonathan Seidman	@jseidman	No	23
Brian Granger	@ellisonbg	Yes	3
Julie Albright	@juliemaya	No	0
Kate Matsudaira	@katemats	Yes	11
Keith Kohl	@keithkohl	No	1
Supreeth Rao	@supreeth_	No	0
Adam Flaherty	@adfm	No	2
Ben Book	@benbook1	No	0
Susan Puccinelli	@susanpuccinelli	No	3
Jordan Mendelson	@aloisius	No	3
Steven Schneider	@slingingsw	No	3
Avik Dey	@avikonhadoop	No	21
Valerie Allman	@vallman	No	1
Kamal Shah	@kdshah	No	0
Brian Swan	@brian_swan	No	0
Erin Davis	@erinclinedavis	No	0
Frank Hangler	@hangler	No	0
Hope Nicora	@hopenic	No	12
Anant Chintamaneni	@anantcman	No	0
Nadieh Bremer	@nadiehbremer	No	0
Nicolas Garcia Belmonte	@philogb	Yes	12
Hoop Somuah	@hoopsomuah	No	1
Curtis Sasaki	@cjsasaki	No	0
Cory Mollet	@corymollet	No	0
Aleksandar Mastilovic	@amastilovic	No	2
Patrick Farrell	@pfarrell	No	0
Isaac Lopez	@_isaaclopez	No	1
Philip Kromer	@mrflip	Yes	53
Andrew Purtell	@akpurtell	No	17
Koby Holzer	@kobyholzer	No	0
Sai Devulapalli	@sdevulap	No	0
Maureen Jennings	@maureenjennings	No	10
Andre Panisson	@apanisson	No	2
Matt Kennedy	@fusionio	No	4
Kevin Lo	@tsg_kevin	No	0
Alex Benik	@abenik	No	5
Steve Pessah	@appfirst	No	4
Bill Jacobs	@bill_jacobs	No	2
Kumar Saurabh	@ksaurabh	No	3
Ewa Dominowska	@medio	No	0
Joshua Mcnary	@spatialcloud	No	0
Drew O'Brien	@drewtweat	No	6
Gopi Chandra Magam	@gopimagam	No	0
Liz Derr	@simularity	No	0
Jonathan Natkins	@nattyice	No	21
Hagen Finley	@finhagen	No	0
Felicia Yiu	@eventbrite	No	7
Richard Miller	@signedbyte	No	0
Steve Trammell	@stevetl2	No	0
Kevin Fink	@kevinfink	No	0
Michelle Guy	@michelleg2	No	0
Boyd Wilson	@boydwilson	No	0
Wojciech Gryc	@wojciech	No	1
Andrew Winterman	@andywinterman	No	2
Molly Turner	@mollysturner	Yes	0
Akihito Kobayashi	@akihito	No	0
David Henry	@ds_henry	Yes	1
Ali Olia	@aaolia	No	0
Anne-Laure Thieullent	@althieullent	No	0
Marc Demarest	@marcdemarest	Yes	3
Shivon Zilis	@shivonz	No	6
Alistair Croll	@acroll	Yes	80
Eric Chan	@ericchansf	No	0
Robert Beebe	@rgbeebe	No	0
Wil Stuckey	@quiiver	Yes	0
Patrick Dirden	@pdirden	No	4
Chad Richeson	@chadricheson	No	1
Sonia Zapien	@soniazatoreilly	No	1
Kay Iwaisako	@kiwaisako	No	0
Bj Wishinsky	@leapingwoman	No	2
Cass Petrus	@mathcass	No	2
David Bainbridge	@dubhe	No	0
Michael Baum	@michaelbaum	No	6
Kevin Cox	@kjcox	No	2
Alexander Rasmussen	@alexras	No	1
Gretchen Malay	@gretchen_malay	No	13
Vincent Dell'Anno	@vdellanno	No	0
Ben Hamner	@benhamner	Yes	12
Matthew Farrell	@mtoddf	No	2
Ryan Rosario	@datajunkie	No	47
Ju-Kay Kwek	@jukaykwek	No	4
Bruno Aziza	@brunoaziza	No	30
Mark Fleming	@fleming_mw	No	0
Lutz Finger	@lutzfinger	Yes	3
Julie Steele	@jsteeleeditor	Yes	45
John Jarve	@johnjarve	No	0
Peter Nachbaur	@peternachbaur	No	0
Larry Carvalho	@robustcloud	No	7
Brian Anderson	@saleschannels	No	0
Ciro Cattuto	@ciro	No	4
Marilyn Craig	@insightvoices	No	0
William Santo	@isominer	No	0
David Beckett	@dajobe	No	7
Frank Lopez	@flopezar	No	0
Ari Gesher	@alephbass	Yes	4
Sam William	@samwilliam	Yes	1
Surendra Reddy	@sureddy	No	7
Gurminder Gill	@genx7up	No	0
Erin Kang	@erinkang	No	0
Noah Iliinsky	@noahi	Yes	41
Daniel Koller	@dakoller	No	2
Jason Beck	@spooksandgeeks	No	8
Sean Murphy	@sayhitosean	Yes	4
Tim Van Ash	@tvanash	No	1
Joseph Turian	@turian	Yes	34
Peter Kourinos	@petekourinos	No	0
Joanne Mcdougald	@jomcdougald	No	0
Barry Fischer	@opoweroutlier	Yes	2
Amir Youssefi	@amiryoussefi	No	1
Ben Sabrin	@bsabrin	No	2
Dipti Borkar	@dborkar	No	0
Tsahi Levent-Levi	@tsahil	No	0
Satyam Priyadarshy	@priyadarshy	No	0
Derick Jose	@judederick	No	2
Mok Oh	@mok_oh	No	19
Gina Dicenzo	@grdicenzo	No	1
Dean Malmgren	@deanmalmgren	Yes	3
Jana Lass	@itsjustjana	No	1
Rick Farnell	@bigorgohome	No	8
Jerome Cukier	@jcukier	Yes	18
Simon Warrington	@spwarrington	No	0
Rocky Yost	@rockyyost	No	0
Daryush Laqab	@dlaqab	No	0
Klaus Roder	@klausroder	No	0
Jowanza Joseph	@jowanza	No	1
Yvan Gauthier	@ygauthie	No	0
Rob Styles	@mmmmmrob	No	0
Jonathan Bruce	@jonbruce	No	1
Douglas Van Der Molen	@dvmolen	Yes	1
Jessica Marie	@jessicamariemba	No	7
Junya Yamaguchi	@junya100	No	0
Tom Hogan	@tomhogan4	No	0
Sandra Crucianelli	@spcrucianelli	Yes	1
Tony Lee	@tosajang	No	0
David Polley	@davidapolley	No	0
Miguel Ortega Hesles	@mohesles	No	0
Alan Dyke	@adyke	No	0
Lara Shackelford	@larashackelford	No	9
Karan Bhatia	@sdksb	Yes	1
Rob Rustad	@healthydata	No	1
Andy Fligel	@afligel	No	0
Paco Nathan	@pacoid	Yes	31
Amr Awadallah	@awadallah	No	70
Tendu Yogurtcu	@tenduyogurtcu	No	0
Zachary Auger	@zacauger	No	0
Ravi Iyer	@ravi_polipsych	No	0
Peter Evans	@na	No	0
Allison Urban	@allisonurban	No	1
Jose Luis Agundez	@ciberjos	No	0
Vicente Moranta	@vicentemoranta	No	0
Tushar Shanbhag	@tusharis	No	1
Michael Masterson	@mlmasterson	No	0
Lynwood Bishop	@maplarge	Yes	1
Sharmila Shahani-Mulligan	@shahanimulligan	Yes	4
Mac Slocum	@macslocum	No	10
Chen Peng	@pengparic	No	0
Nathan Marz	@nathanmarz	Yes	50
Pamela Roussos	@pamelaroussos	No	2
Renald Buter	@dataludentes	No	0
Michael Lang	@mlang2	No	0
Eric Baldeschwieler	@jeric14	No	41
Dj Das	@djdas	No	2

To generate the table, we pulled all of the people with valid twitter handles from the conference attendee directory with a python script and then used the Twitter API to find all of their followers. The data was collected on February 25, and is not being updated as the conference goes on (and as people meet each other at the conference and follow each other!). We'll post a follow up with what this looks like post-conference, but in the meantime we hope you find this useful as you meet folks at Strata. Hope to talk to you soon!

Mike interviewed at Strata 2013

Wed, 27 Feb 2013 00:00:00 -0600

Holiday nerdery! Snowflakes!

Fri, 21 Dec 2012 00:00:00 -0600

As we have done in years past, we wanted to thank all of the people — our clients, colleagues, independent contractors, advisors, friends, and family — that have contributed to our success this year. As a token of our gratitude (and to celebrate Chicago's first snowfall in a long time), we thought we would bring back the good ole' days of paper snowflakes with a d3.js visualization that unfolds before your eyes.

Happy Holidays!

Drag the circles on the perimeter of the triangle to “cut” shapes into the paper. Unfold the paper to view your snowflake!

Thomson Reuters: visualizing the collaborative nature of academic reports

Tue, 11 Dec 2012 00:00:00 -0600

Following the “elaboration phase” workshop where we reimagined the presentation of several analyses in Thomson Reuters’ InCites product, we started working on the first “quick wins” project: redesigning the Collaborating Institutions and Collaborating Authors reports in InCites. These reports were created to address two primary use cases: (i) university administrators that want to quantify the extent to which their institution and faculty are collaborating and (ii) funding agencies that want to target well-connected institutions and authors.

As a first stab at quantifying this information, Thomson Reuters developed the initial collaboration reports by producing tables of data related institution and author collaborations. The two major challenges with presenting this information in a table is that (i) it is very hard to convey a sense of collaboration in a tabular format and (ii) since collaborations are between two authors/institutions, presenting collaboration information for N entities would require a maximum of N2 rows in a table. Here is a screenshot of the original report illustrating these challenges.

Keeping the initial use cases in mind, we started our redesign process by developing several sketches that started to work out the details of how we could visualize the collaborations between different entities in a useful way. Our first design principle was to evoke something that looked like institutions and authors working together. Rather than displaying the data in its raw tabular form, we switched to visualizing these collaborations as a network whose nodes represented institutions or authors and whose links quantified the extent of their collaboration.

Considering that university administrators are primarily interested in who their main collaborators are and that funding institutions want to evaluate the collaborative capability of individual institutions or authors, we made the design decision to limit the visible network to an ego network around a single focal node. It further reduces the complexity of visualizing a massive network of collaborations that, although intriguing, are overly cluttered for making quick decisions.

Combining the network view with a searchable table makes it possible to quickly identify various focal institutions and authors that can be examined along several different bibliometric dimensions. We are particularly proud of this interface as it demonstrates the utility of designing infographics starting with the use cases of real people and iteratively improving the concept.

Thomson Reuters: product reimagination for visual presentation of analytics

Tue, 04 Dec 2012 00:00:00 -0600

Thomson Reuters has long been a leader in providing services for aggregating, organizing and analyzing scholarly publication data. In continuing its leadership position, Thomson Reuters recently developed a new product, InCites, which is intended to offer advanced analytics for university administrators and funding agencies to further set itself apart from the competition. The data behind InCites is world class, but the presentation of the information and product workflow has yet to reach its maximum potential.

Thomson Reuters engaged DsA for a one-week “elaboration phase” workshop at their offices where we engaged people from sales, design, development and management roles to discuss the future of the InCites product. Beginning with Thomson Reuters team’s deep understanding of their customers and their needs, we brainstormed a suite of InCites user personas and their corresponding use cases to better understand what would ideally be required of an advanced analytics product.

These discussions helped guide our development and design of a short- and long-term redesign for the InCites product. Our short-term, “quick wins” design was geared toward keeping the overall layout and workflow of the existing product the same, but redesigning the reports to be much more visual and accessible, with gory details available on demand. The idea is that these quick wins would give the InCites team the ability to modernize the look of the InCites product without a large-scale investment in a new product altogether. Our long-term, “revamped” design was geared toward reimagining how a simpler, more elegant, and more dynamic design could cover each use case in a much more streamlined fashion by building on the quick wins solutions. As a deliverable from the workshop, we provided a detailed action plan about how our proposed implementations will improve the Thomson Reuters product in the short- and long-term.

As our business has become progressively more geared toward the creative development of data-driven solutions, these “elaboration phase” workshops have started to become a staple of our primary offering. These elaboration phase workshops give all stakeholders an opportunity to provide input and result in a clear roadmap on top of which we can develop useful and usable data-driven products.