Jeffrey Veen

Video: Designing for Big Data

jeff — 2009-04-21T08:46:47

This is a 20-minute talk I gave at the Web2.0 Expo in San Francisco a couple weeks ago. In it, I describe two trends: how we're shifting as a culture from consumers to participants, and how technology has enabled massive amounts of data to be recorded, stored, and analyzed. Putting those things together has resulted in some fascinating innovations that echo data visualization work that's been happening for centuries.

I've given this talk a few times now, but this particular delivery really went well. Only having 20 minutes forced me to really stay focus, and the large audience was very engaged. I'll be giving an extended version of this talk in June at the UX London conference, with a deeper look at how we integrated design and research while I was at Google.

Announcing Wikirank: Tracking what's popular on Wikipedia

jeff — 2009-03-26T10:04:09

A few months ago, the four of us in Small Batch Inc were kicking around ideas for what we should build next. We had just launched the Twitter Election site -- a fun real-time data visualization project -- and wanted to keep building things that help people make sense of the enormous amount of information that bombards us each day. Late one night soon after, I followed a link to a repository of Wikipedia server logs. There were gigabytes of data sitting there just begging to be visualized. We got to work.

The result is Wikirank, a tool for exploring what's popular on Wikipedia, discovering comparisons between topics, and sharing them with the world. We launched it yesterday, happily coinciding with the 14th anniversary with Ward Cunningham's invention of the wiki.

There are a bunch of reasons why I think Wikirank is cool, but my favorite is how it helps people find stories in the data. One of the great things about the web is how measuring tiny behaviors reveals patterns that tell stories. The data we get from Wikipedia is no different; as we started playing around with the numbers, we saw loads of interesting shapes emerge in the charts.

For example, big news stories show up as dramatic spikes where there there was no data before. When the astounding story broke that an airliner had made an emergency landing in the Hudson, a page was created on Wikipedia within minutes. Over the next two days, that page was one of the most popular on the site.

US Airways Flight 1549 on Wikirank

The shape takes a different angle as we watch the marketing buzz and fan excitement build towards the release of the Watchmen movie earlier this month.

Watchmen (film) on Wikirank

Comparisons are where Wikirank really shines, however. The weekly viewing habits of television watchers comes into clear view -- as do the day on which the shows air -- when we compare Heroes to Lost. (It's also my guess that the occasionally perplexing plotlines of both of those shows leads a fair number of people to Wikipedia to find out what the heck just happened.)

Lost (TV series) and Heroes (TV series) on Wikirank

I'm really pleased with how this project came out. I'll write more about the technology behind the project in a followup post, but the reality is this launch represents a pretty big shift in how we build web apps. With only a couple developers and a rack of rented machines in the cloud, we pulled this all together in just a few weeks. That simply wasn't possible the last time we did this.

The team behind all of this includes my long-time friends and business partners Bryan Mason, Ryan Carver, and Greg Veen. We were also extremely fortunate to be able to work with Dan Cederholm of Simplebits, who helped us with visual design and identity. Dan rocks.

We'd love to know what you think. You can join the discussion at Get Satisfaction.

Conference hack: Embracing the backchannel at Start

jeff — 2008-07-25T10:29:50

I had a very interesting experience a few months ago while participating in a panel discussion. Once again, I realized that the content on stage is merely the spark of a broader conversation, and that the backchannel is rapidly becoming the whole point. So we've decided to try an experiment at the Start Conference in a couple weeks to see how we might hack traditional presentations.

But let me back up a bit first. I was on a panel at this year's South by Southwest talking about the role of analytics in design. With me were two veterans of the advertising industry who's work included some of the biggest ad campaigns of the past few years - some really amazing stuff. I started the conversation by saying how the remarkable amount of audience data available to us gives designers tremendous power to affect user experiences. My collegues suggested my approach sucked the creativity out of design. I countered that they were mistaking preferential research from behavioral. The argument heated up.

While this was happening, my phone was buzzing non-stop. I slipped it out of my pocket to discretely turn it off, but noticed a stream of Twitters going by - many from audience members in the room. So I set the phone down on the table in front of me and kept an eye on it. I'm so glad I did.

As the conversation on stage continued, the stream of questions and comments from the audience intensified. I changed my tactics based on what I saw. I asked questions the audience was asking, and I immediately felt the tenor of the room shift towards my favor. It felt a bit like cheating on an exam.

I guess it really wasn't cheating, but it does illustrate one of the frustrations I've had at conferences lately. Most of the events I attend have a rich conversation happening in the room, yet the only people not able to participate are those on stage. A couple times, I've seen organizers project a live IRC channel, but that usually bring out the worst in people ("First!!!111") - and is terribly distracting. So I've been wondering for a while if there was something smart we could do at our conference.

Apparently, Bryan had the same idea. As we were planning Start, he said, "We should have someone onstage the whole time to represent the audience. Like an ombudsman does for a newspaper."

So we decided to put a desk on stage and have our friend George Oates fill that roll. She'll be on Twitter, IM, and email listening to what people are talking about. (We'll also have volunteers collecting index cards for those not wanting to be online during the sessions.) And she'll synthesize questions, interrupt us if we get boring, and call bullshit if something sounds like it. George has been the designer at Flickr since it started; her personality has always shown through there, and will be a great fit for what we're trying to do.

What do you think? I've never really seen a conference do something like this before. Have you?

Robot Cross Stitch

jeff — 2008-07-22T09:04:16

This was a project I started a couple years back that I lost between moving, packing and unpacking. I found it a few days ago and finished stitching it over the weekend. It was inspired by my favorite t-shirt back then.

I like cross stitching for it's obvious connection to pixelated icons and images from the web. I made the pattern by grabbing a picture of the shirt, knocking out the background, and posterizing the colors down to just red, yellow, and black. Then I blew up the image 400% and added a grid to show each stitch. Now, you can just upload an image here and have it spit out a pattern. You'll still need to spend time simplifying and sizing the source file, though.

I've always liked images of robots - especially mid-20th Century anthropomorphic representations. I like that style, but I also like the metaphor. Robots represent our approach to technology and, frankly, how wrong we usually are. We had hoped to create tech in our own image, like Rosie, the Jetson's maid who zipped around the house cleaning and cooking. We hoped that robots would enable a life of pure leisure - walking, talking butlers that do our bidding.

Our present reality is far from those idealistic days, but in a way I'm grateful for that. Instead of technology serving us, we've created powerful tools that amplify our abilities. We use computers to make us smarter, the web as an outboard brain, networks to stay connected. Rather than building robots as better humans, we've used technology to become better ourselves.

But that's just me at my most optimistic. The truth is I like to stitch things like this to get me away from the damn computer. I think I'll start another project...

Who is John Snow?

jeff — 2008-07-07T09:05:37

Last month, while attending the fantastic @Media conference in London, we paid a quick touristy visit to a street corner in the heart of Soho. There, we found a small monument - an old hand-drawn water pump on a stone pedestal, its handle notably missing. We climbed up on it, had our photo taken, then ducked across the street for a pint in an old pub named The John Snow.

The pub's name was interesting to me for a couple reasons. I had just finished the Steven Johnson book, "The Ghost Map," a riveting account of the neighborhood's 1854 cholera outbreak. Johnson tells the story of Dr. John Snow, who had lived near where we had our picture taken, and how he followed a hunch that the disease could be spread by water, rather than being airborne. After days of investigation, Snow proved to London's health commission that this Broad Street well was the source of the outbreak, convincing them to remove the handle. It was an early instance of science and evidence winning against superstition and prejudice.

But even more interesting to me was one of the tools he used in his work - a data visualization showing cholera deaths marked on a map correlated to the location of the pump. In the speech I gave that week, I pointed out how his graphic techniques are still a model for good design over 150 years later. Contrary to the popular myth, Snow didn't create that famous map; he was working from an existing visualization first drawn by sanitation engineer Edmund Cooper months earlier. Snow did, however, redesign the work by carefully eliminating elements and emphasizing the casualty data until the narrative of what had happened was perfectly clear. You can't look at that map without thinking, "Yikes! Stay away from that pump!" Snow's early geo-mashup can still inspire.

So back to the eponymous pub. As we stood near the crowded bar, I decided to do a little experiment. Even though there was a plaque on the wall detailing the establishment's history and namesake, I started asking the folks around me, "Hey, why is this place called 'The John Snow'?" hoping to find fans of the historic story - or maybe even someone interested in the visualization. Instead, here's what I got:

The tall Irish guy with a ponytail: "No bloody idea, brother."

The group of women laughing in the corner: "Oh, he's some wanker on the telly who bought this place years ago." (They were thinking of this Jon Snow, who is actually quite respected.)

The bartender, who came close, but really should know better: "Before this was a pub, there was a hospital here and he was the doctor in charge of it."

How quickly history fades - especially after a few drinks.

Stories we'll hear at the Start Conference

jeff — 2008-06-24T08:51:54

One of the nice things about organizing your own conference is getting to choose who you want on stage. That's been particularly fun these last couple weeks as Bryan and I have been thinking about who we'd like to talk to and what we'd like them to tell us.

We started with some good friends whom we've worked with and respect. Ev Williams, for example, was an easy choice. We've known him forever, and worked on a redesign of Blogger together a few years ago. We're talking to him about how his new thing, Twitter, is growing and evolving, what the challenges and opportunities are, and the lessons he's learned so far. I've had similar conversations recently with both Matt Mullengweg of WordPress and Mena Trott of 6apart. Their stories have interesting parallels, but even better divergence.

So, yeah, a lot of our friends are in the blogging world. But we do know a few people outside of that world. Julie Davidson and Narendra Rocherolle, for example, are joining us to talk about their unique partnership, both in business and in life. They've built Webshots, one of the first photo sharing apps, as well as 30boxes, a fantastic calendar app that stands as a fantastic example of how to do the Right Thing when building on the web.

On the other end of the entrepreneurial spectrum is our pal Merlin Mann. He's built a solid brand out of his productivity site 43Folders and we're looking forward to talking to him about how he did that. But also, I'm particularly interested in his ideas for how small teams can find the best ways to work together and communicate with each other. Also, he is a bit, um, energetic on stage. We should have a good time.

We've got a bunch more people joining us on stage - people who follow the industry, comment on it, invest in it, and help companies get started. I'll post more on them soon. I hope you can join us.

Introducing the Start Conference

jeff — 2008-06-11T12:58:40

When I recently decided to leave my job, I began a little personal project I called "100 Lunches." I wanted to sit down with a bunch of people over a meal and just listen. What was interesting to them these days? What where they building? Were they nervous about the state of the industry? Excited by all the new opportunity?

As I started the project, I quickly realized that the stories I was hearing needed a wider audience. My friends and colleagues told tales of late nights, harrowing server crashes, exhilarating growth, and touching emails from their users. Not one of them knew what they were getting into, nor would any of them change their paths.

Wouldn't it be great, I thought, if everyone could hear these amazing start up stories?

So I got together with my long-time friend and business partner Bryan Mason and started planning. We decided to get a great group of entrepreneurs, investors, and other people associated with new web companies together for a day of discussion and advice. And that's how the Start Conference came to be.

So here's the deal: On August 7, we'll spend the day at the beautiful Fort Mason Center talking about starting companies. We've got a lot of fun things planned, including a fantastic party on the Bay afterwards. And we're making it cheap - just $200. We didn't want to turn this into a conference filled with investors hunting for the next big thing. We wanted something for people who make web apps and dream of starting their own thing. (Also, we understand that you probably won't want to submit an expense report for a conference that encourages you to quit your job.)

If you've ever thought of staking out on your own, we hope you'll join us. It's going to be great fun.

All the details are on the Start Conference web site. If you have any questions, let us know in the comments here.

Welcome back Webmonkey

jeff — 2008-05-19T19:36:34

Isn't it nice to get reacquainted with an old friend?

This morning, the fine folks over at Wired relaunched Webmonkey. Somehow, they managed to pry it out of the dying hands of Lycos and revive it as a collaborative site based on MediaWiki. I couldn't be happier to see the site come back; I've just spent a nostalgic afternoon digging through everything that's there.

The original site was founded at HotWired back in 1996 by my friend June Cohen. She pulled a small team together and started publishing tutorials, news, and opinions about how to make the web. They asked me to write a column, which I did for a couple years, and eventually I joined them to redesign the site. I loved that project - it was one of the first sites we built that was dynamically driven through templates rather than as static HTML files - a real separation of content and presentation. It was a big deal for us back then, and it influenced a lot of my future work.

The new site is equally dynamic, but in a much more contemporary way. As a collaborative wiki, it embraces openness in a way that really resonates with the original mission of Webmonkey. The team wrote about it in the new site's introduction:

When Webmonkey debuted in 1996, the site was sort of a soapbox for HotWired's engineers and designers -- a place for them to evangelize emerging web standards, rate the newest browsers and demonstrate their bleeding-edge code hacks. Those engineers also produced stacks and stacks of tutorials on all aspects of building for the web. [...]

Faced with the prospect of going back into the archives to continuously update and rewrite the older articles, we decided to turn Webmonkey into a collaborative project. It was an easy decision.

I completely agree. And I'm thankful that we have another chapter to follow what really looked like the end.

Charles Joseph Minard's visual stories

jeff — 2008-05-14T11:11:49

You may not be familiar with the name Charles Joseph Minard, but it's likely you've seen his work. He served as a civil engineer in 19th century France and developed an interest in cartography later in life. In particular, he was intrigued with showing variable data on maps - how quantities of shipped goods moved along waterways, for example, and later troop movements in military maneuvers.

Minard started drawing in his mid 60s and didn't create his most famous work until he was 80 years old. In it, he shows the progression of Napoleon and his army to Moscow and back in the campaign of 1812. This chart was renown as a masterpiece of economy and insight, and imortalized in Edward Tufte's "The Visual Display of Quantitative Information" - a virtual bible for information design since its publication 26 years ago. Tufte even sells the chart in poster form from his web site, making it a staple in designers' cubes at nearly every creative office I've ever visited.

So I was researching Minard a bit more deeply for a presentation I've been giving and came across the following quote. In it, he talks about his "carte figuratives" - a phrase referring to his particular style of data visualization mixed with geography:

"The aim of my carte figurative is ... to convey promptly to the eye the relation not given quickly by numbers requiring mental calculation."

I really like how this idea of making something visually apparent while reducing intellectual work - a sort of cognitive ergonomics. Minard was suggesting that stories and meaning can be found in any collection of data. It's up to us to uncover those stories and tell them clearly and accurately.

It's a strong lesson for much of the work we do today. Much like the patterns in data visualization, designers seek out stories and meaning when crafting interfaces. Well designed sites "convey promptly to the eye" what's possible, while doing so intuitively as to avoid "requiring mental calculation." That's not to say we should treat people as stupid. Rather, we should help them focus on what they're trying to do, rather than struggle with the means for achieving it.

Or, another way, don't make me think.

Leaving Google

jeff — 2008-05-02T15:26:28

Today marks my last day at Google and an end to a truly remarkable period in my life. If feels like just yesterday we were signing the papers, telling our friends, and announcing to the world that our team had been acquired.

We were nervous, of course. Acquisitions can be tricky - different cultures, different values, different technology platforms can all conspire against successfully merging companies. But moving to Google couldn't have been a better fit. We immediately jumped into the Analytics team and started working with them on a redesign of their product. Their openness to rethink every aspect of the app still amazes me. It's one of the most meaningful professional collaborations I've experienced in my career.

I also had the opportunity to work with Google's User Experience Team - a shockingly talented and effective group of designers and researchers. I wish I could talk about all the amazing things they're working on; all the ways they're changing how we'll connect to the world's information and to each other.

The decision to leave was a tough one. Google clearly is an amazing company to work for. After consulting with many companies during my time at Adaptive Path, it's clear that Google is like no other: they move fast, think clearly, and push strategic decisions out to the people closest to their users. But in my career, I've always swung between the big and the small and it's time for another shift.

So what's next for me? I've got a couple of small projects in the works, but mostly I'm going to take a little break, travel a bit, and catch up on some serious miles on my bike. It's been a crazy couple years ... I could use a nap.

Adaptive Path is looking for a CEO

jeff — 2008-04-15T09:29:55

There's a job opening posted over on the Adaptive Path blog that's particularly interesting - they are hiring a CEO. Considering the company's commitment to openness and transparency, it doesn't surprise me to see them conduct this search in such a public way.

I haven't had a desk there for over two years now, but I still keep very close ties. It has been exciting to watch how my former partners Peter Merholz and Jesse James Garrett have grown the company while hanging on to the core values we started with. They' will continue to serve as co-presidents of the company, but with a new emphasis on the content of their work. I like the way Peter talks about the change:

I tend to think about it as "What do you geek out to?" When I geek out, it's to experience design practice, events programming, supporting the team. I don't geek out to Running A Business. We're looking for someone who geeks out to Running A Business, and who appreciates the impact that design can bring to the world.

If you know someone (or are someone) who would excel at this opportunity, contact Bryan Mason, Adaptive Path's COO, at bryan dot mason at adaptivepath dot com.

At South by Southwest

jeff — 2008-03-08T11:58:57

Just a quick note to say that I'll be part of the Great Geek Migration down to Austin, TX this week for South by Southwest. By all accounts, it should have the largest turnout to date.

I'll be on a panel lead by Clement Mok, a designer I've long admired. We're going to be discussing the role of analytics and measurement in the creative process. This is a topic I've wrestled with in a number of different ways throughout my career - whether it be user research, technical limitations, or quantitive analysis. There never seems to be an end to the constraints designers face. Maybe that's a good thing? That's what we'll be talking about.

Also - and this is very exciting - I'll be announcing a very exciting new thing we're working on at Google. I may even be handing out invites to a private beta if you're there. So come!

The session is on Saturday at 3:30pm and is called "The Elephant in a Creative Designer's Living Room."

Indi Young's new book: "Mental Models"

jeff — 2008-02-15T09:05:41

Congratulations to Indi Young on the publication of her new book, "Mental Models: Aligning Design Strategy with Human Behavior." Indi and I were partners at Adaptive Path and worked together for a bunch of years. I'm thrilled to see her insights into research and design process captured in such an accessible and informative book. And I'm honored she asked me to write the forward, which I've included below.

"You're researching all the creativity out of this project!"

I can't tell you how many times I've heard designers, developers, and even business owners say this. It usually comes just after a project has begun, as I'm preparing for interviews with users. Designers just want to start designing, developers want to start writing code, managers want the thing to ship - so why are we spending all this time talking? And all this stuff just seems so obvious. Do we really need users to tell us what we already know?

I try to be diplomatic. "Maybe a few interviews now will save us lots of grief later," I tell them. "Think of this as insurance: Let's make sure we've got the basics right before we've designed everything and written all the code."

But no matter what I say to convince a team to do research early in their project, I never let them know my dirty little secret: I used to be just as skeptical as them.

I've always believed in a user-centered design methodology. Even early in my career, when I was journalist, we always started with the mantra "know your audience." Later in my career, I'd go to conferences and watch presentations with process diagrams - boxes representing users needs with arrows pointing to boxes representing product requirements. Intellectually, I agreed. But when I started a new project, in that intoxicating first stage when anything is possible, I'd jump straight to solutions. "Let's use Flash for this part! And over here, we'll design some awesome icons for navigation..." Our users were still important, but they were there to bear witness to how cool our designs were.

Then I met Indi Young. Indi and I were among the founding partners of Adaptive Path, a user experience consulting company that focused on research-driven design. We founded the company in the dark days of the Web industry. It was 2001. "Dot com" was a dirty word, companies were cutting their Web budgets, and projects were drying up everywhere.

It was then that "research-driven" started having real meaning to me. As Indi introduced her methodology and resulting visualizations, it became clear that she wasn't just trying to make designs better in some abstract way. Rather, her process was simple enough to resonate with anyone on a Web team. And perhaps more importantly, it would help connect Web teams to other core parts of their organizations who were skeptical of spending even another cent on their web sites.

In the end, using Indi's process, we were able to convince teams that we weren't researching all the creativity out of their projects. We were researching the risk out. And no matter how the industry is faring, that's a story people want to hear.

This book is an excellent guide to a research method firmly grounded in common sense. But don't let the simplicity of the process detract from the power of the change it can enable. Talking to users in a structured way, analyzing in a collaborative way, and diagramming with clarity can transform the way you approach the Web.

And it might just ignite your creativity.

You can buy a copy of the book directly from Rosenfeld Publishing here. Use the code "FOVEEN10" and get 10% off.

Join me at the Heather Gold Show

jeff — 2008-02-11T09:36:04

On Monday, 11 Feb 2008 (tonight!), I'll be joining Heather Gold at her show in San Francisco. I've been to the show a couple times before, and it's a blast - part standup comedy, part interview, part performance. Imagine the Leno show but people you might actually want to talk to.

The topic of the show is "My Needs / Your Needs." We'll be discussing what it means to serve someone else, and how you maintain your own identity when you're in a situation like that. This is something that's long been on my mind as a designer - how do I separate what I want from what our users want? Or, more accurately, how do I blend those things together?

Also on the show is my friend April Rinne, who specializes in microfinance in the developing world, and author Thea Hillman, who will be talking about sex parties. How's that for interesting conversation fodder?

Details are here. Hope to see you there.

Obama: Open formats for government data

jeff — 2007-11-14T18:32:43

Senator Barak Obama visited Google today and spent about 90 minutes in our cafe talking and answering questions. He was in Silicon Valley to introduce his plan for technology and innovation, network neutrality, universal broadband, and more. Wired News has coverage of everything he promised, but the thing that really stood out for me was this:

"Making government data available online in universally accessible formats to allow citizens to make use of that data to comment, derive value, and take action in their own communities."

It was only a quick reference in his prepared statements; he didn't elaborate on what this really means or what "universally accessible" formats would be available. But it was refreshing to hear someone in that position pick up on what those of us developing on the Web have been talking about for years. Government data belongs to us; open formats make access not just possible, but entirely realistic.

Stanford law professor Larry Lessig commented on this in his endorsement of the senator:

"But the big part of this is a commitment to making data about the government (as well as government data) publicly available in standard machine readable formats. The promise isn't just the naive promise that government websites will work better and reveal more. It is the really powerful promise to feed the data necessary for the Sunlights and the Maplights of the world to make government work better. Atomize (or RSS-ify) government data (votes, contributions, Members of Congress's calendars) and you enable the rest of us to make clear the economy of influence that is Washington."

Obama's offical campaign statement on technology and innovation is available (ironically) in PDF format from his web site.