MediaShift Idea Lab

LocalWiki Now Speaks Portuguese, German and Russian

philipn@gmail.com (Philip Neustrom) — Fri, 25 May 2012 09:00:00 -0500

While LocalWiki's roots are in the United States, we've seen increasing interest in starting projects all over the globe. One barrier, however, has been that our interface is entirely in English -- at least it was until now.

Thanks to the hard work of Pedro Lima and Nuno Maltez in Portugal, LocalWiki is now completely internationalized and can be easily translated into any language!

Pedro and Nuno have started a beautiful LocalWiki project for the city of Porto, Portugal: por.to. So far, they've been using their LocalWiki to collect information about the remarkable architecture around the city.

Porto residents are using LocalWiki to document their unique architecture.
This is Casa do Cinema Manuel de Oliveira. (Photo c/o Sérgio Barbosa)

Right now we've got translations for Portuguese (thanks to por.to), German (thanks to some folks in Sissach, Switzerland), and Russian.

We need lots more translations! If you'd like to help with translations, just email contact@localwiki.org.

How the AP's Overview Turns Documents Into Pictures

jstray@ap.org (Jonathan Stray) — Thu, 24 May 2012 09:00:00 -0500

Overview produces intricate visualizations of large document sets -- beautiful, but what do they mean? These visualizations are saying something about the documents, which you can interpret if you know a little about how they're plotted.

Same documents, different visualizations

There are two visualizations in the current prototype version of Overview, and both are based on document clustering.

The first is the items plot, which grew out of the proof-of-concept system we presented a year ago. Every document is a dot. Similar documents get pulled together to form visible groups, that is, clusters. All the dots start grey, but become colored as you apply tags while exploring the document set. You can click on individual documents to view them, or select a whole region of documents to see what they have in common.

Overview also has a "tree" view. Documents are still organized into clusters, but each "node" in the tree is an entire cluster, not just a single document. Also, the clusters are hierarchical, meaning that the larger clusters (higher up the tree) contain all the documents within their child clusters (lower down the tree.) The bottom of the window displays the top words and two-word phrases from the selected nodes. In this case, the selected node contains press releases discussing oil industry subsidies.

The tree view and the items plot show the same thing, just in different ways. When you select documents in one view, the same documents are selected in the other. They're two different ways of looking at the same set of clusters: hierarchically categorized, or laid out visually.

Extracting key words

All of Overview's clustering depends on grouping similar documents together, but what does that mean? Conceivably, two documents might be "similar" because they were written by the same person, talk about the same event, or came from the same place. There are as many potential categorization schemes as there are stories.

But Overview doesn't know any of this. Instead, it breaks down documents by words and short phrases. It starts by counting how many times each word appears in each document. Frequent words are "key" words. But the language processing also discounts words which appear in too many documents. This gets rid of common English words like "the" and "is," but also suppresses words which are very common in your specific set of documents. If most documents from a set of police reports contain the word "crime," Overview will mostly ignore that word.

Two documents are similar if they have overlapping sets of key words. A cluster is a set of documents that are all pretty similar to one another, and less similar to all other documents. This sounds insanely naive; after all, this word counting process throws away pretty much all of the syntactic information in the text, including word order. It can't differentiate between "police hit protesters" and "protesters hit police." But it can group together all the documents that talk about police and protesters, and that by itself is useful enough. In fact, variations of this basic algorithm, called the vector space model, are used by every search engine.

Where do those documents go?

This simple, word-based technique determines where each document is placed in the visualizations. In the Items plot, Overview tries to place similar documents close together. Collections of documents with similar words (and two-word phrases) naturally form groups, or clusters. Clusters with similar topics tend to be nearby. But that's the extent of the process; the exact angle or position of each document and cluster doesn't mean anything at all. In fact, it depends somewhat on the initial, random position of the documents, and every time you run Overview you will get a slightly different visualization -- the same clusters will show up, but possibly in different places.

The tree view finds not only clusters but sub-clusters. In the example above, the yellow branch of the tree contains the key words "fees, credit card, consumers, airlines." The left sub-branch has key words "fees, credit card, consumers" and the right branch has key words "fees, airlines, surcharges." One sub-cluster is about credit card fees, while the other is about airline fees. They've been grouped into one larger cluster because they both contain many occurrences of the word "fees".

For more information, see the discussion of our WikiLeaks visualization. Or if you're really into all the gory details -- including how Overview creates these visualizations efficiently, even for large document sets -- we've recently published a technical paper in collaboration with the University of British Columbia.

With a little practice and experimentation, you can learn how to read Overview's visualizations effectively. If you want to use Overview for your own work, it's important to get a feel for what these visualization can tell you -- and what they can't.

Public Lab's Keys to Developing Low-Cost Science Tools

adamdgriffith@gmail.com (Adam Griffith) — Tue, 22 May 2012 09:00:00 -0500

This piece was co-written by Mathew Lippincot.

The Public Laboratory for Open Technology and Science community is a massive petri dish for low-cost science tools. Our balloon-mapping tool is in its mature phase having evolved out of the agar during the 2010 Deepwater Horizon oil spill. This success was due in large part to the feedback provided by the community of tool users and consumers of tool data and their revisions to the tool. As we've broadened our development, we've asked, how can this success be replicated with other low-cost science tools still in the petri dish?

Rather than looking at specific breakthroughs in the balloon success story, we must examine why the agar in the petri dish was perfect for cultivating such a tool. What environmental conditions existed that pushed rapid development? Four clear features come to mind.

Key to Rapid Development

An urgent need for the tool emerged: The public was hungry for images, and the government had implemented a restriction on flights below 3,000 feet, preventing image acquisition.
The right tool developers were talking to users in the field: We had first-rate programming assistance and an online community of map lovers to tackle problems as they arose. As an example, the need for easier image sorting, as articulated by users in the field, led to Mapmill.org.
Advanced consumer technology was easy to leverage: The low-tech balloon, wedded to cheap, ubiquitous digital cameras produced excellent high-resolution imagery, and the balloon is free of burdensome use restrictions when compared with drones or manned aircraft.
The data is accessible to a broad audience: Real-space photography is interpretable without substantial scientific training.

The urgent need for the tool is the most important factor -- we aim to answer questions in communities that have environmental health threats. Without a use case, a tool is not a tool, it's merely a tech demo-- like a web-enabled toaster. The tool must produce data that answers a compelling question. In order to answer a community's question, the answer, and therefore the form of the data, must share a few features.

how to answer a community's question

Clear data by design: Ideally, no one should have to ask, "What does this mean?" Realistically, asking the question, "What does this mean?" shouldn't be intimidating. Where possible, data should contextualized in real space -- either captured as visual imagery or projected within real space during data acquisition. New information is more legible when couched amid a familiar situation; capturing and preserving the context of data collection should be an integral aspect of the tool and its workflow. Acronyms, jargon, and disassociated quantitative data should be minimized.

Allow users to be tool designers: Those who need data should collect it, and those who use the collection tools should be able to design them. Open source is central, but accessibility of the design process plays out in a variety of material choices -- in our balloon kit, we prefer a rubber band over the camera's shutter to re-programming the camera. Rubber bands aren't intimidating, and don't have an off-putting aura of expertise around them. Everyone suggests newer, better rubber band techniques.

Build data authority through engaging stakeholders: Authority is granted by people, not by any specific technical process. Data standards should conform to the necessary end use -- if those who need to handle data accept it, it's authoritative. If needed in court, then follow appropriate chain-of-custody requirements by engaging lawyers. If needed to compete with industrial data, engage industry professionals in how to meet their standards.

As our tool development process moves forward, the principles outlined here may change shape or be re-prioritized. But the core principle of progress with purpose will remain. A motivated community can make rapid progress, and if the environmental conditions are met, success is inevitable.

Update: This post stated the incorrect altitude for a government-implemented flight restriction. The correct number is below 3,000 feet.

Mathew Lippincott lives in Portland, Ore., where he works on design issues in sanitation through the Cloacina Project, is faculty at the Pacific Northwest College of Art, and designs civic science tools as a founding member of the Public Laboratory for Open Technology and Science.

Adam Griffith, a 2011 Knight News Challenge winner, is a co-founder of the Public Laboratory for Open Technology and Science and serves as their director of Science and Coastal Environments. He received a B.S. in Biology from Roanoke College in 1999 and was subsequently accepted to Teach for America. He taught 6th grade science in the Houston Independent School District in Texas for three years before becoming a kayak instructor and head raft guide for the Nantahala Outdoor Center. He received his M.S. in Biology in 2008 studying the native bamboo Arundinaria gigantea (rivercane) and continues to work with the Eastern Band of Cherokee Indians and community members in Western N.C. as coordinator of the Rivercane Restoration Project. Adam is currently a coastal research scientist in the Program for the Study of Developed Shorelines at Western Carolina University and is based in Asheville, N.C.

Top 12 Social Technologies from MIT Media Lab and Beyond

natematias@gmail.com (J. Nathan Matias) — Mon, 21 May 2012 09:00:00 -0500

During my work on Social Mirror, tablet tech for social checkups, I have been inspired by other amazing Media Lab social technologies. Here are 12 of the projects which I have found most inspiring, including one or two from other universities.

Did I miss a project you love? Post your favorites in the comments.

Social Empowerment through Networks

Can social checkups empower marginalized teenagers? In 2001, Leo Burd, now a researcher at the MIT Center for Civic Media, conducted several paper-based studies at Computer Clubhouse, with positive results. Leo is now an adviser on the Social Mirror software.

Leo Bonnani's SourceMap uses networks to optimize industries and encourage ethical sourcing. Using beautiful touchscreen displays, industry representatives pool information about their supply chains to plan economic development and highlight supply chain ethics (blog post here).

Big Data Network Research

We're bad at identifying the relationships that matter most to us. Nadav Aharony's Ph.D. on Social fMRI with Alex (Sandy) Pentland quantifies the difference between our perceptions and the reality of what our relationships really are. The people we think we trust are often different from the people we actually spend time with or call on the phone. In a future with better data privacy, automated behavior-mining technology such as Funf will outperform Social Mirror, which relies on the perceptions rather than behavior.

What is the economic value of a water-cooler? Ben Waber's Media Lab spinout company Sociometric Solutions tracks face-to-face interactions and extracts social signals from speech and body movements. Using "sociometric badges," Ben's company can quantify the value of social interactions towards factors like social performance. (See Alex Pentland's HBR article "The New Science of Building Great Teams.")

What do our emails reveal about our relationships? In 1995, Judith Donath created the Visual Who, one of the first network analyses of social interactions online. At the lab, Judith's Sociable Media Group created dozens of groundbreaking technologies in the social media space. Judith has moved on to Harvard's Berkman Center, where she continues fascinating research on online communities and virtual identities.

When to invest in plastics. Cesar Hidalgo applies network science to high-impact economic questions. Built together with Alexander Simoes, the Atlas of Economic Complexity breaks an economy into its component parts to tell governments what new industries provide the highest yield within the current capabilities of their infrastructure.

The Power of Online Networks

Were the revolutions tweeted? In this amazing paper and data visualization, researchers from SocialFlow, the Web Ecology Project, and Microsoft Research examine the role of Twitter in the Arab Spring. One of the project's authors, Erhardt Graeff, joins the Center for Civic Media this fall.

Can trust online turn into trust in person? Network scientist Lada Adamic (UMich) has carried out a fascinating study of trust among CouchSurfers. Want to learn social network analysis? This September, she's teaching a massive online course on Social Network Analysis.

How can networks boost the power of sharing? Andres Monroy-Hernandez intentionally designed for remix when creating the online learning community for MIT's Scratch. The network of sharing within the Scratch community is amazing.

Recommending Relationships

Can algorithms suggest good mentor relationships? Catherine Havasi and her company Luminoso use the power of networks and natural language processing to automatically pair hundreds of entrepreneurs with experienced mentors.

A new paradigm for social computing. Sep Kamvar, a luminary of Social Search and Social Datamining, is creating a new programming language, Dog, for social applications. Watch his amazing talk on Community Computing.

A version of this post also appeared on the MIT Center for Civic Media's blog.

Knight Lab Aggregates News, Tweets Around NATO Coverage

richgor@northwestern.edu (Rich Gordon) — Fri, 18 May 2012 09:00:00 -0500

World leaders, diplomats and hundreds of journalists -- as well as protesters with a wide range of grievances -- are coming to Chicago this week because of the North Atlantic Treaty Organization (NATO) summit. NATOinChicago.com, a new project from the Knight News Innovation Laboratory at Northwestern University, aims to help people make sense of what's happening.

The site has launched with two major components:

What sites are saying: An aggregation of top news sources from around the world, allowing users to see how news media in different countries are reporting on NATO and the summit.
What tweets are saying: A look at popular one- and two-word terms included in tweets about NATO.

"What sites are saying" is an application fed by feeds from 292 English-language news sources from around the world. The Lab has pre-selected a sampling of popular terms (such as "Afghanistan," "Secret Service" and "Protest"). By the weekend, users will also be able to enter their own terms. Once a term is selected, a map displays, by country, articles in major English-language news sources that include that term.

"What tweets are saying" displays popular terms being used in tweets related to NATO. It also represents graphically the relative popularity of those terms among all Twitter users and among users who indicate in their profiles that they are from Chicago. As I write this, for instance, the word "protests" is more prevalent among Chicago Twitter users than among Twitter users as a whole. Meanwhile, the opposite is true of "Afghanistan"; that word is more popular overall on Twitter than among Chicago users.

WHAT WE'RE LEARNING
The site is an experiment in live news coverage. We'd like to understand how it might help news consumers -- and journalists -- keep up with the news related to a major event that takes place over a limited period of time.

Tracking conversations on Twitter may be especially important as thousands of people plan to protest in Chicago during the NATO Summit and are using social media to organize their efforts. In addition, ordinary people in downtown Chicago during the summit and the protests may provide first-hand observations via Twitter.

"In a way, the website will give journalists an additional set of eyes and ears that they can train on an event that they are covering in person," said Owen Youngman, Knight professor of Digital Media Strategy at Medill and one of the Knight Lab's founding faculty. "Smart use of social media is not a substitute for the work of journalists on the ground, but it is an interesting supplement."

The Knight Lab will report its findings and observations of the digital conversation surrounding the NATO summit at www.natoinchicago.com/what-we-are-learning/.

The Knight Lab launched last year with a $4.2 million grant from the John S. and James L. Knight Foundation. Other Lab projects include a site that provided information about Illinois congressional primaries and a widely used tool for creating interactive timelines.

ROFLCon Attendees Get a Memes Blast From the Past

stempeck@gmail.com (Matt Stempeck) — Wed, 09 May 2012 09:00:00 -0500

It's 2012. Nerds are in, and Internet memes can actually make you famous IRL. But way back in 2000, things were different. YouTube didn't exist, and a video had to be sent around as an email attachment. (Remember RealPlayer?) Your mom yelled at you for tying up the phone line, and GeoCities plastered banners all over your creations.

At ROFLCon, the past was well-represented during a recent presentation by Eric Wu of Eric Conveys an Emotion (founded in 1998); Zblofu of Zombocom; and Jonti Picking of Weebl's Stuff. They were all online in the '90s, but things really exploded in 2000.

revisiting old memes

With Eric Conveys an Emotion, Wu shot still photos of himself conveying requested emotions, gradually growing more complicated, from sad to conveying sarcastic respect for an authority figure.

During the presentation, the crowd groaned as we revisited hamsterdance.com and saw how commercialized the once-pure GIF-overloaded page has become. But its spirit lives on at sites like omfgdogs.com.

Then we turned to Weebl, who's created an unreal amount of animated GIFs on Weebl's Stuff. We got to revisit one of the Weebl classics, Badger Badger.

Next up was Zombocom. According to BoingBoing's Rob Beschizza, the event moderator, Zombocom perfectly represents the experience on the Internet in 2000: content solely consisting of Flash animations, a permanent loading circle, and a repeating sound clip perpetually welcoming users to Zombocom.

how they got started

Wu came to the internet via AOL, like many of us in the 1990s. He started his site over a summer in college, complete with a Jackie Chan image gallery and movie reviews. A friend encouraged him to post faces and solicit emotion requests. A few months in, people who were not his friends started sending requests. He obliged. "I always said I would stop when it stopped being fun. And I haven't updated since 2006, so ..." Weebl similarly reflected on his beginnings on the Internet with AOL.

Zblofu started Zombocom as a test. At the time, SpunkyTown was a company of 50 or so people paid with venture funding to do nothing but create Flash animations.

where they are now

Wu went to Silicon Valley and worked for Yahoo for six years. He's now a general manager of an ice cream shop in Brooklyn, N.Y., where he's experimenting with a Nyancat flavor: cream cheese ice cream with a swirl of raspberry and chunks of poptart and rainbow sprinkles. He's also working on a new project: He's looking for a huge dataset of faces occupying different emotions to use as an educational tool for helping people with Asperger's and autism.

Weebl continues to make funny animations.

Zblofu makes music and works for a calendar company, which ironically, only takes six months a year. "I don't even want to go into calendars ..." Beschizza asked: "What goes into making a calendar?" "Not much," Zblofu replied.

What's changed online since 2000?

The Internet has "definitely become more social," Wu said. According to Weebl: "Back then, everyone had personal websites. But now, the destinations are always the same: Reddit, Facebook, Twitter, etc." Zblofu agreed: "There are basically only five websites out there ... I miss the individual creativity."

Beschizza sees lower barriers to entry to producing content online. The technical and financial barriers are lower -- you can just go to YouTube. This allows the culture to become more centralized in these privately owned repositories of content.

We're talking about monetization now. Back in the day, website owners sold ad space in order to pay for their own hosting costs. Independent websites found themselves in the unwelcome position of hiring an ad sales person rather than rely on pennies from AdSense.

As the conversation waned, we returned to omfgdogs.com -- because you can't go wrong with animated dogs and rainbows. As Wu said, "I like cool cats and dogs. Every night I trade cat and dog pictures."

Q&A

The following is an edited transcript of the question-and-answer session that ensued.

Q: What's the average visitor time on Zombocom?

Zblofu: A pretty long time. Some people would just leave it on 24 hours a day.

Who's the voice behind Zombocom?

Zblofu: That's the one thing I'm not allowed to talk about.

Speaking to the idea of past, present, future, do you think what you were known for then would have been successful today? If our online environment is more saturated today, how do you stand out?

Picking: I think if you knew the answer to that you'd be very rich.

Wu: Very funny, immediately understood. If I did it today, it'd be much more social. The requests would go to a big list, and the best face would win.

Why are we just finding out who's behind Zombocom?

Zblofu: [The ROFLCon organizers] are the first people who called, so I said, "Sure." My name is on the WhoIs.

Do you like pancakes?

Picking: Yes.

Which creation is your favorite?

Picking: Walk in the woods.

Did anyone ever take Zombocom seriously and think there'd eventually be a real site there?

Zblofu: I did. [There are people who study this sort of thing and have over-analyzed it and written papers on it.]

Who do you wish was on this panel?

Wu: The Homestar Runner guys. I looked them up on Wikipedia, and they're working on Yo Gabba Gabba. Superbad. Not the movie, but the site.

If I buy you a beer later, will you sing one of your Weebl songs?

Picking: If you make it a whiskey, then yeah. [Sings "Kenya." The crowd goes wild.]

Harvard student Lexi Ross contributed to this post. A longer version of this post can be found on the MIT Center for Civic Media blog.

How the Indie Audio Community Is Transforming Storytelling

jesse@metalab.harvard.edu (Jesse Shapins) — Tue, 08 May 2012 09:00:00 -0500

A version of this post also appeared in the Association of Independent's in Radio monthly AIRBlast.

I first started working with independent producer Kara Oehler in 2005. Almost a day didn't pass without her telling me about something that happened on the "AIRDaily" listserve. I'd been on listservs before, but I had never actually talked to other people about them. These conversations with Kara were my introduction to the network of more than 800 makers brought together by AIR.

At the time, I was living in New York but was partially still in Berlin, where I was completing the multimedia project The Colors of Berlin. In Brooklyn, I was enmeshed in starting UnionDocs, a documentary arts center in Williamsburg, and also engrossed in launching Yellow Arrow, a place-based storytelling project combining stickers, mobile phones and the web. A self- taught artist and designer, documentary bled through all of my work, more as a way of seeing the world and approaching artistic practice than as a specific set of rules -- let alone a single medium, despite the fact that when most people hear the word they immediately think of a film.

As I continued working with Kara and better got to know this remarkable community of indie audio producers, what struck me most was its open, heterogeneous approach to documentary. It's very hard to put a finger precisely on the reasons why this exists. But my hunch is that because audio documentary is a relatively fluid genre that operates in so many different contexts at such varied lengths (broadcast news magazines, long-form series, live performance, "listening rooms," etc.), the people engaged in it are more open to redefining and experimenting with its boundaries than those who are entrenched in more established modes of documentary (e.g., classic voiceover-driven video docs).

I am now deeply involved in collaborating with this community through Localore. Since the beginning, Zeega has been involved with AIR's Localore, a broad constellation of producer-led innovation projects embedded at local radio and television stations across the county. Over the next nine months, Zeega is working with eight of the 10 Localore projects. Each project is tackling the same set of problems, but every one will have a distinctive design and conceptual framework.

At Zeega, we love this challenge. We create projects across multiple platforms, connect digital media to physical spaces, and develop open-source tools that enable anyone to experiment with the web as a creative medium. These days, this means spending a lot of time talking about the future of interactive documentary from SXSW to the launch of the OpenDocLab at MIT, from the European i-docs Festiva to dialogues with colleagues at Frontline and PRX to a day-long session at the TriBeca Film Festival to last week's Hot Docs. During all these conversations, I've been thinking about what distinguishes Localore from other initiatives.

Here are a few speculative early stage thoughts.

Audio is driving innovative web experiences

By and large, the public discussion around the future of interactive documentary has been led by the film community (e.g. Mozilla's Popcorn project) and researchers (e.g., Mandy Rose and Sandra Gaudenzi). The most significant venue for experimentation over the past years has been the National Film Board of Canada.

While filmmakers and TV stations are a part of Localore (we are thrilled to be working with Julia Reichert and Steve Bognar!), the primary impetus for the initiative is the audio community. In my mind, this starting point offers a very unique opportunity.

One of the things we've been talking about a lot at Zeega is how important sound is to high-quality immersive experiences online. Probably my favorite interactive documentary is "Welcome to Pine Point." Made by The Goggles, a design duo formerly behind Adbusters, the visual design is certainly a major part of what makes the work so incredible. But it's the sound which grabs you right from the beginning, with the buzz of the fly playing with the loading animation, and is sustained throughout with atmospheric music, archival audio and other simple, but highly evocative effects, such as slight rustling when we see wheat in the foreground. (For more on The Goggles, see this recent Transom post).

Another recent Canadian Film Board project where audio drives the experience is "Bear 71." What's fascinating to me, is that this is basically a linear audio segment that was written for voice, around which users can explore a trove of interactive experiences at their own pace. Overall, I think this works remarkably well. The linear story sustains your attention and narrative engagement, while you control the visual unfolding.

A first wave of thinking about radio's transition to the web seemed to be simply how to make audio files that were produced for broadcast available online. Then there was podcasting, a unique digital distribution mechanism that opened up new audiences and forced a rethinking of format. Increasingly, reporters have been asked to produce text in tandem with their audio pieces, plus images or slideshows that illustrate the sound. The problem with all of these approaches is that they are looking to translate traditional audio practices for online instead of thinking about what unique characteristics drive the web and how audio can inform these.

Neither of these Canadian projects were made by people whose backgrounds were primarily in audio, but I think they illustrate the capacity for audio to be the backbone of rich, interactive experiences that could only be realized in an online environment. And I think these approaches are just the beginning of the potential for creative combinations of sound, story and interaction.

A recent project with audio that Zeega worked on at the foundation is Pejk Malinovski's "Passing Stranger," a sound-rich chronicle of poets and poetry associated with the East Village. Initially developed as an audio tour, we worked with Pejk to conceive of a web-based experience. Instead of trying to illustrate each location with tightly synced visuals, we focused our thinking on how to keep sound at the center of the experience, but use video to draw people into the embodied sense of standing on location. Explicitly rejecting a literal attempt at illustration, each location has a simple full-screen video of the contemporary location, shot from a tripod, coupled with Pejk's produced audio.

To drive home the unique relationship of sound and moving image, you can pause the audio, but the video continues playing, which mimics the experience one would have standing on the street -- you can pause an iPod, but the world outside will always keep moving. (You can see this interaction here.)

Authorship in this new space is essential

The changing nature of authorship is one of the questions discussed over and over in the interactive documentary community. In part, this is fueled by general enthusiasm throughout the media industry for user-generated content and the (I would say misplaced) notion that now that the tools of recording are so widely available, everyone can be a producer and share their story. One thing I've learned from the audio documentary community over the years is that good storytelling is very, very hard -- in any medium.

Because radio can't rely on images to carry a narrative or evoke a mood, radio storytellers tend to be some of the most exceptional at crafting poignant stories and refusing to let a single moment of potential boredom creep into a narrative. In my experience, This American Life and Radiolab are two of the most successful examples in any media form of tying quality reporting to captivating, surprising personal stories. And it's no coincidence, in my mind, that The Moth is a part of the public radio ecosystem and not TV or film.

This expertise in quality, short-form storytelling will be a huge advantage for the radio
community as it makes the transition to creatively combining broadcast and the web from the beginning of projects. This editorial rigor translates not only into the audio components of interactive projects, but to works as a whole.

Another thing we've been talking a lot about at Zeega is the notion of "editing interactive" -- in other words, submitting interface ideas to the same intense editorial process that a story would receive. This forces us to treat interfaces as forms of time-based media, imagining in great detail the sequence of a user's experience. And it requires giving special attention to moments of transition (a classic editorial challenge), which in an interactive context can be addressed in many ways, such as through subtle animations from one click to the next or by atmospheric sound that persists through scenes.

Participatory experiences are also authored. Audiences may be able to participate in new and powerful ways, but they can't necessarily craft extraordinary stories on their own. A great example of this editorial rigor applied to interactive experience is Chris Milk's "Johnny Cash Project." Not a conventional documentary, the starting point is a web-based music video for "Ain't No Grave," Cash's last studio recording. The site has a very simple structure -- you start by watching the video. Music begins in tandem with evocative, grayscale hand-drawn images. Below the main player, there is an unconventional timeline composed of a moving grid of tiny thumbnails. If you click to explore this, the video stops on that frame, and you realize that each of the little images is a different drawing of that frame. And you are then prompted to draw your own interpretation of the frame.

But instead of a totally open format, the site has a built-in drawing feature that provides you a reference image from the original video on top of which to draw. You are provided highly constrained tools (e.g., it must grayscale) for drawing. Instead of feeling limited, these constraints are incredibly enabling, as it's fun and simple. The rule set focuses a contributor's creative energy. After drawing the frame, you can submit it to be included in the website, and your name is added to the credits of the overall project.

Interactive documentary tied to broadcast fosters the unexpected

One of the major challenges facing interactive documentaries is distribution. The Canadian Film Board's work has the benefit of being showcased on a site that receives significant traffic and promotion by the government. Many interactive documentary projects, though, are independent initiatives (even if they have broadcaster support), and to my knowledge, no major projects recently have been tied directly to broadcast series.

A requirement for all of the Localore projects is that they're to be developed in the context of a local public media station. I think this is brilliant and one more quality that sets the initiative apart from others internationally. The potential for this broadcast element is tremendous. It ensures a significant initial audience, enabling novel forms of participatory documentary.

My first experience with tying broadcast to digital was Mapping Main Street, a project made with Kara, Ann Heppermann, and James Burns as part of MQ2, AIR's first-generation innovation project preceding Localore.

What I loved about creating a project like this is that you initiate something open-ended, and you actually have no idea what people will do. And you can learn from what people do -- and the surprising things they do can become the center of a project. For example, Amy Fichter (aka xenia elizabeth) heard the series on her local station, along with the prompt to contribute photos of Main Streets nearby. She proceeded to use the project as an impetus to travel around her region of western Wisconsin, taking photos and talking to people in towns that she had often passed through but in which she had never stopped. Amy documented so many Main Streets that when a gallery owner ran into her taking photos on the gallery's Main Street, the ensuing conversation led to an exhibition of her Mapping Main Street photos.

While it's possible to create sustained engagement with a project that is not tied to broadcast, I think broadcast makes a major difference in generating an initial wave of participation and having a constant connection to a community of contributors. This is a huge advantage for the radio community.

We hear over and over about the death of local news, but local public media stations are unique beacons, where audience is growing and the business model is not driven by advertising, but instead by people invested in content and supporting their local communities.

Jonathan Zittrain Takes the Stage at ROFLCon

natematias@gmail.com (J. Nathan Matias) — Fri, 04 May 2012 14:48:54 -0500

Today with MIT Civic Media Center's Matt Stempeck and Stephen Suen, I'm live-blogging ROFLCon, a conference for things and people who are famous on the Internet. The livenote index is here.

Christina Xu, the event organizer, starts off ROFLCon to cheers. It's an amazingly packed venue. "One out of eight people in this room have done something crazy on the Internet," she says.

Zittrain on memes and society

Jonathan Zittrain is an Internet phenomenon. Emerging from humble beginnings as a longtime CompuServe forum sysop, he is now professor of law at Harvard Law School where he co-founded the Berkman Center for Internet and Society.

He starts by saying that fame can be tricky: "Just before the talk, someone came up to me and said, 'are you the huh guy? I thought you were the huh guy! I'm not that famous. I can aspire. In this room is the engine that makes the Internet sing ... Who's minding the store? Is this going to be a day without memes?"

"Where's Tron Guy?" asks Zittrain. Tron Guy, in full costume, raises his hand, and the room bursts into applause.

Zittrain says he isn't sure if he's one of these "Internet ROFL people" -- hence the tie. It's hard to explain what you're doing this weekend to friends and family who are not part of this tribe, he quips.

But he does have some background in the Internet. He shows us a picture of him using a Texas Instruments home computer with a 300-baud modem, with the obligatory model rockets, and the Webster's Collegiate Thesaurus -- just because you might run out of words.

Zittrain used to work for CompuServe and also got involved in politics. He threw his weight behind Mondale/Ferraro 1984. "At least I carried Minnesota," he says. "And the District of Columbia." When he wasn't doing those things, he was usually spending time stuffed inside a locker. "Whatever that does not stuff you so that you die, makes you stronger," he noted.

Zittrain thinks the image of a nerd stuffed in a locker helps us understand memes -- the dramatic moment of pathos.

"They're all crazy; I'm normal ... they're bad, and we're good. And here's to us for being good," he says. But that opens us up to the charge that this culture, the Internet, is not real life, and is rather a form of retreat. At the base of a lot of memes is some authentic, unguarded voluntary moment, Zittrain says. There's artifice around it, but there is often something authentic beneath it. That's not always the case -- consider Dramatic Hamster. Sometimes a hamster isn't a hamster. But there are other times that it's striking closer to a certain chord.

Wires can be crossed when this culture is commercialized. The nerds struck back against Hot Topic when they produced a T-shirt of Rage Guy.

unstaged authenticity

There's something about commercialization which is always at arm's length of Internet culture. Zittrain talks to us about the most recent Calgary Comic Con, where they invited the entire cast of Star Trek: The Next Generation. Going to the cons involves waiting in lines to get your photo with the cast. It has an Apple Commercial 1984 feel to it -- take a photo with the cast, you cannot touch the cast. He tells us about one of the least proud moments of The Oatmeal, a contest for advice features. There appears to be a negative attitude towards those who intentionally try to "engineer" a meme. People don't like being prompted -- it feels like trying too hard, feels inorganic.

We like unstaged authenticity, like Disaster Girl, who grins deviously as a house burns to the ground behind her. She rather enjoys the attention, and we are pleased to see her embrace her inadvertent success, but there are still lines that you can cross. The point at which you're running your own network and have a store-- maybe not.

Internet Fame is like winning the lottery -- it seems good until someone gets killed. What better example of this ambivalence than Star Wars Kid? So far as he knew, this was an exercise that would be completely private. He didn't realize that when he turned the camcorder in at school that it would be posted to YouTube. Jonathan shows us the video of the the Matrix Version of Star Wars Kid. In Wikipedia, there's a debate on the talk page on whether or not it is right for Wikipedia, the knowledge repository of record for humanity, to include his name in the page. Ultimately, they decided not to name him, despite the fact that the mainstream media has done it several times. And people on Wikipedia fell into line-- upholding the process with which they disagreed.

Can we build an infrastructure of meme propagation that respects people's preferences. He shows us one of the Awkward Family photo sites, with an image that says, "Image removed at request of owner." There are enough yuks to go around, so why not take down private content when someone asks us to?

Jonathan would love to see an infrastructure built native to the web which makes it possible for people to opt out of the celebrity of being a meme. This isn't DRM, but maybe something like robots.txt (a directive that tells web crawlers like Google which subdirectories not to index). Search companies respect robots.txt. No Internet organization created this. But people and companies respect it anyway-- a way to say, "Do you mind?" This is often used with court documents. How could we build this into our technology and our culture? One guy made a T-shirt that reads, "I do not agree to the publication of this photo."

In short, how can we enjoy the culture of Lulz which also respecting people's wishes?

A longer version of this post can be found on the MIT Center for Civic Media blog.

How We Got Here: The Road to Public Lab's Map Project

stewartblong@gmail.com (Stewart Long) — Thu, 03 May 2012 09:00:00 -0500

Last week, Public Laboratory announced that public domain maps are now starting to show up on Google Earth and Google Maps. But how did the projects get there? Here's a timeline of a Public Laboratory map project.

Making a map

Public Laboratory projects take a community-based approach to making maps that differs depending on where you are and the reason you want to create a map. People map areas for a number of reasons, including because there's a need to monitor an area of environmental concern, a dynamic event is happening that there's a desire to capture, or you cannot find adequate aerial image data. Before going out to map, preparing for fieldwork starts with the Public Lab map tools page, where you can discover what type of equipment to use and how to safely use it. Multiple research notes on how to do things such as setting up a dual camera rig and stabilizing the camera with a picavet can help with specific problems, but there are also hundreds of people in the online Public Lab community of mapmakers, sharing tips and experiences on the site.

Upon return

After the mapping flight, the map making begins with backing up the images and sorting through the set, making a subset for map production. Depending on the time in the air, there will be hundreds and sometimes thousands of individual images. Depending on the area of interest, you can hone in on which images will be used in creating the map. Assuming the flight was at a steady altitude, the images that you want to select are the sharpest ones that are vertically oriented. If you have many images for the same area, pick the best one, but also pick overlapping images so that there is plenty of overlap among the different images in the next step.

Public Laboratory's MapMill.

Images can be sorted locally or online. Public Laboratory created an online tool where a group can do collaborative selection. MapMill.org is a web-based image sorting and ranking tool where multiple users can sort through a large dataset simultaneously.

Map production

With a smaller set of the best images on hand, the images can be dynamically placed on the map in a process known as georectification. After all the images have been added to the map, the project is exported. The MapKnitter export tool does all of the geographic information systems crunching behind the scenes with the geospatial data abstraction library (gdal.org) and produces a GeoTIFF map file. The GeoTIFF format is a public domain metadata standard that embeds geographic information into the image TIFF file. At this point, the map is now in an interchangeable format that can be easily distributed.

Public Laboratory MapKnitter web-based aerial image map production tool.

Public Laboratory Map Archive

Public Lab hosts its own map data archive for storing and sharing finished map projects. Each map in the archive has a "map details page" that hosts details such as: title, date, place, location, resolution, field map maker, field notes, cartographer, ground images, oblique images from the flight, and comments from website users. The map participants choose whether to publish the map as Public Domain, Creative Commons Attribution ShareAlike, Creative Commons Attribution, or Creative Commons Attribution Non-Commercial.

Public Laboratory Occupy Oakland, November 2, 2011 -- General Strike map in Google Earth.

Maps are viewable on the archive itself, and you can subscribe to it as an RSS feed. However, it's also a place for distribution of the data. As we announced last week, Google Earth has started licensing our public domain maps. Google Earth plans to continue to publish public domain maps from the Public Lab Archive a few times a year.

It's quite exciting to see these Public Labs maps go online with a ubiquitous data provider such as Google. We look forward to more people participating in this activity, and more publishing of public domain data.

Google published some of the maps to Google Maps as well as Google Earth, which makes those maps widely accessible in the web browser and on mobile applications that use Google Maps.

Mobile Security Survival Guide Helps Journalists Understand Wireless Risks

melissaulbricht@gmail.com (Melissa Ulbricht) — Tue, 01 May 2012 09:00:00 -0500

The Mobile Security Survival Guide for Journalists from SaferMobile helps reporters better understand the risks inherent in the use of mobile technology. The guide covers both local journalists and those on assignment in another country.

As someone working with sensitive information, mobile communications are inherently insecure and expose journalists working in sensitive environments to risks that aren't easy to detect or overcome. This guide is designed to help navigate these challenges. (It should be noted that this guide does not guarantee safety. Rather, it's a foundational resource to understand and minimize the risks of mobile communication in the field.)

The Mobile Security Survival Guide is written with the workflow of a journalist in mind.

Here's a sampling of what the guide offers:

1. Mobile Network Awareness

What does your mobile use say about you?

Did you know? Activity on your phone creates a data trail that is logged on the mobile network, from placing or receiving a call, to sending a message, browsing the web, or just being connected and ready to receive communication. Identifying information logged on the network about you and the people you contact include your IMEI number (the unique handset identifier), the IMSI (the unique SIM card identifier), the time and duration of voice calls, SMS, and the photos or video you take while reporting.

Tip: Have an alternative in case you're unable to access one or more services. Carry SIM cards for other mobile network operators, and if possible, carry more than one phone. In some cases, only an out-of-country operator may have roaming service. Have a backup plan agreed upon in advance with sources and colleagues if you suspect that your specific line or the entire mobile service may be disrupted.

2. Preparing for Assignment

Assess your digital risks and prepare your phone.

You have been given your assignment. You may be traveling to cover a story, or you may reside in a given country. Either way, when it comes to your mobile communications, you should take some precautions and plan ahead. Here are sample tips on practicing on your phone.

Tip: Know your phone. It may sound obvious, but be sure to know how to work your phone. For example, become familiar with how your camera works and how to control its options. The flash of your camera or the sound when you click Capture may draw unwanted attention to you. Take the time to pre-set functions to avoid getting noticed by others.

Practice on your mobile keypad. Learn how to operate your phone without looking at it. You may need to know how to type a text message (SMS) without looking at your mobile keypad or perhaps while it's hidden in your pocket. Plan your shortcuts in advance, and be able to access needed applications without looking at the phone, too.

3. Reporting/In the Field

Have a plan of action ready when you're talking to sources and conducting interviews; checking in with your newsroom; or using your phone in emergency situations.

Tip: Use codes for SMS if needed. If necessary, use prearranged codes to communicate sensitive information to your contacts. Change your codes regularly, and make sure your system incorporates a way to let others know when you think the code may have been broken. To practice, try the code-making exercise in the SaferMobile training guide. Avoid words that could be considered "high profile" or inflammatory if you suspect keyword filtering of SMS is taking place. However, remember that information about the recipient of text messages (i.e., that person's number and other information) is still logged by the network operator. Take care not to put any sensitive sources at risk with your communications.

4. Filing the Story

How do you safely send updates, news bursts, or multimedia content from the field?

Tip: Disable MMS if not needed. Unless you really need MMS functionality on your phone, check the settings to see if it can be disabled. MMS, like SMS, can be intercepted and viewed by the network operator. Delivery rates for MMS tend to be lower as well, making this a more unreliable form of communications as well as a more insecure one. MMS can be an attack vector in another way: There have been cases where it has been used to sneak mobile viruses and malicious mobile software into unsuspecting phones.

5. Social Media

Make safer use of social media to follow news, connect with sources, share breaking stories, and promote your work.

Tip: Set a strong password and keep it safe. Keep your account details safe. Check out this guide from SaferMobile for more on setting a strong password.

While a strong password won't always protect you, it adds an important extra layer of security. Despite the risks involved, Twitter and other social media platforms are very powerful tools that can help report news from the field, especially when events are unfolding quickly, or you have limited options or decreased capacity and staff.

Tip: Avoid older browsers and browse securely. Your phone's web browser needs to support HTTPS. Avoid older browsers, particularly Opera Mini Basic 3 and below. All your communication with Facebook, YouTube, and Twitter should display a lock icon to indicate secure mobile browsing, and a web address starting with https:// rather than http://.

Check out the complete Mobile Security Survival Guide here.

How to Contribute to OpenStreetMap and Grow the Open Geodata Set

bonnie@developmentseed.org (Bonnie Bogle) — Mon, 30 Apr 2012 09:00:00 -0500

Hundreds of delegates from government, civil society, and business gathered in Brasilia recently for the first Open Government Partnership meetings since the inception of this initiative. Transparency, accountability, and open data as fundamental building blocks of a new, open form of government were the main issues debated. With the advent of these meetings, we took the opportunity to expand an open data set by adding street names to OpenStreetMap.

Getting ready to survey the Cruzeiro neighborhood in Brasilia.

OpenStreetMap, sometimes dubbed the "Wikipedia of maps," is an open geospatial database. Anyone can go to openstreetmap.org, create an account, and add to the world map. The accessibility of this form of contribution, paired with the openness of its common data repository, holds a powerful promise of commoditized geographic data.

As this data repository evolves, along with corresponding tools, many more people gain access to geospatial analysis and publishing -- which previously was limited to a select few.

When Steve Coast founded OpenStreetMap in 2004, the proposition to go out and crowdsource a map of the world must have sounded ludicrous to most. After pivotal growth in 2008 and the widely publicized rallying around mapping Haiti in 2010, the OpenStreetMap community has proven how incredibly powerful a free-floating network of contributors can be. There are more than 500,000 OpenStreetMap contributors today. About 3 percent (that's still a whopping 15,000 people) contribute a majority of the data, with roughly 1,300 contributors joining each week. Around the time when Foursquare switched to OpenStreetMap and Apple began using OpenStreetMap data in iPhoto, new contributors jumped to about 2,300 per month.

As the OpenGovernment Partnership meetings took place, we wanted to show people how easy it is to contribute to OpenStreetMap. So two days before the meetings kicked off, we invited attendees to join us for a mapping party, where we walked and drove around neighborhoods surveying street names and points of interest. This is just one technique for contributing to OpenStreetMap, one that is quite simple and fun.

Here's a rundown of the most common ways people add data to OpenStreetMap.

Getting started

It takes two minutes to get started with contributing to OpenStreetMap. First, create a user account on openstreetmap.org. You can then immediately zoom to your neighborhood, hit the edit button, and get to work. We recommend that you also download the JOSM editor, which is needed for more in-depth editing.

Once you start JOSM, you can download an area of OpenStreetMap data, edit it, and then upload it. Whatever you do, it's crucial to add a descriptive commit message when uploading -- this is very helpful for other contributors to out figure the intent and context of an edit. Common first edits are adding street names to unnamed roads, fixing typos, and adding points of interest like a hospital or a gas station. Keep in mind that any information you add to OpenStreetMap must be observed fact or taken from data in the public domain -- so, for instance, copying street names from Google is a big no-no.

Satellite tracing and GPS data

JOSM allows for quick tracing of satellite images. You can simply turn on a satellite layer and start drawing the outlines of features that can be found there such as streets, building foot prints, rivers, and forests. Using satellite imagery is a great way to create coverage fast. We've blogged before about how to do this. Here's a look at our progress tracing Brasilia in preparation for the OGP meetings:

OpenStreetMap contributions in Brasilia between April 5 and April 12.

In places where good satellite imagery isn't available, a GPS tracker goes a long way. OpenStreetMap offers a good comparison of GPS units. Whichever device you use, the basics are the same -- you track an area by driving or walking around and later load the data into JOSM, where you can clean it up, classify it, and upload it into OpenStreetMap.

Synchronizing your camera with the GPS unit.

Walking papers

For our survey in Brasilia, we used walking papers, which are simple printouts of OpenStreetMap that let you jot down notes on paper. This is a great tool for on-the-ground surveys to gather street names and points of interest. It's as simple as you'd imagine. You walk or drive around a neighborhood and write up information that you see that's missing in OpenStreetMap. Check out our report of our efforts doing this in Brasilia on our blog.

Walking papers for Brasilia.

At the International Journalism Festival: Can Data Journalism Save Newsrooms?

lucy.chambers@okfn.org (Lucy Chambers) — Fri, 27 Apr 2012 14:54:48 -0500

PERUGIA, Italy -- Here at the International Journalism Festival the launch of three large initiatives have generated a lot of the buzz around the topic of data journalism.

The School of Data Journalism, organized by the European Journalism Centre and the Open Knowledge Foundation, is composed of three panels and five workshops and dives into some of the key issues that media organizations are currently considering: "Is it worth my while starting out trying to do data journalism?", "Will data journalism make us money?", "How do you get data that you can search, filter and analyze with a computer?" and "How do I make data stories sexy?"

In addition, the 58 nominations for the Data Journalism Awards (DJA) were announced. DJA is the first international competition that recognizes and showcases the great work done in data journalism. Prizes are awarded for data-driven applications, investigations, and storytelling through visualizations. It's hoped that these awards will encourage more news organizations to embark on more ambitious data projects and alleviate the "loneliness in the newsroom" which some data journalists experience when their colleagues don't understand what they do. The six winners will be announced May 31.

And on Saturday, the Data Journalism Handbook will be launched. The handbook was born at the Mozilla Festival in November. It's a collection of tips, anecdotes and case studies from more than 70 leading data journalists and data wranglers, including contributions from The New York Times, Zeit Online, the BBC, the Guardian and many more. The book will be an open educational resource with key lessons a beginner data journalist should know. You can see a chapter overview of the handbook here and an excerpt from the first chapter here. A free version will be available online at datajournalismhandbook.org, and an e-book and print version will soon be published by O'Reilly Media.

So what is data journalism?

The School of Data Journalism, a series of panel discussions and workshops at the festival, was led by leading practitioners from all over the world and aimed to show participants what data journalists can do and why they should take the plunge and learn new skills.

The definition of data journalism varies depending on whom you ask. For some journalists, it's simply the courage to tackle sometimes huge and messy datasets. For others, it's being transparent and open about "showing the working" behind their conclusions, backing up their stories with facts and numbers where one might previously have only evidenced their point with "he said/she said." For others, it's a new way of presenting data through visualizations and interactive news applications; news is no longer simply static words on a page.

Increasingly, though, many are coming to realize that data journalism is a set of skills, involving new methods for acquiring, analyzing and working with data which simply weren't computationally feasible before. In an age that is positively drowning in data, we need more data journalists who typically have better storytelling skills than statisticians and can act as translators of complex datasets for the benefit of the public.

As activist and author Heather Brooke put it in the "Information wants to be free" workshop, data journalism is a misnomer -- one doesn't say "telephone journalism" if you contact your sources via telephone; journalists have to use data to do their job well.

Guerrilla Tactics: how to get started with Data Journalism

In the first panel of the school, "From Computer Assisted Reporting to Data Journalism," Pulitzer Prize winners Sarah Cohen and Steve Doig, highlighted their experiences working in the United States, where the notion of Computer Assisted Reporting (CAR) has been around for several decades -- far longer than the budding data journalism scene here in Europe.

They described their experiences learning how to use tools and techniques -- unfamiliar to journalists but popular in other disciplines such as social science and history -- to stay at the cutting edge of journalism. They also described the "guerrilla tactics" they initially had to use to get their work into print. "If you produce an amazing visualization, your editor is going to find a way to get it published," Cohen said, adding that it's far easier to show someone what data journalism is than to explain what it is.

Next up, Aron Pilhofer described his journey to data journalism at the New York Times. He said it came from a feeling of frustration with the inefficiency of working practices and tools. This sentiment resonated strongly with the other panelists -- a common complaint concerned individual journalists holding onto their data, producing datasets that only they could understand, instead of resources that could be built on and expanded by others on their teams.

On the same panel, Elisabetta Tola of formicablu and Simon Rogers of the Guardian gave a European perspective on data journalism. Rogers demonstrated how the Guardian Datablog's interactive maps of the U.K. riots helped disband false statements by the government that the "riots were not about poverty." Tola then explained some of the more basic problems facing wannabe data journalists in Italy, some of whom would be lucky to get data even on paper, as it's common for officials to simply dictate the numbers to journalists.

The second panel, "How can data journalism save your newsroom?", examined perspectives and business models for data journalism, and attempted to answer the question: "Is it worth it?"

Caelainn Barr of Citywire urged journalists not to consider data journalism as a fix-all for any problem in the newsroom. She warned that editors are unlikely to be considerate and give you more time just because you're using complex data or working hard to present it better. Barr said journalists are constantly playing a game of catchup; advertisers are moving elsewhere; and journalists have less time to produce their stories and are struggling to keep up. All of this means journalists have to be more agile and learn to do things more efficiently.

To solve this problem, Pilhofer said, the New York Times has built resources that live on for future stories, allowing both journalists and the interactive news team to spring into action as soon as a related story breaks.

"What is the simplest thing you can do to start with data journalism?" ProPublica's Dan Nguyen asked rhetorically. "Keep your notes in a spreadsheet." He said often, the skills required to find stories involve sorting, grouping and averaging the data. With skills this simple, can newsrooms really afford not to teach them to their journalists?

The Future of Journalism is Bold

What does the future look like for data journalism? "Data journalism is just becoming journalism," said the Guardian's Rogers -- which was possibly the most encouraging statement from any of the panelists here.

Data journalism is no longer limited to only those who can afford to pay $900 for a piece of visualization software. Now incredibly powerful, open-source solutions are available.

However, a change in culture will be needed to get more journalists into the fold. As Tola explained, collaboration is key, both journalist-journalist and journalist-coder collaboration. As Wired Italia's Guido Romeo put it, "Journalism is a one-man band. Data journalism is clearly not."

Could Italy be a land of opportunity for data journalism? The enthusiasm with which the workshops were met gave the impression that he who dares first will have a serious competitive advantage.

The workshops will continue over the next couple of days, and many have spaces open. Any budding data journalists? Join us!

ScraperWiki: How Legal Is Scraping?

aine@scraperwiki.org (Francis Irving) — Fri, 27 Apr 2012 09:00:00 -0500

Lots of people, when they hear about ScraperWiki, ask, "Is scraping legal? How can you build a business off that?" Usually, they follow up by saying, "We do it in our company, but we would never tell anyone."

This is strange to us, as we have come from a world of good scraping: taking government data and making it easier for people to use for things that benefit all of society. We're in favor of that kind of scraping.

It's obviously a spectrum. At the other extreme, the most evil scraping would be to steal content that somebody else sells, and then to republish it at harm to their business. We're against that kind of scraping.

It's not scraping itself which is good or bad, or legal or illegal, but the circumstances in which you're doing it.

We've written up in full our policy about the legality; it's in our FAQ under "What's your policy on what's legal to scrape?" -- lots of details about robots.txt and take-down notices, and what is our and your legal responsibility.

Getting data into a data hub

Finally, ScraperWiki isn't just about scraping.

We're a data hub, and you need to get data into a data hub. As well as scraping, lots of people make API calls to do that on ScraperWiki, or download their own files from their own servers.

This is much more profound than it sounds -- when you are using data for a new purpose, even if it is already structured, you still need to get it and convert it to your new needs. How you do that is a detail that depends on the circumstances.

The difference between parsing HTML web pages, and using a JSON REST API is surprisingly small. As an example, ScraperWiki's Thomas Levine scraped EventBrite even though it has an API, because it was easier at the time for him.

Whether you're using Nokogiri or Nestful, what matters is getting the data, and converting it into a form where it can do something useful for the world -- and doing that legally.

A version of this post first appeared on the ScraperWiki blog.

Image courtesy of Flickr user bloomsberries.

Knight-Mozilla OpenNews Sponsors Dual Journalism Hack Days

dansinker@gmail.com (Dan Sinker) — Tue, 24 Apr 2012 09:00:00 -0500

There's no better example of the global scale of the Knight-Mozilla OpenNews project than the dualing hack days we recently sponsored in New York City and Buenos Aires.

In New York, we gave money for travel scholarships to bring top-notch developers to town to take part in the Wall Street Journal's Data Transparency Weekend, which brought more than 100 developers and privacy experts to town to create tools to help people see and control their personal data online. The "hackathon" grew out of the Wall Street Journal's excellent ongoing series that looks at how your online footprint is being used by corporations.

The three-day event (documented extensively here, here, and here) resulted in code for almost 30 different projects with winners in "Scanning," "Education," and "Control" tracks.

Five-thousand miles to the south, we sponsored the Hacks/Hackers Buenos Aires ShowTimeLine Hackathon, which brought 45 developers together to work on making new timeline-based visualization tools. The OpenNews sponsorship went to hosting the hack day, as well as a small amount of seed money to keep projects going afterward.

The team of developers and journalists in Buenos Aires took a series of different approaches to displaying data over time, from automatic data-and-date extraction from documents, to translating pre-existing timeline libraries into Spanish, and more.

These are exactly the kind of topic-driven code-based events that we're looking to help sponsor at OpenNews. If you've got an idea brewing for a journalism hack day, we'd love to hear about it. Let's work together to make this year the year of journalism code.

A version of this post first appeared here.

How Ushahidi Deals With Data Hugging Disorder

juliana@ushahidi.com (Juliana Rotich) — Fri, 20 Apr 2012 09:00:00 -0500

At Ushahidi, we have interacted with various organizations around the world, and the key thing we remember from reaching out to some NGOs (non-governmental organizations) in Kenya is that we faced a lot of resistance when we began in 2008, with organizations not willing to share data which was often in PDFs and not in machine-readable format.

This was especially problematic as we were crowdsourcing information about the events that happened that year in Kenya. Our partners in other countries have had similar challenges in gathering relevant and useful data that is locked away in cabinets, yet was paid for by taxpayers. The progress in the Gov 2.0 and open data space around the world has greatly encouraged our team and community.

When you've had to deal with data hugging disorder of NGOs, open data is a welcome antidote and opportunity. Our role at Ushahidi is to provide software to help collect data, and visualize the near real-time information that's relevant for citizens. The following are some thoughts from our team and what I had hoped to share at OGP in Brazil.

Government Data is important

It is often comprehensive - It covers the entire country. For example, a national census covers an entire country, so it has a large sample, whereas other questionnaires have a smaller sample.
Verified - Government data is "clean" data; it has been verified -- for example, the number of schools in a particular region. Crowdsourcing projects done by government can be quite dependable. (Read this example of how Crowdmap was used by the Ministry of Agriculture in Afghanistan to collect commodity prices.)
Official - government data forms the basis of government decision making and policy. If you want to influence government policy and interventions, it needs to be based on official data.
Expensive - Government data because it is comprehensive and verified is expensive to collect -- this expense is covered by the taxpayer.

Platforms are important

Libraries were built before people could read. Libraries drove the demand for literacy. Therefore, it makes sense that data and data platforms exist before before citizens have become literate in data. As David Eaves wrote in the Open Knowledge Foundation blog:

It is worth remembering: We didn't build libraries for an already literate citizenry. We built libraries to help citizens become literate. Today we build open data portals not because we have a data or public policy literate citizenry, we build them so that citizens may become literate in data, visualization, coding and public policy.

Some countries like Kenya now have the data, and now open-source platforms available not just for Kenya but worldwide. What are we missing?

Platforms like Ushahidi are like fertile land, and having open data is like having good seeds. (Good data equals very good seeds.) But fertile land and seeds are not much without people and actions on that very land. We often speak about technology being 10 percent of what needs to go into a deployment project -- the rest is often partnership, hard work and, most of all, community. Ordinary citizens can be farmers of the land; we need to get ordinary citizens involved at the heart of open government for it to powerful.

Ushahidi's role

Accessible data: The ownership debate has been settled as we agree government data belongs to the citizens. However, ownership is useless without access. If you own a car that you do not have access to, that car is useless to you. In the same way, if our citizens own data they have no access to, it's useless to them. Ownership is exercised through access. Ushahidi makes data accessible -- our technology "meets you where you are." No new devices are needed to interact with the data.

Digestible data: Is Africa overpopulated? If Africa is overpopulated or risks overpopulation, what intervention should we employ? Some have suggested sterilization. However, the data shows us that the more education a woman has, the less babies she has. Isn't a better intervention increasing education opportunities for women? This intervention also has numerous additional advantages for a country -- more educated people are usually more economically productive.

Drive demand for relevant data: Governments are frustrated that the data they have released is not being used. Is this because data release is driven mainly by the supply side, not the demand side -- governments release what they want to release, not what is wanted? How do we identify data that will be useful to the grassroots? We can crowdsource demand for data. For example: The National Taxpayer Alliance in Kenya has shown that when communities demand and receive relevant data, they become more engaged and empowered. There are rural communities suing MPs for misusing constituency development funds. They knew the funds were misused because of the availability of relevant data.

Closing the feedback loop: The key to behavioral change lies in feedback loops. These are very powerful, as exemplified by the incredible success of platforms like Facebook, which are dashboards of our social lives and that of our networks. What if we had a dashboard of accountability and transparency for the government? How about a way to find out if the services funded and promised for the public were indeed delivered and the service level of said services? For example: The concept of Huduma in Kenya, showed an early prototype of what such a dashboard would look like. We are working on more ways of using the Ushahidi platform to provide for this specific use case. Partnership announcements will be made in due course about this.

All this, To what end? Efficiency and change

If we as citizens can point out what is broken, and if the governments can be responsive to the various problems there are, we can perhaps see a delta in corruption and service provision.

Our role at Ushahidi is making sure there's no lack of technology to address citizen's concerns. Citizens can also be empowered to assist each other if the data is provided in an open way.

Open Data leading to Open Government

It takes the following to bridge open data and open government:

Community building - Co-working spaces allow policy makers, developers and civic hackers to congregate, have conversations, and build together. Examples are places like the iHub in Kenya, Bongo Hive in Zambia, and Code For America meetups in San Fransisco, just to name a few.
Information gathering and sharing - Crowdsourcing plus traditional methods give not only static data but a near real-time view of what's going on on the ground.
Infrastructure sharing - Build capacity once, reuse many times -- e.g., Crowdmap.
Capacity building - If it works in Africa, it can work anywhere. Developing countries have a particularly timely opportunity of building an ecosystem that is responsive to citizens and can help to leapfrog by taking open data, adding real-time views, and most of all, acting upon that data to change the status quo.
Commitment from government - We can learn from Chicago (a city with a history of graft and fraud), where current CTO John Tolva and Mayor Rahm Emmanuel have been releasing high-value data sets, running hackathons, and putting up performance dashboards. The narrative of Chicago is changing to one of a startup haven! What if we could do that for cities with the goal of making smart cities truly smart from the ground up? At the very least, surfacing the real-time view of conditions on the ground, from traffic, energy, environment and other information that can be useful for urban planners and policy makers. Our city master plans need a dose of real-time information so we can build for our future and not for our past.
Always including local context and collaboration in the building, implementation and engagement with citizens.

Would love to hear from you about how Ushahidi can continue to partner with you, your organization or community to provide tools for processing data easily and, most importantly, collaboratively.

Daudi Were, programs director for Ushahidi, contributed to this post.

A longer version of this story can be found on Ushahidi's blog.