O'Reilly Radar - Insight, analysis, and research about emerging technologies.

Current activities at the Electronic Information Privacy Center

2010-03-19T20:49:50Z

When Marc Rotenberg founded the Electronic
Information Privacy Center in 1994, I doubt he realized how fast
their scope would swell as more and more of our lives became digitized
and networked. Now it seems like everything that happens in society
has an electronic component and a privacy component. I had the chance
to drop in to their office on Monday and heard about the
front-burner items they're working on.

Whole-body imaging in airports, a very hot issue right now. While
Americans push back against it, the European Union has to vote on it
soon.

The Smart Grid: a massive upgrade planned for the American system for
delivering electricity across the nation as well as over the last mile
to your home. Could the Smart Grid tell marketers your life style?
Privacy of text messaging. EPIC is very active on City of Ontario
v. Quon, where the government asserts that using a city-issued
device allows the city to read all of the employee's messages.
Freedom of Information Act. Why are government agencies (except for a
few exemplary ones) fulfilling a smaller percentage of
demands during the Obama administration than they did during the Bush
administration?
Ballot initiatives. EPIC has argued in Doe v. Reed that
signing a petition to put a question on a ballot should be private,
like voting.

And if you visit the EPIC home page this week, or the companion href="http://privacy.org/">privacy.org page, you'll see that
they're following even more diverse issues: the FCC broadband
proposal, consumer privacy, data retention by ISPs, etc. They were
interested to hear what I've been learning recently about privacy in
electronic health records.

EPIC has been remarkably effective over the years as an organization
with about a dozen staff (mostly young and idealistic rather than
canny and seasoned) and no cash-wielding lobbyists. They haven't
compromised their principles in the dozen years I've been following
them, but they not only get to the table most of the time but manage
to bend the decision their way most of the time.

I attribute this success to single-mindedness (they can nail the
privacy chink in any initiative) persistence, coalition-building with
like minded organizations (leading the href="http://privacycoalition.org/">Privacy Coalition,
collaborating with London's href="http://www.privacyinternational.org/">Privacy International,
among other organizations around the world, and work closely with such
natural allies as the ACLU), but mostly knowing their stuff cold. They
sail into debate with a full understanding of technical details as
well as the legal issues that impinge on their position.

The Smart Grid is an excellent example of how EPIC investigates an
issue early in its existence and hones in on the dark underside. The
Smart Grid is a buzzword covering changes that should save us huge
amounts of electricity lost in old, inefficient switches, as well as
improve the efficiency of energy delivery in neighborhoods. A key part
of the Smart Grid is monitoring and logging our electricity usage,
building by building and even machine by machine.

In this futuristic vision, the electric utility would know when you've
started your air conditioner or clothes dryer and could send you
messages suggesting new patterns of behavior that will relieve
pressure on the grid and save you money as well. This is nice, but it
also means the electric utility basically knows how you lead your
life.

Traffic analysis on your device usage could show who stays home during
the day, when kids come home from school, and who plays video games
(heavy electricity usage from a home computer) late at night.

Currently no one has discussed who controls this data. Implicitly, it
is left in the hands of the utility, which is free to sell it like any
other information. There is little doubt that advertisers would love
to get their hands on this information. So would the government, I
bet--remember when police were scanning homes for evidence of
marijuana cultivation? EPIC would like the information to be in the
hands of the consumer.

A bill just introduced by Representative Ed Markey, the href="http://globalwarming.house.gov/mediacenter/pressreleases_2008?id=0209">"Electric
Consumer Right to Know Act" (H. R. 4860), would inform electricity
users of their energy usage in a form they could process on a computer
or other device, typically every 15 minutes. The bill mandates a smart
meter that "provides adequate protections for the security of such
information and the privacy of such electric consumer." It doesn't go
into any more detail about what the utility could do with the
information.

The ambiguous ownership of Smart Grid data illustrates why privacy is
such a hard turf to defend, once you have declared your jurisdiction
over it as EPIC has done. Data flows from one place to
another--whether from the electric meter to your cell phone, your
camera to Facebook, or your vendor to your bank--and is therefore
intrinsically shared. Privacy is an umbrella term that encompass
attempts to set limits or impose rules on all these types of sharing.

In trying to protect our privacy EPIC is swimming against the tide, of
course, but what's really challenging is how data collection and
dissemination has shifted. When EPIC started, most electronic data was
held by large institutions who made ready targets for EPIC's legal
challenges. Now each person is his or her own worst enemy, freely
sharing personal information, pictures, and videos online--a
phenomenon termed Little Brother.

Cameras and sensors are also creating millions of new sources for
data, while advances in data mining and analysis allow people to learn
more from the data than ever before.

I think EPIC is handling this shift well. They stay focused on policy
rather than pursuing the idealistic but impractical course of training
people to use privacy safeguards and protect themselves. There are
just too many ways to weasel data out of us, some of which will never
be under our control, and most people just can't learn everything they
need to know to be safe, whether it be about Web proxies, Flash
cookies, or document metadata.

EPIC demands that institutions take responsibility for privacy,
designing it into their systems. A recent, well publicized example of
this doctrine was their complaint to the FTC about Facebook's changes
to privacy settings in December 2009. EPIC doesn't believe it's enough
to boast about flexibility and user control--something that endangers
the 99.9% of users who don't understand how to change a default is a
violation of users' rights.

But EPIC is neither rigid nor abstentionist. They may complain about
Facebook, but maintain a Facebook page. They're totally into the new
electronic age. But they want it to serve its users rather than a few
centralized institutions, and for privacy advocates they're not shy
about letting us know what they think.

Trapping content on the iPad won't work, even if it's pretty

2010-03-19T13:00:00Z

Wired is one of the few magazines I read cover to cover. It consistently exposes me to new ideas and topics. For that, I'm grateful (and a longtime subscriber).

But when it comes to the iPad, I really don't understand what the Wired crew is doing.

Design-wise, Wired's iPad demo looks beautiful. Take a look:

Yet, reading over this analysis piece by Reuters' Felix Salmon, I'm dismayed to see a return to the days of silos and closed content. Here's how Salmon puts it:

Wired doesn’t want to allow simple links in ads or stories which would open up in the iPad web browser, since opening the browser means closing the Wired app. Instead, web links will open in a pop-up window within the iPad app, which then gets closed, returning you to the position in the magazine that you came from. The whole ethos is a magazine-like one of a closed system with lots of control -- the exact opposite, really, of the internet, which is an open system where it’s very hard indeed to control the user experience. [Emphasis added.]

This type of thing might work on the Kindle, where the "browser" is experimental (for now) and web hooks are limited. But the iPad is an Internet device and limiting web-based functionality from within an app -- even a very pretty app -- is counterintuitive. Web links should open in a web browser. Preferably a very good, very fast web browser, not those lumbering browser wannabes built into some iPhone apps. And whether I return to the app is up to me, not the publisher. I'll be back if the content is valuable. You just have to take my word on that.

Moreover, remember the early days of the web when designers dumped text into GIFs so they could lock down the look and feel? They just had to have control. And remember how that turned out? Trapping text in graphics is a big usability no-no. In time, we'll look at closed iPad apps the same way. It's backwards.

Digital content, like water, will always find a path to freedom. You lock it down, someone else will open it up. That's why the Wired iPad app -- and other offerings from publishers accustomed to silos -- should get ahead of the curve now. These apps need to offer more embedded links, more web hooks, and more opportunities to share. Designers can still create beautiful layouts. The user experience can still be unique. Advertisers can still be wooed. But the content itself needs to be connected to the web because it's being accessed through a web device. So why not expedite the inevitable and turn connectivity into an asset from the onset?

Four short links: 19 March 2010

2010-03-19T10:00:00Z

Tsung -- GPLed multi-protocol (HTTP, PostgreSQL, MySQL, WebDAV, SOAP, XMPP) load tester written in Erlang.
Myth of China's Manufacturing Prowess -- The latest data shows [...] that the United States is still the largest manufacturer in the world. In 2008, U.S. manufacturing output was $1.8 trillion, compared to $1.4 trillion in China (UN data. China’s data do not separate manufacturing from mining and utilities. So the actual Chinese manufacturing number should be much smaller). Also contains pointers to an interesting discussion of lack of opportunities for college grads in China.
OpenSSO and the Value of Open Source -- Oracle are removing all open source downloads and wiki mentions, leaving only the enterprise OpenSSO product on their web site. A Norwegian company has stepped in and will continue the open source project. This is essentially a fork, but for the forces of good. (via normnz on Twitter)
The Internet of Things -- 5m video on sensor networks, etc. (via imran on Twitter)

Four short links: 18 March 2010

2010-03-18T10:00:59Z

Newspaper Club Launches (BBC) -- the uses it has been put to make for good reading: Among the Newspaper Club's first clients were the BBC, Wired UK and Last.fm. Penguin used it to debut a preview of the fifth chapter of The Hitchhiker's Guide to the Galaxy, written by Eoin Colfer.
Machine Learning Algorithm with a Capital A -- It’s claimed to be close to the way the brain learns/recognizes patterns and to be a general model of intelligence and it will work for EVERYTHING. This reminded me of a few other things I’ve come across in the past years that claim to be the new Machine Learning algorithm with Capital A, i.e. the algorithm to end all other ML work, which will work on all problems, and so on. Here is a small collection of the three most interesting ones I remembered.
fityk -- GPL program for nonlinear fitting of analytical functions (especially peak-shaped) to data (usually experimental data). There are also people using it to remove the baseline from data, or to display data only. (via straup on delicious)
Chickenfoot -- Firefox plugin to let you script and manipulate web pages. Useful for automation, like Greasemonkey, but acts on the rendered page and not the HTML source.

Launchpad SF Submissions Are Open Until 3/31

2010-03-17T23:34:53Z

Startups! The Web 2.0 Expo is in two months. We want to highlight your work. Each year we put five of you onstage in a Launchpad session. The deadline for submitting your company is 3/31. The criteria is below.

Entrants do not need to launch their company or a major product/service to qualify.

All proposals will be reviewed before Web 2.0 Expo by our panel.

The judging panel will be comprised of industry experts who will review Launch Pad companies for their value to their market (consumer, enterprise, etc.).

Judges will select about five finalists, each of whom will have five minutes to pitch on stage, in front of the Web 2.0 Expo audience (the largest gathering of the innovators in the Internet industry) and the panel.

Each company will receive feedback on its presentation from both the panel and the audience.

Five companies will be chosen by a panel to present on stage at Web 2.0 Expo San Francisco on May 5, 2010. Each will have five minutes to present their company or product and will receive real-time feedback from a panel of industry experts and the audience. Submit Your Company For Consideration—entries are due by March 31, 2010.

Google Buzz and hybrid blogging

2010-03-17T13:00:00Z

Tim O'Reilly and DeWitt Clinton are both experimenting with Google Buzz as a long form -- well, longer form -- publishing tool. It's an interesting adaptation for Buzz, and I think they're on to something.

Here's why: Blogs are great for getting people to a site. Twitter is great for tossing around short-form ideas and quips. Facebook is great for talking with a defined community.

But blogs are not inherently social. They try to be, with comments and RSS, but they're still built in silos. Twitter is unbelievably social, no doubt about that, but it's also shorthand. It's very hard to have an engaging conversation in 140 characters. And Facebook is like a ping-pong match: lots of back and forth excitement, but very little substance.

Buzz could be the missing link here. It's a hybrid option that's not particularly good at being a blog, or a microblog, or a social network, but it's a good tool for starting conversations and noodling on topics. (Keith Crawford picked up on this early on.) Tim noted during a recent conversation that Buzz is a throwback to blogging's early days, when informal posts were the norm.

Buzz in many ways occupies the same domain as Tumblr and Posterous. All of these services let you dip a bucket in the social/content stream and pour the catch into your own trough. But Because Buzz is constructed in a social environment, as opposed to a publishing environment, it's a bit more natural to share all that conversation and information.

A lot of people just like to get on with it, which is why Twitter and Facebook will always be more popular. And I'm not saying -- nor am I even hinting -- that blogs are dead. Far from it. You need a hub for all those social media spokes, and blogs make great hubs. DeWitt Clinton, in a Buzz update, actually predicts a time when posts and comments from blogs, Buzz, and other networks will "flow seamlessly back and forth between them, such that the syndication will no longer be in only a single direction, but rather a network of threads woven together." That functionality is still a ways off (and I hope it arrives sooner rather than later), but in the interim it looks like Buzz has opened yet another content channel; a social space where you can toss an idea into a pool of willing conversationalists and see what happens.

One last thing ... because a blog post lauding Buzz for its conversation tools carries a hint of hypocrisy, here's my own attempt at a related Buzz conversation starter.

Google's New Marketplace Has over a Thousand Apps

2010-03-17T11:00:00Z

One week^† into its public launch, the Google Apps Marketplace has just under 1,500 (enterprise) apps. Combined with Salesfore.com's app exchange (also with over a thousand apps), enterprises interested in moving to cloud apps have an increasing number of software tools to choose from.

Popular apps (measured in terms of # of installs) includes graphic design and office integration apps (aviary design suite and offisync), a collaboration and project management tool (manymoon), a free travel planner (tripit), a basic ERP app (myerp.com), and a CRM application (Zoho CRM).

The typical supplier has about 2 offerings in the Google Apps Marketplace. Below are the suppliers with the most number of unique apps:

(†) Data for this post was through 2/16/2010.

Four short links: 17 March 2010

2010-03-17T10:00:00Z

Common MySQL Queries -- a useful reference.
MySociety's Next 12 Months -- two new projects, FixMyTransport and "Project Fosbury". The latter is a more general tool to help people organise their own campaigns for change.
riak -- scalable key-value store with JSON interface. (via joshua on Delicious)
Notes from NoSQL Live Boston -- full of juicy nuggets of info from the NoSQL conference.

SkyHook and SimpleGeo Present SpotRank, Now You Can Always Find Where The People Are

2010-03-17T09:20:59Z

In an under-appreciated announcement Skyhook Wireless released a huge set of location trend data. SpotRank, as the data is called, shares out ranking trends for locations around the world. The maps above show the SpotRanks of those locations.

Skyhook has been collecting look-ups for the past five years. CEO Ted Morgan shared these stats "We average about 300 million location requests a day and our reference database has over 200 million wifi access points and over 2 million cell towers." In total Skyhook has over 40 billion data points that are used to derive

As of right now SimpleGeo is the only way to access this data. I spoke with Matt Galligan and Joe Stump, the founders of SimpleGeo, about the new service and the SpotRank data. The data set is massive. There are half a billion 100mx100m boxes. It will be up to the customer to reverse geocode the location. For each location there is an entry for every hour of the past week (Stump said to picture it like a 7x24 grid). Right now the updates lag by an hour or two. The time to update will decrease over time.

For each time-location pair there are four different datapoints. As Skyhook describes them:

Worldside Rank - How one Spot compares to all other Spots worldwide.
City Rank - The City Rank illustrates how one Spot's city compares to other cities at any given time.
Local Rank - Local Rank ignores data from other cities, and shows how one Spot compares to others within that Spot's metro area.
Trending Rank - The Trending Rank is provided on a five-point scale of -2 to +2, indicating whether activity in a Spot will increase, decrease, or hold steady over the next hour.
(I included the full description after the jump)

This is a new type of data. Never before has something like SpotRank been released. It will be used mobile apps and mobile ads (as Stump pointed out, it could cost more to show an ad in a busy part of a city). I can also see it being used city planners and corporate real estate agents everywhere. It will also give us great insights into human behavior.

I asked Morgan what the most interesting thing that he'd learned from looking at this data. He replied "Two things, 1) that human behavior is highly predictive but that 2) they don’t do the things we think they do. We are still learning a great deal about the data and the underlying behaviors and we hope with SpotRank others can now help us figure it out as well. "

Some final thoughts come to mind:

1) How much longer will Skyhook remain independent?
Location is going to become increasingly important. Skyhook's wifi geolocation is one of the best systems out there and is going to keep on growing. Over time SpotRank will become even more accurate, How long till Apple buys them? Skyhook is baked into the iPhone and OS X. Apple's been making other geo moves (like buying Placebase). Will Google give up its own wifi geolocation project in favor of Skyhook? Would Microsoft or Intel (a Skyhook investor) decide that they want to location-enable their stacks?

2) How long will SimpleGeo remain independent?
Stump & Galligan pulled off quite the coup getting this data and being the only provider. The company is creating an ecosystem of geodata. When they throw open the gates developers will have a one-stop shop to access almost a terabyte of data (all hosted on a geo-ified version of Cassandra known as ~~Gazelle~~ Giselle). What large company wouldn't want this business? Heck, Twitter already bought their primary competitor Mixer Labs (who had the product GeoAPI - Radar post).

3) I want more ranks!
Google has PlaceRank. I'd like to know more about it, I think everybody would. Expect John Hanke to talk about it at Where 2.0 on his panel with Danny Sullivan - though I do not expect a SpotRank competitor from Google anytime soon.

Where 2.0 is just two weeks away in San Jose on 3/30-4/1. Both Ted Morgan and Matt Galligan will be speaking along with representatives from Twitter and GeoAPI. Register now with whr10pcb for 25% off.

Here's Skyhook's full explanation of the various data points:
Worldside Rank
How one Spot compares to all other Spots worldwide. A W-2 rank would signify an area that's quiet compared to the rest of the world, like a suburb in Lincoln, Nebraska. W-10 ranks are in the world's most active areas, like SoHo in Manhattan or the busiest neighborhoods of London.

City Rank
The City Rank illustrates how one Spot's city compares to other cities at any given time. On weekday rush hours, a C-10 Rank would give Manhattan a 10 vs. Omaha's 1. Ann Arbor, Boulder and Burlington would all be ranked C-5 at 8pm on a Saturday.

Local Rank
Local Rank ignores data from other cities, and shows how one Spot compares to others within that Spot's metro area. The most active areas of Cheyenne, Wyoming or Cedar Rapids, Iowa will get a L-10 Rank. The least active areas in Paris and San Francisco, although far more active than the Cheyenne and Cedar Rapids hotspots, would be assigned a L-1 Rank.

Trending Rank
The Trending Rank is provided on a five-point scale of -2 to +2, indicating whether activity in a Spot will increase, decrease, or hold steady over the next hour. Penn Station at 6am, just before rush hour, will get T +2 Rank, indicating that a throng of rush hour traffic is about to hit. Denver's Lower Downtown will get a -2 Rank at 1am on a Saturday morning, indicating that the partygoers are about to leave the clubs and head home in the next hour.

Google Fiber and the FCC National Broadband Plan

2010-03-16T19:00:00Z

I've puzzled over Google's Fiber project ever since they announced it. It seemed too big, too hubristic (even for a company that's already big and has earned the right to hubris)--and also not a business Google would want to be in. Providing the "last mile" of Internet service is a high cost/low payoff business that I'm glad I escaped (a friend and I seriously considered starting an ISP back in '92, until we said "How would we deal with customers?").

But the FCC's announcement of their plans to widen broadband Internet access in the US (the "National Broadband Strategy") puts Google Fiber in a new context. The FCC's plans are cast in terms of upgrading and expanding the network infrastructure. That's a familiar debate, and Google is a familiar participant. This is really just an extension of the "network neutrality" debate that has been going on with fits and starts over the past few years.

Google has been outspoken in their support for the idea that network carriers shouldn't discriminate between different kinds of traffic. The established Internet carriers largely have opposed network neutrality, arguing that they can't afford to build the kind of high-bandwidth networks that are required for delivering video and other media. While the debate over network neutrality has quieted down recently, the issues are still floating out there, and no less important. Will the networks of the next few decades be able to handle whatever kinds of traffic we want to throw at it?

In the context of network neutrality, and in the context of the FCC's still unannounced (and certain to be controversial) plans, Google Fiber is the trump card. It's often been said that the Internet routes around damage. Censorship is one form of damage; non-neutral networks are another. Which network would you choose? One that can't carry the traffic you want, or one that will? Let's get concrete: if you want video, would you choose a network that only delivers real-time video from providers who have paid additional bandwidth charges to your carrier? Google's core business is predicated upon the availability of richer and richer content on the net. If they can ensure that all the traffic that people want can be carried, they win; if they can't, if the carriers mediate what can and can't be carried, they lose. But Google Fiber ensures that our future networks will indeed be able to "route around damage", and makes what the other carriers do irrelevant. Google Fiber essentially tells the carriers "If you don't build the network we need, we will; you will either move with the times, or you won't survive."

Looked at this way, non-network-neutrality requires a weird kind of collusion. Deregulating the carriers by allowing them to charge premium prices for high bandwidth services, only works as long as all the carriers play the same game, and all raise similar barriers against high-bandwidth traffic. As soon as one carrier says "Hey, we have a bigger vision; we're not going to put limits on what you want to do," the game is over. You'd be a fool not to use that carrier. You want live high-definition video conferencing? You got it. You want 3D video, requiring astronomical data rates? You want services we haven't imagined yet? You can get those too. AT&T and Verizon don't like it? Tough; it's a free market, and if you offer a non-competitive product, you lose. The problem with the entrenched carriers' vision is that, if you discriminate against high-bandwidth services, you'll kill those services off before they can even be invented.

The U.S. is facing huge problems with decaying infrastructure. At one time, we had the best highway system, the best phone system, the most reliable power grid; no longer. Public funding hasn't solved the problem; in these tea-party days, nobody's willing to pay the bills, and few people understand why the bills have to be as large as they are. (If you want some insight into the problems of decaying infrastructure, here's an op-ed piece on Pennsylvania's problems repairing its bridges.) Neither has the private sector, where short-term gain almost always wins over the long-term picture.

But decaying network infrastructure is a threat to Google's core business, and they aren't going to stand by idly. Even if they don't intend to become a carrier themselves, as Eric Schmidt has stated, they could easily change their minds if the other carriers don't keep up. There's nothing like competition (or even the threat of competition) to make the markets work.

We're looking at a rare conjunction. It's refreshing to see a large corporation talk about creating the infrastructure they need to prosper--even if that means getting into a new kind of business. To rewrite the FCC Chairman's metaphor, it's as if GM and Ford were making plans to upgrade the highway system so they could sell better cars. It's an approach that's uniquely Googley; it's the infrastructure analog to releasing plugins that "fix" Internet Explorer for HTML5. "If it's broken and you won't fix it, we will." That's a good message for the carriers to hear. Likewise, it's refreshing to see the FCC, which has usually been a dull and lackluster agency, taking the lead in such a critical area. An analyst quoted by the Times says "One again, the FCC is putting the service providers on the spot." As well they should. A first-class communications network for all citizens is essential if the U.S. is going to be competitive in the coming decades. It's no surprise that Google and the FCC understands this, but I'm excited by their commitment to building it.

Four short links: 16 March 2010

2010-03-16T10:00:00Z

Government is an Elephant (Public Strategist) -- if Government is to be a platform, it will end up competing with the members of its ecosystems (the same way Apple's Dashboard competed with Konfabulator, and Google's MyMaps competed with Platial). If you think people squawk when a company competes, just wait until the competition is taxpayer-funded ....
Recordings from NoSQL Live Boston -- also available in podcasts.
Modeling Scale Usage Heterogeneity the Bayesian Way -- people use 1-5 scales in different ways (some cluster around the middle, some choose extremes, etc.). This shows how to identify the types of users, compensate for their interpretation of the scale, and how it leads to more accurate results.
Building a Better Teacher -- fascinating discussion about classroom management that applies to parenting, training, leading a meeting, and many other activities that take place outside of the school classroom. (via Mind Hacks)

Open Data Pointers

2010-03-16T10:00:00Z

When I blogged about truly open data, readers sent me a lot of interesting links. I've collected them all below. Enjoy!

The Centre for Environmental Data Archival (CEDA) -- hosts a range of activities associated with evironmental data archives. (Director is on Twitter, @bnlawrence)
CONNECT -- open source healthcare data exchange being developed with Brian Behlendorf, one of the original developers of the Apache web server.
Phil Agre's Living Data -- prescient article in Wired from 1994.
Factual -- web database that permits multiple values in tables, and you can apply different functions to choose which values you'll use when you work with the data (e.g., "most recent", "most popular", ...).
HDF -- BSD-licensed toolkit and format for storing and organising large numeric datasets.
NetCDF -- software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.
Madagascar -- an open-source software package for multidimensional data analysis and reproducible computational experiments with digital image and data processing in geophysics and related fields.
Data Documentation Initiative -- standard for metadata of social science datasets.

The Memento Project

Data Sharing: A Look at the Issues -- presentation from Science Commons data manager Kaitlin Thaney.
SIFN Datasprint -- these folks are planning a sprint around data, the same way coders often have sprints around code.
Get Your Database Under Version Control -- 2008 piece by Jeff Atwood on the need to version control your database.
CKAN -- Comprehensive Knowledge Archive Network. The database of open datasets is itself an open dataset, managed by a versioned database.
Componentization and Open Data, Collaborative Development of Data: OKFN are figuring out packaging and structure of distributed data development. They seem closest to building what I was talking about.
Open Data Maturity Model (Slideshare) - I like the idea of progressing from amateur to professional and identifying milestones along the continuum, but I'm not convinced that the last two stages are based on existing projects. I'm a big fan on building frameworks from successful projects, rather than building the framework in isolation.
Data RSS -- proposal for an API for data feeds.
Fedora Commons -- open source for managing, indexing, and delivering datasets. Islandora integrates that into Drupal.
gEDA -- GPL'd suite of Electronic Design Automation tools, some of which are applicable to non-electronics data projects.
You Cannot Run an Open Data Project Like an Open Source Project, Unless… -- not always coherent disagreement with my post. His most lucid moment comes pointing out that government datasets have a single owner. This is the difference between intrinsic data (crimes reported to police) where the data is about the operations of an agency, and extrinsic data (wild horse populations in Arizona, tree ring climate records) where an agency sends people into the field or otherwise collects and possibly processes data from others' labour. But even intrinsic data can be more collaboratively maintained: take bug reports and corrections from users (there is no 800 block of Main, did you mean the 80 block of Main?). It's true: I can't imagine a lot of collaboration around the preparation and distribution of pure sensor data (e.g., traffic data), but my post talked about more than collaborative generation: revision management, documentation, etc.

The Second Netflix Challenge and Privacy Research

2010-03-15T18:45:52Z

Okay, if you're just catching up with this story, go read this first -- Netflix's announcement that it was canceling its second Netflix Prize challenge over privacy concerns.

Next, head over to 33bits.org, blog of one of the co-authors of the paper on de-anonymizing Netflix users from the first Netflix Prize challenge data, to read the authors' open letter to Netflix about the canceled second challenge.

Data privacy researchers will be happy to work with you rather than against you. We believe that this can be a mutually beneficial collaboration. We need someone with actual data and an actual data-mining goal in order to validate our ideas. You will be able to move forward with the next competition, and just as importantly, it will enable you to become a leader in privacy-preserving data analysis. One potential outcome could be an enterprise-ready system which would be useful to any company or organization that outsources analysis of sensitive customer data.

I find that paragraph from the post particularly interesting. This seems similar to the conversations between security researchers and the companies whose products they find ways to exploit. That has often been a very hostile conversation, but it seems (speaking from the outside of the security community) to have improved over time. (For instance, check out this security research guidelines document from PayPal.) Is there a way for privacy research to head in a similar direction, so that companies view external researchers as in some way beneficial? If anything that seems like a bigger challenge to me; at least everyone usually agrees that security holes should be fixed, while most companies do not agree, publicly at least, that privacy breaches are really a problem (e.g., "Get over it.").

Why HTML5 is worth your time

2010-03-15T13:00:00Z

The debate over HTML5 vs. Flash is great for comments and page views, but all that chatter obscures the bigger issue: Should developers and designers invest in HTML5?

According to Eric A. Meyer, an author and HTML/CSS expert, the answer is a definitive yes. In the following Q&A, Meyer explains why HTML5, CSS and JavaScript are the "classic three" for developers and designers. He also pushes past the HTML5 vs. Flash bombast to offer a rational and much-needed comparison of the toolsets.

HTML5's feature set

Mac Slocum: How is HTML5 different than HTML as we currently know it?

Eric Meyer: It's really the HTML we're all used to plus more elements. But that's the 80/20 answer. HTML5 adds new elements for things like sections of a document and articles, and figures and captions for figures. So it covers things that a lot of us do all the time, like create <div class="figure"> and then <p class="caption"> inside of that to go along with an image. Now there's just an element called "figure" and you insert an image and you have an element after that called "caption."

There's been an attempt to look at what people are doing. What class names are people using over and over again? What structures are they setting up over and over again? Because HTML doesn't have elements that directly address those.

The HTML5 spec also attempts to very precisely and exhaustively describe what browsers should do in pretty much any given circumstance. Older HTML specifications would simply say: "These are the elements. These are the attributes. Here are some basic parsing rules. Here is what you're supposed to do if you encounter an error." HTML5 has these really long algorithms that say: "Do this, then this, then this, then this. And if you hit a problem, here, do this other thing." There's a lot of debate as to whether that's even a good idea. But if the vision that's encoded in those algorithms is brought out -- I'm not saying it will be, but if it is, then browsers will be a lot more interoperable.

But that's the base level answer. As you push further into the more obscure corners, then the answer to "how is HTML5 different?" becomes much more complicated.

MS: Is HTML5 becoming a full-fledged development environment?

EM: I don't see it stepping forward into full-fledged programming. But I do see it pushing HTML forward so that it's a better foundation for web apps. That's one of HTML5's primary goals. There are sections of it that are devoted solely to how to deal with web application environments.

The thing that's most directly applicable to making HTML more web-application friendly is the attempt to include what's known as microdata. That's semantic information and little snippets of data that can be embedded directly into what we think of as pages right now. But these can become the views a web application presents. It's the kind of stuff that we put in cookies now.

But HTML is not getting for loops or switch statements. That's going to stay with JavaScript. In that sense, no, HTML is not becoming a programming language.

What developers and designers need to know about HTML5

MS: What skills do developers need to take full advantage of HTML5?

EM: Developers need to know HTML5. They need to know JavaScript and they need to know CSS. That's the classic three.

MS: How about designers?

EM: Designers need to know mark-up. They need to know HTML5. They need to be able to write CSS and understand web layout. And they need to have at least a decent grasp of what JavaScript does. I don't necessarily insist that everyone who ever touches the web be able to write their own web app by hand, but designers should understand how JavaScript works.

There are a lot of people who call themselves web designers who are really just designers who put their designs on the web. And there's nothing wrong with being just a designer. But they're not necessarily web designers. They're visual designers. There's a difference.

MS: Would you recommend starting with web development skills and then adding Flash and others later?

Yeah. Make that your grounding and then add things to it if you like. You're making a very dangerous bet to not have web tools at your disposal. The developer should be able to do web work. And it's not a bad idea to add Flash to the tool belt.

HTML5 vs. Flash: A rational comparison

MS: Without getting into the "Flash killer" stuff, how does HTML5 compare to Flash?

EM: HTML5 itself and Flash are vastly different. They have different things that they're trying to do. But the HTML5 plus CSS plus JavaScript package is more. I think that's an easier comparison to make to Flash because Flash is supposed to be this total environment. You can put things on the screen and you can script it and you can define interaction. And HTML5-CSS-JavaScript lets you do that as well.

We got to the point a couple of years ago where the HTML-CSS-JavaScript stack can technically do just about anything that the Flash environment makes possible. It's just a lot harder at the moment to do that in HTML5-CSS-JavaScript because Flash has about a decade's head start on authoring environments.

There are a number of people, myself included, who have been observing for a while now that the current web stack feels like Flash did in 1996. Look at the canvas demos, for example. The canvas demos we're seeing now are totally reminiscent of the Flash demos we used to see in the '96 era, where it was like: "Hey, look! I have three circles and you can grab one with a mouse and flick it. And then it bounces around the box and there's physics and collision and animation and they're blobby and woo hoo."

MS: What's your take on plugins? Are they inherently inelegant?

EM: That's been my feeling for a long time. That any plug-in is kind of inelegant and the wrong way to be going about this. And I don't reserve that just for Flash. I really mean any plug-in. The fact that we need plug-ins to play movies has never felt right.

MS: If, for a given application, HTML5 and Flash can provide the same result, why would a developer go with HTML5? What's the motivation?

EM: HTML5 is native to the medium. It's the feeling that if we're going to do web stuff, let's do web stuff. Let's not do Flash stuff that happens to be represented in a web page. So I think that's the philosophical drive.

The technical drive, to a large degree, is that companies don't want to be beholden to somebody else. And doing everything in Flash means that they're effectively beholden to Adobe. With web technologies, the only entity that can reasonably be said to hold the keys to the kingdom is the W3C. And even if the W3C for some reason turned into "evil goatee Spock" tomorrow and said "we want licensing fees," everyone would go, "yeah, no."

HTML5 and mobile applications

MS: Does HTML5 give mobile developers more latitude? Is there benefit in developing applications outside Apple's approval process?

EM: Absolutely. No question. There are some people who have argued that the whole App Store phase is a fad. Granted, a very popular and lucrative and probably long-lived fad, but that it's still a fad.

The argument is that 10 years from now we're going to look back at rebuilding apps for every mobile device and go "What the hell were we thinking?" It's the same way kids who graduate from decent web development programs today don't understand why anyone ever tried to layout a page with tables. I've had conversations with people who literally just can't understand. Even when you explain, "Well, there was no CSS." They're like, "But surely there was something better because that's just awful."

Betting against the web is the sure losing bet of technology. Over the long-term, that's where I see things going.

Note: This interview was condensed and edited.

Four short links: 15 March 2010

2010-03-15T10:00:00Z

A German Library for the 21st Century (Der Spiegel) -- But browsing in Europeana is just not very pleasurable. The results are displayed in thumbnail images the size of postage stamps. And if you click through for a closer look, you're taken to the corresponding institute. Soon you're wandering helplessly around a dozen different museum and library Web sites -- and you end up lost somewhere between the "Vlaamse Kunstcollectie" and the "Wielkopolska Biblioteka Cyfrowa." Would it not be preferable to incorporate all the exhibits within the familiar scope of Europeana? "We would have preferred that," says Gradmann. "But then the museums would not have participated." They insist on presenting their own treasures. This is a problem encountered everywhere around the world: users hate silos but institutions hate the thought of letting go of their content. We're going to have to let go to win. (via Penny Carnaby)
StoryGarden -- a web-based tool for gathering and analyzing a large number of stories contributed by the public. The content of the stories, along with some associated survey questions, are processed in an automated semantic computing process for an immediate, interactive display for the lay public, and in a more thorough manual process for expert analysis.
Google Apps Script -- VBA for the 2010s. Currently mainly for spreadsheets, but some hooks into Gmail and Google Calendar.
There's a Rootkit in the Closet -- lovely explanation of finding and isolating a rootkit, reconstructing how it got there and deconstructing the rootkit to figure out what it did. It's a detective story, no less exciting than when Cliff Stohl wrote The Cuckoo's Egg.