tecosystems

The Friday Grab Bag: Google Gripes Edition

2009-11-06T13:31:13-08:00

There are any number of larger subjects we should be talking about today, but it’s late on a Friday and I’ve been doing too much project work to finish the pieces in the hopper. So instead this will serve as the traditional Friday grab bag, but with a twist: it’s all about Google gripes this week.

As the just launched Google Dashboard reminds me, I use a large number of Google products. Some more actively than others, but it’s not a stretch to say that a great deal of my personal and professional lives is managed by the Big G. In general, this approach works well, because Google is probably the most platform agnostic of the technology providers we could use, and at any given time I may be accessing my data using one flavor of Linux or another, versions of Windows back to 2000, Mac OS X or its close cousin, the iPhone.

Occasionally, however, Google’s products frustrate me. All software has bugs, of course, but the three that follow happen to be particularly irritating.

Chromium Crashes

To be fair to Google, while the builds of Chromium are not deemed production quality, it remains the fastest and lowest UI footprint browser available. For the time being, I’ve mostly cut over to using it as my primary browser, although I still spin up Firefox a half dozen times of day when Chromium misbehaves. And it definitely misbehaves.

One of Chromium’s much appreciated features is its tab-per-process model, which attempts to ensure that a single locked up tab doesn’t bring down the entire browser around it. For the most part, this works as promised. One of the things I’ve been seeing more of, however, is cascading failures amongst tabs. For example, Chromium seems to really dislike Google Reader – ironic, I know – as well as a few of the blogs I follow like The Big Lead. Opening a link to that site, for example, will not only return an Aw Snap (pictured) for that page, but for the originating Google Reader tab as well as every other page I’ve opened from the aggregator.

Whatever happened to tab isolation?

Google Profile

This one is the simplest problem, but it is, frankly, driving me absolutely bonkers. Mostly because I don’t appear to have any real recourse. Google Profiles assures me that I “can control how [I] appear in Google by creating a personal profile,” but I can’t. Really. As you can see from the screenshot above, not only is the picture of me old, Google Profiles crushes it so that I appear approximately a foot wider. Not a good look.

So I tried to change it. Google Profiles dutifully allows me to upload a new picture, crop the portion that I want, and save the change. It will then use the new picture for some undetermined amount of time – at least an hour, as far as I can tell – after which it will silently revert back to the picture you see above. Also known as the picture I’d like to replace.

What can I do about this? Well, changing it multiple times doesn’t work: I’ve switched it to the new picture at least a half dozen times, to no avail. The Help link isn’t one.

Anyway, I might as well document it here just so that you’re aware that I’m not actually five and a half feet wide, no matter what Google Profiles tells you.

Mobile Gmail

One of the things I’ve consistently appreciated about Google’s messaging – the email/calendaring/etc kind, not the PubSubHubbub kind – solutions has been their unwavering support for HTML interfaces for mobile devices. While many of Google’s competitors delivered J2ME clients or RIM specific solutions, Google instead delivered an interface for the masses: basic HTML that would render even on the not-so-smart smartphones I had prior to getting an iPhone.

More recently, they rolled out a elegantly reskinned and HTML5 based version of the mobile interface, which had a few issues at launch and hasn’t gotten much better.

While I haven’t had a repeat of that issue – which resent a specific email every time I logged into the client – I have been having an increasing volume of errors that require the deletion of the mobile Gmail database. Specifically, I open the client and watch it freeze as above; the gear icons just spin endlessly, never returning the Inbox. This can’t be attributed to poor network conditions, because at the same time I’m able to visit other sites and even other Gmail based inboxes. This would indicate to me that it’s a specific problem with the local database. And indeed, when I have this problem, and go into Settings:Safari:Databases and delete the database in question, all is well again. The Inbox pops up normally and with no issue.

If this happened infrequently – say once a month – I wouldn’t think much of it. But between my various Gmail accounts, it’s probably happening once a week – once every two weeks at best. Which isn’t good. I’m certainly not going to switch from Google because of it – worst case I could just use the IMAP client – but it’s certainly a negative experience.

Share This

links for 2009-11-05

2009-11-05T17:03:19-08:00

Hadoop, Pig, and Twitter (NoSQL East 2009)

from a Twitter employee, no less.

(tags: hadoop pig twitter nosql mapreduce bigdata statistics analytics)
Harald Welte's blog

someone doesn't like Android very much

(tags: android haraldwelte linux kernel criticism google)
litl webbook: some technical comments « Lucas Rocha

the technologies that went into litl – lots of clutter, not surprisingly. javascript too.

(tags: cloud web mobile startup laptop clutter javascript)
Personal data warehouse · Nat Friedman

i could not agree more. i want to be able to store, mine, and use my own data with reasonable performance and tools.

democratization of BI, remember.

(tags: personal data warehouse backup ideas projects)
Downeaster "Modified Express Trains" To Be Evaluated

yes please

(tags: downeaster train rail travel portland maine)
EETimes.com – Qualcomm creates open source software group

good decision on their part. now let's see how they execute and engage.

(tags: qualcomm opensource mobile)
It’s Official. Google Wave to Get Its Own App Store (with potentially more than just apps)

"“We’ll almost certainly build a store…So many developers have asked us to build a marketplace—and we might do a revenue-sharing arrangement.” – i told you people marketplaces were coming

(tags: google wave appstore businessmodels marketplaces)
The Scale-Out Blog: Replicating from MySQL to Drizzle and Beyond

"Drizzle is one of the really great pieces of technology to emerge from the MySQL diaspora–a lightweight, scalable, and pluggable database for web applications. I am therefore delighted that Marcus Erikkson has published a patch to Tungsten that allows replication from MySQL to Drizzle. He's also working on implementing Drizzle-to-Drizzle support, which will be very exciting." – very cool

(tags: tungsten replication drizzle scaleout mysql)
Brian "Krow" Aker's Idle Thoughts – Drizzle, InfiniDB, Column Oriented Storage

a potential solution for column oriented storage, as described by Brian

(tags: brianaker drizzle infinidb columnoriented database mysql)
Heroku vs EngineYard Cloud vs Joyent – blog.eliotsykes.com

interesting look at some of the specialized cloud hosting options from a user's perspective

(tags: heroku engineyard joyent cloud user feedback)
Google privacy controls: Most people won't care | The Open Road – CNET News

i concur: there is little evidence to suggest otherwise

(tags: google privacy concerns mattasay)
Talking to DC « Adam Bosworth’s Weblog

Adam Bosworth on the standards process and its lessons. must read.

(tags: adambosworth standards design architecture)
The Silent Number: Top things to do after installing Ubuntu Linux 9.10 Karmic Koala

very useful summary of post-installation tweaks, apps, and more for Ubuntu Karmic users

(tags: karmic ubuntu gnome howto 9.10 tutorial setup tips)
Windows Azure : Windows Azure Platform Launch Update

an update on the Azure platform launch

(tags: windows azure microsoft cloudcomputing)
NoSQL Misconceptions | Viget Extend

just what it says

(tags: nosql database design expectations)
Guest Post: Yahoo's Cloud Team Open Sources Traffic Server

interesting new contribution to open source: load balancing and more.

(tags: opensource apache cloud yahoo traffic trafficserver loadbalancer)
How Thinking Goes Wrong

love this piece. 25 things that lead us away from rational thought.

(tags: rational thought criticalthinking skepticism religion logic philosophy)
Collective Competency – The Underlying Trend Driving Open Source

not sure that i love the term. my primary difficulty is that it attempts to apply a core business management concept on an endeavor that is, at best, only partially a business. much is lost in translation. is multi-entity collaboration transformative in terms of its shared asset ownership and investments? sure. is it a "collective competency?" i'm not sure.

(tags: collectivecompetency gartner opensource)
Gartner in the dock over Magic Quadrant | Irregular Enterprise | ZDNet.com

heard this one before. if i get time, i'll comment, but until then read Dennis' take.

(tags: gartner magicquadrant dennishowlett oversight marketing blogs ethics)
Art Beat | 35 Years of Austin City Limits | Online NewsHour | PBS

summary of the ACL shows, including Pearl Jam sets. awesome.

(tags: acl austincitylimits music pearljam)
I like Unicorn because it's Unix

another must read piece from Ryan. as usual.

(tags: ryantomayko unix coding sockets posix unicorn threads)
Git Magic

good work on git and its capabilities

(tags: git howto documentation dscm)
A Deluge is Underway; is Email Waterproof? :: Return Path Blog

a look at Raindrop and Wave. i'll have more on them shortly, but i think we need to think more broadly to understand where collaboration is going.

(tags: collaboration raindrop wave google mozilla)
Avro: a Format for Big Data » Cloudera Hadoop & Big Data Blog

Doug Cutting on a format for tackling big data

(tags: cloudera dougcutting avro bigdata megadata hadoop)
Flameeyes's Weblog : Distributions *are* the strength of Linux

interesting take on diversity as a strength, rather than weakness, of the Linux community

(tags: ecosystems linux distributions diversity)

Share This

Applications I Can’t Live Without, Volume 1: YubNub

2009-11-05T11:37:50-08:00

Seriously: who wouldn’t appreciate an application named after the Ewok word for freedom?

If you like the command line, you’ll love YubNub. And even if you don’t know what the command line is, you’ll probably still love YubNub because it saves you time just about every time you use the browser. For the first in a series of screencasts on applications I can’t live without, I couldn’t have picked a better one: the Ruby built application is a staple of my everyday life.

Come see why in this video. You may want to view it full-screen in HD to get the full effect.

Notes: Screencast was recorded with gtk-recordMyDesktop, audio was recorded with Audacity, and the final video was rendered using PiTiVi. Thanks to those projects for their hard work.

Share This

Current State of Intel Video, Ubuntu, and Composited Desktops

2009-11-04T08:29:11-08:00

Like Jeremy Zawodny, I’ve had persistent issues with Intel Video since the Jaunty release of Ubuntu. The short version is that the fancy effects that I eventually came to depend on stopped working after the Intrepid release of the distribution. Both Jaunty and Karmic include Intel video drivers – and I’ve tried every bleeding edge PPA version I can find – that simply don’t work with Compiz. For those of you running similar Intel video chipsets (my machine is a Thinkpad X301, which reports the hardware as: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller), then, here are the current options for composited desktops on Ubuntu.

Compiz

Normally, this can be easily activated by selecting System:Preferences:Appearance:Visual Effects:Normal/Extra. The Intel video drivers included with Jaunty and Karmic, however, include a regression which disables Compiz. What you’ll get is a dialog box that says “Searching for available drivers,” a refresh of the display, and no Compiz. As mentioned above, I’ve tried in the past to use the bleeding edge repositories for Intel drivers, but have not had success making Compiz work.

I lied: Compiz now works just fine. The solution? Reinstallation of the Compiz package and its assorted dependencies. I have no idea why this should work, but it worked for someone else and it worked for me. So right now, then, Compiz is your best option.

Metacity

The good news is that composited Metacity – the software Compiz was intended to replace – works. The bad news is that it doesn’t work well. Effects can be enabled, as you can tell from the glassy and transparent panel of GNOME Do, pictured. But enabling them introducing a noticeable and unacceptable lag; ALT-TAB behavior includes a perceptible, unfortunate delay from command to window presence.

If you want to enable this, ensure that Compiz is turned off and execute: gconftool-2 -s '/apps/metacity/general/compositing_manager' --type bool true. And before you ask, yes, false will turn it off.

GNOME Shell

The most interesting of the three options, GNOME Shell is the project upon which the next iteration of the GNOME desktop – version 3 – will be based. While the developers are likely to push the release date out from March to September, you can easily build and test it out on your desktop now.

Like KDE 4, which was a substantial revamp of the previous version of that project, GNOME Shell is a fairly radical rethinking of the desktop paradigm. I like some features more than others – the application activation paradigm suffers greatly in comparison to the non-intrusive GNOME Do, in my opinion – but it’s certainly an attempt to push things forward.

The most interesting aspect of the project, from my perspective, is Mutter. Mutter is essentially Metacity with the Clutter technologies baked into it. As Joe Gregorio noted back at OSCON, Clutter’s a popular project these days, and for good reason. Built by the Intel-acquired OpenedHand team, among others, Clutter offers aesthetically pleasing, thoroughly modern graphics capabilities that, even better, are fast. Indeed, of the available compositing options for my hardware, the only one that performs adequately is the next generation option in the Clutter based GNOME Shell. That is most unusual; the bleeding edge stuff is typically very slow.

I’ll have more on Shell in future, but for now, just be aware that it may be – depending on your hardware – the only compositing option that works.

Share This

Amazon, RDS and the Future of MySQL

2009-11-03T16:24:02-08:00

Last Monday evening, while I was en route from Las Vegas to Seattle, Amazon unwrapped its latest Web Services effort: Amazon RDS – The Amazon Relational Database Service. The addition of a relational offering is no real surprise, to be honest; as much as non-relational persistence mechnanisms and the so-called NoSQL catalog may be a better fit, architecturally, for scale out clouds, there are an order of magnitude more applications that require a relational database than don’t.

What was very interesting, however, was the selection of MySQL as the database. Granted, it is the most popular open source database, and potentially the most popular relational database period. But with the project’s future open to question – the EU has stalled Oracle’s planned takeover of MySQL parent Sun – the timing is interesting. Amazon is effectively launching a product with no certainty of who the future owner of the MySQL assets may be. Which may, in fact, be the point. But let’s come back to that.

Reactions to RDS have been fairly mixed. For some, there’s little that’s new about the service. Reuven Cohen of Enomaly, for example, had this reaction on the Cloud Computing Interoperability Forum:

Let me get this straight. Amazon basically created a Restful Meta API for deploying mySQL clusters for people not smart enough to figure out how to do it themselves using one of the dozens of mySQL cluster AMI’s available at 8.5cent / hour.

Instead of a Web service, what they need is a giant red button labeled DummyDB.

Others, like Lonely Code, are far more appreciative:

Traditionally one would have to set up a block storage device, mount it, hack the mySQL configuration to use the mounted EBS volume for storage (not to mention messing around with XFS locking and snapshotting the DB when the instance goes down). All in all a pain to implement and maintain.

Now with Amazon’s Relational Database Service, you get a MySQL database, hosted in the cloud – auto backed-up, patched and running on a scalable infrastructure (you can re-size it to your hearts content) and at the same time independent of your instances – meaning no more messing around with block storage for your database.

From here, it seems fairly clear that while RDS will not be the best option for every MySQL user, it will find a more than adequate market of customers who are willing to trade money for time, as Mårten Mickos might put it. Assuming that Amazon can realize its typical economies of scale by amortizing the management and administration costs of the service over a wide array of machines, the product should more than pay for itself simply by widening the addressable market.

How much wider will it make the addressable market? At a minimum, it will lower the barriers to entry for customers with relational needs (read: most customers) and a lack of cloud expertise. It will be fascinating to see, however, if Amazon has far grander ambitions in mind.

We’ll need to remember, when we look back at this time, that MySQL is at an interesting crossroads in its existence. Its overwhelming popularity earned it a billion dollar valuation from Sun, but the competitive pressure from technically differentiated forks – made possible by the MySQL licensing model – was mounting even prior to Sun’s decision to sell. As Mårten says:

Indeed, in the MySQL ecosystem, there are already a number of forks. Each one of those forks may perhaps currently be individually weak and unpromising. But the reality remains that if the main steward of an open-source product fails to live up to reasonable expectations, the forces of open source will take over.

MySQL, by virtue of its copyright and trademark ownership among other things, has clear advantages in the marketplace. But it is far from correct to assert that they are insurmountable. Mostly ignored, for example, have been the opportunity for hosting versions of the database. Hosted versions of the database just like RDS, actually.

It’s also worth mentioning that forked vendors are not at all prohibited from leveraging – and exclusively licensing and differentiating – hosted versions of the database, because hosting does not trigger the protections of the GPL. That’s a business model and an additional revenue option that is as viable for them as it is for MySQL.

When I first read about Amazon’s new relational service after touching down in Seattle last Monday, I noticed something interesting. The first three paragraphs of the release talked about MySQL, but didn’t say that it was MySQL. It talked about how it had the capabilities of MySQL, and would work with your existing MySQL databases, but it stopped short of saying that it was MySQL, instantiated in the cloud.

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity while managing time-consuming database administration tasks, freeing you up to focus on your applications and business.

Amazon RDS gives you access to the full capabilities of a familiar MySQL database. This means the code, applications, and tools you already use today with your existing MySQL databases work seamlessly with Amazon RDS. Amazon RDS automatically patches the database software and backs up your database, storing the backups for a user-defined retention period. You also benefit from the flexibility of being able to scale the compute resources or storage capacity associated with your relational database instance via a single API call.

As with all Amazon Web Services, there are no up-front investments required, and you pay only for the resources you use.

Amazon RDS Functionality

Amazon RDS is designed for developers or businesses who require the full features and capabilities of a relational database, or who wish to migrate existing applications and tools that utilize a relational database. It gives you access to the full capabilities of a MySQL 5.1 database running on your own Amazon RDS database instance.

A subtle distinction, to be sure, but potentially an important one. Reading further, it quickly became apparent that Amazon did not, in fact, have any issues declaring its offering MySQL. But the potential, to me, is still there.

Consider that the available forks of MySQL are, for reasons that have been apparent for some time, not only technically differentiated, but regarded by many in the community as superior to the builds available from MySQL itself. In spite of this, they are, as Mårten argues above, individually weak. Forks cannot use the MySQL brand, and they cannot hope to match MySQL’s distribution in the near term through conventional distribution mechanism. But Amazon wouldn’t necessarily need the brand (in the context of EC2, which is stronger – Linux or EC2?), and its cloud offers massive advantages over conventional distribution. What if, compelled by market conditions and Oracle pricing, Amazon eventually stopped talking about its MySQL databases in favor of a compatibility story that would permit it host your MySQL databases. And what if, for whatever the reason, MySQL developers were unhappy with the state of affairs and chose to leave, making them available to Amazon? Or Amazon shifted its partnership to the Open Database Alliance?

Are we looking at MySQL’s successor, then, in RDS? Maybe, maybe not. Compatibility issues would be a major concern, and the feature deltas that Mark discusses could throttle interest. But that’s the wrong question. The right question is could we be looking at the next MySQL in RDS, and the answer to that, I think, is yes.

Share This

Why I Am Voting No On 1

2009-11-02T21:15:52-08:00

“Fourscore and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal.” – Abraham Lincoln

Because I believe in those words. As does Maine native, World War II veteran and self-described lifelong Republican Philip Spooner, above.

If you believe in Lincoln’s proposition, that transcendent idea that we are each of us created equal, I do not believe that you can in good conscience vote yes on Question 1. Question 1, of course, being the ballot referendum here in Maine which seeks to reject the right for same-sex couples to marry.

Stripped of the fearmongering, the issue before us is not complicated: should a citizen of this state be denied a basic human right because of the way that they were born?

This is not about schools, in spite of what the Yes On 1 campaign would have you believe. The Maine Attorney General studied the question at the request of educators, and replied: “Our answer frankly is ‘no,’ there is no impact on the curricula of Maine’s public schools.”

Even if it was about the schools, I’m not sure I see how Proposition 1 represents a solution. Denying same-sex couples the right to marry does not remove them from the state. Children in cities and towns all over Maine are far more resilient and perceptive than we give them credit for, and more importantly are already aware that some households have two moms or two dads. Permitting these couples to marry will improve their lives with no material impact on the lives of children whatsoever.

Nor is this a question about imposing values on religious communities, or the threat of a flood of litigation. As the Portland Press Herald observed,

“Arguments that same-sex marriage would inhibit religious freedom or cause a flood of lawsuits also fall flat. The same claims were made in campaigns against Maine’s anti-discrimination laws and neither of them came true. Maine has strong exemptions for religious organizations in its employment and housing laws, and the marriage law would not require anyone to preside over a ceremony in violation of his or her religious beliefs.

Last year, only 32 out of 1,394 civil rights complaints to the Maine Human Rights commission were based on sexual orientation, and few, if any of them, are ever likely to end up in court. The marriage statute would not provide any new grounds for lawsuits.”

When you remove those arguments – as you must if you are committed to rational discourse – what remains? Nothing, except for your own personal or religious objections.

Which you are entitled to. What you are not entitled to, in this country, is the right to impose those beliefs on others. The Declaration of Independence speaks of “certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness.” For many, happiness lies in the institution of marriage. Just as the Supreme Court of the United States once ruled “that separate educational facilities are inherently unequal,” so may we conclude that civil unions, the alternative favored by some same-sex marriage opponents, can never be a reasonable substitute.

You don’t have to like it, and you do not have to approve. All that is required is that you respect the rights promised to them, as you would expect them to respect yours.

We have, sadly, not always lived up to the promise of our forefathers. It took us 191 years to guarantee people the right to marry irrespective of the color skin they were born with. It is my sincere hope that we don’t deny committed couples the right to marry for another 191 years based on the sexual preference they were born with.

I am fortunate that the law says that I may marry the person that I love. I cannot imagine what I would do if it said otherwise. Please. If you are registered here in Maine and you believe in the rights that make this country worth dying for, vote No On 1.

Share This

You’re Going to Do What With My Data?: Privacy and Data as a Product

2009-11-02T16:16:42-08:00

“You may have heard about our [directory assistance] 1-800-GOOG-411 service. Whether or not free-411 is a profitable business unto itself is yet to be seen. I myself am somewhat skeptical. The reason we really did it is because we need to build a great speech-to-text model … that we can use for all kinds of different things, including video search.

The speech recognition experts that we have say: If you want us to build a really robust speech model, we need a lot of phonemes, which is a syllable as spoken by a particular voice with a particular intonation. So we need a lot of people talking, saying things so that we can ultimately train off of that. … So 1-800-GOOG-411 is about that: Getting a bunch of different speech samples so that when you call up or we’re trying to get the voice out of video, we can do it with high accuracy.” – Marissa Mayer, Google

When I talk about the value of telemetry – a technology that allows remote measurement and reporting of information that I commonly use to refer to generated behavioral, usage and other potentially usable data – one of the first questions I usually get is predictable: “just how valuable is it, really?” One of my answers is Mayer’s example, above: “valuable enough that Google built and offered toll-free 411 services to get it.” So mainstream has the concept become that outlets like CNN are writing about it.

Which is right, because data is easily that important.

The Value of Data

Profits, says 37Signals’ Jason Fried in this Inc interview, should be the focus;not followers. Precious few survivors of the first dot com boom and bust cycle will take issue with that. But absent from that perspective is the realization that one may lead to the other far more directly than is commonly understood.

Much has been made of the lack of an obvious revenue model for properties like Twitter, and to a lesser extent, Facebook. But when looking at the organizations’ balance sheets (hypothetically speaking, of course, as most are private), it seems self-evident that the value of the data assets involved is seriously underreported. When you have, as Facebook does, enough data on hand that you can deliver a plausible “gross national happiness” index, it seems inarguable that there’s commercial value to be mined. Not from such a trivial metric, of course: who can reasonably be expected to pay for a “happiness” index? But consider instead the obvious commercial opportunities: is your brand trending up or down? How about your competitor’s? Is it viewed positively or negatively? And so on. The value of catching production issues, questions or concerns in real-time on social-media properties is potentially enormous, given the alternative: waiting to see monthly and quarterly sales fluctuations.

If you’re looking for concrete evidence of the value of this data, look to last week’s Bing and Google Twitter deals. As John Battele says:

Last week was big for Twitter. After years of speculation about whether the company was going to have a business model, Twitter announced two deals at our Web2 conference – first with Microsoft’s Bing, and second with Google. Details of the deals were not disclosed, but as Google’s Marissa Mayers admitted onstage, there were indeed financial terms.

You could also make the argument that Adobe’s $1.8B acquisition of Omniture was made with this very scenario in mind.

The economic value being assigned to data helps to explain why, while being sympathetic to questions about Twitter business models, I’ve never been overwhelmingly concerned. Where the revenue model for the dot com era “eyeballs” strategy was equal parts indistinct and aspirational, the Web 2.0 businesses are being built out in an era of customers increasingly predisposed to analytics and data driven decision making. In other words, there’s a market for their most valuable asset. True, it’s an asset that they merely enable, rather than pay to create, but for most users it’s a fair trade: they get functionality – social networking or otherwise – in exchange for their data, while the providers can mine that data for insights that in turn subsidize the production of the functionality that keeps users engaged and producing data.

Nor is that model limited to the large properties. There is, or will be, markets for data collected from enterprise of all shapes and sizes. Twitter and Facebook’s will be worth more, of course, because volume matters from a statistical perspective. Some datasets will be valuable merely in the aggregate, say your average small business, but the worth of data will continue to rise for businesses – even individuals, in some cases – that generate it. And most will. As asset values rise, the marketplaces that service them will inevitably follow; unless we’ve seen a repeal of the basic laws of economics.

So the future looks bright for data sales. If the would-be sellers and marketplaces can quell the privacy concerns, that is.

Privacy and Monetization of Data

After the conversation about the real value of data, the second most asked question I get is: “but what about the privacy?” As you might expect, given that violations of privacy laws can result in overwhelming financial penalties, jail time, or both. And even if you escape the official law, the court of public opinion is always willing to try and convinct you for questionable decision making, as Facebook learned (again).

But data collection, marketing and sales need not violate privacy laws, nor provoke customer outrage.

After all, it’s already being done, and the villagers have yet to show up with torches and pitchforks. Examples abound. And not just in monolithic, behind the scenes players such as Acxiom. Tim O’Reilly’s been using his book sales data for pattern analysis for years. Google, meanwhile, tells us what the world was thinking with Zeitgeist. The average consumer might not appreciate what, precisely, that means. But those of in the industry should: Google watches – intimately – how we interact with its search engine, and what we do with it. And uses that information, daily.

Even technically savvy and privacy sensitive communities will volunteer to share their telemetry if it provides an obvious benefit. The Debian Popularity Contest is the product of thousands – or is it millions? – of Debian users the world over, each of whom “phones home” their respective package choices. Individually, they are mundane and uninteresting. Collectively, however, the data is very interesting, highlighting as it does adoption and usage trends. How do they deal with the question of privacy? By acknowledging it openly and transparently, theoretical weaknesses and all:

Each popularity-contest host is identified by a random 128bit uuid (MY_HOSTID in /etc/popularity-contest.conf). This uuid is used to track submissions issued by the same host. It should be kept secret. The reports are sent by email or HTTP to the popcon server. The server automatically extracts the report from the email or HTTP and stores it in a database for a maximum of 20 days or until the host sends a new report. This database is readable only by Debian Developers. The emails are readable only by the server admins. Every day, the server computes a summary and post it on . This summary is a merge of all the submissions and does not include uuids.

Known weaknesses of the system:

1) Your submission might be eavesdropped. We evaluate the possibility to use public-key cryptography to protect the submission while in transit.

2) Someone who knows that you are very likely to use a particular package reported by only one person (e.g. you are the maintainer) might infer you are not at home when the package is not reported anymore. However this is only a problem if you are gone for more than two weeks if the computer is shut-down and 23 days if it is let idle.

3) Unofficial and local packages are reported. This can be an issue due to 2) above, especially for custom-build kernel packages. We are evaluating how far we can alleviate this problem.

Debian is far from alone in these practices. The pink dot chart below was a favorite of Sun’s Jonathan Schwartz; each point indicates a piece of Sun software that had fed back basic telemetry to the firm.

Eclipse also collects user telemetry via its Usage Data Collector component. Specifically, according to the FAQ, it aggregates:

Bundles (also known as plug-ins) that are started by the system.
Commands accessed via keyboard shortcuts, and actions invoked via menus or toolbars.
Perspective changes
View and editor open, close, and activation events (activations occur when a view or editor is given focus).

What about the privacy concerns?

“It’s valuable to also note what we do not capture. We do not capture any personal information. We do not capture IP addresses. We do not capture any information that will allows us to identify the source of the information.”

What can you conclude from such data? Who knows? Here’s a non-authoritative (because I don’t know all the user strings), washed spreadsheet detailing the user counts of dynamic editing clients for PHP, Python and Ruby this year from January through October, 2009:

Definitive? Hardly. An interesting datapoint? I think so.

But maybe you’re in an enterprise so tightly firewalled that that telemetry cannot make it back to the aggregating servers; think you’re not being studied? Think again. If your employees visit websites, they are being studied.

As they are if they use software-as-a-service tools like Google Apps, Salesforce.com or any of the 37Signals applications, for that matter.

The short and sweet version of the above is that more and more of what you do is, or will be soon, analyzed. The question is whether or not you believe that to be inherently and irredeemably wrong. It’s largely an academic question, of course. The SaaS vendors, at least, will go on exploiting your usage data whether you want them to or not.

The answer is that data collection and analysis is a tool: nothing more, and nothing less. It can be used for endeavors good, bad and indifferent. Is it consumer friendly when American Express uses data to punish you based on where you shop? Clearly the answer is no. But is it helpful for Google to be able to identify flu outbreaks via search data? Yes.

If data analytics are neither good or bad, then, success – both economically and in terms of public perceptions – will be determined by execution. Consider carefully what you decide to track, how you track it, how closely you guard your users privacy, and where you sell your data and in what form. Because a reputation once lost is not easily regained, as they say. While Facebook has been able to shrug off headlines such as “Facebook’s New Terms Of Service: ‘We Can Do Anything We Want With Your Content. Forever.’,” and concerns about the mining of Gmail accounts seem to be a distant memory, not everyone will be so fortunate. JetBlue and Northwest mostly recovered from their breaches of customer trust, but at what cost? As much as the economic opportunity that data represents will be an incentive, the user’s needs must always come first: be sure that you have a strong user advocate present and empowered in any and all discussions about the usage of telemetry data.

Marketplaces will have an important role to play with respect to privacy, as well. Just as we’re seeing in application stores – at times overzealously, it must be said – data marketplaces will have a responsibility to buyers and sellers alike to carefully consider what they accept and what they turn down. Certain cases will be obvious: no legal marketplace will broker the sales of stolen credit card number databases, for example. But other datasets may seem more innocuous: are offerings that include IP addresses inevitably privacy infringing? Questions like those will probably create a secondary market for data cleansing and normalization vendors; in partnership with marketplaces, they could offer anonymizing data as a service.

Irrespective of the privacy concerns, the collection and analysis of data is accelerating. If data isn’t already an asset you list on your balance sheet, it soon will be. Why not get a head start on the privacy questions, then, and start thinking about what can be sold and what can’t, now? As soon as the marketplaces open on a volume business, you’ll thank me.

Share This

Infochimps and the Inevitability of Data Marketplaces

2009-10-30T14:02:36-07:00

Let’s say you wanted to try and correlate taxi data, as one hedge fund reportedly did recently, with market performance. Where, precisely, would you go for that data?

Exactly.

That’s why we’re going to see, and soon, the rise of data marketplaces. It’s going to be driven by two simple realizations. First, that every well run organization needs to become more creative in its usage of data analytics, because the margins are slimmer and the competition wider. Second, that just about every organization on the planet is carrying a moderately to heavily underutilized asset on its balance sheet in its data. Yes, even your organization.

The solution to both of those problems, quite obviously, lies in facitilated connections between buyers and sellers. Which means marketplaces. Marketplaces like Infochimps.

The site run by Flip Kormer and Joe Kelly promises to “Find Any Dataset in the World,” and is – to my way thinking – the shape of things to come. Currently hosting 5648 datasets, ranging from “Price Indexes for Personal Consumption Expenditures” to “100,000+ official crossword words” to “Resident Population by Race, Hispanic-Origin Status, and Age” to “Retrosheet: Major League Baseball Awards and Honors,” Infochimps is positioning itself as an independent marketplace for data owners to market and sell their datasets. Thanks to Josh of Jones-Dilworth, I spoke to the founders last week and found their ideas of the opportunity ahead similar. And they’re just great guys, to boot.

There are already information marketplaces, to be sure. SAP, as James passed along, sells data via its Information OnDemand store. And Acxiom and the credit bureaus have been trafficing in your details since the 60’s, according to Wikipedia.

But as the tools of data analytics are being democratized, as I mentioned yesterday, so too will the means of data acquisition. Infochimps illustrates this perfectly; if anything’s, it’s too democratic. The licensing – because it’s set by the user, are all over the place – and there’s not much in the way of client integration.

Consider, however, what becomes possible if they were able to build a bridge between Infochimps and something like IBM’s ManyEyes. What would be the difference between this visualization (Java required for both, sorry) of Brunswick temperatures I did a while back with data that I ferretted out from NOAA’s FTP servers:

and this one, which I did a few minutes ago using Austin weather data downloaded from Infochimps?

Nothing except for the time it took to find the data. Well, that and the fact that one’s Brunswick, ME and one’s Austin, TX.

The importance of data marketplaces is less, though, for data like weather that can be retrieved, albeit with some difficulty, than it is for data that’s currently just not made available, period. The success of marketplaces will depend initially on staples like weather and financial data, but like mobile app stores, broaden significantly as critical mass is gained. Eventually we may even see co-op type sales; while the data from one small business, for example, isn’t likely to be that compelling, the information from 500 of them would be. Marketplaces should eventually be able to facilitate such sales.

In the meantime, I’d settle for one place to get my baseball data, because I’m tired of pulling it down manually from sites like this.

Postscript: Many of you probably read the above and are screaming “you’re going to sell my data??? what about my privacy!?!” Which is a fair question, and as a result one that I’ll tackle shortly.

Share This

QOTD: On Rome and NIH

2009-10-29T22:22:05-07:00

I’ve been meaning to post this for a while, because I think it’s the best indictment of NIH I’ve run across, but I had to wait until I hadn’t had a conversation about the subject for a while so that my posting wouldn’t appear directed.

“[I]t should be noted that the main reason for the Romans becoming masters of the world was that, having fought successively against all peoples, they always gave up their own practices as soon as they found better ones.” Montesquieu C., Considerations on the Causes of the Greatness of the Romans and Their Decline, Hacket, 1999, p.24 (Wikipedia)

The moral of the story is…obvious, I should think.

Share This

IOD 2009: What Do Steve Mills and Nate Silver Have in Common?

2009-10-29T22:12:31-07:00

“Baseball Prospectus’ preseason PECOTA projection for the Boston Red Sox: 95-67, second place
Actual record for the Boston Red Sox: 95-67, second place” – Kiss ‘Em Goodbye: Boston Red Sox

Even if you don’t follow baseball, you can probably appreciate that predicting the performance of a baseball team over a 162 game season is a difficult task. Injuries, abnormally good or bad performances, trades, even weather can affect the outcome of any given season significantly. And yet the Baseball Prospectus guys were able to nail it to the game.

Who says predicting the future is hard?

Or maybe you think that sports are too anomalous, too trivial, and that the above is a superficial and ultimately irrelevant example. How, then, do you feel about politics? Because happily, Nate Silver – the original creator of the PECOTA algorithms used above to predict the Boston Red Sox’s unfortunate second place finish – has turned his extraordinary talents to the problem of predicting elections. Does it work? Yep. Really, really well. As Wikipedia puts it, “Silver’s predictions matched the actual results everywhere except in Indiana and the 2nd congressional district of Nebraska, which awards an electoral vote separately from the rest of the state.”

Still not convinced? Consider then, as we have before, the case of Google Flu trends. It’s using data that indicates, as the diagram below shows, a spike in flu related activity here in Maine.

Which seems like an accurate conclusion in the wake of today’s piece in the Portland Press Herald entitled, “H1N1 widespread in Maine, top health official says.”

How are these predictions made possible? By an awareness and ability to use data. Maybe algorithms aren’t the magic bullet, but they’re damn close.

Which brings us back to the conference. When IBM’s head of Software Group Steve Mills talks, it’s usually in the vernacular of enterprise businesses, with a message tailored at same. Which makes sense; speaking in a language your customers understand is good business, and IBM’s been doing pretty well for itself recently. But one of his Monday comments at this week’s Information On Demand show in Las Vegas should, I think, be fairly universally appreciated. So much so that I’m inclined to believe that even Silver, whose business is a fair distance removed from enterprise software, would subscribe.

He said, in part, “that we’re moving into yet another wave of transformation.” More specifically, that “we’ve been living in a decade of process led transformation, and that we’re moving into an era of information led transformation.” Translated, the time of data driven decision making is at hand. So yes, designers: you may occasionally be asked to prove why 3 pixels is better than 5.

It would be naive, of course, to suggest that data driven decision making is somehow new. Repeat after me: there’s nothing new in technology. Everything from where products are placed within a supermarket to when your favorite TV shows are placed on the calendar is, and has been for a long while now, determined by a particular set of numbers. What’s different these days is that we have exponentially more numbers than we did before, better tools to attack them, and an understanding that if you aren’t using the numbers to improve your business – every aspect of it – your competitors will.

There are three primary components to this transition: market, server and client. There’s the data, too, but that deserves a post of its own.

Market

The technology industry has for the past several years been long on storage growth trivia, detailing for us all by the byte the incomprehensible amounts of data being generated yearly, monthly, even hourly. What we’ve been short on, however, have been stories about how that data is actually being put to good use. The idea certainly isn’t new; warehousing and business intelligence have been staples of the technology industry for decades. But just as cloud computing makes available resource volumes and types that were once out of reach of all but the largest businesses, the growing availability of better and lower cost technologies with an ever increasing supply of datasets dramatically upleveled the visibility of analytics for the average technologist. Perhaps more importantly, it’s received a boost from people like Silver and bestsellers like Freakonomics. For what seems like the first time you can have a conversation with a non-technologist about data and they get it. That’s important, because it widens the addressable market from numbers geeks to include people with actual budget. Not a bad development.

Server

One of the conversations I had at IOD followed a path you’ve heard here before many times: the breadth of persistence options for application developers has never been wider, and while the relational database will remain a popular option for years to come, the days of it being the only option are almost at an end. Consider IBM. For all of its investments in the relational space, the folks from New York seem to get the complementary roles non-relational tools can play as well as anyone; from CouchDB, where they employ Damien Katz, to Hadoop, where their M2 demo is one of the more interesting I’ve seen recently to Cassandra, which they are apparently contributing to. All three of which are part of – along with other projects like Drizzle, MongoDB, Riak, Tokyo Cabinet, and Voldemort – a new set of alternatives.

Whether you know the category as NoSQL or AltDB doesn’t really matter – they’ll still smell as sweet. And be as good a fit for those workloads that just aren’t quite appropriate for the relational databases you grew up with.

Fueling their respective popularity, not surprisingly, is the fact that most of the NoSQL/AltDB options are open source. The tools for making sense of mega data are available to anyone and everyone, which is a far cry from the days when warehouses meant an early retirement for data center systems integrators and software salespeople.

Client

It’s not enough, as anyone who’s viewed raw Apache server logs can tell you, to have access to a lot of data. You need a way to make sense of it; to parse it and present it back in a meaningful fashion. As far as we’ve come on the server side is as far as we’ll need to go with the client. True, the democratization of Business Intelligence tools is well underway – just take a look at what Flowing Data can do with information. It’s so useful it’s pretty. Excel may be enough for geniuses like Bill James, but us mere mortals will needs some help seeing the patterns. Think ManyEyes, but even more accessible.

Look for data visualization clients to be the next category for data driven innovation, then. Because it’s great to be able to manipulate the data, but it’s even better to be able to use it. With something this side of a Java query, I mean.

Disclosure: IBM, the organizer of the IOD show, is a client and comped hotel. Cloudera, a commercial sponsor of the Hadoop project, Basho (Riak), and Sun (Drizzle) are also RedMonk clients.

Share This