DeWitt Clinton

Notes on O’Reilly Radar’s State of the Computer Book Market

2009-02-27T19:47:11Z

For the past several years one of my favorite places to track programming language trends has been the “State of the Computer Book Market” series on O’Reilly Radar.

O’Reilly’s Mike Hendrickson dives deep again this year into the statistics and details of the computer book market in a 5-part series:

My high level summary:

While general book sales were up YoY in 2008 (+30%), computer book sales were down (-8%) with a weak Q3 and Q4.
The only computer book category to grow in 2008 was Consumer Operating Systems, driven by sales of Mac OS X and iPhone books.
The top sellers in the Systems and Programming category were in C#, Mac Programming, and Virtualization. Software Project Management, Windows Administration, and Software Design books were all down.
C# book sales continued to grow for the 4th straight year, passing sales of Java books for the first time (which have been sliding for 5 straight years).
C/C++ and Visual Basic book sales both posted their 5th and 4th straight declines, respectively.
Python rose for the 5th straight year, as did ActionScript (4th straight gain), whereas JavaScript book sales fell for the 3rd straight year.
Online e-book sales (often direct to consumer) are growing strong relative to the overall market

Languages ranked by 2008 book sales (%market share, relative to 2007 rank):

C# – 15.58% (↑)
Java – 12.09% (↓)
PHP – 9.93% (⇈)
JavaScript – 9.89% (↓)
C/C++ – 8.36% (↓)
ActionScript – 5.76% (⇈)
.NET Languages – 5.40% (↓)
VisualBasic – 5.04% (↓)
SQL – 4.57% (↓)
Ruby – 3.51% (↓)
Python – 3.41% (↑)
VBA – 3.18% (↓)
Objective-C – 2.56% (⇈)
Perl – 2.14% (↓)

The most telling charts from the series:

2008 Market Share by Language:

A Treemap view of the Programming Languages:

Percentage of 5 Year Sales Per Quarter by Media Type:

(Images copyright O’Reilly Media, and used without asking permission first — will definitely take them down if necessary.)

Mike goes into far more detail on each topic over on Radar. Start reading the series here.

A Survey of Rel Values on the Web

2009-02-17T06:19:25Z

One of the interesting things about sharing an office with Jyri is that our free-association stream-of-consciousness conversations often lead to places worth exploring further.

On Friday Jyri and I started wondering about the link rel values documented in the XFN 1.1 profile, which include not only the relatively commonplace me and friend values, but also such unconventional values such as colleague, muse, and spouse. But how frequently are the lesser known rel values really used? Rather than speculate blindly, I wrote a simple mapreduce to check the web and find out for sure.

The mapreduce scanned approximately 177 million recently crawled HTML documents, parsing and counting rel values in link and anchor tags along the way. In those 177M documents, I found just over 19 billion <a> and <link> tags in total. And of those 19B tags, 1.8 billion of them contained a non-empty rel attribute.

Following the HTML5 rules for space separated tokens I split each rel value on [\s\t\n\r\f] and extracted each individual value. In total, over 1.9B instances of rel values were found, or an average of just over 10 per HTML document (with some tags having more than one rel value).

I found a staggering 1.8M unique rel value strings in use, with many used only once or twice across all the web. In fact, the top 6 most-frequently-used rel values accounted for 80% of all usage, and the top 11 alone were responsible for 90% of all usage. In fact, less than 1000 of the most frequently unique rel values are sufficient to represent the 99th percentile of all usage. In other words, the tail is long indeed, with the remainder of those 1.8M unique rel values accounting for less than 1% of the total usage.

In passing, I noticed that approximately 3 million rel value strings also contained a comma character; presumably cases where the author may mistakenly have thought that the "," character would be used as a delimiter. However, since these cases account for just 0.18% of all rel value strings, they have little impact in the overall totals.

Here are the top 25 rel values found in <a> and <link> tags in a moderately sized sample of the web today:

Rank	Value	Count	Relative Frequency
1	nofollow	832980014
2	stylesheet	338648161
3	tag	168764800
4	alternate	109150404
5	icon	69183607
6	chapter	56395793
7	forum	55920646
8	shortcut	53906964
9	bookmark	30683701
10	archives	25381711
11	category	24361195
12	external	19181232
13	search	14227485
14	edituri	8109835
15	apple-touch-icon	6753583
16	help	4842211
17	prev	4537344
18	next	4390373
19	pingback	4302068
20	wlwmanifest	4125573
21	contents	3959350
22	contact	3504587
23	service.post	2678873
24	top	2502015
25	me	2501273

The most frequently used values are not surprising at all. The nofollow value is used as a hint to search engines that the target of an <a> tag should not be used in ranking calculations. The stylesheet value is used on <link> tags to indicate that the target is an external CSS document. The tag is a microformat used to indicate a category for the page, as popularized by sites such as Technorati and Delicious. And alternate is frequently used to facilitate the autodiscovery of an RSS or Atom feed for a given site.

Further down we learn that as OpenID continues to gain in adoption the openid.server and openid.delegate rel values come in at #35 and #43 respectively — impressive, since each are only needed once per-page. And even the newer OpenID2-style tags not far behind, with openid2.provider and openid2.local_id reaching #51 and #837 respectively.

Near and dear to my heart, I was pleased to see the search rel value, the OpenSearch discovery mechanism, ranked so high at #13. Again these discovery links are only needed once per page; a sign of strong adoption. Admittedly, not all rel="search" links are OpenSearch related, but I have another more comprehensive analysis of OpenSearch documents that shows similarly pervasive adoption rates.

Even the newly agreed-upon canonical rel value makes a showing at #271, and will surely rise to the top 25 or so over the next year or two.

And the XFN rel values? The contact rel value is the most common at #22, with me and friend just behind at #25 and #28 respectively. Filling out the list are acquaintance (#58), met (#68), colleague (#84), co-worker (#126), neighbor (#180), muse (#196), co-resident (#232), parent (#255), sibling (#414), sweetheart (#446), spouse (#570), crush (#794), kin (#834), child (#879), with date bringing up the rear at #1086.

This survey indicates that rel values are both widely and meaningfully used, with adoption being driven by a wide array of needs, such as semantic markup, search engine hints, client-side rendering, discovery and identity protocols, blogging, and/or content that can be later edited.

But more importantly, we learned that a full 0.0003% of all the links have declared, for all the world to see, that some URI out there is their source of inspiration, their Calliope, their Erato, their muse.

Unbelievable Mental Lapse

2009-01-26T18:44:27Z

I receive a fair bit of misaddressed mail at my gmail.com addresses. Sometimes it is the result of a typo on the part of the sender. But with surprising frequency it is the result of a real person accidentally entering my email address into a web form instead of their own. I’ve seen shipping confirmations, subscriptions to mailing lists, responses to job applications, new account notices on sites like Facebook and Twitter, etc. How someone could enter the wrong email address into one of these forms is beyond me.

But nothing, nothing will ever top the mental lapses responsible for this email I just received:

Internal Revenue Service <refunds@irs.gov>
Date: Mon, Jan 26, 2009 at 9:46 AM
To: DClinton@gmail.com
Dear Dianne Clinton,

Your Stimulus Payment request has beed submited.

A Stimulus Payment can be delayed for a variety of reasons.
For example submitting invalid records or applying after the deadline.

Stimulus Payment request issuer:

Name: Dianne Clinton
Address: [redacted]
City:  [redacted]
State: [redacted]
Postal Code:  [redacted]
Phone: [redacted]
Date of birth:  [redacted]/ [redacted]  (mmddyyyy)
Social Security Number:  [redacted]
Mother name:  [redacted]
Credit card Number:  [redacted]
Credit card expiration:  [redacted]/ [redacted]  (mm/yyyy)
CVV:  [redacted]

Note: For security reasons, we recorded your ip-address, the date and
time.
Deliberate wrong inputs are criminally pursued.
IP:  [redacted]
Date: Mon Jan 26, 2009 6:46 pm

Regards,
Internal Revenue Service

Yes, every single one of those [redacted] fields was filled out completely. Social security #, credit card #, mother’s maiden name. The works. An identity thief could clear out her accounts and bankrupt her by morning.

Want to know the saddest part?

It was an identity thief. (The grammar and spelling errors were a bit of a dead giveaway. Besides, I can’t imagine the real IRS would be so stupid as to send your private details back to you over plain-text email.)

A ten-second perusal of the address headers showed that, not-surprisingly, this message did not originate from irs.gov.

Rather, the mail originated from this site: (And needless to say, don’t you go filling it out!)

http://www.ieaf.es/bbdd/apps/news/stimulus.refund/stimulus.php

This woman was the victim of a phishing scam; she probably thought she was entering her very personal data into a legitimate United States government website, and she may never realize how wrong she was. She didn’t notice the lack of https, or that the domain was ieaf.es, a known IRS phishing site, hosted on a Spanish top-level domain.

I will submit the site to the various phish-tracking websites and make the appropriate notifications at work. That said, I’m on the fence about trying to contact her directly. Morally it would be the right thing to do. However, in this litigious era, it might be exactly the wrong thing to do. Needless to say the email itself will be permanently deleted from my inbox.

This whole episode makes me very, very sad.

Sampling Twitter

2009-01-03T03:37:40Z

Update: I had made a mistake in gathering the samples and took this post down temporarily to fetch new data. A typo I made in one of my scripts radically under-counted the number of friends each account had. Mea culpa, and the lesson I learned is that if a number seems counter-intuitive, then it likely is. I apologize for the mistake!

…

Most of the surveys that have attempted to study Twitter usage do so by scraping the public stream of tweets. This provides reasonable data about how people are publicly using Twitter but it suffers from sample bias insofar as only active Twitter accounts are counted while private accounts and accounts that go dormant are overlooked.

After Dion posted that I was the first person he subscribed to on Twitter, a conversation started in which we speculated about the distribution of used vs. unused user IDs on Twitter. Specifically, I became curious as to how Dion, who signed up for Twitter less than three months after I did, could have a user id 3.5 million higher than mine. Was Twitter really signing up users at a rate of more than one million a month this time last year? Doubtful, but the only way to find out was to gather some hard data.

Instead of mining data from the public feed, I wrote a short script to query a sample across of all the possible Twitter ids. I first created a a test Twitter account, which was assigned a new user ID of 18496098. Using this id as an upper bound on the population of all Twitter IDs, I selected samples at random from the range (0, 18496098) (exclusive), and queried the Twitter API at a metered rate from several machines over the course of a day.

After rerunning the script on queries that returned transient server-side or client-side errors ("502 Bad Gateway", "503 Service Temporarily Unavailable", etc), I arrived at an clean, unbiased sample pool of 4414 ids.

Of the 4414 sampled ids, 1270 ids have been assigned (returned "200 OK") and 3144 are not in use (returned "404 Not Found"). Of the 3144 unassigned ids, 3120 of those were “Not found” (and presumably were never used), and 24 were “User has been suspended”.

By this ratio, we can infer that approximately 5,000,000-5,500,000 accounts have been created since Twitter’s private launch in early 2006.

Of the 1270 sampled ids that have been assigned to a user, 847 accounts have posted at least one update, 759 are being followed, and 735 are following another user.

Further breaking down the 1270 assigned ids, we find that 635 ids are both being followed and follow someone else, 574 ids have posted a status and are being followed, and another 541 have posted a status and follow someone else. And 501 have posted a status, follow someone else, and are being followed.

Of the 1270 ids that have been assigned, 97 have protected their status updates, and 1173 have left their status updates public (the default).

Of the 1173 public ids, 1048 of the accounts were created more than 30 days ago.

Of the 1048 more-than-30-day-old public ids, 691 have posted a status message at least once.

And of those 691 sampled public ids that are over 30 days old and have posted at least one status message, 305 of those accounts have returned to post an update more than 30 days after their account was first created.

This last metric — users that have returned to post again more than 30 days after creating their account — is the best metric I can come up with for a return user on Twitter. Extrapolating this ratio of return vs non-return users back across the segment of users too new to test and the private accounts, we estimate that 29.1% of assigned ids, and thus roughly 8.3% of all possible ids, are assigned to someone that has returned at least to post something more than 30 days their account was initially created.

Given an potential max population of 18,496,098, this ratio implies that there are up to 1,500,000-1,600,000 users that have returned to Twitter to post again after first creating their account, which is a respectable number, and is consistent with the estimates made by others observing the pattern of public status updates.

Of the 305 return accounts, 249 are both following at least one other account and have at least one follower.

Again extrapolating for accounts too new to test and private accounts, this suggests that 23% of all assigned ids, and thus 6.8% of all potential user ids, are assigned to someone who is posting regularly, is following other users, and is being followed by at least one other user. This implies that there there are up to 1,200,000-1,300,000 active, connected users on Twitter.

Of the public users sampled, 470 are followed by no one, 524 are followed by between 1 and 10 people, 159 are followed by 11 to 100 people, and 20 are followed by more than 100.

Of the public users sampled, 499 are following no one, 514 are following between 1 and 10 people, and 139 are following 11 to 100 people, 20 are following more than 101-1000 people, and 1 is following more than 1000.

Of the public users sampled, 403 have no status updates, 535 have posted between 1 and 10 status updates, 141 have posted between 11-100 status updates, 87 have posted between 101-1000 status updates, 6 have posted between 1001-10000 status updates, and 1 has posted more than 10,000 status updates. (The outlier with 10000+ updates is a bot.)

And to return to the question that started the experiment, there is indeed both an upward trend to the number of users created per month, and a sharp transition in early 2008 when huge blocks of ids no longer went unassigned between the creation of each account.

These numbers, while not perfect, should be a reasonably accurate ballpark estimate — ±2% within 2σ over the total population, and ±3% over population of assigned ids — and the numbers likely wouldn’t change significantly with a larger sample. However, there’s always the chance of mistakes, so please feel free to download the data set to confirm. Or if you work at Twitter and would like to verify these numbers, even privately, I’d love to hear how close the sampled numbers come to reality.

Sunsetting Delancey

2008-12-28T22:42:55Z

Three years ago I launched a little service called Delancey. Delancey was an early del.icio.us mashup that kept track of how many times you clicked on each bookmark. This usage metadata was valuable because with it you could sort your bookmarks in order of how often they were used, making for a simple but powerful default browser homepage that learned from your behavior.

In designing Delancey I made several decisions that I knew might eventually cause a maintenance challenge, but I felt that they were justified at the time because of the benefit that they offered.

The first design decision was that users of Delancey would be able to “claim” their del.icio.us account so that they, and only they, would be able to access or update their personal click data. Given that a standard mechanism for claiming external identities was still in its infancy, I hacked together a technique by which a signed-in del.icio.us user would automatically bookmark a secret claim URL that the Delancey application would verify and then delete. It was a surprisingly effective approach that created a strong verifiable claim over a del.icio.us identity without the Delancey application ever requesting the del.icio.us user’s password. However, it took advantage of how the del.icio.us front-end was implemented at that particular moment, and hence it was unlikely to remain stable forever.

The second design decision was that the Delancey application would never store a user’s del.icio.us username or the URLs of the bookmarks the user accessed. This wasn’t strong security, as both were stored as simple one-way md5 hashes of the plaintext data, however it prevented casual abuse as reversing the associations would require a complete dictionary of del.icio.us usernames and the bookmarked URLs. (A truly secure system would require a secret key for each Delancey user and a mechanism to encrypt the data on the client side — a reasonable exercise, but overkill for this type of application.) This kept your data private (even from Delancey itself), but it meant that exporting it would be more difficult down the road.

Then del.icio.us relaunched as delicious.com this fall. I suspected that Delancey would break during the transition, but I couldn’t find a migration guide for del.icio.us developers so I wasn’t exactly sure what to be on the lookout for. So I shelved the project for a while longer and hoped that nothing serious was broken.

This weekend I had the opportunity to investigate, and sure enough, a few of the features that made Delancey possible were no longer supported. Most critically, the automatic bookmarking trick that Delancey used for claim verification was rendered ineffective because the new delicious.com front-end signs (with a key or nonce) the form used to post new URLs, thus blocking Delancey’s attempt to mimic the POST request via an external page. This is a perfectly reasonable decision on the part of of the delicious team, but without it Delancey would need a new mechanism for account verification.

The Delicious API does provide a way of posting new bookmarks, and hence a backdoor mechanism for account verification. However, the official API relies on HTTP Basic auth, which would mean the user would be presented with an unexpected browser-based interstitial login box if Delancey were to use it. Or worse, Delancey application could itself request and proxy credentials for the del.icio.us user — exactly what I was trying to avoid. The latter technique is a form of the password anti-pattern, and I couldn’t in good faith implement that anti-pattern myself. Fortunately, either (or both) of OpenID and OAuth would be sufficient for verifying the account claim if delicious supported them. Unfortunately, Delicious currently supports neither OpenID nor OAuth as far as I can tell. While I’m reasonably confident that I could come with some other hacked-together solution on top of the current delicious.com front-end, I don’t feel comfortable again investing in a solution that isn’t officially supported.

So with that as background, I’ve decided to sunset Delancey. I don’t have a hard date as to when I’ll shut it down, but I won’t be adding new features, nor will I be fixing any bugs beyond those that I fixed today. I promise not to take it down for another 90 days (through the end of March, 2009), but after that I may turn it off completely to save bandwidth. However, before I take it down I will provide links to export your existing click data. That said, since the bookmark URLs are stored as hashes only, you will need to do some work on your end to associate the underlying URL with the click counts. (This is possible because you know your own username and you can easily retrieve a list of all of your bookmarks from delicious.)

For those that want to start extracting their data now, you can play around with the auto-documented Delancey API. For example, to see a list of your tags, you can use the /delancey/tags/{username} method (e.g., as YAML or as JSON). And you can retrieve your click counts via the /delancey/bookmarks/{username}/{tag}/ method (eg. as YAML or as JSON). (You’ll note that the latter API call automatically does the hash association with the public bookmark data — if the del.icio.us account is private, or if the bookmarks are deeply buried, then the titles and the URLs are unavailable. The full Delancey export will contain on the (hash, count) tuple.)

Going forward, if Delicious implements OAuth or OpenID in the next month or two I may change my mind and get this running again, perhaps by porting it over to Python and App Engine in the process (a port I started a while back but never completed). If not, then I hope people understand my reasons for shutting down this service. It was a fun app to build, and if you used it I hope you found it valuable for what it did. I still believe that there is great opportunity in the social bookmarking space, and whether it is to be found the relaunched delicious.com, at a site like digg.com, or at a site yet unseen, I look forward to whomever pushes the technology forward next.

Mobile devices as development machines

2008-11-17T20:29:05Z

Tim O’Reilly tells the story of how Vic Gundotra, my manager at Google, came to the conclusion that the future of software lies somewhere in the intersection of mobile and cloud. Indeed it does, but I jokingly twittered back that we still need PCs — how else are we going to write our mobile and cloud apps?

But in truth we hardly need the heavyweight hardware under our desks to develop software even today. If anything, with a ubiquitous network connection (wifi where available, 3G where it is not), most of the heavy lifting can and should be run on compiler farms, not the desktop. My own development workflow typically involves ssh’ing into a virtual instance from whatever client I’m on (a desktop, a laptop, etc) and running on a distributed compute cluster. Depending on the task at hand even a full graphical IDE can be run remotely over technologies like NX. Or if you’re like me, ssh, screen, and emacs are all you need. And I believe we may even be headed for an era when the IDEs themselves are hosted applications, following the pattern established by email, text editors, and other traditional desktop applications.

So will this mean you can forgo the PC even for software development? Perhaps it does. Looking back, here were the specs on my first personal Linux box, as pieced together in 1994:

Dell Dimension Intel Pentium @ 133 MHz 256MB RAM 1GB HDD 28.8 kbps modem Slackware 2.0 Linux kernel 1.1.18 17″ Dell Trinitron CRT Monitor @ 1024×768

I used this machine throughout my time as a CS undergrad, including the time off I took to write software for one of the first web companies (tripod.com).

By way of comparison, I now carry a HTC G1 device running Android:

HTC G1 Qualcomm MSM7201A @ 528 MHz 192 MB RAM / 256 MB ROM 1GB SSD 3G HSPA/WCDMA, GSM/GPRS/EDGE 802.11b/g, Bluetooth 2.0, USB 2.0 Android RC30 Linux kernel 2.6.25 3.2″ screen @ 320 x 480

These stats are telling: The mobile devices in our pockets are more powerful and more connected in every way than the desktop machines we used to build the first generation of the web. If we can just bridge the peripheral gap and get high resolution external displays, then there is truly nothing stopping us from forgoing the desktop completely in favor of a completely distributed development environment.

Along those lines, I recently purchased an Asus EEE PC 1000 and installed Ubuntu on it. With a 1.6 GHz Intel Atom processor, 2GB RAM, and a 40GB SSD, that little netbook is already faster and more powerful than the desktop PCs we ran just several years ago. As we can expect to see that caliber of hardware in cellphone-sized mobile devices before long, I suspect it is only a short matter of time before we stop thinking in terms of “developer workstations” at all, and instead start thinking in terms of mobile devices as access points to a limitless development environment.

User-Agent headers are out of control

2008-11-09T07:49:16Z

Here’s an actual User-Agent header from my logs:

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; 
  .NET CLR 2.0.50727; Tablet PC 2.0; .NET CLR 3.5.21022; 
  Media Center PC 5.1; Zune 3.0; OfficeLiveConnector.1.2; 
  .NET CLR 3.5.30729; .NET CLR 3.0.30618)

This is absurd.

And it doesn’t help. My server does not care if you own a Zune or have OfficeLiveConnector 1.2 installed. Most servers don’t. And if they do care, there are better mechanisms for negotiating content.

Starting with the next version of every major browser, the User-Agent should simply read:

browser version

For example:

Internet Explorer 8.0
Firefox 3.2
Chrome 0.4
Safari 4.0
Opera 10.0

And that’s it.

None of this “compatible” nonsense (all lies anyway), no more details about the rendering engine. Don’t even include a hint about the platform (Windows, OSX, Linux, etc). If the browser supports multiple operating systems, then it should support them all equally well enough that the server doesn’t need to know.

If necessary, the browser should instead use Accept headers to send stronger signals about what type of content it desires.

Stop the User-Agent madness now. Think of the bytes.

Clearing up inaccuracies about the Google OpenID IDP launch

2008-11-10T15:07:53Z

Not exactly the post I want to write after a long week, but there have been several posts following the developer launch of the Google OpenID IDP. Unfortunately, those posts are inaccurate, and it is worth clarifying what is going on so no one else makes the same mistake and repeats the (incorrect) claim that Google somehow “forked” OpenID.

The first criticism has something to do with the erroneous notion that, simply because the Google IDP supports the indirect lookup of a user’s identity, this is somehow an invalid use of the OpenID protocol. On the contrary, it is using a standard technique known as Directed Identity, which can be found in the OpenID 2.0 specification here. Directed Identity allows users to enter a generic domain name (e.g.., “example.com”), rather than a fully qualified identity (e.g., “example.com/users/bob”), so that they can use their identity provider to make an informed decision about how much personal information to expose to the RP. This is a good thing. You want this. You want to be able to make that disclosure choice yourself.

Another (again mistaken) claim is that because, during the test launch, Google whitelisted the initial set of allowed Relying Parties, that the specification was somehow violated. On the contrary, there is nothing in the spec that requires an IDP to provide access to user data to all RPs (nor should it!), and in the case of the Google launch, the whitelisting was just planned for the initial launch anyway. The initial launch went well, and all RPs are now whitelisted. (This wasn’t a reversal of policy, this was the plan all along.)

The third inaccuracy stems from the second. During the whitelist-only test launch, Google could not publish an XRDS document at the root of the gmail.com domain for all users because it would fail for users using it on non-whitelisted RPs. I can see how this would be confusing if one didn’t know how OpenID 2.0 works with XRDS, but it was the right decision. And now that everyone is whitelisted, Google can publish the XRDS document on the root of gmail.com. (I don’t think that part has launched yet — in the meantime, you can use the explicit path of “https://www.google.com/accounts/o8/id” to fetch the XRDS document.)

And last, there seems to be a huge misunderstanding about email addresses vs. URLs as identifiers. URLs make fantastic identifiers — for the 0.1% of the web population that understands that they “are” a URL. Fortunately, the other 99.9% of the world (our parents, for example) already understand that they have an email address. As has become increasingly clear to everybody doing usability research on OpenID (see here and here), we absolutely need to provide mechanisms for mapping human-friendly identifiers like email addresses to identities. That’s not to say that URLs-as-identifiers should go away. On the contrary, I myself use “dewitt.unto.net” as my identifier, and fortunately, the spec is smart to allow for multiple ways of surfacing that identity. (Or more exotically, I expect we’ll see great things to come with mobile phones or phone numbers as identifiers as well.)

While I don’t fully understand the motivation behind the attacks made in those articles, I do find it rather unfortunate that they’ve been picked up and repeated by people who are similarly unfamiliar with the technology itself. I guess a lesson might be to pause for a second before repeating a rumor you don’t personally understand.

The truth is that OpenID is still in its teenage years and still has a lot of growing up to do, but in the last few months we’ve several of the biggest sites on the web launch as identity providers (nearly everyone on the web has an OpenID now), and continued improvements and additions to the core specifications, and groundbreaking usability research. Bumps along the way notwithstanding, this is forward progress.

Update:

Kevin Marks has a great response about whether or not, as Ben Metcalfe says below, OpenID forked itself by turning into more than a mechanism for claiming resolvable URLs.

I’m personally of mixed opinions about this myself, as I don’t want those more advanced use cases for OpenID overshadow the original role of OpenID as a standard for claiming URLs. In fact, I was among those not entirely in favor of putting these (admittedly valuable) new concepts into the core OpenID 2.0 spec, and would have preferred independent extensions instead. As people try and do more and more with OpenID — such as SSO, profile exchange, and non-URL-based identity mappings — this can indeed lead to what Dare calls the “square peg in a round hole” effect. But to be crystal clear, no one is forking the spec just by implementing the spec in a way it allows for.

And I stand duly chided by Kevin for the rhetorical exaggeration about whether or not the mainstream user can learn to type in a URL to identify themselves. The truth is that no one knows for sure if they can, though the research is showing that that those of us (myself included) who argued for URLs exclusively were probably hoping for too much. But as long as we can always map the user-driven identifiers back to URLs, then I do believe we get the best of both worlds.

As Kevin says, see you all at IIW!

Disclosure: I work for Google, and I represent Google on the OpenID Foundation board. That said, I’m really writing this as a member of the community, just trying to figure out how this story got so mixed up to begin with.

Microblogging syndication formats

2008-08-17T02:49:42Z

I’ve been thinking a little bit about the syndication formats for microblogs lately, and I’m wondering if we might all agree upon some very simple conventions to integrate the various providers. Nothing profound here, just some simple ideas about how to get an easy win with existing technologies.

Taking FriendFeed as the best current example of an aggregation site, I notice that the FriendFeed team has hand-coded various content providers, but they don’t have the same richness enabled for generic feeds.

For example, take Dave Winer’s stream on FriendFeed. His FriendFeed page looks pretty darn good, doesn’t it? (I pick Dave because he does more to syndicate various content sources than just about anyone.)

But then notice that his native FriendFeed bookmarks, his Amazon wishlist, and his Flickr photos all have rich content syndicated and look stunning, but his basic blog feed — itself a very rich content source — is comparably lacking in how it is represented in the aggregated stream.

It’s the difference between this:

And this:

Why is this? Dave’s blog, syndicated over RSS, has plenty of great data to display. Only the blog post title is displayed — missing are the images, an icon, a summary, and more. Couldn’t those be captured as well?

As a strawman proposal, what if content aggregation sites agree to display the following standard elements from RSS feeds:

<?xml version="1.0"?>
<rss version="2.0">
 <channel>
  <title>Scripting News</title>
  <link>http://www.scripting.com/</link>
  <description>
    Dave Winer&apos;s weblog, started in April 1997, 
    bootstrapped the blogging revolution.
  </description>
  <language>en-us</language>
  <copyright>Copyright 1997-2008 Dave Winer</copyright>
  <pubDate>Fri, 15 Aug 2008 07:00:00 GMT</pubDate>
  <lastBuildDate>Fri, 15 Aug 2008 20:35:00 GMT</lastBuildDate>
  <docs>http://cyber.law.harvard.edu/rss/rss.html</docs>
  <generator>OPML Editor v0.73</generator>
  <managingEditor>scriptingnewsmail@gmail.com</managingEditor>
  <webMaster>scriptingnewsmail@gmail.com</webMaster>
  <item>
   <title>Perfect timing!</title>
   <link>
     http://www.scripting.com/stories/2008/08/15/perfectTiming.html
   </link>
   <guid>
     http://www.scripting.com/stories/2008/08/15/perfectTiming.html
   </guid>
   <comments>
     http://www.scripting.com/stories/2008/08/15/perfectTiming.html#disqus_thread
   </comments>
   <description>
      I just read this [...]
   </description>
   <enclosure url="http://www.scripting.com/mp3s/weatherReportSuite.mp3" 
     length="12216320" type="audio/mpeg" />
   <pubDate>Fri, 15 Aug 2008 18:23:10 GMT</pubDate>
  </item>
 </channel>
</rss>

From this feed we’d want to consider the following RSS elements:

rss/channel/title – the name of the blog
rss/channel/link – the link to the blog
rss/channel/item/title – the name of the entry
rss/channel/item/link – the link to the entry
rss/channel/item/description – extract the first 512 characters as the first comment
rss/channel/item/enclosure - inlined media types (images, mp3, video)

And for the Atom case the corresponding elements would be:

feed/title – the name of the blog
feed/link – the link to the blog
feed/author/name – the name of the author
feed/entry/title – the name of the entry
feed/entry/link@rel="alternate" – the link to the the entry
feed/entry/link@rel="enclosure" – inlined media types (images, mp3, video)
feed/entry/summary – extract the first 512 characters as the first comment
feed/entry/content – use this if the summary doesn’t exist

For bonus points in both cases, parse the HTML content of the rss/channel/item/description or feed/entry/summary elements for <img/> elements, extract them, and display them as images along with the post.

And for a custom icon, use the host’s /favicon.ico.

Put all that together and we’d have the following, all without any manual intervention on the aggregator’s side:

This is just the beginning — I feel I’m only scratching the surface of what can be extracted from existing syndication formats. For example, comment stream aggregation (via the comments element or RFC 4685 autodiscovery) is a great next step after this. And I only call out FriendFeed because they’re the best at aggregating multiple content sources, but these concepts apply to any content aggregator, and finding a way to reuse existing formats like RSS and Atom to create rich presentations automatically will enable us to do more with less manual work between aggregators and publishers.

Also note that this isn’t intended to replace a standard format for activity streams. Those are something slightly different, and I know efforts are underway to define conventions there. Rather, this is simply an effort to get more out of the syndication formats we already publish, and in doing so, allow for aggregators like FriendFeed to present a richer experience to their users without requiring as much manual tuning per content source.

Thoughts?

Update:

Bret notes:

We do pull out images and videos for entries that use Media RSS, but those tend to be video and photo sites (the spirit of the Media RSS format is that the post *is* a photo, as opposed to the use case highlighted here, where the picture is simply a thumbnail or accompanying a larger post).

Good point, and sounds like a great thing to do.

On Fighting the Web Itself

2008-08-08T23:42:45Z

Earlier this week I posted a short piece about end user licensing questions in Silverlight. I discussed potential incompatibilities in the Beta 2 licensing terms with respect to open source software development. For most people this is a niche concern, a curiosity, but not something they would normally be thinking about. Besides, one expects that these problems will get fixed in the final Silverlight EULA. So why raise the question at all?

The short answer is that the technology behind Silverlight, and most certainly the company creating it, has the potential of changing how the web itself works.

If you’re a web developer then you’ve felt the acute pain involved in writing applications inside the browser. Even armed with the most state-of-the-art toolkits, such as jQuery, Dojo, etc., you’re still limited to the available runtime of HTML, CSS, and JS, and worse, the absolute morass of cross-browser incompatibilities and restricted access to native client-side capabilities. I remain in awe of what people have accomplished in this environment, but I’m sad that this is all we’ve been able to accomplish so far.

The web revs slowly. Very, very slowly. In 10 years we’ve seen virtually no meaningful advances in the the ubiquitous web client; just a painful slog forward as web developers learn to eek out just a little more functionality in a constrained environment. Progress is slow because revving the ubiquitous client requires the coordination of multiple parties, not all of whom have shown consistent interest in working together to move the web forward.

More recently we’ve seen some earnest attempts at breaking that cycle. Rather than wait for the entire web to catch up, projects like Gears seek to rev the client from the inside out. It may take several years for standards like HTML5 to be widely deployed, but if developers can gain a toehold inside the client and start forcing the issue immediately then we’ll quickly see what works and what doesn’t, and be that much more informed about what to standardize and adopt as part of the long-term web platform.

But there’s another approach, an approach best exemplified today by the Flash runtime, whereby one doesn’t seek to improve the web from the inside, but rather replace it entirely. Sure, technologies like Flash take advantage of the web via http-based delivery mechanisms and in that they run inside the browser, and yes, they can use network connections like anything else, but these alternate runtimes fundamentally divorce themselves from the web ecosystem, and in doing so gain a significant advantage, but at a cost.

In spite of circumventing the web — no, because they circumvent the web — these new runtimes have the potential of offering a far better developer experience, and hence, a far better user experience, then the least-common-denominator of the standard widely-deployed ubiquitous browser runtimes of today.

Which leads us to Silverlight: Silverlight is positioned to take the fork-and-forget approach to the web pioneered by Flash and bring to it an unprecedented wealth of technology and corporate might. With a better underlying runtime and VM, better tool support, far superior multi-language capabilities, and more marketing muscle, Silverlight has all the potential to make rapid and noticeable inroads over the next several months, cleaving a large section clean out of the web.

And the scary thing? That this isn’t entirely a bad idea. The browser itself is anemic, the dependency on a single language is a handicap, the security models simultaneously constricting and flawed, the development environments underpowered, and frankly, the whole ecosystem is deserving of a major disruption. We’ve lived too long thinking that what we have today is good enough.

Granted, these technologies won’t be perfect at first. On the contrary, they might be slow, cumbersome to deploy, buggy, and feature deprived. But right now that doesn’t matter. The strategy is all about getting a wedge in place, a bit of leverage that can be used to further pry open a vector for escaping the existing ecosystem. And over time, as the technology improves and adoption grows, so will the size of that tear in the fabric of the web.

But fighting the web is like holding back the ocean; it will route around you or it will wear you down, but will never go away, and it will never tire or give up. Yet in spite of the futility of fighting the web, Silverlight is being positioned in opposition to the web, not in support of it.

Why in opposition to the web? This stems from the principle that the web is axiomatically defined as an open system, where the underlying technologies are resistant to the centralization of control, where the protocols and formats are extensible and malleable, and where the power to effect change is shared and distributed. The DNA of the web is one of ceding control, and of learning to work with, rather than against, the collective wisdom and power a larger community.

Whereas a development monoculture, a centralization of control, and a tight grasp on the ability to change and adapt, all stand against these basic ideals, and give rise to the forces that, given enough time, will erode and eat away at any temporary advantage gained.

A violation of these principles does not necessarily make for a bad technology, but it does make it something other than the web. The winners here will be those that harness the power of the open web, not those who align themselves against it. Fragmentation wastes time and energy and offers little in return other than short term distractions, whereas as a collaboration offers the potential for long term solutions.

The collective forces on the web are not going to sit idly by and let the Internet operating system be fragmented and dominated like we saw in generations past. Difficult lessons were learned, and there is an inherent will about the web that resists all such attempts with striking efficacy. Success is unlikely when everyone is aligned against you.

But the call to action here is not to go and try to fight the disruptive technology. On the contrary, the ideas are sound and the improvements are very much needed. No, the call is to discover ways in which these ideas can become a part of the web, rather than working to tear it apart.

I do not want to see ambitious attempts like these fail. Just the opposite — I want to see them succeed. But success on the web requires a different kind of DNA, the type of DNA that is difficult to adopt when one’s attention is focused on fighting the web itself.