O'Reilly Radar - Insight, analysis, and research about emerging technologies.

Watching the Retweeted Get Retweeted-er: Power User Secret Retweetist Love

2009-11-22T16:14:21Z

When Twitter decided to slowly roll out a new, official retweeting feature, people waited in anticipation. When they let their users know what it might look like, people debated whether that was the right way to deploy it. When it actually became available, people almost universally disliked it.

But my post is about why I love the new Twitter retweet feature, without ever having to think about it. The reason is that official retweeting represents the new-new arms race for authority among power users. The new-new arms race, you say? Yes, because the new arms race was to get on as many lists as possible, with the most-followed lists having a special significance.

The new-new arms race is the race to get officially retweeted the most. The idea is that in a sea of boring or useless or narrow-topic tweets, people who have "authority" will get retweeted the most. And finally, Twitter has built its own system for keeping track of that - officially. Think of that silly "RT" thing that users generated as a wristwatch at a track meet; Twitter operates the official Rolex timeclock.

Getting officially retweeted has two huge benefits for users that disproportionately benefit the already popular. One, the already popular gain even more authority that will enable their profiles or tweets to be featured, for example, higher in Google and Bing search results. Two, their profile link, photo, and original tweet appear in other people's tweet streams, even if those people don't follow the already very popular person.

Both of these have the potential to drive a tremendous amount of traffic to a person's Twitter account, and the people with the most official retweets will become recommended-users-list version 2.0, I believe (see the ninth paragraph of this story). With all the hub-bub about advertising within one's Twitter stream, driving traffiic is becoming more important to more users than ever before. Who isn't tempted to sign up to push one ad a day and make $30,000 per month in bonus cash?

But not everyone will make $30,000 or $3,000 or even $300 a month. The official retweet system tends to disproportionately favor the already-massively popular. Their authority, already very high, will only become higher relative to that of the average user. To modify the common saying, the common person will watch the retweeted will get retweeted-er.

Not sure if you are part of the retweeted-er class? It's easy to find out. Go to your account on Twitter.com, click the "Retweets" tab, then click on the "Your tweets, retweeted" tab. Is almost every single one of your original tweets in there? Didn't even realize that was happening? Welcome to the club.

Of course, it's not really the fault of the massively popular Twitter users (I don't think Twitter consulted many of them before creating this feature), so don't blame them for trading in on their fame. The petit-bourgeois wealth of authority no doubt creates opportunities for the working-class Twitter users, under the theory of trickle-down tweetonomics. The real question is, will Twitter's proletariat class stand by and watch this happen, or form an uprising?

Addendum: Shortly after I wrote this I came across a Valleywag post with a similar theme.

Robots.Txt and the .Gov TLD

2009-11-20T20:00:26Z

I'm on the board of CommonCrawl.Org, a nonprofit corporation that is attempting to provide a web crawl for use by all. An interesting report just got sent to us about the use of robots.txt files within the .Gov Top Level Domain, a standard known as the Robots Exclusion Standard.

In examining about 32,000 subdomains in .gov, it turns at least 1,188 of these have a robots.txt file with a "global disallow," meaning robots are excluded from indexing this content. Even more curious, on 175 of these sites, while there is a global disallow, there is a specific bypass that allows the Googlebot to index the data. You can look at the raw data on Factual.

At Public.Resource.Org, we've always felt that the use of a robots.txt file by the government should only be used for purposes of security and integrity of the site, not because some webmaster arbitrarily decides they don't want to be indexed. Indeed, on several occasions we have deliberately ignored government imposed robots.txt files because we felt this was an arbitrary and illegal attempt to keep the public out.

And, needless to say, it doesn't make any sense at all to let in some webcrawlers and not let in others. If this is a reaction to a security/integrity issue, such as limited capacity, the proper thing to do is include in the robots.txt file a comment that can be used by other bots to explain what is going on. For example, it could be perfectly reasonable for a government group faced with limited capacity to ask a robot to limit crawls to a certain number of queries per second and only whitelist crawlers that agree to that condition.

Government webmasters should use the robots.txt file sparingly, and should do so in a non-discriminatory fashion.

Asia Continues to be Facebook's Strongest Growth Region

2009-11-20T12:00:00Z

With Facebook topping 330 million active users over the past week, the company's strongest growth region continues to be Asia. Over the last 12 weeks, Facebook added close to 17M active users in Asia alone. Since my previous post, the share of active users from Asia grew by 2% (to 13.5% of all users), and roughly 1 in 7 users now come from the region. With a market penetration under 2%, Facebook is poised to add many more users in Asia (and Africa).

Compared to the U.S., the proportion of Facebook users in their teens (13-17) or in the 18-25 age group are much higher in Asia:

As was the case in other parts of the world, expect the share of users 45 and older to climb as Facebook becomes more mainstream in Asia. Growth was strong across all age groups in Asia over the last 12 weeks, particularly among teens (+90%) and the 18-25 age group (+60%).

In other regions, notably North America, Europe, the Middle East, and South America, growth in the 18-25 age bracket, lagged behind users 45 and older.

In closing I want to highlight countries (within several regions) where Facebook has been growing rapidly:

In Europe, growth has been fastest in the East: as an example, the number of active users in Poland doubled over the last 12 weeks. Growth in Southeast Asia remains strong in countries that have been home to Friendster's core user base. While Facebook added over 800,000 active users in Brazil, for now Orkut remains the dominant social network in South America's most populous country.

Four short links: 20 November 2009

2009-11-20T11:00:00Z

Spokeo -- abysmal indictment of society, first prize in mankind's race to the bottom. Uncover personal photos, videos, and secrets ... GUARANTEED! Spokeo deep searches within 48 major social networks to find truly mouth-watering news about friends and coworkers. PS, anybody who gives their gmail username and password to a site that specializes in dishing dirt can only be described as a fucking idiot. (via Jim Stogdill, who was equally disappointed in our species)
Biologists rally to sequence 'neglected' microbes (Nature) -- The Genomic Encyclopedia of Bacteria and Archaea is project to sequence genomes from more branches of the evolutionary tree of life. Eisen's team selected and sequenced more than 100 'neglected' species that lacked close relatives among the 1,000 genomes already in GenBank. The researchers reported earlier this year at the JGI's Fourth Annual User Meeting that even mapping the first 56 of these microbes' genomes increased the rate of discovery of new gene and protein families with new biological properties. It also improved the researchers' ability to predict the role of genes with unknown functions in already sequenced organisms. (via Jonathan Eisen)
Mail Learning: The What and the How (Simon Cozens) -- a few things that a really good mail analysis tool needs to do. I hope that my mail client and server does these out of the box in the next five years.
Introducing the Open Web Foundation Agreement -- The Open Web Foundation Agreement itself establishes the copyright and patent rights for a specification, ensuring that downstream consumers may freely implement and reuse the licensed specification without seeking further permission. In addition to the agreement itself, we also created an easy-to-read "Deed" that provides a high level overview of the agreement. Applying the open source approach to better standards.

Health gets personal in the cloud

2009-11-19T21:00:00Z

Healthcare is one of the biggest industries in the world. The United States spends over 17% of its GDP on healthcare and the issue of the industry's future is being hotly debated in Congress. Whatever happens to other elements of health reform, health information technology will play a key role in moving us towards the goal of bending the cost curve and improving quality and clinical outcomes. A Personal Health Record (PHR) is one way that patients can have some control of their own health data, while providing an interoperable platform for sharing relevant clinical data between providers. Healthcare is changing rapidly and there are some important trends worth watching.

Healthcare in the near future will be quite different than it is today. Web enabled technology is already changing the way medicine is practiced. As the digital nation comes of age we will see new opportunities, and new challenges, bringing healthcare in America into the 21st century. Health consumers will come to expect they will have control over their own health data. Having secure, interoperable access to clinical data will allow patients to partner with their care providers in new ways incorporating Web 2.0 principles.

For example, Google announced at the Health 2.0 conference that they have entered into a partnership to provide telehealth services through their Google Health platform using MDLiveCare. With the integration of MDLiveCare technology, Google can provide a service that offers patients access to doctors from remote locations, via webcam or telephone, into its personal health record offering. This will be particularly valuable for those who are caring for their loved ones from far away. My family is scattered around the country and caring for our mother with advanced stage Alzheimer's was quite a challenge that would have benefited from this type of service.
Here is a screenshot of Google Health:

"Patients remember less than 25% of what they're told when they consult with a doctor,” said Bob Smoley, CEO, MDLiveCare, in the statement. "By directly synchronizing the information that's shared…we're able to provide patients with a convenient solution to review their physician or therapist encounters."

"We strongly believe that the patient has the right to control their own health data," said Product Manager of Google Health Roni Zeiger, MD a practicing Internist who also works in urgent care. "You can now request an online consultation with a physician. At the end of the visit the doctor documents the encounter and it is immediately sent to your Google Health account and you will have a complete record of the doctor's notes."

Also, Microsoft has introduced My Health Info as part of HealthVault. My Health Info is an interactive and customizable dashboard that allows people to view all their health information: Blood pressure, blood glucose, BMI, immunizations, allergies, lab results, medications, steps walked, health articles and more, in a single, organized, and convenient location. It connects with HealthVault so information updated in one product is automatically updated in the other. This service offers tools and widgets to upload, organize and monitor health information stored in their personal HealthVault accounts. The service also allows people to research medical concerns, read the latest health news, gain guidance from medical experts, learn about nutrition, and monitor conditions such as high blood pressure or diabetes.

This is the main screen for My Health Info: "As consumers are increasingly being asked to manage more of their health and wellness, they are looking for solutions that help them navigate an overwhelming amount of information, enabling them to take control of their personal health data," said David Cerino, General Manager of Consumer Health in Microsoft Health Solutions Group. Marguerite Yeo, Director of Product Marketing for Microsoft HealthVault told me about Online Care deployed by Hawaii Medical Service Association (HMSA) an independent licensee of the Blue Cross and Blue Shield Association. Online Care, enhanced by Microsoft Healthvault, allows patients to see physicians immediately, through live consultations via Web or phone. By providing access to doctors anytime in the patient‘s home, health plans like HMSA have the opportunity to shift healthcare to less expensive care settings when appropriate.

Another company that is doing some interesting work in this area is Practice Fusion. Practice Fusion is a free, Web-based electronic health record service for physicians. They recently announced the launch of Patient Fusion, their new PHR, at Dreamforce 2009 in San Francisco, salesforce.com's user and developer conference.

"The healthcare and life sciences community is a rapidly growing sector," said Clarence So, Senior Vice President of Strategy, salesforce.com. "The Force.com platform allows companies like Practice Fusion to quickly innovate around a common objective for improving health." Through Patient Fusion, doctors grant patients instant access to their medical records, medications and immunization history. Updates to the patient's records are available in real-time in the cloud. Patients will also be able to schedule appointments, request prescription refills, email their physicians, and, most importantly, share their data with other providers at any time.

Here is a shot of the main Health Manager screen:

They also announced ChartShare, a feature which allows users to have real-time access to patient records in a familiar and interactive format. All authorized users can access records simultaneously. This enables care providers to share clinical data and allows real time collaboration and consultation.

"Practice Fusion continues to innovate in the healthcare market by offering a free Web-based PHR that is an extension of the practitioners' EHR. We're unlocking the physician EHR to give patients access and control over their own health data," said Ryan Howard, the CEO of Practice Fusion. He also told me, "The January release of Patient Fusion will allow the same ability that physicians now have using ChartShare for portability of data on the patient side."

Whether it is by using a platform like Microsoft HealthVault or Google Health, or a SaaS model EMR/PHR like Practice Fusion, the options for patients and providers to coordinate care using Web 2.0 technology is making great strides. We will increasingly see platforms that provide virtual visits with care providers, and greater use of the web for tasks like making appointments, medication and therapy reminders, and making payments. I look forward to the day when I can login for a consultation with my doctor as easily as I Skype with my friends around the world. The future of healthcare is here, and it is beginning to be distributed.

Four short links: 19 November 2009

2009-11-19T11:00:00Z

Chumby One (Bunnie Huang) -- new Chumby product released. In addition to being about half the price of the original chumby, the new device added some features: it has an FM radio, and it has support for a rechargeable lithium ion battery (although it’s not included with the device, you have to buy one and install it yourself). There’s also a knob so you can easily/quickly adjust the volume. But I don’t think those are really the significant new features. What really gets me excited about this one is that it’s much more hackable.
Deep Tracing of Internet Explorer (John Resig) -- very sexy deep inspection of Internet Explorer. Finally, something IE does better than Firefox (other than exploits). dynaTrace Ajax works by sticking low-level instrumentation into Internet Explorer when it launches, capturing any activity that occurs - and I mean virtually any activity that you can imagine. (via Simon Willison)
Less Than Free -- begins by talking about Google giving away turn-by-turn directions on Android, and then analyses Google's "less than free" business model: Additionally, because Google has created an open source version of Android, carriers believe they have an “out” if they part ways with Google in the future. I then asked my friend, “so why would they ever use the Google (non open source) license version.” Here was the big punch line - because Google will give you ad splits on search if you use that version! That’s right; Google will pay you to use their mobile OS. I like to call this the “less than free” business model. This is a remarkable card to play. Because of its dominance in search, Google has ad rates that blow away the competition. To compete at an equally “less than free” price point, Symbian or windows mobile would need to subsidize. Double ouch!!
Expert Labs -- a new independent initiative to help policy makers in our government take advantage of the expertise of their fellow citizens. How does it work? Simple: 1. We ask policy makers what questions they need answered to make better decisions. 2. We help the technology community create the tools that will get those answers. 3. We prompt the scientific & research communities to provide the answers that will make our country run better. New non-profit from Anil Dash.

Four short links: 18 November 2009

2009-11-18T11:00:00Z

Memento: Time Travel for the Web -- clever versioning hack that uses HTTP's content negotiation to negotiate about the date!
Ordnance Survey Maps to Go Online -- The prime minister said that by April he hoped a consultation would be completed on the free provision of Ordnance Survey maps down to a scale of 1:10,000, (not the scale of a typical Landranger map set at 1:25,000). The online maps would be free to all, including commercial users who, previously, had to acquire expensive and restrictive licences at £5,000 per usage, a fee many entrepreneurs felt was too high. No word yet on license. (more details here)
Mapsicle -- open source Javascript library to create mashups and application on Google Streetview, from NZ developers Project X. It has been released by Google as part of the Maps Utility library.
Freedom of Creation Shop -- online store for 3D-printed objects. (via Makezine).

The iPhone: Tricorder Version 1.0?

2009-11-17T20:13:46Z

The iPhone, in addition to revolutionizing how people thought about mobile phone user interfaces, also was one of the first devices to offer a suite of sensors measuring everything from the visual environment to position to acceleration, all in a package that could fit in your shirt pocket.

On December 3rd, O'Reilly will be offering a one-day online edition of the Where 2.0 conference, focusing on the iPhone sensors, and what you can do with them. Alasdair Allan (the University of Exeter and Babilim Light Industries) and Jeffrey Powers (Occipital) will be among the speakers, and I recently spoke with each of them about how the iPhone has evolved as a sensing platform and the new and interesting things being done with the device.

Occipital is probably best known for Red Laser, the iPhone scanning application that lets you point the camera at a UPC code and get shopping information about the product. With recent iPhone OS releases, applications can now overlay data on top of a real time camera display, which has led to the new augmented reality applications. But according to Powers, the ability to process the camera data is still not fully supported, which has left Red Laser in a bit of a limbo state. "What happened with the most recent update is that the APIs for changing the way the camera screen looks were opened up pretty much completely. So you can customize it to make it look any way you want. You can also programmatically engage photo capture, which is something you couldn't do before either. You could only send the UI up and the user would have to use the normal built-in iPhone UI to capture. So you can do this programmatic data capturing, and you can process those images that come in. But as it turns out, at the same time, shortly after 3.1, the method that a lot of people were using to get the raw data while it was streaming in became a blacklisted function for the review team. So we've actually had a lot of trouble as of late getting technology updates through the App Store because the function we're using is now on a blacklist. Whereas it wasn't on a blacklist for the last year."

Powers is hopeful that the next release of the OS will bring official support for the API calls that Red Laser uses, based on the fact that the App Store screeners aren't taking down existing apps that use the banned APIs. Issues with the iPhone camera sensors pose more of a problem for him. "In terms of science, it's definitely a really bad sensor, especially if you look at the older iPhone sensor, because it has what's called a rolling shutter. A rolling shutter means that as you press capture or rather as the camera is capturing video frames or as you capture a frame, the camera then begins to take an image. And it takes a finite number of milliseconds, maybe 50 or so, before it is actually exposed to the entire frame and stored that off into a sensor. Because it's doing something that's more like a serial data transfer instead of this all at once parallel capture of the entire frame, what that causes is weird tearing and odd effects like that. For photography, as long as it's not too dramatic, it's not a huge deal. For vision processing, it's a huge deal because it breaks a lot of assumptions that we typically make about the camera. That has gotten better in the 3GS camera, but it's still not perfect. It is getting better, especially when the camera's turned on the video mode."

One thing that has significantly improved with the iPhone 3GS is the actual camera optics. Most people know that the 3G and the first gen phone don't have autofocus at all. So their optics is just a fixed-focus simple plastic lens that doesn't allow you to focus up close. For anybody trying to do macro imagery, something up close, you're just not going to be able to do it on the 3G or the first gen phone. When we set out to build our application, we specifically had to work around that problem. A lot of why our application was successful was because we did focus on that problem. Then in the 3GS, the autofocus mode was enabled which is actually a motor-based autofocus system that can autofocus not only on the center of the image, but also somewhere that you pick specifically. And one more thing is that the autofocus system doesn't just change the focus, it also changes the exposure, which is something a lot of people don't notice. "

Another benefit the 3GS has brought to the table for vision processing is the dramatically increased processor speed. "With the 3GS, it's actually an incredibly powerful device," says Powers. "So we think right now that there's actually a lot of power there that hasn't been exposed. So I mean, there obviously are limits. But I don't think we've seen software that really hits those limits. Honestly, the limits that we're seeing right now are just in the SDK and what you can and can't do. One of the things about the iPhone is, as I was alluding to earlier when I talked about previous problems with the Android which are now being addressed, is that you could code at the lowest level on the iPhone, whereas you could not code at the lowest level on the Android. What that means to the iPhone is that you can actually write on ARM assembly if you want.

Almost everyone who's doing any sort of image processing today on the iPhone isn't taking advantage of that. We are to a very small extent in Red Laser, but there's certainly juice that can be extracted by just spending time optimizing for the platform, which is something that the iPhone lets you do. And the other thing to add to that is there are new instructions enabled by the ARM 7 Instruction Set which is used on the 3GS, which wasn't available previously. And, again, I actually haven't heard of anyone utilizing those functions yet. So there's a lot of power there that is yet to be exposed."

Although the iPhone has been an interesting platform for Powers, he is turning his attention toward the Droid at the moment. "From our perspective, we would love to keep developing our vision software on the iPhone, but because of the fact that the APIs are so restrictive right now and we have no ETA on when that'll be fixed, we're actually looking to the Android now, specifically, the new Droid, as an interesting platform for computer vision and image processing in real-time. Again, if it's not a real-time task, the iPhone's a great platform. If you can just snap an image, process it, you can do anything on the iPhone that has that characteristic. But if you want to process in real-time, Android is really your best bet right now because of the fact that A, the APIs do let you access the video frames and B, you can now actually write on the metal of the device and write things in C and C++ with the new Android OS which, again, you couldn't do before. "

Alasdair Allan is approaching the iPhone from a different direction, using it as a way for astronomers to control their telescopes remotely while "sitting in a pub." While he's seen some primitive scientific applications of the iPhone for thing such as distributed earthquake monitoring, he thinks that the real benefit of the iPhone over the next few years will be as a visualization tool using AR.

That isn't to say that he isn't impressed with the wide variety of sensors available on the iPhone. "You have cellular for the phones. All of the devices have wifi. And most of the devices, apart from the first gen iPod Touch have Bluetooth. You, of course, have audio-in and speaker. The audio-in is actually quite interesting because you can hack that around and actually use it for other purposes. You can use the audio-in as an acoustic modem into an external keyboard for the iPhone, I think that's in iPhone Hacks, the book. It's quite interesting. Then on the main sensor side, you've got the accelerometer, the magnetometer, the digital compass. It's got an ambient light sensor, a proximity sensor, a camera, and it's also got the ability to vibrate. "

According to Allan, the iPhone sensor that the least people know about is the proximity sensor. "The proximity sensor is an infrared diode. I think it's actually now a pair of infrared diodes in the iPhone 3G. It's the reason why when you put your iPhone to your head, the screen goes blank. It basically just uses this infrared LED near the earpiece to detect reflections from large objects, like your head. If you actually take a picture of the iPhone when it's in call mode, with a normal web cam, you'd actually be able to see right next to the earpiece a sort of glowing red dot which is the proximity sensor. Because, of course, web cam CCDs are sensitive in the infrared so it would actually show up. This was a bit of a scandal early on in the iPhone's life. The original Google Search app used undocumented SDK call to use this so you could actually speak into the speech search, and Apple and everyone really was very annoyed about this. So they actually enabled it for everyone in the 3.0 SDK."

Unfortunately, Allan doesn't know of anyone who has been able to make practical use of the prox sensor, partially because it has such a short range. On the other hand, the newly added magnetometer in the 3GS has opened the door to a host of AR applications. But Allan points out that like any magnetic compass, it can be very sensitive to metal and other magnetic interference in the surrounding environment. "It is very susceptible to local changing magnetic field monitors, CPUs, TVs, anything like that will affect it quite badly."

Also, he adds, to do any really accurate AR applications, you need to use the sensors in concert. "By default, what you're measuring, of course, is the ambient magnetic field of the Earth. And that's how you can use it as a digital compass, because there are tables that will show you how to do deviations from magnetic north to true north, depending on your latitude and longitude. Which is why to do augmented reality apps, you need both the accelerometer and the magnetometer, so it can get your pitch and roll to the device and the GPS to get the latitude and longitude so you know the deviation from true north."

Allan thinks that although the current sensor suite has limited uses for scientific data capture, things will improve quickly. "I think the science usage is definitely going to grow. When the sensor get slightly more sophisticated than they are today, for instance gyros or you can imagine slightly better accelerometers or light sensors or sort of other things. You could even put LPG or methane gas sensors in there very easily. They're both sensors that are very small now. You could certainly get going in science doing environmental monitoring, all of that sort of stuff going very easily. And it would quite easily piggyback off sort of social networking ideas as well. I do see the very high-end smartphones contributing to growth in citizen-level science and people in the street getting out to do science and help people build large datasets that can actually be used to predict long-term trends and that sort of thing."

Powers concurs. "It behaves more like a tricorder than a communicator, right, because certainly voice communicating isn't all we're doing anymore. And I think if you take voice communication as a fraction of the utilization of a phone, you're going to see that there's definitely a trend that goes down all the time. I don't think it'll ever go to zero, but it'll certainly go to a smaller fraction. At the same time, the sensors are increasing. I would like to see not necessarily barometric or environment measurement sensors, but things like solid-state gyroscopes on phones and maybe a pair of cameras and maybe even different sensors that can allow us to read credit cards and do transactions on the device. I think there's even some talk of that appearing in the next gen iPhone so you can actually do transactions just by swiping your phone into a register. So I would agree with the assessment that they're becoming more like tricorder."

What Does Innovative Social Engagement Look Like For Businesses and Governments?

2009-11-17T14:00:00Z

I've been thinking about the topic of Government 2.0 a lot lately. Part of this topic deals with the multi-directional engagement between government and citizens. This is what the White House and others have termed a more transparent, collaborative, and participatory government.

Unfortunately, the engagement for the most part is not very authentic nor meaningful. Boring "fan pages" on Facebook are one example I've written about, but there are many others. Often, engagement, when it does happen has so many rules associated with it, or such a high barrier to entry, or such a limited window as to be practically meaningless.

It seems to me that everyone can celebrate the fact that government entities merely have a YouTube channel here, a Twitter account there, or a Blogger profile some other place (the so-called "TGIF revolution"), or we can think a little harder about what the goals of citizen engagement really might be, and how to go about achieving them. But first, a personal example of responsiveness and engagement from the private sector.

On the evening of Nov 2nd, I tweeted from my phone about a local DC restaurant, Co Co Sala, just as I was leaving. We had a nice experience, but the hostess had been a little, shall we say, disinterested in helping us? So I commented as much.

Less than a week later, the co-owner of Co Co Sala sent me an email and cc'd his general manager. He apologized for the treatment I experienced, assured me it was not policy, introduced me to the manager, and said he'd talk to his staff. It was a four-paragraph email. I've never met him before, and furthermore, my personal email is discoverable but not the most easy thing to find.

This is what real social innovation looks like. This is what customer service looks like. This is what true engagement with stakeholders looks like. I want to give this great lounge Co Co Sala a hearty shout-out for not only having a great product, but also really caring about their customers.

Now, imagine we weren't talking about a restaurant here. Imagine we are talking about the Department of Motor Vehicles, or the Patent and Trademark Office, or your Congressman. If you tweeted, would they see it? Would they care? Would they react in any way? I think the answer in many cases is no. And when was the last time you gave the DMV a shout-out for a job well done?

Let's look at a sliver of data. According to TweetStats.com, the people behind the White House Twitter account reply to individuals less than 2% of the time, and seem to have never @ replied to any single more than once (i.e., they have never come close to a conversation). They re-tweet others' tweets about 6.5% of the time, but they only seem to re-tweet other government accounts and the New York Times. Granted, there are more people tweeting about White House issues than Co Co Sala, but does the above data represent any caring in any way, shape or form?

The terrific techPresident blog recently noted that actor Vin Diesel is the single most followed living person on Facebook - and that he recently passed up President Obama. Perhaps that's because Vin Diesel's Facebook fan page is awesome. He is engaged, his fans are engaged, and the tone is informal and fun. There are also many other high-profile people who have taken the plunge into innovative social engagement; my favorite at the moment is Alyssa Milano.

So when exactly did "serious and formal" become a substitute for "informative and meaningful" in government circles? And why is everyone scared of letting their guard down in public? People and entities that innovate and use new social networking tools to engage with stakeholders will be winners. The ones that don't will be losers in the long run. It's that simple.

If a goal of Government 2.0 is to provide citizens better services, and a strategy towards reaching that goal is to use social media tools to communicate better with citizens on multiple channels, it seems to me that listening and responding better to comments and complaints would be a great tactic.

The reason why people still cite the TSA's blog as a good example of citizen engagement is because few other outstanding examples of federal government social media engagement seem to have emerged in 2009. What does 2010 have in store?

It is somewhat outside the scope of this post, but my guess is that more and more local government responsiveness and engagement is happening. We heard some of those stories at the Gov 2.0 Expo Showcase in September. What are some new ones that the feds should hear about?

Four short links: 17 November 2009

2009-11-17T11:30:00Z

Digital Natives (Ze Frank) -- digital natives have grown up in a landscape where access to information and influence has been flattened. they have watched media distribution bottlenecks in the form of networks and studios lose influence to youtube and independent production houses. They have watched companies bow down to viral video critiques, and watched political systems get hacked by social networks. this is a generation that doesn't understand restrictions on access to media if those restrictions are inefficient or obviously detrimental to the system as a whole. this is a generation that has been at war with DRM and copyright right from the start. it is a generation awash with free tutorials and download-able source code. When is a conversation with a precocious 17 year old a glimpse into an inter-generational gulf with implications for the role and status of formal education, and when is it just an encounter with a brat? Ze's piece is worth reading, whichever way it comes out.
ICU64 -- an open source Commodore 64 emulator (Frodo) hacked to visually and textually display memory. Watch the video embedded below, it's hypnotic and seductive. It immediately made me want one for my programs (without having to port my code back to 6502 assembler). (via waxy whose return from pneumonia is greatly welcomed)
Me and Belle du Jour -- interesting story from a UK blog master who guessed her identity but kept it secret, creating a googlewhacked page as a tripwire to let him know when someone else guessed. He tipped her off that her cover was blown. (via waxy again)
The Hail Mary Cloud -- the world's slowest yet effective brute force attack. If you publish your user name and password, somebody who is not you will use it, sooner or later. A botnet is brute-force trying every known username and password combination against every known ssh server. Each attempt in theory has monumental odds against succeeding, but occasionally the guess will be right and they have scored a login. As far as we know, this is at least the third round of password guessing from the Hail Mary Cloud (see the archives for earlier postings about slow bruteforcers), but there could have been earlier rounds that escaped our attention.

Turning Predictions into Opportunities

2009-11-16T18:00:00Z

The view from the eye of a recession isn't great. When companies are going bust, unemployment growing, and everyone's scouring their budgets for costs to cut, it can be hard to see opportunities. However, when Tim pointed to Stephen O'Grady's fine set of 2010 predictions I found myself popping with "oh, so naturally this will happen next ..." thoughts. Think of this as a glimpse of the blue sky after the economic funnelspout that's demolished our economy. (Continuations of the tornado metaphor with "being sucked into the cloud", or "trailer park economics", or "we're not in Kansas any more, Tantek" left as an exercise to the reader)

As every cloud provider creates their own "open API" (itself a fraught term), look to see the rise of brokers who can migrate you from one cloud to another. Deltacloud is an early free project here from RedHat, but there are many business opportunities waiting. It's possible that companies will pay for assurance (you've tested your migration tool, you know it works on corner cases), service vs product (they don't want tools to run, they want to pay you to install and maintain the tools accessible through a web console), or premium services so that you're a partner helping them get the most from the cloud and not simply a vendor.
We're a long way from sated in the world of collaboration tools. The current rage is mail learning, applying machine learning techniques to email so as to better understand social networks and prioritise incoming email messages and these are largely server-based solutions because it's so hard to get access to the desktop/web clients. Should Google Mail create an app store environment with hooks into the backend, the game could be on for consumer plays around email analytics, prediction, and simply smarter behaviour (why does my email client still not tell me when I say "see attached" yet don't have an attachment in the message?).
Beyond email, many interesting tools have sprung up around the Gov 2.0 space that have applicability within organisations. Yammer has done well to bring Twitter to large companies, but there are still opportunities around simple document markup and suggestion gathering and filtering. Solve a real problem and there's money waiting.
Google's low overhead management is made possible by its automated intranet and the visibility into projects from public code repositories, public smoke builds, and public status blogs. The opportunities to sell this into large companies looking to be "more like Google" are huge.
If Stephen's right that datasets are increasingly viewed as "serious, balance sheet-worthy assets" then the world is going to need some serious balance sheet-worthy help in valuing those assets.
Big data is being democratised, but there's a lot of unmet need in businesses around data warehousing. The typical solution is to build a data warehouse team around a product like Oracle, but I've heard plenty of business people grizzling about the result. They want answers, they don't want the headaches and lag that a data warehouse involve. Big Data (or Cloud Analytics or whatever) may be the opportunity to figure out a new minimum viable product for these folks, and offer it without the "data warehouse" baggage. This might be back end, might be UIs, might be visualisation, but all of these have a lot of room for improvement.
The proliferation of developer targets immediately makes me think of the early PC era. It makes sense to proliferate: let the most useful ("successful") bubble to the top and survive naturally. At this point in the evolution of the scaleout of massively multiplayer online programming languages, we don't know exactly what winning looks like: it's a big feedback loop between the people who build the programming languages and the people with problems to solve (there are always more of the latter than former) and each time we go around it we know more about what is and isn't useful in this brave new world of coding for other people's data centres. Opportunity? Join the mob and write your own programming language, or simply take your commercial opportunity for a spin around the many different languages out there and be the first in your niche to find a good fit between problem space and solution tool.
Stephen's throwaway comment "I’ve never subscribed to the idea that only what can be measured can be managed - open source, in particular, belies that claim" seems like a thrown gauntlet on open source analytics. In particular, I suspect there's a tools opportunity around the nebulous "community manager" role that every company seems to need. It's part CRM, it's part developer tool, it's part tech support, and part camp mother. Usefully quantify aspects of open source development and help companies that are doing it to know how they're doing and what they could do better.
Marketplaces are big in mobile, but I look to other areas as ripe for the picking. For example, if Google Apps are catching on in many companies then a plugin marketplace is a natural extension. It would build out the Apps suite faster than Google can, would enable the tight loop between demand and supply that will drive the product along, and make Google's offering very different from other parties. This is also true of Microsoft and others, but I feel like momentum is more with Google's product than the others. (A feature can push a leader further in front, but rarely helps a laggard leapfrog to the lead)
Every marketplace thus far has been flawed. Apple's famously annoys many developers and blocks huge categories of product (the "don't be better than we are" rule, which is hard to justify as being in the customer's interest), but don't forget Palm's impedance mismatch with jwz's open source code. I think the final chapter on how marketplaces work is far from written.
NoSQL tools remain in their infancy and so there are huge opportunities here. Identify a niche ("fast accurate and timely web metrics for decision-making"), a tool that can solve it (MongoDB), and build the deployment, scaling, administration, reporting tools so you can sell a complete package into that niche. Rinse, lather, repeat.

The War For the Web

2009-11-16T14:29:43Z

On Friday, my latest tweet was automatically posted to my Facebook news feed, as always. But this time, Tom Scoville noticed a difference: the link in the posting was no longer active.

It turns out that a lot of other people had noticed this too. Mashable wrote about the problem on Saturday morning: Facebook Unlinks Your Twitter Links.

if you’re posting web links (Bit.ly, TinyURL) to your Twitter feed and using the Twitter Facebook app to share those updates on Facebook too, none of those links are hyperlinked. Your friends will need to copy and paste the links into a browser to make them work.

If this is a design decision on Facebook’s part, it’s an extremely odd one: we’d like to think it’s an inconvenient bug, and we have a mail in to Facebook to check. Suffice to say, the issue is site-wide: it’s not just you.

As it turns out, it wasn't just links imported from Twitter. All outbound links were temporarily disabled, unless users explicitly added them as links via an "attach" dialogue. I went to Facebook, and tried posting a link to this blog directly in my status feed, and saw the same behavior: links were no longer automatically made clickable. You can see that in the image that is the destination of the first link in this piece.

The problem was quickly fixed, with URLs in status updates automatically now linkified again. The consensus was that it was in fact a bug, but it's little surprise that people suspected otherwise, given the increasing amount of effort Facebook puts into warning people that they are leaving Facebook for the big bad unsafe Internet:

All of this is well-intentioned, I'm sure. After all, Facebook is attempting to put in place privacy controls that allow its users to manage the visibility of their information -- and the Web's expectation of universal visibility is not necessarily the best default for much of the information posted on Facebook. But let's not kid ourselves: Facebook is a new kind of web site (or an old kind redux), a world of its own, playing by different rules.

But this isn't just about Facebook.

The Apple iPhone is the hottest web access device around, and like Facebook, while it connects to the web, it plays by a different set of rules. Anyone can put up a website, or launch a new Windows or Mac OS X or Linux application, without anyone's permission. But put an app onto the iPhone? That requires Apple's blessing.

There is one glaring loophole: anyone can create a web application, which any user can save as clickable application on their phone. But these web applications have limits - there are key capabilities of the phone that are not accessible to web applications. HTML 5 can introduce all the new application-like features it wants, but they will work only for web applications, and can't access key aspects of the phone with Apple's permission. And as we saw earlier this year with Apple's rejection of the Google Voice application, Apple isn't shy about blocking applications that it considers threatening to their core business, or that of their partners.

And now, of course, we see the latest salvo in the war against the accepted rules of interoperability on the web: Rupert Murdoch's threat to take the Wall Street Journal out of the Google search index. While most people have repeated the existing wisdom that to do so would be suicide for the Journal, a few contrarian observers have noted the leverage Murdoch holds. Mark Cuban argues that Twitter now trumps search engines when it comes to breaking news. Even more provocatively, Jason Calacanis suggested, a few weeks before Murdoch's announcement, that all big media companies need to do to cut Google off at the knees would be to block Google, while cutting an exclusive deal with Bing to be found only in Microsoft's search index.

Of course, Google wouldn't take that lying down, and would likely make its own exclusive deals, leading to a showdown that would make the browser wars of the 90s seem tame.

I'm not saying that News Corp and other mainstream media publications would adopt Jason's suggested strategy, or that it would work if they did, but it is becoming clear to me that we are heading into a bloody period of competition that could be extremely unfriendly to the interoperable web as we know it today.

If you've followed my thinking about Web 2.0 from the beginning, you know that I believe we are engaged in a long term project to build an internet operating system. (Check out the program for the first O'Reilly Emerging Technology Conference in 2002 (pdf).) In my talks over the years, I've argued that there are two models of operating system, which I have characterized as "One Ring to Rule Them All" and "Small Pieces Loosely Joined," with the latter represented by a routing map of the Internet.

The first is the winner-takes-all world that we saw with Microsoft Windows on the PC, a world that promises simplicity and ease of use, but ends up diminishing user and developer choice as the operating system provider.

The second is an operating system that works like the Internet itself, like the web, and like open source operating systems like Linux: a world that is admittedly less polished, less controlled, but one that is profoundly generative of new innovations because anyone can bring new ideas to the market without having to ask permission of anyone.

I've outlined a few of the ways that big players like Facebook, Apple, and News Corp are potentially breaking the "small pieces loosely joined" model of the Internet. But perhaps most threatening of all are the natural monopolies created by Web 2.0 network effects.

One of the points I've made repeatedly about Web 2.0 is that it is the design of systems that get better the more people use them, and that over time, such systems have a natural tendency towards monopoly.

And so we've grown used to a world with one dominant search engine, one dominant online encyclopedia, one dominant online retailer, one dominant auction site, one dominant online classified site, and we've been readying ourselves for one dominant social network.

But what happens when a company with one of these natural monopolies uses it to gain dominance in other, adjacent areas? I've been watching with a mixture of admiration and alarm as Google has taken their dominance in search and used it to take control of other, adjacent data-driven applications. I noted this first with speech recognition, but it's had the biggest business impact so far in location-based services.

A few weeks ago, Google offered free turn-by-turn directions for Android phones. This is awesome news for consumers, who previously could get this only in dedicated GPS devices or with high-priced iPhone apps. But it's also a sign just how competitive the web is getting, and just how powerful Google is getting, because they understand that "data is the Intel Inside" of the next generation of computer applications.

Nokia paid $8 billion for NavTeq, the leading provider of such turn-by-turn directions. GPS-maker TomTom paid $3.7 billion for TeleAtlas, the #2 provider in the market. Google quietly built an equivalent service, and is now giving it away for free -- but only to their own business partners. Everyone else still has to pay high fees to NavTeq and TeleAtlas. What's more, Google upped the ante by adding in such features as Street View.

Most interestingly, this move sets the stage for the future competition between Google and Apple. (Bill Gurley's analysis is an essential read.) Apple controls access to the dominant device of the mobile web; Google controls access to one of the most important mobile applications, and so far, is making it available for free only on Android. Google's prowess is not just in search, but in mapping, speech recognition, automated translation, and other applications driven by huge, intelligent databases that only a few providers can offer. Microsoft and Nokia control comparable assets, but they too are Apple competitors, and unlike Google, their business model depends on selling access to those assets, not giving them away for free.

It could be that everyone will figure out how to play nicely with each other, and we'll see a continuation of the interoperable web model we've enjoyed for the past two decades. But I'm betting that things are going to get ugly. We're heading into a war for control of the web. And in the end, it's more than that, it's a war against the web as an interoperable platform. Instead, we're facing the prospect of Facebook as the platform, Apple as the platform, Google as the platform, Amazon as the platform, where big companies slug it out until one is king of the hill.

And it's time for developers to take a stand. If you don't want a repeat of the PC era, place your bets now on open systems. Don't wait till it's too late.

P.S. One prediction: Microsoft will emerge as a champion of the open web platform, supporting interoperable web services from many independent players, much as IBM emerged as the leading enterprise backer of Linux.

I'll be speaking on this topic in my keynote at the Web 2.0 Expo in New York on Tuesday. I'll look forward to seeing many of you there.

Four short links: 16 November 2009

2009-11-16T11:00:00Z

Choose Your Own Adventure -- numerical and visual analysis of the Choose Your Own Adventure novels. The distinguishing characteristic of My Kind Of People is that they appreciate the quantitative study of the commonplace. (via Bryan O'Sullivan)
Tracking Droid Numbers -- uLocate, the makers of the Where app for Android, have been tracking the growth of the Droid phone using the data they get from the Android app store. (via BoyGenius Report)
Fly Eyes Makes Better Robot Vision -- to make smaller flying robots, researchers would like to find a simpler way of processing motion. Inspiration has come from the lowly fly, which uses just a relative handful of neurons to maneuver with extraordinary dexterity. And for more than a decade, O’Carroll and other researchers researchers have painstakingly studied the optical flight circuits of flies, measuring their cell-by-cell activity and turning evolution’s solutions into a set of computational principles. [...] Intriguingly, the algorithm doesn’t work nearly as well if any one operation is omitted. The sum is greater than the whole, and O’Carroll and Brinkworth don’t know why. Because the parameters are in constant feedback-driven flux, it produces a cascade of non-linear equations that are difficult to untangle in retrospect, and almost impossible to predict. (via Slashdot)
Meat Band Aids and Mass Production of Living Tissue -- Apligraf is a matrix of cow collagen, human fibroblasts and keratinocyte stem cells (from discarded circumcisions), that, when applied to chronic wounds (particularly nasty problems like diabetic sores), can seed healing and regeneration. This Gizmodo Q&A is informative.

Ignite NYC on 11/16: Gov 2.0, Body Hacks, and Hi-Tech Craft

2009-11-15T22:44:14Z

The Web 2.0 Expo starts tomorrow, 11/16, in NYC. We're kicking off the conference with an Ignite featuring 14 great speakers. The event is at the New World Stages. I'll be co-hosting with Ignite NYC organizer Tikva Morowati.

As always each speaker gets just five minutes on stage. Their presentation will each be just 20 slides that each auto-advance every 15 seconds. The Speakers include:

* Alison Lewis, http://www.iheartswitch.com/ (high tech craft)
* Brady Forrest, http://radar.oreilly.com/brady/ (Burning Man as tech incubator)
* Casey Pugh, http://www.starwarsuncut.com/
* Hilary Mason, http://www.hilarymason.com
* Jennifer Pahlka, http://www.codeforamerica.org/ (Gov 2.0)
* Jonathan Brill, http://Productlust.com, http://www.multitouchmaven.com
* Judy Shapiro, http://trenchwars.wordpress.com
* Kevin Marks, http://epeus.blogspot.com/
* Leesean Hepnova, http://www.leesean.net
* Lauren Schmidt, http://www.mit.edu/~lschmidt
* Molly Wright Steenson, http://www.girlwonder.com
* Nora Abousteit, http://www.burdastyle.com
* Patrick Davidson, http://Whereikeepmythingsontheinternet.com
* Quinn Norton, http://quinnnorton.com/ (body hacks)
* Ray Beckerman, http://recordingindustryvspeople.blogspot.com
* Tony Haille, http://tonyhaile.com

Here is a rough schedule for how the night will go:

7:00 pm - Doors Open for Conference & Expo Plus Pass holders
7:30 pm - Doors Open for Expo Plus Pass holders
7:45 pm - Doors Open for general public attendees (pending capacity)
8:00- 8:15 - Mobile Music Competition
8:15- 9:45PM - Ignite Talks
10PM -- Bar closes

We are going to start the evening off with an Ignite Mobile Music Competition, giving you a chance to win a FREE pass to Web 2.0! We will provide the mini cord, you provide the mobile instrument. Recommended apps include Sonifi, Drumbanger, and Bloom. Please fill out this entry form if you'd like to show off your mobile music making skills!

We will also randomly choose one lucky Twitter user who tweets using http://bit.ly/IgniteWeb20 to win a FREE pass to Web 2.0 Expo NY. You must show up at Ignite to win!

RSVP on Facebook

It's in the Bag! The Apple Tablet Computing Device

2009-11-13T21:11:08Z

In the past 25 years, the personal computing revolution has evolved from tethered (desktop) to luggable (portable) to joined-at-the-hip (mobile).

Via the iPhone Platform (including iPod Touch), Apple has set the bar for mobile computing by seamlessly integrating computation, communications, and media across hardware, software, and service layers.

No less integral, Apple has significantly evolved ecosystem development models by cobbling together developer tools, media relationships, marketplace/e-wallet functions, one-click software distribution, explicit platform governance, and a simple, but compelling, approach to sharing revenue with developers.

But, the pièce de résistance has been a touch, tilt, sensor, and virtual keyboard-based user interaction model that has rendered the traditional physical keyboard plus WIMP-based model (i.e., windows, icons, menus, and pointing device) as so last century, the proverbial horse-and-buggy to Apple's Model T.

The end result is that the iPhone has become the first truly personal computer; more personal to its owners than the PC ever was, a truth that bubbles to the top again and again when you talk to the 50M (combined) iPhone and iPod Touch owners.

Thus, the core thesis of this article is two-fold. One, that while Apple remains committed to cultivating its position in the legacy desktop /portable segment via the Mac, they understand that they will never be the leader of the PC market.

Two, given their dominance in mobile computing platforms, Apple will expand upon their iPhone strategy by attacking an "undefended hill" (an HP axiom) that's less hospitable to desktops/portables; namely, the bag-carrying consumer (think: purses, backpacks, briefcases, and the like).

The Bag-able Device: from Living Room to Classroom, Café to Bus
First, a market-sizing question. How many tens of millions of people carry a bag wherever they go that is large enough to accommodate a bookish-sized device?

From a sniff test, would there be room for a really "phat" version of the iPod Touch in your backpack? Your kid's? Would you make room?

Before answering, imagine that you're kicking back on the couch, with a cappuccino in one hand and a Tablet in the other. After all, this is a device that is recline-able in the sense that you can comfortably use it from any position that suits you (it's neither overly bulky or hot, and input operations can be performed from any angle you desire).

Moreover, owing to its relative absence of moving parts and exposed interfaces, the Tablet is also slob-friendly, a euphemism for saying that it's not the end of the world if you are eating pizza while using it (less susceptibility to spills, sauces and greasy fingers).

In turn, this means that it's kid-friendly since the dearth of moving parts also means fewer to break. On top of this, Apple's governance model provides a more direct path for parental controls on what types of apps can be used, and for how long.

Taking the Tablet out of your bag, you instantly notice that this is a device that can support multiple modalities in a robust fashion. A bigger screen means truer multi-touch, richer interaction possibilities, and a personal home theater experience that simply rocks (especially, when wearing decent headphones). Plus, as iPhone has proven, this is no underpowered computing device. Skype me? Sure. Video chat? In a snap. Day planner? C'mon!

Now, imagine iPhone's current gaming support scaling up to this device (not to mention the other two thirds of the 100K-app-strong App Store).

(Sidebar: I expect a straightforward upgrade path for developers to port their iPhone Apps to also run on the Tablet, offering tremendous platform leverage to the estimated 120K iPhone App developers.)

Moreover, given their iTunes foothold, how much do you want to bet that, coincident to the Tablet launch, Apple pursues a TV 3.0 play (aka, TV Everywhere) powered by a subscription service for music, movies and TV programming? In one fell swoop, the leverage of a TV 3.0 play could be extended not only to the Tablet, but to the Mac, iPhone, iPod, and Apple TV as well.

Who else can match that kind of end-to-end firepower, especially in light of Apple's announcement that the iTunes/App Store Universe is backed by 100M active credit card-backed user accounts?

That's also why Apple rebooting the book marketplace is such a given from where I sit (i.e., look out, Kindle).

A final note: while businesses/enterprises have been less central to the iPhone story to date, I think that the Tablet is a device that is tailor-made for verticals and VAR (value added reseller) channels, with Education, Health Care, Retail, and Field Support as obvious beachheads.

Flies in the Ointment: Avoiding the Tyranny of the Either/Or
So what could go wrong? Apple's challenge is to ensure that consumers never feel like they are being forced to make a binary Either/Or decision between an iPhone, an iPod Touch, a Tablet, and of course, a MacBook.

Under the hood, managing this one touches upon core strategic decisions about form-factors, runtime capabilities, and functional symmetries/asymmetries between the different Apple device offerings (read more about Apple's coming Hardware/Software Matrix decisions).

At the same time, some of this is market segmentation thinking, and allowing consumers to choose the level of integration, the type of computing model, and the depth of Apple-centric leverage that makes most sense for them.

Why? Because Apple's overriding goal is to grow their portion of the consumer's communications, media, entertainment ,and Engagement Time online, and in concert, their Portion of Spend for those services.

To be clear, though, Apple has already proven that they can navigate this one with the release of iPhone, and the related segmenting decisions relative to iPod Touch and iPod (if anything, the net-out has been a total Halo Effect). Hence, I am optimistic that not only will they successfully navigate this path with the Tablet, but that they have been planning for this transition for a long, long time.

As such, for Apple, a successful Tablet launch is not merely a fuzzy ambition, but rather, it's in the bag.

Related Posts:

Rebooting the Book: One Apple iPad Tablet at a Time

Apple, the 'Boomer' Tablet and the Matrix

Touch Traveler: London, Paris and only an iPod Touch