Yesterday Robert Scoble once again declared that the Open Web was dead. His argument was that Apps and proprietary black holes like Facebook are absorbing all the light (read: users, attention, value, investment) and taking our beloved open platform right along with it. In his post, he kindly (but incorrectly) named me as the only person who really cares about the Open Web.

While that’s flattering, I think he’s wrong about me being the only one who cares.

But he is right about the Open Web. It’s in real danger. URLs are fading into the background,  native Mobile apps are all the rage and Facebook threatens to engulf the web into a proprietary black hole.

But I think there’s a bigger problem going on right now. Not just with the web, but with silicon valley (as stewards of the web). We’ve lost sight of the things that matter. We’re obsessed with quick wins, easily digestible VC pitches, stock options and flipping for a Ferrari.

There’s more to this game than that. Let me touch on some of the things I see going on.

1. Lead not just cheerlead
   In our obsession with being seen by our micro-audiences as ‘thought leaders’ or ‘futurists’ it’s always very tempting to watch which way the wind is blowing and shout loudly that THERE is the future. Like a weather vane, it’s easy to point the way the wind is blowing, but our biggest, best opportunity is not to declare a popular service ‘the next big thing’ just because a few visible people are hanging out there. Rather our collective and individual responsibility is to help articulate a direction we think moves the state of the art forward for both the web and for society at large. Something, as leaders of this field, we believe in. Just like VCs develop an investment thesis, we should all have a vision for where the web is going (and how it should get there) and actively seek out, support and promote quiet heros who are building something that moves the needle in the right direction.
2. Add to the web’s DNA
   Almost every startup I see today is focused on building an ‘App’ and calling it a ‘Platform’. Too often (almost every time) though, these apps are nothing more than proprietary, incremental and niche attempts at making a quick buck. We need more companies to think deeper. Think longer term. What are you doing to change the fabric of the web’s DNA forever? How can you contribute to the very essence of the Internet the same way that TCP/IP, HTTP, HTML, JS and so many other technologies have done. Even proprietary technologies have provided valuable evolutions forward – things like Flash and yes, even FB. How are you going to live forever? This is why Facebook used to call itself a ‘Social Utility’ instead of a ‘Social Network’. Mark Zuckerberg was never content to be the next Myspace Tom. He wanted to be the next Alexander Graham Bell. And now he is.
3. Don’t just iterate, innovate
   Of course, someone has to build Apps. We can’t all be working at the infrastructure layer. But too many of the Apps we chose to build (or champion) are incremental. As startup founders, investors and influencers it’s so easy to understand something that can be described as the ‘Flipboard of Monkeys’ instead of thinking really hard about how a completely new idea might fit into the future. Sure there are plenty of good business and marketing reasons why you shouldn’t stray too far from the beaten path, broadening it one incremental feature at a time, but the core essence of what you’re working on can’t be yet another turn of a very tired wheel. If you’re shouting ‘Me too’ then you’re probably not thinking big enough.
4. B2C, not Ego2C
   Silicon valley is clearly a B2C town. We all love the sexy new app that our mother might eventually understand. Something we can get millions of users to use so we can show them lots of ads. Besides the fact that I think we should focus a little more on B2B, the problem is we’re not really a B2C town at all. We’re actually more focused on what I will call Ego2c. That is, we pick our favorite apps based on how famous the founding team is OR how easily we can use the app to build yet another niche audience for ourselves (and brands/marketers). It would be a tragedy if the social web revolution boils down to new methods of PR and marketing. But that’s what we seem to be obsessed with. As soon as any app from a famous founder gets released we give it tones of buzz while plenty of more deserving projects get barley a squeak. If the app gets a little traction (typically the ones that have Ego mechanics baked in) you see a million posts about how marketers can exploit it. Inevitably the app developers start to focus on how to ‘increase social coefficients’ instead of how to help human beings make a connection or find utility in their lives.
5. “Users don’t care”
   Speaking more specifically about the Open vs. Closed debate, too often we hear the criticism ”Users don’t care about open”. This is absolutely true and the reason why most open efforts fail. Users don’t care about open. They care about utility and choice. This is why the only way to continue propagating the open web is to work with BUSINESS. B2B. Startups, Media Brands, The bigco Tech companies. They care about open because the proprietary winners are kicking the losers ass and that usually means there are at least 1 or more other guys who need a competitive advantage. They need to team up and build, deploy and popularize the open alternative.  That’s why open always wins. There’s always plenty of losers around who are going to commoditize the popular closed thing. As technology leaders we’re paid to care about things users don’t care about. Things that shape the future. While users, in the short term, might not care, we should dare to think and dream a little bigger. As a case study look at Android vs. iOS. iOS is more profitable for a single company, but the other is now a force of nature.
6. Death is just a stage of life
   Just because something is no longer interesting doesn’t mean it’s dead. Its spirit, and often times the actual technology, lives on, one layer below the surface. RSS is a great example of this. RSS’s spirit lives on in ActivityStreams and the general publish/subscribe model. It is powering almost every service-to-service interaction you currently enjoy. Is it dead, or has it simply become part of the DNA of the Internet? Could RSS (or something like it) be better exposed higher up in the stack, absolutely, but that will take some time, thoughtful execution and influencers who are willing to champion the cause. The same is true for OpenID and OAuth.
7. The Arc of the Universe Is long but It bends towards Open
   The battle of Open vs. Closed is not a zero sum game. Both have their time. It’s a sin wave. First, closed, proprietary solutions come to define a new way of fulfilling a use case and doing business. They solve a problem simply and elegantly and blaze a path to market awareness, acceptance and commercialization. Open, however, always follows. Whether it’s a year, a decade or a century, Open. Always. Wins. The only question is how long, as an industry, are we going to keep our tail tucked between our legs in front of the the great giant proprietary platform of the moment or are we going to get our act together to ensure the “Time to Open” is as short as possible. It takes courage, co-ordination and vision, but we can all play our part to shorten the time frame between the invention of a proprietary app and the absorption of that value into the open web platform.
8. Acknowledge reality
   FB has won. It’s done. Just like Microsoft won the Desktop OS (in part handed to them by IBM), so too has FB won the Social OS (in part handed to them by Microsoft). For now. Acknowledging the truth is the first step to changing it. The only question now is how long we’re all willing to wait until we get our act together to turn the proprietary innovation of the ‘social graph’ into part of the open web’s core DNA. We need to recognize our power. They have ~1B users? The open web has more. Chances are that the major website or brand you work for has plenty of its own users as well. Are you going to send them to FB, or are you going to invest in your own .com. Trust me, I know it’s really, really easy to take what you’re given because you’re too busy putting out a million fires. But as technology leaders I challenge us all to build something better. We’re the only ones who can.
9. [Edit] Don’t kill Hollywood
   Did you catch the YC post  calling for silicon valley to kill hollywood. Not only was this reckless and short sighted, it’s the exact opposite of what we should be doing. Instead of trying to kill or cannibalize media companies and content creators, how about we work with them to create the next generation of information technology. They have the audiences+information and we have the technology. Instead, most silicon valley companies, by virtue of their B2C focus, are too busy leaching off major media instead of finding ways to help transform it. Sure most of them move slowly – but move they are. Move they must. Helping them is very profitable. I write more about this on the Echo blog – calling it ‘Real-time Storytelling‘
10. [Edit] Today’s data portability problem
    When I started the DataPortability project the issue of the time was personal data portability. That’s not the case anymore. While user-centric data portability is still being done via proprietary mechanisms it’s a) actually possible and b) moving more towards open standards every day. The real issue right now is firehoses. Access to broad corpuses of data so that 3rd parties can innovate is only possible through firehoses (for now). To put it another way, the reason Google was possible was because the open web was crawl-able - for free – with no biz dev deal. The reason FB was possible was because the open web allowed any site to spring up and do what it wanted to do. Today, too much of our data is locked up in closed repositories that can and must be cracked open. Google’s moves to exclude other socnets (besides G+) from their search results until they had free and clear access to them might be inconvenient for users in the short term, but, as a strategic forcing function, is in the best interest of the open web long term.

End of rant.

I was just watching a great TED video about the guy who runs ‘Improv Everywhere’. If you don’t know what that is I strongly suggest you google it and watch the video.

His TED talk had some great themes and some funny videos showing the result of their work, however my key takeaway was from a small innocuous line that he kept repeating as a one of his throw away comments… “and we didn’t even ask permission”.

His final summary in his talk speaks to our collective need to, as adults, remember to play and have fun. But I think that his statement about permission reveals what ‘Improve Everywhere’ is really doing. It’s showing people that it’s ok to impact the world… without asking permission.

Too many people assume that they need to ask permission to change the world.

“I’m studying to be a writer” basically suggests that you are waiting for someone to hand you a permission slip (A degree) which declares you an ‘English Major’. Then maybe you can call yourself a writer? Or maybe after you get your first writing job? Maybe after you publish your first book? Which of these permission slips allow you to declare yourself what you want to be – give you permission to change your little corner of the world?

This applies to everything we do. To be an entrepreneur, to love, to change the rules.

More people need to stop asking for permission – or worse, assuming they could never do something because it’s against the written or unwritten rules.

This theme was echoed by an old interview I recently watched featuring Steve Jobs. He said that the old electronics kits that he played with as a child showed him that ANYONE could build the things they saw around them. A radio, a TV, whatever – these were man made, understandable and attainable things to invent and build for yourself. He mused that these kits were one of the things in his early life that helped him understand that he could build anything he wanted and impact the environment of millions of people.

So for all of you out there waiting for permission to change your life, your career, your perspective or your world – stop waiting. Go do it. Be, Do, Act as my friend and colleague Philippe likes to say.

So at F8 last week Facebook announced Ticker, Timeline and extensions to the Open Graph API to allow for new verbs and nouns.

Here’s what really happened.

• They split their single ‘News Feed’ into 3 levels of filtering. Now (Ticker), Relevant (News Feed), Historical (Timeline). (Side note, we’ve had a ‘Ticker’ style product at Echo that we called ‘Community Stream’ for a long time now – and most of our customers and partners said to us ‘why would we want to show all that data it’s just noisy’. Maybe now they will take a second look.). Question: Will G+, Twitter and the REST of the web adopt the same model? They should.
• This allows FB to collect more ‘noise’ (also known as synaptic firings or Attention data) which, in turn, allows them to find more signal (also known as synaptic inferences or attention management). I’ve long said that the answer to information overload is not LESS information – it’s MORE. The more information you have the more ability you have to find patterns and surface them in relevant places (I said it so long ago I can’t even find the link). Question: Will independent websites think to collect their OWN Attention data BEFORE sending it to FB so they can leverage for their own purposes. The value of this data is incalculable.
• Having these new presentation metaphors in place, they then created a mechanism to collect more data in the form of expanded Verbs and Nouns in the Open Graph API. With this new API, user’s are now expected to abandon explicit gestures of sharing and instead, accept that every action they take is auto-shared to their friends. Question: When will the first horror stories start coming out about engagement ring purchases, personal health issues and sexual orientations being inappropriately revealed due to auto-sharing?
• Using all the bling of the Timeline, along with new messaging and a simple little opt in toggle of ‘Add to my timeline’ they managed to re-launch ‘Beacon’ without anyone noticing (none of the tech blogs I saw even mentioned it). Question: Why did none of the tech media cover that angle of the story?

I continue to be in awe of Facebook’s scale, seriousness, ambition and momentum. There has never been anything like it before.

They have created an Attention Management Platform that rivals Google Search and easily out classes many of my best ideas about Attention Management and Personal Relevancy back when I was thinking about the problem.

It’s breathtaking.

And since it is all done with hard links to a single proprietary hub, it is eating the web like a cancer.

Before F8 it was clear that Google+ was a 1 or 2 years behind FB. Now they are 3 or 4.

Only time will tell who, how and why more open systems will begin to reassert themselves in the ecosystem. My bet is that it wont come from a b2c copy-cat, though. It will come from a well organized, commercially incentivized b2b play.

The part that still confuses me, though, is why ANY serious media company would want their news to load in a ‘FB canvas app’ instead of their own website. It makes zero sense. None of this changes the reality that you need to own your own data and your own point source. I made a little comparison table earlier in the week that explains why.

Today WSJ announced that it has built a news publishing platform that lives inside Facebook - effectively outsourcing their core website to the Social Networking Giant.

The number of reasons this is a bad idea is staggering. I’ve tried to summarize them in a spreadsheet comparing a FB approach verses an Open Web approach.

Please feel free to contribute

There’s a lot of fury on the web right now about ‘Real Names’. FB is trying to use it as a unique feature of their comments system claiming it reduces trolling and low value comments.

Of course that isn’t really true. For one, any commenting system could force FB login. Two, users will troll with or without their name attached and, worse yet, many legitimate users won’t participate for any number of reasons if they can’t use a pseudonym. There are plenty of better ways to increase quality in your comments including participation from the content creators, game mechanics, community moderation and more.

The real debate, however, is about G+ trying to copy FB’s stance on Real Names. They are insisting all user accounts use them and are actively shutting down accounts that violate the policy. They are being so heavy handed about that even people who ARE using their real name are getting notices of violation – most notable Violet Blue.

I’m not really an expert on pseudonyms, shared contexts and anonymity so I’m going to stay out of this debate.

The real question for me, however, is what is Google’s strategic business reason for this policy. There must be a long term plan/reason for it otherwise they wouldn’t be insisting so hard.

My assumption is that it’s related to their intention to become a canonical people directory and identity provider on the internet to compete with FB in this space.

FB, after all, does not just get it’s power from news feeds and photo apps – it gets it from the deep roots it has laid down into the DNA of the internet as the provider of 1st class identity infrastructure and identity information.

In this sense, FB’s social contract has served them very well, and Google’s attempt to copy it is a hint that they understand FB is not just a .com feature set, but a powerful identity utility. They must (and in some cases seem to be) understand that strategy and it’s aggressiveness if they are to properly compete with the monopoly. My only hope, however, is that they are coming up with their own inspired counter strategy rather than just copying the moves they see on the surface – because that’s doomed to fail.

First, to define ‘Real-time’

Real-time is no CDN or Cache latency. When there is new data in the database, it’s available to the end-user.

Real-time is not needing to hit the refresh button to see new information. It’s when information folds into the page while you’re reading it.

Real-time is a new volume and velocity of data. A lot of web data used to consist of ‘Blog Posts’ or ‘News Articles’. Documents. Real-time web data is about activities. Granular, human readable micro-stories about the activities that users make.

“I read this”, “I rated this”, “I commented on this”, “I shared this”, “I edited this” and so on. Why? Because capturing, surfacing and socializing real-time activity data is part of the core essence of the social web. The ability to see not just the result of actions by users, but the play-by-play stream of those actions along side faces, names and time/date stamps takes an experience from a static ‘snapshot’ into a living, breathing stream. Further, by enabling users to like, reply, flag, share and otherwise interact with these activities, sites are creating new opportunities for engagement, conversation and conversion.

Real-time is a presentation metaphor. It often (but not always) takes the form of a reverse chronological stream with nested comments and likes. It helps users understand the order of things and mixes content with conversation in a way that drives engagement and return visits.

Real-time means filters instead of facts. Let the user decide what they want to see – to craft an experience that makes sense for them, and their friends.

Now, what is ‘Real-time as a Service’?

If all the things above are true, then it changes everything we used to know about web infrastructure, databases, user interfaces and tools for moderation or curation.

APIs can no longer be request-response. Databases must now store far more data at far faster rates. User interfaces need to factor in names, faces and actions. Moderation and curation tools must leverage algorithms, crowd sourcing and real-time flows.

Real-time as a service, then, is cloud infrastructure that helps make this transition easier.

It is a database that can handle new magnitudes of scale – handling hundreds or thousands of write events per section. Not just to a flat table, but to a hierarchical tree of arbitrary activities.

Site -> Section -> Article -> Rating -> Comment -> Reply -> Like.

It’s a database that can store all items permanently so that users can visit old streams at any time. Permanent storage that can also handle localized annotations. Localized annotations are the ability to modify the metadata of an activity – say a Tweet (Promote it, tag it, retarget it in the tree etc) – in such a way that that your view of a tweet is different from another customer’s view.

It’s a database that enables not just the ability to perform an SQL-like search query, but also continuously updates you when the data changes – so that you can modify the UI on the fly.

It’s a database that returns not just flat query results, but a hierarchical tree – allowing you to present the activity in context.

It’s a database that handles not just a few hundred users requesting (reading) data, but a few million users swarming to see the latest action in a sports game or a concert.

It’s a database that organically makes connections between items by understanding the relationships of URLs and #tags to make implicit links in the graph where and when they’re needed. For example a tweet mentioning acme.com should be attached to Acme.com in the tree.

And most importantly, it’s a database company that understands that the opportunity of the Real-time, Social Web is far too big and moves far too quickly to possibly be built by a single vendor. A company that, as a result of this understanding, chooses open standards over proprietary formats; Partnership with best-of-breed partners over trying to build mediocre versions of everything by itself.

Polls, Ratings, Comments, Live Blogging, Forums, Data Bridging, Data Enriching, Visualization, Moderation, Curation, Analytics Game Mechanics, Authentication… the list is endless. They are all transformed by the Real-time web. They must all be part of Real-time as a Service.

And finally, Real-time as a Service is about service. Enterprise grade support. Best in class uptime. White label.

That’s Real-time as a Service.

Further Reading

• Another way to describe it here.
• Here are some Case Studies for what you can build with such a service.

It’s certainly very slick, but it’s a few years behind FB.

I mean that not just in timing and network effects, but in the much more strategic sense of platform ambition. FB.com was the FB strategy 4 years ago. FB is now going for the rest of the web. It’s reach and role as an identity provider and social infrastructure player makes it much more important (and harder to beat) than launching a cool new service. So hopefully the Google+ team is thinking WAY beyond this as a destination site when they are thinking Google Social Strategy.

So far the broad ranging announcements from the +1 button to Google Analytics adding Social bode well for this being a company wide, product wide refresh. The key to success will be in thinking about the need to compete with FB beyond the walls and products of Google.

The key to that, of course, will be to get deep adoption by major sites.

Update: Upon thinking about it a little more. Google has once again missed an opportunity to play to their strengths. With the document web they played the role of aggregator and algorithmic signal detection system. With the social web, their ideal strategy would be to build the ultimate social inbox. A place where I can navigate, consume AND interact with Facebook + Twitter + Foursquare + Quora +++ in one place.

Instead they created yet another content source.

I recently wrote down the latest version of my belief system. Here it is.

Part 1

Part 2

Over the last few days I have been debating the NYT pay wall on a private email thread of friends.

I didn’t feel the need to post it on my blog because I thought that pay walls were so obviously a losing strategy that it was a waste of time to comment.

But combined with the recent law suit against the Huffingon Post and Arianna Huffington’s eqloquent response yesterday, I felt it was worth while to re-publish my thoughts here. Most of them are based on thinking and writing that I did many years ago around Attention. Most of that old writing has been lost in the blog shuffle. Hopefully one day I will dig it up and re-post it in a safe place.

On to the issue…

The price of content

I believe that people have historically paid for the medium not the content.

They pay for ‘Cable’ not for ‘CNN News’. They pay for ‘The Paper’ not for the content in the newspaper. They pay for ‘CDs’ not for the music on the album.

Also they paid a lot because the medium was perceived to be scarce (scarce materials, scarce shelf space, scarce advertising dollars), scarce talented people.

Consumers are not stupid, they understand (if only somewhere at the back of their mind) that the COST of creating and distributing things has been deflated by a growing list of converging trends.

We live in a world of abundance (in the area of digital content anyway). Shelf space is infinite (database entries), any kid in a basement can make content and there is no physical media anymore so cost of distribution has disappeared as well.

The scarcity now is on the consumption side – Attention is the scarce resource. Value is derived from scarcity.

That’s why on the Internet, Attention allocation systems (Google Search, FB News Feed etc) are attracting traffic, engagement and ultimately profit.

In this new world, the price of content must be reduced significantly as shakeouts and rebalancing occurs – because the cost of producing it is approaching zero.

The more the Music, TV and News industry fight this, the more they leave themselves open to disruption by Google, FB, Twitter and the rest of silicon valley.

This is not even to mention that everyone is producing content now. Tweets, Photos, Videos – it’s abundant. Of course most of it isn’t very ‘good’ by J school standards – but that’s irrelevant. The world has never rewarded good with any consistency.

Also just because content is not good, doesn’t mean it isn’t personally meaningful.

For example, I care more what my child (theoretical child of course) posts to FB than the most important journalist in all the world says on CNN.

But please don’t confuse my dispassionate assessment of the issue as pleasure or happiness at the demise of mainstream media though.

I am simply stating the facts because without understanding those we can’t begin to change them (if that’s what the media world decided to do).

In terms of making a judgement of those facts, I think that curators who weave and summarize a broader narrative in the form of ‘reporting’ are critical for an informed citizenship and a functional democracy. I believe in it so much that I have dedicate my life to helping mainstream media companies staying relevant and co-writing things like this: http://aboutecho.com/2010/08/18/essay-real-time-storytelling/

But I also believe that mainstream mass media broke an ancient (and by ancient, I mean as old as rudimentary human communication) pattern of people telling each other personal stories vs. getting all their stories/news from editorialized mass broadcasts.

The Internet may just be restoring the balance. The result is some massive restructuring of inflated budgets, processes, offices, costs etc. While we’re in the middle of that restructuring, it looks like a media apocalypse. Until it settles down and a new equilibrium is found.

Here’s what Arianna wrote on the subject:

The key point that the lawsuit completely ignores (or perhaps fails to understand) is how new media, new technologies, and the linked economy have changed the game, enabling millions of people to shift their focus from passive observation to active participation — from couch potato to self-expression. Writing blogs, sending tweets, updating your Facebook page, editing photos, uploading videos, and making music are options made possible by new technologies.

The same people who never question why someone would sit on a couch and watch TV for eight hours straight can’t understand why someone would find it rewarding to weigh in on the issues — great and small — that interest them. For free. They don’t understand the people who contribute to Wikipedia for free, who maintain their own blogs for free, who tweet for free, who constantly refresh and update their Facebook pages for free, and who want to help tell the stories of what is happening in their lives and in their communities… for free.

Free content — shared by people who want to connect, share their passions, and have their opinions heard — fuels much of what appears on Facebook, Twitter, Tumblr, Yelp, Foursquare, TripAdvisor, Flickr, and YouTube. As John Hrvatska, a commenter on the New York Times, wrote of the Tasini suit, “So, does this mean when YouTube was sold to Google that all the people who posted videos on YouTube should have been compensated?” (And Mr. Hrvatska no doubt contributed that original and well-reasoned thought without any expectation he’d be paid for it. He just wanted to weigh in.)

Read more on her post

Update

And here’s a bit of ‘Free Content’ – A conversation I had on Twitter wish someone who disagreed with this post.