As I May Think...

Can Social Media replace News Media in Newark?

2009-12-03T16:27:59-05:00

At a CommonGood lunch meeting today, I asked Cory Booker, Mayor of Newark NJ, if the decline of the mainstream media has made it harder for him to engage with his community. He responded that while he felt that journalism was very important to democracy and society, its recent decline has had little impact on him since he has been able to use social media technology such as Facebook, Twitter and YouTube to build direct, personal and often real-time connections with his constituents.

Today, although Newark NJ has fewer than 300,000 citizens, Mayor Booker has almost a million followers on Twitter, almost 17,000 supporters on Facebook and 550 subscribers to his YouTube channel. Booker regularly posts status updates to both Facebook and Twitter and has uploaded 75 videos including a regular "Week In Review." He regularly posts updates to his schedule, links to speeches he's given, quotes from things he's been reading, calls for participation in "Night Patrols" of Newark's streets, and kudos to local organizations whose work he admires.

Mayor Booker's comments reminded me that at the recent FTC Workshop on the future of journalism, Matthew Gentzkow of the University of Chicago presented research on "The Effect of Newspaper Entry and Exit on Electoral Politics." His research, which spanned the period from 1869 to 2004, indicated a fairly weak connection between the health of newspapers and electoral participation. He concludes that "monopoly paper closings may cause small to moderate declines in local [electoral] participation" and that there was "no evidence that newspaper closings will affect party vote shares or incumbency advantage." This evidence, combined with the personal experience of Mayor Booker, may indicate that communities are much more robust in their ability to obtain the information needed to sustain political and community engagement than some may be suggesting. Perhaps, just as on the Internet where we tend to "route around" anything that impedes the free flow of traffic, communities also find ways to route around forces that would otherwise limit discourse. Thus, it may be that rather than assuming on faith that it is vitally important to prop up and sustain the existing newspaper businesses, we should be seeking instead to determine what other tools and/or media best provide the platforms for discourse and engagement that we need.

bob wyman

PubSub.com V2.0 is live Under New Management

2009-11-18T07:06:56-05:00

Well, it seems that PubSub.com is live again -- over three years after we took it down and handed all our assets over to our creditors in June of 2006. Frankly, I've been wondering what has been taking the new management so long to relaunch the system. PubSub.com was the first "real-time search" engine on the web and given all the attention that real-time search has been getting over the last few years, it is amazing that more effort wasn't put into getting the service retooled and up sooner. Although PubSub V1.0 was widely recognized as the first in its field (even though few people at the time recognized the importance of that field...), PubSub V2.0, as a late entrant into a crowded field, may be seen as simply "yet-another" real-time search engine...

I have no idea if they are using any of the old PubSub.com code but would be surprised if they weren't. The prospective search system that we built for PubSub was the best technology in the market at the time and probably would still be the best among those systems that are currently in public use. In any case, PubSub V2.0 is a somewhat different offering than PubSub V1.0 was. They have added a traditional retrospective search system (we had been experimenting with Lucene for that) and they are now provide streaming updates to HTML pages (something that we had implemented with technology from KnowNow, the company that first popularized "hanging GETs" and with whom we had been planning to merge. Unfortunately, issues caused by a disgruntled ex-employee and apparently irrational investors caused us to cancel the merger and close shop before exposing the live-update capability.) -- PubSub V1.0, as released, had only provided real-time updates via Atom/RSS feeds and XMPP PubSub streams. I can find no evidence of support for syndication feeds or XMPP on PubSub V2.0

It will be interesting to see if PubSub.com V2.0, after so many years of letting the competition get ahead, will be able to build a market for their service. They have lost the "first mover" advantage, but may still be able to have some success. Time will tell.

bob wyman
ex-CTO and founder, PubSub.com V1.0

ASNE sans "Paper"...

2008-12-17T01:29:10-05:00

The American Society of Newspaper Editors is planning to remove "paper" from its name and expand its membership to include editors of online-only news Web sites and journalism educators. (see the release)

Of course, this isn't the first time that a professional society has changed its name in response to changing technology. An example I'm particularly familiar with is the association that began life in 1943 as the "National Microfilm Association" or "NMA." In 1969, in response to a broadening in the technologies used by its members, the association changed its name to the "National Micrographics Association." Then in 1983, as the NMA's members began to focus more and more on digital, non-image data, the association changed its name to "Association of Information and Image Management." Apparently, the association is once again in the process of changing its name. In the future, they will be known as the "Enterprise Content Management Association."

Had the NMA insisted on sticking to the old "microfilm" name, it surely would have closed its doors years ago. Certainly, it could not have been seen to be working in the interests of its members... Each time the old "NMA" changed its name, it was in recognition that the scope and range of opportunities for its members had grown. I'm sure many in the newspaper business will see the loss of "paper" in the ASNE name to be a loss for the industry. Hopefully, they will learn, as have the members of the NMA, that this kind of name changing is a good thing -- a very good thing.

bob wyman

Newspaper Classifieds: A franchise lost.

2008-06-01T12:58:32-04:00

In the early 90's, and for some time afterward, the newspaper industry had an opportunity to lead in the development of online classifieds and, in a number of forums, I actively encouraged them to take the opportunity... Today, I argue that they shouldn't put much effort into online classified ads. What made sense 15 years ago is no longer sensible.

In the early 90's, even as Internet technology was being rapidly deployed, there was still very little commerce on the Internet. The newspapers came to this new environment with an existing database of classifieds, relationships with vast numbers of advertisers, and a clear position in the minds of Internet users who had learned, through years of exposure to the paper-based pre-Internet world, that newspapers is where you went to find classifieds and job postings.

Given this opportunity, the newspapers could have not only maintained the revenue streams that then supported them, they could have vastly increased those revenues. What is the ad business of Google today, what is eBay or Monster today, could have been (some would say *should* have been) a business created, owned and dominated by newspapers. Of course, as we now know, the newspapers forfeited their historical franchise in classifieds and advertising. The result is that they will probably never recover from the loss of those revenue streams. As a secondary result of their forfeiture of these revenues, we, as a society, are now faced with the problem of finding an alternative means to fund and organize the paper-free dissemination of the news and information that we require. The newspapers have done more than just hurt their stockholders, they have failed the society that they once claimed they had a special duty and privilege to support.

But, by simply forfeiting the opportunity, it is probably the case that the newspapers simply sped up the working of inevitable economic processes. The advantage the newspapers once had was a temporary one based on the dynamics of an older and rapidly obsolescing technology. Their advantage wasn't rooted in any inherent binding between the business of journalism and the business of advertising. As such, it was always inevitable that news and ads would become distinct businesses.

The situation of newspapers in the early 90's was much like that of the many organizations that grew up as Internet Service Providers (ISPs). Those companies always knew that their opportunity was only temporary at best. It was always clear that the "proper" provider of Internet connectivity was either the phone or cable TV companies. But, since the phone and cable providers were slow to move, there was a temporary opportunity to profit from their lethargy. Thus, we saw the temporary growth (sometimes spectacular) of companies like AOL, Earthlink, and many thousands of others. Today, of course, inevitable economic processes have caused the "right" or "natural" thing to happen and the independent ISPs are consolidating or simply going out of business. Today, Internet connectivity is normally provided by those who have the cost, service and technology advantage -- the phone and cable companies... Nonetheless, quite a few sport cars, homes, and college educations were paid for by the revenues from those who took temporary advantage of the phone and cable companies' slowness to move. We have also seen the creation of a great number of companies based on what was learned by those who stood in temporarily for the phone and cable companies.

The classifieds and advertising business is very different from the business of journalism. Thus, one might wonder why the two were ever so closely tied. Of course, the connection was as loose one and, as we've seen, a fragile one. The connection between these two came about simply because both required access to the same limited, scarce resource -- the paper on which they were printed, the paper that was distributed throughout communities. What developed early in the history of newspapers was a pattern of printing what was really two publications in one. The paper was split into a news section (sprinkled with non-classified ads) and a printed database of classified ads in the back. Since both rode on the same paper, it was only those who owned the printing presses that owned both of these businesses.

Today, the channel is the Internet. Paper is dying rapidly. No one owns the Internet and access to it is essentially universal. As such, there is no channel-access driver that forces the classified ads and journalism to be owned in common. Given that there is no longer a natural binding between these two businesses, we can be sure that they would have eventually broken apart -- even if the papers had taken the opportunity that they once had to lead in online classified ads. We can also imagine that there would have inevitably grown up competition in classified ads from non-newspaper sources. For instance, we probably still would have seen businesses like eBay, Monster or Craigslist innovate in ways that newspapers didn't. The incumbent's inevitable efforts to expand their offerings to address competition would have, over time, caused those on the classified advertising side of the business to demand that they be set free of their bonds to the news side of the business. In time, we would have gotten to where we are today -- online journalism and online classified ads being distinct businesses. But, what would have been different?

Had the newspapers taken the opportunity to build serious classified businesses online, they would have also seen more clearly the opportunity and value in growing online audiences for news. The result, I'm sure, is that online news today would be very different than it is. We would have benefited from the best minds in the newspaper businesses spending the last 15 years thinking about how to do online news better in order to increase traffic to their revenue producing classified ads instead of what we got -- the best minds trying to figure out how to resist the pressure to go online and maintain their paper-based revenues. My bet is that if the papers had enjoyed years of the same online revenue streams that companies like eBay, Monster, etc. have seen, there would actually be far fewer paper-based newspapers today... The newspaper business would have learned long ago that better margins are found online.

So, where does that leave us today? Should the newspapers try at this late date to recover the online classified business? No. That would be, I am sure, a hopeless task. The opportunity is lost, the window closed. You can only fight economics temporarily and then only at specific moments in the development of an economy or market. The time for this particular battle is long over. For a newspaper to build an online classified business today would be sort of like someone building a new Internet Service Provider to compete with the phone or cable companies... It's just not worth the bother unless the technology is distinctly and greatly different.

bob wyman

Decentralized Twitter isn't hard.

2008-05-12T17:27:52-04:00

Cliff Gerrish writes on his Echovar blog:

The consumption strategy that makes the instant messaging model of Twitter work is to follow a core group and then track keywords of interest. Tracking keywords adds people you don’t follow into your stream and provides a proper level of noise and negative feedback into the information ecosystem. ... It’s tracking that makes a decentralized Twitter nearly impossible. ... This feature radically changes the shape of the social graph underlying the information stream. Since you don’t know who might use a tag you’re tracking, the regular RSS style contract around publication and subscription doesn’t work.

Gerrish's claim that tracking makes it impossible to decentralize Twitter seems to have convinced Steve Gilmore of Techcrunch who writes: "Decentralizing Twitter is unnecessary, if not impractical." Fortunately, both Cliff and Steve are wrong.

Distributed tracking of Twitter-like streams is easily accomplished using what are now well-known systems for distributed publish/subscribe. Certainly, it is easier to implement tracking if you have everything going through a single choke-point in the network, but it isn't necessary. In fact, as long ago as the 80's we had USENET based systems that handled the distributed fan-out of messages (news posts) that were then "matched" against user's local subscriptions (Yes, matching was normally trivial "topic-based" matching, however, "content-based" matching systems were often deployed locally). What we could do in the 80's we can do today -- but do it better. After all, we've learned a great deal since then.

Today, Twitter uses XMPP for both input and output. Via XMPP, Twitter offers topic-based "following" of tweets by pre-specified users and it offers content-based "tracking" of keywords and short phrases across all public tweets. Additionally, Twitter provides an experimental XMPP PubSub (XEP-0060) feed of all public tweets. Thus, almost all of what is needed to build a distributed Twitter is already in place. (Also, all that one needs to offer a "competitive" tracking offering is offered since anyone can get easy access to the full stream of public tweets.)

If Twitter thought that decentralizing their service was a good thing, then, as long as decentralization only dealt with "public" tweets and as long as they were willing to start with a crude system (to be cleaned up later), they would only have to do two things that they haven't already done:

Allow their users to "follow" Twitterers (?) that are hosted at other XMPP servers. (i.e. instead of following user "foo" on Twitter, you might follow "foo@jabber.org".
Accept a stream of "public tweets" generated by other XMPP hosts. This would allow Twitter to implement both topic-based "following" and content-based "tracking" of tweets generated off Twitter.

Of course, more elegant and efficient systems can be designed. But, a basic system is pretty easy to build. In fact, Twitter has already built most of it. However, even though the engineering delta might be small, the key thing that will probably keep Twitter centralized is Twitter's business plan -- whatever that might be... they probably see some advantage to keeping things centralized. Anyway, additional levels of capability would come from doing things like writing a quick "following" XEP (XEP= XMPP Extension Protocol) to permit the exchange of "follow" information between services (different than presence subscriptions). This "Following XEP" would enable directed messages and support private tweets. Once the following XEP was in place, XMPP services would probably adopt the convention that "tweets" should be sent to a standard local JID: "twitter".

Of course, not all XMPP servers would have the capability to do things like content-based tracking of tweets at high volume. But, this is the case with everything on the web these days. Heck, anyone can build a search engine if only they pick up a copy of Lucene, but building a search engine as powerful as Google's or Yahoo!'s is a very different problem. Thus, even if we had a distributed Twitter, it is likely that actual users of Twitter.com would get better service than the users of most other distributed Twitter services. Even over time, there would probably only be a small number of (well financed) players who could usefully support functions like tracking. Thus, we'd move from a Twitter monopoly to an oligopoly. Even so, it would be an improvement.

Distributed Twitter isn't hard -- it only requires the support of Twitter and some powerful partners.

bob wyman

Of Liquidity, Competition and Platforms

2008-04-14T11:28:27-04:00

Fred Wilson, of "A VC" recently sparked a great deal of controversy with his post: "We Need A New Path To Liquidity". Fred suggests that with the IPO market "closed," we now need new thinking about how to reward innovators for their work. Today, getting bought out seems to be the exit strategy that innovators build for, but buy-outs, while enriching those who sell, have tended to result in many innovative solutions languishing or being diluted in the process of assimilation into the purchasing organization. Luminaries such as Jeff Jarvis and Umair Haque have already commented on the main body of Fred's piece, so I'll limit my self to something he writes in response to comments:

Fred Wilson wrote:

The web is decomposing into smaller and smaller pieces And those pieces are sustainable as standalone opportunities

I think we'll find that these pieces aren't quite as sustainable as Fred would like them to be... Its a simple matter of economics.

As the pieces become "smaller and smaller," the barriers to entry are lowered and thus there are more opportunities to enter the market. Lower barriers lead to intensified competition, reduced differentiation and lowered returns for individual developers even as the overall market grows. In such a market, the qualities of your implementation eventually become irrelevant. Sustainability in such an environment comes from one of: lowered expectations, integration with a larger platform or branding. Getting bought out early is the best thing you can hope for...

Branding is ephemeral, lowered expectations tend to reduce the drive for innovation, and integration has a plethora of problems. It is hard to do, but even if done well, integration leads to the problem of the "whole being greater than parts." The result is that it is exceptionally difficult to evaluate the marginal contribution of an integrated part or component. Rewards to the component builders inevitably end up being determined by their negotiating skills or their luck -- not the true value of their work or their skill as innovators.

I learned a great deal about the "component economy" with ALL-IN-1 at Digital in the 80's. Then, I took what I had learned to Microsoft where I was Senior Product Manager for Applications Programmability in the early 90's (91-93). I used to explain to people at Microsoft on a regular basis that by building a platform that encouraged components (Visual Basic, COM, Active/X, VBA, etc.), we were "architecting monopoly" for Microsoft. (Note: I didn't say we should do these things in order to achieve monopoly, only that by doing these things, we would inevitably ensure monopoly. The difference is important.) The logic was simple. By providing ways that people could build components to extend Microsoft's products, we would lower the barriers to entry into the software market at the same time that we lead people away from competing with us. Innovators would focus on developing ever smaller components that produced sufficient rewards to motivate them but they would find themselves in a competitive marketplace that limited their returns and thus ensured that they didn't have enough revenue or energy to tackle competing with Microsoft itself. At the same time, as small developers built better components, those components would strengthen the Microsoft platform and made competition against the platform itself unthinkable. Developers would thank Microsoft for having created opportunities without realizing that the very opportunities that they exploited were precisely the things that made it impossible for them to ever aspire to grow as large as Microsoft itself.

Over time, the aggregation of innovations in any single component space would result in commoditization of the component. (i.e. dozens of small competitors, each trying to build a "better" component of type X would eventually define a "commodity" or standard implementation of the component.) Once a component was commoditized, the idea was that we would simply either build our own version of it or buy in one of the existing implementations. By packaging a component with the platform, we would remove it from the the realm of competition and force innovators to start the cycle over with some other component. (This consolidated the gains of past innovation into the platform and ensured that future innovation was focused on the remaining needs of the platform...) I always argued that the preferred approach would be to *buy* an existing implementation, and even to "over pay" for it, since to do so would invigorate developers by encouraging them to believe that one day they might be relieved of wearying competition by being bought out... The strategy seems to have worked for Microsoft, whether or not it was conscious... And, we're seeing the basic pattern repeating, perhaps less intentionally, in the Web 2.0 space.

As many others have said, much of what people are building today is "features" not products. As long as that is the case, raw economics is the real problem with the software business. Competition strengthens the platform builders while eviscerating the component builders... The rich get richer and the builders of innovative features must be satisfied with the "personal rewards" of doing a job well unless they get lucky in the buy-out lottery game.

It is important to note that while the economics of components and platforms has long-term negative consequences, in the short term the market's interests are served by this rough handling of feature/component innovators. Most customers want integrated, consistent systems and will prefer an integrated solution to a collection of "Best of Breed" implementations even if the integrated system is somewhat less powerful than a cobbled together set of Best of Breed solutions. Thus, the market drives us to integrate our components, yet, as discussed above, we simply don't know how to value and reward a component which is only a small part of a greater whole. Given that we have no model for valuing and rewarding component builders, we're lucky that large companies are willing to buy out "feature companies" and component builders. If developers couldn't look forward to such buyouts, they wouldn't have much motivation to innovate.

The alternative to building components is building platforms, but the market can never sustain more than a very small number of platforms at once -- given the way platforms work today... The platforms that "win" are typically the first in a space since platforms attract component builders and rapidly become too strong to attack. Thus, the only things you should take to IPO are platforms since they are they only things that are really sustainable (other than some classes of services, but that's another post.). And, the only opportunity you have to build platforms is when there is a major market disruption or when the platform defines a new market. Microsoft, Apple and Sun delivered the "Graphical UI" platforms. Google, Yahoo! and Amazon got the opportunity to build Web 1.0 platforms. Now, FaceBook and a few others are hoping that the "Social Web" or Web 2.0 is enough of a disruption to justify platform creation. (I doubt it...)

The "solution" to this problem lies in reducing the anti-innovative power of the platforms but without attempting the impossible which would be the elimination of platforms. I think the solution may be something like what we see in the Linux/Unix market or what we'll see once Google's Android takes hold in the mobile market. In the Linux/Unix market, the open nature of the base platform itself has removed it from the realm of competition at the same time that we see competition between packagers of the platform in the form of "distributions." (i.e. Red Hat, Gnu, Ubuntu, etc.) Each "distribution" contains a selection of components which have been integrated together to address the needs of one or another market. The result is much the same as the creation of a variety of platforms, all working on a common base and thus each platform being much easier to build (or package). As a result, there are more opportunities for component builders to find distribution and thus more opportunities for them to establish the value of their work.

Cutting this short..., in the future, I think what we'll see is a growth in the business of integrators and a growing importance of integration technologies as a source of profit and reward. If we value innovation, we'll hope to see a fragmentation of the market into clusters that build synergy around specific distributions of a common platform which has no market power of its own. For a software market to encourage sustainable innovation, there must be a variety of alternative markets for innovative components. The fewer the platforms, the less sustainable is the business of innovation.

bob wyman

3:00am Phone Calls: Genetic Testing for Presidential Candidates?

2008-04-03T10:32:37-04:00

In the recent debate over who would be best to answer 3:00am phone calls at the While House, most commentators have focused on the relative "experience" of Hilary Clinton, Barack Obama, and John MacCain. However, experience alone is the not the only measure of who is best to be hauled out of bed and faced with a crisis call. We should also consider basic elements of personality such as the degree of emotional response to stressful situations... Which candidate is most likely to answer the call with a level head and respond rationally, not emotionally?

Today, the NIH described research that finds a genetic factor in stress response variability. According to the press release:

Inherited variations in the amount of an innate anxiety-reducing molecule help explain why some people can withstand stress better than others...
Scientists led by David Goldman, M. D., chief of the NIAAA Laboratory of Neurogenetics, identified gene variants that affect the expression of a signaling molecule called neuropeptide Y (NPY)...
"NPY is induced by stress and its release reduces anxiety," said Dr. Goldman. ...
The researchers evaluated the NPY gene variants' effects on brain responses to stress and emotion. Using functional brain imaging, they found that individuals with the variant that yielded the lowest level of NPY reacted with heightened emotionality to images of threatening facial expressions. ...
In a preliminary finding, the low level NPY gene variant was found to be more common than other variants among a small sample of individuals with anxiety disorders. The researchers also found that low level NPY expression was linked to high levels of trait anxiety. "Trait anxiety is an indication of an individual's level of emotionality or worry under ordinary circumstances," explained Dr. Goldman.

So, will we one day be asking presidential candidates to submit genetic profiles so that we can objectively determine which is most likely to respond rationally and unemotionally to these 3:00am phone calls?
What sort of a world will we have once scientists are successful in working out the genetic code? Will genetic testing become a common criteria for a wide variety of jobs?

bob wyman

Am I a "Natural Born" American Citizen? (I think so...)

2008-02-29T17:06:54-05:00

Like Senator McCain, I am an American citizen born overseas to American citizens. As a result, I'm deeply and personally interested in the latest furor over whether McCain's birth in the Canal Zone disqualifies him from running for the US presidency. All my life, I've been told that because I was born in Germany I was a kind of "second class citizen," unable to even consider running for President.

Unlike McCain, whose father was in the military, my father was an American diplomat. Thus, I must admit that I'm not sure what to think about Senate bill S.2678, introduced yesterday in the US Senate by Claire McCaskill (D-MO). The bill is entitled: "A bill to clarify the law and ensure that children born to United States citizens while serving overseas in the military are eligible to become President." Of course, my question is: What about the kids of US Diplomats? What about the kids of folk that worked for Agriculture, Customs, FAA, CIA, or any of the many other US agencies? Are we somehow less worthy, less American, less good than the military kids? In fact, what about the kids born to US citizens who weren't working for the government at all? (Does the fact that one's parents worked overseas or had an unexpected birth while on vacation in the Carribean make one less of an American?)

I strongly encourage Congress to finally resolve the long standing debate over the quality of citizenship experienced by American kids born overseas. However, I feel it will be insulting and counter-productive if only the children of military families are to have their status clarified.

bob wyman

Hackers fear Electronic Voting

2008-01-10T11:30:11-05:00

When the hackers warn that your machines are easy to hack, it is wise to listen.. ComputerWorld reports that the famous German hacker club, Chaos Computer Club, has gone to court to get an injunction against the use of electronic voting machines in German elections. They argue, quite sensibly, that as an organization of hackers, they are quite confident that they could hack whatever machines might be used. Hopefully, people will listen more to these people than they have to organizations of computer professionals like the ACM (Association for Computing Machinery) who have consistently argued against the use of paperless electronic voting machines.

The Chaos Computer Club press release on this issues says, in part:

"There is a growing resistance stirring among the population against the use of the NEDAP voting computer, known to be vulnerable to manipulation", said CCC spokesman Dirk Engling. "After a virtually identical voting machine from the same manufacturer was recently completely rejected in the Netherlands, more and more concerned citizens have turned to the CCC. The voters do not understand why the same rejection cannot also be drawn in Germany. The legal path which has now been chosen is the last chance to save the transparency of the elections in Hesse."

I find it amazing that even though respected members of the computer industry have consistently argued against the use of paperless electronic voting machines, non-technical people and journalists keep suggesting that the industry might be able to build reliable machines that we could trust. Perhaps now that the hackers have spoken, the non-technical types will finally begin to realize that those of us in this industry understand what we do much better than they do...

bob wyman

A New "Classic" blog post -- with no original content

2007-11-26T16:53:43-05:00

Some blog posts (very few) are so brilliantely written that you know the instant you read them that you'll be seeing links to them for years to come. Such is the case with Mark Pilgrim's post "The Future of Reading (A Play in Six Acts)" which will hopefully teach people an important lesson about copyright, DRM, and the future of authorship online.

Mark does a wonderful job of pulling quotes from a variety of sources. By combining these purloined texts into a logical sequence he delivers a powerful message without actually ever writing a single word that is original (other than the title and section names). Mark's original contribution is all in the selection and sequencing of the texts -- a great demonstration of the power of "mashups"

My only criticism of his piece is that he didn't find a way to quote from Spider Robinson's "Melancholy Elephants."

bob wyman