O'Reilly Radar - Insight, analysis, and research about emerging technologies.

Jury to Eolas: Nobody owns the interactive web

2012-02-10T00:20:20Z

As Joe Mullin reported at Wired earlier tonight, a Texas jury has struck down a company's claim to own the interactive web. The decision in this case comes after more than a decade of legal wrangling that has drawn in some of the biggest technology companies and retailers in the world. As Timothy Lee observed at Ars Technica, Eolas, "a patent troll that has been shaking down technology companies for the better part of a decade, now faces the prospect of losing the patent."

It's a rare reversal of two software patents (7,599,985 and 5,838,906), that shouldn't have been granted in the first place. It's also an important victory for the open Internet.

Big news for open web. Eolas patents '906 and '985 ruled invalid today by a jury in Tyler TX. Early web dev (viola) presented as prior art.
— Dale Dougherty (@dalepd) February 9, 2012

As a result of the decision, the eight companies that were resisting the patent lawsuits won't have to pay anything to Eolas. If Google, YouTube, Yahoo, Amazon, Adobe, JC Penney, CDW Corp., and Staples had lost the patent infringement suit, they would have been subject to more than $600 million in damages.

The Eolas patent case represents one of the most infamous claims to ownership of the commons that grew up in universities, garages and labs in the early 1990s.

Here's a quick summary of the history: the '906 patent was applied for in 1994 and granted to Eolas in 1998. Eolas sued Microsoft in 1999. Microsoft lost that trial and settled with Eolas. The World Wide Web Consortium (W3C) and Microsoft both petitioned the U.S. Patent Office to reconsider patent. The Patent Office upheld it, both times.

The Eolas patent covers "embedded application" in a browser, a broad description of a function that was typical of client-server systems of the time. The patent was then used by Eolas founder Michael Doyle to make a broad claim about the invention of interactivity on the web, based upon a medical imaging application that enabled a user to manipulate images on a web browser with computation occurring in the background on a server.

The case appears to have turned on the demonstration of prior art by the defense. A computer science student at the University of California at Berkeley, Pei-Yuan Wei, testified during the trial that he had conceived of making interactive web features as early as 1991, including the creation of the Viola Web browser. Viola, first released in April 1992, was the first web browser with inline graphics, scripting, tables and a stylesheet. The web browser was in development at O'Reilly in 1992-1994. Another UC Berkeley student, Scott Silvey, testified that he had demonstrated such features to engineers at Sun Microsystems in 1993.

That testimony, when combined with that of web pioneers like Eric Bina, the co-founder of Netscape, and Dave Raggett, who invented the HTML "embed" tag, and Tim Berners-Lee, the inventor of the World Wide Web, was enough to convince this jury.

"It was ahead of its time," testified Berners-Lee. "The things Pei was doing would later be done in Java."

One interesting detail that emerged in the case was that the U.S. Patent Office didn't have access to the Internet in 1994 and was apparently forbidden from going on the Internet in 1997, which would make research into prior art in cyberspace somewhat of a challenge.

Patent trolls continue to be a major issue for software companies and the technology industry as a whole in 2012, as an episode of "This American Life" on when patents attack effectively communicated.

As Mike Masnick points out at TechDirt, while today was an important victory for the networked commons and civil society, Eolas still has a lot of settlement money in hand to pursue an appeal.

That said, the jury's decision to invalidate Eolas' claims of ownership regarding the basic technology that enables access to the interactive web means the company won't be suing anyone for a while.

Texas jury agreed Eolas 906 patent invalid. Good thing too!
— Tim Berners-Lee (@timberners_lee) February 9, 2012

Here's to the Open Web.

Related:

PTO Director Orders Re-Exam for '906 Patent(2003

Butting Heads Over the '906 Rebuttal
(2004)

Microsoft Reaches Settlement on EOLAS Patent (2007)

O'Reilly ebooks now optimized for Kindle Fire

2012-02-09T17:00:00Z

Earlier this week, we at O'Reilly regenerated all of our ebook-bundle Mobi files, upgrading them to meet the specifications for Amazon's latest ebook format, KF8.

These files are now available for download in your account on oreilly.com. If your ebook bundle includes a Mobi file (and more than 90% of bundles do), you can download the updated, KF8-compliant file now. (Note: All O'Reilly Media files are now available in KF8. Partner publishers will come soon.)

As always, our ebook bundles are DRM-free. See this page for instructions on loading O'Reilly Mobi files to your Kindle.

We've optimized our Mobi files for Kindle Fire by taking advantage of KF8's support of @media queries. While @media queries have been commonplace on the web for some time, they are just now making their way to ebook ecosystems. KF8's support of @media queries allows you to create an ebook that looks and potentially behaves differently based on your reading device.

For an example of @media queries in action, see the image below, which shows how the same Mobi file appears on a traditional Kindle (left) versus the new Kindle Fire (right):

Click to enlarge.

Amazon's support for @media queries makes this possible, and O'Reilly is among the first publishers to employ this feature across all of its Kindle content. Here are some of the new features that you can expect to see on your Kindle Fire (enhancements vary by book):

Color images
Syntax-highlighted code
Improved layout and design with CSS3
Embedded code font for better legibility and glyph support

Here are some screenshots from our newly optimized Mobis:

Click to enlarge.

Starting this week, our books will begin to be available in KF8 format through Amazon's Kindle Store. However, an unfortunate limitation of buying from Amazon is that they don't normally provide customers with publisher updates. By contrast, buying direct from O'Reilly gives you access to lifetime, DRM-free updates in all standard ebook formats.

Related:

Strata Week: Your personal automated data scientist

2012-02-09T16:00:00Z

Here are a few of the data stories that caught my attention this week:

Wolfram|Alpha Pro: An on-call data scientist

The computational knowledge engine Wolfram|Alpha unveiled a pro version this week. For $4.99 per month ($2.99 for students), Wolfram|Alpha Pro offers access to more of the computational power "under the hood" of the site, in part by allowing users to upload their own datasets, which Wolfram|Alpha will in turn analyze.

This includes:

Text files — Wolfram|Alpha will respond with the character and word count, provide an estimate on how long it would take to read aloud, and reveal the most common word, average sentence length and more.
Spreadsheets — It will crunch the numbers and return a variety of statistics and graphs.
Image files — It will analyze the image's dimensions, size, and colors, and let you apply several different filters.

Wolfram|Alpha Pro subscribers can upload and analyze their own datasets.

There's also a new extended keyboard that contains the Greek alphabet and other special characters for manually entering data. Data and analysis from these entries and any queries can also be downloaded.

"In a sense," writes Wolfram's founder Stephen Wolfram, "the concept is to imagine what a good data scientist would do if confronted with your data, then just immediately and automatically do that — and show you the results."

Strata 2012 — The 2012 Strata Conference, being held Feb. 28-March 1 in Santa Clara, Calif., will offer three full days of hands-on data training and information-rich sessions. Strata brings together the people, tools, and technologies you need to make data work.

Save 20% on registration with the code RADAR20

Crisis-mapping and data protection standards

Ushahidi's Patrick Meier takes a look at the recently released Data Protection Manual issued by the International Organization for Migration (IOM). According to the IOM, the manual is meant to serve as a guide to help:

" ... protect the personal data of the migrants in its care. It follows concerns about the general increase in data theft and loss and the recognition that hackers are finding ever more sophisticated ways of breaking into personal files. The IOM Data Protection Manual aims to protect the integrity and confidentiality of personal data and to prevent inappropriate disclosure."

Meier describes the manual as "required reading" but notes that there is no mention of social media in the 150-page document. "This is perfectly understandable given IOM's work," he writes, "but there is no denying that disaster-affected communities are becoming more digitally-enabled — and thus, increasingly the source of important, user-generated information."

Meier moves through the Data Protection Manual's principles, highlighting the ones that may be challenged when it comes to user-generated, crowdsourced data and raising important questions about consent, privacy, and security.

Doubting the dating industry's algorithms

Many online dating websites claim that their algorithms are able to help match singles with their perfect mate. But a forthcoming article in "Psychological Science in the Public Interest," a journal of the Association for Psychological Science, casts some doubt on the data science of dating.

According to the article's lead author Eli Finkel, associate professor of social psychology at Northwestern University, "there is no compelling evidence that any online dating matching algorithm actually works." Finkel argues that dating sites' algorithms do not "adhere to the standards of science," and adds that "it is unlikely that their algorithms can work, even in principle, given the limitations of the sorts of matching procedures that these sites use."

It's "relationship science" versus the in-take questions that most dating sites ask in order to help users create their profiles and suggest matches. Finkel and his coauthors note that some of the strongest predictors for good relationships — such as how couples interact under pressure — aren't assessed by dating sites.

The paper calls for the creation of a panel to grade the scientific credibility of each online dating site.

Got data news?

Feel free to email me.

Related:

It's time for a unified ebook format and the end of DRM

2012-02-09T14:00:00Z

This post originally appeared on Publishers Weekly.

Imagine buying a car that locks you into one brand of fuel. A new BMW, for example, that only runs on BMW gas. There are plenty of BMW gas stations around, even a few in your neighborhood, so convenience isn't an issue. But if one of those other gas stations offers a discount, a membership program, or some other attractive marketing campaign, you can't participate. You're locked in with the BMW gas stations.

This could never happen, right? Consumers are too smart to buy into something like this. Or are they? After all, isn't that exactly what's happening in the ebook world? You buy a dedicated ebook reader like a Kindle or a NOOK and you're locked in to that company's content. Part of this problem has to do with ebook formats (e.g., EPUB or Mobipocket) while another part of it stems from publisher insistence on the use of digital rights management (DRM). Let's look at these issues individually.

Platform lock-in

I've often referred to it as Amazon's not-so-secret formula: Every time I buy another ebook for my Kindle, I'm building a library that makes me that much more loyal to Amazon's platform. If I've invested thousands or even hundreds of dollars in Kindle-formatted content, how could I possibly afford to switch to another reading platform?

It would be too inconvenient to have part of my library in Amazon's Mobipocket format and the rest in EPUB. Even though I could read both on a tablet (e.g., the iPad), I'd be forced to switch between two different apps. The user interface between any two reading apps is similar but not identical, and searching across your entire library becomes a two-step process since there's no way to access all of your content within one app.

This situation isn't unique to Amazon. The same issue exists for all the other dedicated ereader hardware platforms (e.g., Kobo, NOOK, etc.). Google Books initially seemed like a solution to this problem, but it still doesn't offer mobi formats for the Kindle, so it's selling content for every format under the sun — except the one with the largest market share.

EPUB would seem to be the answer. It's a popular format based on web standards, and it's developed and maintained by an organization that's focused on openness and broad industry adoption. It also happens to be the format used by seemingly every ebook vendor except the largest one: Amazon.

Even if we could get Amazon to adopt EPUB, though, we'd still have that other pesky issue to deal with: DRM.

The myth of DRM

I often blame Napster for the typical book publisher's fear of piracy. Publishers saw what happened in the music industry and figured the only way they'd make their book content available digitally was to tightly wrap it with DRM. The irony of this is that some of the most highly pirated books were never released as ebooks. Thanks to the magic of high-speed scanner technology, any print book can easily be converted to an ebook and distributed illegally.

Some publishers don't want to hear this, but the truth is that DRM can be hacked. It does not eliminate piracy. It not only fails as a piracy deterrent, but it also introduces restrictions that make ebooks less attractive than print books. We've all read a print book and passed it along to a friend. Good luck doing that with a DRM'd ebook! What publishers don't seem to understand is that DRM implies a lack of trust. All customers are considered thieves and must be treated accordingly.

The evil of DRM doesn't end there, though. Author Charlie Stross recently wrote a terrific blog post entitled "Cutting Their Own Throats." It's all about how publisher fear has enabled a big ebook player like Amazon to further reinforce its market position, often at the expense of publishers and authors. It's an unintended consequence of DRM that's impacting our entire industry.

Given all these issues, why not eliminate DRM and trust your customers? Even the music industry, the original casualty of the Napster phenomenon, has seen the light and moved on from DRM.

TOC NY 2012 — O'Reilly's TOC Conference, being held Feb. 13-15, 2012, in New York, is where the publishing and tech industries converge. Practitioners and executives from both camps will share what they've learned and join together to navigate publishing's ongoing transformation.

Register to attend TOC 2012

Lessons from the music industry

Several years ago, Steve Jobs posted a letter to the music industry pleading for them to abandon DRM. The letter no longer appears on Apple's website, but community commentary about it lives on. My favorite part of that letter is where Jobs asks why the music industry would allow DRM to go away. The answer is that, "DRMs haven't worked, and may never work, to halt music piracy." In fact, a study last year by Rice University and Duke University contends that removing DRM can actually decrease piracy. Yes, you read that right.

I recently had an experience with my digital music collection that drove this point home for me. I had just switched from an iPhone to an Android phone and wanted to get my music from the old device onto the new one. All I had to do was drag and drop the folder containing my music in iTunes to the SD card in my new phone. It worked perfectly because the music file formats are universal and there was no DRM involved.

Imagine trying to do that with your ebook collection. Try dragging your Kindle ebooks onto your new NOOK, for example. Incompatible file formats and DRM prevent that from happening ... today. At some point in the not-too-distant future, though, I'm optimistic the book publishing industry will get to the same stage as the music industry and offer a universal, DRM-free format for all ebooks. Then customers will be free to use whatever e-reader they prefer without fear of lock-in and incompatibilities.

The music industry made the transition, why can't we?

Related:

On pirates and piracy

Book piracy: Less DRM, more data

Putting Ebook Piracy into Perspective

The Analog Hole: Another Argument Against DRM

Now available: Best of TOC 2012 anthology

2012-02-09T13:30:00Z

We just released "Best of TOC 2012," a free anthology that brings together key interviews and analysis from Radar's publishing area.

The material in Best of TOC falls into four sections:

The adaptation of publishing — The disruption in publishing is just getting started. Journalists are experimenting with ebook options over traditional outlets, readers are wrapping their heads around the concept of paperless books, and authors are wondering if they even need publishers.

Digital publishing and the legal landscape — The emerging global market for books is stirring up all sorts of legal questions concerning copyright, public domain and digital publishing rights for authors and publishers. Existing laws are slowly adapting to new media platforms as well.

Publishing tech and tools — Digital publishing is requiring tech education for everyone, from publishers to authors to readers. In addition, the rise of mobile is driving the development of publishing's next toolset.

The edge of publishing — Adaptation to a new publishing landscape starts with a change in thinking — not only in how we think about technology and books as objects, but in how we define our various roles and how we choose to collaborate.

You can download a free copy of "Best of TOC 2012" here (available in EPUB, Mobi and PDF formats).

Four short links: 9 February 2012

2012-02-09T11:00:00Z

Weave -- web-based visualization platform designed to enable visualization of any available data by anyone for any purpose. GPL and MPL-licensed. (via Flowing Data)
Flotr2 -- MIT-licensed Javascript library for drawing HTML5 charts and graphs. It is a branch of flotr which removes the Prototype dependency and includes many improvements. (via Javascript Weekly)
What Silicon Valley Gets Wrong About Math Education Again And Again (Dan Meyer) -- nicely said: it's hard to test true understanding, easy to automate only part of the testing and assessment support for learners.
mitmproxy -- GPLv3-licensed SSL-aware HTTP proxy which lets you snoop on the traffic being sent back to the mothership from apps.

Tip for B&N: Don't just follow Amazon

2012-02-08T17:00:00Z

This post is part of the TOC podcast series. You can also subscribe to the free TOC podcast through iTunes.

I follow dozens of publishing blogs and tweet streams, but there's one that always rises above the rest for me. Any time I see something from Joseph Esposito (@JosephJEsposito), president of Portable CEO consulting, I make sure I read it. He's a frequent contributor to the Scholarly Kitchen blog, and one of his recent articles there got me thinking about the need for better competition in the publishing industry. I sat down with Joe to discuss Amazon's dominance, what B&N should do to improve its position and much more.

Key points from the full video interview (below) include:

"B&N needs an 'MCI solution'" — Amazon is the clear market leader and, as #2, B&N must avoid just following Amazon's lead and come up with a completely new and different product and content model. What B&N is doing with in-store Nook merchandising is great, but they've got to go much further. [Discussed at the 1:00 mark.]

Can B&N do anything to disrupt Amazon Prime? — Amazon and anyone else creating a Prime-like service will start to run into the same challenges Netflix has encountered. [Discussed at 4:07.]

Broad content repositories vs. narrow, vertical ones — Specific genres lend themselves more to this sort of offering, and each one could have a different pricing model. Safari Books Online is a great example. [Discussed at 5:52.]

Pay-for-performance is the only option — Amazon has publicly stated that the Kindle Owner's Lending Library program pays most publishers a flat fee. I strongly believe that's the wrong model, and Joe talks about why the flat fee probably won't be a viable long-term option. [Discussed at 6:45.]

Apps vs. HTML5/EPUB — Publishers are starting to figure out that platform-specific investments often aren't wise. Development costs for a single platform, even if that's iOS, are still high, so the future leads to more open, portable solutions. [Discussed at 8:26.]

DRM — Joe makes an excellent point when he notes that, "the pro-DRM stance that many publishers have is not really getting them anywhere." [Discussed at 11:05.]

Discoverability & recommendations — Discoverability will continue to get worse before it improves, but better integration with the social graph can provide a way forward. [Discussed at 15:06.]

You can view the entire interview in the following video.

TOC NY 2012 — O'Reilly's TOC Conference, being held Feb. 13-15, 2012, in New York City, is where the publishing and tech industries converge. Practitioners and executives from both camps will share what they've learned and join together to navigate publishing's ongoing transformation.

Register to attend TOC 2012

Related:

Coming soon to a location near you: The Amazon Store?

Barnes & Noble: It's Time to Disrupt the Industry!

Hating Amazon is not a strategy

Open Question: Is it realistic for publishers to cut Amazon out of the equation?

More TOC Podcasts

The NoSQL movement

2012-02-08T14:00:00Z

In a conversation last year, Justin Sheehy, CTO of Basho, described NoSQL as a movement, rather than a technology. This description immediately felt right; I've never been comfortable talking about NoSQL, which when taken literally, extends from the minimalist Berkeley DB (commercialized as Sleepycat, now owned by Oracle) to the big iron HBase, with detours into software as fundamentally different as Neo4J (a graph database) and FluidDB (which defies description).

But what does it mean to say that NoSQL is a movement rather than a technology? We certainly don't see picketers outside Oracle's headquarters. Justin said succinctly that NoSQL is a movement for choice in database architecture. There is no single overarching technical theme; a single technology would belie the principles of the movement.

Think of the last 15 years of software development. We've gotten very good at building large, database-backed applications. Many of them are web applications, but even more of them aren't. "Software architect" is a valid job description; it's a position to which many aspire. But what do software architects do? They specify the high-level design of applications: the front end, the APIs, the middleware, the business logic — the back end? Well, maybe not.

Since the '80s, the dominant back end of business systems has been a relational database, whether Oracle, SQL Server or DB2. That's not much of an architectural choice. Those are all great products, but they're essentially similar, as are all the other relational databases. And it's remarkable that we've explored many architectural variations in the design of clients, front ends, and middleware, on a multitude of platforms and frameworks, but haven't until recently questioned the architecture of the back end. Relational databases have been a given.

Many things have changed since the advent of relational databases:

We're dealing with much more data. Although advances in storage capacity and CPU speed have allowed the databases to keep pace, we're in a new era where size itself is an important part of the problem, and any significant database needs to be distributed.

We require sub-second responses to queries. In the '80s, most
database queries could run overnight as batch jobs. That's no
longer acceptable. While some analytic functions can still run as
overnight batch jobs, we've seen the web evolve from static files
to complex database-backed sites, and that requires sub-second
response times for most queries.

We want applications to be up 24/7. Setting up redundant
servers for static HTML files is easy, but a database replication
in a complex database-backed application is another.

We're seeing many applications in which the database has to
soak up data as fast (or even much faster) than it processes
queries: in a logging application, or a distributed sensor
application, writes can be much more frequent than reads.
Batch-oriented ETL (extract, transform, and load) hasn't
disappeared, and won't, but capturing high-speed data flows is
increasingly important.

We're frequently dealing with changing data or with
unstructured data. The data we collect, and how we use it, grows
over time in unpredictable ways. Unstructured data isn't a
particularly new feature of the data landscape, since unstructured
data has always existed, but we're increasingly unwilling to force
a structure on data a priori.

We're willing to sacrifice our sacred cows. We know that
consistency and isolation and other properties are very valuable,
of course. But so are some other things, like latency and
availability and not losing data even if our primary server goes
down. The challenges of modern applications make us realize that
sometimes we might need to weaken one of these constraints in order
to achieve another.

These changing requirements lead us to different tradeoffs and compromises when designing software. They require us to rethink what we require of a database, and to come up with answers aside from the relational databases that have served us well over the years. So let's look at these requirements in somewhat more detail.

Microsoft SQL Server is a comprehensive information platform offering enterprise-ready technologies and tools that help businesses derive maximum value from information at the lowest TCO. SQL Server 2012 launches next year, offering a cloud-ready information platform delivering mission-critical confidence, breakthrough insight, and cloud on your terms; find out more at www.microsoft.com/sql.

Size, response, availability

It's a given that any modern application is going to be distributed. The size of modern datasets is only one reason for distribution, and not the most important. Modern applications (particularly web applications) have many concurrent users who demand reasonably snappy response. In their 2009 Velocity Conference talk, Performance Related Changes and their User Impact, Eric Schurman and Jake Brutlag showed results from independent research projects at Google and Microsoft. Both projects demonstrated imperceptibly small increases in response time cause users to move to another site; if response time is over a second, you're losing a very measurable percentage of your traffic.

If you're not building a web application — say you're doing business analytics, with complex, time-consuming queries — the world has changed, and users now expect business analytics to run in something like real time. Maybe not the sub-second latency required for web users, but queries that run overnight are no longer acceptable. Queries that run while you go out for coffee are marginal. It's not just a matter of convenience; the ability to run dozens or hundreds of queries per day changes the nature of the work you do. You can be more experimental: you can follow through on hunches and hints based on earlier queries. That kind of spontaneity was impossible when research went through the DBA at the data warehouse.

Whether you're building a customer-facing application or doing internal analytics, scalability is a big issue. Vertical scalability (buy a bigger, faster machine) always runs into limits. Now that the laws of physics have stalled Intel-architecture clock speeds in the 3.5GHz range, those limits are more apparent than ever. Horizontal scalability (build a distributed system with more nodes) is the only way to scale indefinitely. You're scaling horizontally even if you're only buying single boxes: it's been a long time since I've seen a server (or even a high-end desktop) that doesn't sport at least four cores. Horizontal scalability is tougher when you're scaling across racks of servers at a colocation facility, but don't be deceived: that's how scalability works in the 21st century, even on your laptop. Even in your cell phone. We need database technologies that aren't just fast on single servers: they must also scale across multiple servers.

Modern applications also need to be highly available. That goes without saying, but think about how the meaning of "availability" has changed over the years. Not much more than a decade ago, a web application would have a single HTTP server that handed out static files. These applications might be data-driven; but "data driven" meant that a batch job rebuilt the web site overnight, and user transactions were queued into a batch processing system, again for processing overnight. Keeping such a system running isn't terribly difficult. High availability doesn't impact the database. If the database is only engaged in batched rebuilds or transaction processing, the database can crash without damage. That's the world for which relational databases were designed. In the '80s, if your mainframe ran out of steam, you got a bigger one. If it crashed, you were down. But when databases became a living, breathing part of the application, availability became an issue. There is no way to make a single system highly available; as soon as any component fails, you're toast. Highly available systems are, by nature, distributed systems.

If a distributed database is a given, the next question is how much work a distributed system will require. There are fundamentally two options: databases that have to be distributed manually, via sharding; and databases that are inherently distributed. Relational databases are split between multiple hosts by manual sharding, or determining how to partition the datasets based on some properties of the data itself: for example, first names starting with A-K on one server, L-Z on another. A lot of thought goes into designing a sharding and replication strategy that doesn't impair performance, while keeping the data relatively balanced between servers. There's a third option that is essentially a hybrid: databases that are not inherently distributed, but that are designed so they can be partitioned easily. MongoDB is an example of a database that can be sharded easily (or even automatically); HBase, Riak, and Cassandra are all inherently distributed, with options to control how replication and distribution work.

What database choices are viable when you need good interactive response? There are two separate issues: read latency and write latency. For reasonably simple queries on a database with well-designed indexes, almost any modern database can give decent read latency, even at reasonably large scale. Similarly, just about all modern databases claim to be able to keep up with writes at high-speed. Most of these databases, including HBase, Cassandra, Riak, and CouchDB, write data immediately to an append-only file, which is an extremely efficient operation. As a result, writes are often significantly faster than reads.

Whether any particular database can deliver the performance you need depends on the nature of the application, and whether you've designed the application in a way that uses the database efficiently: in particular, the structure of queries, more than the structure of the data itself. Redis is an in-memory database with extremely fast response, for both read and write operations; but there are a number of tradeoffs. By default, data isn't saved to disk, and is lost if the system crashes. You can configure Redis for durability, but at the cost of some performance. Redis is also limited in scalability; there's some replication capability, but support for clusters is still coming. But if you want raw speed, and have a dataset that can fit into memory, Redis is a great choice.

It would be nice if there were some benchmarks to cover database performance in a meaningful sense, but as the saying goes, "there are lies, damned lies, and benchmarks." In particular, no small benchmark can properly duplicate a real test-case for an application that might reasonably involve dozens (or hundreds) of servers.

Changing data and cheap lunches

NoSQL databases are frequently called "schemaless" because they don't have the formal schema associated with relational databases. The lack of a formal schema, which typically has to be designed before any code is written, means that schemaless databases are a better fit for current software development practices, such as agile development. Starting from the simplest thing that could possibly work and iterating quickly in response to customer input doesn't fit well with designing an all-encompassing data schema at the start of the project. It's impossible to predict how data will be used, or what additional data you'll need as the project unfolds. For example, many applications are now annotating their data with geographic information: latitudes and longitudes, addresses. That almost certainly wasn't part of the initial data design.

How will the data we collect change in the future? Will we be collecting biometric information along with tweets and Foursquare checkins? Will music sites such as Last.FM and Spotify incorporate factors like blood pressure into their music selection algorithms? If you think these scenarios are futuristic, think about Twitter. When it started out, it just collected bare-bones information with each tweet: the tweet itself, the Twitter handle, a timestamp, and a few other bits. Over its five-year history, though, lots of metadata has been added. A tweet may be 140 characters at most, but a couple KB is actually sent to the server, and all of this is saved in the database. Up-front schema design is a poor fit in a world where data requirements are fluid.

In addition, modern applications frequently deal with unstructured data: blog posts, web pages, voice transcripts, and other data objects that are essentially text. O'Reilly maintains a substantial database of job listings for some internal research projects. The job descriptions are chunks of text in natural languages. They're not unstructured because they don't fit into a schema. You can easily create a JOBDESCRIPTION column in a table, and stuff text strings into it. It's that knowing the data type and where it fits in the overall structure doesn't help. What are the questions you're likely to ask? Do you want to know about skills, certifications, the employer's address, the employer's industry? Those are all valid columns for a table, but you don't know what you care about in advance; you won't find equivalent information in each job description; and the only way to get from the text to the data is through various forms of pattern matching and classification. Doing the classification up front, so you could break a job listing down into skills, certifications, etc., is a huge effort that would largely be wasted. The guys who work with this data recently had fits disambiguating "Apple Computer" from "apple orchard." Would you even know this was a problem outside of a concrete research project based on a concrete question? If you're just pre-populating an INDUSTRY column from raw data, would you notice that lots of computer industry jobs were leaking into fruit farming? A JOBDESCRIPTION column doesn't hurt, but doesn't help much either, and going further, by trying to design a schema around the data that you'll find in the unstructured text, that definitely hurts. The kinds of questions you're likely to ask have everything to do with the data itself, and little to do with that data's relations to other data.

However, it's really a mistake to say that NoSQL databases have no schema. In a document database, such as CouchDB or MongoDB, documents are key-value pairs. While you can add documents with differing sets of keys (missing keys or extra keys), or even add keys to documents over time, applications still must know that certain keys are present to query the database; indexes have to be set up to make searches efficient. The same thing applies to column-oriented databases, such as HBase and Cassandra. While any row may have as many columns as needed, some up-front thought has to go into what columns are needed to organize the data. In most applications, a NoSQL database will require less up-front planning, and offer more flexibility as the application evolves. As we'll see, data design revolves more around the queries you want to ask than the domain objects that the data represents. It's not a free lunch; possibly a cheap lunch, but not free.

What kinds of storage models do the more common NoSQL databases support? Redis is a relatively simple key-value store, but with a twist: values can be data structures (lists and sets), not just strings. It supplies operations for working directly with sets and lists (for example, union and intersection).

CouchDB and MongoDB both store documents in JSON format, where JSON is a format originally designed for representing JavaScript objects, but now available in many languages. So on one hand, you can think of CouchDB and MongoDB as object databases; but you could also think of a JSON document as a list of key-value pairs. Any document can contain any set of keys, and any key can be associated with an arbitrarily complex value that is itself a JSON document. CouchDB queries are views, which are themselves documents in the database that specify searches. Views can be very complex, and can use a built-in MapReduce facility to process and summarize results. Similarly, MongoDB queries are JSON documents, specifying fields and values to match, and query results can be processed by a built in MapReduce. To use either database effectively, you start by designing your views: what do you want to query, and how. Once you do that, it will become clear what keys are needed in your documents.

Riak can also be viewed as a document database, though with more flexibility about document types. It natively handles JSON, XML, and plain text, and a plug-in architecture allows you to add support for other document types. Searches "know about" the structure of JSON and XML documents. Like CouchDB, Riak incorporates MapReduce to perform complex queries efficiently.

Cassandra and HBase are usually called column-oriented databases, though a better term is a "sparse row store." In these databases, the equivalent to a relational "table" is a set of rows, identified by a key. Each row consists of an unlimited number of columns; columns are essentially keys that let you look up values in the row. Columns can be added at any time, and columns that are unused in a given row don't occupy any storage. NULLs don't exist. And since columns are stored contiguously, and tend to have similar data, compression can be very efficient, and searches along a column are likewise efficient. HBase describes itself as a database that can store billions of rows with millions of columns.

How do you design a schema for a database like this? As with the document databases, your starting point should be the queries you'll want to make. There are some radically different possibilities. Consider storing logs from a web server. You may want to look up the IP addresses that accessed each URL you serve. The URLs can be the primary key; each IP address can be a column. This approach will quickly generate thousands of unique columns, but that's not a problem — and a single query, with no joins, gets you all the IP addresses that accessed a single URL. If some URLs are visited by many addresses, and some are only visited by a few, that's no problem: remember that NULLs don't exist. This design isn't even conceivable in a relational database. You can't have a table that doesn't have a fixed number of columns.

Now, let's make it more complex: you're writing an ecommerce application, and you'd like to access all the purchases that a given customer has made. The solution is similar. The column family is organized by customer ID (primary key), you have columns for first name, last name, address, and all the normal customer information, plus as many rows as are needed for each purchase. In a relational database, this would probably involve several tables and joins. In the NoSQL databases, it's a single lookup. Schema design doesn't go away, but it changes: you think about the queries you'd like to execute, and how you can perform those efficiently.

This isn't to say that there's no value to normalization, just that data design starts from a different place. With a relational database, you start with the domain objects, and represent them in a way that guarantees that virtually any query can be expressed. But when you need to optimize performance, you look at the queries you actually perform, then merge tables to create longer rows, and do away with joins wherever possible. With the schemaless databases, whether we're talking about data structure servers, document databases, or column stores, you go in the other direction: you start with the query, and use that to define your data objects.

The sacred cows

The ACID properties (atomicity, consistency, isolation, durability) have been drilled into our heads. But even these come into play as we start thinking seriously about database architecture. When a database is distributed, for instance, it becomes much more difficult to achieve the same kind of consistency or isolation that you can on a single machine. And the problem isn't just that it's "difficult" but rather that achieving them ends up in direct conflict with some of the reasons to go distributed. It's not that properties like these aren't very important — they certainly are — but today's software architects are discovering that they require the freedom to choose when it might be worth a compromise.

What about transactions, two-phase commit, and other mechanisms inherited from big iron legacy databases? If you've read almost any discussion of concurrent or distributed systems, you've heard that banking systems care a lot about consistency. What if you and your spouse withdraw money from the same account at the same time? Could you overdraw the account? That's what ACID is supposed to prevent. But a few months ago, I was talking to someone who builds banking software, and he said "If you really waited for each transaction to be properly committed on a world-wide network of ATMs, transactions would take so long to complete that customers would walk away in frustration. What happens if you and your spouse withdraw money at the same time and overdraw the account? You both get the money; we fix it up later."

This isn't to say that bankers have discarded transactions, two-phase commit and other database techniques; they're just smarter about it. In particular, they're distinguishing between local consistency and absolutely global consistency. Gregor Hohpe's classic article Starbucks Does Not Use Two-Phase Commit makes a similar point: in an asynchronous world, we have many strategies for dealing with transactional errors, including write-offs. None of these strategies are anything like two-phase commit. They don't force the world into inflexible, serialized patterns.

The CAP theorem is more than a sacred cow; it's a law of the database universe that can be expressed as "Consistency, Availability, Partition Tolerance: choose two." But let's rethink relational databases in light of this theorem. Databases have stressed consistency. The CAP theorem is really about distributed systems, and as we've seen, relational databases were developed when distributed systems were rare and exotic at best. If you needed more power, you bought a bigger mainframe. Availability isn't an issue on a single server: if it's up, it's up, if it's down, it's down. And partition tolerance is meaningless when there's nothing to partition. As we saw at the beginning of this article, distributed systems are a given for modern applications; you won't be able to scale to the size and performance you need on a single box. So the CAP theorem is historically irrelevant to relational databases: they're good at providing consistency, and they have been adapted to provide high availability with some success, but they are hard to partition without extreme effort or extreme cost.

Since partition tolerance is a fundamental requirement for distributed applications, it becomes a question of what to sacrifice: consistency or availability. There have been two approaches: Riak and Cassandra stress availability, while HBase has stressed consistency. With Cassandra and Riak, the tradeoff between consistency and availability is tunable. CouchDB and MongoDB are essentially single-headed databases, and from that standpoint, availability is a function of how long you can keep the hardware running. However, both have add-ons that can be used to build clusters. In a cluster, CouchDB and MongoDB are eventually consistent (like Riak and Cassandra); availability depends on what you do with the tools they provide. You need to set up sharding and replication, and use what's essentially a proxy server to present a single interface to cluster's clients. BigCouch is an interesting effort to integrate clustering into CouchDB, making it more like Riak. Now that Cloudant has announced that it is merging BigCouch and CouchDB, we can expect to see clustering become part of the CouchDB core.

We've seen that absolute consistency isn't a hard requirement for banks, nor is it the way we behave in our real-world interactions. Should we expect it of our software? Or do we care more about availability?

It depends. The consistency requirements of many social applications are very soft. You don't need to get the correct number of Twitter or Facebook followers every time you log in. If you search, you probably don't care if the results don't contain the comments that were posted a few seconds ago. And if you're willing to accept less-than-perfect consistency, you can make huge improvements in performance. In the world of big-data-backed web applications, with databases spread across hundreds (or potentially thousands) of nodes, the performance penalty of locking down a database while you add or modify a row is huge; if your application has frequent writes, you're effectively serializing all the writes and losing the advantage of the distributed database. In practice, in an "eventually consistent" database, changes typically propagate to the nodes in tenths of a second; we're not talking minutes or hours before the database arrives in a consistent state.

Given that we have all been battered with talk about "five nines" reliability, and given that it is a big problem for any significant site to be down, it seems clear that we should prioritize availability over consistency, right? The architectural decision isn't so easy, though. There are many applications in which inconsistency must eventually be dealt with. If consistency isn't guaranteed by the database, it becomes a problem that the application has to manage. When you choose availability over consistency, you're potentially making your application more complex. With proper replication and failover strategies, a database designed for consistency (such as HBase) can probably deliver the availability you require; but this is another design tradeoff. Regardless of the database you're using, more stringent reliability requirements will drive you toward exotic engineering. Only you can decide the right balance for your application. The point isn't that any given decision is right or wrong, but that you can (and have to) choose, and that's a good thing.

Other features

I've completed a survey of the major tradeoffs you need to think about in selecting a database for a modern big data application. But the major tradeoffs aren't the only story. There are many database projects with interesting features. Here are a some of the ideas and projects I find most interesting:

Scripting: Relational databases all come with some variation of the SQL language, which can be seen as a scripting language for data. In the non-relational world, a number of scripting languages are available. CouchDB and Riak support JavaScript, as does MongoDB. The Hadoop project has spawned a several data scripting languages that are usable with HBase, including Pig and Hive. The Redis project is experimenting with integrating the Lua scripting language.

RESTful interfaces: CouchDB and Riak are unique in offering
RESTful interfaces. These are interfaces based on HTTP and the architectural
style elaborated in Roy Fielding's doctoral
dissertation and Restful Web
Services. CouchDB goes so far as to serve as a web application
framework. Riak also offers a more traditional protocol buffer
interface, which is a better fit if you expect a high volume of
small requests.

Graphs: Neo4J is a special
purpose database designed for maintaining large graphs: data where
the data items are nodes, with edges representing the connections
between the nodes. Because graphs are extremely flexible data
structures, a graph database can emulate any other kind of
database.

SQL: I've been discussing the NoSQL movement, but SQL is a
familiar language, and is always just around the corner. A couple
of startups are working on adding SQL to Hadoop-based datastores:
DrawnToScale (which
focuses on low-latency, high-volume web applications) and "http://www.hadapt.com/">Hadapt (which focuses on analytics and
bringing data warehousing into the 20-teens). In a few years, will
we be looking at hybrid databases that take advantage of both
relational and non-relational models? Quite possibly.

Scientific data: Yet another direction comes from SciDB, a database project aimed at the
largest scientific applications (particularly the Large Synoptic Survey Telescope).
The storage model is based on multi-dimensional arrays. It is
designed to scale to hundreds of petabytes of storage, collecting
tens of terabytes per night. It's still in the relatively early
stages.

Hybrid architectures: NoSQL is really about architectural
choice. And perhaps the biggest expression of architectural choice
is a hybrid architecture: rather than using a single database
technology, mixing and matching technologies to play to their
strengths. I've seen a number of applications that use traditional
relational databases for the portion of the data for which the
relational model works well, and a non-relational database for the
rest. For example, customer data could go into a relational
database, linked to a non-relational database for unstructured data
such as product reviews and recommendations. It's all about
flexibility. A hybrid architecture may be the best way to integrate
"social" features into more traditional ecommerce sites.

These are only a few of the interesting ideas and projects that are floating around out there. Roughly a year ago, I counted a couple dozen non-relational database projects; I'm sure there are several times that number today. Don't hesitate to add notes about your own projects in the comments.

In the end

In a conversation with Eben Hewitt, author of Cassandra: The Definitive Guide, Eben summarized what you need to think about when architecting the back end of a data-driven system. They're the same issues software architects have been dealing with for years: you need to think about the whole ecosystems in which the application works; you need to consider your goals (Do you require high availability? Fault tolerance?); you need to consider support options; you need to isolate what will change over the life of the application, and separate that from what remains the same. The big difference is that now there are options; you don't have to choose the relational model. There are other options for building large databases that scale horizontally, are highly available, and can deliver great performance to users. And these options, the databases that make up the NoSQL movement, can often achieve these goals with greater flexibility and lower cost.

It used to be said that nobody got fired for buying IBM. Then nobody got fired for buying Microsoft. Now, I suppose, nobody gets fired for buying Oracle. But just as the landscape changed for IBM and Microsoft, it's shifting again, and even Oracle has a NoSQL solution. Rather than relational databases being the default, we're moving into a world where developers are considering their architectural options, and deciding which products fit their application: how the databases fit into their programming model, whether they can scale in ways that make sense for the application, whether they have strong or relatively weak consistency requirements.

For years, the relational default has kept developers from understanding their real back-end requirements. The NoSQL movement has given us the opportunity to explore what we really require from our databases, and to find out what we already knew: there is no one-size-fits-all solution.

Related:

Four short links: 8 February 2012

2012-02-08T11:00:00Z

Mavuno -- an open source, modular, scalable text mining toolkit built upon Hadoop. (Apache-licensed)
Cow Clicker -- Wired profile of Cowclicker creator Ian Bogost. I was impressed by Cow Clickers [...] have turned what was intended to be a vapid experience into a source of camaraderie and creativity. People create communities around social activities, even when they are antisocial. (via BoingBoing)
Unicode Has a Pile of Poo Character (BoingBoing) -- this is perfect.
The Research Works Act and the Breakdown of Mutual Incomprehension (Cameron Neylon) -- an excellent summary of how researchers and publishers view each other and their place in the world.

Unstructured data is worth the effort when you've got the right tools

2012-02-07T14:00:00Z

It's dawning on companies that data analysis can yield insights and inform business decisions. As data-driven benefits grow, so do our demands about what more data can tell us and what other types we can mine.

During her PhD studies, Alyona Medelyan (@zelandiya) developed Maui, an open source tool that performs as well as professional librarians in identifying main topics in documents. Medelyan now leads the research and development of API-based products at Pingar.

Pingar senior software researcher Anna Divoli (@annadivoli) studied sentence extraction for semi-automatic annotation of biological databases. Her current research focuses on developing methodologies for acquiring knowledge from textual data.

"Big data is important in many diverse areas, such as science, social media, and enterprise," observes Divoli. "Our big data niche is analysis of unstructured text." In the interview below, Medelyan and Divoli describe their work and what they see on the horizon for unstructured data analysis.

How did you get started in big data?

Anna Divoli: I began working with big data as it relates to science during my PhD. I worked with bioinformaticians who mined proteomics data. My research was on mining information from the biomedical literature that could serve as annotation in a database of protein families.

Alyona Medelyan: Like Anna, I mainly focus on unstructured data and how it can be managed using clever algorithms. During my PhD in natural language processing and data mining, I started applying such algorithms to large datasets to investigate how time-consuming data analysis and processing tasks can be automated.

What projects are you working on now?

Alyona Medelyan: For the past two years at Pingar, I've been developing solutions for enterprise customers who accumulate unstructured data and want to search, analyze, and explore this data efficiently. We develop entity extraction, text summarization, and other text analytics solutions to help scrub and interpret unstructured data in an organization.

Anna Divoli: We're focusing on several verticals that struggle with too much textual data, such as bioscience, legal, and government. We also strive to develop language-independent solutions.

What are the trends and challenges you're seeing in the big data space?

Anna Divoli: There are plenty of trends that span various aspects of big data, such as making the data accessible from mobile devices, cloud solutions, addressing security and privacy issues, and analyzing social data.

One trend that is pertinent to us is the increasing popularity of APIs. Plenty of APIs exist that give access to large datasets, but there also powerful APIs that manage big data efficiently, such as text analytics, entity extraction, and data mining APIs.

Alyona Medelyan: The great thing about APIs is that they can be integrated into existing applications used inside an organization.

With regard to the challenges, enterprise data is very messy, inconsistent, and spread out across multiple internal systems and applications. APIs like the ones we're working on can bring consistency and structure to a company's legacy data.

The presentation you'll be giving at the Strata Conference will focus on practical applications of mining unstructured data. Why is this an important topic to address?

Anna Divoli: Every single organization in every vertical deals with unstructured data. Tons of text is produced daily — emails, reports, proposals, patents, literature, etc. This data needs to be mined to allow fast searching, easy processing, and quick decision making.

Alyona Medelyan: Big data often stands for structured data that is collected into a well-defined database — who bought which book in an online bookstore, for example. Such databases are relatively easy to mine because they have a consistent form. At the same time, there is plenty of unstructured data that is just as valuable, but it's extremely difficult to analyze it because it lacks structure. In our presentation, we will show how to detect structure using APIs, natural language processing and text mining, and demonstrate how this creates immediate value for business users.

Are there important new tools or projects on the horizon for big data?

Alyona Medelyan: Text analytics tools are very hot right now, and they improve daily as scientists come up with new ways of making algorithms understand written text more accurately. It is amazing that an algorithm can detect names of people, organizations, and locations within seconds simply by analyzing the context in which words are used. The trend for such tools is to move toward recognition of further useful entities, such as product names, brands, events, and skills.

Anna Divoli: Also, entity relation extraction is an important trend. A relation that consistently connects two entities in many documents is important information in science and enterprise alike. Entity relation extraction helps detect new knowledge in big data.

Other trends include detecting sentiment in social data, integrating multiple languages, and applying text analytics to audio and video transcripts. The number of videos grows at a constant rate, and transcripts are even more unstructured than written text because there is no punctuation. That's another exciting area on the horizon!

Who do you follow in the big data community?

Alyona Medelyan: We tend to follow researchers in areas that are used for dealing with big data, such as natural language processing, visualization, user experience, human computer information retrieval, as well as the semantic web. Two of them are also speaking at Strata this year: Daniel Tunkelang and Marti Hearst.

This interview was edited and condensed.

Related:

Four short links: 7 February 2012

2012-02-07T11:00:00Z

Integrated Content Editor (GitHub) -- a track changes implementation, built in javascript, for anything that is contenteditable on the web, written by the NY Times team and open sourced.
Data Tables -- featureful jQuery plugin for tables of data. (via Javascript Weekly)
Creating a Developer Community (Slideshare) -- treat the problem like a channel conversion funnel: turn visitors into downloaders, downloaders into users, users into contributors. His screenshots of shitty conversions are great! (via Kohsuke Kawaguchi)
Sex Differences in Intimate Relationships (PDF) -- Albert-Laszlo Barabasi and others use social graph analysis to analyze communications patterns in relationships. Notice that not only does the preference for an opposite-sex “best friend” kick in significantly earlier for females than for males (~18 years vs mid-20s, respectively), but females maintain a higher plateau value for much longer. More reality mining to understand ourselves. (via Sean Gourley)

Small Massachusetts HIT conference returns to big issues in health care

2012-02-06T19:46:10Z

I've come to look forward to the Massachusetts Heath Data Consortium's annual HIT conference because--although speakers tout the very real and impressive progress made by Massachusetts health providers--you can also hear acerbic and ruthlessly candid critiques of policy and the status quo. Two notable take-aways from last year's conference (which I wrote up at the time) were the equivalence of old "managed care" to new "accountable care organizations" and the complaint that electronic health records were "too expensive, too hard to use, and too disruptive to workflow." I'll return to these claims later.

The sticking point: health information exchange

This year, the spears were lobbed by Ashish Jha of Harvard Medical School, who laid out a broad overview of progress since the release of meaningful use criteria and then accused health care providers of undermining one of its main goals, the exchange of data between different providers who care for the same patient. Through quantitative research (publication in progress), Jha's researchers showed a correlation between fear of competition and low adoption of HIEs. Hospitals with a larger, more secure position in their markets, or in more concentrated markets, were more likely to join an HIE.

The research bolsters Jha's claim that the commonly cited barriers to using HIEs (technical challenges, cost, and privacy concerns) are surmountable, and that the real problem is a refusal to join because a provider fears that patients would migrate to other providers. It seems to me that the government and public can demand better from providers, but simply cracking the whip may be ineffective. Nor should it be necessary. An urgent shortage of medical care exists everywhere in the country, except perhaps a few posh neighborhoods. There's plenty for all providers. Once insurance is provided to all the people in need, no institution should need to fear a lack of business, unless its performance record is dismal.

Jha also put up some research showing a strong trend toward adopting electronic health records, although the small offices that give half the treatment in the United States are still left behind. He warned that to see big benefits, we need to bring in health care institutions that are currently given little attention by the government--nursing home, rehab facilities, and so forth--and give them incentives to digitize. He wrapped up by quoting David Blumenthal, former head of the ONC, on the subject of HIEs. Blumenthal predicted that we'd see EHRs in most providers over the next few years, and that the real battle would be getting them to adopt health information exchange.

Meanwhile, meaningful use could trigger a shake-out in the EHR industry, as vendors who have spent years building silo'd projects fail to meet the Stage 2 requirements that fulfill the highest aspirations of the HITECH act that defined meaningful use, including health information exchange. Meanwhile, a small but steadily increasing number of open source projects have achieved meaningful use certification. So we'll see more advances in the adoption of both EHRs and HIEs.

Low-hanging fruit signals a new path for cost savings

The big achievement in Massachusetts, going into the conference today, was a recent agreement between the state's major insurer, Blue Cross Blue Shield, and the 800-pound gorilla of the state's health care market, Partners HealthCare System. The pact significantly slows the skyrocketing costs that we've all become accustomed to in the United States, through the adoption of global payments (that is, fixed reimbursements for treating patients in certain categories). That two institutions of such weight can relinquish the old, imprisoning system of fee-for-service is news indeed.

Note that the Blue Cross/Partners agreement doesn't even involve the formation of an Accountable Care Organization. Presumably, Partners believes it can pick some low-hanging fruit through modest advances in efficiency. Cost savings you can really count will come from ACOs, where total care of the patient is streamlined through better transfers of care and intensive communication. Patient-centered medical homes can do even more. So an ACO is actually much smarter than old managed care. But it depends on collecting good data and using it right.

The current deal is an important affirmation of the path Massachusetts took long before the rest of the country in aiming for universal health coverage. We all knew at the time that the Massachusetts bill was not addressing costs and that these would have to be tackled eventually. And at first, of course, health premiums went up because a huge number of new people were added to the roles, and many of them were either sick or part of high-risk populations.

The cost problem is now being addressed through administrative pressure (at one point, Governor Deval Patrick flatly denied a large increase requested by insurers), proposed laws, and sincere efforts at the private level such as the Blue Cross/Partners deal. I asked a member of the Patrick administration whether they problem could be solved without a new law, and he expressed the opinion that there's a good chance it could be. Steven Fox of Blue Cross Blue Shield said that 70% of their HMO members go to physicians in their Alternative Quality Network, which features global payments. And he said these members have better outcomes at lower costs.

ACOs have a paradoxical effect on health information exchange Jha predicted that ACOs--while greatly streamlining the exchanges between their member organizations, because these save money--will resist exchanging data with outside providers because keeping patients is even more important for ACOs than for traditional hospitals and clinics. Only by keeping a patient can the ACO reap the benefits of the investments they make in long-term patient health.

As Doris Mitchell received an award for her work with the MHDC, executive directory Ray Campbell mentioned the rapid growth and new responsibilities of her agency, the Group Insurance Commission, which negotiates all health insurance coverage for state employees, as cities and towns have been transferring their municipal employees to it. A highly contentious bill last year that allowed the municipalities to transfer their workers to the GIC was widely interpreted as a blow against unionized workers, when it was actually just a bid to save money through the familiar gambit of combining the insured into a larger pool. I covered this controversy at the time.

A low-key conference

Attendance was down at this year's conference, with about half the attendees and vendors as last year's. Lowered interest seemed to be reflected as none of the three CEOs receiving awards turned up to represent their institutions (the two institutions mentioned earlier for their historic cost-cutting deal--Blue Cross Blue Shield and Partners HealthCare--along with Steward Health Care).

The morning started with a thoughtful look at the requirements for ACOs by Frank Ingari of Essence Healthcare, who predicted a big rise in investment by health care institutions in their IT departments. Later speakers echoed this theme, saying that hospitals should invest less in state-of-the-art equipment that leads to immediately billable activities, and more in the underlying IT that will allow them to collect research data and cut waste. Some of the benefits available through this research were covered in a talk at the Open Source convention a couple years ago.

Another intriguing session covered technologies available today that could be more widely adopted to improve health care. Videos of robots always draw an enthusiastic response, but a more significant innovation ultimately may be a database McKesson is developing that lets doctors evaluate genetic tests and decide when such tests are worth the money and trouble.

The dozen vendors were joined by a non-profit, Sustainable Healthcare for Haiti. Their first project is one of the most basic health interventions one can make: providing wells for drinkable water. They have a local sponsor who can manage their relationship with the government, and an ambitious mission that includes job development, an outpatient clinic, and an acute care children's hospital.

Business-government ties complicate cyber security

2012-02-06T14:00:00Z

From time to time, we like to check in with "Inside Cyber Warfare" author Jeffrey Carr to get his thoughts on the digital security landscape. These conversations often address specific threats, but with the recent release of the second edition of Carr's book, we decided to explore some of the larger concepts shaping this space.

Are corporate and government interests in the U.S. becoming one and the same? That is, an attack on an American business' network may be regarded as an assault on the country itself?

Jeffrey Carr: Due to the dependence of the U.S. government upon private contractors, the insecurity of one impacts the security of the other. The fact is that there are an unlimited number of ways that an attacker can compromise a person, organization or government agency due to the interdependencies and connectedness that exist between both.

Are national network security and media piracy becoming interrelated and confused?

Jeffrey Carr: It has definitely become confused to the point where the Department of Homeland Security (DHS) is now the enforcement arm of the Recording Industry Association of America (RIAA), which I find utterly disgraceful. It's due entirely to the money and power that entertainment industry lobbyists have to wave in front of members of Congress. It has absolutely nothing to do with improving the security of our critical infrastructure or reducing the attack platform used by bad actors.

Flipping this around, how much of a cyber threat does the U.S. pose to other countries?

Jeffrey Carr: The U.S. is probably as capable or more capable at conducting cyber operations than any of the other nation states who engage in it. It's not a question of "they do it to us, but we don't do it to them." It's a question of how to defend your critical assets in light of the fact that everyone is doing it.

What recent technologies concern you the most?

Jeffrey Carr: We are racing to adopt cloud computing without regard to security. In fact, many customers wrongly assume that the cloud provider is responsible for their data's security when the reverse is true. Not only is security a major problem, but there's no telling where in the world your data may reside since most large cloud providers have server farms scattered around the world. That, in turn, makes the data susceptible to foreign governments that have cause to request legal access to data sitting on servers inside their borders.

Inside Cyber Warfare, 2nd Edition — Jeffrey Carr's second edition of "Inside Cyber Warfare" goes beyond the headlines of attention-grabbing DDoS attacks and takes a deep look inside recent cyber-conflicts, including the use of Stuxnet.

This interview was edited and condensed.

Related:

Four short links: 6 February 2012

2012-02-06T11:00:00Z

Jirafe -- open source e-commerce analytics for Magento platform.
iModela -- a $1000 3D milling machine. (via BoingBoing)
It's Too Late to Save The Common Web (Robert Scoble) -- paraphrased: "Four years ago, I told you all that Google and Facebook were evil. You did nothing, which is why I must now use Google and Facebook." His list of reasons that Facebook beats the Open Web gives new shallows to the phrase "vanity metrics". Yes, the open web does not go out of its way to give you an inflated sense of popularity and importance. On the other hand, the things you do put there are in your control and will stay as long as you want them to. But that's obviously not a killer feature compared to a bottle of Astroglide and an autorefreshing page showing your Klout score and the number of Google+ circles you're in.
iBooks Author EULA Clarified (MacObserver) -- important to note that it doesn't say you can't use the content you've written, only that you can't sell .ibook files through anyone but Apple. Less obnoxious than the "we own all your stuff, dude" interpretation, but still a bit crap. I wonder how anticompetitive this will be seen as. Apple's vertical integration is ripe for Justice Department investigation.

O'Reilly Radar - Insight, analysis, and research about emerging technologies.

Jury to Eolas: Nobody owns the interactive web

O'Reilly ebooks now optimized for Kindle Fire

Strata Week: Your personal automated data scientist

Wolfram|Alpha Pro: An on-call data scientist

Crisis-mapping and data protection standards

Doubting the dating industry's algorithms

Got data news?

It's time for a unified ebook format and the end of DRM

Platform lock-in

The myth of DRM

Lessons from the music industry

Now available: Best of TOC 2012 anthology

Four short links: 9 February 2012

Tip for B&N: Don't just follow Amazon

The NoSQL movement

Size, response, availability

Changing data and cheap lunches

The sacred cows

Other features

In the end

Four short links: 8 February 2012

Unstructured data is worth the effort when you've got the right tools

How did you get started in big data?

What projects are you working on now?

What are the trends and challenges you're seeing in the big data space?

The presentation you'll be giving at the Strata Conference will focus on practical applications of mining unstructured data. Why is this an important topic to address?

Are there important new tools or projects on the horizon for big data?

Who do you follow in the big data community?

Four short links: 7 February 2012

Small Massachusetts HIT conference returns to big issues in health care

The sticking point: health information exchange

Low-hanging fruit signals a new path for cost savings

A low-key conference

Business-government ties complicate cyber security

Are corporate and government interests in the U.S. becoming one and the same? That is, an attack on an American business' network may be regarded as an assault on the country itself?

Are national network security and media piracy becoming interrelated and confused?

Flipping this around, how much of a cyber threat does the U.S. pose to other countries?

What recent technologies concern you the most?

Four short links: 6 February 2012

Top stories: January 30-February 3, 2012