• Graph data model basics
• Relationship analytics definition
• Relationship analytics applications
• Analysis of large graphs (this post)

My series on graph data management and analytics got knocked off-stride by our website difficulties. Still, I want to return to one interesting set of issues — analyzing large graphs, specifically ones that don’t fit comfortably into RAM on a single server. By no means do I have the subject figured out. But here are a few notes on the matter.

How big can a graph be? That of course depends on:

• The number of nodes. If the nodes of a graph are people, there’s an obvious upper bound on the node count. Even if you include their houses, cars, and so on, you’re probably capped in the range of 10 billion.
• The number of edges. (Even more important than the number of nodes.) If every phone call, email, or text message in the world is an edge, that’s a lot of edges.
• The typical size of a (node, edge, node) triple. I don’t know why you’d have to go much over 100 bytes post-compression*, but maybe I’m overlooking something.

*Even if your graph has 10 billion nodes, those can be tokenized in 34 bits, so the main concern is edges. Edges can include weights, timestamps, and so on, but how many specifics do you really need? At some point you can surely rely on a pointer to full detail stored elsewhere.

The biggest graph-size estimates I’ve gotten are from my clients at Yarcdata, a division of Cray. (“Yarc” is “Cray” spelled backwards.) To my surprise, they suggested that graphs about people could have 1000s of edges per node, whether in:

• An intelligence scenario, perhaps with billions of nodes and hence trillions of edges.
• A telecom user-analysis case, with perhaps 100 million nodes and hence 100s of billions of edges.

Yarcdata further suggested that bioinformatics use cases could have node counts higher yet, characterizing Bio2RDF as one of the “smaller” ones at 22 billion nodes. In these cases, the nodes/edge average seems lower than in people-analysis graphs, but we’re still talking about 100s of billions of edges.

Recalling that relationship analytics boils down to finding paths and subgraphs, the naive relational approach to such tasks would be:

• Store a table with one row per edge.
• Do an (n-1)-way join, where n is the number of edges in the path or subgraph.

In many cases the cardinality of intermediate result sets would be high, and you’d basically be doing a series of full table scans. Those could take a while.

There are various approaches to dealing with this challenge. For example:

• Graph analysis has been around long enough that much of it has surely been done relationally.
• I wrote about some specific relational strategies for graph analysis five years ago.
• A lot of graph analysis these days is being done in Hadoop (or other MapReduce, notably Aster Data’s).
• Objectivity Infinite Graph  and Google Pregel emphasize pre-fetching (or pre-shipping) edges that might soon be needed.
• Yarcdata, with its Cray genes, tries to optimize hardware (single RAM image across a cluster, with a whole lot of multithreading) for in-memory Apache Jena performance. Unfortunately, I’m not clear as to which data structure(s) Jena uses.

When trying to figure out which of these techniques is likely to win in the most demanding cases, I run into the key controversy around analytic graph data management — how successfully can graphs be partitioned? Opinions vary widely, with the correct answers in each case surely depending on:

• The topology of the graph.
• The size of the graph.
• The length of the paths that need to be examined.

But in the interest of getting this posted tonight, I’ll leave further discussion of graph partitioning to another time.

If you notice any counterexamples, please be so kind as to ping me.

A comment is, of course, an update. So we’re closing off comments across DBMS 2, Strategic Messaging, Text Technologies, Software Memories, and the Monash Report. I hope to turn them back on shortly.

The sites should remain readable all the way through — unless, of course, there are more hosting company outages.

When I pull the trigger on the move, there will be a short period when I turn off comments across all my blogs. I’ll post again here to announce when that is happening.

I apologize for the inconvenience.

• Graph data model basics
• Relationship analytics definition
• Relationship analytics applications (this post)
• Analysis of large graphs

In my recent post on graph data models, I cited various application categories for relationship analytics. For most applications, it’s hard to get a lot of details. Reasons include:

• In adversarial domains such as national security, anti-fraud, or search engine ranking, it’s natural to keep algorithms secret.
• The big exception – influencer analytics, aka social network analysis — is obscured by a major hype/reality gap (so, come to think of it, is a lot of other predictive modeling).

Even so, it’s fairly safe to say:

• Much of relationship analytics is about subgraph pattern matching.
• Much of relationship analytics is about identifying subgraph patterns that are predictive of certain characteristics or outcomes.
• An important kind of relationship analytics challenge is to identify influential individuals.

Notes on that middle point include:

• Pattern identification could be done through trial-and-error visualization, through predictive modeling, or through any form of investigative analytics in between.
• I presume what’s hardest about all this from a processing-performance standpoint would often be enumerating the subgraphs of a particular candidate pattern.

So I’m tempted to say “it’s all about subgraphs.” But it might be more accurate yet to say “It’s about paths”. Arguably, that’s saying the same thing; paths are subgraphs, and subgraphs are made up of paths, so a way of finding one is also a way of finding the other. But referring to paths nods to such standard tasks as:

• Finding the shortest path between two nodes.
• Calculating centrality metrics.

Paths are also simpler than subgraphs, and hence also simpler to think about.

Let’s drill down a bit more on the cases of influencer analysis and centrality. Telecom service providers around the world compete with relatively few of their peers (because they’re so geographically bound), and hence are pretty good about sharing technical ideas with each other. One application that has spread like wildfire is influencer analysis for churn control. The idea is to identify influential subscribers who, if they left your service, would be particularly likely to take other people with them, so that you can make great efforts to retain them. The key data used is CDRs (call detail records).

As in many things, it’s tough to separate influencer analysis adoption fact from fiction.

• The telecom case is surely real; I’ve heard of many examples.
• Social networking is a harder call. Top-down, the story sounds good; but bottom-up, I’m not so sure.*
• I’m quite dubious about attempts to use influencer analysis based on, say, credit card records; the detailed information about person-to-person connections isn’t there.
• National security clearly uses similar kinds of techniques, albeit for slightly different purposes.

Specific conclusions I’ve heard include:

• Who calls you is a better predictor of whether you influence cellular subscribers to churn along with you than who you call.
• Length of calls is an indicator of involvement influence in terrorist networks (short ones suggest there’s serious business being done).

*For example my Klout profile asserts I’m more influential about Airlines than about Databases or Software. A bit of manual intervention could surely change that — which just serves to underscore my doubts about the effectiveness of social network analytic automation.

One more thing — relationship analytics on social networks rarely works unless you take out a few spurious highly-connected nodes. The paradigmatic example is the local pizza parlor, which receives many phone calls, but is neither a terrorist mastermind nor a major influence  upon telecom service churn. More on that point when I write about the partitioning of large graphs.

• Graph data model basics
• Relationship analytics definition (this post)
  
• Relationship analytics applications
• Analysis of large graphs

In late 2005, I encountered a company called Cogito that was using a graphical data manager to analyze relationships. They called this “relational analytics”, which I thought was a terrible name for something that they were trying to claim should NOT be done in a relational DBMS. On the spot, I coined relationship analytics as an alternative. A business relationship ensued, which included a short white paper. Cogito didn’t do so well, however, and for a while the term “relationship analytics” faltered too. But recently it’s made a bit of a comeback, having been adopted by Objectivity, Qlik Tech, Yarcdata and others.

“Relationship analytics” is not a perfect name, both because it’s longish and because it might over-connote a social-network focus. But then, no other term would be perfect either. So we might as well stick with it.

In that case, “relationship analytics” could use an actual definition, preferably one a little heftier than just:

Analytics on graphs.

At the risk of sounding circular, I’ll try:

Relationship analytics is analytics that focuses upon relationships encoded in data.

Notes on that proposed definition include:

• The more directly the relationships are encoded — for example by a node-edge-node graph data model — the more applicable the term is likely to be.
• It can still be relationship analytics if the nodes of the graph are ultimately more important than the edges. The edges just have to be central — no pun intended — to the analytics.
• “Analytics” is a vague term, and “relationship analytics” inherits the vagueness. That said, I think of relationship analytics as being more about investigative analytics than the operational kind.

So what do you think — does this definition of “relationship analytics” work?

• Graph data model basics (this post)
• Relationship analytics definition
• Relationship analytics applications
• Analysis of large graphs

Interest in graph data models keeps increasing. But it’s tough to discuss them with any generality, because “graph data model” encompasses so many different things. Indeed, just as all data structures can be mapped to relational ones, it is also the case that all data structures can be mapped to graphs.

Formally, a graph is a collection of (node, edge, node) triples. In the simplest case, the edge has no properties other than existence or maybe direction, and the triple can be reduced to a (node, node) pair, unordered or ordered as the case may be. It is common, however, for edges to encapsulate additional properties, the canonical examples of which are:

• Weight. Usually, the intuition here is that the weight is a number indicating the strength of the connection. This is generally derived from more basic data.
• Kind. The edge can encapsulate one or more descriptors indicating the kind of relationship between the nodes.

Many of the graph examples I can think of fit into four groups:

• Networks of people, aka social networks.Three (overlapping) areas of particular importance are:
  • People/communications.
    • One canonical example is influencer-finding in telecommunications customer bases. The nodes are subscribers; the edges are call details (raw or aggregated).
    • Other examples may be found as subgraphs of our next category, namely …
  • … people/places/things.
    • This is the classic structure for anti-terrorism, law enforcement, or anti-fraud use cases.
    • Nodes are people, buildings/addresses, cars, businesses, etc. , except that …
    • … nodes can actually be ordered pairs (tangible thing, timestamp). After all, it’s more interesting if two people were, not just in the same place, but in the same place at the same time.
  • People/connections/recommendations.
    • Similarly, there are use cases in which various people have social network connections, and then also recommend products of some kind.
    • Edges can carry information about the evident strength of the social network connection …
    • … but also about apparent similarities in taste.
• Graphs of IT objects.Various sets of conceptual IT objects can be viewed as graphs. For example:
  • I visited Workday recently. They refer to their Java object model as a “graph.”
  • Neo Technology (the neo4j guys) started out doing a content management system, and eventually decided that what they really wanted underneath it was a graph-oriented DBMS.
  • Now one of Neo’s major application areas is MDM (Master Data Management).
  • Most dramatically, there’s Tim Berners-Lee’s “Semantic Web”, which is built on RDF, which models things as “a directed, labeled graph”. SPARQL, OWL and so on are in the mix as well. To date, the Semantic Web has been a lot of hot air, only without the hot aspect; still, it’s obviously influenced many people’s thinking about graphs.
  • Edit: Please see Marie’s comment below for a rather major example I left out.
• Taxonomies, ontologies, and/or semantic networks.
  • To a large extent this overlaps with my previous category …
  • … but I’m particularly fond of the example of straightforward taxonomies of words, e.g. WordNet. The nodes are the words themselves, or more precisely word senses (i.e., specific meanings of a word); edges are typically chosen from a limited set of alternatives such as is_a, is_part_of, or entails.
• Finally, there are representations of physical graphs. Examples might include telecom networks, utility grids, or locations and routes for physical deliveries.

My main reason for reciting these diverse examples is to illustrate that, for any really interesting technical discussion, it is necessary to focus on a subset of the possible use cases.

This post is intended to start a short series. When the next one goes up — focusing on a particular set of use cases — this footer will be edited accordingly.

• My comments, lightly edited, are in plain text below.
• Further thoughts are in italics.
• Of course I also linked him to my post “Big Data” has jumped the shark.
• Overall, my responses boil down to “Of course there’s some hype.”

1. A great example of hype is that anybody is calling Birst a “Big Data” or “Big Data analytics” company. If anything, Birst is a “little data” analytics company that claims, as a differentiating feature, that it can handle ordinary-sized data sets as well.

When I checked Birst’s website, “Big Data” was nowhere to be found. On the other hand, the term was all over its press pitch for the financing.

2. The great growth in database sizes is both caused and balanced out by Moore’s Law. The net effect is healthy but not enormous growth in the overall data management and analytics markets.

I’ve made versions of that point many times before.

3. Incumbent data and analytic technology vendors such as Oracle, IBM, and Microsoft are vulnerable, but are competing very hard. Favorable exits have ensued for companies such Netezza, DATAllegro, Vertica, and Aster Data.

The connection between those two points is that the big companies will hold a lot of share, but part of how they’ll hold it is through acquisitions. For example, IBM, Microsoft, HP, Teradata, and Greenplum all bought newish analytic RDBMS vendors, at an aggregate cost of several billion dollars. And SAP bought Sybase.

But while there have been billions of dollars in fairly recent analytics-related acquisitions, the pace of acquisition would have to accelerate much further yet to justify current valuations.

Upon reflection, I may have overestimated the acquisition/IPO total-value-created ratio somewhat. Even so, what’s the last enterprise technology vendor to create huge investor value by going public, continuing to prosper, and so on? Red Hat and Autonomy may be as good as it gets. VMware isn’t really an example, because of its ownership structure.

4. I’m worried that people may be overestimating the business benefit of accurate analytics, great thought that value truly is. For example, it’s not plausible that all enterprises in the world use better analytics to all improve their respective market shares.

Yes, it’s great to be an arms dealer to all sides. But “Big Data” technology is just another chapter in the ever-growing importance of IT.

• Advising vendors what/how to sell.
• Advising users what/how to buy.

One complication that commonly creeps in is that different groups of users have different buying practices and technology needs. Usually, I nod to that point in passing, perhaps by listing different application areas for a company or product. But now let’s address it head on. Whether or not you care about the particulars, I hope the sheer length of this post reminds you that there are many different market segments out there.

Last June I wrote:

In almost any IT decision, there are a number of environmental constraints that need to be acknowledged. Organizations may have standard vendors, favored vendors, or simply vendors who give them particularly deep discounts. Legacy systems are in place, application and system alike, and may or may not be open to replacement. Enterprises may have on-premise or off-premise preferences; SaaS (Software as a Service) vendors probably have multitenancy concerns. Your organization can determine which aspects of your system you’d ideally like to see be tightly integrated with each other, and which you’d prefer to keep only loosely coupled. You may have biases for or against open-source software. You may be pro- or anti-appliance. Some applications have a substantial need for elastic scaling. And some kinds of issues cut across multiple areas, such as budget, timeframe, security, or trained personnel.

I’d further say that it matters whether the buyer:

• Is a large central IT organization.
• Is the well-staffed IT organization of a particular business department.
• Is a small, frazzled IT organization.
• Has strong engineering or technical skills, but less in the way of IT specialists.
• Is trying to skate by without much technical knowledge of any kind.

Now let’s map those considerations (and others) to some specific market segments.

• Traditional large enterprises’ central IT organizations commonly:
  • Favor large, proven vendors and well-accepted IT methodologies.
  • Would like to consolidate their IT vendors as much as possible.
  • Have major challenges with legacy systems and data integration …
  • … which are often exacerbated by mergers.
  • Spend a lot of cycles on bureaucracy and company politics.
  • Notwithstanding the forgoing, have resources to invest in some “sizzle” initiatives.
• The very largest enterprises are more likely than their slightly smaller counterparts to:
  • View IT as a potential area of competitive differentiation.
  • Believe much of what they do should be custom, due to their unique needs and resources.
  • Experiment with unproven technologies.
• Smaller enterprises may:
  • Have small, generalist, overwhelmed staffs.
  • Hope for turnkey application solutions (SaaS or otherwise).
  • Get very committed to/reliant on a small number of vendors.
• In particular, IBM or Microsoft loyalists can be:
  • Extremely locked into their preferred vendor’s strategies.
  • Not very fruitful for rival vendors to attempt to sell to.
• Humongous consumer internet companies tend to:
  • Have very high opinions of themselves and their technical abilities.
  • Be open source zealots, for reasons both of free-like-beer and free-like-speech.
  • In particular, not want to buy anybody else’s software.
  • Not be big fans of relational database designs.
• Other large consumer internet companies tend to:
  • Be like the humongous ones they look up to, but maybe not to the same extremes.
  • In particular, be more willing to pay for software.
  • Be mired in company politics only/mainly to the extent they are both large and old(er).
• Smaller consumer internet companies tend to:
  • Be like the large ones they look up to, but …
  • … be quite short on traditional IT skills, and work around that shortage by reinventing various wheels.
• Business-oriented SaaS (Software as a Service) companies commonly:
  • Are drawn to the cool open source technologies consumer internet companies use …
  • … but may wind up using more traditional kinds of DBMS, for the same reasons those DBMS are used in other business applications.
  • Are more primitive in the analytic capabilities they offer their customers than I think they should be (analytics-only vendors sometimes excepted).
  • Are refreshingly free of traditional IT politics, because technology is too important to them to mess around with too badly. (Of course, any other kinds of company politics may still come into play.)
• Internet operations of traditional enterprises:
  • Sometimes are just like stand-alone internet businesses.
  • Sometimes are just like — and part of — the rest of the enterprise’s IT operations.
  • More commonly are somewhere in between.
• Marketing departments of traditional enterprises sometimes:
  • Want to do their own data acquisition, management, and/or analysis …
  • … without having great IT resources of their own.
  • Invest in departmental analytics efforts or even …
  • … have line executives who are analytically proficient.
  • Make heavy use of SaaS, as an alternative to relying on central IT, or as a natural byproduct of acquiring third-party data.
• Large investment firms commonly:
  • Have numerous departments, each with its own IT experts.
  • Care about sub-millisecond latency …
  • … and sub-week time-to-value.
  • Experience return-on-investment in a very different way than most businesses do.
• Telecom service companies commonly differ from other similarly-sized enterprises in that:
  • They are more aggressive about using innovative technology to manage (and analyze) data.
  • Somewhat resemble investment firms in having multiple departments that each have broad engineering discretion.
• National security customers often:
  • Want the best, cutting-edge, sometimes custom technology, yet …
  • … make themselves very cumbersome to sell to and support.
  • Are not forthcoming about how they use what they buy.

I could keep going for quite a while — but for now I won’t. Vertical markets I’m thus omitting include but are not limited to:

• Pharmaceutical researchers
• Hospitals
• Insurers
• Academic scientists

Finally, for yet another omission — in my original outline I contemplated distinguishing among various geographical areas, with my first-pass segmentation being:

• North America
• Europe
• Japan
• China
• Smaller geographies

• Cloudera now has 220 employees.
• Cloudera now has over 100 subscription customers.
• Over the past year, Cloudera has more than doubled in size by every reasonable metric.
• Over half of Cloudera’s customers use HBase, vs. a figure of 18+ last July.
• Omer Trajman — who by the way has made a long-overdue official move into technical marketing — can no longer keep count of how many petabyte-scale Hadoop clusters Cloudera supports.
• Cloudera gets the majority of its revenue from subscriptions. However, professional services and training continue to be big businesses too.
• Cloudera has trained over 12,000 people.
• Hortonworks is training people too.
• Hortonworks now has 70 employees, and plans to have 100 or so by the end of this quarter.
• A number of those Hortonworks employees are executives who come from seriously profit-oriented backgrounds. Hortonworks clearly has capitalist intentions.
• Hortonworks thinks a typical enterprise Hadoop cluster has 20-50 nodes, with 50-100 already being on the large side.
• There are huge amounts of Elastic MapReduce/Hadoop processing in the Amazon cloud. Some estimates say it’s the majority of all Amazon Web Services processing.
• I met with 4 young-company clients who I regard as building vertical analytic stacks (WibiData, MarketShare, MetaMarkets, and ClearStory). All 4 are heavily dependent on Hadoop. (The same isn’t as true of older companies who built out a lot of technology before Hadoop was invented.)
• There should be more HBase information at HBaseCon on May 22.
• If MapR still has momentum, nobody I talked with has noticed.