... and points beyond

Graph databases as big data stores for every business

Jay Jakosky — Thu, 17 May 2012 04:59:04 +0000

Here’s another angle on big data applied to the world surrounding the common business: graph databases. This is a more natural way to model the complete context of a transaction. It’s a big data problem and every business can benefit from analyzing it.

For example, each business transaction is a node. And each person that touches it, or is responsible for it. Each customer and vendor and the communication with them. And each component of the transaction, such as the product, or serial number. And the billions of relationships between all these nodes.

New IBM BPM Software Integrates Social, Cloud and Mobile

Jay Jakosky — Thu, 17 May 2012 02:31:21 +0000

Validation from IBM. Combine BPM, social communication and mobile interfaces. Ok, now just add 2-way business intelligence and some BigMetaData concepts for a revolutionary product. Related posts: Social Explorer — Interactive Census Data There Is No Desktop. Only Mobile. The … Continue reading → Related posts:

Rapid Survey Feedback

Jay Jakosky — Wed, 16 May 2012 06:15:48 +0000

Great company, great examples.

eBay makes a quick decision with bottom line results using simple surveys.

“This used to be a job for outside specialists. Now basically anyone inside Thomson Reuters can do this,” says Nicole Gagnon, a senior director of market research.

Qualtrics

QlikView Scalability In A NUMA World

Jay Jakosky — Wed, 09 May 2012 06:21:45 +0000

The scalability material for QlikView states it will scale linearly and take advantage of all cores. I think this is misleading because as I tell my customers, “QlikView scales linearly but hardware does not.” At the most recent Qonnections there was a realistic discussion of this problem in two cases.

The first is Non-Uniform Memory Access, or NUMA. This technology divides the memory among the processors, creating “local” and “remote” banks. QlikView would rather treat memory as a uniform segment. So the advice from QlikTech is to disable NUMA. It’s not always possible so there is also a QlikView configuration option to better spread the data throughout memory. What these options do is enforce average-case memory access times and avoid worst-case situations for one object versus another. If you had an app that could fit entirely into a single memory area, you’d do better to leave NUMA enabled.

I expect that future releases of QlikView will take advantage of NUMA. One of the advantages of QlikView’s batch processing is that memory storage requirements are known ahead of time and therefore optimizable. QlikView’s current data structures are compact, but still tabular. I expect to see QlikView intelligently divide its workload across processors using a hybrid of row-based and column-based storage.

The second issue is inter-processor communication. At Qonnections there was a specific concern about using an 8-way (8 processor socket) design instead of 4-way because the communication starts to take a significant percentage of peak performance. Although the cores are calculating on a subset of rows, the row details are stored across all memory areas, which means constant communciation among processors. QlikView’s current bit-stuffed indexes could be split into row or column chunks to push aggregation to be local to the processors.

There Is No Desktop. Only Mobile.

Jay Jakosky — Tue, 01 May 2012 06:02:37 +0000

“…through the last quarter, I should say, which is just 2 years after we shipped the initial iPad, we’ve sold 67 million. And to put that in some context, it took us 24 years to sell that many Macs and 5 years for that many iPods and over 3 years for that many iPhones. And we were extremely happy with the trajectory on all of those products. And so I think iPad, it’s a profound product.”

Tim Cook, Apple CEO

Great Reading

Jay Jakosky — Sat, 28 Apr 2012 07:38:35 +0000

Manufacturing: The third industrial revolution

Thingiverse

Massive Health

All The Interesting Problems Are Scalability Problems

Qonnections 2012

1 Million Lessons For Netflix About Big Data

Jay Jakosky — Tue, 17 Apr 2012 23:00:09 +0000

Netflix opted not to implement the entire algorithm that won the Netflix Prize because a) it would have cost too much to engineer, b) it no longer fit the direction of the company, and c) just two algorithms from the prize submission yielded the most significant gains.

This is such a great example of the reality of Big Data! Before embarking on a mathematical journey, companies need to assess the reward of developing models and balance against the risk of process changes, and the cost of implementation. Also, simpler models that are cheaper to implement and better understood in the general case are more likely to get business traction than deep models, highly tuned through iteration.

Netflix’s factors changed, meaning the pieces of data fed into the system. Factor selection is repeatedly identified as the most significant step in model building. With streaming, a partial viewing of a video means something: so do clicks to go back, forward, or browse for something else while you’re watching. All of this reflects your behavior.

Diagnostic Heat Maps

Jay Jakosky — Wed, 04 Apr 2012 05:52:51 +0000

Wonderful examples of diagnostic heat maps!

Change Data Capture

Jay Jakosky — Wed, 21 Mar 2012 06:03:33 +0000

The last missing enabling technology for my vision of business software was recently discussed on High Scalability as a future open source project. Soon all the pieces will exist in the open source ecosystem.

Big Metadata: The Storm of Business Data All Around Us

Jay Jakosky — Wed, 14 Mar 2012 03:19:02 +0000

Big Metadata is my phrase for the application of Big Data tools and techniques to the larger ecosystem of data surrounding business software. While Big Data is focused on unwieldy data sets at large firms or new-technology companies, Big Metadata is about every company or organization taking advantage of the massive amounts of relevant metadata that is completely ignored.

Business software only capture the bits of each transaction that will calculate changes in such things as inventory, bank accounts, and production. This reductionist view of each transaction is disconnected with the reality of daily business: full of exceptions and rewrites and human conversations. Just as consumer software is making strides in conforming to our habits and needs, business software should capture the reality of business and not enforce an ideal.

How we got here is simple. The constraints on storage, processing, software development and error checking left us focused only on critical features. We now design business databases to be as narrow as possible. We write code to run the system for today and struggle to reconcile original decisions as processes, people and the whole business landscape continues to change.

When we talk of business software being more human, more predictive, and more like the consumer software we experience from Apple and others, I think we are looking for the same kind of Big Data solutions that come from Google, Apple, Amazon, and Facebook. These are solutions based on massive amounts of data, good choice in analysis factors, and a sprinkling of algorithm and statistics.

There are clear individual business uses for social networks, voice and gesture recognition, cloud infrastructure, schema-less databases, in-memory storage, distributed algorithms, graph processing, domain-specific languages, development frameworks and other technologies. I believe the combination of these technologies to solve the common challenges in business will fuel the next great wave. My hope is to sketch out the future of business software.