We should have seen it coming. When we stopped even thinking about how we store data for our applications, when we just assumed some DBA would give us a database – and some SysAdmin would give us a file system. Sure, we can talk about W-SAN (what WLAN was to the LAN, but for storage) solutions like Amazon S3 and Rackspace Cloud – but they didn’t fundamentally change anything.

Big Data forces us to re-think storage completely. Not just structured/unstructured, relational/non-relational, ACID compliance or not. It forces us – at the application level – to rethink the current model exemplified by

I’m storing this because I may need it again in the future.

Where storage means physical, state aware object persistence and future means anywhere between now and the end of time.

Data Persistance – A Systemic Approach to Big Data for Applications

What Big Data applications require is a systemic approach to data. Instead of applications approaching data as only a set of if/then operations designed to determine what (if any) CRUD operations to perform it demands that applications (or supporting Data Persistence Layers) understand the nature of the data persistence required.

This is a level of complexity developers have been trained to ignore. The CRUD model itself explicitly excludes any dimensionally – or meta information about the persistence. It is all or nothing.

Data Persistance is primarily the idea that data isn’t just stored – it is stored for a specific purpose which is relevant within a specific time slice. These time slices are entirely analogous to those discussed in Preemption. Essentially, any sufficiently large real time Big Data system is simply a loosely aggregated computer system in which any data object may generate multiple tasks each of which have a specific priority.

For example, in a geo location game (Foursquare) the appearance of a new checkin requires multiple tasks which are prioritized based on their purpose, for example:

1. Store the checkin to distribute to “friends” (real-time)
2. Store the checking association with the venue (real-time)
3. Analyze nearby “friends” (real-time)
4. Determine any game mechanics, badges, awards, etc
5. Store the checkin on the user’s activity
6. Store the checkin object

NOTE: Many developers will look at this list above and ask: “Why not a database?” While a traditional database may suffice for a relatively low volume system (5k users, 20k checkins per day) it would not be sufficient at Big Data scale (as discussed here).

This Data Persistence solution is comprised of four vertical persistence types:

Transitory

Transitory persistance is for data persisted only long enough to perform some specific unit of work. Once the unit of work is completed the data is no longer required and can be expunged. For example: Notifying my friends (that want to be notified) that I’m at home.

Generally speaking (and this can vary widely by use case) Transitory persistence must be atomic, extremely fast and fault tolerant.

Volatile

Volatile persistance is for data that is useful but can be lost and rebuilt at any time. Object caching (how memcached is predominantly used) is one type Volatile persistence, but does not describe the entire domain. Other examples of volatile data include process orchestration data, data used to calculate decay for API Rate Limits, data arrival patterns (x/second over the last 30 seconds), etc.

The most important factor for Volatile data persistence is that the data can be rebuilt from normal operations or from long term data storage if it is not found in the Volatile dataset.

Generally speaking, data is stored in Volatile persistence because is offers superior performance, but limited dataset size.

ACID

Relational databases and atomicity, consistency, isolation and durability (ACID) are not obsolete. It is important for specific types of operations – done for specific purposes to maintain transactional compliance and ensure the entire transaction either succeeds in an atomic way, or fails. Examples of this include eCommerce transactions, Account Signup, Billing Information updates, etc.

Generally speaking, this data complies with the old rules of data. It is created/updated slowly over any given time slice, it is read periodically, there is little need to publish the information across a large group of subscribers, etc.

Amorphous

Amorphous persistence is the new kid on the block. NoSQL solutions fit nicely here. This non-volatile storage is amorphous in that the content (think property, not property value) of any object can change at any time. There is no schema, table structure or enforced relationship model. I think of this data persistence model as raw object storage, derived object storage and the transformed data that forms the basis of what Jeff Jones refers to as Counting Systems. Additionally, these systems store data in application consumable objects – with those objects being created on the way in.

Systems in this layer are generally highly scalable, fault tolerant, distributed systems with enhanced write efficiency. They offer the ability to perform the high volume writes required in real time Big Data systems without significant loss of read performance.

What Does All This Mean?

Most notably it means, that after years of obfuscating the underlying data storage from developers, we now need to re-engage application developers in the data storage conversation. No longer can a DBA define the most elegant data model based on the “I’m storing this because I may need it again in the future.” model and expect it to function in the context of a real time Big Data application.

We will hear a chorus of voices who will attack these dis-aggregated data persistence models based on complexity or the CAP Theorem or on the standard “the old way is the best way” defense of ACID and the RDBMS for everything. But all of this strikes me as a perfect illustration of what Henry Ford said:

If I had asked customers what they wanted, they would have told me they wanted a faster horse

I saw this on TechCrunch earlier today and thought it was an awesome add to my big data series. Jeff Jonas is clearly a big thinker and I agree with almost everything he says. The only thing I take issue with is the recurring theme in this interview that Big Data is primarily about commerce – and specifically ad targeting.

In my next post I’ll be talking about Big Data for One – which I think Jeff hints at but never fully develops.

Part 1: What is Data?

Part 2: Why data makes us more ignorant.

Data is actually evidence that you already knew, but failed to act on it. Amnesia.

Part 3: Why Big Data is the next big thing.

From pixels to pictures… This agrees with my idea of Big Data – it isn’t about the size of the dataset, but about using pieces of data in context by understanding context.

Why he goes to ad based is beyond me however…

Part 4: How data makes us average.

Very similar to the points I made here: Living In Public – Facebook, Privacy and Frictionless Distribution

Why the future is irresistible.

Both hiring and job seeking are about people. You should hire based on the person. The intangibles of who they are, how they interact, the strength of the cultural fit are more important than the checklist of skills from the job posting. As a job seeker your number one concern should be:

Do I want to work with these people, in this environment, every day?

Your CV doesn’t tell the prospective employer any of that. It certainly doesn’t help the job seeker learn anything about the people and culture of the company. So what is the point?

The question isn’t do you need the skills I have, it is do you need me…

I don’t get it… I really don’t. When (and why) did silicon valley abandon the idea that technology – particularly the web based variety – had intrinsic value. The kind of transformative value that people would pay for…

Find me two business plans for an online consumer product that don’t have “and monazite via ad sales” somewhere in the business plan – go ahead, I dare you.

On the backs of Google and their transformation of advertising based on keywords and relevance we’ve become lazy. Why deliver real solutions to the problems of consumers when you can just aggregate eyeballs and information? Make something “cool”, something that will “go viral” and “generate buzzzz”. Get people to LOOK, not buy.

Think about it for a second, how different is Facebook from Yahoo circa 1998? Yahoo had tons of users. Yahoo knew a lot about their users based on what they made part of their “portal”. Yahoo sold ads.

I’m not talking about how advertising is evil, I’m not demonizing marketers. I’m asking why we’d have conversations like these: SEO vs. Usability

I’m also not suggesting that if you simply build a better mousetrap the world will beat a path to your door… I’m just suggesting that we should get back to trying to build better mousetraps – and by extension trying to change the world.

Following up on yesterday’s Big Data post.

Here is the same data from the NLCS final last night. I’ll let you draw conclusions regarding the delta from the ALCS numbers.

One aspect of Big Data is the arrival pattern of the data being stored/analyzed.

Yesterdays Rangers win in the ALCS is a excellent example of this. At justSignal we monitor all of the MLB teams and when the Rangers won we experienced a phenomenal spike in data arrival. Right now – which is fairly representative of normal volumes – we are receiving 7.87 Tweets per Second from Twitter. Last night we peaked out at 43.5 Tweets per Second.

Here is what that looks like graphically:

Managing that kind of data arrival while continuing to generate meta data dimensions (transformations) and deliver data to clients in real time is no small task.

I’ve been thinking about writing a series of posts about Big Data for months now… which is entirely too much thinking an not enough doing, so here we go.

And, by Big Data we mean…

Wikipedia offers a very computer science oriented explanation of Big Data – but while the size of the dataset is a factor in Big Data there are several others worth considering:

• The arrival pattern (how data is created in the large dataset)
• Dimensions (how the data is grouped, sub-grouped and organized for analysis)
• Transformation (how the data is re-dimensioned or transformed to meet new analysis needs)
• Consumption (how will the data be consumed by client applications and ultimately humans)

Classically Big Data was regarded is extremely large datasets, usually acquired over a long period of time, involving historical data leveraged for predictive analysis of future events. In the simplest terms, we use hundreds of years of historical weather data to predict future weather. Big Data isn’t new, my first exposure to these concepts was in the early 1990′s dealing with Contact Center telecom and CTI (computer telephony integration) analytics and predictive analysis of future staffing needs in large footprint (3 to 5 thousand agent) contact centers.

Another example of the classic Big Data problem is the traditional large operational dataset. In the 90′s a 2 Terabyte dataset – like the one I worked with at Bank of America – was so massive that it created the need for a special computing solution. Today large operational datasets are measured in Petrabytes and are becoming fairly common. The primary challenges with these types of datasets is maintaining acceptable operational performance for the standard CRUD operations.

There are Big Data storage models emerging and maturing today with the two default options being hadoop – which relies on a distributed file system and distributed key value store systems such as Cassandra and CouchDB. These systems (referred to as NoSQL solutions) differ from standard RDBMS systems in two important ways:

• The underlying organization of data is amorphous and does not implement relationships.
• There is no support for Transactional Integrity (also know as ACID).

While all of these are interesting engineering problems, they still lack a crucial component. As a matter of fact, most conversations about Big Data fail to adequately address what is, perhaps, the most important problem with Big Data systems today.

The Intersection of Real Time and Big Data

Today’s big datasets are manageable in RDMBS systems. That being said, a significant amount of complexity is inserted in the managament process, most notably:

• Database Sharding and the complexity of managing multiple databases.
• Downtime associated with Schema changes or complexity of CLOB fields used in a amorphous manner.

Given that, large datasets that change slowly over time, or more accurately, those that have a relatively low volume of creates (including those that occur as a result of large transformations) as compared to read, update and delete can be managed using RDBM systems.

Where Real Time – specifically as related to Social Media, user generated content and other high create applications (such as the Large Hadron Collider) – intersect with Big Data is – to me – the most interesting Big Data topic.

This model presents three distinct challenges:

• High data arrival – potentially hundreds of thousands of new objects “found” per second.
• High transformation rate.
• Real Time Client updates.

The intersection of all three of these challenges was exactly the what we dealt with at justSignal. We needed to consistently collect hundreds of thousands of Social Media objects per second, generate hundreds of meta data elements (transformations) for each object, and make all of that data available in real time to our client applications. This view of Big Data is slightly different. In this view the size of the dataset – while still significant – isn’t as important as the challenges presented by a very high volume of CRUD operations over very short time slices.

The most important thing I’ve learned is that there is no silver bullet. While the traditional relational database isn’t effective in real time big data scenarios neither is a standalone Hadoop or Distributed Key Value Store. Essentially, you must evaluate each use case against the suite of technologies available and select the one best suited to that particular use case. Selecting a NoSQL solution for order processing – which has heavy ACID/Transactional requirements isn’t a good idea. However, staying with your RDBMS for high insert/transform data processing isn’t going to work either.

The approach we took at justSignal – which I will go into in more detail in a future post – was to create a unified data persistence layer designed to leverage the right long/short term data store (and sometimes more than one) based on the requirements of the application. This data persistance layer is made up of:

• MySQL
• memcached
• Cassandra
• Beanstalkd

Each plays a critical role in our ability to collect, transform, process and serve hundreds of thousands of social media mentions per minute.

More to follow…

Seth Godin – without meaning to – has pointed at the root cause of the education slide in the US.

Many people in the United States purchase one or fewer books every year.

Many of those people have seen every single episode of American Idol. There is clearly a correlation here.

Access to knowledge, for the first time in history, is largely unimpeded for the middle class. Without effort or expense, it’s possible to become informed if you choose. For less than your cable TV bill, you can buy and read an important book every week. Share the buying with six friends and it costs far less than coffee.

Or you can watch TV.

Blaming teachers is lazy. Blaming politicians is just stupid.

Until we value intelligence as a society you can’t expect children actively seek it. When intelligent well educated people of all stripes (but most notably politicians) are branded elitist while those for whom complete sentences seems to be a significant struggle are lauded as “folksy” or “accessible” you see the heart of the problem.

When our popular culture values thuggish-ness or vapid preening over substance you see the heart of the problem.

Fixing education starts with how you regard the educated. It begins with what you demand in your entertainment, books, newspapers, politics and TV news. It begins when we seek knowledge and the knowledgable instead of the common and comfortable.

I’m a big fan of the internet. It is the single most disruptive technology in the history of mankind because it has the ability to disrupt so much of the old ways we did things.

I’m a huge fan of Social Media, because it fully leverages the internet of things to enable human connections without regard for location.

I’m a huge fan of Google, because the idea that we can index and make “findable” the entire repository of human knowledge is transformative.

What I’m not a fan of is the “Ad-Unit”. I’m not a fan of the ad supported business. I’m not a fan of the idea that the only way to monetize the internet of things is by plastering it with adverts.

I know this is impolitic, but is this really the best we can do? Is the internt nothing more than an advertising platform?

I’m not offended that Facebook apps are selling my data to advertisers – I’m offended that we’ve reached a point where it is taken for granted that those tactics are effective. I’m offended that Michael Arrington would suggest on TechCrunch that the real point is:

If you do stuff online, people are tracking it and putting it into a database and trying to sell you stuff based on that. There’s not much you can do about it except not be online. And it’s not all that bad, really, to get ads for diapers when you’re having a baby, or ads for cars when you are looking to buy a car. Life will go on.

As if I’m too stupid to find diapers or a damn car without the “help” of a bunch of adverts.

But this is how things work now. You build a service people want to use, and then you help marketers beat your (stupid) users about the head neck and shoulders with adverts for adult diapers, erection pills and soap that is guaranteed to get you laid. You aren’t users, they aren’t solving your problem. You are a demographic with disposable income, a short attention span and a willingness to click.

The internet of things is becoming the sole domain of two things:

• Content designed to attract as much attention as possible (without regard for merit).
• Services and tools to plaster adverts all over above content.

Two years ago we all woke up and realized that the best minds of our generation were being used to create highly lucrative but utterly meaningless financial schemes. These Quants weren’t adding value to our economy, they were merely finding creative ways to extract as much value as possible from our economy.

I have to wonder if the new Quants aren’t the internet of things wunderkinds who – in exchange for our attention – are extracting as much value from the internet as quickly as they can. Adding little while creating fortunes.

I find this completely unacceptable. My internet of things is about diversity, frictionless distribution of knowledge, and the ability to negate the tyranny of geography. If you are interested in that internet of things – and creating real value without generating a dime in advertising revenue… let me know, those are projects, non-profits and startups I’d love to be involved in.

From an interview with Kyung Hee Kim

Decrease in Elaboration after 1984: Elaboration scores (ability to develop and elaborate upon ideas and detailed and reflective thinking and motivation to be creative) decreased more than other subscales of the TTCT. Elaboration scores decreased by 19.41% from 1984 to 1990, by 24.62% from 1984 to 1998, and by 36.80% from 1984 to 2008.

The interview is dense and contains a lot of information and statistics – but what jumped out at me was the section above. Elaboration is a critical skill in making things that solve people’s problems. Without elaboration entrepreneurship suffers.