DataStax » Blog Post – Corporate

DataStax Enterprise 1.0.2 Service Pack Now Available

Robin Schumacher — Fri, 17 Feb 2012 13:38:33 +0000

We’re please to let you know that DataStax Enterprise service pack 1.0.2 is now available for download. Please see the release notes for the changes included in the service pack and the online documentation for upgrade instructions.

Why people care about big data and multi-datacenter replication

Billy Bosworth — Sun, 05 Feb 2012 03:00:17 +0000

This is my final post in a series (here, here, and here) breaking down the following paragraph from our recent press release that touches on some of the key reasons why people choose Cassandra. It reads:

Customers this year chose Cassandra time and time again over competing solutions. The peer-to-peer design allows for high performance with linear scalability and no single points of failure, even across multiple data centers. Combine this with native optimization for the cloud and an extremely robust data model and Cassandra clearly stands apart from the competition for enterprise, mission-critical systems. [emphasis added]

For many, the idea of spanning multiple datacenters with a single database conjures images of late nights, amazing complexity, and delicate “bubble gum and shoestring” solutions that once created, you would never even think about touching for fear of watching it all crumble. The architectures were so challenging that the cost and complexity simply outweighed the benefits. But in today’s world, it’s becoming more than a benefit — it’s now a requirement. Let me share a few examples:

Disaster avoidance. Some companies need to plan for a worst-case scenario where they lose contact with an entire datacenter (or “region” in Amazon’s cloud). During the outage, their application running on the database must not fail.
Performance. Taking the processing to the locality where the application is interacting with its users.
Scale. One customer of ours is running an application simultaneously collecting massive amounts of device data from its infrastructure in more than ten (10) datacenters across the globe.
Security. Certain industries have regulations that require local copies of the data, but nobody wants to wait on things like log shipping and batch loads.

The world has shrunk for businesses and IT is often in a race to catch up. The fact is that now big data comes from a wide variety of geographic locations. It only makes sense that developers and operations teams are going to require that their underlying database keep pace. But because of all the “scar tissue” formed from many years of trying to handle even keeping just two (2) datacenters in sync (let alone multiple) people have a healthy skepticism as to whether or not it can really be done. Not only “can” it be done, it “must” be done as that is going to be a common requirement in this big data world. I think about it a little like wireless. When wireless networking became a reality, many were ultra skeptical. Remember how hard everyone fought hard to get their computers and phones in their homes and offices in just the right places with just the right wiring? Wireless, at first, seemed far too complicated and a “pie in the sky” vision. But once we started using it, there was no going back. The benefits were too great, and we all started demanding it. When I talk to customers whose businesses now live and love multi-datacenter replication, I can’t help but think that in a few years, we’ll all be looking back and asking: “How did I ever build applications on just one, or even two, datacenters?

Big Statements about Big Data

Robin Schumacher — Wed, 01 Feb 2012 15:13:38 +0000

I was surprised at how many news stories at the end of 2011 contained predictions that big data would be ‘big’ in 2012, and thought that the chorus of voices talking about big data would lessen a bit in the early part of this year. How wrong I was. Instead, the momentum and grandeur of the statements being made has only gotten stronger. Witness this story in the Wall Street Journal that calls out big data as being one of the three technologies that will help pull the present economy out of recession just as electrification, telephony, and the automobile did for the recession of 1912. That’s bold. ZDNet columnist Dion Hinchcliffe was ahead of the WSJ and put big data as the foundation of what he believes are the five major shifts in 21^st century information technology. As a reminder, big data doesn’t necessarily only mean dealing with petabytes of data, although that can certainly be part of the equation. Big data, as defined by Gartner Group, encompasses:

Velocity – the speed at which data is coming in. That data may only sum up to be GB’s and not TB’s/PB’s in the end, but the pace at which the data enters the database (be it in burst format or continuous) is extremely rapid in nature
Variety – this includes structured, semi-structured, and unstructured data, perhaps all in the same database
Volume – indeed big data can mean “big” data. The desire of many companies to have large volumes of corporate data at the ready for either real-time or batch analysis has never been higher
Complexity – big data can bring a lot of complication in the forms of heavy data distribution, multi-geography and data center topographies, separation of workloads (e.g. real-time, batch), ETL processes that keep data on the move, and more

These four dimensions comprise very real challenges and very real opportunities for those who can effectively tame big data and make it work for them. A good example of the types of benefits that flow from big data is seen in this recent article from CNN Money that highlights the use cases of big data in the energy marketplace. We’re excited at DataStax about seeing our customers succeed in their big data projects through a blend of open source technology and commercial software. Our DataStax Enterprise big data platform provides the best of both worlds and supplies proven technology (Cassandra and Hadoop) that future-proof’s any application, and cost savings that range from 80-90% less than the RDBMS vendors charge. For more information, download our big data white paper along with a copy of DataStax Enterprise (free for development use – no limits or strings attached).

Apache Cassandra on Windows? Absolutely!

Robin Schumacher — Tue, 31 Jan 2012 13:14:56 +0000

Outside of politics and religion, nothing can get the blood rolling in tech folk’s bodies more than a healthy debate on the merits of operating systems, such as Linux vs. Windows vs. Mac. Personally, I’ve never used Mac in a production environment, but I have plenty of experience with databases on Linux and Windows. And I have a confession to make: I’ve always found Windows to be a pleasant experience. I’ve worked with Oracle, SQL Server, MySQL, and PostgreSQL on Windows and in general was happy with the results. I remember a case at a large company I worked where we benchmarked Oracle for a major trading system on a large HPUX system against a Windows config that cost 75% less and saw the Windows box win in 90% of the test cases. In my experience, I also never saw all the supposed outages, performance issues, etc., that some report on Windows. That doesn’t mean my experience of managing databases on Windows was perfect, but overall, Windows did a very nice job for the systems I was assigned to care for. A few months back we ran a poll asking what development platforms you used, and you said you used Microsoft Windows a lot (it was second on the list behind Mac). So we’ve listened to you and now made available a bundled installer for Windows that includes the latest version of Apache Cassandra, all utilities including the CQL interface, and our DataStax OpsCenter community edition. The MSI installer does everything for you, including creating all the Windows services you need, and takes about one minute to complete. For a run-through of what the new Windows package looks like along with some pointers on getting started on Windows, see a new article I’ve posted here on using our new installer on a single Windows box. The installer is primarily designed for single Windows installs on workstations and laptops, but you can create multi-node setups with a little manual tweaking – something I demonstrate in this article that shows how to create and monitor a new Cassandra cluster on Windows. Today, we support Windows 7 and Windows 2008 Server, both 32 and 64-bit, for development work only. Production support will be coming soon. Download our new Windows package and let us know what you think. And thanks for continuing to support Apache Cassandra and DataStax!

DataStax OpsCenter Now Available on Mac

Robin Schumacher — Tue, 31 Jan 2012 13:02:42 +0000

Well, you asked and we’ve responded. You said you wanted OpsCenter on Mac on one of the polls we recently took, and now you’ve got your wish. You can now download and run DataStax OpsCenter on your Mac. If you want a detailed run-through of how easy it is to install both Cassandra and OpsCenter on Mac, see a new article I’ve posted here. Keep in mind that, currently, we only support OpsCenter on Mac (both Community and Enterprise versions) for development purposes right now. Download DataStax OpsCenter for Mac and let us know what you think. And thanks, as always, for your support of DataStax and Apache Cassandra.

DataStax Community Server 1.0.7 Now Available

Robin Schumacher — Mon, 23 Jan 2012 13:21:14 +0000

We’ve updated the DataStax Community Server to Apache Cassandra 1.0.7 and it’s now available for download on our community downloads page. All changes can be found in the CHANGES.txt file in the main installation directory. Enjoy!

My thoughts on Amazon’s DynamoDB

Billy Bosworth — Thu, 19 Jan 2012 01:41:39 +0000

Earlier today Amazon announced DynamoDB, a hosted database for Amazon Web Services, which prompted several friends and colleagues of mine to ping me for my thoughts. The reason they asked is because Cassandra was initially created as the combination of Amazon Dynamo’s fully distributed architecture and Google Big Table’s rich data model. So, naturally, they wondered if I saw this as a competitor to Cassandra. I don’t typically get into the “feeds and speeds” differences in offerings, but if you are interested in that, I suggest you take a look at this post from Jonathan Ellis, the Apache Cassandra Chairman. For me, I’ll share a few of my thoughts at the business level.

Personally, I have never believed that other post-relational (aka NoSQL/Hadoop) database companies were our primary competition. The brute fact of the matter is that if you put us all together, we are still not statistically relevant compared to the overall DBMS market. In order to change that, we need to change the ecosystem itself. Many decision makers who are excited about big data technologies are also frustrated at the lack of human resources who really know how to leverage them. That isn’t just about ease of use either. It’s about a fundamental shift in thinking to solve problems in new and interesting ways.

When it comes to that kind of ecosystem evolution, we need more momentum than any of us new players can generate on our own — or even collectively. That’s why I got excited last year when Oracle entered the market with a real-time, NoSQL solution of their own. Today, I feel the same sense of excitement about Amazon’s announcement. Their release is another huge validation stamp on what now looks like an irreversible path toward the need for real-time, big data databases. It is a big step on the journey toward building a massive ecosystem of people who understand how to harness the power of these new technologies.

At this point you may say: “All well and good, but he’s dodging the question… are they competitive?”

Sure they are. So is every other database technology out there today, in some form or fashion. And you know who wins when we are all competing? Customers! It forces us all to create better and better solutions, like what we’re doing here at DataStax to leverage Cassandra in really exciting ways, not to mention the massive amount of time and improvements that have been put into Cassandra itself.

Competition is always exciting. So much so, that if no real competition exists, we humans will go to great lengths to manufacture it through games and contests! (Just put two kids in a room for a while and watch what happens.) In the big data world, it’s more than exciting … it’s necessary.

I had only one real personal fear coming into this market: That I would sink a big portion of my life into something that would never take hold in the mainstream. I suspect that would be a truly awful ending for all of us in this space. But thanks to companies like Amazon and Oracle, that feels highly unlikely now, and that is a great thing.

When failure is not an option for your big data system

Billy Bosworth — Mon, 16 Jan 2012 18:09:52 +0000

Having tackled the fundamentals of the peer-to-peer architecture in my last post, I now want to take a look at the section of this paragraph from our recent press release that touches on one of the most important aspects of running a mission-critical, big data system. The paragraph reads:

Customers this year chose Cassandra time and time again over competing solutions. The peer-to-peer design allows for high performance with linear scalability and no single points of failure, even across multiple data centers. Combine this with native optimization for the cloud and an extremely robust data model and Cassandra clearly stands apart from the competition for enterprise, mission-critical systems. [emphasis added]

While there are definitely some difficult concepts in the world of big data, “no single point of failure” isn’t one of them. Someone a little more business oriented may well prefer the phrase “continuous availability”, which is the result of having a system with no single points of failure. In either case, it basically means your system will remain available even under extreme circumstances because it is designed for failure. You may have read that last sentence and said, “Typo! You surely didn’t mean your system is designed for failure!” But that’s precisely what I mean. Notice that I didn’t say “your system is designed to fail.” I said your “system is designed for failure,” meaning, that architecturally it is built in such a way that assumes the components that make up that system will individually fail (maybe even very frequently) but that the larger system as a whole will remain available. As Google said many years ago in this paper on its distributed file system: “First, component failures are the norm rather than the exception”. This idea of accounting for component failures is often equated with the idea of “scaling out” or “scaling horizontally”, which is the opposite of “scaling up” or “scaling vertically”. When you scale horizontally, you add more machines to your system to increase capacity. When you scale vertically, you add more capacity to your single machine. For years, relational databases have handled increased capacity by scaling vertically, and aside from other challenges that causes, it introduces a single point of failure that jeopardizes continuous availability. But here’s the thing that many people don’t realize: scaling horizontally does not eliminate the challenge of a single point of failure. To truly achieve continuous availability, you have to understand the system architecture, which goes back to the discussion in my last post around the differences between “master/slave” and “distributed peer-to-peer.” Read slaves in the master/slave architecture introduce certain limitations, and there’s no easy way of getting around that. Conversely, Cassandra’s fully distributed architecture means every node is the same. Every node is a master, and every node is a slave. You never have to worry about how and when to add nodes of a certain type to increase capacity. And that also means you never have to worry about losing a node in your system. Failover testing isn’t required in Cassandra because Cassandra is constantly failing over from the moment you start your first cluster. Or said more succinctly, Cassandra is truly built for continuous availability when failure is not an option.

Does anyone see the irony in Oracle’s Big Data Appliance (asked a friend of mine…)

Billy Bosworth — Wed, 11 Jan 2012 06:26:56 +0000

Indulge me in a little fun on this one.

Did you hear the one about the world largest database company partnering with… umm…. an open source… uhh… database company… to serve as a software layer on their…. hardware? Oh, never mind — who would believe a tale like that? But in the world of big data, anything is possible. Last year we saw Oracle do a sudden about face on NoSQL in general, and now they are partnering to try and quickly fill the gaps in their offering. I think there are two broad takeaways here:

Big data solutions are going mainstream fast.
These ain’t your father’s data problems.

2012 is going to be such an exciting year in the big data space. In fact, I think I just figured out what happened to the Mayan calendar… they hit a big data problem and their calendar app crashed! We’ll rewrite it on Cassandra.

Choosing the right architecture for big data scale

Billy Bosworth — Wed, 11 Jan 2012 00:06:00 +0000

Following up from my last post, let’s now take a look at the section of this paragraph from our recent press release that deals with the foundation of Cassandra: it’s architecture. The paragraph reads:

Customers this year chose Cassandra time and time again over competing solutions. The peer-to-peer design allows for high performance with linear scalability and no single points of failure, even across multiple data centers. Combine this with native optimization for the cloud and an extremely robust data model and Cassandra clearly stands apart from the competition for enterprise, mission-critical systems. [emphasis added]

When dealing with new technologies, one of the easiest things to overlook is the architecture. Talking about architecture is not sexy or glamorous, but it is the absolute foundation of everything you will do for years to come. Make a mistake up front with most systems, and unwinding later it can be difficult. Make a mistake with your big data architecture, and unwinding it later can be downright ugly if not impossible.

Today’s big data architectures come in primarily two flavors: one where a single machine coordinates all activities for other machines in the cluster (aka Master/Slave); and one where all machines in the cluster are equal in type and function (aka peer-to-peer, or others may call it “fully distributed”). Cassandra is built on the latter–a fully distributed peer-to-peer architecture based on something called Amazon Dynamo. (For those who want to geek out on the details of Amazon Dynamo, you can read this paper.)

The decision between the two is vitally important. In master/slave architectures you have, by definition, a single point of failure in your master coordination node and you have introduced some complexity into scaling. There are techniques and tricks to try and mitigate this issue, but at the end of the day there is simply no free lunch and dealing with it at some level is inescapable. We will touch more on this in my next post.

Another benefit of the architecture comes in terms performance. The Cassandra developers are absolutely fanatical about performance and it shows. But what really still stuns me about Cassandra is not just its amazing performance, but that the performance scales linearly. Think of the advantages this provides to the operations and capacity planning teams. You don’t have to worry about what node types to add at what point along the way, you just keep adding nodes to the cluster and scale keeps going up exactly, mathematically, how you would expect it to.

An even more incredible aspect to this linear scale is that it is not limited to on-premise solutions. This graph shows how one of our customers achieved perfect linearly scalability that takes place entirely in the cloud.

When it comes to big data, choosing the right backend system is absolutely critical to the long-term success of your application and it’s worth a little up-front time investigating it.