What we’re doing with membase.org definitely falls in line with this description although with a slightly different twist. NorthScale saw the need for a simple, fast, and elastic NoSQL database that we felt wasn’t being met by existing technologies. When it became clear that many prominent companies shared this view and were committed to an open source solution, NorthScale stepped in to shepherd the development of a broad community around the membase.org project. Consistent with Matt Aslett’s description of open source 4.0, the result is a project with an “emphasis on collaboration and community rather than control.” While NorthScale has contributed the bulk of the code to the project, our customers Zynga and NHN are co-sponsors of the project who have a strong commitment to its success. This blurring of the line between vendor and customer – the collaboration between two seemingly opposite sides of a transaction – has long set open source apart from the large proprietary vendors who want nothing more than a lock on their customers.

Traditionally, the primary attraction to open source, and what enabled it to make inroads in the enterprise, has been cost. This is the “cheaper Oracle than Oracle” model where the technology is not necessarily solving any new problems in the market, but provides a cheaper open source version of something enterprises are already paying for.

However, when I talk to enterprise companies, lowering costs no longer cuts it as a sole driver for open source technology adoption. On the other hand, if we engage our customers around a very real and painful problem they’re dealing with – in our case, the mismatch between relational databases and the needs of interactive web applications – and demonstrate how we’re solving this with innovative new technology, then we can have a discussion.

In a nutshell, the fourth stage of open source is much more than just a return to community and collaboration – it’s about putting open source front and center as an engine of innovation. We’re seeing an emergence of open source projects that solve a new problem and create a new solution that eases this pain point. The source code just happens to be open because it’s what we have all come to expect. This is particularly true of infrastructure software going forward, where it’s expected that some component, if not all of it, is available as open source.

We believe open source 4.0 is characterized in part by projects that solve new problems with innovative solutions and use a highly collaborative model. We encourage the participation of both “corporate sponsors” and passionate individuals who are willing to contribute to the membase roadmap and strengthen the community.

http://wiki.membase.org/bin/view/Main/ClientImplementationGuide

The easiest approach is to start with your favorite memcached client library (that speaks memcached binary protocol) and proceed from there.  The fun part  is with handling the cases during Rebalance operations to allow for seamless cluster elasticity without data loss, but who doesn’t like fun challenges like these?

This beta release adds a lot of new functionality and reflects most of what you’ll find in the final product. Highlights include:

• Windows support
• Multi-tenancy – allows multiple buckets on a single cluster including bucket quotas
• “Cluster Overview” as a new monitoring dashboard
• And lots of small improvements and bug fixes, of course!

Let’s take a look at these features in a bit more detail:

Windows support is by far one of the most frequently requested features, and we are very pleased to offer it with this beta release. Beta3 provides 32-bit Windows support, with 64-bit support on the way (Note: The 32bit binary runs just fine on Windows 64-bit but is subject to the 32-bit memory limits). The Windows version provides the same feature set as our Linux version.

Multi-tenancy is the mechanism for creating multiple buckets on one membase cluster. Each bucket represents a separate namespace, but more importantly it also provides a resource control mechanism on a per bucket basis, allowing buckets to have different behavior. For example if you have some data you consider very important, you may want to create a bucket with a replica count of 3; for other less crucial data, a replica count of 0 might make sense. This way you can decide how to divide the cluster resources to accommodate different requirements for different applications or different types. No more one size fits all!

Bucket quotas are worth a bit more explanation. Each time you create a cluster, you set a fixed amount of memory that each server node in the cluster will contribute to the total cluster memory that buckets can consume. Once set, this value will be inherited by any server joining the cluster and cannot be changed. Hence, the total memory available for membase use in the cluster increases by this amount with each addition of server to the cluster.

Similarly, each bucket defines a memory quota that sets the amount of memory it can use out of the cluster total memory. This quota does not change as you add servers to your cluster, but you can manually edit this on the “Manage Bucket” screen.

In addition to the memory quota, there is also a disk quota associated with each bucket. In contrast to the memory quota, there is no fixed limit of disk space that each server brings to the cluster; all free disk space on the assigned storage path may be used. It is up to sysadmin to make sure that each node provides sufficient space to accommodate the data written (and you can track free disk space in the new Cluster Overview dashboard). Disk quotas are not yet enforced in Beta3, but you can already use it to monitor your bucket’s usage versus the quota.

The Cluster Overview provides a single cluster overview dashboard, showing you the most crucial stats of your cluster in one place.

As you can see you get a single page to keep track of the memory and disk usage of all your buckets, as well as how many operations your cluster is performing. The “disk fetches per second” serves as a potential issue indicator. If you are seeing a lot of disk reads it means that the working set for at least one of your buckets does not fit into RAM alone anymore. Disk reads are much higher latency than memory reads, so should this happen you can use the Data Bucket monitor section to drill down and understand which bucket is encountering the issue. If you need to take action you can increase the bucket memory quota in the Manage Data Bucket section. Issue resolved!

As you see we packed a lot of great new features into Beta3. But there is still more to come. You might be able to guess from the new bucket creation dialog that we have another bucket type in store, which will make multi-tenancy even more exciting – but for more on that you’ll have to check back later.

Enjoy Beta3 and let us know how you are getting on with the new features!

Wear a t-shirt, win an iPad!
And speaking of t-shirts, pick up your own yellow membase t-shirt and wear it around the show floor to win a chance for an iPad. We’ll be giving away iPads at our booth on Tuesday (8/31) and Wednesday (9/1) at 5:30pm, so stop by during the day to get your t-shirt and find out more about how to win.

Can’t make the show?
Test drive Membase anyway.
If you won’t be in San Francisco next week, you can still take Membase for a spin by downloading it here. There are also a number of webinars available for getting started, and a very active user forum as well.

Hope to see you at the show!

Things are moving at the speed of light over here and I wanted to take a second to come up for air.

We just had our 7th weekly beta webinar and this week I did a demo/preview (albeit quick) of some of the features and functionality coming in our soon-to-be-released beta 3.

Check out the recorded webinar for a sneak-preview and then download the real thing when it’s available.

Thanks for all the feedback and please keep it coming.

P.S. If you happen to be in the area today, stop by for a beer and some eats at our parking lot BBQ (behind our Mountain View offices from 3-8pm.)

We’ve sent you this email to let you know that we have observed that you may be running memcached in an insecure configuration. Specifically, we have noticed that you have at least one security group that allows the whole internet to have access to the port most commonly used by memcached (11211).

Regards,
The Amazon Web Services Team

Great email and service from the AWS team, and the suggested fix is spot on.

This posting is meant to provide some background on the issue and the alluded to “recent attention” the issue has received. The issue is relevant to all users of memcached, not just those deploying on Amazon EC2.

The vulnerability
The genesis of this bulletin was almost certainly the result of the development of go-derper by the team at sensepost, highlighted at the blackhat USA 2010 conference on July 30, 2010.

The highlighted vulnerability can be summarized as: if you deploy memcached on a server, leave the TCP port on which memcached is configured to listen (11211, by default) exposed to the Internet, leave the memcached ASCII protocol enabled, AND you are not using SASL authentication with the memcached binary protocol, then there is a trivial way for Bad Guys to retrieve and replace most of the contents of your cache. go-derper.rb is a simple Ruby application, built by sensepost, that can be used to exploit the vulnerability.

Eliminating the vulnerability
Let’s examine the vulnerability, clause-by-clause, and highlight what can be done to eliminate it, starting at the top:

“If you deploy memcached on a server,”

This may seem silly to consider, but there are actually options here. Not everyone needs to deploy and configure memcached on a server themselves in order to use the technology. If you are deploying memcached on a cloud platform, for example, you may simply leverage a pre-built image or even an add-on service.

We run the memcached add-on service for Heroku (itself run on Amazon infrastructure), the leading platform-as-a-service cloud provider for Ruby applications. Because we manage the memcached add-on, our deep expertise with memcached is implicitly brought to bear on behalf of the thousands of applications deployed on Heroku that leverage our memcached add-on.

Additionally, we are working closely with our friends at RightScale to make pre-configured memcached images available for those who want to deploy pre-configured memcached and membase instances on Amazon AWS.

If you are using one of these deployment options, we’ve ensured the configuration is secure.

“[if you] leave the TCP port on which memcached is configured to listen (11211, by default) exposed to the Internet,”

If you have deployed your own instance of memcached, either on your own equipment or in a cloud computing environment, then you need to ensure a firewall is protecting the system.

Amazon provides a rich set of capabilities for expressing and enforcing access control for instances running on EC2.

NorthScale co-founder Dustin Sallings also weighed in over the weekend; his blog provides great additional detail, especially regarding firewalling.

“[if you] leave the memcached ASCII protocol enabled,”

As built, the go-derper exploit depends on use of the ASCII protocol.

Memcached provides both an ASCII protocol and a binary protocol. The binary protocol was co-developed by NorthScale’s own Trond Norbye while employed by SUN Microsystems.

The fact of this vulnerability is that it also exists in the binary protocol, but the binary protocol supports authentication and access control, providing a mechanism for securing the data.

The ASCII protocol, the original protocol developed for memcached, does not have any facility for authentication or access control, and thus is not suitable for hanging on the public Internet. This protocol was explicitly developed for use behind a firewall, as a “back-end,” protected system.

In the unlikely event that you have some good reason to make the memcached port available to any host on the public Internet, but want to control access to the data, then you should disable ASCII protocol support (and enable SASL authentication on the binary protocol, as described next). The NorthScale distribution of memcached makes it easy to configure memcached to NOT bind the ASCII protocol listener to the memcached port.

“AND [if] you are not using SASL authentication with the memcached binary protocol,”

As mentioned above, the memcached binary protocol in recent releases of memcached does support authentication and access authorization via the SASL protocol.

The NorthScale distribution of memcached makes it very easy to leverage this capability. Creating a new “bucket” in our memcached distribution provides both multi-tenancy capability (allowing multiple applications to securely bind to a single memcached cluster) and serves as the vehicle for SASL credential binding. It is this capability that allows us to securely support thousands of memcached add-on users up on Heroku without running thousands of individual servers.

If you are using an older version of memcached (most linux distributions ship with antiquated versions of the software), and you need authenticated access support, you should look at a more recent version of the software. I certainly recommend our distribution.

The memcached historical context
The vulnerability is not surprising. Memcached was initially built by Brad Fitzpatrick for use at LiveJournal, in an environment where control over servers and network security was managed by a skilled team of system administrators. With many lines of defense in front of memcached, there was little need to build yet another layer of security into memcached itself, where the inevitable price would be development effort (effort better spent building blogging features) and performance (in an environment where many millions of memcached transactions are processed per day, and every single microsecond counts).

In a perfect world, every person developing and deploying software should fully understand the characteristics of all the underlying software infrastructure components on which their software is dependent; have a firm understanding of network security, policy formulation and policy enforcement; and regularly audit their operational environment while tracking emerging threats. Few systems would get deployed. In fact, I think it is fair to say that some of the most popular web applications on the Internet today would never have seen the light of day under those constraints.

In the real world, there is a lot of interesting software being developed and deployed by people who are not themselves, and who frequently do not have the resources to employ, experienced system administrators and network security specialists. They just want to get their software or service in the hands of as many users as possible, as quickly as they possibly can. If and when the billions of operations per second materialize, then the microseconds can be wrung out, hopefully by a competent team of system administrators which the organization can then attract, and afford.

An aside on cloud computing
Ultimately, this is one of the promises of cloud computing, as outlined in one of our white papers. Cloud computing is not just about transforming capital to operating expenses, or about leveraging service provider economies of scale. Managed hosting providers have been doing that for over a decade. Cloud computing ultimately enables software developers to develop and deploy software, without also building up expertise in system administration and network security. Ultimately the world is a better place as a result. More developers are empowered to build and deliver software solutions.

Amazon has demonstrated part of the value they provide to their customer base: they have tracked a newly highlighted vulnerability that is widely relevant (given the broad deployment of memcached) to their users, identified specific users at risk (possible given the metadata used to configure the virtual machine instances that ultimately underlie running systems on EC2) and notified them in a timely manner precisely how to deal with the problem. Serious value add.

Where do you get your memcached?
The team here at NorthScale provides the vast majority of the contributed source code to the memcached and membase open source projects. We respect the clearly expressed desire of the larger memcached development community that the core of memcached should remain raw, fast and best suited for those who know what they are doing.

We also make available commercially-supported, certified, less “raw” versions of those systems, making it easier for organizations to deploy, configure, secure and manage the software. While the memcached development community doesn’t want the code base polluted with “ease of use” features, there is a much larger potential community of users of the software that will be better off with those features present. Same goes for things like replication and live cluster reconfiguration. Many users want these capabilities, but the core community would prefer to keep them out of memcached proper. We make them freely available in our distribution and make the source code available in related projects (e.g. http://github.com/northscale/bucket_engine).

It’s been a whirlwind (in a good way!) of activity and I wanted to personally thank everyone who’s been involved and provided feedback. Being the beta program manager, I wanted to touch specifically on the last month as it relates to the beta program at large:

NorthScale Membase Server (aka – The product) – It’s been a great pleasure to work with such a talented and dedicated group of individuals here at NorthScale. We are radically focused on making Membase simple, fast and elastic. We’ve met that charge with flying colors but there’s still a lot more work to be done. Here’s where we were and where we are:

Beta 1 – Made available on June 23, it contained the basic features of complete memcached compatibility, replication and persistence of data as well as the ability to rebalance the data across multiple nodes. Alongside the beta release we introduced the Membase open-source project at membase.org.

Beta 1.1 – A quick refresh of the beta 1 code to address some known issues and introduce Ubuntu support! One of the earliest and most sought after features, we now support Ubuntu as well as RedHat/CentOS. Windows support is coming soon (see below).

Beta 2 – A major update of the code to fix bugs and add new features. Some of the key additions were:

• Disk > RAM. The first release required that your RAM was equal to your persistent storage. No more! Now you can define more disk than you have RAM available and we will handle moving data “up” and “down” based upon access patterns.
• Statistics in the UI – Our first pass at providing statistics of the cluster through the UI. Much more to come.
• Standalone Moxi – There is a proxy running on every Membase server in order to maintain compatibility with the open source memcached protocol. Running it on the server introduces some performance overhead and so now you can run it locally on the client to eliminate any unnecessary network traffic. See Steve Yen’s post to learn more.

The release notes for all versions are available here.

Beta Webinars – We’ve presented weekly webinar calls since the start of Beta1. I’ve fielded dozens of questions from participants and the feedback has been invaluable in making this beta program successful. A quick recap of the webinars to date (recordings are available):

• July 1 – The first webinar! We had just launched the beta a few days before and I demo’ed the initial setup and installation of Membase.
• July 8 – A demo on the rebalancing operation within Membase which allows you to add/remove nodes and automatically distribute the data and load across the cluster.
• July 15 – Matt Ingenthron presented our REST interface and how it can be used to programmatically control your Membase cluster.
• July 22 – I showed off the slick new features of our Beta 2 release: statistics in the UI, disk > RAM, client-side proxy for improved performance (Moxi)
• July 29 – The most recent webinar where I showed off the TAP interface which is an extremely powerful piece of technology upon which is built replication, rebalancing, querying/indexing, Hadoop integration and much more!

Forums – The forums have provided a direct and public channel from users to NorthScale. The most popular topics have been:

• Using Membase as a memcached server
• Performance overhead
• Deleting/flushing the storage bucket
• Querying/search feature
• Lack of capability in the GUI (yes, a bit of a black eye, but we’ve heard your cries!)

Looking forward – We’re not there yet people! There’s more to come including Windows support, multi-bucket support, and more UI cleanup/additions and …

Your call to action! – Now is where I need YOUR help. A beta program is only as successful as the users who participate. It is my job to make sure that you guys are actively engaged and that beta feedback is heard and understood. Please send me (perry@northscale.com) any and all use cases, feedback, requests, problems and questions. I have been engaging with people on a daily basis and have been getting some great information but I always want more.

And on a personal note – please get in touch with me or NorthScale the minute you experience an issue. Beta code has its bugs, but we don’t want them getting in the way of showing you what Membase can do…which is pretty darn cool, imho.

I had the opportunity to do this through a session at the conference.  Slides are available for download.

Unfortunately, this presentation was not recorded, but you can certainly post questions here and I’d be glad to answer.

There are several slides in there about their current cache deployment and their involvement in membase.  This is serious stuff.  Daily, their cache and membase deployment are right in the critical path of the top dozen or so applications at NHN.  These service 17M users daily across their 250 M  unique users.  In China alone, their services touch up to 750K concurrent users.

The success in deployment of moxi and memcached there have both given confidence in and lended to the design and ongoing deployment of membase at NHN.

Please look over the slides and join us over on the mailing list/IRC/XMPP off of membase.org or send me a note if you’d like to learn more.