GridCentric Blog

Creating VMs with JavaScript and Node

2011-05-27T09:47:00.000-07:00

Seriously, JavaScript?

Node is a framework built around the speedy V8 JavaScript interpreter. It enables you to use JavaScript to create back-end services for your web application (or any other application). I/O is based around the same event-driven model that is familiar to front-end JavaScript programmers. Putting aside any claims about the inherent performance benefits of that model, it's powerful to use the same language in all parts of a web application and I have no doubt that Node will grow to stand next to the big players (Rails, Django, Struts) in the world of web frameworks.

Copper is focused on enabling dynamic and flexible services on the back-end. We empower programmers and administrators with a programmatic model for scaling applications, embedded within the control flow of the applications. Given that Node is an upcoming web framework, a few weekends ago I embarked on the fun exercise of creating some GridCentric V8 bindings.

When I co-founded a systems company a couple years ago I would never have guessed that I'd write a line of JavaScript, but technology brings us to unexpected places.

This post has two simple goals.

To enable GridCentric API calls in Node applications.
To use those bindings in a nifty demo, creating VMs on-demand to scale an application.

For a nifty demo, I've chosen to take the standard Node chat demo and make it scale automatically as users join chat rooms, using only about a hundred lines of JavaScript.

Building the extension

To skip the details of building the extension, click here to jump straight to the demo.

Starting points for V8

If you want to write a Node extension, without a doubt the best place to start is the useful blog post from cloudkick. Figuring out where to go next once you've exhausted their example however, is a bit tricky. I found that the V8 embedder's guide was mostly useless, but maybe you'll have a different experience. I think the best way to learn more is by looking at the Node source and the source for other native extensions (a relatively complete list of extensions can be found here).

Synchronous bindings

Let's start with the easy stuff.

Simple, synchronous function bindings for Node are quite straight-forward. Many of our API functions can be considered non-blocking (reading a value from the kernel, equivalent to a system call) so they are quite simple to wrap. For example, the C function below will return the current vmid.

#include <gridcentric/gc-guest.h>

int func() {
  return gc_vmid();
}

In the JavaScript world, we really want to this look like:

var gridcentric = require("gridcentric");
var vmid = gridcentric.vmid();

To get those semantics, we can take the C example above and wrap it in some V8 voodoo.

#include <node/v8.h>
#include <node/node.h>

#include <gridcentric/gc-guest.h>

using namespace node;
using namespace v8;

static Handle<Value> VmId(const Arguments& args)
{
    HandleScope scope;
    Local<Integer> result = Integer::New(gc_vmid());
    return scope.Close(result);
}

The V8 function is complete.

Before we have a usable extension however, we must bind this function appropriately within the native module. When Node loads a native extension, it executes the init symbol, passing in a variable representing the scope created for the module. Using a FunctionTemplate wrapper, we define an init† symbol that does the appropriate binding within the module.

extern "C" {
  void init (Handle<Object> target)
  {
    Local<FunctionTemplate> vmid = FunctionTemplate::New(VmId);
    target->Set(String::NewSymbol("vmid"), vmid->GetFunction());
  }
}

† The init function must be wrapped in an extern "C" declaration to prevent g++ from name mangling.

Almost done -- we only need the build script. To build the module with our vmid function, we first create a wscript file (assuming that our source file is src/gridcentric.cc) used by the Node build tool node-waf.

def set_options(opt):
  opt.tool_options("compiler_cxx")

def configure(conf):
  conf.check_tool("compiler_cxx")
  conf.check_tool("node_addon")

def build(bld):
  obj = bld.new_task_gen("cxx", "shlib", "node_addon")
  obj.cxxflags = ["-Wall"]
  obj.ldflags = ["-lgridcentric"]
  obj.target = "gridcentric"
  obj.source = "src/gridcentric.cc"

Finally, we run node-waf configure && node-waf build to build our extension.

We now have a basic gridcentric module, and the following code works.

var gridcentric = require("./build/default/gridcentric");
var vmid = gridcentric.vmid();
console.log("My vmid is " + vmid);

Now we can move on to adding some meat to the module.

Understanding non-blocking operations

Node event-driven semantics require that any function doing significant work (i.e. I/O) be structured using callbacks. Much of the heavy-lifting of a binding is caused by the need to restructure calls to your library using the callback mechanism provided by Node.

This can be a bit of a pain, but fortunately many of the functions in our guest bindings were well-suited to the asynchronous callback style required by Node (and I think that it's generally not too difficult to find a nice mapping). For example, the request ticket operation may take a few hundred milliseconds to make the round-trip to the scheduler, allocate the requested resources and return the result. Similarly, the clone operation may take seconds, but in a complex control flow you'll likely need to be doing other things during that time.

With our C bindings, the request ticket function call looks like:

#include<gridcentric/gc-guest.h>
...
gc_uuid_t ticket;
if( gc_request_ticket(1, 1, 1, 1000, &ticket) < 0 ) {
   perror("Couldn't request ticket");
}

If we were to translate directly into JavaScript using a synchronous style this might look like:

var gridcentric = require('gridcentric');
...
ticket = gridcentric.request_ticket(1, 1, 1, 1000);
if( ticket ) {
  console.log("Successfully allocated ticket " + ticket + ".");
} else {
  console.log("Unable to allocate ticket in 1000 milliseconds.");
}

But because the request_ticket operation will block up to 1000 milliseconds in this case, this function doesn't conform to the non-blocking semantics required by Node.

Instead, we must structure this function to use an asynchronous callback when the ticket request is completed, as follows:

var gridcentric = require('gridcentric');
...
gridcentric.request_ticket(1, 1, 1, 1000, function(ticket) {
  if( ticket ) {
    console.log("Successfully allocated ticket " + ticket + ".");
  } else {
    console.log("Unable to allocate ticket in 1000 milliseconds.");
  }
});

Notice that we pass in a function as the last parameter. This function will be called asynchronously with the return value of the request_ticket function after it has completed. We don't have any guarantees about when that function will be executed.

Once this style is adopted for all functions, we can easily see how to chain operations using closures. For example, in order to fork() the VM we can extend the above:

var gridcentric = require('gridcentric');
...
gridcentric.request_ticket(1, 1, 1, 1000, function(ticket) {
  if( ticket ) {
    console.log("Successfully allocated ticket " + ticket + ".");
    gridcentric.clone(ticket, function(vmid) {
       if( vmid > 0 ) {
           console.log("On a clone VM.");
       } else if( vmid == 0 ) {
           console.log("Still on the master VM.");
       } else {
           console.log("Error during clone operation.");
       }
    });
  } else {
    console.log("Unable to allocate ticket in 1000 milliseconds.");
  }
});

Implementing callbacks

Before implementing these functions, you'll notice above that my simple example did not require any arguments. The first thing that I will do is define a number of processor macros to sanity check passed in arguments.

#define REQUIRE(I, ISTYPE, CASTTYPE, NAME)                     \
  if( args.Length() <= (I) || !args[I]->Is##ISTYPE() )         \
    return ThrowException(Exception::TypeError(                \
      String::New("Argument " #I " must be a " #ISTYPE "."))); \
  Local<CASTTYPE> NAME = Local<CASTTYPE>::Cast(args[I]);

#define REQUIRE_STRING(I, NAME) \
        REQUIRE(I, String, String, NAME)
#define REQUIRE_INTEGER(I, NAME) \
        REQUIRE(I, Number, Integer, NAME)
#define REQUIRE_FUNCTION(I, NAME) \
        REQUIRE(I, Function, Function, NAME)

We could now implement a no-op request ticket function that takes the appropriate arguments.

static Handle<Value> RequestTicket(const Arguments& args)
{
    REQUIRE_INTEGER(0, maxcpus);
    REQUIRE_INTEGER(1, minvms);
    REQUIRE_INTEGER(2, mincpuspervm);
    REQUIRE_INTEGER(3, timeout);
    REQUIRE_FUNCTION(4, cb);
    return Undefined();
}

It sanity-checks it's input. Now it needs to do something.

Node uses libeio as the basis for its thread pool (which, assuming you are not working with raw file descriptors and sockets, you will likely be using). To use libeio, you schedule two functions for future execution: one that does the work and one which will be called when the work is completed. The function called when the work is completed will be executed in the main thread, so it needs to be quick. You are also permitted to pass an opaque pointer, which will be (indirectly) passed to each of the two functions.

For our example below, we will first define a new class that we can use as an opaque pointer. This class will hold all data related to the ticket request, the callback function passed in, and the return value to be given. Since we've going to have three functions involved: the one called by V8, the one scheduled by libeio and the one executed after the work is complete, this class will be used to pass around shared information to each of them.

We will also declare two functions ahead of time that will use for our libeio work, EIO_RequestTicket and EIO_Post.

class CallbackData
{
public:
    Handle<Value> This;      // The this scope we were called in.
    Persistent<Function> cb; // The callback function passed.
    Handle<Value> rval;      // The return value to be given.

    // The parameters required for request_ticket.
    int maxcpus;
    int minvms;
    int mincpuspervm;
    int timeout;
}

static int EIO_RequestTicket(eio_req* req);
static int EIO_Post(eio_req *req);

We use the This variable to track the scope, cb to record the callback the user passes in and rval to store the return value once the work is done. The rest of the parameters are required for the actual ticket request.

Given these declarations, the actual RequestTicket function is straight-forward.

static Handle<Value> RequestTicket(const Arguments& args)
{
    REQUIRE_INTEGER(0, maxcpus);
    REQUIRE_INTEGER(1, minvms);
    REQUIRE_INTEGER(2, mincpuspervm);
    REQUIRE_INTEGER(3, timeout);
    REQUIRE_FUNCTION(4, cb);

    // Create the opaque pointer.
    CallbackData *data = new CallbackData();

    // Set the scope variable (in case its needed).
    data->This = args.This();

    // Set the parameters associated with the ticket request.
    data->maxcpus = maxcpus->Value();
    data->minvms = minvms->Value();
    data->mincpuspervm = mincpuspervm->Value();
    data->timeout = timeout->Value();

    // Save the passed callback.
    data->cb = Persistent<Function>::New(cb);

    // Schedule the EIO functions to be run.
    eio_custom(EIO_RequestTicket, EIO_PRI_DEFAULT, EIO_Post, data);
    ev_ref(EV_DEFAULT_UC);

    return Undefined();
}

As required, it doesn't do any real work. It allocates the opaque data pointer (the CallbackData class we defined), schedules the work in the thread pool (lines 15 and 16), and returns Undefined() immediately.

All that remains is for us to actually implement the missing functions.

The first EIO_RequestTicket does the work required (called gc_request_ticket) and sets the return value (rval) in the opaque pointer.

static int EIO_RequestTicket(eio_req* req)
{
    CallbackData *data = static_cast<CallbackData*>(req->data);
    gc_uuid_t uuid;

    if( gc_request_ticket(
            data->maxcpus, data->minvms, data->mincpuspervm,
            data->timeout, &uuid) < 0 ) {
        // Set the result to undefined.
        data->rval = Undefined();
    } else {
        // Save the resulting ticket as a string.
        data->rval = String::New(uuid.value);
    }

    return 0;
}

The second function, takes the given opaque pointer, creates a V8 array using the return value and calls the callback function that was passed in as a argument. This will also not block.

static int EIO_Post(eio_req *req)
{
    CallbackData *data = static_cast<CallbackData*>(req->data);
    ev_unref(EV_DEFAULT_UC);
    Local<Value> argv[1] = { *(data->rval) };
    TryCatch try_catch;
    data->cb->Call(Context::GetCurrent()->Global(), 1, argv);
    if (try_catch.HasCaught()) {
        FatalException(try_catch);
    }
    data->cb.Dispose();
    delete data;
    return 0;
}

That's it! All that's left to do is to bind the RequestTicket function appropriately within the extension (see vmid example above), then our asynchronous request ticket function will be working like a charm.

Wrapping objects

Some of the GridCentric API functions return more complex structures. Although I would recommend mapping values to V8 primitives wherever possible, the need may arise to return more complex JavaScript objects.

After negative experiences with wrapped objects in V8, I think that unless you require complex interactions with the JavaScript world -- you can return complex objects as simple JavaScript Objects (i.e., no prototype). Below is my example for creating a TicketInfo object.

#define SET_VALUE(VAR, NAME, TYPE, VAL) \
    VAR->Set(String::New(NAME), TYPE::New(VAL))
#define SET_INTEGER(VAR, NAME, VAL) \
        SET_VALUE(VAR, NAME, Integer, VAL)
#define SET_STRING(VAR, NAME, VAL) \
        SET_VALUE(VAR, NAME, String, VAL)

class TicketInfo {
public:
    static Handle<Object> Create(gc_ticket_info_t info)
    {
        HandleScope scope;
        Local<Object> obj = Object::New();
        SET_STRING(obj, "id", info.id.value);
        SET_STRING(obj, "status",
           gc_ticket_status_string(info.status));
        SET_INTEGER(obj, "cpus", info.cpus);
        SET_INTEGER(obj, "vms", info.vms);
        SET_INTEGER(obj, "mincpuspervm", info.mincpuspervm);
        return scope.Close(obj);
    }
};

Pre-processor tricks and gotchas

If you look at the source for my extension on bitbucket, you'll see that I didn't explicitly define separate classes and functions for each of the callbacks. Due to the repetitive nature of the wrapping, I wrapped most of the callback code into hacky pre-processor macros.

I also encountered one annoying gotcha while building the Node extension. During the clone operation, libgridcentric executes the scripts at /etc/gridcentric/pre-clone and /etc/gridcentric/post-clone. This execution is simple. Here's some pseudo-C.

pid_t child = fork();
if( !child ) {
  exec(script);
} else {
  int rc = waitpid(child,...);
}

When executed from within Node, the waitpid fails with return value -1 and causes the clone operation to be aborted if there is an /etc/gridcentric/pre-clone script. Why? Ostensibly, the waitpid fails because a different part of Node gobbles up all child processes and their associated return values. Presumably this is prevent Zombie processes, but it's not a great solution. The workaround for the gridcentric extension is to remove these scripts, but then you lose this functionality.

The application

Enabling the GridCentric API in an application running on our platform allows it to dynamically scale horizontally by requesting resources and cloning itself, much in the same way fork() works in UNIX. The cloning operation is handled transparently from under the VM in seconds, with state magically propagated. With a Node application, our stack will look something like this.

With the bindings I've just built, this operation in JavaScript looks like this.

var gridcentric = require("gridcentric");
gridcentric.request_ticket(1, 1, 1, 1000, function(ticket) {
  if( ticket ) {
    gridcentric.clone(ticket, function(vmid) {
      if( vmid < 0 ) {
        console.log("There was an error.");
      } else if( vmid == 0 ) {
        console.log("I'm on the original VM.");
      } else {
        console.log("I'm on a clone with id " + vmid + ".");
      }
    });
  } else {
    console.log("Unable to allocate resources.");
  }
});

Service structure

More logic is required to scale a service than simply cloning a VM. To scale any service horizontally, you'll need to implement same kind of proxy or load-balancing mechanism.

Using the completed bindings, I created an autoscale.js module which turns the master VM into a proxy (based on this) and routes requests to clones which are created automatically. In other words, it turns a regular Node application into a auto-scaling service.

In this case, we create a new VM for every two active users we have and don't synchronize state across different VMs (think of it as a Node chat roulette -- only with cloning VMs).

More specifically, the autoscale.js implements the following simple algorithm.

Every second, we fetch the list of clone domains and store their IPs in a global array.
If there is less than one clone for every two active connections, we create the appropriate number of clones.
When a new connection arrives, it is mapped to the latest clone VM.

Integration

To leverage this service, I modified the chat demo to use the auto-scaling module. This required adding the following lines at the bottom of the server.js file:

setTimeout(function() {
  as = require("./autoscale");
  as.setup(80, PORT);
}, 3000);

I add the 3 second delay so that the service can get started before the first clone operation.

The following video is a quick demo of the service. I log on to the auto-scaling chat service with five different users. Because the service has been configured to create new VMs for every two users, the five users in the demo are routed to three different VMs that are created on-demand, in the span of seconds. Enable annotations for notes during the video.

Caveats

Much like Node chat, the demo described here is not intended to be a serious service. Were you to add auto-scaling to a real application, you'd definitely need to do a better job of tracking active hosts, synchronizing necessary state across slaves and handling errors in general.

It's worth pointing out however, that this is miles easier with the gridcentric extension than in the case where you have to provision new VMs from scratch. Provisioning from scratch, you'll likely need to involve lots of languages and tools (init scripts, chef or puppet, configuration files, proxies, synchronization servers) before you even touch the application. The semantics of clone() give the programmer a very powerful primitive on top of which they can build reliable distributed services. Plus, it's pretty awesome.

Do-it-yourself

If you have a Copper installation, feel free to install the bindings from NPM and try them out for yourself. There are likely a few bugs, but I'd love to hear feedback or complaints. You will need to have the gc-guest-base package installed in the VM to provide libgridcentric, then the module can installed simply:

$ npm install gridcentric
gridcentric@0.0.1 ./node_modules/gridcentric

If you want to dig more into the bindings (or steal macros -- please go ahead), the full source is available here. This source also includes the simple autoscale.js module used above.

Enjoy!

Infrastructure performance

2011-03-17T11:01:00.000-07:00

There's a lot of FUD surrounding private cloud. Without getting in to definitions, I figured I would add a bit of noise and have some fun with infrastructure performance in the process.

Cloud vs Claude

I'm a virtualization guy -- I know that cloud can be interpreted a lot of ways, but for now I'm using it to describe Infrastructure-as-a-Service (IaaS). Public cloud means public infrastructure providers such as Amazon's EC2 or Rackspace's cloud.

From my perspective, public IaaS offerings fundamentally provide two forms of value:

Outsourcing of infrastructure costs (hardware, management, etc.)
Cloud technology as a platform (APIs, billing, image libraries, etc.)

It's clear that the value of the first item is fully realized by traditional (non-cloud) hosting and co-location services. Yet, based on how these companies are scrambling to release cloud offerings, the cloud providers must be killing these shops in the market. It follows that there must be something more than just vapor to that second point. Hourly billing alone is not driving people to choose cloud providers over traditional hosts.

Private Claude

That brings us to the definition of private cloud. It's what you get when you have cloud technology running on your own infrastructure -- no outsourcing of hardware. But WTF is cloud technology? While virtualizing your infrastructure buys you a lot of agility, there's definitely more to it than just virtualization. Here's my take.

The key difference between vanilla virtualization and cloud is automation.

Automation isn't just one thing -- there are lots of ways to automate. Automated provisioning means that you can deploy virtual machines based on standard templates with the click of a button (an or API call). Automated billing means that resources are accounted for and reports are automatically generated or fed-back into a billing system. Automated scaling means that services are architected to automatically grow and shrink their footprints over time, using infrastructure APIs.

To me, automation is a fundamental tenet of cloud technology.

So where does private cloud make sense? In environments where automation makes sense.

Automation sees return on investment when things change often: workloads, requirements, environments, usage. These environments are very common, but not in every business. Technical computing, content creation, service providers and hosting all may have demanding and unpredictable workloads but may also have strict data requirements (legal or bandwidth) that prevent them from using public infrastructure. Private cloud makes immediate sense here.

If you're not using hosted infrastructure today but are faced with dynamic workload and automation needs, then it's likely that cloud is a revolution that will change how you compute in your own datacenter.

Where do we come in?

Our technology transforms how applications interact with infrastructure, simplifying automated provisioning and scaling.

Instead of requiring virtual machine templates to be stored and maintained as complete disk images, Copper enables on-demand cloning of live, running virtual machines in real-time via a powerful API. One second you have a single server, the next you have dozens -- and each server is aware of its unique ID allowing complex, distributed services to be created easily.

You provision virtual machines in seconds. So what? I can deploy a new server on <favorite platform> in just a few minutes. That's good enough, I've never needed anything better.

The above remark is a classic (and ludicrous) response I occasionally get when I explain what our technology does. It's a total straw man. That's good enough for what you do today because you've never been able to do anything more. It's like saying that you'll never need a hover car, or that "640Kb [of RAM] ought to be enough for anybody".

Taking minutes to provision an additional virtual machine means that you need to be able to predict fluctuations in demand and provision in anticipation of load. It also means that if you want to run a large-scale parallel computation that can take as little as a few minutes given a few hundred machines then you'll probably wait many, many times the required time or most of the time and cost in your analysis will consist of deploying virtual machines instead of actually running code.

Being able to scale in seconds changes the name of the game. Software can predict and react in response to demand, in real-time. That's the basis for our upcoming hosted cloud offering, RiotCloud, and our powerful grid queueing solution GCQ.

Taking it on the road

If automation is a fundamental tenet of cloud technology, then measuring the speed at which infrastructure software reacts to programmatic automation is a logical next step. I'm a massive fan of Top Gear, despite having a complete lack of interest in cars. After realizing that this was quite normal, I thought it would be fun to do a Top Gear tribute by boasting about our raw infrastructure performance.

The below tests were performed on our test ~~track~~ cluster by our tame ~~race driver~~ infrastructure programmer, ~~the Stig~~ Steve, seen here. Some say that Steve's brain, if hooked up to an IPv6 backbone, is capable of routing over 50GiB per second.

Acceleration

A fun test for infrastructure software is how quickly it allows an application to scale, which is tantamount to how fast new machines can be provisioned and booted. A faster time-to-scale means smoother upgrades, fewer resources wasted transferring state and an overall more dynamic infrastructure. For example, consider scaling an analysis application (like Hadoop) to perform a time-critical computation. Our software requires zero time to create and bundle virtual machine templates (it's not necessary), so for us this is simply a matter of how quickly we can scale a running virtual machine. In fact, everything here is done within the virtual machine being scaled, showing the power of our API.

Before Steve did his thing, we were lucky enough to have Jeremy Clarkson of Top Gear take our software for a spin and give his first impressions.

Don't worry: he's fine now, just a little shaken up.
Also, that's obviously not really Jeremy Clarkson.

In the tests performed by the our tame infrastructure programmer, we found a blazing average time of just a little over 9 seconds to scale from a single virtual machine to sixty-one, making repurposing infrastructure a staggeringly fast operation. Remember, that when Copper clones virtual machines, they are already running with all the necessary application state -- 9 seconds is everything, there is no boot time.

Cornering

Although acceleration is important, a huge factor in how fast you can make it around the track is how your machine can handle corners. Application cornering is a lot like cornering in a race car: you need to slow down in one direction, then speed up in another.

The below video shows Steve apexing a nearly 180 degree corner. Our infrastructure starts off running one application with over one hundred virtual machines, destroys them all, then scales a second application out to over one hundred virtual machines. This process takes less than 20 seconds.

The fine print

These demos show the power and speed of dynamic infrastructure, but I'm glossing over a few details. When you're scaling applications (in or out), they'd better be ready for it. Although a lot of applications can be adapted, there are plenty of frameworks that fit the bill already, especially in spaces accustomed to dynamic workloads.

To give some concrete examples of what they might be, suppose that application one is for distributed data crunching (like SETI@Home or a batch processing system) and application two does distributed web load testing (like Selenium Grid). In that case, cornering might consist of putting aside the opportunistic analysis that we have our infrastructure doing in order to load test the latest website release.

When provisioning is so simple and takes seconds, we can easily imagine building new infrastructures that react and scale like never before. Applications can scale in response to load or needs in real-time. The example above, where infrastructure is taken over at the flick of a switch, could easily happen every night, every day, every hour or even every time some stupid developer (or umm... CTO) pushes code and breaks the build.

That's it folks!

In other news, we've recently released Copper 1.4.2 and updated all the guest images available include Debian 6.0, Fedora 12, Fedora 13, and CentOS 5.5. Happy updating.

I also want to add that although we're excited to work every day at changing the world with innovative virtualization software, our team's thoughts and hopes are with those affected by the tragedy in Japan.

How fast can you add more worker nodes?

2011-01-04T11:13:00.000-08:00

There is a very important, and often overlooked, question when it comes to the scalability of the system: How fast does the system scale? The main question is still how will the system scale in the first place. But in the age of Internet powered applications that revolves around sudden, unexpected peaks of traffic, just answering this question is not good enough. The system doesn't just need to be able to scale, but to also scale instantaneously.

The producer-consumer programming model is a good example of a software design that has a very simple conceptual path to scaling. The model basically consists of three parts: A Producer that creates work, a Queue that stores a backlog of pending work the producer has created, and finally a Worker that takes the pending work from the queue and executes it.

Take for example uploading a video to YouTube. The producer would be the web server that creates the work of "process this video file". A worker then takes the next video file to process, and performs all the codec conversions, formatting, etc. on the video file. Once it is done, it picks the next waiting video file from the queue. Suppose suddenly a couple hundred people decide to all upload their video at once. The work queue would get huge, and the system will start to get bogged down. However, more workers can be added to the system in order to scale it up to handle the increased load. Essentially the system's throughput is proportional to the number of workers.

Here is a quick video that demonstrates the scalable nature of the producer-consumer model. There is a producer node that creates jobs that will take a worker ~ 5 seconds to complete. The producer creates, or submits, 2 jobs every second. Obviously this is more than the single worker can handle, so more workers are added to the system to keep up to pace with the producer node, and to tackle the backlog of jobs.

There are two main virtual machines in the video: A producer machine and the worker machine. The producer machine is running a Django application, which is used to visualize the system, RabbitMQ, which is an AMQP system used to support the producer-consumer model, and finally a simple shell loop that hits the Django application to submit jobs to the work queue. Here is the code snippet from the the Django view.py that handles the job submission:

from django.shortcuts import render_to_response
from prdconapp.models import Job, Server

# pika is a simple AMQP library that is used to interact with the
# RabbitMQ instance.
import pika

def give(request, vm_id, num_jobs):
    """
    This gives jobs to the system. It will create num_jobs more
    jobs and submit them to the queue, and update the 
    visualization state with these jobs. The vm_id is just a 
    string to indicate what machine is giving these jobs.
    """
    
    # Get a connection of the queue instance called "task_queue" 
    # that is running on the local host.
    connection = pika.AsyncoreConnection(
            pika.ConnectionParameters(host='localhost'))
    channel = connection.channel()
    channel.queue_declare(queue='task_queue')
    
    # Submit the desired number of jobs to RabbitMQ.
    for i in range(int(num_jobs)):
        j = Job(owner=vm_id, status='pending')
        j.save()
        message = "Job id: " + str(j.id)
        channel.basic_publish(exchange='', 
                routing_key='task_queue',
                body=message, 
                properties=pika.BasicProperties(
                delivery_mode = 2, # make message persistent
            ))


    return render_to_response("index.html", job_dict())

The worker virtual machine simply runs a single python script that takes the next job from the RabbitMQ queue, updates the Django application visualization, runs the job, and then loops to take the next job. When the Copper platform does the live-cloning of the virtual machine, this same python script is already automatically running in each of the cloned machines. This is how the new workers are able to connect to the RabbitMQ system, and start pulling off the next pending job. Here is the worker code:

import time
import urllib2
import sys

# pika is a simple AMQP library that is used to interact with the
# RabbitMQ instance.
import pika

# The gridcentric library binding used to get some information
# about the virtual machine on which this script is running.
from gridcentric import guest as gc

# The host that has the RabbitMQ instance. It is also the host
# running the Django app we need to update.
host = '192.168.1.80'

# We use the gridcentric library to get the unique vmid of this
# machine. This is used to determine if we are a cloned virtual 
# machine, or the original master machine.
VMID = gc.vmid()

# Start a connection to the RabbitMQ server and listen to the 
# task_queue queue.
connection = pika.AsyncoreConnection(
        pika.ConnectionParameters(host=host),
        True,
        pika.connection.SimpleReconnectionStrategy())
channel = connection.channel()
channel.queue_declare(queue='task_queue')

print ' [*] Waiting for messages. To exit press CTRL+C'

# This is the call back function that gets called when this 
# worker receives a message from the server. It essentially 
# updates the visualizer, and runs the task.
def callback(ch, method, header, body):
    # Use the gridcentric library to get the IP address of 
    # this machine in the cluster's private virtual network. 
    # We will use this to uniquely identify this machine in
    # the visualization.
    ip = gc.my_ip()
    
    print " [%s] Received %r" % (ip,body,)
    job_id = body.split(":")[1].strip()
    try:
        # Update the visualizer to tell it that we have taken
        # the job and are going to be running it.
        urllib2.urlopen("http://%s:8000/prdcon/take/%s/%s" % 
                (host, ip, job_id) )

        # All jobs themselves are just sleep jobs. We could 
        # decode the message, and run the job, or better yet 
        # use something like Celery that does all of this for 
        # us. For the purposes of the demo, sleeping 5 seconds 
        # should be fine.
        time.sleep(5)

        # Update the visualizer to tell it that we have completed
        # the job.
        urllib2.urlopen("http://%s:8000/prdcon/done/%s/%s" % 
                (host, ip, job_id) )
        print " [x] Done"
    except Exception:
        pass

    # Tell the RabbitMQ server that we have processed the message, 
    # and we are ready for another one.
    ch.basic_ack(delivery_tag = method.delivery_tag)

    # If we detect that our VMID has changed, we basically want 
    # to reconnect back to the RabbitMQ server because we are a 
    # new machine. This is a bit of a hack to get a demo worker, 
    # but basically the worker will exit. This is why we run the
    # worker in a while loop in the shell.
    if gc.vmid() != VMID:
        sys.exit(0)

# Tell the RabbitMQ server that we only want to receive a single 
# message at a time.
channel.basic_qos(prefetch_count=1)

# Register the callback function.
channel.basic_consume(callback,
                      queue='task_queue')

# Just keep looping and waiting for new messages to process.
pika.asyncore_loop()

The producer-consumer model is a straightforward design that has many simple semantics that make it easy to reason about how it will scale. But, as I mentioned in the beginning of this post, the mere fact that this model can scale is not good enough. The next, and almost equally important question, is how fast does it scale?

Let's take a look at how long it would take to scale the producer-consumer programming model throughout time:

Days: In the 1980s adding new workers to the system would be in the magnitude of days because it was a very manual process, both physically setting up the computer and manually configuring it. Taking days to scale an application is simply not good enough.
Hours: In the 1990s ghosting computer images enabled computers to be preconfigured, however they still needed to be physically setup and it was still a big manual process. Scaling in hours is better, but it is still not able to capture a lot of potential.
Minutes: In the 2000s virtualization and cloud computing started to become more main stream removing a lot of the manual process in setting up new workers. This has helped enabled the current explosion in Internet based applications, but there is still a window of lost potential as the system scales up.
Seconds: GridCentric's Copper platform can add functioning workers to the system an order of magnitude better than everyone else. With delays of only seconds, the system has become a lot more responsive, and reactive, to peaks in demand. Now the window of lost potential has gone from minutes, down to seconds. The added bonus is it is often simpler, and easier, to add workers using the Copper platform, than using other platforms that can only accomplish the task in minutes.

Here is the video again, and as you watch it remember to ask yourself how fast can you add workers to your scalable system, and what are you losing as you scramble to react to a temporary peak in demand?

Tis the season...

2010-12-20T09:09:00.000-08:00

We've been meaning for a while to put up some videos demoing how Copper can be used to implement load-testing environments with minimal setup. Well, here they are:

In retrospect, it's a simple use case that's easy enough to set up and demonstrate, and makes for some cool visuals... so it's about time we got around to it.

There are two videos, one by me and one by Dave. Each of them follows the same basic format: we start off with a single master VM running a simple hitcounter webapp, and then spawn clone VMs that run an HTTP client and ping the hitcounter. The hitcounter is observed via the webapp's main page, which updates itself dynamically. My hitcounter's main page just counts the hits, while Dave's gets all fancy about it ;)

The way our scripts gets clones to do their work is also different. In my case, a script running on the master VM initiates the cloning, then branches into the code that runs the http client. In Dave's case, he backgrounds a shell command which runs in an infinite loop pinging the hitcounter, and then just creates clones directly using the 'gc clone' command. His background command continues executing on each clone.

Also.. I know.. this demo is not actually a DDOS or a load test. However, throwing a couple of while loops around the http client would get the behaviour pretty close to it.. ;) It's not particularly hard to topple a Django standalone server running in test mode. Actually, in Dave's video you can see the server start struggling to fulfill requests towards the end..

Anyway, if what you see intrigues you, feel free to download and play with Copper yourself.

Cheers and Happy Holidays

Join The Revolution!

2010-11-08T14:02:00.000-08:00

Scaling your services can be a pain -- a big one. Using conventional cloud hosting is easier than building and managing your own infrastructure, but service scaling is still painful. At GridCentric, we think service and infrastructure scaling should be easy and goes hand-in-hand.

For example, current automated cloud scaling typically includes learning new and complex APIs along with specifically prepared server images. There are third party services you can work with – some really good ones – but the increased cost and dependence doesn’t really make things any easier to swallow. For everybody that finds this as frustrating as we do – we’re proud to announce RiotCloud: an easy-to-use, managed hosting service that makes automated scaling trivial.

RiotCloud is designed for simple deployment – users won’t need to learn new APIs, languages or have to manage/configure images; they just rent a server, change a few settings, and work away. The servers will scale automatically with the same application, run time and configuration states without you having to update or manage the system. In fact, the only thing missing is the administrative complexity. RiotCloud employs the revolutionary live-cloning technology of Copper to ensure consistent server state as you scale, exactly as you expect.

The entire team is excited and prepping for launch! How about you? Comment and let us know your thoughts/feedback – we’d love to hear them!

Join the Revolution: http://riotcloud.com/ - Register now for a free $50 credit!

How do I clone thee? Let me count the ways...

2010-10-05T11:34:00.000-07:00

Dynamically provisioning stateful VMs with Copper is fast and easy. It occurred to me that we don't often show how fast and easy it is in ways other than using the shell (because it's simple and accessible). Not everyone's workflow is bash-based, so I thought I would show a few examples of our API used with different language bindings.

Our API is powerful and fully featured, allowing for synchronization, resource reservations, and the equivalent of process control for VMs. For the purpose of these examples, I've created four different extremely simple "VM fork" programs below. Each one duplicates the running VM machine (which takes seconds with Copper) and prints the VM's id ala fork().

C/C++

/* Must be linked with -lgridcentric */
#include <gridcentric/gc-guest.h>

int main(int argc, char** argv);
    gc_uuid_t ticket;
    int vmid;
    gc_request_ticket(1, 1, 1, 1000, &ticket);
    vmid = gc_clone(ticket);
    printf("%d\n", vmid);
}

bash

#!/bin/bash
ticket=`gc rt 1 1 1 1000 | awk '{print $2;}'`
gc clone $ticket # Could have pulled the vmid from here.
vmid=`gc vmid`   # But this is much simpler.
echo $vmid

python

#!/usr/bin/env python
import gridcentric.guest

ticket = gridcentric.guest.request_ticket(1,1,1,1000)
vmid = gridcentric.guest.clone(ticket)
print vmid

java

// Be sure to include '/usr/share/gridcentric/gridcentric.jar'
// in the classpath when running this program.
import ca.gridcentric.guest.API;

class VMFork {
    public static void main(String[] args) {
        API api = new API(); 
        API.Uuid ticket = api.requestTicket(1,1,1,1000);
        int vmid = api.clone(ticket);
        System.out.println("" + vmid);
    }
}

Got it? These programs all implement fork() for VMs.

The result of running any of these programs is the same: two nearly-identical VMs, printing our two different vmid values. The operation takes seconds, and can easily be incorporated into complex workflows. It's pretty nuts. Of course, you need not limit yourself to fork with Copper. Say hello to endless possibilities for distributed and scalable applications!

GridCentric: Testing Mincemeat.py MapReduce Scalability

2010-09-21T11:47:00.000-07:00

Introduction
The MapReduce programming paradigm has been getting a lot of buzz the past couple of years because in general developers are facing larger and larger datasets. Google has shown that MapReduce is very effective at processing extremely large datasets by indexing the entire web using it. As a result of all this buzz there have also been numerous different implementations popping up. The front runner is probably Hadoop, and we have a post about how to set up a Hadoop cluster in 5 minutes. But there are many more, and in this post I am using a lightweight python implementation called mincemeat.py.

Essentially the promise of MapReduce is that we can easily throw machines at the dataset and process it faster. What I am interested in is the affect to the processing power of the system when adding additional machines. In other words, what does the performance function look like in relation to the number of machines in the system. To do this I am basically going to have a fixed dataset, and then test how fast it takes to process the dataset using a different number of worker machines.

Test Harness
Being able to dynamically vary the number of workers that I have is an interesting problem. I am a believer in the DRY principle so I don't want to have to configure 10 machines with mincemeat, or synchronize them so that the correct number of them connect when running my tests. Basically I have a very simple experiment I want to run and I want a very simple solution that allows me to easily vary the number of machines in my tests.

Fortunately the GridCentric Copper virtualization platform allows me to create an arbitrary sized cluster very easily. Copper provides a very simple, but powerful, API call that allows a virtual machine to live-clone itself. Within seconds a running virtual machine can create exact replicas of itself that have the same memory loaded, the same disk state and even the same instruction pointer in the CPU. In other words, I can use the Copper API to create clusters with a single node, 5 nodes, 10 nodes, and so on within a matter of seconds without any need for additional configuration, synchronization, or any maintenance on my part. In addition, since this is an API call I can include it directly within my test harness.

My test harness is composed of two scripts: one that will create my test cluster and execute the MapReduce program, and the other that will vary the size of the cluster and collect the time information. The first script will basically take as a parameter the number of additional workers to add. It will use the Copper API to live-clone the machine to build the cluster with the additional workers, start up the MapReduce server and then connect all the workers together. It will then clean up the cluster to free up the resources for other tests, or other applications we want to run on our cluster.

This is what my test script looks like that uses Copper to dynamically scale out a test cluster and executes my tests:

############################################################
# 
# Simple test script that will create a cluster of arbitrary
# size and then use mincemeat.py to run a MapReduce job
# on the cluster.
#
############################################################


# This will synchronize the process on the ticket and act
# as a barricade so that the process won't resume until
# everyone has synchronized on the ticket.
#
function sync_ticket {

 TICKET=$1
 IS_MASTER=$2

 if [ $IS_MASTER -eq 0 ]
 then
   gc sync $TICKET 60000
 else
   gc sync
 fi
}



if [ $# -eq 0 ]
then
  echo "Usage: $0 num_workers"
else

  # The number of additional workers to create
  WORKERS=$1
  # The IP address of the server
  SERVER_IP=`gc my-ip`

  if [ $WORKERS -gt 0 ]
  then
    # If we need additional workers then we will request a 
    # ticket for them. This reserves the resources in Copper
    # for our cluster.
    TICKET=`gc rt $WORKERS $WORKERS 1 60000 | awk '{ print $2 }'`
    if [ ${#TICKET} -eq 0 ]
    then 
      # We failed to get a ticket for that number of workers.
      # Basically the resources in our system is in use, so
      # we just exit with an EBUSY error.
      echo "Failed to acquire the resources"
      exit 16
    else
      # We have the resources so we clone out. This will create
      # all the worker machines, and they will be created as
      # exact replicas of this machine, and each one will
      # be executing this script at this point!
      gc clone $TICKET

      # The gc clone API will create the replica virtual machines.
      # Essentially we will now have new $WORKERS machine all 
      # running this script because they are clones of a virtual
      # machine that is running this script. Moreover, they will
      # all be running this script from this point onwards because
      # gc clone was the last command executed by the virtual 
      # machine prior to cloning.
    fi
  else
    # We just get the ticket that the master was created.
    # There is no need to create additional workers.
    TICKET=`gc lt | grep ticket | awk '{ print $2 }'`
  fi

  gc ismaster
  IS_MASTER=$?

  if [ $IS_MASTER -eq 0 ]
  then
    # We want to start up the server on the master
    python example.py > results.out &
    # Give the serve some time to startup
    sleep 3
  fi
  
  # This is a barricade and ensures all workers have checked
  # in before we start the process.
  sync_ticket $TICKET $IS_MASTER

  # We start up the mincemeat worker and connect to the server
  python mincemeat.py -p changeme $SERVER_IP
  
  # We have finished processing so we want to destroy the 
  # additional workers that were created.
  if [ $IS_MASTER -ne 0 ]
  then
    # We are a worker so we should join back
    gc join noreport
  fi
fi
############################################################

Results
I execute the above script using 1, 2, 3, 4, 5, 6, 7, 8, 9 and 10 hosts. For each host configuration I ran the script five times to calculate the average time it takes for this configuration. Here are my results running MapReduce using mincemeat.py on a 28 MB dataset.

The overall MapReduce time decreases as we add more workers, while at the same time the overhead and cloning of virtual machines remains constant. The time spent synchronizing the virtual machines, and requesting a ticket, is essentially zero.

We can clearly see a remarkable improvement when moving from 1 worker to 3 workers, and in fact it looks like 3 nodes is a local minimum performing fast than having either 4 or 5 workers. However, things do continue to improve and once we are in the 8 to 10 workers territory we start to see the performance improvements leveling off. Obviously this is specific to the 28 MB dataset, and I expect us to be able to get significant gains with larger number of nodes with larger data.

Conclusion
The results presented above highlight that there is a point of diminishing returns whereby adding new hosts to the MapReduce system does not provide any significant gains. I feel like this is a intuitive property of the MapReduce paradigm, but it is nice to have the data to support it. However, what is interesting is how easy it was to test this prediction. In fact, any hypothesis on distributed applications can similarly be easily tested. This is because GridCentric's Copper virtualization platform makes it extremely simple to build a cluster in which to test the distributed application, as well as provides the API so that the test harness itself can dynamically create this cluster.

The real result of this post is that Copper can open up a doorway into testing and experimenting with distributed applications that was otherwise closed due to the high cost of configuring, maintaining and synchronzing the distributed appication.

Video: installation tutorial

2010-09-14T15:03:00.000-07:00

We think that our installation tutorial is straight-forward. Hopefully it addresses most questions that come up when installing Copper. But it's lengthy. And a picture is worth a thousand words.

In that spirit, we made a video version of the tutorial! From bare-metal to cloning virtual machines in about ten minutes. Most of that is the download time! Enjoy!

Nimble Test Clusters

2010-08-30T13:00:00.000-07:00

I have been a big fan of Selenium since I first used it about 4 years ago. It is a very flexible web testing framework that has the advantage of actually running within a real browser. This means that it can do complete end-to-end testing starting with client side javascript code down to SQL queries hitting the database server, and it can also do cross browser testing to ensure that everything works between IE, Firefox, Safari, Chrome, etc. The test cases can be written in a wide variety of syntaxes, from a simple HTML table to full-fledged programming languages like Java or Ruby. By using Selenium it is very easy for any team building a web project to create automated end-to-end regression tests that can be kicked off by a Continuous Integration (CI) system whenever a developer commits. As I said before, I am a big fan of it.

Unfortunately, automated tests in general have a downside especially when a project matures. The number of tests increase causing the time it takes to run the test suite to increase. This either causes the test suites to become a bottleneck in the development cycle, or people just run them less frequently. Running tests less frequently has a compounding effect because tests start going stale, which increases maintenance cost on the tests, which causes them again to be run less frequently. Unfortunately, Selenium has a handicap in this realm because it runs within the actual web browser which inherently adds time to each test.

Fortunately, automated tests are a good candidate for parallel execution even though they are generally executed serially. This is great news because we can significantly reduce the time a test suite takes to execute by distributing it across a mini-cluster of computers. Both JUnit and TestNG have projects for distributing their execution, and there is the Selenium Grid project that will distribute Selenium tests. Basically the goal of the project is to build a small test cluster where Selenium tests can be farmed out onto different machines to greatly reduce the time it takes to execute the full test suite. It is definitely a good idea, and it will address the problem of stale tests and their associated maintenance cost. With it taking less time to run the suite, we can easily add it back into a CI system and run the tests more frequently.

Unfortunately, as I mentioned in my post on Elastic Build Systems, once we start to distribute our environments we open up a host of other problems. Namely, ensuring that all the machines in the cluster have been updated with the same version of Selenium, and Selenium Grid, that the browser versions being tested are properly synchronised, and in general be confident that the machines are identical: a test running on one machine is guaranteed to give the same results when executed on another machine. And of course the flip side of having tests run faster by distributing it across multiple machines means that there are more machines running idle for longer. Finally with a special consideration for Selenium it might be desirable to rerun the test suite across multiple browsers to regression test what is supported. With this in mind, your test cluster could start growing to include specialised Windows machines running IE7 and IE8, Linux machines running FireFox, etc. to ensure that you get good coverage, in addition to a speed boost. Of course, when adding more machines, the complexity of managing the cluster also increases, as well as the wasted resources of idling machines.

Fortunately, GridCentic's Copper high-performance virtualization platform makes managing a computer cluster extremely easy and allows a cluster to be re-purposed within seconds.

At the core of the Copper platform is the ability of virtual machines to live-clone themselves. In other words, Copper allows running virtual machines to instantly create multiple copies of themselves where each copy is an exact replica of the original machine, from the memory that has been loaded down to the instruction pointer in the CPU. This whole process of taking a single running virtual machine to saturating the physical limitation of your hardware with cloned virtual machines happens within seconds. In addition, the Copper platform also takes care of networking all of these virtual machines so that they can communicate with each other, and if desired they can also communicate with the rest of the network.

Lets suppose for example that we manage to scrounge together a little test cluster of 4 machines that we can use for running Selenium Grid to optimize our Selenium test runs. Ideally we would like to ensure that our web platform tests run using our supported configurations: Windows running IE8, OS X running Safari, and Linux running Firefox. Traditionally, we would divide up our cluster for these configurations with a single machine per configuration and one spare machine we could give to a single configuration. We have just lost the benefits of Selenium Grid and we have an extra computer that cannot be shared.

However, we can make use of Copper's ability to dynamically and quickly re-purpose an entire cluster. Each testing environment (operating system, browser, tools and Selenium Grid) will be encapsulated in its own virtual machine that will run on top of the little test cluster. When it becomes time to run the tests, the virtual machine will scale itself out using the fast live-cloning provided by Copper and then run Selenium Grid as normal. Once it has finished executing the tests, it will clean up its clone machines and scale back down freeing up the cluster resources. Then another test environment can scale out onto the recently freed resources, run the tests, and again scale back down. This can be done for each environment.

Moreover, supporting a new test environment is nothing more that creating a virtual machine for it and configuring the machine by installing the test environment software. This new configuration can now scale up, run tests using Selenium Grid, and scale back down.

In a previous post I showed how Hudson can be easily transformed into an Elastic Build System. Basically, it allowed Hudson to dynamically create new slave machines that are identical to the master machine and then farm out build jobs to them. Hudson could also destroy the slaves when they were no longer needed to free up resources. Now let's combine everything together to show how we can build an ultimate continuous integration environment using the Copper platform.

There will be a single virtual machine with Hudson installed that will be our master machine. We will continue to have a virtual machine per test environment that we want to support, and we'll configure Hudson to treat each of these machines as a slave. We'll then configure some Hudson jobs that will run on our test environment slaves that will basically clone out the test environment, start up Selenium Grid, run the tests and then scale back down. We'll also configure Hudson to have a couple of elastic slaves as described in a previous post. So now we have it. A developer commits some change, Hudson farms the build out to a dynamically created slave. Once the build is completed, it can trigger the Selenium Grid testing agents in turn so that they will scale out, run the regression integration tests, scale back down and produce a test report of any possible regression issues.

Amazing.

Slashdotted!

2010-08-11T08:41:00.000-07:00

Well, the Slashdot referral certainly stirred things up over here.

I think it's important to clarify some core points about our system and how it works. Adin's done a good job responding to some of the posts on Slashdot, but we have our hands full cutting the Copper 1.1 release and I figured it would be best to provide a quick overview of what we are up to here, so we can all get back to work :).

The memory oversubscription is one neat application of our core technology, which basically amounts to COW-based cloning of entire virtual machines. Our goal with Copper is to use this primitive to enable whole-cluster virtualization. We take a single physical cluster, and transparently run multiple virtual clusters on top of it. Each of these virtual clusters can grow and shrink independently (by cloning new VMs and killing off clones), to accomodate different demands.

Our ultimate aim is to take 'Virtual Machine Appliances' and turn them into 'Virtual Cluster Appliances': minimal-configuration disk images that are able to spin up a virtual cluster on demand, serve up a single app, and dynamically adjust resource usage based on whatever internal metrics they want to use.

To this end, we've exposed the cloning primitive inside the master virtual machine of Copper virtual clusters. The master VM, through either an API call or a shell tool (which basically wraps the API call), can invoke a 'clone' operation. For example, with the shell, it would look like this:

'gc clone 3'

This will request resources for 3 clone VMs from the Copper controller, create those clones, and then exit. The clones are exact duplicates of the master VM in terms of the state of memory and the root disk. Peripheral devices, external storage resources, and network configurations of the master can also be transparently replicated on the clones.

The semantics of the clone operation closely match those of the familiar UNIX process fork().

There are several additional API calls that can help with managing the resulting virtual cluster - e.g. API calls for listing all clones, killing individual clones or entire generations of clones, etc. Bindings have been created to expose these APIs in several languages so applications can make full use of Copper's capabilities. For instance, David had a very informative post on how to use Copper's APIs to create a dynamic build system (Elastic Build System in Action).

Copper's use of lightweight virtual machine cloning is deceptively powerful for enabling 'virtual cluster appliances'. For example, we can create a 'memcached' virtual cluster that just has memcached installed on the root disk, and can be instantly expanded or contracted in a matter of seconds, with no extra configuration steps. Same can be done for apps like Hadoop and Cassandra. Check out some of the earlier blog posts (Howto: Build and Scale a Cassandra Cluster,Howto: Build a Hadoop Cluster in 5 Minutes , Howto: Build a 10 Node Memcached Cluster) for enabling cluster appliances with Copper.

Virtualization is growing up, and it can't be just about single machines anymore. Copper's name was chosen to be reminiscent of 'cluster operating system', and that's really what we're trying to build at GridCentric.

Virtualization and over-subscription: breaking the 100% utilization barrier

2010-08-10T12:51:00.000-07:00

Something that we think about a lot at GridCentric is how to ensure that resources are used efficiently. In fact, that's what led us to create Copper in the first place -- seeing how difficult it was to share large clusters amongst multiple users, groups or even software versions. We think that the ideas behind Infrastructure-as-a-Service have huge potential to revolutionize computing.

We realize that a lot of enterprises have focused on deriving maximum efficiency from their resources for a long time. I thought that I'd take a ground-up look at some of the mechanisms used by virtualization to do that, and introduce something that we're developing for our next products in order to take resource multiplexing to the next level.

Example and demo

Our developers are hard at work on our next generation of products, which in addition to allowing instantaneous scaling and solving configuration nightmares, allows you to over-commit your physical resources (both processors and memory). This quick demo provides a great example of over-subscription, showing a development version of our platform cramming 16 gigabytes worth of clones onto a single host with 8 gigabytes of memory. Awesome.

If you're unfamiliar with Copper, each of the clone VMs created in the video is an independent machine with its own memory, disk and network initially identical to the master at the time of cloning. Each of the 16 VMs could have been allocated to a user or used to perform a specific task independently of the rest. At the end, if we look at the resources allocated by VMs on this host within our Copper installation, we see the following:

$ gc-host stat node8

                   CPU(s) 16 of 4 (400%)
                      RAM 17408 of 5631 (309%)
               Local disk 0 of 192258490368 MB (0%)
  Available Named Storage ['kv-local(10485760000)']
       Used Named Storage []
    Available PCI Devices ['VGA(0000:01:05.0)']
         Used PCI Devices []
       Available Networks ['default', 'gridcentric']
           Available Tags ['development']

To be fair, the memory shown is without the extra overhead from the management domain (2.5 gigabytes). Pessimistically, we are achieving an actual memory over-subscription of ~212%, not 309%. Still, not too shabby.

We achieve over-subscription by leveraging our novel cloning mechanism. In this post, I take a look at over-subscription in general and touch on some of the mechanisms used to support squeezing more out of your resources. For simplicity, I will focus on two crucial computing resources which are typically managed by an operating system or hypervisor: the processor (CPU) and memory. This post assumes a basic familiarity with virtualization concepts, but it will be gentle.

CPU over-subscription with multi-processing

Traditional operating systems have been over-subscribing and multiplexing resources since the days of the mainframe. Contrary to what it might seem, multiplexing a resource may actually allow you to use it far more efficiently than just giving it to a single process or user (when you consider everything that it is doing). Multiplexing CPUs with virtual machines is a primary motivator for consolidation, since CPUs are often an underutilized component on a typical enterprise server.

Suppose we have four virtual machines (VMs) that perform some work (the dark color) followed by some waiting for an I/O event (the light color). If you had an ssh connection to a VM, this might consist of processing an incoming TCP packet, the terminal and shell doing a bit of work, generating and sending a response packet (with appropriate ACK number) then waiting for the next packet to arrive.

Virtual machines are not much different than processes in an operating system. Because most processes inside each VM are I/O-bound, VMs are also generally I/O-bound as a whole (depending, of course, on the workload). This means that VMs often spend significant time waiting for I/O. If we were to just schedule just a single VM on a CPU, the execution would look much like the following.

Instead of maximizing efficiency, the CPU would actually be doing nothing most of the time. If the VMs B, C, and C were similar and we had four distinct processors (one assigned to each), we would imagine them to running as follows.

I've staggered them to show that we could actually come up with a very efficient schedule of execution on a single CPU. Since each VM spends most of its time waiting (and the CPU is not required for I/O), we could simply overlay the above schedules.

When a virtual machine doesn't have work to do, we instead run another virtual machine. When an I/O request completes, we reschedule the appropriate VM which can once again perform useful work. We make this a bit more complex with limits in how long each virtual machine can run consecutively (the quantum), priorities, interrupt handling, gang scheduling, etc. But the above is the undeniably essence of all CPU over-commit: simple time-sharing.

Memory over-subscription

Memory over-subscription is where things get interesting. There are several different approaches that we can take in order to over-commit our physical memory.

In a normal virtualization situation, VMs each have a fixed amount of memory, the sum of which is less than or equal to the total memory available on the physical machine. Memory of the physical system is divided up into pages by the hardware, which the hypervisor can arbitrarily remap (therefore the VMs do not need contiguous memory). In a non-over-committed situation, there is an injective mapping from the memory of the VMs to the pages of physical memory. This situation is shown below, with colors corresponding to VMs and the numbers corresponding to the contents of their memory.

Paging / swap

Paging is the process in which a page is taken out of memory (possibly to a swap partition or pagefile), the corresponding page table entry is marked as not present, and the page is returned to memory on-demand. Operating systems have been paging out process memory to disk for a long time. This allows you to run more processes than you have space for or a single processes that requires more memory than the physical memory you have available. Most optimization and research around paging is focused on selecting the correct pages to remove from physical memory, as the penalty incurred for unexpectedly having to bring a page back in is quite high (disks are extremely slow compared to memory).

In the below example, we see that we want to run VMs that have more memory than is available on the physical machine. In this case, several pages from VM C have been marked not present by the underlying hypervisor. If VM C is scheduled and attempts to access these pages, it will be blocked while the system fetches those pages from secondary storage. When that happens, we can assume that another page (belonging to one of A, B or C) will be paged out to disk to make room for the incoming one.

For virtualization specifically, it's useful to divide hypervisors into two different camps: type-I hypervisors run on the bare metal (e.g. VMWare ESX, Xen) while type-II hypervisors run on top of an existing operating system (e.g. VMWare Workstation, KVM, VirtualBox).

Type-II hypervisors often leverage existing operating system infrastructure in order to provide paging for virtual machines. The fact that they are running on an existing operating system means that they can leverage many pre-existing mechanisms. For type-I hypervisors, as far as I know, only VMWare's ESX products offer a paging mechanism today. Several research projects have explored adding forms of paging to Xen, but as far as I know it's still a work-in-progress in the main tree.

Paging is a useful fall-back mechanism, and does not require modifications within guest VMs. Unfortunately, it can be very expensive (in terms of a performance hit) and may suffer from poor interactions with the guest operating systems (double paging).

Ballooning

Ballooning is a very common mechanism used to support creating VMs with more memory than the host can support. In essence, ballooning simply forces VMs to share with each other by requiring that they return some of their memory to the hypervisor. This is done by dynamically inflating a balloon driver within the guest operating system, then informing the hypervisor which pages have been allocated so that they may be used by other VMs. This is a very effective mechanism, and generally does not suffer from poor interactions with guest operating systems (unless memory requirements change rapidly). However, it is not transparent to the guest operating systems -- they are aware that they are missing memory and require the special balloon driver to be installed.

An example of ballooning is shown below. The grey pages have been allocated by the balloon driver within the guest and returned to the hypervisor (we could say that the balloons have been inflated slightly within the VMs).

Page sharing

By far the neatest of the space-saving techniques, some of VMWare's products support a mechanism they call content-based page sharing. Essentially, the hypervisor continually spends a relatively small amount of CPU time crawling memory and hashing pages. If it identifies two pages with the same contents, it maps those pages to the same underlying physical page and frees one of the copies. Of course, this single copy is marked read-only and will be copied if one of the VMs needs to change the contents (copy-on-write). Thus, if you are running a lot of similiar or identical VMs, there can be significant savings.

Similiar to paging, research projects (1, 2) have explored this capability for Xen, but it has not quite made it into mainline. In practice, savings for production workloads will likely be very small. According to the few numbers I've found from VMWare, it seems to be in 5-10% for simple workloads. Windows rewrites binaries heavily, so one is unlikely to find many similar pages outside of the kernel code pages, some fixed-address DLLs (win32 probably always gets its preferred address) and pages in the buffer cache. I suspect that most savings (in whitepapers, datasheets, etc.) likely come from identification of unused zero-pages which would also be nicely gobbled up by an automated and co-operative balloon (an example -- each VM is only using about 5% of its memory). An example of page sharing is shown below, where the hypervisor has identified the common pages and remapped appropriately, leading to significant savings.

Summary

For clarity, the three approaches for over-subscription memory that I've touched upon are outlined here.

Technique	Advantage	Disadvantage	Whitepaper symptom
Paging	(Mostly) transparent. As much as you want.	Slow. Poor interactions with guest VMs.	They don't show how much swap is active.
Ballooning	Co-operative. Safe.	Guest VM modifications required. Limits guest VM memory.	Big VMs with tiny memory usage (big balloons).
Content-based sharing	Transparent. Negligible overhead.	Probably limited gains.	VMs run the same application with the same data.

Virtualization has come a long way in delivering techniques that allow organizations to get the most out their resources. It's a very tricky problem however, and there is no magic bullet. If you see 5x over-commit with real workloads, it's almost guaranteed that there's either a lot of ballooning or a lot of paging. We'll be talking more about some of these ideas over the next few months, as we put the finishing touches on our own approach to cramming in as much stuff as possible into as little time and space as we can. :)

Elastic Build System in Action

2010-08-06T12:50:00.000-07:00

GridCentric's Copper platform enables multiple diverse applications to securely, and easily, share the same underlying physical resources. Moreover, it allows these applications to effortlessly scale up to create on-demand homogeneous clusters, and to scale back down again. My last post described how this capability is ideally situated to solve the problems with Distributed Continuous Integration (CI) servers. In this post I will show how we can make Hudson, a popular CI server, aware of the Copper platform, and to use its powerful API to dynamically create identical slave machines and become an elastic build system.

Requirements

A Copper deployment (60-day free fully featured license)

A virtual cluster named elastic-hudson created in the Copper platform. If you are unfamiliar with creating virtual clusters in Copper please take some time following the tutorials.

Configure the Virtual Cluster

This section will describe all the configuration we need to do in Copper to boot the virtual machine that will host our Hudson installation. Once the virtual machine is booted, it will be able to use the Copper API to clone itself.

As mentioned earlier I have a virtual cluster already created called elastic-hudson. For the root container I downloaded the pre-configured Ubuntu image, but there are other distros available if you would prefer something else. The only special configuration for this virtual cluster is that I am going to attach a public network to it.

Whenever a virtual machine uses the clone API, the new virtual machine will automatically be configured to be apart of a private network between all of the clones and the original machine. Since Hudson is a build system, it will need access to my Mercurial respository which is hosted on a separate machine outside this private network.

To remedy this problem, I configured my virtual cluster to also have a public network. In this case, all the virtual machines in the cluster now have access to two networks: the original private network and the public network. I will be using the private network for communication between the Hudson master and the Hudson slaves, and then the public network for the machines to communicate with the repository.

That's all the configuration needed, and now I can boot the elastic-hudson virtual cluster.

Configure Hudson

Once the elastic-hudson virtual cluster is booted we should be able to log into it. By using the built-in DNS Server that comes with the Copper platform I can access the virtual cluster by simply using the domain name elastic-hudson.clusters. Otherwise, I can use the management tools to determine the virtual machine's public IP address.

Once logged into the virtual machine I will setup a vanilla Hudson install, as I would do in any other environment. So I follow the Hudson install instructions, install the tools (e.g. Mercurial, Ant, unzip, wget, make, etc.) that are required to build my project, and finally configure Hudson to build my projects. So far we have not done anything different when operating Hudson.

Now comes the interesting part: making Hudson clone slaves of itself to create an elastic build system. Access to the Copper API is controlled using the familiar Unix permission scheme. Basically a user needs to have read/write access to /proc/xen/xenbus in order to utilize the API. By default only the root user has permission to it and since Hudson runs as the special hudson user, we need to also give this user permission to it in order for the application to clone itself. There are many ways to do this, but we are just going to give everyone access to it:

root@elastic-hudon# chmod og+rw /proc/xen/xenbus

Now that the hudson user has permission to use the Copper API, the next piece of configuration is that we have to allow Hudson passwordless ssh within the private network. Simply follow this guide and use localhost (or 10.0.0.1) for the remote machine as the hudson user. This should enable the hudson user to ssh throughout the private network without the need of a password.

Finally we are ready to finish off our setup by configuring a slave within Hudson. Log into the Hudson web interface (e.g. http://elastic-hudson-0.clusters:8080) and go to Manage Hudson -> Manage Nodes -> New Node. Enter the name of the slave (e.g. clone-slave) and select the dumb slave option and finally click ok.

In reality the only thing special about this configuration is that I've specified a custom launch script called /var/lib/hudson/clone_slave_launcher.sh. This is the script that Hudson will call when establishing a connection to the slave machine. The first thing we do in this script is use the Copper API to clone the Hudson machine, then we determine the clone's private network IP address, and finally proceed as normal to establish a connection to the newly created slave machine.

Before revealing the script the other interesting configuration is Hudson's slave availability. I have set it so that Hudson will connect to a slave (or in our case create one) if there is a pending job in the queue for more than a minute. It will then disconnect from the slave (or in our case destroy the machine) if the queue becomes empty for more than a minute. In other words, Hudson will automatically scale up its footprint when there are pending jobs it needs to complete, and then automatically scale down, freeing up resources for other users (potentially other Hudson system for different teams), once it no longer needs that many resources.

Here is the script that makes Hudson aware of the Copper platform, and performs the magic to make it into an Elastic Build System:

#!/bin/bash
###################################################################
# This script can be used with the Continuous Integration server
# Hudson to allow it to use the GridCentric Copper platform to
# create on-demand slaves that are preconfigured to be identical to
# the master, at the time the slave is created.
#
# It will also destroy the slaves once Hudson has finished with it
# and turns it offline.
###################################################################

# The SSH command. We are not doing strict host checking because
# these are clone machines on a private network. We are sure there
# will be no man in the middle attacks.
SSH="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"

# The amount of time (in seconds) the master should wait before
# connecting to the slave. By default it is 5 seconds, but can be
# passed as the first argument to the script.
WAIT_BEFORE_CONNECT=5
if [ $# == 1 ]
then
WAIT_BEFORE_CONNECT=$1
fi

# Acquire a ticket from the GridCentric Copper platform. The ticket
# basically reserves resources (CPUs, Memory, etc.) on the physical
# cluster to host our clone.
echo "Acquiring ticket for resources to clone on to..."
TICKET=`gc rt 1 1 1 60000 | awk '{ print $2 }'`

if [ ${#TICKET} == 0 ]
then
echo "Failed to acquire ticket in 1 minute."
# exit with an EBUSY signal
exit 16
else
# We reserved the resources, so now we will clone using the
# ticket. This is the API call that allows this virtual machine
# to grow its footprint.

echo "Cloning on ticket $TICKET..."
gc clone $TICKET

# The clone operation is kinda like fork(), afterwards there will
# be 2 copies of this script running. One on the master virtual
# machine (this one) and another on the clone virtual machine.
# They will both be at the instruction after the clone (i.e. here),
# so we do a quick check to see if we are the master or clone and
# then execute differently.

gc ismaster

if [ $? == 0 ]
then

# This is the master's task and will be executed on the same
# machine that the Hudson instance is running. Basically it
# will determine the clone's private network's IP address, wait
# a little bit of time for the clone to take care of its setup,
# and then SSH to the clone and run the Hudson slave jar.
gc ltd $TICKET
IP_ADDRESS=`gc ltd $TICKET | grep ip | awk '{ print $2 }'`

echo "Connecting to clone on $IP_ADDRESS after waiting for $WAIT_BEFORE_CONNECT seconds..."
sleep $WAIT_BEFORE_CONNECT

# This command is what lets Hudson communicate with the slave.
# Basically it uses the stdin and stdout for communication.
$SSH $IP_ADDRESS java -jar /var/run/hudson/war/WEB-INF/slave.jar

else

# This is the slave's task that will be executed on the clone
# machine. It first needs to kill the Hudson process to ensure
# no conflicts, and then it will sit and poll to see if the
# slave.jar job is done. Hudson just kills the script on the
# master, so we can't assume we'll get a signal from it. So, we
# just poll and once the slave.jar stops executing, we
# 'gc join' back and kill ourselves.

# Kill any running Hudson instance
HUDSON_PID=`ps aux | grep hudson.war | grep java | awk '{ print $2 }'`
kill $HUDSON_PID

# Kill any running SSH connection to another slave. Remember,
# this is a clone of the master virtual machine, so any process
# that was running there is also running here.
SLAVE_PIDS=$(`ps aux | grep slave.jar`)
kill $SLAVE_PIDS

# Just wait a bit for things to happen
sleep 60

# Now we periodically check if slave.jar is still running. If
# it is, we are good. Otherwise, we are finished.
while true;
do
SLAVE_PID=`ps aux | grep slave.jar | grep java | awk '{ print $2 }'`
if [ ${#SLAVE_PID} == 0 ]
then

# Slave process has gone. We join, but since the master is
# not waiting for us to join, we join with 'noreport'. In
# other words, this virtual machine will die and the master
# will get no report.
gc join noreport
else
wait $SLAVE_PID
fi
done

fi
fi

Here is what it looks like in Hudson when we launch our new slave.

Finally, we can add as many of these slaves to Hudson as we like, using the exact same configuration, if we need more machines in the cluster. These machines will never sit idle because Hudson will create them only when it needs them, then destroy them when it is done with them, and they will be guanteed to be properly configured because they are exact replicas of the master machine at the instance they are created.

Howto: Build and scale a Cassandra cluster in five minutes

2010-08-05T16:36:00.000-07:00

Introduction

In the spirit of the software recipes we've been posting recently, I decided to give Cassandra a shot. Cassandra is a distributed key-value store that was open-sourced by Facebook in 2008 and is now under the umbrella of the Apache foundation. It advertises high-performance and robustness to individual node failures while providing eventual consistency. It's been receiving quite a bit of attention recently, and Riptano has appeared to provide commercial support.

Cassandra is a bit of an odd name for a piece of software in my opinion. Cassandra is a figure from Greek mythology, who was both blessed with the ability to see the future and the curse that no one would ever believe her predictions. I struggled for a while to come up with some hilarious software-equivalent joke, but I think we are all better off without one.

I spent a bit of time and installed Cassandra on a Copper virtual cluster. I then wrote a few quick scripts to automatically have clones join a Cassandra cluster when they are created automatically. GridCentric Copper supports persistent local storage through an awesome VFS abstraction. As with Hadoop, I've omitted this part of the setup for now to simplify this post, but I reserve the right to make that post in the near future.

Cassandra does not have different classes of nodes, so I had no need to run anything special on the master of the virtual cluster. Cassandra only requires a seed node, so that a freshly started instance can learn about the others. I use the master for this purpose, since we can assume that it's always around. Other than that, this article details an unmodified Cassandra installation running inside a virtual cluster -- with transient nodes that can be created and destroyed in seconds.

Requirements

A GridCentric Copper deployment (free trial available here, software downloadable here).

Just as in my Hadoop post, I'll assume that you have a virtual cluster already. If you don't know how to create a virtual cluster, you can see the tutorials. I just used the cluster-in-a-box script to fire up a new Ubuntu cluster.

Installing Packages

Cassandra needs a Java6 JRE installed and configured. In Ubuntu, I just did an apt-get install sun-java6-jre, but the exact steps will vary depending on your distribution. The next step is to grab the latest Cassandra packages. Choose a mirror from
http://www.apache.org/dyn/closer.cgi?path=/cassandra/0.6.4/apache-cassandra-0.6.4-bin.tar.gz. Once I had downloaded the latest Cassandra, I extracted the contents and moved the directory to /opt, so I could make a nice little system that corresponds to the Linux Standard Base (or at least closer).

wget <url>
tar -zxvf apache-cassandra-0.6.4-bin.tar.gz
mv apache-cassandra-0.6.4 /opt

Cassandra is actually ready to go with the default configuration and your Java install can be tested at this point. Simply run /opt/apache-cassandra-0.6.4/bin/cassandra -f and you should see it start up. You can press Ctrl-C to kill it, because we want to write a few more scripts and make a cluster.

Tweaking files and writing Scripts

Okay, so one annoying thing about Cassandra is that you need to specify an address inside the configuration file for binding. This doesn't lend itself well to any kind of cluster environment and it actually confused me -- an interface would make much more sense here. We need to abstract away this detail so we can tune the bind address on the clone. In order to do this, we copy /opt/apache-cassandra-0.6.4/conf/storage-conf.xml to /opt/apache-cassandra-0.6.4/conf/storage-conf.xml.in and change the following keys in our new storage-conf.xml.in:

...
  <Seeds>
      <Seed>127.0.0.1</Seed>
  </Seeds>
  ...
  <ListenAddress>localhost</ListenAddress>
  ...

to:

...
  <Seeds>
      <Seed>10.0.0.1</Seed>
  </Seeds>
  ...
  <ListenAddress>PRIVATE_ADDRESS</ListenAddress>
  ...

I changed the Seed key because in our virtual cluster I am going to use the master as the seed. I changed the ListenAddress to PRIVATE_ADDRESS because we are going to write a quick script that replaces that token and generates a unique storage-conf.xml on each clone with the correct addresses.

In order to write that script, I need a quick helper. First, I create a script named get-private-ip and put it in /usr/local/bin.

#!/bin/bash
ifconfig eth0| grep inet | cut -d: -f2| cut -d' ' -f1

Next, I create
recreate-cassandra-config in /usr/local/bin to actually grab the IP we want, and recreate storage-conf.xml from our tokenized storage-conf.xml.in.

#!/bin/bash
cd /opt/apache-cassandra-0.6.4
address=`get-private-ip`
cat conf/storage-conf.xml.in | sed -e "s/PRIVATE_ADDRESS/$address/" > conf/storage-conf.xml

Finally, because I am not going to be using persistent local storage (saving for a future post), I create a script clear-cassandra-data in /usr/local/bin in order to clear the data directory. We will run this on clones before kicking Cassandra.

#!/bin/bash
rm -rf /var/lib/cassandra/data/*

I'm not a fan of hacking ways of starting and stopping services. Before running my mini-Cassandra cluster, I threw together a script that lets me start and stop nicely (at least somewhat nicely). I put this in /etc/init.d/cassandra and then run update-rc.d cassandra defaults in order to install the appropriate symlinks from /etc/init.d/rc*.d.

#!/bin/bash
#
# Simple Cassandra init script.
#
# chkconfig: 345 99 99
# description: Cassandra

### BEGIN INIT INFO
# Provides: 
# Required-Start: $network
# Required-Stop: 
# Should-Start: 
# Should-Stop: 
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Short-Description: start and stop cassandra daemon
# Description: Cassandra
### END INIT INFO

PROG="/opt/apache-cassandra-0.6.4/bin/cassandra -f"
PROGNAME="Cassandra"
LOGFILE=/var/log/cassandra.log
PIDFILE=/var/run/cassandra.pid

running(){
    PID=`cat $PIDFILE 2>/dev/null`

    # Check that the pid is sane.
    if [ "x$PID" == "x" ] ; then
        return 1
    fi

    # Check that the process is alive.
    ps $PID >/dev/null 2>&1 || return 1

    # Looks okay.
    return 0
}

start(){
    echo -n $"Starting $PROGNAME: "

    # Try to start the program.
    if running; then
        echo "Failed.  Maybe remove $PIDFILE?"
        return 1
    fi

    mkdir -p `dirname $LOGFILE`
    $PROG > $LOGFILE 2>&1 &
    PID=$!
    mkdir -p `dirname $PIDFILE`
    echo $PID > $PIDFILE

    echo "Success."
    return 0
}

stop(){
    echo -n $"Stopping $PROGNAME: "

    # Check if it's already stopped.
    if ! running ; then
        echo "Failed.  Already stopped."
        return 1
    fi 

    # Find the PID and kill it.
    PID=`cat $PIDFILE 2>/dev/null`
    if [ "x$PID" == "x" ] ; then
        echo "Failed."
        return 1
    fi
    # (Try five times to kill it).
    for i in `seq 0 5`; do
        kill $PID
        sleep 1
        if ! running ; then
            break
        fi
    done

    # Check if it is finished.
    if running ; then
        echo "Failed."
        return 1
    fi

    # Clear out the pidfile.
    echo "Success."
    rm -f $PIDFILE
    return 0
}

restart(){
    stop
    start
}

status(){
    echo -n $"Status of $PROGNAME: "

    if running ; then
        echo "Running."
        return 0
    else
        echo "Not running."
        return 1
    fi
}

# See how we were called.
case "$1" in
    start)
 start
 RETVAL=$?
 ;;
    stop)
 stop
 RETVAL=$?
 ;;
    status)
 status
 RETVAL=$?
 ;;
    restart)
 restart
 RETVAL=$?
 ;;
    *)
 echo $"Usage: $0 {start|stop|status|restart}"
 RETVAL=2
esac

exit $RETVAL

Tying everything together, I create a simple post-clone script that will stop Cassandra, reconfigure, reset the data directory, and start Cassandra. This I will put in /etc/gridcentric/post-clone-on-clone.d/cassandra so that it runs on every clone immediately after creation. Recall that we set the Cassanda seed to point to the master, so after cloning each clone will automatically join the ring and discover the others.

/etc/init.d/cassandra stop
recreate-cassandra-config
clear-cassandra-data
/etc/init.d/cassandra start

Creating the cluster

We're finally ready to create the cluster! To start a single instance of Cassandra on the master, we simply run:

/etc/init.d/cassandra start

Now I am free to use nodetool to query Cassandra and start storing data, etc. As an example, you can run /opt/apache-cassandra-0.6.4/bin/nodetool -h localhost ring. This shows the current nodes that are in the Cassandra cluster (should be just one). In order to automatically grow this cluster, we simply clone this virtual machine. Here, I'll create a cluster of size 5:

gc clone 4 # Creates a cluster of size 5.

Now check out /opt/apache-cassandra-0.6.4/bin/nodetool -h localhost ring!

Check out the video below. I start just before starting Cassandra and creating the cluster. I show the ring continuously after starting Cassandra. This is all in real-time (except when I pause, so you can read the labels!), including the cloning of virtual machines.

Update: Sorry for the somewhat poor quality of the video. I'll see if I can get a higher quality version uploaded soon so that it's clear what's going on. Note that you can click on the video and it will take you to Youtube.

Elastic Build Systems

2010-08-04T15:20:00.000-07:00

I am a firm supporter of Continuous Integration (CI) and throughout my development career I have used CruiseControl, TeamCity and finally at GridCentric we are using Hudson. I am actually really proud of the setup that we have going on here. In addition to the usual tasks Hudson performs, such as compiling our source, running unit tests or packaging our distribution, we are also creating up-to-date documentation of both the JavaDocs and the database schema. It is really cool to have your database schema document automatically updated whenever you push a change set.

Eventually as the project grows -- more sub-projects being supported by the CI server, more people committing, more tasks being done on each commit -- it will start to become too much for a single machine to handle. TeamCity supports distributing the build over multiple machines using what it calls agents, and Hudson does something similar with its notion of slaves.

Great! We distribute the CI server and we can grow indefinitely living in our happy world of Continuous Integration.

Wait, one second...

Instead of having a single machine that is easy to update we now have a handful of machines. Each one in isolation is easy to manage, but now we have to worry about ensuring they are kept in sync. Suppose we realize we need a new version of Ant, or our project's Maven settings.xml changes. Instead of simply modifying these changes in one spot, we now need to log into each machine in our build cluster and modify them. That doesn't sound like fun, it goes against my DRY philosophy and I can foresee some things falling through the cracks every now and then.

Hudson even comes with this warning:

Also note that the slaves are a kind of a cluster, and operating a cluster (especially a large one or heterogeneous one) is always a non-trivial task. For example, you need to make sure that all slaves have JDKs, Ant, CVS, and/or any other tools you need for builds. You need to make sure that slaves are up and running, etc. Hudson is not a clustering middleware, and therefore it doesn't make this any easier.

Things only start to look worse when you go from a single team, to view a company with multiple teams all trying to keep their mini-build clusters synchronized. The probability that something will go overlooked increases, and at any given time probably half of those machines are just idling waiting for a build job. This is a waste of resources -- both in terms of physical machines consuming power doing nothing, and in developer's time tracking down build issues because one machine wasn't upgraded properly.

Fortunately, GridCentric's Copper Virtualization Platform makes managing multiple homogeneous clusters extremely easy.

Copper provides a very simple, but powerful, API call within the virtual machine: clone. Within seconds the virtual machine can create multiple running replicas of itself at the state just prior to being cloned. In addition, the clone virtual machines are automatically networked with the original virtual machine on a private network only visible to them. In other words, by using the clone API call we can very easily create a homogeneous cluster based on the configuration of a single machine within seconds.

Need to make a configuration change to your build cluster, such as manually installing a library into the Maven repository, or installing an interesting Ant plugin? Simply destroy your existing cluster (a couple of seconds), perform the configuration changes on your original virtual machine, then clone the cluster back into existence (just a couple of seconds). Keeping your build cluster's configuration in sync could not be easier, or faster.

Of course Copper gives this powerful cloning ability to any virtual machine booted into the platform, enabling it to support multiple build clusters. In fact Copper makes consolidating all the mini-build clusters within an organization onto the same physical hardware possible and simple.

If a new CI server is required, simply boot up a virtual machine in Copper and install the server on it. When it becomes time to distribute the server, clone the virtual machine running it, and its footprint will grow without any additional configuration.

The next piece of the puzzle is to teach the Continuous Integration servers about this powerful clone API call so that they can automatically scale themselves up into a distributed system when needed, but can also scale back when running idle. Once this piece is solved we'll have a truly remarkable CI server and there will be no excuse about lacking resources for every team within an organization to be doing Continuous Integeration.

Howto: Build a Hadoop cluster in five minutes

2010-07-30T11:38:00.000-07:00

One of the key benefits of Copper is that it allows very different applications to easily and securely share physical resources. We have standard packages that make grid queueing and MPI integration dead simple, but I figured it might be valuable to show off a newer paradigm, like Hadoop. This continues what Kannan started, demonstrating how Copper makes deploying real-world services simple.

Hadoop is quickly gaining traction as a powerful tool for data analysis. A key benefit of Hadoop is that it integrates data management and a map-reduce engine, so that analysis and data processing jobs can be scheduled and performed in a data-aware fashion (e.g. computation goes to where the data is). This post will show a relatively simple setup, without integrating Copper's local storage containers (I'll reserve the right to do this step in a future post! :).

Requirements

A GridCentric Copper deployment (free trial available here, software downloadable here).

I'll assume that like me, you have a virtual cluster named 'hadoop'. If you don't know how to create a virtual cluster, you can see the tutorials. I just used the cluster-in-a-box script to fire up a new Ubuntu cluster.

Install Cloudera Hadoop Packages

The first step is to jump in to our virtual cluster (using gc-vm console or via ssh) and install the Cloudera Hadoop distribution. I'm using Ubuntu Jaunty, so I'll be following the Cloudera instructions for debian-based distributions (it's a small delta for the RPM-based ones). First, we'll edit /etc/apt/sources.list:

Make sure that the Ubuntu repositories include universe and multiverse.
Add the Cloudera repositories:

deb http://archive.cloudera.com/debian jaunty-cdh3 contrib

Next, we'll install all the Hadoop packages (you may have to type 'Y' to accept unsigned packages or you can go through the Cloudera instructions and install the key for their repository).

sudo apt-get update
sudo apt-get install hadoop-0.20-conf-pseudo

That was easy! Hadoop is installed and ready to run in pseudo-distributed mode. Don't run it just yet though. Psuedo-distributed mode involves all the services running on a single node. Although that's a good start, I really want to run it across dozens of machines in the next couple of minutes.

Tweak Configuration Files

Before we edit files to create a fully distributed Hadoop, we need to know a few things. Each Hadoop cluster requires exactly one namenode and one jobtracker. These services co-ordinate the activities of the datanodes and tasktrackers, which host storage and co-ordinate computation respectively. In our setup, the virtual cluster master will host the namenode and jobtracker, and the clones will each run a datanode and a tasktracker. We will automate this so that whenever we create new clones, they will just start the appropriate services and join the Hadoop cluster!

First I'm going to edit /etc/hadoop/conf/mapred-site.xml to ensure that the tasktracker that runs on the clones always talks to the jobtracker on our master. In our configuration, our master always gets the address 10.0.0.1 on its virtual cluster private interface. So, in mapred-site.xml, I simply changed localhost:8021 to 10.0.0.1:8021. It now reads:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>10.0.0.1:8021</value>
</property>
</configuration>

Second, I will do something similiar for the datanodes. Because we will run the namenode on the master of the virtual cluster, we need to edit /etc/hadoop/conf/core-site.xml so that the datanodes on the clones talk back to the master. As seen in my new core-site.xml below, I changed hdfs://localhost:8020 to be hdfs://10.0.0.1:8020:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://10.0.0.1:8020</value>
</property>

<property>
<name>hadoop.tmp.dir</name>
<value>/data/hadoop/${user.name}</value>
</property>
</configuration>

In the above file, I also changed the "hadoop.tmp.dir" to be "/data/hadoop/${user.name}". This is really a move for later, when I hook in Copper local storage containers. For now I'll just create this directory and make sure that the permissions are correct.

root@ubuntu# mkdir -p /data/hadoop/hadoop
root@ubuntu# chown hadoop:hadoop /data/hadoop/hadoop

Start Services

Finally, before starting the namenode and jobtracker on the master, I need to format the files used for the namenode.

root@ubuntu# su - hadoop -c "hadoop namenode -format" 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ubuntu.gridcentric.ca/192.168.1.76
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2+320
STARTUP_MSG:   build =  -r 9b72d268a0b590b4fd7d13aca17c1c453f8bc957; compiled by 'root' on Mon Jun 28 23:15:26 UTC 2010
************************************************************/
Re-format filesystem in /var/lib/hadoop-0.20/cache/hadoop/dfs/name ? (Y or N) Y
10/07/30 18:09:29 INFO namenode.FSNamesystem: fsOwner=hadoop,hadoop
10/07/30 18:09:29 INFO namenode.FSNamesystem: supergroup=supergroup
10/07/30 18:09:29 INFO namenode.FSNamesystem: isPermissionEnabled=false
10/07/30 18:09:29 INFO common.Storage: Image file of size 96 saved in 0 seconds.
10/07/30 18:09:29 INFO common.Storage: Storage directory /var/lib/hadoop-0.20/cache/hadoop/dfs/name has been successfully formatted.
10/07/30 18:09:29 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ubuntu.gridcentric.ca/192.168.1.76
************************************************************/

At last, I start-up the namenode and jobtracker on the master.

root@ubuntu# /etc/init.d/hadoop-*-namenode start
root@ubuntu# /etc/init.d/hadoop-*-jobtracker start

Now I am running the services that will stay on the master. On each of the clones that I create, I would like to have a tasktracker and a datanode running. In order to automate this from the begining, I simply install a post-clone hook by creating the file /etc/gridcentric/post-clone-on-clone.d/hadoop with the following contents and making sure that it is executable.

/etc/init.d/hadoop-*-namenode    stop
/etc/init.d/hadoop-*-jobtracker  stop
/etc/init.d/hadoop-*-datanode    start
/etc/init.d/hadoop-*-tasktracker start

Creating Your Cluster and Running Jobs

I am ready to create my hadoop cluster. To create four clones running datanodes and tasktrackers, I simply run:

root@ubuntu# gc clone 4

I wait for 30 seconds or so for the datanodes and tasktrackers to co-ordinate with the master and then I run sample Hadoop applications!

root@ubuntu:~# hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar pi 2 100000
Number of Maps  = 2
Samples per Map = 100000
Wrote input for Map #0
Wrote input for Map #1
Starting Job
10/07/29 22:31:21 INFO mapred.FileInputFormat: Total input paths to process : 2
10/07/29 22:31:21 INFO mapred.JobClient: Running job: job_201007292045_0008
10/07/29 22:31:22 INFO mapred.JobClient:  map 0% reduce 0%
10/07/29 22:31:51 INFO mapred.JobClient:  map 100% reduce 0%                      
10/07/29 22:32:12 INFO mapred.JobClient:  map 100% reduce 100%
10/07/29 22:32:22 INFO mapred.JobClient: Job complete: job_201007292045_0008
10/07/29 22:32:22 INFO mapred.JobClient: Counters: 18
10/07/29 22:32:22 INFO mapred.JobClient:   Job Counters
10/07/29 22:32:22 INFO mapred.JobClient:     Launched reduce tasks=1
10/07/29 22:32:22 INFO mapred.JobClient:     Launched map tasks=2
10/07/29 22:32:22 INFO mapred.JobClient:     Data-local map tasks=2
10/07/29 22:32:22 INFO mapred.JobClient:   FileSystemCounters
10/07/29 22:32:22 INFO mapred.JobClient:     FILE_BYTES_READ=50
10/07/29 22:32:22 INFO mapred.JobClient:     HDFS_BYTES_READ=236
10/07/29 22:32:22 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=170
10/07/29 22:32:22 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=215
...

To grow my virtual cluster, I can simply clone again. When I am finished, I run gc killall to remove my clones and free up the resources for someone else. That was pretty easy!

Howto: Build a ten node memcached cluster in five minutes

2010-07-28T13:55:00.000-07:00

Introduction

Memcached is used to cache everything from dynamically generated web pages to database query results. Multiple memcached instances are used in situations where a single memcached server does not have enough CPU or RAM to fulfill application requirements.

In a distributed setup, memcached servers run independently of each other, and have no knowledge of other servers. Memcached's design pushes the logic of dealing with multiple servers to the client, which makes things very simple on the server end.

GridCentric Copper is a bare-metal-up virtualization and management stack built to manage "virtual clusters". Copper's fast virtual machine cloning (which works similarly to fork() in UNIX, except it creates clones of entire running virtual machines in just a few seconds) can be used to quickly expand and contract a collection of memcached servers. We're going to use this to make a single memcached virtual machine, then grow that machine to a pool of 10 virtual machines in a couple seconds.

Requirements

GridCentric Copper (free fully featured, 60-day trial available here, software downloadable here).
The GridCentric DNS mapper (part of the Copper distribution).
Enough free hardware resources to host 10 virtual machines.

Process

Create a New Virtual Cluster

First, create a new virtual cluster using either the included "cluster-in-a-box" script or via the Admin Web Console. In this walkthrough we will name our virtual cluster "memcached". Make sure to give the virtual cluster a managed public network interface, and make sure the "master only" checkbox is unchecked.

Let's boot the memcached virtual cluster.

gc-vc boot memcached

This will bring up a single virtual machine - a cluster of size 1. This virtual machine will be the basis of our memcached cluster, so we need to set it up. We get a console on the new master with:

gc-vm console memcached-0

This will give us a login prompt on the master.

Install memcached

We install memcached according to the README and configure it to listen on all addresses (edit the /etc/memcached.conf file and change '-l 127.0.0.1' to '-l 0.0.0.0'), and then start it up.

We now have a Copper virtual cluster, which for now contains a single virtual machine running memcached.

Create some clone virtual machines

To start up more memcached instances, we invoke the gc command line tool from within the virtual machine to create some clones of itself:

[ubuntu]$ gc clone 9

After a few seconds, the clone command returns, we have now have 10 virtual machines running memcached.

Note: we just scaled 1 memcached instance to 10 with zero extra configuration, in just a few seconds!

Configure the client address pool

Now we have to let the memcached clients know about the servers so that the clients can add the servers to their pool.

One easy method is just to use the GridCentric DNS service. This service runs on any computer with access to the Copper head node, and does dynamic mapping of DNS lookups to virtual clusters within Copper.

One of its particularly nice features is that it maps the name of a running virtual cluster to the list of all public managed IPs on that virtual cluster.

On some machine which has access to the GridCentric DNS service (let's say it's running on host gridcentric-headnode), we execute the following:

$ dig +short memcached @gridcentric-headnode
192.168.1.144
192.168.1.142   
192.168.1.143   
192.168.1.140   
192.168.1.146   
192.168.1.141   
192.168.1.145   
192.168.1.149   
192.168.1.147   
192.168.1.148

This returns a list of public IPs for all 10 of the memcached servers running within the virtual cluster. You could even design your web application to automatically query this DNS server and periodically update its view of the set of servers running.

That's it! You're now ready to start using memcached on your client nodes.

Homework:

Try to accomplish the above on a traditional cluster setup ;)

A new grid computing paradigm explained

2010-04-09T10:11:00.000-07:00

Copper represents a new paradigm for grid computing.

Why? Copper flips some of the standard practices in grid computing on their head. I'll illustrate this by starting with a typical cluster environment.

(click for larger version)

In this environment, users log in to a cluster headnode (or login node) and submit jobs (as scripts) to a scheduler that copies those scripts to several pre-configured slave nodes (a.k.a. compute nodes).

The software environment that runs on the headnode must be carefully crafted by an administrator to meet all of the users' needs and accommodate each application that they may want to run. Note that users do not have any administrative control over the cluster in this context, so they may not freely install new software. This leads to users building software in their personal home directories, using large, statically-linked binaries and often banging their head against the wall.

Additionally, administrators must ensure that the slave nodes are kept in sync with the headnode. If a new software package is installed on the headnode, it must be installed on each of the slaves. If a configuration or library changes, then it must be changed on each of the slaves.

A Copper computing environment is fundamentally different.

(click for larger version)

In this case, users may each be given a virtual headnode on which they have complete administrative control. This means that they can freely change the configuration and install new software. They are also free to use the distribution and tools that they are familiar with (and possibly have on their workstations).

When jobs are submitted to a queue in a Copper cluster, slave nodes are created on the fly, which are exact clones of the headnode at the time. This means that the user does not need to worry about updating software on any slaves nodes, as everything will be exactly as the same as it was on their virtual headnode. If their software ran on the headnode, then it is guaranteed to run on their slave nodes. When the jobs complete, the slave nodes are automatically destroyed and their associated resources are freed.

How is this possible? Our technology allows virtual machines to clone nearly instantaneously and with almost zero overhead. Using this cloning for grid computing means that you don't have to worry about keeping environments in sync. Ever.

In fact, the above diagram is not really accurate. Scripts are not copied to the slave VMs, because they are already there. Because the slaves are guaranteed to up-to-date from the point when you actually submitted the job, they can just run a command that you specify instead of having to copy a script. That's fundamentally cool; it's like we copy the whole headnode whenever you run a job.

As an example, this will run flawless on Copper, but not on traditional clusters:

apt-get install python     # Install python.
vim myprogram.py           # Edit my python program.
gcqsub python myprogram.py # Submit my python program to the queue.

In the above, I install python, edit a program and run it through the grid queue in steps that are just are natural as using my workstation.

GridCentric Inc. Announces Copper™ Cluster Management Software

2010-04-08T07:19:00.000-07:00

Copper™ Combines Cloud and Grid Computing Technologies to Provide Unprecedented Ease-of-Use and Flexibility

Toronto, Canada – April 8th, 2010 – GridCentric Inc., a Toronto-based software company, today announced the availability of Copper™, a cluster management system for high performance computing workloads. Copper™ combines virtualization and grid computing technologies to enable simple, efficient and flexible cluster deployment, management, and use.

The GridCentric Copper™ platform enables real-time, on-demand sharing of high performance computing resources and makes cluster administration a one-click task. With Copper™, organizations reduce IT expenditures through significantly lower installation and management costs, higher utilization of their physical infrastructure, and improved sharing of their data assets.

Operators of large clusters face enormous challenges due to system complexity – compute clusters are typically composed of hundreds or even thousands of individual computers, each requiring separate software components for hardware provisioning and management, resource allocation, job scheduling, and application support. Initial setup of a compute cluster can take several months because of these challenges.

By combining technologies from cloud and grid computing, GridCentric’s Copper™ platform provides server provisioning, resource management, job control, and high-level application support in a single integrated software package. With Copper™, the time required to setup and configure a compute cluster goes from months to days. Additionally, Copper™ has built-in support for operating hundreds of “virtual clusters” on the same set of physical resources, scaling their compute footprints on-demand in real time – the industry’s first true high-performance cloud computing platform. Copper™ gives cluster users the power of a supercomputer with the ease-of-use of a PC.

“GridCentric's approach to cluster management will enable many organizations to service users in ways that would have otherwise been impractical or impossible. Our experiences with it to date have been very positive. Copper™ is the way of the future for many organizations,” said Professor Michael Bauer of the University of Western Ontario, and Associate Director of the SHARCNET academic supercomputing consortium.

Copper™ has been in limited trial on a SHARCNET compute cluster located at York University in Toronto, Canada since late 2009, and is now open to all researchers from SHARCNET’s 17 academic member organizations.

“GridCentric’s Copper™ product represents a new class of cluster management software,” said Tim Smith, co-founder and CEO of GridCentric Inc. “Systems integrators, cluster operators, and end-users will all benefit from Copper’s unprecedented ease of installation, management, and use.”

Availability and Pricing

Copper is immediately available through authorized systems integrators and resellers at a list price of $1250.00 per compute node. Volume and Academic discounts may apply. For more information, please visit www.gridcentriclabs.com/copper or e-mail sales@gridcentriclabs.com.

About GridCentric Inc.

GridCentric Inc. is a technology-leading systems, networking, and virtualization software company. It is the mission of GridCentric to make high performance computing easy, without sacrificing performance and flexibility. GridCentric is a privately held corporation, and is funded in part by Rogers Ventures. GridCentric was recently named as one of the “Top 25 Canadian ICT Up and Comers” by the Branham Group. For more information, please visit www.gridcentriclabs.com.

About SHARCNET

SHARCNET was established in 2001. It is one of seven world-leading Compute Canada (www.computecanada.org) supercomputing consortia. SHARCNET currently serves 14 universities, 2 colleges, and one research institute across western Ontario. For more information, please visit www.sharcnet.ca.

Grid Queueing, The Copper Way

2010-02-04T07:10:00.000-08:00

So let's talk about grid queueing. Before I wax poetic about how GridCentric's cluster operating system makes grid queueing better, it's probably a good idea to start with a delineation of what the current state of grid queueing is, and some of the problems we're trying to solve for people who use it.

Conceptually, grid queueing is simple: you have a cluster, and you want to utilize the free compute nodes in the cluster to execute programs. So you install a grid queueing engine, which is aware of all the computers in your cluster, and you submit your jobs to it. The queueing engine figures out what computers are capable of executing your jobs, what other jobs have been submitted by other people, prioritizes them according to its configuration, and communicates with the hosts on the cluster to run them. It monitors and stores the execution and state of these jobs and reports them to the user as requested.

Pretty straightforward, yes? Well.. conceptually yes.. but the devil, as always, is in the details.

The Setup

Let's say Uma the user, an analyst, has three types of heavyweight programs she uses routinely:

1. A reporting program that analyzes logs and generates and XML reports. This program is written in python and uses a wide set of python libraries, including bindings to a C XML library.

2. A JDBC client that connects to her company's database, retrieves a dataset, and performs a long-running analysis on it, storing the results back into the database.

3. An internal C++ application that takes in raw input data collected from surveys and stored as flat files on a network filesystem, and prepares them for entry into the database.

The Old Way

Let's assume we already have a traditional, non-Copper, cluster, installed and configured appropriately with operating systems, networking, etc.

What does the cluster administrator need to do to let Uma run her jobs on the cluster?

1. He needs to install and configure the 3 different applications on all the machines in the cluster. This might be straightforward for one or two machines, but tedious once you get up to dozens, let alone hundreds or thousands of machines. They all need Python, the appropriate Python libraries, java, the appropriate JDBC adaptors, the c++ program, etc.

2. He needs to be wary of conflicts that Uma's tools might have with some other users' tools. He needs to ensure that if another user already has a tool on a host that requires, say, an XML library that conflicts with Uma's requirements, then Uma's tools don't get installed on that machine.

3. He needs to ensure that the network filesystem that Uma's tools access is configured at the same path on all the machines in the cluster.

4. He needs to add a queue to the queueing engine specific to the machines capable of executing Uma's jobs.

5. Even after all this, when Uma actually submits her jobs, she needs to be careful to send it to the appropriate queue as configured by the administrator. If she does that incorrectly, then her jobs fail.

By no means is this a trivial amount of work. In fact, once you scale it up to 100s, or 1000s of machines, it's extremely difficult to manage, as cluster administrators are already well aware.

The Copper Way

So how does Copper help with grid queueing? It helps by eliminating nearly all of the redundancies in the above steps.

Let's assume, as before, that the cluster has been set up, but this time with the Copper platform. None of Uma's applications have been installed or configured.

1. The administrator allocates a new virtual machine image to Uma with the queueing daemon installed (installing the daemon is trivial: just download the appropriate package and install it. There is zero configuration). He then configures Uma's VM to have access to the appropriate network filesystem containers. This can be done quickly and easily through Copper's web interface.

2. The administrator installs and configures the applications on this virtual machine image. This setup is done exactly once, and on a virtual machine independent of all other users and types of jobs that might execute on the cluster, obviating any possible issues with software conflicts. The administrator can even skip this step if Uma is able to install and configure the applications herself. Since Uma's VM is independent of all other VMs, and exists in its own private virtual cluster, the administrator need not be concerned with any security or conflict issues.

3. Uma logs into the virtual machine, and submits jobs to the queueing engine running on it. The jobs she submits can be any command that can be executed on the virtual machine. Copper will ensure through the magic of VM cloning that wherever the job executes, the host it executes on will look and behave exactly like the host it was invoked on. As far as the job is concerned, it is for all practical purposes running on Uma's original VM.

4. There is no step 4.

With Copper, we've dropped the overhead of cluster setup as close as possible to zero, for both the cluster administrator and the user. Uma can even trivially add new applications to execute on the cluster: she simply installs it on her VM, and it implicitly and automatically becomes available as an application that can be queued up to execute on the cluster.

The Lesson

Grid queueing doesn't have to be hard or tedious, no matter what the size of your cluster. Using virtualization, cloning, and the GridCentric high-level toolkit, Copper makes implementing grid queueing a snap.

Copper 1.0 on its way

2010-01-30T13:33:00.000-08:00

The months are flying by! As always, we've been busy here preparing to rock the world of virtualization and High Performance Computing with Copper. But as Copper 1.0 nears, I thought that I'd provide a bit of background on what exactly it is.

GridCentric's platform has been named Copper in light of the fact that it really is the first true "Cluster Operating System". That means that it lets you do on your cluster all the things that we're used to doing with a modern multi-tasking operating system. Let me provide a table with a few simple analogies:

Feature	Desktop OS (e.g. Windows, Linux)	Cluster OS (Copper)
Multi-tasking	Run multiple programs simultaneously.	Run different cluster applications simultaneously (which include operating system stacks, libraries, across hundreds of machines); these are virtual clusters.
Resource accounting	See memory and CPU usage of different processes.	Account for memory, CPU, disk and network usage of every virtual cluster.
Job control	The ability to kill misbehaving processes (some of the time).	The ability to pause, suspend, limit or kill any virtual cluster.
Developer API	An API for performing system-related functions, such as creating processes, reading and writing files.	An API for growing the size of your virtual cluster instantaneously, killing machines within it, accessing data sets, reserving resources, etc.

Simple, eh? Now for some Copper eye candy. Some of those following Copper may know that the fundamental primitive that enables all this amazing stuff is blazingly fast virtual machine cloning. Just for fun: here's a video of me growing my virtual cluster from one machine to a few, then back to one, then to eleven and back to one, then to over twenty and back to one. Each clone is fully-functional and independent -- waiting for ssh is actually slower than creating the machine itself.

Copper deployment at York open to researchers

2010-01-30T13:13:00.000-08:00

We're proud to open up the deployment at York University to researchers there who need a bit more computing power and want to experiment with the next generation of High Performance Computing.

You will be able to harness the power of your own personal easy-to-use cluster - without any of the hassles of keeping compute nodes consistent with your binaries, worrying about cross compilers, restrictions installing packages on the headnode. Copper gives everyone the simplicity of one machine with the power of hundreds.

This hardware for this trial is generously provided by SHARCNET and we expect that we will be able to open up this deployment to all SHARCNET users in the near future. For now, York researchers can request a Copper virtual cluster.

GridCentric Copper deployment to York University

2009-12-18T10:28:00.000-08:00

In December GridCentric performed a beta installation of its Copper Cluster Operating System on a SHARCNET cluster at York University in Toronto. The cluster has 32 compute nodes, each with two dual-core Opteron processors and 8GB of RAM, for a total of 128 compute cores and 256GB of memory. The cluster also includes 4TB of RAID5 storage, as well as Myrinet 2G low-latency interconnect between each node.

Total installation time was around five hours, 4.5 of which were spent installing Fedora on the login node, head node, and storage node. After that was completed, we booted the compute nodes into our Copper compute image and the cluster was ready for action.

Here's a picture of the Copper admin console showing an Ubuntu 9.04 virtual cluster running a single virtual machine (click for larger image):

Here's the same "Virtual Cluster" after expanding out to 85 VMs (click for larger image):

This "footprint expansion" takes a few seconds. Try that on EC2!

For a better idea of what's going on, take a look at out our demo at DCT24.

GridCentric SC09 Wrap-up

2009-11-24T14:17:00.000-08:00

We spent last week in beautiful Portland, Oregon for the SC 2009 International Conference on HPC, Networking, Storage, and Analysis. In between visits to various brewpubs and our gastronomic home base, we managed to get the following done:

Hung out at the Compute Canada booth with Mike Bauer, John Morton and others from SHARCNET, Susan Baldwin from Compute Canada, as well as Cameron Kiddle from the U of Calgary / WestGrid (who showed us a cool demo of GeoChronos) at the Monday night Opening Gala.
Attended the IBM Customer Reception, where we had beers with Neil Bunn (IBM), Jill Kowalchuk (Cybera), Florent Parent (Clumeq), and many others (thanks to Neil for the invite!).
Went to the SGI Innovators Breakfast where we met with Paul Lu (U of Alberta) and talked about the GridCentric platform's technological underpinnings (thanks to Daniel St-Germain for the invite!). This was one of the few alcohol-free events we attended.
Spoke with Josh Simons (Sun) about the convergence of HPC and Enterprise technologies. You should read his blog, especially his HPC writings.
Showed our demo to Daniel Chavarría from PNNL, who gave us helpful feedback and words of encouragement.
Spoke with folks from Mellanox, QLogic, and Myricom about hardware support for virtualization in their Infiniband and 10GigE adapters. Turns out they've supported it for years.
Spoke at length with Gregor von Laszewski about the NSF FutureGrid project, which will act as a testbed for HPC grid/cloud platforms and software stacks - perfect for evaluating the GridCentric platform and getting ourselves on the U.S. government's radar.
Had beers with Daniel Gruner (SciNet) at the Thursday Closing Event (which was quite nice!), then went on to have more beers with Danny at another pub whose name escapes me (although I recall it had over 100 beers on tap).

Overall, it was an exhausting week (bookended by some atrocious connector flights), but we made a lot of friends and are much the wiser. We're really looking forward to SC10!

Office move

2009-11-13T11:48:00.000-08:00

It's been a while since our last update, but we've been pretty busy (more on that later). With the investment from RCI also came a move to their newly renovated space for Rogers Ventures companies. This transformation has been quite significant, so I wanted to share it.

Those that had visited us in our old offices know that they were a bit cramped (especially the weeks leading up to the move as we grew to six). We like to think that it was a stereotypical start-up, and gives us a great "it all started in an attic" (literally) story.

First: the developer's desks. In the main room of our office, four people were in fairly close quarters. Meanwhile, Tim and I shared two sides of another small Ikea desk. Great for extreme programming, but not so great on those hot august days! :)

Workstations 1.0

Workstations 2.0

Our new workspaces are much more spacious -- enough for multiple machines and monitors and a good place to organize and post notes. Unlimited productivity, here I come!

In our old office, Tim and I shared a room with three fairly noisy servers and several pedestal servers. Not that the others had it much better; noise wasn't really containable in our space. You'll also see our enormous 12,000 BTU air conditioner at the bottom left of this photo.

Server Room 1.0

Server Room 2.0

I would say that one of the best features of our new office is the rack space. Now we can hide away our servers and not worry about tripping over the power cord! (Also we don't have to worry about blowing a fuse!)

We nick-named a portion of our old office "the idea board". This was the coffee space and the white board. Shown below is the idea board in the last few days before we moved -- when it also served as storage space for the blue boxes to be packed up with our stuff.

Idea Board 1.0

Idea Board 2.0

As you can see from the photo on the right, the size of the white board has at least tripled (it's yet to been seen whether the ideas will triple). We've also got some great space to relax and meet. The only thing left to do is decide what to do with the old idea board.

GridCentric receives funding from the Rogers Venture Group

2009-10-18T14:29:00.000-07:00

On September 2nd GridCentric Inc, a Toronto-based startup developing a platform for the provisioning and management of High Performance Compute (HPC) clusters using virtualization, closed a seed round of financing with the Rogers Venture Group (http://rogersventures.ca).

GridCentric was founded in early 2009 by Andres Lagar-Cavilla, Adin Scannell, and Tim Smith. GridCentric's platform is based on technology that enables stateful cloning of a running virtual machine to dozens of physical hosts in less than a second.

Mike Lee, Rogers Communications' Chief Strategy Officer, has joined Adin Scannell and Tim Smith on GridCentric's Board of Directors.