Recently BlueMetal has announced opening a new Chicago office. We’re all very excited to have more smart and amazing people joining our adventure.

You can read more about this in the Mass High Tech Business article “BlueMetal opens new Chicago office, names director”.

I really advise to check it out: http://www.infoq.com/presentations/Designing-Consuming-a-Cloud-2-API

I think it’s time to start talking about Cloud Virtualization. Today cloud diversity is driving customers out of the clouds instead of embracing them. Multiple cloud providers are trying to lure customers in by developing feature specialization and this may well serve to fracture the markets and the community even more.

Specialized clouds have a good reason for existence, but an explosion of many different ‘flavors’ of clouds are creating a similar problem to choosing an insurance policy – there are thousands of them, you have to be an insurance agent to understand it and there is a lot of fine print that hides the details of the policy.

Cloud Virtualization should provide a common ground for the generic solutions. This allows standardization of the knowledge and principals by which the general population can abide by. Where the "cloud citizens" can thrive and teach the juniors the basic rules of the environment and the survival practices. When the specific needs of your product or your solution require some special sauce – you’re welcome to shop around and choose a provider that can satisfy your specific taste.

The Cloud Virtualization is a necessity that should be recognized by now. You see some standardization in the IaaS platforms around Amazon stack (I.e. OpenStack, Eucalyptus, etc…) but still no specifications around PaaS.

Some of the major providers are starting to recognize the problem. Check out the recent VMWare announcement (VMware seeks Linux-ization of cloud with any-cloud app platform) where it claims:

“The key, …, is allowing developers to build applications using just about any framework, with the ability to deploy to just about any cloud.”

"Just as Linux provides a high degree of application portability across different hardware, Cloud Foundry provides a high level of application portability across different clouds and different cloud infrastructure," VMware CTO Stephen Herrod wrote in a blog post.

The calls for Cloud Virtualization are coming from everywhere, but, since developers have higher tolerance for complexity (they just have fun solving it by writing more code), you hear more vocal calls from the IT industry:

“What’s more important than cost savings for companies — at least in the U.S. and Asia-Pacific regions — is the ability to standardize their software and business processes across the company, according toa new survey of 600 large companies by Tata Consultancy Services, the $8 billion IT service provider. In Europe and Latin America, the primary rationale was the ability to ramp systems up and down faster.”

– http://gigaom.com/cloud/in-cloud-computing-moves-money-isnt-everything/

More and more organizations are beginning to recognize the need and advantages of operationally consistent clouds. The recent announcement from Zynga to move to an internal cloud architecture is the proof of my point:

“We own the base, rent the spike. We want a hybrid operation. We love knowing that shock absorber is there.” – Allan Leinwand, Zynga’s Infrastructure CTO

Zynga uses Citrix Systems CloudStack as its virtual machine management interface superimposed on all zCloud VMs, regardless of whether they’re in the public cloud or private cloud.

– Inside Zynga’s Big Move To Private Cloud by InformationWeek’s Charles Babcock

To achieve progress in this direction we need to talk not only about different clouds and about private data clouds in the data centers (so called “private clouds”), but about a standard that can span across both of these (for the moment) different domains – hybrid clouds. The hybrid cloud should become the main topic in these conversations. It is a commonality between the public and the private clouds (we, developers, call it an abstraction – hybrid cloud with “empty” public component will give you the “private cloud” and vice versa).

“Thus if hybrid cloud is to become the architectural model of choice, it becomes necessary to unify operations across all environments – whether public or enterprise.”

– The Conspecific Hybrid Cloud, HighScalability.com

“This ability to invoke and coordinate both private and public clouds is "the hidden jewel" of Zynga’s success, says Allan Leinwand, CTO of infrastructure engineering at the company.”

– Lessons From FarmVille: How Zynga Uses The Cloud

So, in my opinion, Cloud Virtualization, (or in “dev speak” Cloud Abstraction) is the next evolutionary step of our industry. For the moment Platform-as-a-Service (PaaS) systems spanning the same APIs across different cloud providers are a great (and realistic) promise to achieve the cloud unification.

Here is why I think it’s true:

1. The PaaS layer, in its core definition, has to provide an API for all the Cloud infrastructure control knobs. Otherwise it’ll not be considered as a CLOUD.
2. The same control API will enable the development unified control tools ecosystems that can control in the same way multiple clouds from different cloud providers. Thus unifying all the IT and DevOps processes.
3. Unified PaaS API layer will enable a new evolution cycle of the application development that will enable the same applications to run seamlessly on different clouds as well as on local data centers without modifications. Hence enabling IT and DevOps to make decisions how and where to deploy them.

Today there is only one publicly available project that shows some promise to achieve the Cloud Abstraction for the developers – JClouds. Lately, VMWare has been making some changes with their Cloud Foundry platform by promising to change the balance with BOSH – an open source tool chain for release engineering, deployment and lifecycle management of large scale distributed services.

So, to sum all this up, I believe that Cloud Virtualization is unavoidable. Customers are looking to be able to scale across multiple cloud factors, and vendors are looking to lock in with feature creep. This tension will continue to grow in the models which are marketed today. The goal of reaching the cloud promise of true elasticity and flexibility based on what the application demands is the opportunity for our industry to deliver on, and we can wait for it passively or we can do something about it by architecting for this now.

For me personally, I choose the latter.

The new REST interface allows you to see what are the currently running services by sending a GET request to the dashboard URL (i.e. http://dasboard_url/services).

As a reply, you’re going to get a JSON block that contains small snippets of information about each seevice and shelf that is running inside the TopShelf process.

[...
   {
      "Name":"clock",
      "ServiceType":"Shelf",
      "CurrentState":"Running",
      "Action":"stop"
   }
...]

The most important piece of the block above is the URL-encoded Name of the service. You can use this name as an ID to control the service’s lifetime via HTTP GET requests, i.e.: start, stop, unload:

• /service/{name}/start
• /service/{name}/stop
• /service/{name}/unload

The latest addition to the control interface is the ability to upload shelves remotely by uploading a ZIP file with all the relevant bits of your service.

You can do the upload via the dashboard by navigating to the ZIP package or by dragging and dropping the shelf archive onto a designated area.

In addition, you can do it programmatically via file upload POST request. This sounds complicated, but, in fact, if you’re using .Net framwork, it may look as simple as this:

using (WebClient client = new WebClient())
{
     client.UploadFile("http://localhost:8087/service/upload", @"C:\Packages\clock.zip");
}

All the above functionality can be configured and fine-tuned via the new optional configuration settings that you can put in the TopShelf host configuration file:

<appsettings>
	<add key="ServicesLocation" value="Services" />
	<add key="DashboardUri" value="http://localhost:8087" />
	<add key="DashboardUploadsEnable" value="true" />
	<add key="DashboardUploadsPackageFolder" value="Packages" />
</appsettings>

Here is a short explanation of the settings:

• Services – contains a full or relative path of the shelves folder
• DashboardUri – defines a Uri (ip and port) that the dashboard module (and REST web service) will use to listen to the incoming requests
• DashboardUploadsEnable – this value controls if the uploads of the shelves are enabled (both through the dashboard and the REST service)
• DashboardUploadsPackageFolder – defines a temporary location where the ZIP archives of the shelves will be stored before unpacked into the shelves location (defined by the Services setting)

If you adding Nancy to the existing MVC site – make sure to remove the sub-route that controlled by Nancy from the routing table.

Here how it works:

1. Follow all the steps from the Nancy Documentation Wiki

2. Add your module and make sure that it has an “offset” path (via inheritance)

public Module() : base("/nancy")
{
   ...
}

3. Remove the “nancy” route from the Routing table in the Global.asax.cs

MvcApplication.RegisterRoutes(...)
{
   ...
   routes.IgnoreRoute("nancy/{*pathInfo}");
   ...
}

I’ve had a chance to build the realistic distributed application live-coding from scratch. The application consisted from 2 roles – web role and a worker role. During the demo I’ve used the Amazon AWS cloud, but the similar technique can be applied to Azure deployments.

The audience was great and engaging even though we didn’t have enough and I was flying through the code.

The code I’m publishing (Amazon AWS demo) is a beautified version of the one we’ve created during the Code Camp.

Stay tuned for the mirrored Azure demo code…

NOTE: This is a simplified version and should not be treated as a real-life application.

I liked their in-depth explanation of how the internals of the Host OS and Application VMs are interconnected.

Let’s take a simple example of understanding dependencies between your assemblies.

If you’re running in a Visual Studio and you have a full access to the source code, it is possible to see the referenced assemblies but it’s pretty hard to track what assembly depend on the assembly you’re working on. [Note: if you’re really in need of a ready made tool for this purpose, you can check the “Architecture Tools” for Visual Studio 2010 or NDepend tool]

A visualization like this is priceless:

On this graph you can see a graph of the assemblies dependencies for the Paint.Net. The snapshot above highlights the dependencies of the PaingDotNet.SystemLayer (highlighted in blue) by showing all the assemblies it depends on in “yellow” and all the assemblies dependent on it in “green”. The rest is left in a "gray” color.

Let’s see how you can create interactive dependency graphs for your data.

In my case I chose to use the QuickGraph module from CodePlex that can be download from NuGet [PM> Install-Package QuickGraph]. It provides a very simple yet robust API that allows you to create the directional and non-directional graphs in memory and then save them in different formats.

Here is a small example of how to create 2 vertices (V1 and V2) and a directional edge (from V1 to V2):

var graph = new AdjacencyGraph<int, TaggedEdge>();
graph.AddVertex("v1");
graph.AddVertex("v2");
var edge = new TaggedEdge("v1", "v2", "depends");
graph.AddEdge(edge); 

As soon as all the graph is created it can be serialized, for example, in a GraphML standard format like this:

using(var xwriter = XmlWriter.Create(...))
    g.SerializeToGraphML(xwriter);

This is it. GraphML files can be opened by a range of different tools, but, if you’d like to show these visualizations in your WPF applications you can use a Graph# from CodePlex.

Here is a sample WPF application rendering a GraphML dependency graph:

As you can see, you don’t have to be afraid of interactive data visualizations but you can make your applications shine and provide useful and easily digestible information for your users.

Note: Source code for this article you can download …here…

Check out this slide presentation for Netflix insights on how they are running large production system on AWS cloud: