In State of the Cloud August 2009, I noted that Amazon EC2 (which I’d covered in a previous one-off post) had grown 9% in just a month, hinting at 181% CAGR. One year later, let’s see if EC2 lives up to these expectations.

While EC2’s overall growth in our survey hasn’t lived up to 181%, there is hardly any reason for disappointment. With the exception of Joyent, the evidence shows that all of the providers tracked have seen incredible growth over the past year. The two leaders have doubled their share and continue their neck-to-neck race (more on that below). The smaller GoGrid and Linode have more than doubled in size, however their is still a large gap between them and the major league duo.

If we tally up all the numbers, we find that overall the cloud has doubled in size over the past year: back in August 2009, State of the Cloud found 3,635 websites on cloud providers. This month that number is 7,278. That’s 100.2% growth.

Snapshot for August 2010

Let’s turn back to this month. The race continues, but takes a surprising twist:

For the first time since this survey began, Amazon EC2 has dropped. Last month we counted 3,043 sites on EC2. This month, that figure is 3,011 – negative growth of 1.2%. Granted, that’s a pretty small figure and could well be a statistical anomaly; nonetheless it will no doubt be a concern to Amazon, what with Rackspace breathing down its neck.

A Word of Appreciation

It’s been a fantastic year. I’d like to take this opportunity to extend my thanks to all of you out there in the cloud industry who have pitched in, to the providers who readily cooperate (and to those who don’t…) – and above all to you, my loyal readers, whose feedback and encouragement are a real driving force. Where will we be in August 2011? Let’s wait and find out.

Snapshot for July 2010

Here are the results for this month. Welcome Linode.

The top pair continue their steady march forward with 13% and 19% growth for Amazon EC2 and Rackspace Cloud Servers respectively, as compared to the last report two months ago. Amazon EC2 is the first of our contenders to smash the 3000-site barrier.

Linode is this month’s surprise, jumping straight into third place. I was deliberating whether Linode should be included in the report. Following some lively discussions on Twitter, the consensus was that Linode looks like a duck and walks like a duck, even if it doesn’t bother quacking. (This, as opposed to some providers that work hard to market themselves cloud while they don’t really seem to be.)

Linode offer a rich feature set and have an outstanding reputation among their customer community. The last time I saw this picture it was Slicehost, who ended up being acquired by Rackspace to jumpstart their Cloud Servers offering. What does the future hold for Linode?

Trends

Uncovering Linode’s footprint in the historical data collected, we witness remarkable growth. Linode has grown by 270% over the past 11 months, more than any other of the providers tracked.

Alternative Data Sets

Last month we skipped a report due to a bug in the top site list published by Quantcast, which serves as the input for this research. This prompted me to consider other data sets that could also be used. So what would our results be like if we used another well-known source of top site rankings? I chose to take Alexa, a well-known source, for a spin and see what happens. Here’s what I found -

I was pleased to find the results do not differ greatly between the different sets. The overall rankings are preserved although some providers do exhibit some variation (particularly Rackspace, which loses over 20% when we use Alexa). Even if it is tempting to switch data sets, continuity is of grave importance for research like this one. The findings above do confirm that the standings are in the same ballpark even when viewed through a different lense. Hence I am happy to continue using Quantcast (well, unless last month’s issues pop up again!).

Next month marks a year since the first State of the Cloud. This milestone will serve as an opportunity to look back and see what progress and changes we’ve seen over the past year – as well as to take a guess at what the future may hold for this industry.

Unfortunately, this month we have no regular post in the State of the Cloud series. Quantcast’s Top 1M Site list, which is the input data set used by the research, is broken. Instead of 1 million sites, the list contains only 78,000. Clearly, this makes it difficult to continue tracking using the same, consistent methodology we’ve had in place for almost a year.

Quantcast are aware of the problem, but “do not have an ETA on the fix”.

What you can do

Help us prioritize Quantcast’s investigation of this issue: go to http://www.quantcast.com/contact and tell Quantcast what you think. Here’s an example of what you could write:

But first, this month’s figures -

Looking at this month’s trends, we find a relatively slow month for the cloud. Overall growth was just 1.7%. The largest month-to-month percentage growth goes to OpSource which grew by 11%. In terms of absolute size, Rackspace grew the most with 70 new sites. Amazon gained just 10 new sites this month.

Let’s take a look at cloud providers from another angle. We last did this back in February.

Our first conclusion: Amazon now controls more than 50% of cloud-hosted sites. The second conclusion: by this metric at least, the cloud race continues to be a two-horse race: Amazon and Rackspace together control 94%, and all the rest of the providers retain but a sliver of control.

Many of the smaller providers have had a weak month, some even showing up less this month in the sample than they did previously. Only Amazon and Rackspace continue to plough ahead with Amazon gaining 6% and Rackspace 3%.

Overall cloud growth this month is at 3.9% (=58% CAGR).

Here’s the video of my talk and the presentation.

Snapshot for March 2010

Total Cloud Adoption

It’s easy to get caught up in the differences between each provider, so this time let’s try to zoom out for a moment. Last month I noted that all these providers combined still hold less than 1% of the sites surveyed. Well, this month that threshold was crossed. The cloud now constitutes 1.01% of our sample. There is plenty of room to grow…

Like many in the cloud community, these days I am gearing up for CloudConnect in Santa Clara next month. If you’re going to be there, be sure to catch me and say hi. Don’t miss my appearance on the main stage on the morning of Tuesday 16th.

See you there!

Be sure to come and watch, as well as to post your questions during the show.

Tuesday, Feb 16th at 11am PST. Register here.

This week, the black box became a little bit whiter.

Sören Bleikertz, a computer science student writing his Masters thesis on EC2 security, poked into the Xen hypervisor used by EC2 and made some observations regarding EC2’s underlying architecture. Among his findings on the storage and networking configurations, Sören pointed out that each instance was given a unique name (the “Xen domain”) such as dom_32504936 and that this seemed to behave like a serial number, growing from day to day. Sound familiar yet?

Well, it turns out that this Xen domain is none other than the underlying instance ID uncovered in my previous research! This revelation gives us an important conclusion: the decoding method was accurate. The serial number exists and based on everyone’s input we even got the formula right.

With Sören’s technique at hand we can now uncover the constants needed for all EC2 regions. Except for us-east-1 which thanks to RightScale enjoyed a 3-year history, we did not have enough data to extract the constants for other regions. Surprisingly, it turns out that the constants are in fact identical for all regions. What threw us off the scent is that as opposed to us-east-1 which very likely started the serial number from zero, the other regions do not. For example, the serial numbers for the 3-month-old us-west-1 region are already in the range of 752 million. Those for eu-west-1 are in the 500 million range. We can safely assume that hundreds of millions of instances have not in fact been spun up. What makes more sense is that each region was assigned a different starting point in order to ensure globally unique instance IDs.

An additional finding of Sören’s is that the image file for the root disk points to a filename on the VM host such as /mnt/instance_image_store_3/262768. It turns out that the number at the end of this file is, again, simply the AMI ID – decoded. For example, we can re-encode 262768 to yield ami-19a34270, which is Alestic’s Ubuntu Karmic Base image. Similar to instance IDs, the underlying image ID also seems to have different ranges in each AWS region.

As a bonus of Sören’s discoveries and the connection to the IDs, it’s now possible to infer your instance ID (and image ID) locally, without even consulting the EC2 user-data. Why someone would prefer this to the user-data is a good question, but it’s a fun exercise nonetheless. Here’s a Ruby script that does just that:

#!/usr/bin/ruby
$stderr.puts("Detecting VM domain ID (may take a few moments)")
dom_id = nil
(1..65535).each do |i|
        if system("xenstore-ls /local/domain/#{i} > /dev/null 2>&1")
                dom_id = i
                break
        end
end

$stderr.puts("VM domain ID is #{dom_id}")

dom_name = `xenstore-read /local/domain/#{dom_id}/name`
$stderr.puts("VM domain name is #{dom_name}")

numeric_id = dom_name.split("_").last.to_i
c1 = numeric_id >> 24
c2 = (numeric_id >> 16) & 0xFF
c3 = numeric_id & 0xFFFF
c3_1 = (numeric_id >> 8) & 0xFF
c3_2 = numeric_id & 0xFF

d1 = c1 ^ c3_2 ^ 0x69
d2 = c2 ^ c3_1 ^ 0x40 ^ 0xe5
d3 = c3 ^ 0x4000

instance_id = sprintf("i-%02x%02x%04x", d1, d2, d3)
puts(instance_id)

This requires xen-utils to be installed on the machine (on Ubuntu, run apt-get install xen-utils-3.3). Here’s an example run:

# ./get_instance_id.rb
Detecting VM domain ID (may take a few moments)
VM domain ID is 1423
VM domain name is dom_32900610
i-6a554602

Thanks once more to Sören for the great detective work.