Scaling Out

SimpleDB Open to the Public with Lower Prices

2008-12-01T16:13:00.001-08:00

As of today, anyone can try SimpleDB with no waiting… so what are you waiting for??

And they’ve reduce the storage costs by 83% to $0.25 per GB per month.

AND finally (and best of all), they’ve added a FREE tier giving you 25 Machine hours + 1GB of data transfer + 1GB of storage for free. This is huge news. Now people can try and potentially use SimpleDB for free indefinitely.

New Social Search Product Running Completely on Amazon Web Services

2008-10-13T17:02:00.001-07:00

A new social search service called SideStripe has been released into the wild. The service lets you find information as well as ask questions and get answers from the people in your social network. They also have a nice widget that embeds itself right in your Google results.

SideStripe is running on EC2 with app servers behind load balancers, SimpleDB is the database and S3 is used for larger data that doesn’t fit in S3 as well as for shared caching.

Amazon Reducing S3 Storage Costs via Tiers

2008-10-09T10:44:00.001-07:00

Starting November 1st, the following pricing structure will take effect.

Current storage price (through Oct 31, 2008)
US: $0.150 per GB
EU: $0.180 per GB
New storage price (effective Nov 1, 2008)
US
$0.150 First 50 TB / month of storage used
$0.140 Next 50 TB / month of storage used
$0.130 Next 400 TB / month of storage used
$0.120 Storage used / month over 500 TB
EU
$0.180 First 50 TB / month of storage used
$0.170 Next 50 TB / month of storage used
$0.160 Next 400 TB / month of storage used
$0.150 Storage used / month over 500 TB

I’m still hoping that they drop the per request price at some point.

EC2 to Support Microsoft Windows Server and SQL Server this Fall

2008-10-01T08:03:00.001-07:00

Currently EC2 only supports Unix based operating systems, but coming this fall you’ll be able to run Windows Servers. This may not sound like much, but sometimes you just need it and it will be nice to be able to have it. We needed Windows as part of our infrastructure for one project and we actually we had to lease a windows box from another host and send messages between them to get things done… not pretty. This should solve that problem.

This is also great news for you ASP.NET developers. Welcome to the cloud!

Amazon Will Soon Have a CDN

2008-09-18T10:03:00.001-07:00

Wow, the hits just keep on coming. Amazon is quickly becoming the one stop shop for every possible hosting scenario. They will soon add a content delivery network for low latency delivery for things such as video's, etc.

I love how easy it will be to use. Just upload your file to S3 like you do now and then a single API call will make it part of the CDN.

Use Amazon S3 like Memcached

2008-08-30T15:17:00.001-07:00

A new open source project called CloudCache enables people to use a simple cache/map interface into Amazon S3. Check it out at:

http://code.google.com/p/cloudcache/

How Much Would it Cost to run Twitter on Amazon's SimpleDB?

2008-07-01T11:12:00.001-07:00

As you may or may not know, Twitter has had an amazing amount of downtime in the past few months and it may be due in part to running only three MySQL servers to run one of the most popular sites on the Internet. So lets say they wanted to use SimpleDB to get rid of their database issues, well lets try to figure out their costs based on some publicly available usage numbers and traffic numbers along with these BoxUsage numbers.

Twitter Usage

March 2008
Total Users: 1+ million
Total Active Users: 200,000 per week
Total Twitter Messages: 3 million/day

14 Million Visits per month with an average of 7 pages per visit = 98 million pageviews per month

Now lets assume that each page only does one (assume a small amount of caching) and each object has item has 5 attributes.

Translated to SimpleDB usage:

3 million PUT's per day
98 million QUERY's per month
98 million GET's per month

3 million PUT's = (3 * $3.079) + (3 * 0.0000280 * 5^3) = $9.25 / day = $277.43 / month

98 million QUERY's = (98 * $1.96) + (98 * 0.00112 * 5) = $192.63 / month

98 million GET's = (98 * $1.305) + (98 * 0.00028 * 5^2) = $128.58 / month

For a grand total of $598.64 per month.

This only includes box usage for SimpleDB and does not include bandwidth (which is free if you're using EC2) or storage.

Wow, a lot cheaper than I would have expected. Also, you could reduce this cost significantly by doing some serious caching (if you develop in Java, check out SimpleJPA to get easy caching).

UPDATE: I miscalculated this when I first posted it by counting 98 million pageviews per day, when it should be per month. Should be all corrected now.

SimpleDB BoxUsage Calculations

2008-06-30T19:01:00.001-07:00

Colin Percival wrote a great article that dissects SimpleDB's BoxUsage figures. It's very interesting what he found and he breaks it down in a nice table that you can use to estimate how much using SimpleDB will cost you.

Google Plans to Launch BigTable as a Service

2008-04-06T10:15:00.001-07:00

Google's BigTable might be open to the public soon, so says TechCrunch, essentially competing directly with Amazon's recently launched SimpleDB.

BigTable is a massively scalable database and it is what Google uses for over 60 of their own applications including Google Maps and Orkut.

BigTable allows Google to have a very small incremental cost for new services and expanded computing power (they don't have to buy a license for every machine, for example). BigTable is built atop their other services, specifically GFS, Scheduler, Lock Service, and MapReduce.

This is a huge move from Google that I hope we see more of in the near future... is an EC2 clone on the way?

EC2 Availability Zones Protect Your Applications from Data Center Outages

2008-03-27T11:12:00.001-07:00

Amazon just released another new feature for the Elastic Compute Cloud that allows you to specify physically different locations when launching an EC2 instance. It's called Availability Zones. This means that if there is a major data center issue such as a network outage or even a fire in one Availability Zone, it will not affect an instance running in a different zone. High availability on the cheap... a beautiful thing.

"Previously, only very large companies had the scale to be able to distribute an application across multiple locations, but now it is as easy as changing a parameter in an API call."

Amazon EC2 Now Has Static IP Addresses

2008-03-27T11:02:00.001-07:00

Or as they like to call them, Elastic IP Addresses. It is actually a pretty interesting concept that allows you to get a static IP address assigned to your account which you can then point to any particular EC2 instance. So if you take a server down, you point the Elastic IP to a new instance and the outside world still uses the same IP. Here's Amazon's description:

Elastic IP addresses are static IP addresses designed for dynamic cloud computing. An Elastic IP address is associated with your account not a particular instance, and you control that address until you choose to explicitly release it. Unlike traditional static IP addresses, however, Elastic IP addresses allow you to mask instance or Availability Zone failures by programmatically remapping your public IP addresses to any instance in your account. Rather than waiting on a data technician to reconfigure or replace your host, or waiting for DNS to propagate to all of your customers, Amazon EC2 enables you to engineer around problems with your instance or software by quickly remapping your Elastic IP address to a replacement instance.

Of course, you can still use dynamic dns (explained here) too.

Pricing:

No cost for Elastic IP addresses while in use
$0.01 per hour when not mapped to a running instance
100 free Elastic IP remaps per month per account and $.10 per remap thereafter

Microsoft Responds to Amazon SimpleDB with SQL Server Data Services (SSDS)

2008-03-06T12:18:00.001-08:00

There's not a lot of information on what exactly this will be (relational db? bigtableish? will it even be SQL Server behind the scenes?) other than users can host an unlimited amount of data on SSDS. My guess is it's not relational and not SQL Server, but rather more like BigTable/SimpleDB since that appears to be the new school way of massively scaling your data like how the big green giant Google does it.

They do have some interesting features that take the edge over SimpleDB such as full-text search and a LINQ based query language.

You can get your name on the beta list here.

Why is it that when Microsoft does something like this, it's just not quite as exciting as when a different company does it? If Google announced something like this instead of Microsoft, I would be so much more excited. Oh well, at least it's good to see some competition cropping up.

Smugmug says Amazon EC2 is not Slower than they Advertise - it just Depends on How you Define a GHz

2008-02-29T11:45:00.001-08:00

All GHz aren’t created equal.

Who knew? Here's a good read backed by real numbers by one of the most notable Amazon customers Smugmug, not by people complaining in the forums.

The Future Just Called...

2008-02-29T11:36:00.001-08:00

I had a written a post back in November about how the New York Times converted 11 million old articles to pdf in under 24 hours using Amazon EC2, S3 and Hadoop and then I came across a post today that had this to say about it:

The future just called and wanted to explain how "supercomputing" was actually going to work.

Amazon S3 Goes Down for 3 Hours

2008-02-17T13:16:00.001-08:00

Amazon's Simple Storage Service (S3) had it's first major outage in years going down for 3+ hours and taking down 1000's of web 2.0 sites with it. Many new companies rely on Amazon's services to run their business so when Amazon goes down, they go down (myself included).

There is a big thread that you can read about it on the AWS forums and see Amazon's Post Mortem response:

Here’s some additional detail about the problem we experienced earlier today.
Early this morning, at 3:30am PST, we started seeing elevated levels of authenticated requests from multiple users in one of our locations. While we carefully monitor our overall request volumes and these remained within normal ranges, we had not been monitoring the proportion of authenticated requests. Importantly, these cryptographic requests consume more resources per call than other request types.
Shortly before 4:00am PST, we began to see several other users significantly increase their volume of authenticated calls. The last of these pushed the authentication service over its maximum capacity before we could complete putting new capacity in place. In addition to processing authenticated requests, the authentication service also performs account validation on every request Amazon S3 handles. This caused Amazon S3 to be unable to process any requests in that location, beginning at 4:31am PST. By 6:48am PST, we had moved enough capacity online to resolve the issue.
As we said earlier today, though we're proud of our uptime track record over the past two years with this service, any amount of downtime is unacceptable. As part of the post mortem for this event, we have identified a set of short-term actions as well as longer term improvements. We are taking immediate action on the following: (a) improving our monitoring of the proportion of authenticated requests; (b) further increasing our authentication service capacity; and (c) adding additional defensive measures around the authenticated calls. Additionally, we’ve begun work on a service health dashboard, and expect to release that shortly.
Sincerely,
The Amazon Web Services Team

Of course it's bad that this happened, but really not so bad when you think about it. How fast could you recover your own database and servers when they go down? And we all know, things go down at some point or another. And since Amazon puts out the fires, you get to keep more of your hair.

via: InfoWorld

Amazon Drops the Price for Simple Queue Service (SQS), but there are some Caveats

2008-02-06T10:55:00.001-08:00

Amazon announced a new pricing model for their Simple Queue Service (SQS) today which looks much more reasonable than the old system.

Here are the pricing changes:

Previous Price (prior to February 6, 2008)

Messages

$0.10 per 1,000 messages sent ($0.0001 per message sent)

Data Transfer

$0.10 per GB—all data transfer in
$0.18 per GB—first 10 TB / month data transfer out
$0.16 per GB—next 40 TB / month data transfer out
$0.13 per GB—data transfer out / month over 50 TB

New Price (effective February 6, 2008)

Requests

$0.01 per 10,000 Amazon SQS requests ($0.000001 per request)

Data Transfer

Data transfer rates are unchanged. However, because many customers want to use Amazon SQS in conjunction with Amazon EC2, all data transferred between Amazon EC2 and Amazon SQS is free of charge.

So to get the benefits, you should use EC2 to get the free data transfer and second, poll the queue as little as possible (throttle back if the queue is empty and pick it up when there's stuff in it again).

Message Retention Time Changes

The message retention time has DECREASED from 15 days to 4 days. This is not a good thing.

Message Size Changes

The maximum message size has DECREASED from 256 KB to 8 KB for both Query and SOAP requests. Seriously?

Number of Messages Received

They've DECREASED the maximum number of messages you can receive in one request from 256 to 10. So now you have to make 25 requests (which cost money) to get what you could have got out of 1 request.

Conclusion

I've always had a problem with how SQS was priced. I had planned to use it for a service that uses high volume queues locally, but the pricing is just off the hook and didn't make any sense. I'm not sure if the new pricing will make that better or worse, but they've also decreased the functionality quite a bit.

It almost sounds like they are trying to sunset this service.

In any case, you have 180 days to switch to the new API folks, so get on it. For the Java folks out there, Typica is your best bet.

Why Tier Based Architectures Don't Scale

2008-02-03T14:07:00.001-08:00

Nati Shalom from GigaSpaces wrote an interesting piece on his blog about why tier based architecture's don't scale. Think "weakest link" and your half way there.

Read it here.

While it sounds like he has a good point, some real world examples would have been nice.

Another Cloud Database Coming Soon

2008-01-29T11:15:00.001-08:00

EnterpriseDB plans to release a cloud version of it's database offering which will run on Amazon EC2 and store it's data on S3. This will apparently compete directly with SimpleDB, but provide all the advanced relational database features we all know and love.

In an interview conducted last week in advance of today's announcement of the Cloud Edition plans, Zurek declined to disclose specifics about the company's pricing plans. But the pricing "will be disruptive," he said. "I don't think Oracle will like this."

Sounds good to me. I'm not sure how these guys will package it up, but what I would really like to see is a simple on-demand model like SimpleDB with a real relational database. Create databases and tables on demand, pay only for storage and processing power, automatic/transparent scaling, no maintenance.

Java has Achieved Cockroach Status

2008-01-25T09:33:00.001-08:00

I just read through this article on Ted Neward's blog where he attempts to answer this question. He makes a good point about how prolific the JVM really is today.

Here's a thought: Let's leave Java where it is, and just start creating new JVM languages that cater to specific needs. You can call them Java, too, if you like. Or something else, like Scala or Clojure or Groovy or JRuby or CJ or whatever suits your fancy. Since everybody compiles down to JVM bytecode, it's all really academic--they're all Java, in some fundamental way. Which means that Java can thus rest easy, knowing that it fought the good fight, and that others equally capable are carrying on the tradition of JVM programming.

Beyond that, I'll add that it's also found everywhere: computers, phones, and all sorts of other devices.

Don Box from Microsoft responded to the article on his blog saying:

Java has achieved cockroach status and its inventors should be proud.

Mission accomplished.

All I can say is that I'm glad I'm on the cockroach team.

EC2 Teaches Bad Habits

2008-01-18T10:21:00.001-08:00

A twitter from Eric Hammond:

EC2 teaches bad habits: I almost shutdown a remote non-EC2 server when I was done using it

Why SimpleDB Matters

2007-12-16T11:21:00.001-08:00

Here's a good post on SimpleDB usages and why SimpleDB matters.

Very, very simplistically speaking, domains are like tables, with items like rows and attributes like columns. A query cannot cross domains, so in this analogy you can’t “join” domains. But that sort of thinking is a holdover from the relational database normalized model.In reality a domain is much more like a database, so we have to stop thinking in terms of tables and joins.

And another interesting post:

To Rule the Clouds Takes Software: Why Amazon SimpleDB is a Huge Next Step

I love how this one starts with a good Lord of the Rings analogy:

One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them…

J. R. R. Tolkien

Amazon is handing out the nine rings to us humans, that's for sure, but I don't think they are as evil as Sauron... or are they? ;)

Amazon Rolls Out The Missing Piece to Its Web Services Empire: SimpleDB

2007-12-14T16:55:00.001-08:00

Amazon announced a limited beta for SimpleDB, their latest web service offering. A reliable, scalable database has been a particular pain point for EC2 users since EC2 is unreliable. Running a database on an EC2 instance is like driving your car over the Bay Bridge; you just know it's going to fall down at some point, you just hope you're not driving that day (and that your parachute (backup) is freshly packed).

To use Amazon SimpleDB you:

CREATE a new domain to house your unique set of structured data.
GET, PUT or DELETE items in your domain, along with the attribute-value pairs that you associate with each item. Amazon SimpleDB automatically indexes data as it is added to your domain so that it can be quickly retrieved; there is no need to pre-define a schema or change a schema if new data is added later. Each item can have up to 256 attribute values. Each attribute value can range from 1 to 1,024 bytes.
QUERY your data set using this simple set of operators: =, !=, <, > <=, >=, STARTS-WITH, AND, OR, NOT, INTERSECTION AND UNION. Query execution time is currently limited to 5 seconds. Amazon SimpleDB is designed for real-time applications and is optimized for those use cases.
Pay only for the resources that you consume.

Some key points:

Scalable
Amazon SimpleDB allows you to easily scale your application. You can quickly create new domains as your data grows or your request throughput increases. For the Beta release, a single domain is limited in size to 10 GB and you are limited to a maximum of 100 domains; however, over time these limits may be raised.
Fast
Amazon SimpleDB provides quick, efficient storage and retrieval of your data to support high performance web applications.
Reliable
The service runs within Amazon's high-availability data centers to provide strong and consistent performance. To prevent data from being lost or becoming unavailable, your fully indexed data is stored redundantly across multiple servers and data centers.

How many smart people does Amazon have working on this stuff?

Amazon Announces $100,000 Start-Up Challenge Winner

2007-12-08T10:33:00.001-08:00

Out of over 900 business plan entries submitted by developers building their products and services using the AWS platform, Ooyala is announced the winner of the Amazon Web Services Start-Up Challenge. Ooyala is a video company delivering a high quality interactive video experience with monetization and analytics tools for video publishers and advertisers. As the grand prize winner, Ooyala will receive $50,000 in cash, $50,000 in Amazon Web Service credits and an investment offer from Amazon.com. Click here for more on the contest.

Ooyala has built a platform to deliver video content for publishers and targeted advertising via an interactive video experience. Ooyala is built on Amazon Web Services including Amazon EC2, used for video and analytics processing, and Amazon S3 for content storage and delivery. Ooyala leverages Amazon FPS to process both content and ad serving payments. Amazon Mechanical Turk and Amazon SQS are also utilized by Ooyala.

“When we started Ooyala, we were warned that we would spend most of our time in datacenters at 2 a.m. making sure everything worked,” said Sean Knapp, Founder and President of Technology for Ooyala. “Very few companies in the world can provide the infrastructure services Amazon can provide. Why would you do it yourself? AWS has enabled Ooyala to build, deploy and scale our product in record time, raising the bar for rapid innovation.”

Redhat Living in the Clouds

2007-11-14T13:34:00.001-08:00

The Amazon Elastic Compute Clouds that is. Redhat announced the availability of Redhat Enterprise on EC2, so now you can get Redhat Enterprise benefits but pay for it on demand.

Available at a starting price of $19/month per customer plus $0.21 per hour for every deployed server, plus additional bandwidth and storage fees.

From the sounds of it, it looks like they've just packaged RHEL up as a paid AMI so it'll be as easy as launching any old instance. Not a bad idea for enterprise Linux customers.

The Perfect Excuse to use Facebook at Work: Server Monitor for Facebook

2007-11-13T09:47:00.001-08:00

Here's a great, free tool to monitor your servers from within Facebook. This is the description on the app's about page:

Monitor your web servers in Facebook! Get a notification when your site goes down and when it comes back up. Show your server status on your profile too so your friends can see if you're a network admin rockstar or not.
Features

Notifications via Facebook feed, email and SMS

Checks every ten minutes

10 second setup

FREE!