Amazon SQS: Simple Queue Service
A few days later than expected, it was the....erm....writers strike!
Amazon's Simple Queue Service provides reliable storage and delivery of messages between any number of computers that have at least periodic access to the Internet.
SQS has everything you expect from a messaging system, including loose coupling and fault tolerance, with some things you might not, like a distributed infrastructure that stores messages redundantly over multiple data centres. And of course, being web service based, clients can be built on any platform that can use HTTP requests.
SQS Architecture
Like the other Amazon Web Services, SQS is elegant in its simplicity. It contains just 2 constructs:
- Messages. A message is a piece of textual data up to 8KB in size.
- Queues. Queues store related messages together with a common configuration.
SQS is implemented as a distributed system. It stores messages in multiple servers and potentially across multiple data centres. Although this has a positive impact on redundancy, reliability and scalability, it also has its drawbacks. The distributed nature brings in a few quirks that you need to bear in mind when developing an application with SQS.
Message retrievals might return incomplete results. When a request comes in from a client, the SQS system samples only a subset of physical servers for messages to return. Although the subset changes, you cannot be sure that any particular retrieval contains all the messages in a given queue.
Messages may not be delivered speedily. You cannot rely on SQS as a massaging system if you require instant message delivery. In the normal run of things, messages can take from 2 to 10 seconds to be delivered.
Messages may not be delivered in sequence. Although SQS will try to deliver your messages in the order in which they were sent, in a distributed system it's not always possible. If you need to have ordered delivery, you will need to add some sort of sequencing layer on top of SQS.
Messages may be redelivered. SQS uses 2 criteria to determine whether or not a message should be delivered; whether it still exists in a queue, and its visibility state (see below). In a distributed system however, one can never be certain that these properties are synchronised across multiple physical servers, so your application must gracefully handle the delivery of a message that should be invisible or have already been deleted.
Managing Messages
SQS will deliver a message as many times as it needs to in order to ensure that it is properly received, processed and acknowledged. This approach means that no message can be lost. Even if a message-receiving component crashes, or a network goes down before a message can be processed.
Messages are managed by changing a message's visibility state. A messages state can be set to invisible for a certain amount of time. This amount of time varies from 0 seconds to 2 hours, and is set by the queue.
Because a message only remains invisible for a set period of time, the only way to prevent eventual redelivery is to delete the message from the queue. Setting the visibility timeout appropriately is essential for efficient running of your application. If you set it too short, the message might be delivered again whilst it is still being processed. Ideally, a message's visibility timeout should be slightly higher than the time it would take to process the message.
SQS Roles
Message Sender. A message sender contacts SQS and asks it to create a message on a given queue with some data.
Message Receiver. The message receiver periodically polls SQS to see if there are any new messages available on a specific queue. If there are messages on a specific queue, and on the subset of sampled physical servers, SQS will return up to 10 of these messages for processing. If a message is processed successfully, the receiver will delete the message from the queue. If it is not processed successfully, the receiver can opt to do nothing, and let the visibility timeout expire.
Administrator. The administrator monitors the queues and keeps the infrastructure running smoothly. This role can bounce quite nicely off of EC2, in the sense that if a queue starts to back up, it can simply spawn a new server to process the messages!
Pricing
$0.01 for 10,000 requests.
Data transfer is on a sliding scale - $0.18/GB from 0 to 10TB, $0.16/GB from 10 to 50TB and $0.13/GB for any amount over 50TB.
All in all, an enterprise level messaging system, but for a very low cost. Just the right formula for the micro ISV!
Comments