AWSNinja Code Library

Name: awsninja_cloudfrontprivatestreaming

AWS Services: CloudFront, S3

Source: at GitHub

In a Sentence: Setting up Private Streaming on CloudFront is hard, so I automated it.

Setting up private streaming on CloudFront is way more difficult that it probably needs to be. There are all sorts of little niggling details that will have you pulling your hair out. My hope is that this post will help ease the pain, by gliding you right past the rough edges so you can get it done and maintain your sanity.

To do that, this tutorial will set up a CloudFront Streaming Distribution, set up the security, upload some videos, and set up a webpage to view those videos using private streaming.  In addition to the AWS Ninja code that I’ve written, the package includes the JW Player and the AWS SDK for PHP, both of which are included consistent with the licenses granted by those works.

If you’re just trying to learn more about the architecture of CloudFront and the API, you might just want to take a look at the source of oneTimeSetup.php, pushFilesToCloudFront.php, and cleanup.php.  The steps for signing requests are in index.php.

CloudFront doesn’t do everything

The first way I will attempt to save your time is by informing you that CloudFront Private Streaming lacks certain security features. It is not the most secure way to deliver your streams. It has been noted in CloudFront Forums that there are are two specific security techniques that CloudFront doesn’t support:

1. While CloudFront supports RTMPE (encrypted streaming), it does not not have a way to stop end users from unencrypted streaming. This means that if a user has access to your stream, he will have the capability of saving a copy of it using certain third-party tools which can be found online. A user with some level of technical sophistication could obtain a copy of your content in this way.
2. CloudFront does not support SWF verification. This is a technique whereby the streaming server will only stream to a SWF file that is “approved.” This is another method of preventing your stream from being copied.

Despite these limitations, CloudFront Private Streaming does give you a lot. It prevents unauthorized streaming of your content by IP address of the user and/or date range.  This allows you to effectively set up a pay-walled or limited access video site or intranet for content you do not want to make available to the world. Also, CloudFront is one of the AWS services that is still in beta, and we can reasonably expect Amazon to add these and other missing security features at some point.

You should also keep in mind, nothing is 100% secure. It’s a truism that Anybody who can watch your video can steal your video. Your objective is to employ cost-effective means of making your content hard to steal.

If you decide that CloudFront does meet your requirements, what follows is your guide to go from zero to streaming.

Set Up

First, download the awsninja_cloudfrontprivatestreaming project from GitHub. The project also includes a copy of the newly-relaunched AWS SDK for PHP from Amazon and the JW Player flash player. Once you’ve downloaded and uncompressed the library, you need to set up the SDK for PHP config.  Rename the “./aws/sdk/config-sample.inc.php” file to “”./aws/sdk/config.inc.php.”  Then, specify all of the parameters that are listed there:

• AWS_KEY
• AWS_SECRET_KEY
• AWS_ACCOUNT_ID
• AWS_CANONICAL_ID
• AWS_CANONICAL_NAME
• AWS_CLOUDFRONT_KEYPAIR_ID
• AWS_CLOUDFRONT_PRIVATE_KEY_PEM

Instructions for how to set all of these can be found in the “config.inc.php” file itself.

One-Time Steps

Once you’ve completed the SDK config file, you should then run the “oneTimeSetup.php” script on the command line.  This is found in the “examples” directory. This script automates and ensures  completion of all of the steps of the setup process. Here are the pitfalls the script helps you avoid:

• You must specify a “Caller Reference” when you create an Origin Access Identity (OAI). You’ll probably never use it, but the CloudFront API requires that you have one.  When using the “oneTimeSetup.php”, a Caller Reference is created for you automatically.
• When creating a private streaming distribution, there are several required pieces of information:
  1. The OAI Id – You must specify the OAI Id, so you need to create the OAI before creating the distribution.
  2. The Streaming Flag – A “Distribution” and a “Streaming Distribution” are really different things. In order to create a “Streaming Distribution” you must add the Streaming=true argument. If you forget to include it you will create a normal Distribution. There is no way to change a normal Distribution to a Streaming Distribution or vise-versa.
  3. The TrustedSigners Flag – In order to control how your streams are used (which is bascially what we mean by “private”), you must make the stream unavailable to requesters who don’t properly sign their requests. To activate that requirement, you must set the TrustedSigners flag to either “Self” or to the Canonical Id of another AWS user. There is no setting called “Require Signed Request” or anything like that. Whether or not a signed request is required is determined by whether or not you specify a TrustedSigner.Furthermore, if you do not set up any TrustedSigners for your Streaming Distribution but the requester sends a signed request – the stream will still work. This might mislead you into thinking that your stream is protected when in reality it would work equally well with an unsigned request.

The “oneTimeSetup.php” script will take up to 15 minutes to run because it takes that long for your new Streaming Distribution to become active and deployed. Once that completes successfully, rename the “config-sample.php” file to “config.php.” The “oneTimeSetup.php” script will echo out the PHP define() methods that you need to put in the “config.php” file (not the AWS SDK “config.inc.php” file!). This completes the configuration of the other scripts.

To see that your Streaming Distribution is all set up, log into the AWS Console and click the CloudFront tab:

Your Streaming Distribution is Ready

Uploading Files

Now that your CloudFront Origin bucket, OAI, and Streaming Distribution are all set up and configured, it’s time to upload some videos to your streaming bucket.

Included in the package is a file called “getCrockfordVids.sh.” If you run this file on your command line, it will download five videos of Douglas Crockford’s talks on JavaScript from YUI Theater. If that’s not enough JavaScript instruction for you, there are about 15 more videos in the file that are also available for download if you uncomment those lines. If you’re wondering what to do next New Year’s Eve, I highly recommend downloading them all and hosting a “Douglas Crockford Viewing Marathon.” Best New Years EVER! (BTW – Douglas Crockford is awesome and I’ve learned a lot from his presentations even though I’ve only watched a small portion of what he’s put out.)

Anyway, the “getCrockfordVids.sh” file will download Crockford videos to the “video” directory (warning: the five files that are set to download by default comprise 1.6GB of video). Once that is complete, you can run the “pushFilesToCloudFront.php” script to upload the files to the S3 bucket that was created for you by the “oneTimeSetup.php” script. In this script, there are two calls to the S3 API. The first one simply puts the object to S3, specifying that the owner has full control:

echo("Put $file on S3\n");
$cbr = $s3->create_object(NINJA_STREAMING_BUCKET, $file, array(
	'fileUpload'=>NINJA_BASEPATH . 'awsninja_cloudfrontprivatestreaming/videos/' . $file,
	'acl'=>AmazonS3::ACL_OWNER_FULL_CONTROL
));

The second step sets the Access Control List (ACL) so that the Origin Identity we previously set up has READ access. This is the step that allows your CloudFront Distribution to read your files from S3.

$acl = array(
	array(
		'id'=>AWS_CANONICAL_ID,
		'permission'=>AmazonS3::GRANT_FULL_CONTROL
	),
	array(
		'id'=>NINJA_STREAMING_CANONICALID,
		'permission'=>AmazonS3::GRANT_READ
	)
);
echo("Update the ACL for $file on S3.\n\n");
$sas = $s3->set_object_acl(NINJA_STREAMING_BUCKET, $file, $acl);

Note that the $acl array of arrays holds the permissions for the owner (indicated by the AWS_CANONICAL_ID constant) and the Origin Access Identity (indicated by NINJA_STREAMING_CANONICALID).

Streaming your Content

Next, you need to set up the webpage that will be used for streaming. All of the files you need to stream to are in the “www” directory. If your project is not in your web root, you will need to either move the “www” directory (and fix the include paths in “index.php”) or use a symbolic link so that you can view that directory in your web browser.

Extending the AmazonCloudFrontClass
In order to make this tutorial work, there is a minor change that needed to be made to the AmazonCloudFront service class in the SDK for PHP. The original AmazonCloudFront service contains a method called get_private_object_url() which is designed to create the signed url requests that are used to serve private content. The method returns a complete signed url with domain name, path and query string. This is a problem because JW Player actually wants the domain name and the path as two separate arguments. To solve this, I extended the AmazonCloudFront class from the SDK and added a method called get_private_object_path() which just consists of the portion of the get_private_object_url() which generates the path. The get_private_object_url() remains the same (except that the logic that was moved to get_private_object_path() was replaced with a call to the new method). You can inspect this work in the “ninja.cloudfront.class.php” file.

Protection Options
There are a few different options for protecting your stream. At a minimum, you need to set an expiration date (called “DateLessThan”) after which the stream stops working. Additionally, you can set a start date (“DateGreaterThan”) and/or an IP Address. If you set the IPAddress, the request must come from that IP Address in order to work.

If you want to make sure that your content can only be streamed from a web page that you control, the best way to do that is to sign the request using the $_SERVER['REMOTE_ADDR'] PHP global. Your arguments in “index.php” would look like this:

$expiration = strtotime('+4320 minutes');  //three days

$opt = array(
//	'BecomeAvailable'=>strtotime('+10 minutes'),  //starts working in 10 minutes
	'IPAddress'=> $_SERVER['REMOTE_ADDR']  //limit to the current IP address (prevents the user from embedding the flash player to serve the content to other users)
);

$cfsn = new AmazonCloudFrontNinja();
$urlcloud = $cfsn->get_private_object_path($item, $expiration, $opt);;

Note: “BecomeAvailable” is the argument you send to the AmazonCloudFront service (the SDK for PHP). Inside the service class, it gets changed to “DateGreaterThan” which is the correct name for the CloudFront API. Since the “BecomeAvailable” parameter is commented out in the code above, the stream would be available immediately.

Load-er Up!

Pull up the “index.php” file in your web browser  - and you should see your videos.  Happy New Year!

Tear-er Down!

Once you’re done working though this tutorial, the “cleanup.php” script will remove all of the Streaming Distributions and OAIs. The process of removing these objects is nearly as complicated as creating themm but I an not going to describe it. The source code has documentation and explains what’s going on. Take Heed: If you’re currently running a live Streaming Distribution, this script will delete that one too.

AWSNinja Code Library

Name: awsninja_cloudfrontjavascript

AWS Services: CloudFront, S3

Source: at GitHub

In a Sentence: A system for optimal JavaScript management and performance in both production and development environments.

Today I’m going to try to answer the question: JavaScript – How do you organize this mess?!? The StackOverflow user who posted this question was primarily concerned with organizing JavaScript classes and namespaces – but many of the replies were about managing and serving script files. Both topics are extremely important, and in this post I’ll be focusing on the latter.

I’ve previously written about managing your site’s images on CloudFront, and this article will basically tackle the same topic for JavaScript. Someday, I will post a third post in this series that deals with CSS.

The Headaches

My headache has JavaScript written all over it.

The headaches that your site’s JavaScript cause can be grouped into two categories, Maintainability Considerations and Performance Considerations. Here are the main points:

JavaScript Maintainability Considerations:

• It’s a lot of code doing a lot of complex stuff. In addition to the functionality you write yourself, you’re including more third-party libraries and plug-ins than ever before.
• You need to manage dependencies. When you include a component on the page, you need to make sure you include the other required components.
• The order that you include the code in your page matters. A dependency needs to appear before the component that depends on it.
• The development environment is completely different from production.
  • In development, your primary concern is finding where the error is, so you want your application to be broken down logically into files and you want the code to be readable for a human being like yourself.
  • On production, you want peak performance which means combining the JavaScript into one file and minimizing the code.

JavaScript Performance Considerations:

• Minimize your code.
• Compress your files.
• Combine your files.
• Get the browser to cache your files (but also make sure the browser isn’t using an outdated version).

These performance considerations (and many, many others) come mainly from the work of Steve Souders. If you’re not familiar with his work in front-end optimizations, go read up. We’ll be here when you get back.

The CloudFrontJavaScript package is my strategy for implementing the best practices in all of these areas. The idea is to keep track of all of your JavaScript files in a couple of database tables, then use a class called the CloudFrontJavaScriptService to manage the resources on CloudFront and generate the SCRIPT tags with the proper source URLs to be inserted into your HTML page.

Before I describe the technical details and how to install and use the package, I am going to highlight a couple of key features of CloudFrontJavaScriptService:

Maintenance Mode

To facilitate website development, the CloudFrontJavaScriptService can be run in Maintenance Mode. In Maintenance Mode, several things are different:

Next time, try flushing your cache.

• JavaScripts are sent to the browser as the original source files. They are not minimized, compressed or combined. This is so that your browser development tools (e.g. Firebug) can work effectively.
• A random string (called “cacheBust”) is appended to the script paths on each page refresh. This is done just to make sure that your web browser does not cache any of your JavaScript, avoiding that frustrating development scenario that goes like this:
  • Adjust your code
  • Refresh
  • See no change
  • Make Another Change
  • Refresh
  • Still no change
  • Make Still Another Change
  • Refresh
  • “What’s wrong with me!?!”
  • Make a change that is absolutely guaranteed to completely break the program.
  • Refresh
  • &%^#(%$)!!!
• The script files are served from your web server, not CloudFront.

By providing Maintenance Mode, the CloudFrontJavaScriptService accommodates the needs of the development environment without adding to the work required to maintain the code and without making any compromises in performance on the production environment.

Compression

One of the major downsides of Amazon CloudFront compared to some other CDNs is that CloudFront will not automatically compress (i.e. gzip) your content for web browsers that support it. Amazon is almost certainly developing this feature and could announce its availability at any time. In the meantime, the CloudFrontJavaScriptService handles this for you. The service does this by storing both the compressed and uncompressed versions on CloudFront. When a page request comes in, it examines the headers for the HTTP_ACCEPT_ENCODING header to determine if gzip is supported. It then returns the URL to either the compressed or uncompressed version of the file.

Versioning

Both CloudFront edge locations and your users’ web browsers are going to cache your content. This is what you want them to do in order to get the best possible performance from your website. However, this creates an issue when your content changes because you can’t fully control when your users will see the updated content.

The CloudFrontJavaScript service handles this by maintaining a version number. The version number is part of the URL of the JavaScript file. The number is reset whenever a new version of your site is deployed. This causes the URL of the JavaScript files to change. Because a new URL indicates a completely new file, both CloudFront and the user’s browser are guaranteed to request the new version. Consider the URL of “script 25” in CloudFrontJavaScriptService:

http://d5zgyj1bk6lvu.cloudfront.net/javascript/1285535650_25.gz.js

After deploying a new version, the URL of this resource might look like this

http://d5zgyj1bk6lvu.cloudfront.net/javascript/1285362461_25.gz.js

The large number is the version number and is also a Unix timestamp. That value matches the time that the version was deployed, ensuring that caching does not prevent users from getting the most recent version of your JavaScript.

The Database

To help you get a sense of the underlying architecture of the service, here are the two database tables that keep track of the details:

The tbl_javaScriptScript table contains the following fields:

• id – The auto-incrementing primary key.
• lookupId – This is a unique number that you will assign to the script file and will be the number you use to indicate which script you’re requesting from the CloudFrontScriptService. I prefer to increment the numbers by five and to make the lookupId be the same as the sortOrder.
• fileName – This is the file name of the script. Each script will appear in your filesystem in two versions (one normal and one minimized). In this field, the filename is displayed in the form to be used for the minimized version of the file. The minimized form will be like this “yahoo_yahoo.js.” In your development environment (where you will actually work the files) this file would appear in “yahoo” subdirectory and would appear like this “yahoo/yahoo.js” The CloudFrontScriptService has a method in it for translating one form to the other.
• dependencies – This field contains a comma-delimited string of lookupIds which represent the dependencies of the script. A typical value in this field would be “25,35,40” where the scripts with lookup ids of “25”, “35”, and “40” are dependencies of the current script.
• sortOrder – This field specifies the sort order the scripts. When the CloudFrontScriptService determines which scripts (including dependencies) it needs to include in your webpage, it will order them in the order specified here. Typically, I like to start out with the lookupIds and sortOrder as the same value. For example, the file with a lookupId of 10 will start out with a sortOrder of 10. If I find out later that things need to be reordered, I can just adjust the sortOrder without the much bigger hassle of rearranging the lookupIds.

The tbl_javaScriptCDN table contains the following fields:

• id – An auto-incrementing id. Not the primary key and not currently used. (The primary key is a composite of scriptIds and gzip.)
• scriptIds – This is a comma-separated list of the scriptIds that were requested. If you request the script with lookupId 25, then the value in this field will be “25”. If you request scripts 25 and 35 then the value will be “25,35”. The dependencies of scripts 25 and 35 won’t be included here even though they would be sent to the browser.
• version – This indicates the version number of the scripts that are currently being served from Cloudfront. If the version number in this row is out of date, the CloudfrontJavaScript service knows to rebuild the script package, upload it to CloudFront and update the version number in this table.
• gzip – This indicates whether the combined JavaScript file is gzipped.

Installation and Usage

Download the package from GitHub and put it on your webserver. You will also need the awsninja_core package.

YUI Components
To demonstrate how the CloudFrontJavaScriptService works with a large JavaScript project, the javascript directory in the package contains the JavaScript files from the YUI Library. If you want to test out the ability of the service to serve files to a web browser in Maintenance Mode, you will need to move the files to a location within your web root and change the CDN_JAVASCRIPT_SOURCE_PATH in config.php to match the new path.

Steps to Get Set Up

1. Update the settings on config.samp.php and rename the file to config.php.
2. Run the sql in the ninja_cloudfrontimages.sql against your database to create the needed tables.
3. In the examples directory, run the yuiStoke.php script from the command line (e.g. type “php yuiStoke.php”) to populate the tbl_javaScriptScript table with the YUI script file information. The yuiStoke.php script uses a dependency tree created by jassem.shahrani to set up the YUI files.
4. Run the deploymentTest.php script from the command line. This will minimize and copy the YUI JavaScript files to the js-minified directory. It will also delete the version file, which will cause the CloudFrontJavaScriptService to create a new version number at the first opportunity.
5. Run the serviceTest.php script to see a demonstration of how the CloudFrontJavaScriptService generates HTML Script Tags.

At step five, you know that you’ve got it set up correctly if you see output similar to this:

The code in the example requests script 25, which is the YUI datasource.js script. This script is dependent on event.js (script 10) which itself is dependent on yahoo.js (script 5). In all of the examples, we use the getScriptHTML() method with the first argument 25. See the serviceTest.php source for more details.

The following uses are demonstrated:

• Under “Here’s the uncombined script” you see that the datasource.js script and it’s dependencies are individually included. The scripts are ordered according to the sort_order field in the database and a random “cacheBust” string is included at the end of the URLs. This is ideal for the development environment, because your code will be properly broken out into files that are not minimized. Your browser development tool (e.g. Firebug) will have no problem directing you to your errors by file and line number.
• Under “Here’s the combined uncompressed script:” you see that there is a single file and it is being served from a CloudFront domain name. The three JavaScript files are combined into one and minimized. In this example, they are not compressed because this is an example of how the scripts would be served to browsers that do not support compression.
• Under “Here’s the combined compressed script:” you see something very similar to the previous one. The difference is that before the “.js” extension there is an extra “.gz” in the path. This is the path to the combined, minimized and compressed file.
• Under “Here’s how to let the web browser determine whether to compress or not:” the script echos out the most common way to use the service to insert script into HTML. Basically, you just give it the script number you want and let the service figure out the best way to serve the script.
• Finally, under “In the current context, that code produces this:” the script gives an example of what the script in the previous example outputs in the current context. The context of this output is the command line, so it sends the combined and minimized (but not compressed) script. The reason it does not send the compressed script is that in the command line context, there is no browser HTTP_ACCEPT_ENCODING header.

How I Organize and Add New Scripts to my JavaScript Project

There is one more thing that I would like to share about how I like to manage my scripts. My general strategy is to separate my third party scripts (third-party libraries and plug-ins) from the scripts that I write specifically for the project that I’m working on. I do this by including a large gap of lookupIds between the third-party components and my own components. Here’s an example from one of my projects where I’m using jQuery and a bunch of jQuery plug-ins:

Note that the third party code (jquery and jquery plug-ins) start at lookupId 1000 and my code (the “marketplace” files) start at lookupId 1500. The gap between 1000 and 1500 provides plenty of room for me to add more third party components as needed. The third-party components will always appear above my components, which is good because the third-party scripts are the dependencies.

When I need to add a new JavaScript file (either third-party or my own component), I add the file to the filesystem, then add the row directly to the database table.

Conclusion:
I hope that you find this method of managing your JavaScript useful, and I hope that I was successful in clearly explaining how it works. As always, I’d very much like to know what you think.  If you read this far, then for gosh sakes leave a comment and let me know!  It makes me happy.

AWSNinja Code Library

Name: awsninja_ec2resourcemanagement

AWS Services: EC2

Source: at GitHub

In a Sentence: A script framework for automating operations on EC2 resources

When I first moved beyond simply playing around with EC2 and made the decision that I was actually going to build and deploy a web application on it, I quickly discovered that it’s next to impossible to keep everything straight.

The reason for this is the heart of the difference between using actual computer hardware and using virtualization. Virtualization is more flexible because it has more parts. That’s also the reason why it is more complicated.

With traditional hosting, you (or your hosting provider) build a server to a specification. This involves capital investment because you have to purchase the hardware and locate it somewhere. Once you’ve bought the hardware, you are stuck with it. Changes to your configuration require new capital investment and may result in unwanted legacy components in your possession. This costs time and money.

EC2 and other cloud-hosting services exist mainly for the purpose of eliminating these burdens. But as is often the case, elimination of one problem creates a new one. Since the problems associated with virtualization are newer, developers are not as adept at dealing with them yet. We’re still learning.

The biggest problem is essentially this: The power to rearrange your hardware in real-time means that you now have much more to keep track of. With traditional hosting, you were constrained by your inability to quickly add disk storage, RAM, CPU, or whatever. You also may have been constrained in terms of your ability to quickly add new software.

Those constraints are now gone:

• Need more hard disk space? Add it in five minutes.
• Need a backup? Take a snapshot.
• Running low on RAM? Spin up a high-memory instance and you’re back in business.

But virtualization has created new questions, which you may not have answers for:

• How will you keep track of that new volume you created?
• How can you find that snapshot you created six weeks ago?
• What’s the Instance-Id of that high-memory instance you started yesterday?

Here is a partial-list of EC2 objects you will need to track:

• Your Amazon Machine Images. Each different type of server you use in your application will have an AMI. Additionally, you may have multiple versions of those AMI’s that you need to keep track of. More about this below.
• Your Instances. Traditionally you will have a Development, Staging and Production environment, each consisting of one or more instances. For instance you may have web and database servers in each environment. That’s at least six instances right there.
• Your Elastic Block Store Volumes. EBS volumes provide wonderful flexibility for managing your data stores. You may have volumes for your database, application root or for storing files such as images or PDF files. Wonderful, if you can keep track of them.
• Your EBS Snapshots. The most common use of EBS Snapshots is to make backups of your EBS volumes. You might have hundreds of these. You may also need a plan for purging them when the time is right.

How I Organized my EC2 Resources

For my applications, I’ve settled on using SimpleDB for the purpose of keeping track of all of my EC2 resources. This has several advantages:

• It’s free. The free tier of SimpleDB is more than adequate for this purpose.
• It’s always on. This is very important. See below.
• It’s Amazon. If you’re familiar with other AWS services (ahem – why else would you be reading this?), you have a leg-up in learning SimpleDb.

The “Always On” characteristic of SimpleDb is extremely useful for our purposes here. It would make sense to keep track of your EC2 resources in a relational database like MySQL, except for the fact that you probably want to run your MySQL database on EC2. There’s a chicken and egg problem in here. Suppose you want to start up your MySQL instance? If the Instance ID and other information is sitting in your MySQL database, which isn’t running yet. . . . you see what I mean.

By keeping your EC2 resource information on SimpleDB, you can use it even when none of the resources of your application are running by running a SimpleDB client like SimpleDBAdmin or SimpleDB Explorer.

Another advantage of using SimpleDB to store information about your EC2 resources is that it makes it possible for your EC2 instances to identify themselves at boot time by running a boot script that queries SimpleDb. This is an elegant and seamless way to start a new instance. You can include a boot script which checks for its own EC2 Instance ID, looks itself up in SimpleDb and then automatically configures all of the EBS Volumes and other resources that belong to it. The EC2ResourceManagement package includes an example of this.

The Code

The EC2ResourceManagement library consists of a few different services and two types of classes. The EC2Service and EC2Classes interact with the Amazon EC2 API. The EC2Classes represent objects like instances, volumes, AMIs and snapshots that exist in your EC2 account. The EC2ManagementService and the SimpleDBService interact with SimpleDB. The classes used by the EC2ManagementService (found in the classes directory of the library) represent records that are stored in SimpleDB and which are designed to represent EC2 objects and to keep track of how they are intended to be used.

Getting Started

To get started, download and unpack the awsninja_ec2resourcemanagement and awsninja_core packages. You’ll also need an AWS account with EC2 and SimpleDB enabled.

Add your credentials to the config.samp.php file and rename it to config.php. Then try running a few of the scripts found in the examples directory (these are designed to be run from the command line).

The following example scripts are included:

• buildEC2ResourceInventory.php – This is a good script to start with. This script will examine your EC2 account and find the Images, Instances and Volumes that belong to you and will create records in SimpleDB to help you keep track of them. To do this, it will ask you a few questions about what all of the different objects are for. You’ll have better luck if you know the answers.
• attachVolume.php – This is a simple script that shows how to attach a volume to an instance. You give it the Volume object, and the script will determine which instance to attach to and what device to use by looking up the details in SimpleDB.
• backups.php – This script will make snapshots of your Volumes and create Snapshot records in SimpleDB. Perfect for backing up your data – and keeping track of what your snapshots are for.
• cloneVolume.php - This script will clone a volume and assign it to whatever instance you want. For instance, if you wanted to clone the MySQL database volume on your development server so that you could use an exact copy on your staging server, this script will do that. Clones are accomplished by creating a snapshot, then creating a volume from that snapshot. A record for your new Volume is created in SimpleDB and the new Volume becomes one of your managed EC2 resources.
• launchInstance.php - This will launch an instance from one of your images. You only need to specify an Image and a role. The role is something like “development,” “staging,” or “production.”
• listInstances.php – This simply lists your managed instances. Simple enough.

If you’ve got everything configured, you can run the buildEC2ResourceInventory.php to get your environment to create SimpleDB records for your Images, Instances and Volumes. The script is intended to be run from the command line and will prompt you to answer some questions about your Images, Instances and Volumes.

When you run the buildEC2ResourceInventory.php, here are the questions you will get:

The answer given here is "WebServer." The idea is to describe the purpose of the AMI that is being asked about. Your " WebServer" could be launched into instances in multiple roles (i.e. Development, Staging, Production) - but presumably all of these instances would use the WebServer image.

Next, it asks about your Instances:

This question (asked regarding two instances in the image above) inquires about he role of individual Instances you are running. I have answered that one is the Development and one is Staging.

Then, it asks about your EBS Volumes:

Next, the script inquires about our volumes. It asks the role of the Volume ("application" and "database" are the answers because these volumes hold the application root and the database files), then it asks what path on the Instance that these volumes should be mounted to. Note that the script does not ask what instance the volume is attached to or what the device address is. It gets that information from the EC2 api and will automatically configure those values.

Once the script is finished, you can look in your SimpleDB domains to see how your EC2 resources are mapped to records in SimpleDB

Final Note

The scripts in this package are not intended to be a turn-key solution for getting your stuff organized. You will need to adjust and build on them for your own projects. The main idea here is to provide you with a foundation on which you can start keeping track of your objects and automating operations on them to save time, reduce errors and increase the overall quality of the service you provide.

I recently had an issue on DocMonk when the script that sends out the emails to the PDF recipients was crashing.  After checking it out, I discovered that the crashes coincided with the startup of another script which handles backend aggregating for the reporting pages.  I was initially skeptical that this other script could really be causing such a problem.  I decided to try to see what else was going on on the server.  I started CloudWatch Monitoring on the server and went away for a few hours.

When I came back, this is what the CPU Utilization Chart looked like:

Some Spiky CPU

Those spikes were happening every 15 minutes, which happens to be the frequency of the reportAggregator.php script – the reporting script I originally suspected.

Armed with new certainty that I was on the right track, I set about diagnosing the problem.  One SQL query was running very slowly, but only under certain conditions.  It looked like this:

SELECT * FROM tbl_pdfRecipient WHERE pdfOrigialId=:pdfOriginalId ORDER BY ID ASC LIMIT 0,100

Assuming you’ve got intelligent indexes on the table, this is a very fast query, but change the limit clause to “LIMIT 100000, 100″ an it will be much, much slower. If you’re going to be processing a lot pages of data, this is a very inefficient way to do it.

The solution was to query the minimum and maximum id values of the rows I’m querying for, then step through the pages by comparing the id values to ranges within the max and min range of the entire set.  The following queries demonstrate:

SELECT min(id), max(id) FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId
/* Assume :min=1 and :max=1534.  We will grab these pages in groups of 100*/
/* Here's the efficient way to get all of the records */
SELECT * FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId and id BETWEEN 1 AND 100;
SELECT * FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId and id BETWEEN 101 AND 200;
SELECT * FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId and id BETWEEN 201 AND 300;
SELECT * FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId and id BETWEEN 301 AND 400;
SELECT * FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId and id BETWEEN 401 AND 500;
SELECT * FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId and id BETWEEN 501 AND 600;
SELECT * FROM tbl_pdfRecipient WHERE pdfOriginalId=:pdfOriginalId and id BETWEEN 601 AND 700;
/* and so fourth. . .*/

With this method, the 10,000th page would be retrieved just as quickly as the first.  Here’s the PHP code from the reportAggregator application.  Note that the SQL queries are not shown because they are abstracted away to other parts of the application.  This code is intended to show how to handle the looping and the startId and endId for the queries.

$pageSize = 100;  //whatever your page size will be

//Initialize the start and end id values
$minId = PDFTO_domain_PDFRecipient::findMinByPDFOId($pdfo->getId());
$maxId = PDFTO_domain_PDFRecipient::findMaxByPDFOId($pdfo->getId());

//Initialize the first $startId and $endId
$startId = $minId;
$endId = $startId+$pageSize-1;

//Get the first page of results
$pdfRecpts = PDFTO_domain_PDFRecipient::findByPDFOIdAndIdRange($pdfo->getId(), $startId, $endId);
while($startId <= $maxId)  //See how we will know when to stop it?
{
	foreach($pdfRecpts as $pdfRecpt) //Loop though the members of the page
	{
		/* DO SOMETHING WITH THE $pdfRecpt */
	}
	//Done with page, now set $startId and $endId for the next page
	$startId = $endId+1;
	$endId = $startId+$pageSize-1;

	//Run the next query - (this is at the bottom of the while loop because the first instance happens before we entered it)
	$pdfRecpts = PDFTO_domain_PDFRecipient::findByPDFOIdAndIdRange($pdfo->getId(), $startId, $endId);
}
//All done!

A few hours later after deploying this change, here what my CPU Utilization Chart looked like:

All Fixed

Much better, I think.

AWSNinja Code Library

Name: awsninja_simpledbadmin

AWS Services: SimpleDB

Source: at GitHub

In a Sentence: A web interface for Amazon SimpleDB.

Amazon SimpleDB is a highly available, scalable, and flexible non-relational data store that offloads the work of database administration. Developers simply store and query data items via web services requests, and Amazon SimpleDB does the rest.

Why SimpleDBAdmin is Needed

I started building SimpleDBAdmin because I wanted a package to administer my SimpleDB data that met the following criteria:

• Web-based - so it could be used by anybody anywhere
• Free – for obvious reaons
• Intuitive to use

There are a couple of web-based SimpleDB managers out there, but I found them to be incredibly difficult to use for actually manipulating data. They seem to be designed for the purpose of teaching basic SimpleDB concepts to programmers.  That’s a laudable goal but it’s not what I was looking for.  AWSZone and Amazon’s own JavaScript Scratchpad expose all of the intricacies of the SimpleDB API and require you to perform actions like “Invoke Request” or use “PutAttributes.”  Rather than deal with all that, I wanted a package that provides a higher level of abstraction and focuses mainly on letting the user view and manipulate data. SimpleDBAdmin is designed to meet that need.

How it was Built

Because I wanted to make it possible to use the application on any platform, the vast majority of the application is written in JavaScript and based on JQuery.  In addition to JQuery and JQuery UI, the following packages are also used:

• The AWS Query Signer from SonaCWS
• Chris Domigan’s Context Menu plug-in for JQuery
• Carhartl’s Cookie plug-in for JQuery
• The xmltojson php library published on IBM developerWorks.

Also included is a PHP component called relay.php that handles relaying the API commands to SimpleDB.  This is needed because the JavaScript Same Origin policy prevents a JavaScript application from sending requests to a separate domain name.  The relay.php component receives the completed request from the SimpleDBAdmin JavaScript package, passes it on to SimpleDB, receives the XML response from SimpleDB, converts it to JSON and passes it back to the JavaScript package.

Note: Your AWS Secret Key is not sent to the relay.php script.  Instead, the signed request is created in JavaScript and sent to relay.php.  This prevents your Secret Key from being exposed in the internet traffic between the browser and server.

SimpleDBAdmin Features

• Supports saving multiple sets of AWS account credentials, useful if you manage SimpleDB data on multiple accounts.
• Secret keys are encrypted and saved as cookies protected by a simple password.  You can access your SimpleDB account from the same computer without having to enter the AWS Access and Secret keys every time.
• An elegant interface for storing, modifying and removing multiple values in the same attribute name.  Read about SimpleDB’s data model if you’re unfamiliar with SimpleDB’s ability to hold multiple values with the same attribute name.
• Results are presented in a table format and are pageable.
• Easily add and remove SimpleDB Domains.

Limitations and To-dos

• No Sorting – Currently, Domain contents are returned in SimpleDB’s default order.  I believe that this is the same as the chronological order that the items were added, but not sure if this is aways true.
• No Searching – The ability to search the items has not been added yet.
• Large Data – Domains with a large number of attribute names or large attribute values will not display optimally in the web browser.
• No Counts – The interface does not indicate the number of items in each Domain.

Installation

To install SimpleDBAdmin, download the package from GitHub and place the contents somewhere on your server’s web root.  Then navigate to the index.html with your web browser.  When you get it set up, it looks like this:

Login Dialog

The Login Dialog

If it doesn’t work, you may need to adjust some of the paths within the index.html, relay.php or SimpleDbAdmin.js.

Usage

Once you’ve got the package installed and have the dialog shown above displayed in your browser, you then must enter your SimpleDB credentials.  To do this, fill in the form shown above with the following values:

• Name – This is just the name that you want to give your account.  If you manage multiple SimpleDB accounts, this will help you keep track of them.
• Access Key – This is your AWS Access Key Id, which can be found from the AWS Credentials page when you are logged into the website.
• Secret Key – This is the AWS Secret Key associated with the Access Key and also found on the AWS Credentials Page.
• Password – This is a password chosen by you.  When you return to SimpleDBAdmin from the same web browser, the accounts you created will be available to you if you can enter the password.  The password is used to encrypt your Secret Key, so that it is not stored in plaintext in your browser’s cookies.

Once you’ve entered your credentials, you should see your SimpleDB Domains (if any) listed on the left side of the screen.  You can click any of those Domains to browse through the data.  You can right click for the option to Delete a Domain (you will be asked to confirm this action).  You can also add a new Domain by clicking the “Add Domain” button.

Browse Data

Browse Data

You can edit an item row by clicking on it.  Doing so brings up the Edit Item dialog box:

Editing Data

Editing Data

Clicking the “Add Attribute Name” link will open a dialog that allows you to enter another name-value pair to the record.  When you are done making edits, click “Put” to save your changes.

Online Sample

You can give the application a try here:

http://samples.awsninja.com/awsninja_simpledbadmin/

Please don’t use the sample application to manage live web applications or to view or manipulate any sensitive data.  The application is not configured to examine or log your data -but you should install your own copy of SimpleDBAdmin to be completely secure.

If you’d like to install a copy on your own server, download it from GitHub.

I welcome questions, comments or suggestions.  Thanks for reading.