To my surprise, this actually got a response. Someone monitoring the @awscloud account opened a trouble ticket to my email address asking for clarification. The exchange was friendly and hopefully, and I think it’s worth sharing here.

It’s a pretty compelling situation: cloud service offerings and web browser technology have advanced to the point where S3 and CloudFront should be all one needs to deliver an incredibly performant and cost-effective user experience, letting small startups compete in the time-to-first-render game on an even playing field with the likes of Google and Yahoo. Instead, developers are forced to settle for ugly workarounds and outright hacks due to a few crucial shortcomings.

My team at Boundless is has been working on a cutting edge single-page HTML5 app. We are hosting it on S3 and CloudFront, and its underlying API lives on EC2. Without getting into too much detail, the architecture is a lot like #newtwitter.

1. S3 Restricts Response Headers

Despite initial appearences, and without much justification from Amazon, the S3 API severely restricts which headers can be attached to an object.

• Cache-Control
• Expires
• Content-Disposition
• Content-Type
• Content-Language
• Content-Encoding

Users can apply their own metadata, but it will always be prefixed with x-amz-meta. CSS3 brings the ability to embed arbitrary fonts on the web. Fonts are the clothes words wear, and CSS3 is why the web is looking so sharp lately. The difficulty is that W3 puts fonts under a same-origin restriction. Thus, embedding these fonts requires these additional headers:

• Access-Control-Allow-Headers
• Access-Control-Allow-Origin

And the complete CORS specification has yet more headers to contend with:

• Access-Control-Max-Age
• Access-Control-Allow-Methods
• Access-Control-Allow-Credentials

This leaves CloudFront users who wish to embed fonts with a handful of undesirable options.

1. Serve the entire domain through CloudFront. This is fine unless there’s anything on your domain which shouldn’t be cached, and I’m sure things get even more complicated if you throw SSL into the mix.
2. Skip S3 and serve everything from EC2. S3 has eleven nines of durability. Go ahead, reproduce that with a couple of NginX boxes.
3. Insert some proxy servers to add the headers. I think what you mean is, add yet another hop in your network while increasing your attack footprint and your EC2 bill.
4. Mix fonts into stylesheets using data: URIs. Now every time you adjust a <div> tag, your visitors have to download your fonts again. You could break your CSS into multiple files, but this is in direct opposition to one of the tenants of website optimization: minimize the number of HTTP requests. Also, 7-bit encoding means your fonts are now 37% fatter on the wire.

Here is a forum post from 2009 bringing this to Amazon’s highly dismissive attention. What really irks me about this is that Amazon chose to bless a few headers instead of letting the end-user decide what is best for our customers.

2. S3 and CloudFront Won’t Compress Anything

RFC2616 allows that a client may suggest to a server that it would like to have the response encoded as something other than raw bytes before transmission. One common encoding is gzip, and lots of HTTP traffic includes a header like Accept-Encoding: gzip. Most web servers will comply with this suggestion, reducing plain text like HTML, CSS, and JavaScript by well over 50%.

Two notable exceptions to “most web servers” are S3 and CloudFront. A possible workaround involves Content-Encoding being among the allowed HTTP headers for S3 objects. The image below has been compressed with gzip -9 before uploading, and has Content-Encoding: gzip set in S3.

If you can see Success Kid, this hack will work on your browser.

This exploits the fact that most browsers usually send Accept-Encoding: gzip, or they will handle Content-Encoding: gzip in the response even if they didn’t request it. Users of IE7 and previous versions will see a broken image icon. wget or curl will also result in corruption unless those tools are explicitly configured to always use compression. This is really a quasi-violation of RFC2616 Section 14.3, but it does sort of work.

If you want to be compliant, you most choose between S3 and compression. CloudFront, at least, will cache both compressed and raw versions of each object depending on the clients which have requested it.

3. CloudFront’s TCP Stack Lacks Tuning

I have harped on this issue before, but Amazon CloudFront exhibits one of the smallest initial TCP congestion windows in the CDN marketplace. They’re at 2. Consensus is growing that it should be closer to 10. Rather than making an argument for it here, I’ll let some Googlers do it for me.

An Argument for Increasing TCP’s Initial Congestion Window”, Nandita Dukkipati, Tiziana Refice, Yuchung Cheng, Jerry Chu, Tom Herbert, Amit Agarwal, Arvind Jain, Natalia Sutin, ACM SIGCOMM Computer Communications Review, vol. 40 (2010), pp. 27-33.
[ccr.sigcomm.org] [pdf] [search]

The @awscloud guys are apparently considering my request. If I get any answers back, I will be sure to post them here.

Update - May 12, 2012

Amazon’s Senior Evangelist for AWS @jeffbar has reached out to me to clarify one point about my post.

There you have it. CloudFront definitely deserves credit for upping the amount of content websites can render before round trip latency starts getting in the way.

In the eyes of those lovers of perfection, a work is never finished—a word that for them has no sense—but abandoned; and this abandonment, whether to the flames or to the public (and which is the result of weariness or an obligation to deliver) is a kind of an accident to them, like the breaking off of a reflection, which fatigue, irritation, or something similar has made worthless.

What’s the right amount of time to stay at a startup? My first startup (Livewire Mobile, née Groove Mobile) was the fastest and most eye-opening 18 months of my life. Before that was six years of running IT for The Real Estate Company, which was at the time the largest independently owned real estate agency in southeastern Massachusetts. That company was founded the year I was born and is still operating today. In my memory, Groove and T.R.E.C. occupy about the same amount of space.

Ultimately my four years at LocaModa was just the right amount of time. I am immensely proud of what I built while there. Some of it is open source (on my github account, with more to be released later on theirs). In a later post I may write about some of the cool work we did with Solr, Chef, ActiveMQ, Varnish, HAProxy, and Nagios. There’s a lot to be excited about happening over there, but that will be for others to ship, not me.

Try it out here.

To complete the picture, you must also structure your permalinks like Blogger does.

1

permalink: /:year/:month/:title/

I’m still looking for the right way to get Cache-control and Expire headers from s3cmd sync. Without that, the best YSlow score you can get on a default Octopress blog is about a C (76). CloudFront doesn’t do any transparent compression for you (no Accept-Encoding support) so that’s a no-go as well. It would also be nice if Amazon would acknowledge that initcwnd of 2 is nonsensical on today’s Internet. Nearly all other CDNs use something higher, with Akamai using up to 6, and Google using 10.

See Initcwnd settings of major CDN providers for a comparison.

Here’s the documentation I submitted along with the (comparitively small) patch.

Amazon Web Services (AWS) is a collection of web services that together make up a cloud computing platform, offered over the Internet by Amazon.com. The most central and well-known of these services are Amazon EC2 and Amazon S3. This page will walk you through setting up a blog on S3.

If you haven’t already, go sign up for an Amazon AWS account right now. It’s very inexpensive and you only pay for what you use.

You will need to install s3cmd. This is available in most Linux distributions, via Homebrew, or directly from s3tools.org. You’ll need to copy & paste access_key and secret_key from the Security Credentials page into s3cmd’s configuration wizard.

1

s3cmd --configure         # Begin interactive configuration

When finished, your ~/.s3cfg file should end up looking something like this.

1
2
3
4
5

[default]
# ...snip...
access_key = ALRJCALDDZUDEGNSSIPE 
secret_key = 9DJWHga1Y+uBAFXntDM1Ujd6FrnnUZb/9dLMOqzn
# ...snip...

S3 is really just a huge, fast, and very reliable, key-value store which happens to be accessible over HTTP. It doesn’t recognize index.html as being special (or even the concept of directories, really) unless you enable “website hosting.”

Unfortunately, the latest version of s3cmd doesn’t let you do this yet, so you’ll have to use the AWS Console. Create a bucket having the same name as the fully-qualified domain name of your blog (e.g., blog.example.com). Open the Properties pane for your bucket, and then the Website tab to turn this on. Make note of the “endpoint”, something like blog.example.com.s3-website-us-east-1.amazonaws.com. You can use this directly, but most people will create a DNS CNAME pointing to it instead.

In your Rakefile, set the deploy default to s3 and configure your bucket.

1
2

deploy_default = "s3"
s3_bucket      = "blog.example.com"

Now if you run

1

rake deploy     # Syncs your blog to S3

in your terminal, your public directory will be synced to your bucket.

(In other words, testing was successful.)

But what about the new hotness, Varnish?

That’s what. Save this to /etc/varnish/newrelic.h and include it in your vcl_recv declaration.

We use the excellent Varnish HTTP accelerator in front of our application servers. True, we could easily be serving crossdomain.xml from Tomcat, or Apache, or NginX, but Varnish is the only part of the infrastructure I’m really happy with for the long-term. This is why I decided to serve a static file from Varnish which, officially, doesn’t do this.

“Error 843? What the hell is that?” vcl_recv is intercepting requests for crossdomain.xml and returning an illegal, non-standard, stupid error. 843 is meaningful to those familiar with Flash socket connections as the TCP port number for socket policy requests. (I wrote a server for this a while ago.)

Having had an opportunity to take better notes on this process for my employer, I present to you a few brief commands to make renewing your SSL certificates almost painless.

1. First, check your expiration date.
   
   

   openssl s_client -connect host01.example.com:443 < /dev/null 2> /dev/null |\
      sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' |\
      openssl x509 -enddate
   
   

2. Renewal time is an great opportunity to rotate your key. This is technically optional but it’s good practice:
   
   

   openssl genrsa -out example_com.key 1024
   

3. You’ll need both the key and the about-to-expire certificate in the current directory for the next step. This will create a new CSR with all of the same parameters you used last time. Take this over to your preferred certificate authority and begin their renewal process.
   
   

   openssl x509 \
       -x509toreq \
       -signkey example_com.key \
       -in example_com.crt
   

4. The Java keytool command doesn’t offer any way to import an x.509 key by itself. The workaround is to first merge the x.509 certificate and key to a new PKCS12.
   
   

   openssl pkcs12 \
       -export \
       -in example_com.crt \
       -inkey example_com.key \
       -out example_com.p12
   

5. Now use keytool to convert that PKCS12 file into a Java keystore file.
   
   

   keytool \
       -importkeystore \
       -deststorepass changeme \
       -destkeypass changeme \
       -destkeystore keystore.jks \
       -srckeystore example_com.p12 \
       -srcstoretype PKCS12 \
       -srcstorepass changeme \
       -alias 1
   

mysql --batch information_schema -e "select concat('ALTER TABLE ', TABLE_NAME,' ONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;') as '--' from TABLES where TABLE_SCHEMA='somedb' and TABLE_COLLATION not like '%utf8%'" | mysql somedb

The addition of INFORMATION_SCHEMA combined with CONCAT makes it possible to do arbitrarily cool things with MySQL. Love it.

Superman, The Jetsons, James Kirk and a black and white TV told me things, beautiful things. Adventure and possibility beyond imagination. Perfect worldwide peace without disease or poverty. Ubiquitous, effortless flight. Probably, a black president with a stage presence like Morgan Freeman. Above all, I expected to see the limitless reach of human exploration and expression. I want to talk about that last point for a second.

Imogen Heap confirms for me that the future has not only already arrived, but has delivered fully on one promise. Here we see a young woman controlling a fantastic array of electronic equipment. Any single component, if presented to NASA’s best engineers in 1969, would have seemed impossibly sophisticated. She, alone on stage, uses this future artifact not to explore space or design pharmaceuticals, but to project without restraint her imagination onto our consciousness.

That’s the highest quality recording I can find on YouTube. I strongly suggest you go support this artist in any way you can, and with that being said, here’s my favorite rendition of Hide and Seek.

This movie, built with data collected during the European Space Agency’s Huygens probe on Jan. 14, 2005, shows the operation of the Descent Imager/Spectral Radiometer camera during its descent and after touchdown. The camera was funded by NASA.

The article doesn’t do justice to how much beauty exists in this visualization of data. The brief and glorious life of the Huygens lander, dropped from the Cassini probe into the atmosphere of Titan, is compressed from 4 hours to just a few minutes. A unique chime as if from a child’s xylophone accompanies each sensor reading. As Huygens’ last meters close, the final intrusion of poetry into science is utterly disarming.

If anyone can find a higher resolution form of this video it would be greatly appreciated.

Unlike Spirit and Opportunity, Sojourner could not directly communicate with Earth and had to rely on the lander to transmit messages about its health and travels. So when contact with the lander, which was designed to last one month, was lost after three months, ground controllers were not sure what became of Sojourner.

Now, it seems the rover kept moving, apparently trying to reach its companion. “The rover was programmed so that if it didn’t get commands from the lander, it would assume it somehow got out of radio contact behind a ridge or rock,” Parker told New Scientist. “So it would drive to the lander as best it could and keep trying to re-establish contact”, circling it as it got close, he says.

When it was last seen, the rover was 13 metres away from the Pathfinder lander. Now, it appears to be about 6 metres away, according to the new MRO images. “I think the simplest explanation is it started to drive back, got about half way, and stopped for whatever reason - it may have thought it got there,” says Parker. “The other possibility is it drove around and around for who knows how long and simply failed at that location.”

curl http://jelder.s3.amazonaws.com/git-1.6.1-leopard.tar.gz | sudo tar xvzf - -C /

This will put git in /usr/local/bin/git (with other support files in /usr/local/libexec and /usr/local/share). The nub script in /usr/local/bin/nub will launch GitNub from the current directory.

Most of my Mint’s value for us users comes from knowing how our money is getting spent, but one critical part of that use case is missing: a workflow for rapid categorization.

Mint needs a really slick way to let me sit down once a month and quickly categorize all of my uncatagorized transactions. I should be presented with a queue of transactions from unknown merchants and some method for applying labels, splitting bills, etc. Ideally, I should be able to keep my hands on the keyboard for this, and it should take like 5 minutes a month or less.

This can be done manually now, of course, but it’s far from efficient. The ajax stuff in the transactions tab has improved but it’s still too chunky and slow on a modern MacBook Pro and Firefox 3.0. The experience should be less hunt-pick-wait and more next-next-next.

With more complete vendor preference profiles for each user, Mint should be able to monetize via more focused opportunities. Good luck guys! Mint is great but it’s not complete! The thread is here. ]]> 2008-12-29T22:44:00Z http://blog.jacobelder.com/2008/12/git-is-exquisite Possibly the best thing about git is that you can start from an existing directory. This makes it amazing for tracking configuration files.

cd /etc/ldap ; git init ; git add . ; git commit -m “got some config religion”

Too bad one can’t use GitHub for this, but remote backups and batch config changes get really elegant with kind of setup.