Ken's Blog

AutoScaling Amazon SQS Queue Processors

2011-12-14T10:20:00.000-08:00

One of my favorite things about running servers in Amazon EC2 is the ability to use AutoScaling to automatically add and remove nodes as web traffic increases and decreases. Not only does this generally save money, but it also helps prepare a system to handle traffic spikes.

Recently I had some fun setting up AutoScaling for a different use case -- a cluster of machines processing messages on an Amazon SQS queue. The idea was to add and remove nodes as the number of visible messages on the queue fluctuated. Again, this keeps costs lower by only running as many nodes as are necessary to process the current workload and handles workload spikes.

Here are sample steps and commands that you can use for setting up AutoScaling for SQS queue processors:

Create an AMI for a node that will process SQS messages. The node that boots from this AMI should automatically launch one or more queue processing processes. A user-data script may be useful for this.

Create a Launch Configuration for the queue processor node:

as-create-launch-config MyQueueConfig --image-id [INSERT YOUR AMI ID HERE] --instance-type c1.medium --key [INSERT YOUR KEYNAME HERE] --user-data [INSERT YOUR USER DATA SCRIPT HERE]

Create an AutoScaling group for queue processors:

as-create-auto-scaling-group MyQueueGroup --launch-configuration MyQueueConfig --availability-zones us-east-1b --min-size 1 --max-size 10

Create Policies that add/remove 1 node to/from the cluster. They will be invoked when the number of messages on the queue grows excessively high or decreases to an acceptable level:

as-put-scaling-policy MyScaleUpPolicy -g MyQueueGroup --adjustment=1 --type ChangeInCapacity

as-put-scaling-policy MyScaleDownPolicy -g MyQueueGroup --adjustment=-1 --type ChangeInCapacity

Create Alarms to scale up/down when the number of messages on the queue grows excessively high or decreases to an acceptable level. Use the Policy ARN's returned by the previous as-put-scaling-policy commands.

mon-put-metric-alarm --alarm-name MyHighMessagesAlarm --alarm-description "Scale up when number of messages on queue is high" --metric-name ApproximateNumberOfMessagesVisible --namespace AWS/SQS --statistic Average --period 60 --threshold 1000 --comparison-operator GreaterThanThreshold --dimensions QueueName=MyQueue --evaluation-periods 10 --alarm-actions [INSERT MyScaleUpPolicy ARN HERE]

mon-put-metric-alarm --alarm-name MyLowMessagesAlarm --alarm-description "Scale down when number of messages on queue is low" --metric-name ApproximateNumberOfMessagesVisible --namespace AWS/SQS --statistic Average --period 60 --threshold 100 --comparison-operator LessThanThreshold --dimensions QueueName=MyQueue --evaluation-periods 10 --alarm-actions [INSERT MyScaleDownPolicy ARN HERE]

In this example the Alarms cause the cluster to scale up when the number of visible messages on the queue remains above 1000 for 10 consecutive minutes and scale down when the number of visible messages falls below 100 for 10 consecutive minutes.

The Policies above adjust the amount of nodes in the cluster by a specific amount, but it is also possible to specify the adjustment in terms of percentages. Using --adjustment 10 --type PercentChangeInCapacity would adjust the amount of nodes by 10 percent.

It also would have been possible to base scaling activities on other AWS/SQS metrics such as:

NumberOfMessagesSent
NumberOfMessagesReceived
NumberOfMessagesDeleted
NumberOfEmptyReceives
ApproximateNumberOfMessagesVisible
ApproximateNumberOfMessagesNotVisible
ApproximateNumberOfMessagesDelayed
SentMessageSize

Here are a few online references relevant to AutoScaling SQS queue processors:

Happy SQS AutoScaling!

Online Advertising Formula Reference

2011-10-20T15:07:00.000-07:00

Occasionally I need to manipulate online advertising formulas to make a quick calculation. The following serves a quick cheat sheet for this purpose. Note that the term Cost is from the perspective of the Advertiser. The web site or publisher can translate that into Revenue.

CPM Campaigns

Cost = Impressions / 1000 * CPM

CPC Campaigns

Cost = Impressions * CTR * CPC

CPM = CTR * CPC * 1000

CPA Campaigns

Cost = Impressions * CTR * AR * CPA 

CPM = CTR * AR * CPA * 1000

Notation	Description
CPM	Cost Per 1000 Impressions
CPC	Cost Per Click
CPA	Cost Per Action
CTR	Click Thru Rate (Clicks / Impressions)
AR	Action Rate (Actions / Clicks)

Giant Ampersand - Washington/Centinela Redevelopment Part 6

2011-07-18T21:53:00.000-07:00

It's been almost 5 years since my last post about the corner lots of Washington and Centinela which are supposedly destined for redevelopment. That's because pretty much nothing has happened on those lots - until this past week.

All of the sudden a 8-foot tall white wooden ampersand appeared in the middle of the lot on the north east corner. I had no idea what it meant, but I was really happy to see it. To me it meant that someone out there is thinking about this vacant plot of land.

A little Googling produced an answer. The mysterious ampersand is a public project by Los Angeles-based artist Shannon Ebner. According to the LAXART web site:

The ampersand is a conjoiner; it signals an incomplete thought and acknowledges that there is more to be said.

This is part of a Shannon Ebner exhibition at the Hammer Museum which started July 16, 2011. Apparently this isn't the first time she's used an ampersand in her work. See this photo from a year ago in Milan.

I've been waiting for the redevelopment on this corner to get underway since I moved to this Del Rey neighborhood 11 years ago. I really hope, as the giant ampersand acknowledges, that there is more to be said...

Using a Canon Pixma MX882 All-In-One with Ubuntu 10.10

2011-04-26T21:51:00.000-07:00

Are you thinking about buying a Canon Pixma MX882 Wireless Office All-In-One Printer and wondering if it will work with your Ubuntu computer? Although Canon does not make this clear in any way, the answer is YES!

Instead of using software that comes on a CD with the printer, you'll have to download and install drivers yourself. They are not easy to find if you live in the US since they're only available on European and Asian Canon websites. I'll tell you how to download and install them.

Printing

Finding and installing the drivers

Visit the Linux IJ Printer Driver page on the Canon Europe web site.
Click the Linux IJ Printer Driver MX880 series.tar link in the Downloads section to download the 7299.5 kb tarball.
Inside the tarball, find and unpack cnijfilter-mx880series-3.50-1-deb.tar.gz.
Inside the resulting cnijfilter-mx880series-3.50-1-deb/packages directory, locate and double click cnijfilter-common_3.50-1_i386.deb and cnijfilter-mx880series_3.50-1_i386.deb in that order. Ubuntu will guide you through the installation.

Note: The guide that came with this driver asks you to run the install.sh program, but it didn't work for me. When I ran it, I got an error: An error occurred. The package management system cannot be identified.

Adding the printer

From Ubuntu's menu, select: System, Administration, then Printing.
From Printing window that opens, select Add.
After a few seconds, you should see a Canon MX880 printer show up under Network Printer.
Select it and click Forward.
Finish entering your printer Name, Description, and Location.

Scanning

Finding and installing the drivers

Visit the Linux IJ Scanner Driver page on the Canon Europe web site.
Click the Linux IJ Scanner Driver MX880 series.tar link in the Downloads section to download the 3072 kb tarball.
Inside the tarball, find and unpack scangearmp-mx880series-1.70-1-deb.tar.gz.
Inside the resulting scangearmp-mx880series-1.70-1-deb/packages directory, locate and double click scangearmp-common_1.70-1_i386.deb and scangearmp-mx880series_1.70-1_i386.deb in that order. Ubuntu will guide you through the installation.

Scan Gear MP

With the drivers, an app called Scan Gear MP is also installed. This app serves as an interface to your scanner and has a Basic Mode:

And an Advanced Mode:

You can launch the app from the terminal by typing scangearmp and it is also available within Gimp by selecting File, Create, Scan Gear MP...

I have not yet figure out a way to use the scanner via XSane. I'd love to hear comments about whether this is possible and how to do it. Without XSane support, I can't use familiar programs such as SimpleScan and gscan2pdf.

VueScan

One more thing to mention. Before I figured out Canon made drivers for this scanner, I came across a program called VueScan which claimed to support MX880 series printers on Linux via the network without installing any drivers at all. It took seconds to download, launch, and see that this program could communicate with my MX882 scanner with ease! It may be worth the $79.95 if you can't get Canon's drivers to work.

Happy printing, scanning, copying, and faxing!

Edit Remote EC2 Text Files with TextWrangler

2011-02-03T01:02:00.000-08:00

This post explains how to use TextWrangler on a Mac to edit text files that live on a remote Amazon EC2 instance with Ubuntu.

I assume you already know how to SSH to Amazon using an Identity File:

ssh -i ~ec2-keys/id_your-keypair ubuntu@ec2-public-dns-name.amazonaws.com

You can use a similar command to connect to your EC2 instance via SFTP:

sftp -o IdentityFile=eec2-keys/id_your-keypair ubuntu@ec2-public-dns-name.amazonaws.com

TextWrangler allows you to open a file from an SFTP server, but it doesn't allow you to configure an Identity File as you did when SSHing or SFTPing from the command line. To work around this limitation, you simply have to configure your SSH client so that it knows the Identity File you want to use. Create ~/.ssh/config if it doesn't already exist and add a line specifying your Identity File location:

echo 'IdentityFile ~ec2-keys/id_your-keypair' >> ~/.ssh/config

Now you're ready to use TextWrangler to connect to your server and browse files. Choose: File -> Open from FTP/SFTP Server... Then enter your server publisher DNS name, check the SFTP box and enter ubuntu as the User name. You can leave the Password: field blank.

After you hit connect, you TextWrangler will allow you to browse the file system of your EC2 instance, choose a file, and edit it. When you save the file, it will update directly on the remote EC2 instance. That's all there is to it!

Monitor HBase & Hadoop with Ganglia on EC2

2010-10-27T01:08:00.000-07:00

This post is a recipe on setting up Ganglia to monitor an HBase and Hadoop cluster on the Ubuntu OS on Amazon EC2. Ganglia is a monitoring system for grids and clusters consisting of the following 3 components:

gmond
A Ganglia Monitoring Daemon (gmond) runs on each node in the cluster and collects statistics from the node it runs on as well as other nodes in the cluster. Normally it is a multicast system where each gmond node receives data from its peers. However, since Amazon EC2 does not support multicast at this time, you must setup Ganglia Monitoring Daemons in unicast mode where each node in a cluster is configured to send its data to one pre-designated node.

gmetad
A Ganglia Meta Daemon (gmetad) runs for each grid and collects data from the Ganglia Monitoring Daemons, one from each cluster. It stores the data it collects on the file system. We'll only be configuring one grid and therefore one gmetad. We'll be running gmetad from the same node that the PHP Web Front End is installed on.

PHP Web Front End
A PHP application reads the data and provides a UI to visualize the data over time with pretty graphs. Example from UC Berkeley.

Approach
Our HBase cluster has one Master node (with Hadoop's Name Node and Job Tracker) and the rest Region Server nodes (each with a Hadoop Data Node and Task Tracker). Since we must use Ganglia in unicast mode on EC2, we need to choose one of these nodes to receive data from each of the other nodes. We'll configure each Region Server gmond to send data to the gmond running on the Master. A separate node will be used to run gmetad and the PHP web front end. The gmetad will be configured to poll the Master's gmond for aggregated cluster data.

Getting the right version of Ganglia is important because of compatibility issues with Hadoop and HBase's Ganglia clients. We're using HBase 0.20.6 and Hadoop 0.20.2. Therefore, we're going to use Ganglia 3.0.7, the latest of 3.0.x at the time of writing. You could use the latest Ganglia 3.1.x if you're willing to apply the patch found in HADOOP-4675 to your Hadoop installation.

Setting up gmond
Add a ganglia user.

adduser --disabled-login --no-create-home ganglia

Download and unpack Ganglia 3.0.7.

wget http://downloads.sourceforge.net/project/ganglia/ganglia%20monitoring%20core/3.0.7%20%28Fossett%29/ganglia-3.0.7.tar.gz
tar -xzvf ganglia-3.0.7.tar.gz -C /opt
rm ganglia-3.0.7.tar.gz

Install dependencies

apt-get -y install build-essential libapr1-dev libconfuse-dev libexpat1-dev python-dev

Compile and install.

cd /opt/ganglia-3.0.7
./configure
make && make install

Generate a default config file.

gmond --default_config > /etc/gmond.conf

Configure /etc/gmond.conf

Modify the globals.

globals {
  user = ganglia
}

Define the cluster.

cluster {
  name = HBase
  owner = "Your Company"
  latlong = "N34.02 W118.45"
  url = "http://yourcompany.com/"
}

Disable multicast and define the host, the HBase master, where nodes in the cluster send data.

udp_send_channel {
  # mcast_join = 239.2.11.71
  host = master.yourcompany.com
  port = 8649
  ttl = 1
}

udp_recv_channel {
  # mcast_join = 239.2.11.71
  port = 8649
  # bind = 239.2.11.71
}

Run gmond:

gmond

Note: As far as I know, there is no way to stop gmond other than killing the process.

You can test each gmond node to make sure it is working. From each node:

telnet localhost 8649

You should see XML output.

Once gmond is installed and running on each node, move on to setup gmetad.

Setting up gmetad
We're going to setup a new EC2 instance to run gmetad and the PHP web front end. A t1.micro should be sufficient for modest clusters. gmetad will be configured to poll the gmond running on the HBase master for aggregated cluster data.

Add a ganglia user.

adduser --disabled-login --no-create-home ganglia

Download and unpack Ganglia 3.0.7.

wget http://downloads.sourceforge.net/project/ganglia/ganglia%20monitoring%20core/3.0.7%20%28Fossett%29/ganglia-3.0.7.tar.gz
tar -xzvf ganglia-3.0.7.tar.gz -C /opt
rm ganglia-3.0.7.tar.gz

Install dependencies

apt-get -y install build-essential libapr1-dev libconfuse-dev libexpat1-dev python-dev librrd2-dev

Compile and install.

cd /opt/ganglia-3.0.7
./configure
make && make install

Copy the default config file.

cp /opt/ganglia-3.0.7/gmetad/gmetad.conf /etc/gmetad.conf

Configure /etc/gmetad.conf

setuid_username "ganglia"
data_source "HBase" master.yourcompany.com
gridname "YourCompany"

Create directories where Ganglia will save its data.

mkdir /var/lib/ganglia
mkdir /var/lib/ganglia/rrds/
chown -R ganglia:ganglia /var/lib/ganglia/

Run gmetad:

gmetad

For troubleshooting, it might be helpful not to daemonize gmetad:

gmetad -d 1

Note: As far as I know, there is no way to stop gmetad other than killing the process.

Setting up the PHP Web Front End
Install Apache and PHP libraries:

apt-get -y install rrdtool apache2 php5-mysql libapache2-mod-php5 php5-gd

Copy PHP web app into Apache's HTML directory:

cp -r /opt/ganglia-3.0.7/web /var/www/ganglia

Restart Apache:

/etc/init.d/apache2 restart

You should now be able to visit the Ganglia console on http://ganglia.yourcompany.com/ganglia and see pretty graphs and charts for machine metrics like load, memory, disk use, etc.

Configuring Hadoop and HBase
The next step is to configure Hadoop and HBase to start sending metrics to the master gmond. Detailed instructions are given here and here for Hadoop and HBase respectively.

Here's what we end up with inside HBase's conf/hadoop-metrics.properties:

hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext
hbase.period=10
hbase.servers=master.yourcompany.com:8649

jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=master.yourcompany.com:8649

rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
rpc.period=10
rpc.servers=master.yourcompany.com:8649

And inside Hadoop's conf/hadoop-metrics.properties:

dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
dfs.period=10
dfs.servers=master.yourcompany.com:8649

mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
mapred.period=10
mapred.servers=master.yourcompany.com:8649

jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=master.yourcompany.com:8649

Once the hadoop-metrics.properties files are modified on all nodes in your cluster, you'll need to restart HBase and Hadoop.

Metrics
After restarting, you should see the following metrics show up in Ganglia for your HBase and Hadoop cluster:

hbase.regionserver.blockCacheCount
hbase.regionserver.blockCacheFree
hbase.regionserver.blockCacheHitRatio
hbase.regionserver.blockCacheSize
hbase.regionserver.compactionQueueSize
hbase.regionserver.fsReadLatency_avg_time
hbase.regionserver.fsReadLatency_num_ops
hbase.regionserver.fsSyncLatency_avg_time
hbase.regionserver.fsSyncLatency_num_ops
hbase.regionserver.fsWriteLatency_avg_time
hbase.regionserver.fsWriteLatency_num_ops
hbase.regionserver.memstoreSizeMB
hbase.regionserver.regions
hbase.regionserver.requests
hbase.regionserver.storefileIndexSizeMB
hbase.regionserver.storefiles
hbase.regionserver.stores

rpc.metrics.RpcProcessingTime_avg_time
rpc.metrics.RpcProcessingTime_num_ops
rpc.metrics.RpcQueueTime_avg_time
rpc.metrics.RpcQueueTime_num_ops
rpc.metrics.addColumn_avg_time
rpc.metrics.addColumn_num_ops
rpc.metrics.checkAndPut_avg_time
rpc.metrics.checkAndPut_num_ops
rpc.metrics.close_avg_time
rpc.metrics.close_num_ops
rpc.metrics.createTable_avg_time
rpc.metrics.createTable_num_ops
rpc.metrics.deleteColumn_avg_time
rpc.metrics.deleteColumn_num_ops
rpc.metrics.deleteTable_avg_time
rpc.metrics.deleteTable_num_ops
rpc.metrics.delete_avg_time
rpc.metrics.delete_num_ops
rpc.metrics.disableTable_avg_time
rpc.metrics.disableTable_num_ops
rpc.metrics.enableTable_avg_time
rpc.metrics.enableTable_num_ops
rpc.metrics.exists_avg_time
rpc.metrics.exists_num_ops
rpc.metrics.getClosestRowBefore_avg_time
rpc.metrics.getClosestRowBefore_num_ops
rpc.metrics.getClusterStatus_avg_time
rpc.metrics.getClusterStatus_num_ops
rpc.metrics.getHServerInfo_avg_time
rpc.metrics.getHServerInfo_num_ops
rpc.metrics.getOnlineRegionsAsArray_avg_time
rpc.metrics.getOnlineRegionsAsArray_num_ops
rpc.metrics.getProtocolVersion_avg_time
rpc.metrics.getProtocolVersion_num_ops
rpc.metrics.getRegionInfo_avg_time
rpc.metrics.getRegionInfo_num_ops
rpc.metrics.getRegionsAssignment_avg_time
rpc.metrics.getRegionsAssignment_num_ops
rpc.metrics.get_avg_time
rpc.metrics.get_num_ops
rpc.metrics.incrementColumnValue_avg_time
rpc.metrics.incrementColumnValue_num_ops
rpc.metrics.isMasterRunning_avg_time
rpc.metrics.isMasterRunning_num_ops
rpc.metrics.lockRow_avg_time
rpc.metrics.lockRow_num_ops
rpc.metrics.modifyColumn_avg_time
rpc.metrics.modifyColumn_num_ops
rpc.metrics.modifyTable_avg_time
rpc.metrics.modifyTable_num_ops
rpc.metrics.next_avg_time
rpc.metrics.next_num_ops
rpc.metrics.openScanner_avg_time
rpc.metrics.openScanner_num_ops
rpc.metrics.put_avg_time
rpc.metrics.put_num_ops
rpc.metrics.regionServerReport_avg_time
rpc.metrics.regionServerReport_num_ops
rpc.metrics.regionServerStartup_avg_time
rpc.metrics.regionServerStartup_num_ops
rpc.metrics.shutdown_avg_time
rpc.metrics.shutdown_num_ops
rpc.metrics.unlockRow_avg_time
rpc.metrics.unlockRow_num_ops

dfs.datanode.blockChecksumOp_avg_time
dfs.datanode.blockChecksumOp_num_ops
dfs.datanode.blockReports_avg_time
dfs.datanode.blockReports_num_ops
dfs.datanode.block_verification_failures
dfs.datanode.blocks_read
dfs.datanode.blocks_removed
dfs.datanode.blocks_replicated
dfs.datanode.blocks_verified
dfs.datanode.blocks_written
dfs.datanode.bytes_read
dfs.datanode.bytes_written
dfs.datanode.copyBlockOp_avg_time
dfs.datanode.copyBlockOp_num_ops
dfs.datanode.heartBeats_avg_time
dfs.datanode.heartBeats_num_ops
dfs.datanode.readBlockOp_avg_time
dfs.datanode.readBlockOp_num_ops
dfs.datanode.readMetadataOp_avg_time
dfs.datanode.readMetadataOp_num_ops
dfs.datanode.reads_from_local_client
dfs.datanode.reads_from_remote_client
dfs.datanode.replaceBlockOp_avg_time
dfs.datanode.replaceBlockOp_num_ops
dfs.datanode.writeBlockOp_avg_time
dfs.datanode.writeBlockOp_num_ops
dfs.datanode.writes_from_local_client
dfs.datanode.writes_from_remote_client

jvm.metrics.gcCount
jvm.metrics.gcTimeMillis
jvm.metrics.logError
jvm.metrics.logFatal
jvm.metrics.logInfo
jvm.metrics.logWarn
jvm.metrics.memHeapCommittedM
jvm.metrics.memHeapUsedM
jvm.metrics.memNonHeapCommittedM
jvm.metrics.memNonHeapUsedM
jvm.metrics.threadsBlocked
jvm.metrics.threadsNew
jvm.metrics.threadsRunnable
jvm.metrics.threadsTerminated
jvm.metrics.threadsTimedWaiting
jvm.metrics.threadsWaiting

As you can see, the metrics are incredibly useful for understanding the health of your HBase and Hadoop cluster over time.

Swivel4J - A New Java Client for Swivel

2009-11-30T21:52:00.000-08:00

At work, we were looking for a simple way to visualize business data without having to build and host it ourselves. This led us to Swivel (the new Swivel as they call it which is geared towards private data shared within an organization). With Swivel, we can upload our data, create pretty charts and graphs, and share them privately within our company.

Swivel has a REST API for managing the charts. Though not difficult, we found it a little unwieldy to have to use an HTTP client to setup each API call. This naturally led to a small library that wrapped the REST API calls in a native Java API.

We decided to make this library available to everyone as Swivel4J which is open source and hosted on Google Code. We hope this makes it easier for Java, Groovy, and Scala developers to get started with Swivel and promotes the use of Swivel in general.

Please try it out and give feedback, submit patches, etc.

Amazon ELB - Capturing Client IP Address

2009-09-12T20:19:00.000-07:00

If you're using Amazon EC2's Elastic Load Balancer (ELB) for load balancing web applications, you may have noticed that in your web access logs, the remote host IP address is the same for every request. The IP address you see is the private IP address of the load balancer.

If you want to see the IP address of the client (called remote host in access log documentation), you'll need to look at the value of the X-Forwarded-For request header which ELB populates when it forwards the request.

This can be achieved in an Apache access log by using the syntax:

%{X-Forwarded-For}i

You Apache log format would then look something like this:

LogFormat "\"%{X-Forwarded-For}i\" %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined-elb
CustomLog log/acces_log combined-elb

See Apache Custom Log Formats

If you're using a Tomcat application server, you could define an access log Valve like this:

<Valve className="org.apache.catalina.valves.AccessLogValve"
    directory="logs" prefix="access_log." suffix=".txt"
    pattern="%{X-Forwarded-For}i %l %u %t &quot;%r&quot; %s %b &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot;"
    resolveHosts="false"/>

See Tomcat 6 Valve Configuration Reference.

If you're trying to capture the client IP address within your application code, simply use whatever API you have to read the X-Forwarded-For request header. For example in Java use the HttpServletRequest:

String clientIpAddress = request.getHeader("X-Forwarded-For");

instead of

String clientIpAddress = request.getRemoteAddr();

Serving Gzipped Javascript Files from Amazon CloudFront

2009-08-31T22:49:00.000-07:00

It is well-known that using a content delivery network (CDN) and compressing an HTTP response with gzip can significantly improve the performance and reduce the cost of a web site. If your CDN is Amazon's CloudFront, you'll face difficulties serving gzipped content to browsers that support it. Most web servers are able to examine the HTTP request headers sent by the browser and dynamically choose whether to deliver compressed or uncompressed content. CloudFront, however, does not yet offer this feature.

People have overcome this problem by manually inspecting the Accept-Encoding request header for a dynamically served page and writing references to either original or compressed files accordingly using a naming convention like myscript.js and myscript.js.gz for regular and compressed files respectively.

If all your pages are statically served from CloudFront, however, there is no opportunity to inspect the request headers. Javascript running on a statically served page has no ability to ask the browser if it supports gzipped content. We found ourselves in this situation and resorted to the following solution to determine if the browser supports gzip:

1) Create a small gzipped file, gzipcheck.js.jgz, and make it available in CloudFront. This file should contain one line of code:

gzipEnabled = true;

2) Use the following code to attempt to load and run this file. You'll probably want to put it in the HTML HEAD section before any other Javascript code.

<script type="text/javascript" src="gzipcheck.js.jgz">
</script>

If the file loads, it sets a flag, gzipEnabled, that indicates whether or not the browser supports gzip.

Use the result to drive a file naming convention for references to other static files. For example, you can upload and reference each compressed Javascript file with an additional .jgz extension. Why .jgz instead of .gz? Because of an annoying limitation of Safari.

When you upload your files to S3/CloudFront, make sure to set the proper HTTP response headers on your files that end in .jgz:

Content-Encoding = gzip
Content-Type = application/x-javascript

And if you want browsers to cache your files forever (almost):

Cache-Control = max-age=315360000
Expires = Tue, 31 Dec 2019 20:00:00 GMT

Thanks to Ricardo Rangel who helped design and code this solution.

Ubuntu License Plate on the Santa Monica Freeway

2009-04-08T09:49:00.000-07:00

I came across this car while driving eastbound on the Santa Monica Freeway this past Saturday. Do you think the license plate is referring to the Ubuntu Linux OS or just the Ubuntu philosophy?

It also had some funny stickers and decals:
* Evolve FISH Logo Decorative Silver Car Emblem
* Goodnight Bush: A Parody
* Got Hope Bumper Sticker

Products I Can't Live Without - 2009

2009-01-14T21:50:00.000-08:00

Inspired by this post from Michael Arrington of TechCrunch, I thought it would be interesting to make my own annual list of products and services that I can't live without. Here we go for 2009 (in the order that they came to mind):

Gmail, Google Calendar, Remember The Milk, iPhone, F-Spot, Quicken, Google Reader, Facebook, Google Sites, Flickr, iTunes, Twitter, Jungle Disk, Dropbox, Delicious

Gmail
I use this every minute every day for both personal and work email (my company uses Google Apps for email and calendar). I just love the user interface and ease of accessing it on the web and the iPhone. Beats the hell out of Outlook for work email.

Google Calendar
Again, this is how I keep track of temporal events in my life both at home and at work. I love that I can get text messages with alerts for calendar events.

Remember The Milk
Ever since reading David Allan's Getting Things Done, I was in search of a tool to put his ideas to work. About a year ago, I settled with Remember The Milk (RTM) and have used it ever since for task management. The user interface is outstanding and I can access it from the web and iPhone. Perhaps the best feature is that it integrates perfectly with Gmail as a "Gadget".

iPhone
I don't identify with being an Apple person, but I couldn't resist the lure of the iPhone. It really adds a lot of value to my life as I can manage email, read blogs, listen to Podcasts, use Google Maps, locate GeoCaches, and even SSH to a server at work. Oh, and visual voice mail rocks!

F-Spot
This GNOME desktop application is what I use to manage photos. It stores photo metadata in a SQLite database which I can easily access if I ever want to export the data somewhere else. It also makes it easy to upload photos to Flickr.

Quicken
I have been using Quicken for years to manage my personal finances. I upgrade it to the latest version every even year, so right now I have Quicken 2008. Honestly, I would love to stop using Quicken and switch to a web-based solution, but I can't find one that is fully-featured enough for me. I just have to have the ability to create arbitrary asset accounts for my wallet (cash), reimbursements, etc. Most web-based services don't offer this yet. I've got my eye on Mint which, I have a feeling, will get good enough for me to make the switch one day.

Google Reader
I read a lot of blogs. I couldn't possibly keep up if I didn't use a blog reader. In late 2008, I switched from Bloglines to Google Reader because the user interface for Google Reader just kept getting better and eventually it won me over. A big part of this was the excellent user interface of Google Reader on the iPhone.

Facebook
This barely made it to my list. Maybe I could live without it, but I do find myself checking it almost daily because so many of my friends use it and somehow it makes me feel like I'm in touch with all of them without ever really talking to them :). Also, Facebook is the platform I used to develop and launch my book sharing application We Read which continues to grow slowly, but steadily each day and recently passed 1000 users.

Google Sites
This past year I finally committed to using a wiki for my family. I have been using wikis for years for work and open source software projects, and always thought it would be useful for keeping track of personal things like lists of gifts to get people, account information for utilities, financial services, and contractors, etc, etc. I tried setting up and hosting wikis like XWiki and Confluence, but ultimately went with the free, online Google Sites which was formerly JotSpot.

Flickr
I purchased the Pro Account and now use Flickr for all my online photo sharing needs. The pro account allows me to upload an unlimited amount of photos for about $25/year. My family depends on this to see the many photos I take of my son as he grows up.

iTunes
I hate the fact that iTunes doesn't run on Linux machines, but I can't resist it because it is really the only good option for managing music and videos on an iPod. I use it daily to load up my iPhone with podcasts.

Twitter
I use Twitter a lot. I blame Twitter for killing my urge to blog. It is just so much easier to share things 140 characters at a time. I have Twitter linked with Facebook so that each Tweet I send out ends up updating my Facebook status.

Jungle Disk
This might be one of the most important services I use. It keeps my music, videos, photos, and other important files backed up on Amazon's S3 network. At only $.15/GB, it ends up being pretty cheap to store a lot of data. This way, if my house ever burns down or someone steals my computers, or I experience a hard disk failure, I will always be able to restore my precious data.

Dropbox
I have started storing all files that I access often in a special directory on each computer (home, work, laptop, etc) called the Dropbox. Each directory gets automatically synchronized so I have easy, local, access to the files I need wherever I am. It also makes it extremely easy to share files with other people that use Dropbox.

Delicious
I store all my bookmarks in Delicious and take advantage of its Firefox plugin to easily tag sites and recall sites I've already tagged. I don't do a lot with the social features of Delicious like bookmark sharing, though. Maybe it's just because I don't know which of my friends use Delicious too.

Well, there you have it! I can't wait to see how this list compares to the one I'll make in January 2010.

Groovy Best Fit Line

2008-12-03T20:32:00.000-08:00

An engineer on my team at work asked if I could help with coding an algorithm for calculating a line that best fits a given set of data points. I thought, "what a great excuse to practice some Groovy", and got started right away.

A few Google searches led me to this 10-yr-old page on Regression Functions by Stefan Waner from Hofstra University. Stefan outlined the exact algorithm I was looking for.

Here is my implementation in Groovy:

// Prints the start/end points of a line that best fits 4 sample data points
pts = [[1, 1.5], [2, 1.6], [3, 2.1], [4, 3.0]]
println bestFit(pts)

/**
* Given a set of points, uses a linear regression algorithm to find the start and end points
* of a line that best fits the set of points.
* Returns the two points as [[x1, y1], [x2, y2]].
* The algorithm is from
* http://people.hofstra.edu/stefan_waner/realworld/calctopic1/regression.html.
*/
def bestFit(pts) {
  // Find sums of x, y, xy, x^2
  n = pts.size()
  xSum = pts.collect() {p -> p[0]}.sum()
  ySum = pts.collect() {p -> p[1]}.sum()
  xySum = pts.collect() {p -> p[0]*p[1]}.sum()
  xSqSum = pts.collect() {p -> p[0]*p[0]}.sum()

  // Find m and b such that y = mx + b
  // m is the slope of the line and b is the y-intercept
  m = (n*xySum - xSum*ySum) / (n*xSqSum - xSum*xSum)
  b = (ySum - m*xSum) / n

  // Find start and end points based on the left-most and right-most points
  x1 = pts.collect() {p -> p[0]}.min()
  y1 = m*x1 + b
  x2 = pts.collect() {p -> p[0]}.max()
  y2 = m*x2 + b

  [[x1, y1], [x2, y2]]
}

Running this script prints the following:

[[1, 1.3], [4, 2.8]]

You gotta love Groovy!

A Personal Hosted Confluence Wiki for Free

2008-11-22T22:26:00.000-08:00

I had been searching for a way to use Confluence as a personal wiki for me and my wife. I couldn't find a cheap hosted solution where I could just get an account for 2 people on an instance of Confluence maintained by someone else. I also didn't really want to pay a hosting company like Linode or Slicehost $20 a month for my own virtual machine on which I'd have to install and maintain Confluence myself.

Recently I heard about Morph Labs which describes themselves as a Platform as a Service (PaaS). They offer free hosting accounts that give you up to a 1GB database. I immediately got the idea to try running the personal edition of Confluence on this service. Eventually I got it working, but the process to get up and running was not at all easy. In fact, I struggled for serveral nights.

The rest of this post documents the process I went through. I hope it will help someone else avoid the many hours I spent learning how to work within Morph AppSpace with a third-party Java web application. Good luck, and if you have any tips or corrections, please comment and I'll update this post.

Start by signing up for a Morph AppSpace account with Morph Labs. Create a new subscription and setup a new database for that subscription. Be sure to choose Java, not the default Ruby on Rails, when setting up your subscription. Choose PostgreSQL for your database as this seems to work best with Confluence. Once your subscription is setup, download the Properties File, morph_deploy.properties, and Deployment Jar File, morph-deploy.jar, that can be found in the Java Tools for Morph AppSpace Deployment section.

Next, go get your free license for personal Confluence. It allows you to register 2 users. Hold on to that license text. You'll need to activate Confluence once it is installed. Then download the latest version of Confluence which is currently 2.9.2.

Unpack the file you downloaded somewhere on your computer. Inside, you should find an edit-webapp directory. Any files you need to modify should get copied into this directory. The contents of this directory get overlayed on top of the contents of the eventual Confluence WAR file that you're going to build, so make sure to adhere to the standard WAR directory structure. The whole purpose of the edit-webapp directory is to keep your local changes separate from the distribution so when it comes time to upgrade Confluence, you can just replace it and you'll know what you had to modify.

Copy confluence/WEB-INF/classes/confluence-init.properties to edit-webapp/WEB-INF/classes/confluence-init.properties.
Edit the copied confluence-init.properties, specifying the home dir as /var/java/ where APP_NAME is whatever name you chose as your subdomain at Morph. For example, if your domain is family-wiki.morphexchange.com, then your home directory path would be /var/java/family-wiki.

Copy confluence/WEB-INF/web.xml into edit-webapp/WEB-INF/web.xml.
Edit web.xml adding the jdbc datasource:


<resource-ref>
  <description>Morphlabs Datasource</description>
  <!-- any name will do for the res-ref-name -->
  <res-ref-name>jdbc/morph-ds</res-ref-name>
  <res-type>javax.sql.DataSource</res-type>
  <res-auth>Container</res-auth>
</resource-ref>

and the JavaMail session:


<resource-ref>
  <description>Morphlabs Mail Session</description>
  <!-- any name will do for the res-ref-name -->
  <res-ref-name>mail/Session</res-ref-name>
  <res-type>javax.mail.Session</res-type>
  <res-auth>Container</res-auth>
</resource-ref>

I had to disable Confluence's JMX-based monitoring because of problems registering MBeans. This isn't a big deal since, for a personal instance of Confluence, I don't really have any need to monitor performance and gather statistics. This web page explained how to disable JMX. Download jmanageContext.xml from the URL shown on that page. Then, copy jmanageContext.xml into edit-webapp/WEB-INF/classes.

Remove mail.jar and activation.jar from confluence/WEB-INF/lib. Morph AppSpace already includes those in the Jetty server and having an extra copy in your WAR might cause class loader issues that would prevent JavaMail from working properly. This is the only change that you'd have to remember to do again if you were to ever download a new version of Confluence.

Now it is time to build the WAR file. Run ./build.sh which creates dist/confluence-2.9.2.war. If you need to make any changes to any files, be sure to run this build script again.

Download morph_deploy.properties and morph-deploy.jar from your subscription within the Morph Control Panel.

Run


java -Xmx512m -jar morph-deploy.jar --config morph_deploy.properties confluence-2.9.2/dist/confluence-2.9.2.war



Now, sit back, relax, and wait while your WAR is being uploaded and deployed on Morph AppSpace.  This could take up to 15 minutes, so be patient.

Hit your-subscription-name.morphexchange.com and if all went well, you should see a Confluence installation web page.  Following the Confluence installation wizard, entering your personal license text.  Choose Custom Installation, External Database, Connect via a Datasource.
Enter java:comp/env/jdbc/morph-ds as the datasource.

Confluence takes a long time to setup the database, so the page may appear to hang.  Each time I tried this (I had to try many times before figuring all this out), the page timed out.  I don't know if something went wrong or Morph AppSpace just has a time limit on waiting for a page to render.  Even though the page timed out and I saw some kind of "Proxy error" message, the database still seems to have been initialized properly, but the application wasn't able to recover.  I had to go into the Morph Control Panel to restart the application.  Once I did that, Confluence was up and running!

Coming soon to this post: Setting up a mail server and using Groovy and XML-RPC to perform a daily backup of your confluence data.



MacBook to HDTV via Mini DVI to DVI to HDMI
2008-09-28T22:17:00.001-07:00
Since the MacBook is so good at playing different kinds of media, I plan to make it the media center of my life.  I will store all music, videos, and photos there.

This led me to try to connect my MacBook to my Sharp Aquos LC46D64U 46-Inch 1080p LCD HDTV.  I did some research and found that there was only one option if I wanted to have digital picture quality.  I needed 2 cables because there is no such thing yet as a Mini DVI to HDMI cable:
Mini DVI to DVI Adapter ($19.95)
DVI to HDMI Cable ($19.95)
I picked these up, connected the Mini DVI end of the first cable to my MacBook, connected the DVI end of the second cable to the first cable, and finally the HDMI end of the second cable to an open HDMI slot in my TV.  Then I switched the TV's input to the HDMI port I just plugged into, and voila! - it worked immediately! The 1920 by 1080 picture looked great.

The MacBook lets you choose between having 2 monitors and mirroring the same monitor.  The settings are all under System Preferences > Displays.

Now, I can show photo slide shows when the family is over, I can watch video podcasts and YouTube videos on the big screen, and I can even enjoy the Twitter election feed from my couch.

Next I will look for the best way to get the sound from my MacBook hooked up to my Denon receiver.





Giving In and Buying a MacBook
2008-09-23T21:37:00.000-07:00
I finally caved in to the marketing pressure from Apple and a lot of my Apple-religious friends.  I bought a MacBook.  Why would I do this after many years of trying to do everything with open source software?  The main reasons are iTunes and my wife's need to use MS Office.

I need iTunes to manage my family's 3 iPods (2 of which I got for free at tech conferences and users groups).  I also will need iTunes if I decide to buy an iPhone.  I have been thinking about it because I am surrounded by friends and co-workers who swear up and down that the iPhone is the answer to all of life's problems. Anyway, Apple won't produce a version of iTunes that runs on Linux and that's a shame, but there's nothing I can do about it.

My wife has been using our Ubuntu linux desktop for the last few years, and has gotten along quite nicely until recently when she needed some advanced spreadsheet functionality like mail merge. I am disappointed to say that Open Office just wasn't cutting it.  The feature set was either lacking or buggy.  She was longing for MS Excel.  MS Office is available for the Mac, so that problem could get solved too.

So far, the best thing about that Mac has been its packaging.  It came in a slick box with fancy designer Styrofoam.  It has stylish body and the l.e.d. lights have a pleasing glow.  Also the screen has a great, kind of glossy, picture.

I do have some gripes:

For one thing, I can't find a way to turn of that annoying sound the computer makes when booting.  A few blog posts I read say that you have to download some 3rd party software to disable it.

After 20 minutes of surfing the web when I first booted up, the computer crashed and the wireless card stopped working.  I had to take the computer back to the Apple store.  Thankfully they exchanged it even though I had purchased the MacBook from the online Apple store.

There doesn't appear to be any way to maximize a window.  There is a green plus button on each window that makes the window larger, but it doesn't make it go full screen.  Why can't I have this functionality on a Mac?

There is no concept of right-click!  To get a context menu, I have to put 2 fingers on the trackpad and click the button.  I guess I can get used to this, but come-on, would it be so hard for Apple to make the right side of the button do what Windows and Linux users expect?

You can only resize a window from the lower-right corner.  This is unbelieveable.  Both Windows and Linux allow you to resize a window from any edge of the window.  I find this really annoying.

There are no Page Up, Page Down, Home, and End keys!   The function key + up or down arrows will do the same thing as Page Up and Page Down.  Also the Apple key + the left and right arrow will do the same thing as Home and End, but I'm just not used to it yet.

Pressing the red X button on a window closes the window, but doesn't close the application even if it is the last window open in that application.  You have to remember to explicitly quit the application with an Apple key + Q.  I always forget to do this and am left with many open applications as a result.

Despite all these gripes, I am happy to be getting to know my way around a Mac now.  I have always felt completely lost on a Mac and these days, it feels like everyone and their mother has one.





Walking L.A. #20 - Lower Beachwood Canyon
2008-07-13T21:01:00.000-07:00
Again, Erin Mahoney, the author of our guide book, took us back to Hollywood.  This time we visited Lower Beachwood Canyon which is just a little bit east of the last walk in Whitley Heights.  Again there were many steep roads, but I enjoyed this walk a little more than last week's because we went earlier in the morning to avoid the heat, and I saw an area I've never seen before, even though I've lived in LA for over 15 years.

We began at the bottom of Vista del Mar Ave where it meets Franklin Ave.  We stopped for coffee at the 101 Coffee Shop which was a 70's-looking diner with some character.  I'd return there sometime to try the food, but never again for a cup of coffee which was pretty lousy even for a diner.

Again this week we saw beautiful houses, many with enviable views of Hollywood and Downtown LA.  Many parts of the neighborhood were quiet with narrow, curvy streets, which reminded me of hillside towns in Europe.  Other parts were extremely noisy from the roaring cars on the adjacent 101 Freeway.

One of the noisiest areas, ironically, was home to the Vedanta Society of Southern California. We stopped in to admire their white temple which looked like a mini Taj Mahal.  We also browsed through their bookstore where we almost bought our son a Peaceful Piggy Meditation book.




The next walk is Upper Beachwood Canyon.

Walking L.A.: 36 Walking Tours Exploring Stairways, Streets and Buildings You Never Knew Existed







Walking L.A. #19 - Whitley Heights and Hollywood Boulevard
2008-07-06T20:31:00.000-07:00
This walk was just on the other side of Highland Ave from the last walk.  You can tell that Whitley Heights was once a very prestigious neighborhood by the many fancy Mediterranean style homes.  Unfortunately the 101 Freeway was built right through it and now the residents have to put up with the noise.

It was really hot today and the many steep roads on this walk contributed to a lot of sweating.  After walking through Whitley Heights we descended down Whitley Ave to Hollywood Blvd where we walked west to the Hollywood and Highland complex.  The boulevard was full of the usual tourists.  We had been here many times before so instead of sightseeing, we headed into a Build-a-Bear store to cool off.  Yes, there is actually a store that is like a Subway Restaurant for teddy bears.  You start with a naked bear and then pick its name, clothes, and accessories.

After the bear store, we stopped to watch some dancers rehearsing their moves at the Hollywood Pop Academy while we fed our baby.  Finally, we grabbed some coffee at Starbucks and then headed back to Whitley Heights to end our walk.

The next walk is Lower Beachwood Canyon.

Walking L.A.: 36 Walking Tours Exploring Stairways, Streets and Buildings You Never Knew Existed







Walking L.A. #18 - High Tower and the Hollywood Bowl
2008-06-08T20:46:00.000-07:00
Before the walk, we stopped for breakfast at the Mar Vista Farmers Market getting omelettes and French toast from Cafe Laurent.  Two good friends who were visiting from San Francisco joined us today.

The first part of the walk was interesting.   We walked through the Hollywood Heights neighborhood which is just south of the Hollywood Bowl.  At the center of this hillside neighborhood stood a tall elevator tower known as High Tower. The tower is closed to everyone except residents with a key, so we had to climb many cement staircases to get to the top.  This was a challenge considering we had our baby and stroller with us.  Luckily my friends took turns helping me out.   One of the things I liked the most about this neighborhood was that many of the homes were accessible only by footpath.  This gave it a quiet, neighborly feel similar to places I've seen in Europe.

The next part of the walk took us to the famous Hollywood Bowl.  There were no shows going on and the bowl was open for people to just walk in and check it out.  We walked to the boxed seat area, took a seat in one of the boxes, and dreamt about how nice it would be to attend a concert or show in one of the boxes eating cheese and sipping wine.

The next walk is Whitley Heights and Hollywood Boulevard.

Walking L.A.: 36 Walking Tours Exploring Stairways, Streets and Buildings You Never Knew Existed







My Twitter Account Has Disappeared!
2008-06-02T23:38:00.000-07:00
I was about to Twitter just now and for some reason Twitter wouldn't let me login.  I tried visiting my Twitter feed page at http://twitter.com/kweiner and Twitter responded with the message "That page doesn't exist!".  Figuring that Twitter was just down, I tried a friend's page, and that page loaded just fine.  In the "Following" section of my friend's page, I noticed that my picture was missing!  Next I tried to reset my password, and when I entered my email address, Twitter responded with "Oh, snap! We couldn't find you!".

What is going on?  Is my Twitter account really gone and my data lost forever?  I know they have had problems scaling recently, but losing someone's account is way worse than having a temporary outage.   I submitted a support ticket and also posted a message to Get Satisfaction.  Let's see what happens...  If this doesn't get resolved, I am just going to give up on Twitter which will probably make my wife happy.





Walking L.A. #17 - Carthay Circle and South Carthay
2008-06-01T20:31:00.000-07:00
There wasn't anything that exciting about today's walk which was a little south of last week's walk in the Miracle Mile area.  We explored 2 of 3 neighborhoods within a residential district called Carthay: Carthay Circle and South Carthay.  Our book says that these neighborhoods are in the Miracle Mile district, but Wikipedia says Carthay is its own district.  Both neighborhoods contain HPOZ's for their architectural integrity and cohesiveness.  The styles include Spanish Colonial Revival and Tudor Revival.  I can't think of much else to say about this walk.

The next walk is High Tower and the Hollywood Bowl.

Walking L.A.: 36 Walking Tours Exploring Stairways, Streets and Buildings You Never Knew Existed







Walking L.A. #16 - Miracle Mile
2008-05-25T19:55:00.000-07:00
Our friend Kevin joined us for this walk which meandered through the Miracle Mile district where he works at E! Networks.

The walk began at Hancock Park (a real park, not the wealthy LA neighborhood of the same name).  This park is home to the infamous La Brea Tar Pits and the adjacent LA Country Museum of Art.  We paused briefly by the tar pits to see if we could get the smell to trigger our memory of childhood playground asphalt on a hot day, something our book mentioned we might experience.  It didn't work at first, but as we walked away from the tar pits, a gentle breeze indeed carried the scent we were waiting for.  We skipped the Page Museum which is dedicated the ancient history of the area since we'd all been there at some point in our lives.

Next we passed by a new part of LACMA called BCAM which stands for Broad Contemporary Art Museum.  In front of BCAM was a giant red toy firetruck and closer to Wilshire was a display of lamp posts.  We paused here to take some pictures.

We stopped for lunch at Farmers Market on 3rd and Fairfax.  I had fried catfish and jambalaya at one of my favorite eateries, The Gumbo Pot.  My friend Kevin grabbed a savory crepe from the French Crepe Company.  After lunch we walked quickly through The Grove, one of those generic, outdoor, fake-urban, Disneylandesque shopping malls that everyone but me seems to love.

We left The Grove to walk through a nice neighborhood called Miracle Mile North, one of the district's several HPOZ's.  Very nice homes in a very central location.  Wish we could afford to live there.  We finished up the walk passing through the Park La Brea high-rises.  It was the first time I had walked through Park La Brea, and I wasn't impressed, although it seems that many people like it.

The next walk is Carthay Circle and South Carthay.

Walking L.A.: 36 Walking Tours Exploring Stairways, Streets and Buildings You Never Knew Existed







Cloud Studio Released
2008-05-17T19:32:00.000-07:00
Cloud Services just launched Cloud Studio which is a java-based desktop application for managing images and instances in Amazon's EC2.  I've yet to try it, but from the screenshots, it looks really nice.  I'm not sure, though, if it does anything more than the very capable Elasticfox which runs right in your Firefox browser.  Cloud Studio is also available as an Eclipse plugin which is really convenient for Java developers and users of the popular open source IDE.

GUI's like Elasticfox and Cloud Studio make working with Amazon EC2 really easy.  Now I wish someone would create similar management tools for Amazon's SQS and SimpleDB.  I'd like to be able to list, see stats for, and manage my SQS queues and SimpleDB domains, items, and attributes without having to run command line scripts.  Hint, hint, Cloud Services!





ejabberd on Amazon EC2 Ubuntu AMI
2008-05-11T10:21:00.000-07:00
It turned out to be harder that I expected to setup a Jabber (XMPP) server on an Ubuntu virtual machine (ami-ce44a1a7) within Amazon EC2.  I chose to setup ejabberd since it was an easy to install via apt-get.  I was a little nervous about working with a server built  with Erlang since I knew nothing about it, but it was a reputable server and I was counting on not needing to have any Erlang knowledge to work with ejabberd.  That was a correct assumption for the most part.

First, I logged into the VM and installed ejabberd:
apt-get install ejabberd
Next I edited the ejabberd config file, /etc/ejabberd/ejabberd.cfg, making kweiner@jabber.pop140.com an admin user and setting the hostname as jabber.pop140.com:
%% Admin user
{acl, admin, {user, "kweiner", "jabber.pop140.com"}}.

%% Hostname
{hosts, ["jabber.pop140.com"]}.
Next, I restarted the server and registered kweiner as a new user:
/etc/init.d/ejabberd restart
ejabberdctl register kweiner jabber.pop140.com mypasswd
Then I authorized traffic on ports 5222 (Jabber), 5223 (Jabber encrypted for old clients), 5269 (Other Jabber servers), and 5280 (Jabber web admin tool):
ec2-authorize default -p 5222
ec2-authorize default -p 5223
ec2-authorize default -p 5269
ec2-authorize default -p 5280
I could now hit the admin screen on which I logged in as kweiner@jabber.pop140.com:
http://jabber.pop140.com:5280/admin/
From here it was possible to add new users and browse server statistics. I experimented with adding users and using various Jabber clients like Gajim to send messages from one user to another.

My first problem came when I tried to communicate with users registered in other Jabber servers like Jabber.org and Google Talk.  I struggled for hours trying to figure out why users on my server couldn't communicate with users from these other servers.  Thankfully James Murty gave me a bit of help on this Jabber on EC2 message board thread.  It turned out that I needed to configure SRV records in my DNS settings.

I logged into GoDaddy where my domain is registered and configured SRV records as follows:


After this, I was able to use nslookup to verify that the SRV records were setup properly:
kweiner~$ nslookup
> set type=srv
> _xmpp-server._tcp.pop140.com
Server:         66.75.160.63
Address:        66.75.160.63#53

Non-authoritative answer:
_xmpp-server._tcp.pop140.com    service = 10 10 5269 jabber.pop140.com.
That did it!  My jabber server was finally federating with other jabber servers and my users could talk to their users.

I encountered my next big problem when I tried to use jabber again after terminating and relaunching my AMI. ejabberd failed to start and I found the following error message in the /var/log/ejabberd/ejabberd.log:
application: ejabberd
exited: {bad_return,{{ejabberd_app,start,[normal,[]]},
           {'EXIT',{{badmatch,{aborted,{no_exists,config}}},
                    [{ejabberd_config,set_opts,1},
                     {ejabberd_app,start,2},
                     {application_master,start_it_old,4}]}}}}

After some googling, I found that ejabberd associates itself with an Erlang node name, a concept I don't really understand that well.  By default the node name was dynamically set based on the hostname for the machine.  It looked something like this: ejabberd@domU-12-31-38-00-9D-63.  This node name is somehow linked to the Mnesia database stored as files within /var/lib/ejabberd.  The problem is that the hostname and therefore the node name changes everytime the AMI is relaunched which confuses ejabberd.

One solution I found is to explicitly set the node name.  I did this by modifying /etc/default/ejabberd adding the line:
export ERLANG_NODE=ejabberd@jabber
This requires adding jabber as a host name inside /etc/hosts:
127.0.0.1 localhost.localdomain localhost jabber
I made those changes, removed all the database files from /var/lib/ejabberd, and restarted ejabberd. That did it! The node name was the same regardless of the hostname associated with the particular AMI instance.

This was a lot of effort, but it probably would have been easier if I had been familiar with Erlang applications and SRV DNS settings.  I hope this post helps someone else struggling to setup ejabberd on EC2 as I did.





Amazon EC2 - My First Step into the Cloud
2008-05-06T21:42:00.000-07:00
Ever since returning from the Web 2.0 Expo in San Francisco last month, I have been excited about learning how to setup a machine in the Amazon Elastic Compute Cloud (EC2).  My first goal was just to get a Linux machine running in the cloud on which later I will try to install enough software to get a basic web application running.

To learn what to do, I consulted a the Programming Amazon Web Services book which is fortunately available on Safari Online.  In parallel I read the Getting Started Guide on Amazon's website.  The steps to get a virtual machine running were roughly the following:
Register for Amazon EC2.
Download a X.509 certificate private key and public key pair and store them in a ~/.ec2 directory on my computer.
Take note of my AWS account number.
Download the Amazon EC2 Command-Line Tools
Set environment variables: EC2_HOME, EC2_PRIVATE_KEY, EC2_CERT
Run ec2-describe-images -o self -o amazon to search for images.
Generate keypair using  ec2-add-keypair gsg-keypair
Fired up the Getting Started AMI with  ec2-run-instances ami-2bb65342 -k gsg-keypair
Open SSH and HTTP ports: ec2-authorize default -p 22, ec2-authorize default -p 80
SSH into the instance: ssh -i id_rsa_gsg_keypair root@ec2-75-101-209-13.compute-1.amazonaws.com
Access the instance's web server: http://ec2-75-101-209-13.compute-1.amazonaws.com/
Shutdown the instance: ec2-terminate-instances i-db6ea2b2
That's it! Everything worked as advertised and I got through all of that in less than an hour.  This whole procedure cost me about 10 cents.  I am looking forward to my next goal which is to find an Ubuntu server image, install a database with data, and learn how to preserve the data so that it survives an image restart.

  





Server-Side Language Detection with Ruby + Google Language API
2008-04-20T22:07:00.001-07:00
Have you ever wanted to detect the language of a piece of text?  Google's AJAX Language API makes this possible on the client side.  This Belgian startup's blog post shows a PHP example of how you can use Google's detection service on the server side.  Here is a port of that example in Ruby:

(the 'json' gem must be installed prior to running this program)

require 'rubygems'
require 'net/http'
require 'open-uri'
require 'cgi'
require 'json'

base_url = 'http://www.google.com/uds/GlangDetect?v=1.0&q='
url = base_url + CGI.escape("See if you can guess what language this is!")
response = Net::HTTP.get_response(URI.parse(url))
result = JSON.parse(response.body)
lang = result['responseData']['language']
puts "Language code: #{lang}"