Server side:

• node.js – Used to process the long polling requests
• ngnix – Front for multiple nodejs instances and serving some static files
• underscore.js
• rabbitmq – Used to monitor the realtime feed
• Redis – Used as a datastore for everything
• ruby – Used to glue a bunch of things together, tons of gems used
• Resque – Used to handle the screen captures

Browser side:

• jQuery
• jQuery flot
• jQuery sparklines

Hosting:

• AWS EC2 – Used for processing browser snapshots
• AWS Cloudfront – Used for storing CSS, etc
• Rackspace Cloud Servers – Used to run the node servers
• Linode – Used to run the main web server

1. A simple daemon
2. An event driven daemon with RabbitMQ
3. RobustThread, timeouts, monit/god, capistrano
4. Other Ruby daemon options

A simple daemon

Creating a simple daemon is easy. This is a simple walk through so look at the project's documentation for much more information. First you will need to get the gem installed:

gem install daemon-kit

After the gem is installed you create a daemon with the following command:

daemon-kit asimple

Now you have some generated code in the asimple directory. In that directory you will find a README that tells you want each directory is used for. The main file to look at for this example is:

libexec/asimple-daemon.rb

A fresh file will have a lot of comments in it pointing you in the right direction. Here is a modified version where I'm displaying the contents of /tmp over and over:

DaemonKit::Application.running! do |config|
end

loop do
  DaemonKit.logger.info Dir.entries('/tmp/').join(' ')
  sleep 5
end

To run the daemon you would use a command like the following:

RUBYOPT=rubygems ./bin/asimple

The documentation found in the project is great so look there for more information. There is a lot of flexibility built in like multiple environment support and command line options. The daemons and angels blog post is educational and worth a read for more on the basics of daemon-kit.

After experimenting for a while I think I may want to go back and redo the live segmenter so that it uses daemon-kit instead of my hand rolled process. That is probably a good example of where the simple functionality comes in.

An event driven daemon with RabbitMQ

What tilted me towards looking into Ruby daemon options was actually another event driven process I was thinking about. As it turns out daemon-kit supports creating evented daemons as well. I picked RabbitMQ for the following example but anything that supports AMQP should work. I'm going to assume you have RabbitMQ installed already.

First off you need to install the AMQP gem:

gem install eventmachine
gem install amqp

At this point you might want to verify that you can connect to your MQ server using some AMQP examples.

To generate an event based daemon use the following command:

daemon-kit -i amqp eventbased

For some reason the generated code didn't work right out of the box for me but a slight modification fixed the issue. I added the following to the top of the config/environment.rb file:

require 'amqp'
require 'mq'

Again the file generated in libexec is the starting point and gives you good hints as to where you want to start. I've modified my example above to take a directory to gather the contents of and respond with those contents as another message:

DaemonKit::Application.running! do |config|
end

DaemonKit::AMQP.run do
  amq = ::MQ.new

  amq.queue('dir_request').subscribe(:ack => true) do |h, msg|
      h.ack # just ack the message

      DaemonKit.logger.debug "Received message: #{msg.inspect}"

      amq.queue('dir_response').publish(Dir.entries(msg).join(' '))
  end
end

One thing to note is that I've set the message queue to require an acknowledgement for each message. I did this to simulate a process that requires that each request be executed once. Now all that is needed is a simple application to produce request messages and display the response:

require 'rubygems'
require 'amqp'
require 'mq'

AMQP.start(:vhost=>"/", :host=>"localhost", :user=>"guest", :pass=>"guest") do

  def produce_periodic_requests
    mq = MQ.new
    EM.add_periodic_timer(1) do
      puts "Creating request..."
      mq.queue('dir_request').publish("/tmp/")
    end
  end

  def watch_for_response
    mq = MQ.new
    mq.queue('dir_response').subscribe(:ack => true) do |h, msg|
      puts "Got response: #{msg}"
      h.ack
    end
  end

  produce_periodic_requests
  watch_for_response
end

I'm assuming that the MQ server that is being used has the user guest set up with the password guest, this is the default in the default that is generated by daemon-kit in the config/amqp.yml as well. At this point you should be able to execute both the daemon process and the producer and see the output on each end as the requests are completed. Note that the messages that come in are processed one at a time and this may not be optimal, you would probably want to spawn a new thread for each message or use a thread pool so that multiple messages could process at the same time.

RobustThread, timeouts, monit/god, and capistrano

The above gets you a long way but there are a number of open ends that need to be tied up. The first of those is what happens if the requested action fails? In the above evented example if the directory request fails the entire daemon will end and that isn't going to be good. The first step to fixing this is to use RobustThread. Wrapping the action with RobustThread will make sure any exceptions thrown don't cause everything to stop.

Next you may have tasks that end up never completing for some reason. In this case you will want to use a timeout to kill the processing and move along. This actually works well with RobustThread since it can catch TimeoutErrors that get raised after a certain amount of time. Here is a modified version of the evented daemon above that takes two parameters in the incoming message to give both RobustThread and timeouts a test:

require 'robustthread'

DaemonKit::Application.running! do |config|
  RobustThread.logger = DaemonKit.logger
end

DaemonKit::AMQP.run do
  AMQP.conn.connection_status do |status|
    DaemonKit.logger.debug("AMQP connection status changed: #{status}")
    if status == :disconnected
      AMQP.conn.reconnect(true)
    end
  end

  amq = ::MQ.new

  RobustThread.exception_handler do |exception|
    RobustThread.logger.debug "Exception: #{exception.inspect}"
    amq.queue('dir_response').publish("Exception: #{exception.inspect}")
  end

  amq.queue('dir_request').subscribe(:ack => true) do |h, msg|
    rt = RobustThread.new(:args => [h, msg, amq], :label => "message processing") do |h, msg, amq|
      h.ack # in this case we always want to just ack the message

      DaemonKit.logger.debug "Received message: #{msg.inspect}"
      dir_name, sleep_time = msg.split(",")

       # give the request 5 seconds to complete before ending it
      Timeout.timeout(5) do
        sleep sleep_time.to_i
        amq.queue('dir_response').publish(Dir.entries(dir_name).join(' '))
      end
    end
  end
end

A modified version of the producer is also needed:

require 'rubygems'
require 'amqp'
require 'mq'

AMQP.start(:vhost=>"/", :host=>"localhost", :user=>"guest", :pass=>"guest") do

  def produce_periodic_requests
    mq = MQ.new
    EM.add_periodic_timer(1) do
      puts "Creating request..."
      mq.queue('dir_request').publish("/tmp/,1") # directory to list,seconds to sleep before listing
    end
  end

  def watch_for_response
    mq = MQ.new
    mq.queue('dir_response').subscribe(:ack => true) do |h, msg|
      puts "Got response: #{msg}"
      h.ack
    end
  end

  produce_periodic_requests
  watch_for_response
end

To see RobustThread in action just modify the request so it asks for a directory listing of a directory that doesn't exist. To see the timeout handling in action change the number after the directory to a value greater than 5. One thing to keep in mind with this example is that every request will spawn a new thread and if you have a large number of requests at one time you may end up starving your threads so that none of them can finish. A more complete implementation of this example would have some type of limit on the number of threads that would spawn at one time or use some type of thread pool (the Servolux project listed later has such a thread pool).

Now you have a fairly sturdy daemon process but there is still some possibility that what gets processed causes Ruby itself to die. In this case daemon-kit also has support for Monit and God that can both restart the daemon if needed. This is probably also a good place to mention that instead of using threads it may be better to use processes and something like the process worker pool from Servolux to provide the ability to withstand process crashes.

On top of all that there is also support for Capistrano deployments.

Other Ruby daemon options

After building a the above examples and some more digging I ran into a few other options that are worth mentioning. I haven't looked into any of these in detail so I may be overlooking features and I also believe that these all fall outside of the simple case where all you want is a long running process. The first few are BackgrounDRB, Beanstalkd and Servolux. Another option I will mention is Resque that was released by github and lives on top of Redis. From an initial look it seems like Resque may be a better option for situations where evented tasks being executed may crash the entire process and also has a lot of other features right out of the box. Out of these I'm probably going to investigate Resque more to see how it survives with workers that may crash.

The first thing to note is that they are only supporting 64 bit systems officially. Having said that it isn't too hard to modify the code to make it work on a 32 bit system although it may turn out that such early modifications are missing some fundamental bits on why they were only support 64 bit systems. I'm going to assume at first that you are using a 64 bit system and then end with what you need if you are still using a 32 bit system.

I don't actually have a 64 bit system myself so I used an EC2 instance for the following instructions. To do the same start with Amazon's Basic 64-bit Fedora Core 8 (AMI Id: ami-86db39ef) instance (note that this is EBS backed so you will end up with an EBS volume after you start it) and then upgrade to Fedora 12 using my previous instructions on building a EBS bootable Fedora 12 instance. You will need to remove a few packages to get the 64 bit version of Fedora 8 to upgrade that I didn't have to do for the 32 bit version, here are all the commands you need to get to a running 64 bit Fedora 12 instance (the entire upgrade takes about 20 minutes):

# Fedora 8 to Fedora 10
yum -y remove dmraid-1.0.0.rc14-4.fc8.i386 dmraid-1.0.0.rc14-4.fc8.i386 curl-7.18.2-7.fc8.i386
yum clean all
rpm -Uhv http://archive.kernel.org/fedora-archive/releases/10/Fedora/i386/os/Packages/fedora-release-10-1.noarch.rpm http://archive.kernel.org/fedora-archive/releases/10/Fedora/i386/os/Packages/fedora-release-notes-10.0.0-1.noarch.rpm
yum -y update

# Fedora 10 to Fedora 11
yum -y remove gpm-1.20.5-2.fc10.i386
yum clean all
rpm -Uvh http://mirrors.usc.edu/pub/linux/distributions/fedora/linux/releases/11/Fedora/i386/os/Packages/fedora-release-11-1.noarch.rpm http://mirrors.usc.edu/pub/linux/distributions/fedora/linux/releases/11/Fedora/i386/os/Packages/fedora-release-notes-11.0.0-2.fc11.noarch.rpm
yum -y update

# Fedora 11 to Fedora 12
yum -y remove cryptsetup-luks-1.0.6-7.fc11.i586
yum clean all
rpm -Uvh http://mirrors.kernel.org/fedora/releases/12/Fedora/i386/os/Packages/fedora-release-notes-12.0.0-4.fc12.noarch.rpm http://mirrors.kernel.org/fedora/releases/12/Fedora/i386/os/Packages/fedora-release-12-1.noarch.rpm
yum -y update

# Make sure the basics are installed
yum -y install gcc-c++ git

To start with there are some prerequisites you need. This can be taken care of in one command with yum:

yum -y install git cmake boost pcre-devel libicu-devel libmcrypt-devel oniguruma-devel mysql-devel gd-devel boost-devel libxml2-devel libcap-devel binutils-devel flex bison expat-devel

Next create a directory to hold everything in, change into that directory and create another directory to hold the customized libraries needed to compile HipHop PHP:

mkdir hiphop
cd hiphop
mkdir local

Next it is time to pull down the HipHop PHP source along with the source for some libraries it depends on (these all go into the hiphop directory created above):

git clone git://github.com/facebook/hiphop-php.git

wget "http://downloads.sourceforge.net/project/re2c/re2c/0.13.5/re2c-0.13.5.tar.gz?use_mirror=cdnetworks-us-2"
wget "http://www.threadingbuildingblocks.org/uploads/77/142/2.2/tbb22_20090809oss_src.tgz"
wget http://curl.haxx.se/download/curl-7.20.0.tar.bz2
wget http://www.monkey.org/~provos/libevent-1.4.13-stable.tar.gz

tar xvjf curl-7.20.0.tar.bz2
tar xvzf libevent-1.4.13-stable.tar.gz
tar xvzf re2c-0.13.5.tar.gz
tar xvzf tbb22_20090809oss_src.tgz

Next the customized patches get applied to some of the library sources and each is built to install in the custom directory:

export CMAKE_PREFIX_PATH=`pwd`/local

cd tbb22_20090809oss
gmake
cp -Rp include/tbb/ /usr/include/
cp `pwd`/build/*_release/*.so /usr/lib/
ldconfig
cd ..

cd re2c-0.13.5
./configure --prefix=`pwd`/../local
make install
cd ..

cd libevent-1.4.13-stable
cp ../hiphop-php/src/third_party/libevent.fb-changes.diff .
patch < libevent.fb-changes.diff
./configure --prefix=`pwd`/../local
make install
cd ..

cd curl-7.20.0
cp ../hiphop-php/src/third_party/libcurl.fb-changes.diff .
patch -p0 < libcurl.fb-changes.diff
./configure --prefix=`pwd`/../local
make install
cd ..

There is one problem at this point that requires a little surgery on the HipHop PHP source itself. There is more about this in issue #6 and once it gets fixed this won't need to be done.

cd hiphop-php
echo "#ifndef LHASH" >> src/cpp/ext/ext_openssl.h
echo "#define LHASH LHASH_OF(CONF_VALUE)" >> src/cpp/ext/ext_openssl.h
echo "#endif" >> src/cpp/ext/ext_openssl.h

And at last it is time to compile HipHop PHP itself:

git submodule init
git submodule update
export HPHP_HOME=`pwd`
export HPHP_LIB=`pwd`/bin
cmake .
make

It takes about 20 minutes to compile everything. Once the compile is done you are ready to roll. Check out the running HipHop wiki page to learn how to run the resulting binary. One important thing to note is that you need to make sure you have the correct environment variables set when you go to compile things. I created a little file I can source with the following in it:

export HPHP_BASE=<path to the first directory>
export CMAKE_PREFIX_PATH=$HPHP_BASE/local
export HPHP_HOME=$HPHP_BASE/hiphop-php
export HPHP_LIB=$HPHP_HOME/bin

For those who just want it to go I've put all of the above into one script that can be found here. If you are going from Fedora 8 to Fedora 12 on an EC2 node you can get a script for that here.

Now if you want to do this on a 32 bit Fedora 12 install you will need to modify the source first. The easiest way I know of doing this is to look at this commit log or clone my version that can be found here:

git clone git://github.com/carsonmcdonald/hiphop-php.git

Please note that my version my not be up to date and the modifications to get the source to build on the 32 bit system may not be 100% correct. My goal was to get it to build and run on a 32 bit system but I don't have the time to very much more than that.

I'm going to assume you already understand how to do things like create instances, create EBS volumes and ssh into your running instance using key based authentication. I use the AWS management console for a lot of what follows with the exception of needing to register the AMI and for that you will need the Amazon EC2 API Tools and Amazon EC2 AMI Tools

There are two ways to get to a bootable EBS backed Fedora 12 instance and they start off the same. The first thing to do is fire up the AMI named "Basic Fedora Core 8 (AMI Id: ami-84db39ed)" that is provided by Amazon.

Once the Fedora Core 8 EC2 instance is ready ssh into it. Fedora 12 requries a newer version of RPM to install so you now need to upgrade the instance to Fedora 10. This is pretty easy and can be done by following my instructions on upgrading from Fedora 9 to Fedora 10 (don't worry about skipping 9 it will work). Here are the commands needed to do the upgrade:

yum clean all
rpm -Uhv http://archive.kernel.org/fedora-archive/releases/10/Fedora/i386/os/Packages/fedora-release-10-1.noarch.rpm http://archive.kernel.org/fedora-archive/releases/10/Fedora/i386/os/Packages/fedora-release-notes-10.0.0-1.noarch.rpm
yum -y update

After a few minutes the instance will be upgraded and ready for the next step. This is where the two paths diverge depending on how you want the final product constructed. The options are to install Fedora 12 on a freshly minted volume or continue upgrading the instance you just created.

Upgrade path

I will start with the upgrade path since that is probably the easier of the two although may leave you with a messier instances after it is done. The next step for the upgrade path is to do what I outline in upgrading from Fedora 10 to Fedora 11 and upgrading from Fedora 11 to Fedora 12. Here are the commands all in one place to make it easy:

yum clean all
rpm -Uvh http://mirrors.usc.edu/pub/linux/distributions/fedora/linux/releases/11/Fedora/i386/os/Packages/fedora-release-11-1.noarch.rpm http://mirrors.usc.edu/pub/linux/distributions/fedora/linux/releases/11/Fedora/i386/os/Packages/fedora-release-notes-11.0.0-2.fc11.noarch.rpm
yum -y update
yum clean all
rpm -Uvh http://mirrors.kernel.org/fedora/releases/12/Fedora/i386/os/Packages/fedora-release-notes-12.0.0-4.fc12.noarch.rpm http://mirrors.kernel.org/fedora/releases/12/Fedora/i386/os/Packages/fedora-release-12-1.noarch.rpm
yum -y update

Once you have everything upgraded to Fedora 12 you will have a 15G root partition that has less than 2G used. This may not suite your needs very well if you really don't need that extra 13G but thankfully if you want to shrink the root EBS partition you can.

I found some instructions in this article on EBS backed AMIs that describes using the following command to copy the entire file system over. Assuming you have created a smaller volume and attached it to the instance as sdh you should be able to do something like the following to copy everything to the new volume:

mkfs.ext3 /dev/sdh
mount /dev/sdh /mnt
tar cpS / | cpipe -vt -b 1024 | gzip -c | tar zxpS -C /mnt
rm -rf /mnt/mnt/*
rm -rf /mnt/proc/*
umount /mnt

One thing to note in the above is that the entire sdh drive is formatted for the file system (you will actually get a prompt asking if that is ok). As far as I can tell this is the way it has to be or the instance will not boot correctly. I assume this is because the root device is hidden behind a partition already as /dev/sda1 and so shouldn't have a second partition table.

Skip to the common part now to learn how to make the final bootable AMI.

From scratch path

This path is similar to and mostly an update/extension to my post on creating a Fedora 7 AMI setup. I'm going to leave out most of the details and just provide you with a script that will take an empty volume (assumed to be attached as /dev/sdh) and turn it into a bootable EBS backed Fedora 12 volume. Download the script createfedora12bootebs.sh instead of trying to cut and paste the following, it gets formatted in such a way as to lose a newline that is important. Please note that you will need at least 1G of space on the given volume.

#!/bin/sh

echo "y" | mkfs.ext3 /dev/sdh
mount /dev/sdh /mnt

mkdir /mnt/dev
mkdir /mnt/proc
mkdir /mnt/etc

for i in console null zero ; do /sbin/MAKEDEV -d /mnt/dev -x $i ; done

cat <<EOL > /mnt/etc/fstab
/dev/sda1               /                       ext3    defaults 1 1
none                    /dev/pts                devpts  gid=5,mode=620 0 0
none                    /dev/shm                tmpfs   defaults 0 0
none                    /proc                   proc    defaults 0 0
none                    /sys                    sysfs   defaults 0 0
/dev/sdc1               /mnt                    ext3    defaults 0 0
/dev/sdc2               swap                    swap    defaults 0 0
EOL

mount -t proc none /mnt/proc

cat <<EOL > /tmp/yumec2.conf
[main]
cachedir=/var/cache/yum
debuglevel=2
logfile=/var/log/yum.log
exclude=*-debuginfo
gpgcheck=0
obsoletes=1
reposdir=/dev/null

[base]
name=Fedora 12 – i386 – Base
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-12&arch=i386
enabled=1

[updates-released]
name=Fedora 12 – i386 – Released Updates
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f12&arch=i386
enabled=1
EOL

yum -c /tmp/yumec2.conf --installroot=/mnt -y groupinstall Base
yum -c /tmp/yumec2.conf --installroot=/mnt -y install openssh-server

yum -c /tmp/yumec2.conf --installroot=/mnt -y clean packages

echo "UseDNS no" >> /mnt/etc/ssh/sshd_config
echo "PermitRootLogin without-password" >> /mnt/etc/ssh/sshd_config

cp /etc/rc.local /mnt/etc/
cp /etc/sysconfig/network /mnt/etc/sysconfig/network
cp /etc/sysconfig/network-scripts/ifcfg-eth0 /mnt/etc/sysconfig/network-scripts/ifcfg-eth0
cp /usr/local/sbin/* /mnt/usr/local/sbin/
cp -Rp /lib/modules/2.6.21.7-2.fc8xen/ /mnt/lib/modules/

echo "/sbin/MAKEDEV /dev/urandom" >> /mnt/etc/rc.sysinit
echo "/sbin/MAKEDEV /dev/random" >> /mnt/etc/rc.sysinit
echo "/sbin/MAKEDEV /dev/sdc" >> /mnt/etc/rc.sysinit
echo "/sbin/MAKEDEV /dev/sdc1" >> /mnt/etc/rc.sysinit
echo "/sbin/MAKEDEV /dev/sdc2" >> /mnt/etc/rc.sysinit

cat <<EOF >> /mnt/etc/rc.sysinit
# The following will partition the local drive and set up swap
cat <<EOL | fdisk /dev/sdc
n
p
1
1
+140G
n
p
2

w
EOL
mkswap /dev/sdc2
EOF

mv /mnt/lib/tls /mnt/lib/tls.disabled
echo "hwcap 0 nosegneg" >> /mnt/etc/ld.so.conf.d/kernelcap-2.6.21.7-2.fc8.conf

chroot /mnt chkconfig --level 2345 NetworkManager off
chroot /mnt chkconfig --level 2345 network on

sync
umount /mnt/proc
umount /mnt

Common wrap up

At this point you will need to create a snapshot of the volume that was created for one of the paths above. Once the snapshot is available you will need to then register the snapshot as an AMI that is bootable from EBS. To do that you would issue something like the following command substituting the correct data in where it relates to your volume and snapshot.

ec2-register -n "AMIName" -d "AMI Description" --block-device-mapping /dev/sdc=ephemeral0 --snapshot your-snapname --architecture i386 --kernel aki-a71cf9ce --ramdisk ari-a51cf9cc

One thing to note in this command is the –block-device-mapping option. That option is what gives you access to the local drive on your node once it is booted. This gives you extra storage for things you don't need to keep after the life of the running node. In the from scratch option I'm turning part of the local drive into swap as well as creating a partition that could be used as a large temporary storage. If you want to know more details on the ephermeral storage look at this post

After all that you should have a bootable EBS backed Fedora 12 install to work with.

"Commands out of sync; you can't run this command now"

Details about this error can be found in the mysql docs. Reading those details makes it clear that the result sets of a prepared statement execution need to be fetched completely before executing another prepared statement on the same connection.

Fixing the issue can be accomplished by using the store result call. Here is an example of what I initially was trying to do:

<?php

  $db_connection = new mysqli('127.0.0.1', 'user', '', 'test');

  $post_stmt = $db_connection->prepare("select id, title from post where id = 1000");
  $comment_stmt = $db_connection->prepare("select user_id from comment where post_id = ?");

  if ($post_stmt->execute())
  {
    $post_stmt->bind_result($post_id, $title);

    if ($post_stmt->fetch())
    {
      $comments = array();

      $comment_stmt->bind_param('i', $post_id);
      if ($comment_stmt->execute())
      {
        $comment_stmt->bind_result($user_id);
        while ($comment_stmt->fetch())
        {
          array_push($comments, array('user_id' => $user_id));
        }
      }
      else
      {
        printf("Comment statement error: %s\n", $comment_stmt->error);
      }
    }
  }
  else
  {
    printf("Post statement error: %s\n", $post_stmt->error);
  }

  $post_stmt->close();
  $comment_stmt->close();

  $db_connection->close();

  printf("ID: %d -> %s\n", $post_id, $post_title);
  print_r($comments);
?>

The above will result in the following error:

Here is what needs to be done to make it work correctly:

<?php

  $db_connection = new mysqli('127.0.0.1', 'user', '', 'test');

  $post_stmt = $db_connection->prepare("select id, title from post where id = 1000");
  $comment_stmt = $db_connection->prepare("select user_id from comment where post_id = ?");

  if ($post_stmt->execute())
  {
    $post_stmt->store_result();
    $post_stmt->bind_result($post_id, $title);

    if ($post_stmt->fetch())
    {
      $comments = array();

      $comment_stmt->bind_param('i', $post_id);
      if ($comment_stmt->execute())
      {
        $comment_stmt->bind_result($user_id);
        while ($comment_stmt->fetch())
        {
          array_push($comments, array('user_id' => $user_id));
        }
      }
      else
      {
        printf("Comment statement error: %s\n", $comment_stmt->error);
      }
    }

    $post_stmt->free_result();
  }
  else
  {
    printf("Post statement error: %s\n", $post_stmt->error);
  }

  $post_stmt->close();
  $comment_stmt->close();

  $db_connection->close();

  printf("ID: %d -> %s\n", $post_id, $post_title);
  print_r($comments);
?>

A couple things to note about the above example:

• The bind and fetch on the statement still works correctly.
• Make sure the results are freed when the processing is done.

The Aptana Air plugin supports developing Adobe Air applications using HTML and Javascript. It even support the 2.0 release of Air that is currently in beta. Aptana uses the Eclipse framework as an editor so if you are familure with Eclipse it will be even easier to use.

I started by downloading and installing the latest version of the Air runtime. Next I grabbed the Air SDK, the SDK doesn't come with the plugin so it is something you have to get directly from the Air developers site. After getting the SDK unpacked I installed the latest Aptana core release. Once the core is installed there is a big plugin button on the startup screen that currently has Air listed.

The install went smoothly except for a few issues. The first one I ran into was very noticeable since it kept any dialog buttons from working when they were clicked although they did work when I clicked them and then hit enter or navigated to them with the keyboard. Luckily someone has already figured out that there is an issue with Eclipse and GTK+ that is the cause (even though the post is for Ubuntu the same problem and solution worked for me on Fedora). The fix is to set the GDK_NATIVE_WINDOWS variable before running the Aptana binary:

The next thing I noticed was the application.xml descriptor that Aptana created didn't generate correctly. It needs to start with the correct xmlns or the following error will be thrown on run: "invalid application descriptor: descriptor version does not match runtime version". To fix this check the version of the Air SDK by running the following command:

For the version of the Air SDK I downloaded the correct xmlns was http://ns.adobe.com/air/application/1.5 so I needed the following application tag:

Once I had that working I was able to compile and execute a demo application. I was also able to create an Air application package from within Aptana using File > Export > Adobe AIR > Adobe AIR Package. Before creating the Air package I had to create a signing certificate. Creating the certificate can be done within Aptana too but because I had not yet fixed the above button issue I created a cert on the command line with the Air SDK and then imported it. To create the Air signing certificate from the command line I used the adt command from the SDK:

Remember the password that gets used to generate the certificate because it will have to be used before a package is signed.

Finally Adobe has a lot of information on developing Air applications on their Air devnet site. The Air ajax section is especially important.

Some of the following grew out of the requirement that you verify for a lot of the Heroku addons and verification requires a credit card. For some of these tips there are easier paths if you don't mind paying a little extra or just verifying your account.

• Configuration
• Set up a custom domain name
• Poor man's cron
• Sending mail with GMail
• Keeping your logs
• Rack works too
• Compiled gems
• Understanding dynos

Configuration

For configuration Heroku has a way but for some reason it didn't seem like the best or easiest way to do things. I started off setting up a configuration file as described in this railscast on using yaml configuration files. I took a while to dig a little more and found something similar but even better way of doing configuration with Heroku.

Set up a custom domain name

To make a more professional looking app you probably want to have a non-Heroku base domain name. They make it very easy to set that up using a CNAME DNS entry.

Poor man's cron

One of the things I ran into pretty quickly was the need to run a task once every minute. Heroku offers both cron and delayed job. The cron jobs offered are a little limited since the finest grained execution for cron is once an hour. I believe delayed jobs could be made to run every minute but the next issue with both cron and delayed job on Heroku is that they are addons that require verification.

The solution for me was to create a poor man's cron. I added a controller that would execute the task and then ran wget to hit the controller from an external server. This isn't a great solution but for testing the service it worked fine. The main note on doing this is to keep in mind that you will tie up a dyno for the length of the request, see Understanding dynos for more.

Sending mail with GMail

If you want to send email you have a number of different options on Heroku. First off it is important to note that Heroku doesn't support sending email from their systems directly but instead they support outgoing SMTP. Sending mail with the Sendgrid addon is probably the most flexible option but the GMail SMTP option is the least costly. The GMail option gives you 500 emails a day and that was plenty for my use.

Even though there is an addon for the GMail option you don't actually need to use it. Instead install the action mailer optional TLS plugin. Follow the readme to get it installed and configured. Then you put something like the following into your production.rb file:

  config.action_mailer.delivery_method = :smtp
  config.action_mailer.raise_delivery_errors = true
  ActionMailer::Base.smtp_settings = {
    :tls            => true,
    :address        => 'smtp.gmail.com',
    :port           => 587,
    :domain         => 'example.come',
    :authentication => :plain,
    :user_name      => 'support@example.com',
    :password       => 'password'
  }

Keeping your logs

One downside to Heroku is that they only retain a small portion of your logs. They indicate in their docs that they only retain 100 lines of logs. So if you want to track your log output you will need to store them outside of the service. They give two suggestions in the docs for logging but there are other ways to do something similar on your own.

The solution I found that seems like the easiest is to use MongoDB to log since it is "fantastic for logging". You will need to have a MongoDB server available first. If you don't want to install MongoDB on your own external server you can try the new MongoDB hosted solution.

Once you have the MongoDB server ready you will need to add the MongoDB gem to your .gems file:

mongodb-mongo --source gems.github.com

Then you will want to install the mongo db logger plugin in your rails project. Just follow the instructions they give on the project page to get it installed.

Your app/controllers/application_controller.rb will look something like this:

class ApplicationController < ActionController::Base
  include MongoDBLogging

  helper :all # include all helpers, all the time
  protect_from_forgery # See ActionController::RequestForgeryProtection for details
end

After you get the plugin installed you will have to hard code the MongoDB information into the plugin. This could probably be fixed and pulled out of a configuration file but it won't work the way it is set up out of the box. Edit the file vendor/plugins/mongo_db_logger/lib/mongo_logger.rb and change the db_configuration to match your setup. It should be something like this:

  db_configuration = {
    'host'    => 'my.mongohost.com',
    'port'    => 56700,
    'database'    => 'testapp',
    'capsize' => default_capsize}

Now you should be able to deploy to Heroku and see your logs show up in your MongoDB database.

A couple very important things to note are that this logging won't catch exceptions that happen above the application itself and if your MongoDB server goes down your app may hang trying to connect to it.

Rack works too

You aren't limited to just Ruby on Rails with Heroku, rack works as well. That opens the door for other frameworks like Sinatra, Merb and Camping on Heroku. It is also possible that more in depth logging could be done using clogger or even rack a mole.

Here is an example Camping application:

The .gems file:

camping

The config.ru file:

require 'hello'
run Rack::Adapter::Camping.new(Hello)

The hello.rb file:

require 'camping'

Camping.goes :Hello

module Hello::Controllers
  class Index < R '/'
     def get
        render :hello
     end
  end
end

module Hello::Views
  def hello
     p  "Hello World!"
  end
end

Compiled gems

Heroku also lets you use gems that need to be compiled before they are installed. You can see this if you try to use something like memprof however you will also notice that in the memprof case it won't actually work because there are missing libraries. So using gems that require libraries may be hit or miss.

If you are lucky you may find that the gem is already part of Heroku like RMagick is. Here is a little example of using Heroku and the RMagick gem to produce an image using Camping:

The .gems file:

camping
rmagick

The config.ru file:

require 'hello'
run Rack::Adapter::Camping.new(Hello)

The hello.rb file:

require 'camping'
require 'RMagick'

Camping.goes :Hello

module Hello::Controllers
  class Index < R '/'
     def get
       @headers["Content-Type"] = "image/gif"
       img = Magick::Image.new(200, 200)

       gc = Magick::Draw.new
       gc.gravity = Magick::CenterGravity
       gc.pointsize = 32
       gc.font_family = "Helvetica"
       gc.font_weight = Magick::BoldWeight
       gc.stroke = 'none'
       gc.annotate(img, 0, 0, 0, 0, "Hello world!")

       img.format = "GIF"
       img.to_blob
     end
  end
end

Understanding dynos

It is important to understand how dynos work. The bottom line is that you need one for every concurrent request. This is easy to demonstrate with the following modification of the Camping example above:

require 'camping'

Camping.goes :Hello

module Hello::Controllers
  class Index < R '/'
     def get
        render :hello
     end
  end
end

module Hello::Views
  def hello
     sleep 10
     p  "Hello World!"
  end
end

If you deploy this application to the free version of Heroku and then open two requests to it at the same time the second request will hang until the first one completes. This should illustrate why it is important to keep your processing quick and have enough dynos to match the concurrent request needs of your application.

When I first started down the path of using EC2 I thought I would grab the source each time I wanted to build. I quickly ran into a snag however because it took forever to sync the source and download the Ubuntu repo. Once I had the initial sync of the source I decided I would copy it all to an EBS volume and keep that volume up to date. Using EBS to store the source feels better too since I assume Google expects people to be syncing changes only as opposed to pulling the entire source tree down every time they want to build.

I started out by finding this Ubuntu AMI for a base to work from. For the most efficient compile times I ended up using the High CPU (c1.medium) instance. I started with the default small instance but it was just too slow. With the high cpu instance you are looking at about 45 minutes to build the OS after you have the source synced for the first time and if you add building Chromium in there you are looking at around 55 additional minutes. All told you can have a complete build in less than 2 hours even if there are some source updates needed. For EBS you need a 3G volume for the Chrome OS source plus Ubuntu package repo and a 4G volume for the Chromium source.

I've bundled everything up into one script called builder.sh and if you don't care to know the details you can download it and give it a try. There are some things you need to know about it however even if you don't want to follow all the details. First there are some of the assumptions made by the script:

1. The EC2 AMI ami-ccf615a5 image is used on the node it is run on.
2. If you are storing the source then you have attached the EBS volumes and you have initialized a filesystem on both with mkfs.ext3
3. If you are storing the source then you have attached the EBS volumes and indicated what devices they are using OSVOLDEV and BROWSERVOLDEV in the following config section
4. You are running the script as the root user

Next is a small configuration area at the top of the script that lets it know what devices the two volumes will be mounted on. If you don't intend on having stored source then you don't need to worry about setting them to anything as long as what they are set to doesn't match a real device.

There are a few command line options that will let you tune the script if you want to skip parts of the build process:

• --skip-sync – This will skip trying to sync any of the repos. If you aren't using EBS to store the source the script will fail if you use this option since there will be nothing to build.
• --skip-chrome-build – This will skip trying to build chrome. You have to build chrome at least once for the chrome os build to work.
• --skip-chrome-os-build – This will skip trying to build chrome os.

Here are a few other important things to make note of:

• The build script adds a user named "gogo" that can be used to log in if you don't want to use your Google login.
• The build script sets the password for root access to "gogo".
• Running the build script should result in an image named /mnt/builder/.usb.img.bz when everything is done. This is a compressed bootable image that can be written to a USB key (use something like dd if=usb.img of=/dev/usbkeydevice bs=10M).
• I could imagine the same instructions and script being re-purposed for building on a non-EC2 system too.
• The script can be run multiple times on the same node and will skip the parts that should only be done once.
• There are probably ways to speed up the build to make it even faster. With some more effort the build for the browser could be kicked off while the source for the OS was still be synced. I also wondered if using a 64 bit EC2 node and creating a large ramdisk would help but I didn't want to fiddle with 64 bit builds of Chrome, I may revisit if I find time.
• I was tempted to build an AMI that included all the prerequisites but it only takes 5 minutes to pull all of those together so I decided not to. It could streamline things to do that but I'm not sure that it is worth the effort.

If you are looking for more information about the script, places where it might make sense to modify the script and the build process in general then read on.

To get started there are a number of prerequisites that need to be installed. These are the required parts for both Chromium and Chromium OS. One note here is that there is some issue with lighttpd on Ubuntu that causes it to die while the apt-get is downloading all the packages so I had to resort to installing Apache as well. I make sure lighttpd isn't running since the prerequisites install it.

The next step is to create a user that will actually run the build. It turns out to be important that this user is not root so that is why there are some hoops being jumped here. The two source volumes get mounted into the user's home directory here as well. The last part of this section builds a filesystem to be used as temporary storage for output images. The way they build is set up it will push the final images into the source tree and because that is stored on EBS it could end up eating up a lot of room, there is more to this later in the Chrome OS build section.

The next sections each sync either source or the apt-get repository. The first is syncing the apt-get repository using apt-mirror. In case you missed it the mirror was defined in the prerequisites section above.

The next is the Chromium OS source repository.

There are a few extra parts to syncing the Chromium browser. One of those extras is to make sure the third party test suites don't get synced since they are large.

Building the Chromium browser. This isn't needed every time. When it is done it sticks the browser in place for the Chromium OS build.

Finally the place where the OS actually gets compiled. The majority of this is right out of the build instructions. One thing to understand here is that the build process creates a chroot environment and then maps the source into that before the build starts. The build is done inside the chroot environment so the build script has to create a temporary script with all the commands that need to run there. One other note here is that the images filesystem created above has to be mapped into the chroot environment and I did that by setting it up as a loopback device then mounting it from inside the chroot environment. After the build is complete I can then get to the filesystem outside of the chroot environment to grab the final image. If you want to change the test user, the system password or get rid of either of those options this is the section to change.

A final bit of glue looks for command line arguments ties everything together.

• Updated window managers Gnome 2.28, KDE 4.3 and Fedora Moblin
• Delta RPM support
• i686 as the base architecture
• Lots of virtualization changes: KSM, KVM huge page support, KVM QCow2 performance improvements, KVM Stable Guest ABI, libguestfs, Virtual network management and improved virtual privileges to name a few
• An easier to use bug reporting interface Abrt 1.0
• Better Webcam Support

You can find the complete list of Fedora 12 enhancements as well if you want more details.

I'm again starting with PreUpdate since it worked well last time. The steps are pretty much the same as last time but I did have more problems after the upgrade:

1. You have to be at Fedora11 before you try this. If you aren't there follow the steps to get to Fedora11.
2. yum clean all
3. yum udpate
4. Make sure to back up your xorg.conf since it can disappear with the upgrade.
5. preupgrade-cli "Fedora 12 (Constantine)" The total download for the upgrade from Fedora 11 to Fedora 12 was 1.1GB for me
6. After I rebooted the first time I got an error that the /boot directory didn't have enough space. It turns out I needed about 27M of free space so I ended up having to delete old kernels until I had 28M of free space.
7. reboot and wait

I lucked out again and didn't have to remove anything to fix dependency issues. I recompiled my existing NVIDIA driver but I ran into an issue with the nouveau NVIDIA driver that comes with Fedora 12. I needed to remove the nouveau package, rebuild initrd and reboot before I could build the NVIDIA driver:

If that is too much work you can also get the NVIDIA driver from atrpms but you will still need to remove the nouveau driver first. I also needed to pull down the Fedora 12 version of Virtualbox but so far that was all.

If you prefer the yum upgrade option here are the steps for that as well:

1. yum clean all
2. yum update
3. rpm -Uvh http://mirrors.kernel.org/fedora/releases/12/Fedora/i386/os/Packages/fedora-release-notes-12.0.0-4.fc12.noarch.rpm http://mirrors.kernel.org/fedora/releases/12/Fedora/i386/os/Packages/fedora-release-12-1.noarch.rpm
4. yum clean all
5. yum -y update
6. You may need to resolve dependencies and then do another yum -y update
7. reboot

I had to resolve dependencies to get this to work. I had to remove tigervnc-server-1.0.0-2.fc11.i586 and VirtualBox-3.0.10_54097_fedora11-1.i586. The resulting update was about 1.4G so it took a little while to apply.

I've also put together a few videos and screenshots if you want to get a quick preview of what the different versions available look like. You can also view the videos on Youtube: Moblin, KDE and Gnome

This post goes over how I went about configuring Sphinx and gives an example of how to integrate it with PHP. I'm using MySQL as the data store filled with the Stack Overflow CC data dump although it should be easy to adapt the instructions to other data sources. To follow along just download a copy of the data dump and use my schema and loader to get the same MySQL database.

I've broken the setup down into the following 4 steps:

1. Configuring Sphinx
2. Building an Index with Sphinx
3. Searching with Sphinx
4. Using the Sphinx PHP API

As a side note, Sphinx is available as a package under Fedora 11 and I assume the same of other Linux distributions so it shouldn't be too hard to install. I am using version 0.9.8.1 compiled from source.

Configuring Sphinx

The Sphinx configuration file can be be broken down into different parts with some of those parts being used by the indexer and some by the search service. I have broken the configuration file I am using into different sections but I include the full file for download at the end. The first segment of the configuration describes where to get the data for the index, in this case it is a MySQL database:

There are a few notable parts to the above. The field "sql_query" defines the query that grabs data from the database for the index. The "sql_attr_*" values point out the columns in the data that will be used in search queries later to sort or group by, more on that in the searching section.

The next section defines the index:

In the above there are two index definitions. The first index is very basic and has no morphology defined. The second index is derived from the first index and uses a stemming morphology. I'm also having any HTML stripped out of the content since there is some in the data and it doesn't need to be searchable in the index.

The next block in the configuration file defines how the indexer will run. Here I'm limiting the amount of memory used to 32M as an example of what could be done:

The final part of the configuration file sets options for the search daemon:

You can download the full configuration file here: sphinx.conf

The above configuration gets you the basics. If you want to find out more you can check out the indexing documentation.

Building an Index with Sphinx

The complexity of building and maintaining an index goes up with the frequency of the updates that are required to keep it up to date. Luckily the data set I'm working with gets updated once a month so it is fairly static. Because the data is updated monthly I only have to update the index once a month so this makes the indexing simple.

This is the command I use to create the index:

indexer --all --config sphinx.conf

Indexing the entire data set I have only takes a few minutes so re-indexing speed isn't an issue for me and the resulting index is in the 250M range as of this post so size isn't necessarily an issue either. However I currently create the index on one machine and then upload it to the server so a 250M transfer could become a bandwidth hog if I needed to update the index more often. If I were able to get deltas from the data dump I would look into using the index merging feature and then transferring the delta index.

Searching with Sphinx

Sphinx offers a lot of flexibility in configuring how a search is run and I found myself needing that flexibility to get better results. The search command line interface (see the reference for all the options) can be used to test a few of the options but you can't do everything with it that you can with the various APIs available.

The first place I started changing defaults with the search is in how it matches the search words. The default is to find all words:

search --config sphinx.conf "php full text search"

Here is an example of the output you get from the CLI search:

The output shows you the "document id" as well as the weight and any "sql_attr_" values. I used the CLI search to get the weights for each search change I made before I started working with the API. Please note that the extended information in the above example output only shows up if the sql_query_info query is set correctly in the configuration file.

Next I tested with "any" and "extended search version 2" before settling on extended version 2:

search --any --config sphinx.conf "php full text search"

search --ext2 --config sphinx.conf "php full text search"

The CLI search will also perform sorts. It is important to note that you can only sort on values that were indexed using the "sql_attr_"* configuration options. Here is an example of sorting by an indexed value:

search --sortby "date_added desc" --config sphinx.conf "php regex"

This is what happens when you try to sort based on an un-index attribute:

search --sortby "id desc" --config sphinx.conf "php regex"

Sphinx 0.9.8.1-release (r1533)
Copyright (c) 2001-2008, Andrew Aksyonoff

using config file 'sphinx.conf'...
index 'so_2009_10': search error: failed to create sorting queue: sort-by attribute 'id' not found.

One of the things that the CLI search interface can't do is change weightings for the different data points. I found that weighting the title more than the body made sense and that is reflected in the PHP code that follows in the API section.

The last part of searching to cover is searchd server. This is the integration point for the various Sphinx APIs and I assume it is running for the examples in the PHP API section. If the configuration example given in the first section is used all that needs to be done is to start the daemon. The following is a simple init script I use to start and stop searchd:

#!/bin/sh

case "$1" in
'start')
searchd --config /etc/sphinx/sphinx.conf
;;
'stop')
killall -9 searchd;
;;
*)
echo "Usage: $0 { start | stop}"
exit 1
;;
esac
exit 0
#

Using the Sphinx PHP API

The Sphinx PHP API is included in the Sphinx source. The API is contained in one file named sphinxapi.php that is located in the api directory of the source. Make sure the library is in a place where it can be included before trying out the examples. There is also some documentation for the PHP API.

In the following example I'm running the search query then serializing it to JSON format. Notice that the title is given a weight of 70 and the body_text is given a weight of 30 so the results will focus more on the title than what is in the body:

Although I picked PHP there are a wide range of language specific libraries available.

Overall it didn't seem too difficult to set up Sphinx and have it serving search results quickly. I liked that it is very light weight and doesn't need a lot of resources or require a lot of extra parts to be installed.