TJ Stein

Running WordPress on Heroku + Amazon RDS

2011-09-16T00:00:00-07:00

With the announcement of the Heroku and Facebook partnership yesterday, Heroku quietly confirmed support for two new languages, Python and PHP. The self-proclaimed polyglot platform, Heroku's Celadon Cedar Stack now has a considerable advantage over newcomers like PHP Fog, and Orchestra.io by avoiding mostly separate, language-specific products.

To test out Heroku's PHP support (version 5.3.6), I deployed WordPress. One of the current limitations is the lack of native MySQL support, outside of hooking into the Xeround Cloud DB or Amazon RDS. I've found one blog post that suggested using the default PostgreSQL backend that Heroku provides with the 'PG4WP' WordPress plugin, enabling WordPress to be used with a PostgreSQL database. While this may work, it lacks most plugin support and is more of a bandaid for the platform limitations. Instead, you can use Amazon's Relational Database Service (RDS) addon.

Amazon RDS is a service that allows you to set up, operate and scale a dedicated MySQL database server on top of EC2. In addition to standard MySQL features, RDS offers the following functionality:

Automated backups
Point-in-time recovery
Seamless vertical scaling between instance types

The free Amazon RDS add-on lets you connect your Heroku app to an RDS instance and seamlessly use it in place of the standard, Heroku-provided PostgreSQL database. To get started, you should configure the RDS command line toolkit and Heroku gem if you haven't already. Let's start by creating the RDS database instance on your local machine:

rds-create-db-instance --db-instance-identifier [name]\
  --allocated-storage 5 \
  --db-instance-class db.m1.small  \
  --engine MySQL5.1 \
  --master-username [user] \
  --master-user-password [pw] \
  --db-name [name] \
  --headers

This will take a few minutes. Once the database is provisioned, add your local IP address to the security group -- assuming your workstation’s public IP is 1.1.1.1:

rds-authorize-db-security-group-ingress default --cidr-ip 1.1.1.1/32

Heroku also needs to be able to access your RDS instance. To allow Heroku’s cloud through the RDS firewall, run the following command:

rds-authorize-db-security-group-ingress default \
    --ec2-security-group-name default \
    --ec2-security-group-owner-id 098166147350

Now we can begin building the application layer. Since we're going to be using Git for version control, I'd suggest cloning WordPress from Mark Jaquith's GitHub repository; it is synced from Automattic's SVN repository every 30 minutes, including branches and tags:

git clone git://github.com/markjaquith/WordPress.git
cd WordPress

Before we start making any changes to file structure, we should make our own Git repository and start committing. Note here that you'll want to populate the wp-config.php with the MySQL credentials from the rds-create-db-instance command above:

git init
mv wp-config-sample.php wp-config.php
git add .
git commit -m 'initial commit'

Now, create the stack and enable the RDS addon with your MySQL credentials. Once the stack has been created, you can deploy:

heroku create --stack cedar
heroku addons:add amazon_rds url=mysql://user:pass@rdshostname.amazonaws.com/databasename
git push heroku master

The output should look something like this:

➜ wp-heroku-test git:(master) git push heroku master
Counting objects: 985, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (965/965), done.
Writing objects: 100% (985/985), 3.65 MiB | 221 KiB/s, done.
Total 985 (delta 66), reused 0 (delta 0)

-----> Heroku receiving push
-----> PHP app detected
-----> Bundling Apache v2.2.19
-----> Bundling PHP v5.3.6
-----> Discovering process types
       Procfile declares types -> (none)
       Default types for PHP   -> web
-----> Compiled slug size is 24.9MB
-----> Launching... done, v4
       http://evening-waterfall-3372.herokuapp.com deployed to Heroku

To git@heroku.com:evening-waterfall-3977.git
 * [new branch]      master -> master

Notes

A commenter on HackerNews noted that the slug is still read-only, but the ephemeral filesystem is writable. The slug is what gets deployed on each new dyno spawned. The ephemeral filesystem is the individual file system on each dyno. So a plugin like WP Super Cache would be able to write to the file system, but that cache would only exist for the individual dyno that wrote it.
Because of the usage concerns of media and content uploads, I'd suggest using a CDN or Amazon S3 for storing images and attachments.
The zlib extension for PHP is not compiled on the Celadon Cedar Stack. Theme and plugin uploads through the WordPress admin panel will fail. To get around this, set up your themes and plugins on your local workstation first, commit and deploy.
Do not include a phpinfo page in the document root as it will contain your database credentials in plain text.
If you want to have pretty permalinks, create the .htaccess file on your local machine and populate the mod_rewrite rules prior to deploying.

Live Data Replication Using lsyncd

2011-08-29T00:00:00-07:00

One of my favorite Linux utilities I've discovered recently is lsyncd, a live syncing (mirror) daemon. Following the traditional Unix philosophy, it does data replication simply and it does it very well. Using some fancy inotify magic, lsyncd will spawn one or more processes to synchronize the targets after changes have been made.

After determining that a client would need multiple web-servers running in sync, I evaluated a few different tools that perform data replication and live syncing. This is one of the larger problems I've encountered with scaling web applications horizontally -- storage management among multiple web services. In traditional deployments, it might make sense to use something like NFS or even DRBD, operating on the block device level. While this works for write-heavy systems under high load, it isn't practical mirroring a whole block device for a high-traffic WordPress site -- I just needed a basic client-server model.

This is where lsyncd really shines. The lsyncd configuration file is written in Lua and super easy to set up. Below is the lsyncd.conf that duplicates /var/www/tjstein.com on the master server to the 4 targets with the same path:

settings = {
   delay        = 1,
   maxProcesses = 5,
   statusFile   = "/tmp/lsyncd.status",
   logfile      = "/var/log/lsyncd.log",
}

targetlist = {
 "10.0.1.23:/var/www/tjstein.com",
 "10.0.1.24:/var/www/tjstein.com",
 "10.0.1.25:/var/www/tjstein.com",
 "10.0.1.26:/var/www/tjstein.com"
}

for _, server in ipairs(targetlist) do
  sync{ default.rsync,
    source="/var/www/tjstein.com",
    rsyncOpts="-rltvupgo",
    target=server
  }
end

To make the rsyncing seamless, you'll need to send the public key generated on the master server to all of the duplicated nodes for easy access. With the client I was working with, we were also using Nginx to load balance across the web nodes. Since we only wanted writes going to the primary master server, we redirected all requests to /wp-admin to the primary node. Then, any changes made through the WordPress backend were synced over to the targets in less than 1 second. Here is the Nginx configuration:

upstream backend  {
  server 10.0.1.20; #master
  server 10.0.1.21; #www-01
  server 10.0.1.22; #www-02
  server 10.0.1.23; #www-03
}

upstream admin {
  server 10.0.1.20;
}

server {
  server_name www.tjstein.com tjstein.com;

  location / {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass  http://backend;
  }

  location ~ /wp-admin/* {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass  http://admin;
  }
}

Notes

You'll want to install lsyncd from source on most distributions. Older versions (pre 2.x) contained gross XML style configuration; the most recent version, as of 8/30/2011, is 2.0.5.

APC Cache Purge Plugin for WordPress

2011-08-23T00:00:00-07:00

After a recent WordPress update, I noticed that the admin panel did not correctly reflect the new version as indicated in the footer of the backend panel. I realized that the APC opcode cache was not flushed and therefore held on to cached versions of many updated files. I thought about what other ways this cache could be flushed without restarting the web server or PHP daemon (PHP-FPM) without compromising service availability.

Some subsequent googling later, I found a dirty hack that, although helpful, was somewhat limited by design -- it suggests adding the script to the functions.php file of your theme. This means you'd need to manually insert the code for each theme. Also, apc_clear_cache() did not work by default with APC version 3.1.3p1. From this, I created a simple, single purpose WordPress plugin to flush the APC cache.

Once activated, you'll get a 'Purge APC' option under the tools menu. Once the option is clicked, the APC cache is flushed and the entries are displayed on the page:

Feel free to fork or submit pull requests to the GitHub repository.

<?php
/**
 * @package APC Cache Purge
 * @version 0.1
 */
/*
Plugin Name: APC Cache Purge
Plugin URI: http://tjstein.com
Description: This is a simple, single purpose plugin to flush the APC cache.
Author: TJ Stein, inspired by Kaspars Dambis of konstruktors.com
Version: 0.1
Author URI: http://tjstein.com
License: GPLv2
*/
function apc_purge() {
    return apc_clear_cache('opcode');
}
// Add Purge APC menu under Tools menu
add_action('admin_menu', 'php_apc_info');
        
function php_apc_info() {
    add_submenu_page('tools.php', 'Purge APC', 'Purge APC', 'activate_plugins', 'flush_php_apc', 'php_apc_options');
}
        
function php_apc_options() {
    if (apc_purge() && apc_purge('user'))
        print '<p>Success!</p>';
    else
        print '<p>Clearing Failed!</p>';
    print '<pre>'; print_r(apc_cache_info()); print '</pre>';
}
// Add Purge APC in the favorite actions dropdown
add_filter('favorite_actions', 'clear_apc_link');
        
function clear_apc_link($actions) {
    $actions['tools.php?page=flush_php_apc'] = array('Purge APC', 'edit_posts');
    return $actions;
}
?>

Installation

Upload apc-cache-purge.php to the /wp-content/plugins/ directory
Activate the plugin through the ‘Plugins’ menu in WordPress
When needed, flush the cache by clicking on ‘Purge APC’ under the Tools section

Alsatian Darn

2011-04-22T00:00:00-07:00

"Alsatian Darn" from the album "Tomboy" (2011) performed by Panda Bear.

Classy Web Development with Sinatra and Heroku

2011-04-01T00:00:00-07:00

Since my recent switch from WordPress to Jekyll, I've really enjoyed playing around with Ruby. Although I don't have much programming experience, I've been fascinated with one particular Ruby framework, Sinatra.

Sinatra is a DSL (domain-specific language) for quickly creating web applications in Ruby with very little effort. It is a minimalist framework, built right on top of Rack, a standard interface for Ruby web frameworks. Unlike rails, you won't find many bells and whistles here. There are no models, views or controllers by default, but that's where Sinatra really shines. You can use Sinatra to create lean, focused web applications in just a few lines of code.

So this morning, I published a Sinatra template for Heroku with Haml, Sass & jQuery. The repository includes a base HTML 5 template for Sinatra, ready for Heroku deployment. Use bundle install to grab all of the gem dependencies. If you get lost during deployment, check out Getting Started with Heroku

One of the fun parts of the project was using Haml in place of erb. I'm still pulling up the Haml reference here and there but it's a breeze to work with. Here is an example of the layout page:

!!! 5
%html{html_attrs('en-en')}
  %head
    %meta{:'http-equiv' => "Content-Type", :content => "text/html; charset=utf-8"}
    %meta{:name => "lang", :content => "en"}
    %title Sinatra Template for Heroku w/ Haml, Sass & jQuery | TJ Stein
    %link{:href => "assets/css/style.css", :rel => "stylesheet", :type => "text/css"}/
  %body
    .container
    = yield
    = haml :analytics, :layout => false

The actual application file doesn't have much in it. It loads the required gems, builds the routes including the Sass support. You can add or modify any of the routes to create new pages or functions:

require 'sinatra'
require 'haml'
require 'sass'

set :haml, :format => :html5

get '/' do
  haml :index
end

get '/:path' do
  haml params[:path].to_sym
end

get '/style.css' do
  content_type 'text/css', :charset => 'utf-8'
  scss :style
end

If you prefer using a Rakefile to deploy, I've included an example Rakefile with a rake task for deployment to Heroku using rake deploy:

require 'rake'

desc "Deploy to Heroku."
task :deploy do
   require 'heroku'
   require 'heroku/command'
   user, pass = File.read(File.expand_path("~/.heroku/credentials")).split("\n")
   heroku = Heroku::Client.new(user, pass)

   cmd = Heroku::Command::BaseWithApp.new([])
   remotes = cmd.git_remotes(File.dirname(__FILE__) + "/../..")

   remote, app = remotes.detect {|key, value| value == (ENV['APP'] || cmd.app)}

   if remote.nil?
   raise "Could not find a git remote for the '#{ENV['APP']}' app"
   end

   `git push #{remote} master`

   heroku.restart(app)
end

If you end up using the template in your own application or want to contribute, send a pull request or just fork the project. Keep it classy.

Mounting S3 Buckets Using FUSE

2011-02-22T00:00:00-08:00

Over the past few days, I've been playing around with FUSE and a FUSE-based filesystem backed by Amazon S3, s3fs. Until recently, I've had a negative perception of FUSE that was pretty unfair, partly based on some of the lousy FUSE-based projects I had come across. I'm sure some of it also comes down to some partial ignorance on my part for not fully understanding what FUSE is and how it works.

What is FUSE?

FUSE is a loadable kernel module that lets you develop a user space filesystem framework without understanding filesystem internals or learning kernel module programming. This basically lets you develop a filesystem as executable binaries that are linked to the FUSE libraries. So, if you're not comfortable hacking on kernel code, FUSE might be a good option for you.

FUSE + Amazon S3

So, now that we have a basic understanding of FUSE, we can use this to extend the cloud-based storage service, S3. Using a tool like s3fs, you can now mount buckets to your local filesystem without much hassle. Although your reasons may vary for doing this, a few good scenarios come to mind:

Your server is running low on disk space and you want to expand
You want to give multiple servers read/write access to a single filesystem
You want to access off-site backups on your local filesystem without ssh/rsync/ftp

To get started, we'll need to install some prerequisites. I've set this up successfully on Ubuntu 10.04 and 10.10 without any issues:

aptitude -y install build-essential libfuse-dev fuse-utils libcurl4-openssl-dev 
aptitude -y libxml2-dev mime-support pkg-config

Now you'll need to download and compile the s3fs source. As of 2/22/2011, the most recent release, supporting reduced redundancy storage, is 1.40. Check out the Google Code page to be certain you're grabbing the most recent release.

cd /usr/local/src
wget http://s3fs.googlecode.com/files/s3fs-1.40.tar.gz
tar zxvf s3fs-1.40.tar.gz && cd s3fs-1.40
./configure
make && make install

This will install the s3fs binary in /usr/local/bin/s3fs. You can add it to your .bashrc if needed:

echo "export s3fs='/usr/local/bin/s3fs'" >> ~/.bashrc
source ~/.bashrc

Now we have to set the allow_other mount option for FUSE. Using the allow_other mount option works fine as root, but in order to have it work as other users, you need uncomment user_allow_other in the fuse configuration file:

perl -p -i -e 's|#user_allow_other|user_allow_other|g;' /etc/fuse.conf

To make sure the s3fs binary is working, run the following:

root@tjstein.com:~# s3fs
s3fs: missing BUCKET argument
Usage: s3fs BUCKET MOUNTPOINT [OPTION]...

So before you can mount the bucket to your local filesystem, create the bucket in the AWS control panel or using a CLI toolset like s3cmd. Then, create the mount directory on your local machine before mounting the bucket:

s3cmd mb s3://bucketname
mkdir -p /mnt/s3
s3fs bucketname -o use_cache=/tmp -o allow_other /mnt/s3

To allow access to the bucket, you must authenticate using your AWS secret access key and access key. You can either add the credentials in the s3fs command using flags or use a password file. Depending on what version of s3fs you are using, the location of the password file may differ -- it will most likely reside in your user's home directory or /etc.

I also suggest using the use_cache option. If enabled, s3fs automatically maintains a local cache of files in the folder specified by use_cache. Whenever s3fs needs to read or write a file on S3, it first downloads the entire file locally to the folder specified by use_cache and operates on it. When FUSE release() is called, s3fs will re-upload the file to s3 if it has been changed, using md5 checksums to minimize transfers from S3.

To confirm the mount, run mount -l and look for /mnt/s3.

Notes

While this method is easy to implement, there are some caveats to be aware of.

Because of the distributed nature of S3, you may experience some propagation delay. So, after the creation of a file, it may not be immediately available for any subsequent file operation. To read more about the "eventual consistency", check out the following post from shlomoswidler.com.
You can't update part of an object on S3. If you want to update 1 byte of a 5GB object, you'll have to re-upload the entire object.
The software documentation for s3fs is lacking, likely due to a commercial version being available now.

Minimum Viable Frameworks

2011-02-18T00:00:00-08:00

During the process of redesigning my personal blog, I evaluated several pieces of software including Jekyll, Flask (a Python micro-framework) and Toto among a few others. I decided to go with Jekyll as it seemed to meet most of my requirements:

No database; posts are stored in plain text, markdown or some other equivalent
No server-side language requirements
Can integrate easily with Git
Can be themed and extended
Has well-written documentation and good community adoption

Since the redesign, I've really embraced this type of site-building platform and I think many others are starting to see it's appeal. The popularity of these micro-frameworks has grown tremendously over the last year or two. I've read numerous blog posts from users leaving larger CMS-based blogging platforms like WordPress in favor of more minimalist, git-powered projects. Their reasons for switching vary, but one common theme is apparent: It's simple.

"I really wanted something simpler than WordPress. I didn’t need a CMS. I barely need a blogging engine. I update so infrequently. I want something that creates well formed html (hah), static content and is easy to use."

Source: Harper Reed

A few web service providers are also catching on, offering extremely affordable (and sometimes free) hosting. Amazon recently announced the availability of website hosting on S3, their storage service. Heroku has offered a free, albeit somewhat diminutive, tier for rapid-prototyping, staging, and testing purposes, as well as actually running lightweight apps. Google App Engine also offers cloud-based solutions, which can be pushed further with tools like DryDrop. DryDrop enables you to host your static site on Google App Engine and update it by pushing to GitHub. That's pretty rad.

If you're looking into micro-frameworks, check out the following articles for more information:

Helplessness Blues

2011-01-31T00:00:00-08:00

"What's my name, what's my station. Oh just tell me what I should do. I don't need to be kind to the armies of night. That would do such injustice to you."

Source: Fleet Foxes | Helplessness Blues

Using WebFont Loader

2011-01-22T00:00:00-08:00

In the process of designing this site, I decided to use some custom fonts using the @font-face CSS attribute. In comparison to some of the other web font options (Cufon, sIFR, FLIR), the @font-face CSS method is simple, easy to implement and well supported in most modern browsers. Although the Google Font API is probably the most well-known web font library, I decided to roll my own kit from Font Squirrel and self-host the fonts. The really nice part about Font Squirrel is that they provide all of the different font formats (TTF, EOT, OTF, and SVG), compatible with every browser on the market.

For all of my H1-H6 headings, I use the following markup:

@font-face {
    font-family: 'YanoneKaffeesatzRegular';
    src: url('YanoneKaffeesatz-Regular-webfont.eot');
    src: local('☺'), url('/fonts/YanoneKaffeesatz-Regular-webfont.woff') format('woff'), url('/fonts/YanoneKaffeesatz-Regular-webfont.ttf') format('truetype'), url('/fonts/YanoneKaffeesatz-Regular-webfont.svg#webfont1BSMunJa') format('svg');
    font-weight: normal;
    font-style: normal;
}

I then load the font in like I would any other family:

h1, h2, h3, h4, h5, h6 {
    font-family: YanoneKaffeesatzRegular, Arial, Helvetica, sans-serif;
    font-weight: normal;
    color: #111111;
}

Everything looked great besides one small, yet extremely annoying, caveat.

I noticed in Firefox and Opera that for a split second, just before the page finished rendering, unstyled text would be displayed. This drove me crazy. Some subsequent Googling let me know that this was common, often referred to as FOUT (Flash of Unstyled Text). The fix was pretty trivial. What you can do is use the WebFont Loader from Google & Typekit.

The WebFont Loader is a JavaScript library that gives you more control over font loading than the Google Font API provides. The key is using the events system to hide the font until it's ready to be shown. There are quite a few implementation options so I'd suggest checking out some of the following articles for help:

Note: If you see a horizontal scroll-bar after implementing the WebFont Loader on body text, try adding overflow attributes:

body {
    font-size: 75%;
    font-family: DroidSansRegular;
    overflow: -moz-scrollbars-vertical;
    overflow-x: hidden;
    overflow-y: scroll;
}

WordCamp LA

2011-01-18T00:00:00-08:00

In September of last year, I was invited to speak at WordCamp LA on developing fast and scalable servers for WordPress. During the 30 or so minute presentation, I covered quite a few topics. I strayed away from the application layer side of things and really focused on server improvements and software considerations that offered the most performance. I got some really nice feedback and met some great people involved with WordPress in the LA area. The two other WordCamps I attended before (New York and their flagship conference, San Francisco) were extremely well organized and WordCamp LA was no different.

Although I wasn't able to post my slides, I've posted the video here. I look forward to speaking at more events like this and I highly recommend attending a WordCamp event if one comes into your town.

A Response To Quality

2011-01-15T00:00:00-08:00

"Doing things the right way is hard, which is why most businesses take the lazy path and settle for mediocrity. People respond to quality, though: they reward it, in no small part because quality is such a rarity in today’s marketplace. Do right by your customers and they’ll do right by you."

Source: Shortchanging Your Business with User-Hostile Platforms

PHP-FPM Plugin Suite For Munin

2010-09-25T00:00:00-07:00

Since switching from a spawn-fcgi implementation several months ago, I’ve been really pleased with PHP-FPM. Given some of the new statistical features included in newer versions (5.3.2+), I put together a plugin suite for Munin. I am not proficient in Perl so I encourage feedback and suggestions to make these better. This suite contains plugins to measure average process size, total memory usage, connection count, process count and most importantly, pool status.

Before installing, make sure you have the most recent version of PHP-FPM:

[www-data@lenny:/var/www]# php-fpm -v
PHP 5.3.2-1ubuntu4.5ppa5~lucid1 (fpm-fcgi) (built: Sep 22 2010 08:04:01)
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies
    with Suhosin v0.9.29, Copyright (c) 2007, by SektionEins GmbH

Now we can get into our plugin directory, clone the plugins from my Git repo and make them executable:

cd /usr/share/munin/plugins
git clone git://github.com/tjstein/php5-fpm-munin-plugins.git
chmod +x PHP5-FPM-Munin-Plugins/phpfpm_*

To enable the plugins, you’ll need to create these symlinks:

ln -s /usr/share/munin/plugins/PHP5-FPM-Munin-Plugins/phpfpm_average /etc/munin/plugins/phpfpm_average
ln -s /usr/share/munin/plugins/PHP5-FPM-Munin-Plugins/phpfpm_connections /etc/munin/plugins/phpfpm_connections
ln -s /usr/share/munin/plugins/PHP5-FPM-Munin-Plugins/phpfpm_memory /etc/munin/plugins/phpfpm_memory
ln -s /usr/share/munin/plugins/PHP5-FPM-Munin-Plugins/phpfpm_status /etc/munin/plugins/phpfpm_status
ln -s /usr/share/munin/plugins/PHP5-FPM-Munin-Plugins/phpfpm_processes /etc/munin/plugins/phpfpm_processes

Note: For the phpfpm_status and phpfpm_connections plugins, you’ll need to enable the status feature included in newer versions (5.3.2+) of PHP-FPM. Open up the php5-fpm.conf, in /etc/php5/fpm, and uncomment line 186 for the pm.status_path directive:

pm.status_path = /status

Jérôme Loyet from the Nginx forums provided some useful insight on how to get this working with Nginx. You’ll essentially set up the status location directive like this:

location ~ ^/(status|ping)$ {
    include fastcgi_params;
    fastcgi_pass backend;
    fastcgi_param SCRIPT_FILENAME $fastcgi_script_name;
    allow 127.0.0.1;
    allow stats_collector.localdomain;
    allow watchdog.localdomain;
    deny all;
}

You’ll need to make sure that from within your box, you can curl /status with # curl http://localhost/status. You should get a response similar to this:

accepted conn: 40163
pool: www
process manager: dynamic
idle processes: 6
active processes: 0
total processes: 6

Once the plugins have been set up, you run any of the plugins manually using the munin-run command line tool:

cd /etc/munin/plugins && munin-run phpfpm_status

The output should look like this:

idle.value 6
active.value 0
total.value 6

Note: The phpfpm_status plugin is particularly useful if you’re using dynamic process management. You can choose static or dynamic in the php5-fpm.conf.

Requirements: libwww-perl

Rails 3.0 with RVM, Nginx and Phusion Passenger

2010-09-14T00:00:00-07:00

Rails 3.0 is ponies and rainbows! Well, kinda. I've never been into Rails much but 3.0 has brought on some much needed features over previous versions, some of which deserve bullet points:

Brand new router with an emphasis on RESTful declarations
Unobtrusive JavaScript helpers with drivers for Prototype, jQuery, and more coming (end of inline JS)
Explicit dependency management with Bundler

The newest version of Rails came out on August 29 so I decided to jump right in. Since Ruby 1.8.6 and earlier are no longer supported, we'll be using RVM in this article. RVM is a command line tool which allows us to easily install, manage and work with multiple ruby environments from interpreters to sets of gems.

Before we get started, we'll be using a fresh (ve) Server from (mt) Media Temple running Ubuntu 10.04 Lucid Lynx for the install. I should also note that we're not installing a full-featured application -- this is simply a tutorial on how to get the Rails 3.0 test application up and running with Nginx + Passenger.

Prerequisites: You'll need these to get through the entire install process for RVM, Nginx and Phusion Passenger (mod_rails).

aptitude -y install curl git-core build-essential zlib1g-dev libssl-dev

aptitude -y install libreadline5-dev libc6 libpcre3 libssl0.9.8 zlib1g

Now we'll need to do a system-wide install of RVM. For your convenience, the dude that wrote RVM provided a script you can run on most that does all the heavy lifting:

bash < <( curl -L http://bit.ly/rvm-install-system-wide )

Every piece of documentation and website I read on this had this particular part wrong. I've updated the correct path from the documentation for the purposes of this article:

echo "[[ -s '/usr/local/lib/rvm' ]] && source '/usr/local/lib/rvm'" >> ~/.bashrc
source ~/.bashrc

Doing so ensures RVM is loaded as a function (versus as a binary), ensuring commands such as rvm use work as expected. Please note that you can confirm this worked correctly by opening a new shell and running:

type rvm | head -n1

If this was performed correctly, you should see:

rvm is a function

So if that looks good, we can continue installing Ruby:

rvm install 1.9.2

Now sit back and relax. This took about 8 minutes on my system. To make sure that the system uses this version, we can use RVM for this and install the rails and passenger gems:

rvm --default ruby-1.9.2
gem install rails passenger

Now comes the fun part of installing Nginx. This will grab the stable binary, bake in Phusion Passenger and compile it:

rvmsudo passenger-install-nginx-module

You will be presented with two options right from the start. Just choose Option 1 and let passenger download, compile and install Nginx for you. You'll also be prompted to choose a prefix directory. You can leave it as the default, /opt/nginx. When it completes, it should give you some Nginx configuration snippets for your rails apps. Save those as you'll need them later.

Now, you'll want a init script so you can stop/start/restart Nginx. I've provided mine here:

cd /etc/init.d
wget -O nginx http://bit.ly/8XU8Vl
chmod +x nginx
/usr/sbin/update-rc.d -f nginx defaults

We're almost done. Now we can create the test application:

cd /opt/nginx/html
rails new testapp

You'll need to update the Nginx configuration to make sure it's using the right document root now. The file will be located in /opt/nginx/conf/nginx.conf:

sed -i".bak" '47d' /opt/nginx/conf/nginx.conf
sed -i '47 a\
            root   /opt/nginx/html/testapp/public;' /opt/nginx/conf/nginx.conf
/etc/init.d/nginx start

The spacing looks a little weird in that middle sed command for a purpose -- to keep the syntax of the nginx.conf file consistent. Now that we've restarted Nginx, we should see the 'Welcome aboard: You're riding Ruby on Rails!' image:

If you're interested in an all-in-one installation, I've put together the steps above into a simple bash script. Enjoy!

Wilco and Intelligentsia Mug

2010-08-22T00:00:00-07:00

I love Wilco. I also love Coffee. So when the coffee-connoisseurs Intelligentsia teamed up with Chicago-based band, Wilco, to offer an exclusive Wilco/Intelligentsia co-branded mug, an impulse purchase was made.

If you're interested in purchasing one, head over to the Intelligentsia store.

APC Plugin For Munin

2010-05-25T00:00:00-07:00

If you're like me, you want to monitor and graph every little piece of your server. Until recently, I've been relying on a simple PHP script to display APC statistics. Although the tool worked well, I wanted to integrate the statistics into Munin, as most all other server properties and system services are monitoring with it. The problem was that finding a suitable APC plugin for Munin proved to be very difficult. Fortunately I stumbled across this German gem (Vielen Dank, dass Sie meine deutsche Freundin!) that met my needs. To set this up on your system, just follow the directions below:

First, let's set up the apcinfo.php file in the document root of your web server and retrieve the file from a private gist:

cd /var/www/html
wget -O apcinfo.php http://goo.gl/Lgfmz

Now, make the file executable:

chmod +x apcinfo.php

The script itself is very simple. It simply ouputs the total cache memory, memory available and memory used:

<?php
$mem = apc_sma_info();
$mem_size = $mem['num_seg']*$mem['seg_size'];
$mem_avail= $mem['avail_mem'];
$mem_used = $mem_size-$mem_avail;
$out = array(
    'size: ' . sprintf("%.2f", $mem_size),
    'used: ' . sprintf("%.2f", $mem_used),
    'free: ' . sprintf("%.2f", $mem_avail)
    );

echo implode(' ', $out);

To confirm that the script is working, check out /apcinfo.php on your site. You should see something like the following:

size: 31457200.00 used: 19478104.00 free: 11979096.00

Now, let's set up the Munin plugin that will use that information:

cd /usr/share/munin/plugins
wget -O apc http://goo.gl/gUgkj 
chmod +x apc
ln -s /usr/share/munin/plugins/apc /etc/munin/plugins/apc

What that does is create the apc plugin, makes it executable and then symlinks it to the proper directory. Now, we'll need to make sure Munin loads the new plugin. To do this, we'll edit the /etc/munin/plugin-conf.d/munin-node to add the following:

[apc*]
user root
env.url http://example.com/apcinfo.php

Obviously, you'll want to replace example.com with the host you intend to use. Once the file has been updated and saved, restart Munin:

/etc/init.d/munin-node restart

Once the node is running, it'll take a few minutes for Munin to update so don't worry if you don't see any new graphs right away. One thing to keep in mind is that you will need the LWP::UserAgent Perl module. If you are on a Debian/Ubuntu OS, just run the following to install it:

aptitude install libwww-perl

Now, go check your pretty graphs.

Simple Load Balancing Using Nginx

2010-04-23T00:00:00-07:00

While the concept of load balancing has been around for a while, using Nginx to do this is fairly new to me. Other common load balancers in use today are LVS, HAProxy, Perlbal and Pound. In this example, I am using 3 (ve) servers from Media Temple, each running on Ubuntu 9.10. To get started, log into the server you want to set up as the load balancer and install Nginx:

aptitude install nginx

Then we'll need to make a new Nginx default virtual host, /etc/nginx/sites-available/default. In the two server directives under the upstream backend section, be sure to put in the IP addresses or hosts you are balancing:

upstream backend  {
  server 123.123.123.123;
  server 123.123.123.124;
}

server {
  location / {
    proxy_pass  http://backend;
  }
}

This is the simplest configuration possible. After you've completed the proxy configuration, test and restart Nginx:

nginx -t
/etc/init.d/nginx restart

At this point, requests handled by the load balancer will serve equal requests from each upstream server. The really nice thing is that if one of the upstream servers is not responding, the load balancer will automagically stop routing requests to it. So although the configuration has the unavailable server loaded, Nginx sees that it is down and then routes to all other available upstream servers. If all upstreams are down, Nginx will halt the proxy, simply showing a 502 http error. This also makes it a very useful tool for balancing mongrels for those running rails.

It's also worth noting that you can add as many backend nodes as you want; I'm just using two as an example. This makes using a (ve) server an ideal choice; spin up a new (ve) and simply add it to the upstream pool. There are a few other configuration options that provide the ability to add weight to backends to force an uneven load distribution. You can also use proxy_set_header to make sure the user's IP address is logged instead of localhost:

upstream backend  {
  server john.tjstein.com;
  server paul.tjstein.com;
  server ringo.tjstein.com;
  server george.tjstein.com weight=30; # George was always my favorite
}

server {
  server_name www.tjstein.com tjstein.com;
  location / {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass  http://backend;
  }
}

Nginx provides many other configuration directives so I would recommend checking the official Nginx documentation on the NginxHttpUpstreamModule for more information. In the next post, we'll look at using a few file synchronization tools like Unison and rsync for data replication between backends.

Fixing Unsupported Protocol Error with PECL

2010-01-26T00:00:00-08:00

After setting up PHP 5.2.10, I wanted to install a few additional modules using PECL. I found soon after that I was unable to do so, receiving the following error:

pear.php.net is using a unsupported protocol - This should never happen. install failed

The problem seems to exhibit itself with corrupted PEAR installations in PHP 5.2.9 and 5.2.10. To fix this, just backup and update the channels:

mv /usr/local/lib/php/.channels /usr/local/lib/php/.channels.bak
pear update-channels

After that, you should be good to go.

Media Temple (gs) Grid Service Domain & MySQL Backups

2009-10-25T00:00:00-07:00

Having been on many shared hosting platforms before, I am fortunate enough to have a few (gs) Grid Services with (mt) Media Temple. Although no native backup tool currently exists for the (gs), I've created some simple backup scripts to backup the domain directories and MySQL databases. Both are very simple bash scripts that can be set up as cron jobs from with the AccountCenter. The first backup we'll create is for domain backups:

If you haven't done so already, enable SSH from the (mt) Media Temple AccountCenter.

Once you've logged in via SSH, navigate to the /data directory. Keep in mind you'll need to add your site number in the 00000 for this to work. You can find your site number in the Server Guide > Access Domain section of the AccountCenter.

cd /home/00000/data

Run the following commands to create and set permissions of the backup directory:

mkdir backup
chmod 777 backup

Create a file called backup.sh and make it executable:

touch backup.sh
chmod +x backup.sh

Open the file in a text editor or vim and paste in the following:

#!/bin/bash
today=$(date '+%d_%m_%y')
echo "* Performing domain backup..."
tar czf /home/00000/data/backup/example.com_"$today".tar.gz -C / home/00000/domains/example.com
# Remove backups older than 7 days:
MaxFileAge=7
find /home/00000/data/backup/ -name '*.gz' -type f -mtime +$MaxFileAge -exec rm -f {} \;
echo "* Backed up..."

This really simple script creates a tar.gz backup of the example.com directory, stamps it with the current date and places it in the the backup folder we just created. It will also prune any backup older than 7 days old. This number can be adjusted by tweaking the MaxFileAge value. Before saving the file, be sure to make the necessary adjustments for your site number after the /home directory. You'll also need to modify example.com for the actual domain you'll be backing up. Before adding it as a cron, run the script via SSH -- the output should look like this:

example.com@n10:/home/00000/data$ ./backup.sh
* Performing domain backup...
* Backed up...

Depending on the size, the backup may hesitate after the initialization while the domain directory is being compressed. You can now verify that the new backup is in the /home/#####/data/backup/ directory with the today's date: example.com_25_10_09.tar.gz. Once you've confirmed the backup works, add the backup.sh file as a cron within the AccountCenter following this KnowledgeBase article. You can schedule the cron as frequent as you would like.

Now, you've probably got at least 1 MySQL database that should be backed up as well. This can be accomplished just as easily.

SSH back into the /data directory and create a new bash script file:

touch db-backup.sh
chmod +x db-backup.sh

Now, open the file in a text editor or vim and paste in the following:

#!/bin/sh
#############################
SQLHOST="internal-db.s00000.gridserver.com"
SQLDB="db00000_dbname"
SQLUSER="db00000"
SQLPASS="db_user_password"
SQLFILE="db00000_example.com_$(date '+%d_%m_%y').sql"
LOCALBACKUPDIR="/home/00000/data/backup"
#############################
echo "* Performing SQL dump..."
cd $LOCALBACKUPDIR
mysqldump -h $SQLHOST --add-drop-table --user="$SQLUSER" --password="$SQLPASS" $SQLDB > $SQLFILE
# Remove backups older than 7 days:
MaxFileAge=7
find /home/00000/data/backup/ -name '*.sql' -type f -mtime +$MaxFileAge -exec rm -f {} \;
echo "* Backed up..."
exit 0

This bash script uses mysqldump to dump a MySQL database which stamps it with the current date and places it in the the backup folder we created before. This will also prune any database backups that are older than 7 days. Before saving the file, be sure to make the necessary adjustments for your site number and database credentials -- all of this can be found under the Manage Databases section of the AccountCenter. Before adding it as a cron, run the script via SSH -- the output should look like this:

example.com@n10:/home/00000/data$ ./db-backup.sh
* Performing SQL dump...
* Backed up...

Once you've confirmed the script completes without errors, add it as a cron job so you'll never have to worry about losing your MySQL data again!

Ditching Apple Mail for Postbox

2009-08-26T00:00:00-07:00

Until recently, I've been using Apple Mail for all of my email needs. Although we typically loathe Apple Mail for it's overall bugginess in my workplace, I've been apathetic and used it for nearly 2 years. In doing so has warranted some abuse from my coworkers, all of whom swear by Thunderbird. My hesitance to switch was purely based on my willful ignorance toward other applications.

Meet Postbox.

Derived from Thunderbird and Mozilla labs, Postbox is a cross platform e-mail client that brings some of the great features from Gmail to the desktop. Features like tagging, topics, conversation view, tabs, and smart search made this an ideal replacement for Apple Mail. Setting up new accounts was also easy; if you're familiar with Thunderbird, you should feel right at home. Within an hour of installation, I had set up 4 IMAP accounts (2 Work, 2 Personal) and set up mail filters and synchronized additional IMAP folders.

I'm still running beta 10 however the newest version is said to have significantly decreased CPU and memory usage. AOL Mail account support (ugh) has been added as well. The full list of bug fixes and improvements can be found here.