shapeshed

Compiling Node.js from source on Ubuntu 10.04

2012-01-04T00:00:00+00:00

Compiling Node.js from source is easy enough so here’s how.

Installing from a tarball

Node.js needs a few things to compile so make sure they are installed.

Installing Node.js dependencies

sudo apt-get update
sudo apt-get install build-essential openssl libssl-dev pkg-config

Get the link for the latest tarball from the Source Code link on the Node.js homepage. Then download it and extract it.

Downloading and extracting the tarball

cd /usr/local/src
sudo mkdir node
cd node
sudo wget http://nodejs.org/dist/v0.6.6/node-v0.6.6.tar.gz
sudo tar -xzf node-v0.6.6.tar.gz

Now you can compile and install the binary. By default Node.js will be installed to /usr/local/bin/node and npm will be installed to /usr/local/bin/npm.

Compiling Node.js

cd node-v0.6.6/
sudo ./configure
sudo make
sudo make install

Upgrading via a tarball

To upgrade Node.js simply download the new tarball, extract and repeat the installation process.

Installing from Git

If you plan to upgrade each time a new release comes out you may find cloning the Git repository more convenient than downloading and extracting a tarball each time.

You will need Git and the dependencies to be installed first.

Installing Node.js dependencies

sudo apt-get update
sudo apt-get install git-core build-essential openssl libssl-dev pkg-config

Using Git means that instead of downloading extracting a tarball you can just clone the repository and checkout the latest version. If you need to check the versions that are available run git tag.

Cloning the Node.js Git repository

cd /usr/local/src
sudo git clone git://github.com/joyent/node.git
cd node
sudo git checkout v0.6.6

Now you can compile the binary as before

Compiling Node.js

sudo ./configure
sudo make
sudo make install

Upgrading via Git

To upgrade to a new release of Node.js first pull the latest source.

Pulling the latest Node.js source

cd /usr/local/src/node
sudo git checkout master
sudo git pull origin master

Then checkout the latest version where v.x.x.x is the version you want.

Checking out the latest version

sudo git checkout vx.x.x

Then follow the installation process as before

Compiling Node.js

sudo ./configure
sudo make
sudo make install

This will upgrade Node.js and overwrite the previous version.

Using rbenv to manage rubies

2011-10-12T00:00:00+01:00

rbenv is great

If you haven’t seen rbenv yet I highly recommend it. It is a small collection of shell scripts that lets you manage rubies on UNIX type machines. To date I’ve used rvm to install versions of ruby and manage gemsets. Don’t get me wrong - I’m hugely greatful to the work that has gone into the rvm project and it is still a great tool. But I prefer the UNIX philosophy of doing one thing well and rbenv embraces that. I also really like that with rbenv you just need to amend your PATH and you are done. It follows UNIX conventions making integration with other UNIX tools easier - for me at least.

rbenv basics

We have a dev machine at work which runs a variety of Sinatra and Rails projects (including a Rails 1 project). As such these projects need lots of different versions of Ruby. To date I’ve used rvm to manage this and it has worked well but I always came up against issues when trying to integrate tools like Monit, puppet or init.d scripts. So I bit the bullet and switched to rbenv.

Each app has a UNIX user so rbenv is installed locally for each user when logged in as the user account

Cloning rbenv

cd
git clone git://github.com/sstephenson/rbenv.git .rbenv

You need to load rbenv into the shell and you are good to go

Adding rbenv to your PATH

echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(rbenv init -)"' >> ~/.bashrc
source ~/.bashrc

If you have installed ruby-build you can then install rubies with rbenv install.

Installing a ruby

rbenv install 1.9.3-rc1

You can set a global ruby for the user account with

Setting a global ruby

rbenv global 1.9.3-rc1

And you can also use an .rbenv-version file to set the version. I choose to use this in projects and check it into git.

Deployment with capistrano

I like that rbenv doesn’t try and manage rubies for you - we have bundler for that. I deploy as the same user as the app accounts so I just need to add the following to my cap recipe to use rbenv since I’m already including require 'bundler/capistrano'.

Adding the PATH to Capistrano’s Shell

set :default_environment, {
  'PATH' => "/home/youruser/.rbenv/shims:/home/youruser/.rbenv/bin:$PATH"
}

This loads rbenv into the shell that capistrano uses. That’s pretty much all you need to use your rbenv installed ruby.

Following a thread on github you can also apply a clever technique to allow you switch versions of ruby by pushing a new .rbenv-version file with capistrano. From version 1.1rc bundler allows you to specify a shebang for binstubs. To use this add the following to your capistrano recipe.

Adding binstubs for easy ruby switching

set :bundle_flags, "--deployment --quiet --binstubs --shebang ruby-local-exec"

This generates executables in the bin folder of your Rails app which reference ruby-local-exec as the shebang. This command will execute whichever ruby is specified in the .rbenv-version file. In my case I have an init.d script to manage unicorn that references bin/unicorn. If I wanted to upgrade the ruby used by the app I would install it on the remote machine and then just update my .rbenv-version file commit, cap deploy and I’m done.

Integrating with other tools

I like to use monit to monitor processes. Monit exexutes scripts with a very limited shell but since rbenv follows UNIX conventions we can easily create a small bash script to load rbenv into the path and then do what we want.

Here’s the script I’m using for starting a resque_worker

resque_worker bash script

#!/usr/bin/env bash
USER=youruser
APP_PATH=/srv/yourapp.com/current
PATH=/home/$USER/.rbenv/bin:/home/$USER/.rbenv/shims:$PATH
RAILS_ENV=production

cd $APP_PATH
RAILS_ENV=$RAILS_ENV bin/rake environment resque:work& > $APP_PATH/log/resque_worker.log
echo $! > $APP_PATH/tmp/pids/resque_worker.pid

Then my monit task looks like this

resque_worker monit script

check process yourapp_resque_worker
    with pidfile /srv/yourapp.com/current/tmp/pids/resque_worker.pid
    start program = "/bin/sh -c '/home/youruser/bin/start_resque_worker.sh'" as uid youruser and gid youruser
    stop program = "/bin/sh -c 'cd /srv/yourapp.com/current && kill -s quit `cat tmp/pids/resque_worker.pid` && rm -f tmp/pids/resque_worker.pid; exit 0;'"

Better separation

By using rbenv, bundler, capistrano and monit together we have great separation between what these tools do

rbenv manages rubies
bundler manages gems
capistrano manages deployments
monit monitors processes

This feels clean and manageable to me.

By using binstubs with the ruby-local-exec shebang we also separate our app from versions of rubies, init.d and monit scripts, making it super easy to upgrade.

Thanks rvm, onwards with rbenv

I’m really grateful to Wayne E Seguin and the work he has put into rvm and I hope the project continues to thrive. rbenv fits the way I work though and I’m really enjoying using it. It makes piecing tools like puppet, monit, init.d scripts, capistrano and bundler much easier, so a big thanks to Sam Stephenson for creating it.

Managing Unicorn Workers with Monit

2011-09-12T00:00:00+01:00

Riding Unicorns

For many reasons I like using Unicorn as a web server for Ruby apps. Ryan Tomayko wrote a great article and the many Unix features that it has. It has the killer feature of supporting hot restarts so you can deploy new releases with zero downtime. This means that in most cases you can deploy as often as you want and end users won’t even notice.

More power == more responsibility

With all of the great stuff that comes with Unicorn there is more configuration to do. This is the point at which the less intrepid opt for Passenger, that to be fair is stil a good option. But if you want a web server that uses great features of Unix and enables hot restarts you should persevere.

For Unicorn you need to set up a reverse proxy with nginx. I’ve written before about how to do that if you need a steer. You’ll also need to use upstart or script in /etc/init.d/ to manage Unicorn. There are a few scripts around on the web. I’ve posted the one that I use up as a gist on Github. You can then use something like this in your capistrano recipes to do a hot restart.

Hot restarts with unicorn

namespace :deploy do
  task :restart do
    sudo "/etc/init.d/unicorn upgrade"
  end
end

This sends a USR2 signal to the Unicorn master process to upgrade itself. It will create a new master process and then switch over when it is ready. See - zero downtime!

Up the workers

The Unicorn master process looks after workers and will reap workers that die from broken apps. In my scenario I was working on a memory constrained box and I wanted to make sure that workers did not consume too much memory. I like to use Monit to monitor processes. You can also use god or bluepill but I’ve found these tools consume much more memory that Monit. Call me old fashioned but I like to use Unix tools if they are available over Ruby.

So I want to monitor the Unicorn workers that are a child processes of the master Unicorn process. Bluepill supports monitoring child processes but it turns out this is easy enough to do with Monit.

Unicorn’s config file has an after_fork method that allows you to write out the pid of a worker. Here’s how

Writing worker pids in Unicorn

after_fork do |server, worker|
  defined?(ActiveRecord::Base) && ActiveRecord::Base.establish_connection
  child_pid = server.config[:pid].sub('.pid', ".#{worker.nr}.pid")
  system("echo #{Process.pid} > #{child_pid}")
end

If you want to see my full unicorn.conf it is here. This will write out the pid of each worker /tmp/pids/unicorn.[pid_id].pid, so now we can use Monit to watch it.

Adding in Monit

Now that we have the pids of workers and an init.d script that can manage workers too (see kill_worker) we can use Monit to keep everything in check. The only downside is that you need to tell Monit about each worker process that you have. If you are spawning and reaping Unicorn workers dynamically with TTIN and TTOU this probably isn’t going to work for you. But if the numbers of workers are fixed you are good.

A monit script to monitor unicorn workers

check process unicorn_worker_0
    with pidfile /path/to/your/app/shared/pids/unicorn.0.pid
    start program = "/bin/cat /dev/null"
    stop program = "/etc/init.d/unicorn kill_worker 0"
    if mem is greater than 175.0 MB for 1 cycles then restart
    if cpu is greater than 22% for 2 cycles then alert
    if cpu is greater than 25% for 1 cycles then restart

Monit will watch the worker for memory growth and kill it off if it consumes too much. The unicorn master will spawn a new worker when it is killed so it is a neat solution to keep things running.

Much of this article was derived from Andrew Grim’s article ‘Where Unicorns go to die: Watching unicorn workers with monit’, so thanks to Andrew for the pointers.

Migrating from Slicehost to Linode

2011-08-29T00:00:00+01:00

Slicehost was a game changer

I’ve been a long standing Slicehost customer, after moving from Media Temple. Slicehost was perfect for my needs - I knew my way around Linux so I just wanted SSH access and the ability to configure it myself. At $20 a month Slicehost was a ground-breaking service. Coupled with a blog that spoke to developers about how to do things developers were doing every day it provided a range of distributions and was a massive hit with the developer community. On October 22nd 2008 Slicehost announced that they had been acquired by Rackspace. This was a just reward for what Slicehost had achieved but immediately there were concerns as to the direction of the service.

From public accounts Rackspace has used Slicehost’s underlying technology to build their Rackspace Cloud. In May 2011 Rackspace contacted Slicehost customers to say they would be migrated to the Rackspace Cloud product.

PR Car Crash

The email to customers raised more questions that it gave answers causing many customers to leave Slicehost there and then. There were so many questions that a further post was added to the forum which in fairness addressed many of the questions that customers had. At that point it became clear that the service Rackspace were offering was very different from Slicehost:

DNS moving to Rackspace’s service. Unknown if it would be free
Lower bandwidth allowance with pricing moving to pay what you use
Some servers moving datacenters and changing IP addresses
Removal of some Slice sizes
Slicehost API likely to close
A requirement to migrate to Rackspace by May 2012

Particularly if you use more than 60GB of bandwidth a month the service Rackspace’s Cloud Service would be more expensive.

Free markets are good

Thankfully the hosting market is very competitive and even Slicehost has competition in the form of Linode, a service launched in 2003. Linode offer a very similar service to Slicehost with the ability to provision servers, get SSH access and build it yourself. In fact Linode’s offering to customers is more competitive than Slicehost with 512MB of RAM offered for the entry price of $20 a month. Linode also offer free DNS servers, a great API and has also been a strong hit with the developer coummunity.

After looking at the direction of Rackspace Cloud and what Linode offer for the same price the clear choice was to migrate to the better product: Linode.

The pain of migrating

I’ve been with Slicehost for a while so with that comes many configurations, DNS zones and a server setup exactly how I like it. Rebuilding servers is tedious and sadly I built mine before Puppet was around. I use Puppet at work and this has shown me there is huge value in automating server management, especially in the context of Cloud providers.

After the decision to migrate I was staring at a terminal with many hours of work ahead of me, until the power of open source came to the rescue.

Migrating DNS

Firstly for migrating DNS records Rob Schultz created a short Ruby script to migrate DNS records from Slicehost to Linode. Once you’ve installed the script and necessary gem dependencies migrating your records is as simple at

./slicedns2linode.rb domain1.com.

Within a few minutes I had migrated all of my DNS records.

Migrating the server

If you are not already using Chef or Puppet there is a project called Blueprint that lets your reverse engineer a server configuration to a bash script, Puppet manifests or Chef cookbook. There is a good tutorial on a basic migration. Once you’ve installed blueprint and the dependencies you can create a shell script that will build another server with

blueprint create -S tutorial

Once you’ve created your scripts, just run them on the target server and it will build the server in the image of the old one.

My experience of Blueprint

The migration via blueprint went very smoothly. I had installed lots of packages adding third party repositories and compiling from source. Other than gitosis packages were mostly copied over and iptables was successfully configured.

Finishing up with scp

Finally I had to copy over website, home folder files and a few config files. The perfect tool for this was scp. First I created a tar.bz2 archive for folders I wanted to copy.

tar -cjf vhosts.tar.bz2 /var/www/vhosts

Then I copied over the archive to my new Linode instance

scp vhosts.tar.bz2 george@123.456.78.90:/var/www

Finally I unarchived it on the Linode instance

tar -xjf vhosts.tar.bz2

The great thing about using this approach is that permissions are preserved.

The last part was to change the name servers for my domains over to Linode.

Thanks Slicehost

Slicehost was a great service and the articles in particular helped me to skill up on UNIX and respond to many different scenarios. I’m sad that Slicehost will no longer be around and in particular that the service offering is so different from Rackspace. Slicehost was cloud hosting before it became trendy and now cloud hosting is something entirely different. I hope Linode can make me as happy - so far so good.

Going realtime with node.js, Express and socket.io

2011-07-18T00:00:00+01:00

Counting is fun!

This is a walkthrough on how to create a simple realtime counter of visitors to a page using node.js, socket.io and express. This example is very simple but could easily be extended to a range of applications.

Just show me the app

http://counter.nodester.com

Pulling the pieces together

Express is a great web framework for creating sites in node.js. I’ve written before about creating a basic site. This time we are going to use a couple more libraries to add realtime communication to the application.

socket.io is an amazing library that takes the pain out of websockets and provides a reliable fallback depending on browser capabilites. Here’s a simple example from the socket.io site.

Setting up socket.io on the server side

var io = require('socket.io').listen(80);

io.sockets.on('connection', function (socket) {
  socket.emit('news', { hello: 'world' });
  socket.on('my other event', function (data) {
    console.log(data);
  });
});

This sets up the server side. Then you just need to add some client-side JavaScript and you are ready to go

Setting up socket.io on the client side

<script src="/socket.io/socket.io.js"></script>
<script>
  var socket = io.connect('http://localhost');
  socket.on('news', function (data) {
    console.log(data);
    socket.emit('my other event', { my: 'data' });
  });
</script>

I’ve used this setup to push the number of connected clients out to the browser and a simple counter to increment and decrement the number of connected clients on the relevant events. I’ve had some issues with different browsers on hosting environments. For now the demo works with Safari and Chrome.

Smoothie Charts is a great library that will draw graphs in JavaScript. It is very customisable, and lightweight and is specifically designed for live, streaming data. It uses canvas to draw the graphs and the tutorial shows just how easy it is to create a realtime charts.

Bringing these libraries together gives us live, realtime graphs. There’s no persistence here but if we wanted to we could add something like Redis into the mix to persist the maximum number of connections.

You can browse the source and see how the libraries are used on Github. Fork it!

Potentials uses

We could use this kind of setup to show realtime stats on server resources of the kind provided by Munin, to create realtime statistics for a page, or to stream live data from a third party API to a page. I’m excited about what people are going to make with these libraries!

Instant Rails dev environments with Tmuxinator and Foreman

2011-07-06T00:00:00+01:00

The problem

To begin working on a Rails app I need to do things like starting a server, starting a console, ensure background processes are running, start a terminal window so I can watch logs, start a text editor etc etc.

Doing all this is tedious. Wouldn’t it be nice if we could just do something like?

start_myapp

Well you can!

Tmux

Tmux is a ‘terminal multiplexer’. What the hell’s that? It basially allows you to create terminal sessions - similar to GNU Screen where you can detach from a session and come back later. If you are not familiar with tmux there are a few good resources available

You can install tmux with homebrew

brew install tmux

It takes a little reading and configuring but is well worth effort if you are a heavy terminal user.

Tmuxinator

Tmuxinator is an excellent gem that lets you define the layouts of your tmux sessions.

You can install tmuxinator from RubyGems

gem install tmuxinator

By defining a yaml file we can layout things exactly how we want them.

tmuxinator project layout file

project_name: myapp
project_root: ~/Sites/myapp
tabs:
  - zsh:
  - vim: vim .
  - foreman: bundle exec foreman start
  - git: git pull
  - console: bundle exec rails console
  - server: bundle exec rails server
  - logs: tail -f log/development.log

Then you can start your session with start_myapp and your layout will be ready for you. Here’s a screenshot from the documentation:

There’s excellent documentation on using tmuxinator in the project README

Foreman

Enter the final piece of the jigsaw - foreman. Foreman is a gem that lets you manage background processes associated with your application through a Procfile. You just add any processes you want to be started into the Procfile and you are done. Here’s an example

Procfile

worker: bundle exec rake resque:work QUEUE=sweep_orders
apn_sender: bundle exec rake apn:sender
resque_scheduler: bundle exec rake resque:scheduler
resque_web: bundle exec resque-web --foreground --server thin --port $PORT --no-launch
sphinx: bundle exec rake ts:run_in_foreground

You can then start these process by running

Procfile

foreman start

The processes will run in the foreground and spit out any logs messages to standard output - perfect for development.

Using foreman you can add a tab to tmuxinator that will start your processes and have a nice log of what they are doing. We have a neat way of starting any background processes in our app within the context of our unified tmux development environment.

So with one command we can set things up exactly the way we want and get straight into developing.

Alternatives

This technique will also work well for screen and regular terminal users with the Screeninator and Terminitor gems so if you are not sold on tmux you can still use it.

Building a Rails server with Puppet

2011-06-22T00:00:00+01:00

The goal

On a regular basis at pebble.code where I work we have a requirement to provision servers for clients, mostly for Rails apps. In the past we have used a combination of Cloud PaaS providers like Heroku or Engine Yard, we have built servers manually (gasp) or used bash scripts.

Puppet is a tool for building and managing servers and looked like a great fit for our requirements.

Learning curve

Puppet has a learning curve but there are lots of excellent resources available on the web. I found Pulling Strings With Puppet to be an excellent resource and there is also some good documentation on the Puppet site. I’ve written before on how to set up Puppet on Ubuntu 10.04 and also how to connect clients to a Puppet master.

Once you learn about resources and modules you are pretty much ready to get going.

Setting up Puppet Dashboard

Puppet comes with a dashboard that can be run on the Puppet Master to give a graphic view into what your nodes are doing, providing reports of updates and letting you know if there are any problems. There’s some good documentation offered by PuppetLabs on this to get you started.

I had some problems getting this going getting the error

can't activate rack (~> 1.0.1), already activated rack-1.2.2

This thread on the puppet-users mailing list gives a fix

cd <path-to-dashboard> 
git clone git://github.com/puppetlabs/puppet-dashboard.git 
rm -r vendor/gems/rack-1.0.1 
sed -i -e 's,~> 1.0.1,~> 1.2.2,' vendor/rails/actionpack/lib/action_controller.rb 

After that I was able to see the nodes and updates.

Rails stack choices

Then came the hard part - creating Puppet modules and using third party ones to install software on the server. I ended up writing my own for some but used existing modules where possible.

The ones I found most useful in building a Rails stack were

One issue is where to put the responsibility for some items in the stack. The web server is a good example. In my case I chose to use Unicorn and manage the installation of this via bundler and the Gemfile of the application. I could have chosen to use Puppet for this with something like Passenger. For automation Passenger is difficult to upgrade as it needs to be recompiled against Apache or Nginx. By chosing Unicorn I can upgrade it easily using bundler. To manage Unicorn I use an init script and hook this up to Monit and capistrano recipes. I’m still not sure about this approach but it works at least.

Switching the Puppet Master to Unicorn

As the number of nodes grew, and particularly where a new node was added I found that the default WEBrick server that ships with Puppet was not good enough and I started to hit some errors. Thankfully there is an excellent wiki article on the Puppet Labs site walking through how to switch Puppet to using Unicorn. Unicorn is a great web server and handles load balancing itself - perfect for my requirements.

Workflow

I chose to deploy using Capistrano and a git based workflow. I have an init script for the Unicorn Master powering Puppet so I can take advantage of the rolling restarts offered by Unicorn.

Still to do…

I still need to create a good framework for testing. Currently I run things against a blank VM. There are a few patterns for testing which I would like to become a bit more familiar with and use. I’d like to extend a few existing modules and write a few new ones, particularly for user managemet.

The good bit

The good bit is that we can now create a secure, bootstrapped server that can host any rack application with any cloud provider within about 20 minutes. Moreover we can manage SSH keys with Puppet so it is easy to grant and revoke developer access.

There’s more to learn but in my opinion the investment of time was well worth it.

After reading this article Matt Tanase contacted me about a great project called Blueprint. If you are porting existing infrastructure to Puppet have a look.

Building a Rails development server

2011-06-06T00:00:00+01:00

At pebble.coe where I work we have a Ubuntu 10.04 LTS server for our Rails and Ruby projects. We use this mostly internally but sometimes expose sites outside of our network to allow clients to quickly review work. When we started the business we built the Rails stack using the standard tools - Apache and Passenger. As our client list grew and we took on a couple of legacy projects we found that this stack no longer matched the environments that we were deploying too and didn’t give us the flexibility we wanted. So here’s how we changed it.

Promoting nginx to port 80

Nginx is quite simply an awesome web server. It is simple to configure, the memory footprint is small and it can be used to proxy traffic through the web server of our choice and can serve static assets like html, images and css directly before it hits the Rails app.

From having the ability to serve Rails sites only from Apache and Passenger we have moved to this.

We’ve moved Apache to port 81 so if a site needs to be tested on that setup we can still do it. Otherwise we can proxy from nginx directly to Apache. The real flexibility comes from the server we use to serve the Rails app. We deploy by choice on Thin or Unicorn so we can replicate this stack easily. We are looking at Goliath at the moment so we could easily add this in beneath Nginx. We are playing with some other technologies like node.js too. With Nginx we can support this too.

Getting nginx

If you want to make the switch on Ubuntu there is a PPA repo you can add

sudo -s
nginx=stable # use nginx=development for latest development version
add-apt-repository ppa:nginx/$nginx
apt-get update
apt-get install nginx

If you are currently using Apache you’ll need to decide whether or not to keep it on port 80. We switched Apache to port 81 and promoted Nginx to port 80.

Example configs

The Ubuntu install uses sites-available and sites-enabled folders to allow you to create site configs. Here are some examples that you can use to get you started.

Logs are disabled in these examples. If you want logging just add a path to the log file.

Default proxy settings

To keep your Nginx configs nice and DRY you can add a file to hold common proxy settings under /etc/nginx/conf.d called in my case proxy.conf

proxy.conf

proxy_redirect     off;

proxy_set_header   Host             $host;
proxy_set_header   X-Real-IP        $remote_addr;
proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
proxy_max_temp_file_size 0;

client_max_body_size       10m;
client_body_buffer_size    128k;

proxy_connect_timeout      90;
proxy_send_timeout         90;
proxy_read_timeout         90;

proxy_buffer_size          4k;
proxy_buffers              4 32k;
proxy_busy_buffers_size    64k;
proxy_temp_file_write_size 64k;

Proxy to Apache

This expects that you have setup an Apache virtual host on port 81.

Nginx proxy to Apache

server {
    listen       80;
    server_name  site1.com;

    access_log  off;
    error_log off;

    location / {
        proxy_pass         http://127.0.0.1:81/;
    }
}

Proxy to Thin

Nginx proxy to Thin

upstream thin {
    server unix:/var/www/vhosts/site2.com/httpdocs/current/tmp/sockets/thin.0.sock;
}
server {
    listen       80;
    server_name  site2.com;

    access_log  off;
    error_log off;

    location / {
        proxy_pass         http://thin;
    }
}

Proxy to Unicorn

Nginx proxy to Thin

upstream unicorn {
    server unix:/var/www/vhosts/site3.com/httpdocs/current/tmp/sockets/unicorn.sock fail_timeout=0;
}
server {
    listen       80;
    server_name  site3.com;

    access_log  off;
    error_log off;

    location / {
        proxy_pass         http://unicorn;
    }

So which one is best?

Having played a fair bit recently with servers to serve up Rails apps there isn’t a clear answer I’m afraid. I may get round to publishing my rudimentary benchmarks but I’ve found that for low memory deployments Nginx and Thin is a great choice. The response times are excellent and memory usage is much lower than Passenger or Unicorn. Thin doesn’t come with the same in-built load balancing that you get with Unicorn so HAProxy comes in to play here. HAProxy is another piece in the stack though which may put some off.

If memory isn’t an issue then Unicorn behind Nginx is a great choice. The rolling deployments are a great feature, but you also get free load balancing with Unicorn thanks to the power of UNIX.

If you are not interested in memory or tweaking you are probably best with Passenger behind Nginx.

If you are not interested in doing any sys admin work use Heroku, EngineYard or Brightbox.

Conclusion

If you are rolling your own servers and using Apache switch to Nginx!

Creating a basic site with node.js and Express

2011-04-18T00:00:00+01:00

What we are going to do

This walkthrough will go over setting up a basic site using node.js and Express. The walkthrough is aimed at beginners exploring node.js as I’ve had many questions from friends and colleagues about creating and deploying node apps. If you are not a beginner the article probably won’t be of much use to you. We are going to use express, an excellent web framework for node created by TJ Holowaychuk who seems to be pumping out node.js libraries like he was ten men.

Here is the site we are going to create. You might also want to grab the source code.

Setup

First we need to setup our development environment. If you are on OSX I’ve covered how to setup node.js and npm on OSX in a previous article. If you haven’t got everything installed follow that article.

If you are on Linux there are plenty of articles on Google.

For Windows users there are also resources on Google but it is a bit more tricky.

Prerequisites

If everything has installed ok you should now have node.js and npm running on your machine. At the terminal type node -v and npm -v and you should see something like:

node -v
v0.4.5
npm -v
1.0.1rc7

You’ll see I’m using the RC of version 1 of npm. I encourage you to install this as this will soon become the default.

Create an Express site

Still with me? We’ve covered a lot already! Now let’s create an Express site.

First let’s create a folder for our project and install Express.

mkdir express_example
cd express_example
npm install express

This installs Express into the directory we created. Now we can create the skeleton site. We are going to use jade and stylus for templating and css. More on those later.

./node_modules/express/bin/express -t jade -c stylus
destination is not empty, continue? y
   create : .
   create : ./app.js
   create : ./public/stylesheets
   create : ./public/stylesheets/style.styl
   create : ./public/images
   create : ./public/javascripts
   create : ./logs
   create : ./pids
   create : ./test
   create : ./test/app.test.js
   create : ./views
   create : ./views/layout.jade
   create : ./views/index.jade
   - make sure you have installed stylus: $ npm install stylus
   - make sure you have installed jade: $ npm install jade

You’ll see that we need to install a couple of dependencies for stylus and jade support so let’s do that.

npm install stylus jade

Boot the app

That’s all the setup you need. Phew. Now you can boot the app:

node app.js

You should see Express server listening on port 3000 and if you open http://0.0.0.0:3000 you’ll see the default Express page.

Using Git

Git is a version control system that is used heavily in the node.js ecosystem, particulary with Github. If you aren’t familiar with Git Scott Chacon is your go-to man. He’s written extensively and eloquently on Git for beginners and experts. Checkout Gitcasts for if you are a beginner and ProGit for more advanced stuff. We are going to use git to version our site and publish it so let’s set up our repo now. If your Express server is still running hit CTRL + C to stop it.

git init
git add .
git commit -m 'initial commit'

Developing node.js sites

Normally when you develop a node.js site you’ll need ot restart your application each time you make a change. Thankfully our home-grown British JavaScript genius Remy Sharp has solved this problem with nodemon. Nodemon will reload your application each time it changes so you don’t need to restart it. If you have used Shotgun for Ruby with Sinatra it is similar to that. To install run

npm install -g nodemon

Then you can start your app with

nodemon app.js

Using nodemon means you don’t have to restart your app each time you make a change. For more infomation on nodemon see the README

HTML in Express

Express is agnostic as to which templating language you use. Templating languages can be a hot topic of debate but for this article I’m going to use jade. If you’ve used haml it is similar to that. In the example we use jade to setup a layout template.

!!! 5
html
  head
    title= title
    link(rel='stylesheet', href='/stylesheets/style.css')
    link(rel='stylesheet', href='/stylesheets/chunkfive-fontface.css')
  body
    header
      nav
        ul
          li
            a(href="/") Home
          li
            a(href="/about") About
          li
            a(href="/contact") Contact
    section#wrapper!= body
        footer
          section.css-table
            section.four-column
              section.cell
                p Mauris porttitor <br />felis eu leo aliquet<br /> ac rutrum odio aliquet
              section.cell
                p Mauris porttitor <br />felis eu leo aliquet<br /> ac rutrum odio aliquet
              section.cell
                p Mauris porttitor <br />felis eu leo aliquet<br /> ac rutrum odio aliquet
              section.cell
                p Mauris porttitor <br />felis eu leo aliquet<br /> ac rutrum odio aliquet

This is a common template we can reuse. The line section#wrapper!= body pulls in content from the page it is used on. Express also supports variables that you pass through to the template. In this case we pass the title variable. If you are coming from Sinatra this will all be familiar to you. If you are not I recommend consulting the Express documentation.

CSS in Express

Again Express is agnostic to what you use to generate your CSS - you can use vanilla CSS but for this example I’m using Stylus. This is very similar to Sass and supports variables, mixins, functions and more. I really like it! Here’s an example from our stylesheet

body
  font 62.5%/1.5  Helvetica, Arial, "Lucida Grande", "Lucida Sans", Tahoma, Verdana, sans-serif
  text-align center
  background #000

#wrapper
  width 920px
  text-align left
  margin-left auto
  margin-right auto
  background #fff
  padding 20px
  border-bottom-radius(15px)

You’ll see that stylus is very terse - you don’t need brackets or commas.

Routing in Express

Routing is similar to Sinatra, allowing you to set up RESTful routes.

In this example we setup three routes in app.js

app.get('/', function(req, res){
  res.render('index', {
    title: 'Home'
  });
});

app.get('/about', function(req, res){
  res.render('about', {
    title: 'About'
  });
});

app.get('/contact', function(req, res){
  res.render('contact', {
    title: 'Contact'
  });
});

See the Express documentation for more.

Publishing your site

We’ve now developed a basic node.js site using express and we want to host it somewhere.

There are a number of hosting options for node. It isn’t so difficult to host it yourself but for me the easiest and best is Nodester. If you have used Heroku the deployment process is very similar. You’ll need to request an invite for Nodester (it won’t come through immediately though).

curl -X POST -d "email=your_address@gmail.com" http://nodester.com/coupon

Chris Matthieu has created a handy video going over how to deploy to Nodester so go and watch that if you don’t fancy reading the documentation.

Let’s create the site on nodester

nodester app create express_example app.js

Now we can get the information on the site

nodester app info express_example
nodester info Gathering information about: express_example
nodester warn express_example on port 9451 running: false (pid: unknown)
nodester info gitrepo: ec2-user@nodester.com:/node/hosted_apps/shapeshed/1298-ec0117a54b696d7a9781c79e5283692e.git
nodester info appfile: app.js

To publish the site we need to add the git repo as a remote our git repo

git remote add nodester ec2-user@nodester.com:/node/hosted_apps/shapeshed/1298-ec0117a54b696d7a9781c79e5283692e.git

And finally we can publish the site by pushing

git push nodester master

Nodester will let you know about the deploy - if the deploy was successful you can see your app at http://[your_app_name].nodester.com/. In this example you can see the site at http://express_example.nodester.com

Some other node.js hosting providers include nodejitsu, Joyent, Cloud Foundry and Duostack.

Conclusion

This article has showed how to create a very basic site using node.js and Express. It has introduced a number of things from the node.js ecosystem and showed you how to deploy your app to Nodester.

The strengths of node.js as a technology are not so much in building static websites like this. I encourage you to explore some of the node.js libraries to see what it can do. Particularly for real-time applications node.js is extremely exciting and I think we’ll see some great apps built on node.js. Try starting with socket.io for a taste of what to expect.

If you find any inaccuracies in the post send me an email and I’ll update the post.

Cloud Foundry - a Ruby and node.js developer's perspective

2011-04-15T00:00:00+01:00

Open Source PaaS

VMware this week announced Cloud Foundry, an open source PaaS. You can use VMware’s cloud (which is where they’ll monetise this investment) or you can host your own. I believe you can even have a hybrid of both. Cloud Foundry joins other players in the market like Heroku, Joyent, Engine Yard and Nodester.

Why PaaS?

Developers don’t want to be Sys Admins. Especially in small businesses developers take on a lot - some or all of business analysis / requirements gathering, server admin, development, testing, design, client management. The list goes on. So the emergence of services like Heroku for Ruby and Nodester for node.js is very attractive. Developers can remove one thing from the list of things that they need to be responsible for. It is an easy sell to the management layer too - it is cheaper in terms of man hours, with (almost) no maintenance.

The providers doing well in the market are the ones that realise that developers have the keys to the choices that get made. Heroku has built a kick-ass platform with a super slick API. Frankly it is a joy to use. Nodester emerged from nowhere as an open-source weekend project and for me is beating players like Joyent in their node.js offering.

One Paas to rule them all?

VMware have put considerable effort into building a PaaS to accommodate three technologies - Java, Ruby and node.js. They must be applauded for open-sourcing the project and making it available on GitHub. There are a considerable amount of man hours there and a significant investment from VMware. Thank you.

Taking it for a spin

So I took it for a spin firing up a VirtualBox Ubuntu 10.04.2 LTS VM. Installation was simple enough following the README. I created the SSH tunnel as advised and within about half an hour had built a private cloud capable of running Java, Ruby and node.js apps, with support for MySQL, Redis and MongoDB. Impressive.

The VMware team have thought hard about the API. If you are a RubyGems user you can check it out by installing the vmc rubygem and then running help

Installing the vmc gem

gem install vmc --no-ri --no-rdoc
vmc --help

Once you have setup a user on your private cloud deploying an app is super simple. Here’s the process of deploying an app

Deploying with vmc

vmc push [yourappname]

Err.. that’s it. Amazing!

Testing deployment on my cloud

Then I set about deploying a node.js app. It was a simple express based node.js application with express, jade and stylus as dependencies. So let’s give it a go

Deploying a Node.js app

vmc push myapp
Would you like to deploy from the current directory? [Yn]: y
Application Deployed URL: 'myapp.vcap.me'?
Detected a Node.js Application, is this correct? [Yn]: y
Memory Reservation [Default:64M] (64M, 128M, 256M, 512M, 1G or 2G)
Creating Application: OK
Would you like to bind any services to 'myapp'? [yN]: n
Uploading Application:
  Checking for available resources: OK
  Processing resources: OK
  Packing application: OK
  Uploading (613K): OK
Push Status: OK
Staging Application: OK
Starting Application: ..........................Error 306: Error retrieving file 'logs/startup.log'

It failed. I used the log commands but there wasn’t any useful output - I guess this is early days. But I did find this article which required changing a couple of things in the app file. After that it was as easy as

vmc update myapp
Uploading Application:
  Checking for available resources: OK
  Processing resources: OK
  Packing application: OK
  Uploading (15K): OK
Push Status: OK
Stopping Application: OK
Staging Application: OK
Starting Application: OK

Lovely! The API follows exactly the same process for deploying a Sinatra / Rails which is really attractive.

At pebble.code where I work I’m responsible for managing servers and setting up development sites for the team. This can be a time-consuming chore and I’ve been working towards a Puppet setup to allow me to automate much of this. A private cloud where developers can spin up apps with three words is massively better.

What’s good about Cloud Foundry

I love the fact that VMware understand that PaaS is about being open and not being tied to one platform. Being open makes me more likely to use a platform than one that makes it hard for me to get my data in and out. I can easily move my apps in and out at any time. I can create my own cloud and move my apps onto that. I’m free and I like that.

I love the fact that Cloud Foundry is open source and created using Ruby. The Ruby community is full of talented and smart developers and it is a wise move by VMware to engage that community. I anticipate the product will be enhanced by open source contributions.

What could be improved

It is a real shame that VMware didn’t go with git based deployment. Both the Ruby/Rails and node.js communities use git heavily and are used to git based PaaS services. I’m not sure why this design decision was made - perhaps the old skool Java heads at VMware won out. It would be great to see git deployment added to the platform. Currently there is no versioning on the platform - it would be a quick feature win to leverage the power of git to add this.

That said with the project being open source the developer community can easily fork it and enhance the platform. The fact that the project is open source is massive.

Will I use it?

Maybe. Currently I’m happy with the PaaS providers I’m using - Heroku and Nodester. I feel both are doing a great job. Heroku is closed source, whilst Nodester is open-source. I have the option to host a Nodester private cloud somewhere but Cloud Foundry is the only solution to offer both Ruby and node.js. I like that. Currently I mostly use PaaS providers for quickly prototyping an idea and getting it out there. I could see a Cloud Foundry private cloud working well for this.

The API is a big thing for me and I think VMware has done a great job here. Where Heroku has led others have followed and that’s great news for developers.

But.. for many of our production apps they have more complex technical architectures involving Resque, Redis, Sphinx, cron jobs and background processes. In short we need to have full control over the server. Cloud Foundry goes a long way to solving this by offering MongoDB and Redis but currently unless I’m mistaken I’d be unable to host apps with complex requirements. This is an issue with PaaS in general. Heroku’s add-ons solve most of this issue but there are times when simply you need root access.

So overall thank you VMware! I also need to say thank you for your support of Redis!

What are you waiting for? Give it a go, or fork it.

Vim eighteen months on

2011-04-12T00:00:00+01:00

Goodbye TextMate?

About 18 months ago I made the decision to give Vim a go as my primary text editor. I had been using vi extensively for server admin work for a number of years so I wasn’t a total noob. TextMate was my default editor and broadly I was happy with it but decided to give Vim a go.

Baby steps

I started out with baby steps, looking back to TextMate and doing all of the common things a switcher does, especially looking for TextMate’s features in Vim. Vim’s contexts were familiar to me from working on servers so I was at an advantage there but I quickly found Derek Wyatt’s awesome video series on Vim. Derek’s videos were one of the main reasons my adoption was so positive. If you are thinking of switching to Vim I strongly recommend this resource. The videos start at a beginner level and work through to advanced techniques, are excellently produced and give a particular helping hand for newcomers. If you’ve never used Vim the Welcome to Vim video is the best.

Many people recommend using MacVim to ease the transition. Henrik Nyh has a great article on switching from TextMate using MacVim. I didn’t follow this route but it may be helpful to some.

Installation of MacVim via homebrew on OSX is easy

Installing macvim

brew install macvim

MacVim works like a standard application so you don’t need to edit directly in the terminal and comes with many common keyboard shortcuts that you’ll likely be used to.

The setup phase

As with with many things in UNIX you get back what you put in. The RTFM mantra is true for Vim - it has massive and excellent documentation that you could spend months going through. The default install of Vim needs to be configured which takes time. Initially I scoured github for dotfiles and basically took configuration from developers that I admire - Tim Pope and Mislav Marohnić. I tweaked a little here and there, marrying up a color scheme with my terminal color scheme of choice ir_black. If you want to pick through my dotfiles they are here.

Pretty much after that I had a text editor that looked good, had sensible defaults for web programming and recognised the file formats I was editing. Then came the hard bit - getting to intermediate level.

Learn as you go

Vim does take time to learn. Initially it would be fair to say that I became slower. Probably for around a month. But as you go you quickly learn how Vim works and are delighted by the time-saving over other editors in almost every feature. Learning to stop using the mouse is a paradigm shift but one I highly recommend. When you get to a stage where you can quickly split windows and navigate round without using a mouse you’ll see the light. I challenge anyone to commit Vim’s many commands to muscle memory but the more you work with Vim the more it becomes automatic and that’s where Vim’s power lies.

My tip is to take a little at a time and when you come up against a common problem read the manual and learn how to do. For example you might want to lowercase some text. You could do this through deleting the text and rewriting it or you could you use Vim’s excellent help to learn how Vim does it:

:help lowercase

This takes you straight to the documentation where you can learn the keystrokes that Vim uses before doing it in a split window context. Typically it takes me four or five lookups before I remember what the keystrokes are. If I ever forget looking it up again in the help text is so easy it is not an issue.

Around about the time I was moving from a novice to intermediate Vim user Vimcasts arrived on the scene. Drew Neil produces high quality screencasts on all aspects of using Vim and I highly recommend this resource. A book is also apparently in the offing from this sage so stay tuned for more from Drew.

Pumping iron, building muscle memory

The more you use Vim the more you build on your muscle memory. More and more I’m finding I consult the documentation less and just do it. In short I’m starting to feel like I’m over the hump and am seeing the many benefits of the learning stage. But it would be fair to say that I had to pump a lot of iron to get to this stage. Months 6 - 12 were a big learning curve in that respect, and I’m still learning all the time. I don’t feel I’ll ever learn everything about Vim but I now feel I’m at a stage where I know how to get better.

Now

Fast forward 18 months and I feel I would be significantly less productive without using Vim not just for coding but also for text editing. So I can use vim for writing emails I now use Mutt, a lightweight powerful email client for the terminal. I use the Vimium extension for Chromium, and of course Vim for all of my coding. The experiment has been an extremely happy one for me.

Not for everyone

I accept that Vim is not for everyone. Casual coders, or anyone who doesn’t want to get their hands dirty should probably stick with TextMate. Yahuda Katz famously blogged that switching to Vim was not easy and I accept many of his points. Vim isn’t going to be something you can pick up and use straightaway.

But if you stick with it you might just see the light. I did and can only give Vim a massive ringing endorsement.

Translations of this article

Bulgarian - Albert Ward

Nifty Unix Tools - sort

2011-03-21T00:00:00+00:00

Sort

To demonstrate some of the features here is an example CSV file with some of my favorite cheeses:

1,Brie de Meaux,1.99
2,Maroilles,1.13
3,Stinking Bishop,1.65
4,Munster,1.29

The file is sorted by the id number in the first column. Let’s say we want to sort this by the name of the cheese.

sort -k2,2 -t , test.csv

The -t flag allows us to specify a delimiter which in this case is the comma character. The -k flag lets us specify a key to sort on. The values are the start and end point for the key. This option says take the second value on the line according to the delimiter.

This gives us an alphabetically sorted list:

1,Brie de Meaux,1.99
2,Maroilles,1.13
4,Munster,1.29
3,Stinking Bishop,1.65

Sort also lets us sort numerically so if we wanted to sort on the third price column we can use

sort -n -k3,3 -t , test.csv

Here we use the -n flag which say sort numerically and we have shifted the key to the third column.

1,Brie de Meaux,1.99
3,Stinking Bishop,1.65
4,Munster,1.29
2,Maroilles,1.13

If we want to sort by the cheapest price we can reverse the sort with the -r flag

sort -r -n -k3,3 -t , test.csv

This gives us the cheapest price first

2,Maroilles,1.13
4,Munster,1.29
3,Stinking Bishop,1.65
1,Brie de Meaux,1.99

Conclusion

The sort tool can sort strings and numbers in text files and streams. By using keys you can narrow down the search criteria and get the information you want quickly. It also has options clean up dirty data, although for anything non-trivial you’ll probably want to use something like sed.

It is a great little tool for daily use on in the shell and can make output information much easier to read in shell scripts and standard output.

Nifty Unix Tools - wc

2011-03-19T00:00:00+00:00

Word Count

Text files have lots of statitics that can be really useful when you want to extract information from them. With the word count tool it is possible to access

The number of bytes
The number of lines
The number of characters
The number of words

wc /usr/share/dict/words

This gives us

234936  234936 2486813 /usr/share/dict/words

The output is [number of lines] [number of words] [number of bytes] [filename].

We can access each of these statistics individually by passing an option to the command.

# The number of bytes
wc -c /usr/share/dict/words
# The number of lines
wc -l /usr/share/dict/words
# The number of words
wc -w /usr/share/dict/words

Piping is powerful

The wc command really starts to become useful when it is piped to other commands. Here’s an example we have 5 csv files that are full of data. We want to find out a sum of how many records there are in all five files. We can do this easily by piping the output of the cat command to wc.

cat *.csv | wc -l
1866

Done - we have 1866 records across the 5 files.

Another example might be looking for the number of occurences of a word or pattern in a file. We can combine grep with wc to achieve this.

grep "union" /usr/share/dict/words | wc -l
41

wc is a really useful tool for summing up results and gathering information on text files or streams.

Connecting clients to a Puppet Master

2011-03-13T00:00:00+00:00

Overview

Following on from a previous post where I outlined how to set up a Puppet Master server we are now going to look at how to connect a client to the Puppet Master and start to manage the client server. This walkthrough is based on two Ubuntu 10.04.2 LTS servers.

Assumptions

For the purpose of this walkthrough I’m going to assume you have a working Puppet Master server as outlined in the previous post. I’m also assuming that you have a client server and that you have either have DNS set up of that you have amended the hosts files of each server so that each can ping the other by hostname. If you are using hosts files your /etc/hosts files should look something like this

192.168.3.162 puppetmaster.example.com puppetmaster puppet
192.168.3.165 puppetclient.example puppetclient

Connect client to server

Once we can ping each server make sure the Puppet Master is running

sudo puppet master

Then we can connect from the client to the server

puppet agent --server puppetmaster --waitforcert 60 --test

You should see

info: Creating a new SSL certificate request for puppetclient

Now on the server we can see the request that has come in with

puppet cert --list

You should see ‘puppetclient’. Now we can sign the certificate with

puppet cert --sign puppetclient

The first time I did this I encountered an issue that complained that the hostname did not match the server certificate. This can be resolved by setting this explicitly in /etc/puppet/puppet.conf

[master]
    certname=puppetmaster

Now you should have both servers talking to each other and everything configured to start configuring the client with the Puppet Master. After a minute or so you should see the puppet client cache the certificate - this shows that everything is ok.

Adding a module

Now you can start building modules to be sent to the client server. I started with a module to change the message of the day banner. You can find it on github. Once you’ve added your new module restart puppet.

Then you can send modules to nodes that you manage. In /etc/puppet/manifests/nodes.pp add this

node puppetclient {
  include motd
}

When the client requests an update from the server it will be sent the new configuration and the motd will be updated.

The Puppet documentation has some good documentation on creating manifests.

Setting up Puppet on Ubuntu 10.04

2011-02-25T00:00:00+00:00

Overview

Puppet is the not quite so new automated server provisioning hotness. It allows you to provision and manage servers automatically from one Puppet Master Server. It is a great solution for quickly bringing up new VPS instances. Here is a quick walkthrough on how to get started on Ubuntu 10.04 LTS

Assumptions

I’m setting this up on two clean installs of Ubuntu 10.04 LTS Server Edition. If you are using other Linux flavours things should work but your experience may vary.

Setup

To help learn Puppet I used two Ubuntu server instances in VirtualBox. If you choose to use VirtualBox in the network settings for the virtual machine choose Bridged Adapter to allow a network IP address to be assigned.

Next I installed the two servers from the .iso and gave the Puppet Master the hostname of puppetmaster and the Puppet Client the name of puppetclient. The rest of the install is standard, selecting no automatic updates, and installing no additional software.

Installing Puppet

Once you’ve installed the servers we can get going with the interesting stuff - installing and configuring Puppet. There is a package available for Puppet in aptitude but I found I had issues with this so instead I recommend that you install it via RubyGems so let’s get that installed

First install packages required by Puppet via aptitude

sudo apt-get update
sudo apt-get install irb libopenssl-ruby libreadline-ruby rdoc ri ruby ruby-dev

Now we can install RubyGems

cd /usr/local/src
sudo wget http://production.cf.rubygems.org/rubygems/rubygems-1.5.2.tgz
sudo tar -xzf rubygems-1.5.2.tgz
cd rubygems-1.5.2
sudo ruby setup.rb
sudo update-alternatives --install /usr/bin/gem gem /usr/bin/gem1.8 1
sudo gem update --system

And finally Puppet

sudo gem install puppet

All done!

Configuring Puppet

I’m assuming you have completed the installation of Puppet on both the puppet master and client servers. I’m working locally so I need to explicity set the ip addresses of my Puppet Master and client in my /etc/hosts file. To find out the internally assigned ip address run

ifconfig

You want the entry that say inet addr: - it should be something like 192.168.3.162

Then you need to update the /etc/hosts file on each machine with each machine’s ip address

192.168.3.162 puppetmaster.example.com puppetmaster puppet
192.168.3.165 puppetclient.example puppetclient

Now on the Puppet Master you need to set up the puppet configuration in /etc. John Arundel of Bitfield Consulting has created a handy tarball that you can use as a template. Let’s get it

cd /etc
sudo wget http://bitfieldconsulting.com/files/powering-up-with-puppet.tar.gz
sudo tar -xzf powering-up-with-puppet.tar.gz

This provides a basic stucture for Puppet configuration. We’ll come onto writing manifests in a later post. Now we can start the Puppet daemon, making the relevant system users in the process

sudo puppet master --mkusers --verbose --no-daemonize

If you see it start then great. I encountered a bug here that meant the Puppet Master Deamon failed to start. The issue here is that file ownership isn’t set correctly I fixed this with

chown -R puppet:puppet /var/lib/puppet

Once you are sure everything is ok you can daemonize Puppet with

sudo puppet master

Finally we can run a local test to make sure that everything is running as expected

sudo puppet agent --test --server=`hostname`

You should see something like

info: Caching catalog for puppetmaster
info: Applying configuration version '1298651839'
notice: Finished catalog run in 0.25 seconds

Next we’ll look at connecting the client to the Puppet Master, creating manifests before moving on to managing users. Stay tuned!

Abstracting the design layer

2011-02-02T00:00:00+00:00

Overview

Whilst working on a node.js project I realised that using various tools I had extracted the entire design layer of the site.

CSS

There are now several mature and stable frameworks available to developers to help fill the gaps in CSS. Sass is my tool of choice - I’ve written about it before, also I’ve also been using LESS on a few projects recently. Using the Sass project you can now also use syntax that is very close to vanilla CSS. Here’s an example:

blue: #3bbfceg;
$margin: 16px;

.content-navigation {
  border-color: $blue;
  color:
    darken($blue, 9%);
}

.border {
  padding: $margin / 2;
  margin: $margin / 2;
  border-color: $blue;
}

I’m agnostic as to which framework is best but unless the site is very small I always use a frameowrk. Sass can be pre-compiled so it doesn’t add any overhead when the site is delivered and also provides options on whether the CSS file is pretty printed or compressed for minimum file size. Compass also provides some very useful features for CSS3 support. One common scenario is adding a border radius. As the spec is still in progress browser vendors are using prefixes. Compass takes account of this:

@include border-radius(4px, 4px);

compiles to

 -webkit-border-radius: 4px 4px;
 -moz-border-radius: 4px / 4px;
 -o-border-radius: 4px / 4px;
 -ms-border-radius: 4px / 4px;
 -khtml-border-radius: 4px / 4px;
 border-radius: 4px / 4px;

This is particularly good as if and when the spec is formalised and a universal syntax is agreed then it is very likely that compass will be updated and you won’t need to do anything other than regenerate your files.

HTML

HTML also has a number of templating languages that abstract writing raw HTML. The advantages are mainly syntax checking - the tools won’t complile if you have an error. If you care about writing valid HTML this ia great idea. For Ruby my tool of choice is HAML. In node.js a dialect of HAML is available in jade. Here’s an example of jade.

!!! 5
html(lang="en")
  head
    title My site
  body
    h1 Site title
    #wrapper
      p Oh hai!

Particularly when you are interacting with a database and outputting data to views I find these templating languages really useful. Again there are many templating languages available. I’m agnositc other than my opinion that you should be using one.

JavaScript

The final piece of the puzzle is JavaScript. JavaScript is a hated and loved language with a great object model at heart. I’ve been writing native JavaScript for quite a while now so I was interested to find the CoffeeScript project. In the last year JavaScript has become trendy again with node.js and many developers coming from Ruby found JavaScript’s syntax ugly. CoffeeScript abstracts JavaScript and presents JavaScript in a more Ruby like syntax. In compiling to JavaScript it also runs the code through JSLint. I can take or leave the syntax improvements but automatically running the code through a tool that empasises quality is a great sell for me. Here’s are the examples from the documentation:

song = ["do", "re", "mi", "fa", "so"]

singers = {Jagger: "Rock", Elvis: "Roll"}

bitlist = [
  1, 0, 1
  0, 0, 1
  1, 1, 0
]

kids =
  brother:
    name: "Max"
    age:  11
  sister:
    name: "Ida"
    age:  9

Compiles to

var bitlist, kids, singers, song;
song = ["do", "re", "mi", "fa", "so"];
singers = {
  Jagger: "Rock",
  Elvis: "Roll"
};
bitlist = [1, 0, 1, 0, 0, 1, 1, 1, 0];
kids = {
  brother: {
    name: "Max",
    age: 11
  },
  sister: {
    name: "Ida",
    age: 9
  }
};

Conclusion

Abstracting the entire front-end layer may seem drastic. There is new syntax and frameworks to learn for a start. If you working in a team of more than one developer there may also be a reluctance to move to frameworks. But for me the value is immense. With CSS you can improve code maintainability and defend against the evolving CSS3 spec and implementations. With HTML you can ensure your code is valid and make it more readable with indentation based languages like HAML or jade. With JavaScript you get a prettier syntax (which I can take or leave) but more importantly your code is automatically checked for quality by JSLint. Many JavaScript developers do this anyway but automating this has to be a good thing.

Frameworks and abstractions are not for everyone and I certainly would not consider using frameowrks on micro or nano projects. But for large complex applications they make a lot of sense to me.

Getting email through spam filters from a Rails App

2010-12-06T00:00:00+00:00

The Skinny

This is a walkthrough on how mail is delivered successfully into the inboxes of Gmail, a Google Hosted Mail account, Hotmail, Yahoo and AOL Mail from a Rails application using Postfix on an Ubuntu 10.04 server.

Knowledge of Ruby, DNS and Unix administration is required. If all of that seems to frightening I recommend a service like Sendgrid to handle delivery of mail from your Rails app.

The Task

I followed a test driven approach to this task so set out what I wanted to achieve first. Success criteria for a pass was for mail to be delivered to the inboxes of

Gmail
Google Hosted Mail Account using Postini
Hotmail
Yahoo Mail
AOL Mail

Initial tests

Sending mail from the application without any changes resulted in the following test results

FAIL: Gmail, Google Hosted Account, Hotmail, Yahoo, AOL

Content Level

First we looked at the content of the email. It was three lines of text each with a link in. This looked spammy so we decided to flesh out the email a little, taking other web based services as an example. This presented an opportunity for some help text at the end of the email. Useful to the user and useful for the purpose of getting mail through.

FAIL: Gmail, Google Hosted Account, Hotmail, Yahoo, AOL

Application Level

I decided to take HTML email out of the equation entirely and deliver plain text mail only. It may be that this gets added back in at a later date but it seemed easier to test things by taking out one variable.

It is easy to ensure that your Rails app sends email as plain text. Make sure your templates end with .text.plain.erb and the correct mime type will be sent. Note that if you have templates with .text.html.erb these will need to be removed or a multipart email will be sent.

Following some excellent advice from Ilya Grigorik I then ensured that the headers on the emails were correct. In my case the reply to address was different from the from header.

At this stage messages were getting through Postini but still ending in SPAM in Gmail but were being rejected by Hotmail and Yahoo.

Partial PASS: Google & Google Hosted Account (delivered but in Spam folder)

FAIL: Hotmail, Yahoo, AOL

Server

A great article by Kenn Wilson really helped to configure the mail server correctly. To start with there was no fully qualified domain name in place on the server and the reverse DNS was set to the dynamically generated Slicehost address. Firstly a FQDN was set on the server and then a reverse DNS entry was set to point to the FQDN rather than the Slicehost address. This is all detailed in Kenn’s article so I won’t duplicate the details here.

Partial PASS: Google & Google Hosted Account (delivered but in Spam folder)

FAIL: Hotmail, Yahoo, AOL

SPF Records

The Sender Policy Framework allows you to specify a TXT DNS entry that identifies which servers are allowed to send mail for a domain, or subdomain. The Project homepage has a good overview and a wizard to help you set up DNS records.

Partial PASS: Google & Google Hosted Account (delivered but in Spam folder)

FAIL: Hotmail, Yahoo, AOL

Spamhaus

Spamhaus by default blacklists some IPs until they are manually removed. This was the case with our Slicehost IP. There are a few services to see if your domain is blacklisted. MX Toolbox offers and excellent free service. If you discover that your IP is listed in Spamhaus there is a process for removing it. Once DNS had propagated we saw messages making it through to Hotmail and Gmail. Still though we were seeing messages ending up in the Spam folder for Yahoo!

PASS: Google & Google Hosted Account, Hotmail, AOL

FAIL: Yahoo

Domain Keys

Domain Keys is an idea that Yahoo is behind so I felt there was a good potential that if Domain Keys were implemented that Yahoo Mail would start to trust the mail. I found a great walkthrough detailing how to do this on Ubuntu. I won’t duplicate that article here, but the short answer is that once DNS had propagated Mail was now being delivered successfully to Yahoo.

PASS: Google & Google Hosted Account, Hotmail, AOL, Yahoo

Wrapping up

Getting mail delivered from a Rails application to all of these email providers was not an easy task and all in took about a day’s work. I can now understand why many developers just opt for delivery via a service like SendGrid.

But having completed the exercise the application now has reliable email delivery to all of these email services without the additional overhead of using a third-party service. I’m also sure this requirement will come up again so the experience should stand me in good stead.

If you have anything to add from your own experience please email me and I’ll update the article.

Nifty Unix Tools - head and tail

2010-11-03T00:00:00+00:00

Head - view the top of files

Head is simple tool to view the top of files. In its simplest form you can use it to see the first ten lines of a file.

head /usr/share/dict/words
A
a
aa
aal
aalii
aam
Aani
aardvark
aardwolf
Aaron

By default it outputs the first ten lines of a file but you can modify that by passing the -n option

head -n 1 /usr/share/dict/words
A

In fact you don’t even need to use -n. You can just use a number.

head -1 /usr/share/dict/words
A

This is useful for viewing comments and the the top of files in general but becomes really useful when you pipe things together.

FILES=/some/dir/*.zip
SEARCH_STRING="findme"

for f in $FILES
do
  zipinfo -1 $f | grep $SEARCH_STRING | head -1 | awk '{ print "Found in: " $1 }'
done

Here we are searching the contents of zip files with a search string but we only want the first occurence. So we can use head to filter the results to just one.

Tail

Tail is the opposite to head but operates on the end of files.

tail /usr/share/dict/words
zymotoxic
zymurgy
Zyrenian
Zyrian
Zyryan
zythem
Zythia
zythum
Zyzomys
Zyzzogeton

It also accepts the -n argument for the number of lines you want

tail -n 1 /usr/share/dict/words
Zyzzogeton

This also works passing just a number

tail -1 /usr/share/dict/words
Zyzzogeton

Watching log files

Tail has the option to watch files in real time so you can keep an eye on log files. You can enable this by passing the -f option.

tail -f /Users/george/Sites/myrailspp/log/development.log

This is really useful for watching logs. You can use it for watching apache, mail logs - any file!

head and tail are simple tools but are extemely useful for everyday usage, especially in piping so worth understanding!

Installing Passenger 3 with RVM and Nginx on OSX

2010-10-21T00:00:00+01:00

Nginx

My preferred way of installing nginx on OSX is using homebrew. You need to compile it with passenger support

brew install nginx --with-passenger

RVM

RVM makes it easy to manage multiple rubies. Installation is easy

bash < <( curl http://rvm.beginrescueend.com/releases/rvm-install-head )

You can then install different versions of ruby. In this case let’s install Ruby Enterprise Edition

rvm install ree

Then we can make Ruby Enterprise Edition the default Ruby on this machine with

rvm --default ree

Passenger

This is the most tricky part of the install. To start you’ll need the passenger gem

gem install passenger

We need to use the nginx source code for this process. If your homebrew setup is a default one run

tar -xzf /Users/george/Library/Caches/Homebrew/nginx-0.8.53.tar.gz

Note your version of nginx may be different from this!

Then we need to run the nginx passenger module installer

passenger-install-nginx-module

The default install downloads an entirely new binary for Nginx - a bit annoying. If you choose option 2 you’ll be able to choose the source you’ve downloaded with homebrew. For the source directory choose the one we’ve just extracted

/Users/george/Library/Caches/Homebrew/nginx-0.8.53

Then choose to install Nginx to (if you’ve used the default homebrew install)

/usr/local/Cellar/nginx/0.8.53/sbin

Hit return on the next two prompts. Drink tea whilst it compiles.

The installer spits out some configuration for you. Open up /usr/local/etc/nginx/nginx.conf in your favourite text editor and paste this in.

Then you just need to create a virtual host

server {
    listen 80;
    server_name foo.local;
    root /webapps/foo.com/public;
    passenger_enabled on;
}

Provided you’ve added foo.local to your /etc/hosts file you should be able to start the server and see your site using

sudo nginx

Running multiple virtual hosts

Passenger 3 adds a killer feature - the ability to run a standalone passenger and proxy using Nginx and TCP sockets. The process for this is to first setup a virtual host to proxy through to a socket

upstream bar_upstream {
    server unix:/tmp/bar.local.socket;
}

server {
    listen 80;
    server_name bar.local;
    root /webapps/bar.local/public;
    location / {
        proxy_pass http://bar_upstream;
    }
}

Then start a standalone instance of Passenger. This downloads and complies a new binary of Nginx for some reason

passenger start --socket /tmp/bar.local.socket -d --nginx-version 0.8.53

This is explained in more detail on the Phusion Blog. Because the standalone passenger is invoked within an RVM context it is completely possible to have different ruby versions and gemsets. TOTAL WIN!

It should then be trivial use something like Monit to make sure that the process for virtual hosts is running and start / stop it.

This setup can be applied to a Linux server too to create a flexible Rails stack with multiple hosts, Rubies and gemsets.

Background job nirvana with Resque, Redis and God

2010-09-08T00:00:00+01:00

Resque

Resque is a Ruby library that lets you place jobs on queues and process them later. It was created by defunkt and is in production use on Github. The documentation is excellent. I’m using Resque on a Rails app but it can be used with any Rack application.

First you need to install Redis. On OSX this is trivial with homebrew

brew install redis

If you installed Redis with homebrew it spits out some helpful information about how to start Redis on boot or you can start the server with

redis-server

The installation notes for Resque are excellent so I won’t repeat them here - head over to the README.

Booting the GUI

Resque ships with an great GUI that lets you track jobs and workers. To launch it from the root of your site run

resque-web

Then you’ll probably want to spawn some workers

COUNT=5 QUEUE=* rake resque:workers

In the GUI you should now see 5 workers waiting to do your bidding

Now you can start sending jobs to Resque and watch your workers eat them up. The GUI also provides a great way to review failures. The stack trace is even included!

Settings for production

There are a few issues to consider in production. Firstly if your Redis server is used by other applications you might want to specify a namespace for the application to use. You can do this by adding the following to an initialiser after Redis is configured.

Resque.Redis.namespace = "resque:YourApp"

As Redis supports running multiple databases on a server it may be possible to specify which database to use but that wasn’t a requirement for me. I couldn’t find anything in the documentation about this. If anyone knows anything about this let me know and I’ll update the post.

In production it will be important that Redis and the workers have high availability. Redis can easily be configured to start on reboot but God can also monitor a live system and restart Redis if necessary. God is another Ruby based library for monitoring services on a server. The configuration files are written with Ruby so it is easy to extend it to whatever you’d like. Using God we can easily make sure our workers are well-managed and that if Redis does go down it will be restarted. There is even an example script in Resque that will get you started with managing workers.

With God running on the server we can be sure that Redis and our workers will have high availability, or that we’ll know about it quickly if not.

General win

Adding a highly available and robust background queue to the application took less than a day. Thanks to defunkt and mojombo for their excellent open source libraries!

shapeshed

Compiling Node.js from source on Ubuntu 10.04

Installing from a tarball

Upgrading via a tarball

Installing from Git

Upgrading via Git

Using rbenv to manage rubies

rbenv is great

rbenv basics

Deployment with capistrano

Integrating with other tools

Better separation

Thanks rvm, onwards with rbenv

Managing Unicorn Workers with Monit

Riding Unicorns

More power == more responsibility

Up the workers

Adding in Monit

Migrating from Slicehost to Linode

Slicehost was a game changer

PR Car Crash

Free markets are good

The pain of migrating

Migrating DNS

Migrating the server

My experience of Blueprint

Finishing up with scp

Thanks Slicehost

Going realtime with node.js, Express and socket.io

Counting is fun!

Just show me the app

Pulling the pieces together

Potentials uses

Instant Rails dev environments with Tmuxinator and Foreman

The problem

Tmux

Tmuxinator

Foreman

Alternatives

Building a Rails server with Puppet

The goal

Learning curve

Setting up Puppet Dashboard

Rails stack choices

Switching the Puppet Master to Unicorn

Workflow

Still to do…

The good bit

Building a Rails development server

Promoting nginx to port 80

Getting nginx

Example configs

Default proxy settings

Proxy to Apache

Proxy to Thin

Proxy to Unicorn

So which one is best?

Conclusion

Creating a basic site with node.js and Express

What we are going to do

Setup

Prerequisites

Create an Express site

Boot the app

Using Git

Developing node.js sites

HTML in Express

CSS in Express

Routing in Express

Publishing your site

Conclusion

Further reading

Cloud Foundry - a Ruby and node.js developer's perspective

Open Source PaaS

Why PaaS?

One Paas to rule them all?

Taking it for a spin

Testing deployment on my cloud

What’s good about Cloud Foundry

What could be improved