kris.me.uk

Send Email on OS X Yosemite through GMail

2015-03-05T00:00:00+00:00

This guide covers how to utilise the installed postfix program so that it relays local email through google mail. I predominately use this for notifications such as logs of cron tasks that have run. This guide borrows heavily from the references listed at the end.

Configuration

Launch configuration

Edit /System/Library/LaunchDaemons/org.postfix.master.plist to include OnDemand key. The resulting file should look like:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>org.postfix.master</string>
  <key>Program</key>
  <string>/usr/libexec/postfix/master</string>
  <key>ProgramArguments</key>
  <array>
    <string>master</string>
    <string>-e</string>
    <string>60</string>
  </array>
  <key>QueueDirectories</key>
  <array>
    <string>/Library/Server/Mail/Data/spool/maildrop</string>
  </array>
  <key>AbandonProcessGroup</key>
  <true/>
  <key>OnDemand</key>
  <true/>
</dict>
</plist>

Postfix main configuration

Edit /etc/postfix/main.cf to include necessary configuration. Add the following to the end of the file:

relayhost = smtp.gmail.com:587
smtp_generic_maps = hash:/etc/postfix/generic
smtp_tls_loglevel=1
smtp_tls_security_level=encrypt
smtp_sasl_auth_enable=yes
smtp_sasl_password_maps=hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_sasl_mechanism_filter = plain

Note: the smtp_sasl_mechanism_filter option was required since Yosemite.

Postfix aliases

First, note the username of the user you wish to receive root’s email. If logged in as the user, you can check the value by running:

whoami

Edit /etc/postfix/aliases enabling root email to be aliased to your user. Change the following line:

#root:    you

by uncommenting and changing you to your username, e.g.:

root:    username

You need to then run newaliases to pick up the modifications:

cd /etc/postfix
sudo newaliases

Postfix local email to gmail mappings

In order to configure correctly, note the user and hostname you want to map:

whoami
hostname

Edit /etc/postfix/generic adding mappings of local email to gmail accounts, substituting username, hostname and gmailaccount as appropriate:

username@hostname gmailaccount
@hostname gmailaccount

You need to then run postmap to pick up the modifications:

cd /etc/postfix
sudo postmap generic

SMTP password

Ensure that the password file exists:

cd /etc/postfix
sudo touch sasl_passwd

Edit /etc/postfix/sasl_passwd adding the details for gmail:

smtp.gmail.com:587 gmailaccount:gmailpassword

Rebuild the postmap database file and delete the raw file containing the raw password:

sudo postmap sasl_passwd
sudo rm sasl_passwd

Forwarding using .forward

Individual users can use sendmail forwarding by creating ~/.forward containing only the email address to forward to. This will override the accounts configured above, allowing someone without sudo permissions to configure their desired email address.

Run postfix

Run the following commands to ensure postfix is running with the latest configuration:

sudo launchctl unload -w /System/Library/LaunchDaemons/org.postfix.master.plist
sudo launchctl load -w /System/Library/LaunchDaemons/org.postfix.master.plist

Testing

Sending to root

Run the following command:

echo $(date) | mail -s "test" root

Sending to username

Run the following command:

echo $(date) | mail -s "test" username

Sending an external email

Run the following command:

echo $(date) | mail -s "test" user@example.com

Problems

If email is not working, the following may help.

Tailing the mail logs

You can look at and tail the mail log for details of what is happening.

tail -f /var/log/mail.log

Spool directory not found

If you get the following error:

sendmail: fatal: chdir /Library/Server/Mail/Data/spool: No such file or directory

Run the following commands to fix it:

sudo mkdir -p /Library/Server/Mail/Data/spool
sudo gzip /usr/share/man/man1/{postalias.1,postcat.1,postconf.1,postdrop.1,postfix.1,postkick.1,postlock.1,postlog.1,postmap.1,postmulti.1,postqueue.1,postsuper.1,sendmail.1}
sudo gzip /usr/share/man/man5/{access.5,aliases.5,bounce.5,canonical.5,cidr_table.5,generic.5,header_checks.5,ldap_table.5,master.5,mysql_table.5,nisplus_table.5,pcre_table.5,pgsql_table.5,postconf.5,postfix-wrapper.5,regexp_table.5,relocated.5,tcp_table.5,transport.5,virtual.5}
sudo gzip /usr/share/man/man8/{anvil.8,bounce.8,cleanup.8,discard.8,error.8,flush.8,local.8,master.8,oqmgr.8,pickup.8,pipe.8,proxymap.8,qmgr.8,qmqpd.8,scache.8,showq.8,smtp.8,smtpd.8,spawn.8,tlsmgr.8,trivial-rewrite.8,verify.8,virtual.8}
sudo /usr/sbin/postfix set-permissions

References

Send Email on OS X Lion through GMail

2013-02-23T00:00:00+00:00

Configuration

Launch configuration

Edit /System/Library/LaunchDaemons/org.postfix.master.plist to include OnDemand key. The resulting file should look like:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>org.postfix.master</string>
  <key>Program</key>
  <string>/usr/libexec/postfix/master</string>
  <key>ProgramArguments</key>
  <array>
    <string>master</string>
    <string>-e</string>
    <string>60</string>
  </array>
  <key>QueueDirectories</key>
  <array>
    <string>/Library/Server/Mail/Data/spool/maildrop</string>
  </array>
  <key>AbandonProcessGroup</key>
  <true/>
  <key>OnDemand</key>
  <true/>
</dict>
</plist>

Postfix main configuration

Edit /etc/postfix/main.cf to include necessary configuration. Add the following to the end of the file:

relayhost = smtp.gmail.com:587
smtp_generic_maps = hash:/etc/postfix/generic
smtp_tls_loglevel=1
smtp_tls_security_level=encrypt
smtp_sasl_auth_enable=yes
smtp_sasl_password_maps=hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous

Postfix aliases

First, note the username of the user you wish to receive root’s email. If logged in as the user, you can check the value by running:

whoami

Edit /etc/postfix/aliases enabling root email to be aliased to your user. Change the following line:

#root:    you

by uncommenting and changing you to your username, e.g.:

root:    username

You need to then run newaliases to pick up the modifications:

cd /etc/postfix
sudo newaliases

Postfix local email to gmail mappings

In order to configure correctly, note the user and hostname you want to map:

whoami
hostname

Edit /etc/postfix/generic adding mappings of local email to gmail accounts, substituting username, hostname and gmailaccount as appropriate:

username@hostname gmailaccount
@hostname gmailaccount

You need to then run postmap to pick up the modifications:

cd /etc/postfix
sudo postmap generic

SMTP password

Ensure that the password file exists:

cd /etc/postfix
sudo touch sasl_passwd

Edit /etc/postfix/sasl_passwd adding the details for gmail:

smtp.gmail.com:587 gmailaccount:gmailpassword

Rebuild the postmap database file and delete the raw file containing the raw password:

sudo postmap sasl_passwd
sudo rm sasl_passwd

Forwarding using .forward

Run postfix

Run the following commands to ensure postfix is running with the latest configuration:

sudo launchctl unload -w /System/Library/LaunchDaemons/org.postfix.master.plist
sudo launchctl load -w /System/Library/LaunchDaemons/org.postfix.master.plist

Testing

Sending to root

Try the following at a command prompt, filling in the prompts:

mail root
Subject: Test root
<CTRL+D>

Sending to username

Try the following at a command prompt, filling in the prompts:

mail username
Subject: Test username
<CTRL+D>

Sending an external email

Try the following at a command prompt, filling in the prompts:

mail user@example.com
Subject: Test email
<CTRL+D>

Problems

If email is not working, the following may help.

Tailing the mail logs

You can look at and tail the mail log for details of what is happening.

tail -f /var/log/mail.log

Spool directory not found

If you get the following error:

sendmail: fatal: chdir /Library/Server/Mail/Data/spool: No such file or directory

Run the following commands to fix it:

sudo mkdir -p /Library/Server/Mail/Data/spool
sudo gzip /usr/share/man/man1/{postalias.1,postcat.1,postconf.1,postdrop.1,postfix.1,postkick.1,postlock.1,postlog.1,postmap.1,postmulti.1,postqueue.1,postsuper.1,sendmail.1}
sudo gzip /usr/share/man/man5/{access.5,aliases.5,bounce.5,canonical.5,cidr_table.5,generic.5,header_checks.5,ldap_table.5,master.5,mysql_table.5,nisplus_table.5,pcre_table.5,pgsql_table.5,postconf.5,postfix-wrapper.5,regexp_table.5,relocated.5,tcp_table.5,transport.5,virtual.5}
sudo gzip /usr/share/man/man8/{anvil.8,bounce.8,cleanup.8,discard.8,error.8,flush.8,local.8,master.8,oqmgr.8,pickup.8,pipe.8,proxymap.8,qmgr.8,qmqpd.8,scache.8,showq.8,smtp.8,smtpd.8,spawn.8,tlsmgr.8,trivial-rewrite.8,verify.8,virtual.8}
sudo /usr/sbin/postfix set-permissions

References

Mountain Lion with Favourite Apps

2012-09-22T00:00:00+01:00

With mountain lion being out, I decided it was time to do a clean install and rid my system of the clutter I’ve accumulated over the last few years.

When choosing applications to use I am mindful of not only how useable they are but also whether or not they are open source or low cost and can be used on other platforms so the knowledge and skills are transferable where possible. I’ve tried to loosely group these into sensible themes. Where these apps are available in the app store I’ve included a link for that also.

File System Browser

Finder does not offer the filesystem visibility that I like, so I use PathFinder when I need a little more power.

Web Browser

I use Google Chrome and occasionally Firefox depending on what I’m doing and how well the sites work in the browsers.

To synchronise browser information, I use Xmarks and LastPass.

For Chrome I use the following plugins for general use:

Add to Amazon Wishlist
Google Calendar Checker
Google Dictionary
Google Mail Checker
Google Reader Notifier
Google Similar Pages
Lastpass
Microformats for Google Chrome
Pocket
RSS Subscriptions Extension
Session Buddy
The Camelizer
Xmarks Bookmarks Sync

And the follow plugins for development:

Firebug Lite
JSONView
LiveReload
PageSpeed Insights
Speed Tracer

Resource Monitoring

iStat Menus provides the slickest resource monitoring for the menubar. I end up ditching most of the standard menubar icons by holding down ⌘ and dragging them off.

Notifications

Growl was the defacto notification system for OS X, though this may change now that mountain lion has notifications built in. For now I’m keeping it around till there is more support for native notifications. Many applications utilise growl if installed. I use HardwareGrowler too.

Instant messaging

Adium is a generic chat client with support for most chat protocols that also integrates with Growl. Skype is also very good and can be used for messaging, voice calls and video calls.

FTP

FTP Client

Filezilla is a free, open source, cross platform ftp client. Cyberduck is a free, open source FTP, SFTP, WebDav and cloud browser for Mac OS-X. For those happy to spend a few coins, Transit is highly commended.

Media

Player

VLC is a really good cross platform media player with support for most formats.

Transcoder

Handbrake is a good cross platform video transcoder.

Little App Factory

Little App Factory do a number of cool little apps, such as ripit and tagalicious.

Sync

Dropbox

Dropbox only syncs files you put into it but has the advantage that its software as a service and therefore a backup is stored on a server and you can also access files through the website or mobile phone applications. Its free for up to 2GB of storage and this can be extended through referrals and rewards through participation in events, and you can pay for larger storage needs.

Text Editor

There are quite a number of text editors available, however, there is a lot of support for TextMate on the mac. If you are happy with the command line, vim is very popular especially when used with tmux. Sublime Text has been gaining a lot of popularity recently also.

Screen grabber

Skitch is a really good screen grabber and editor.

Image Editor

There are numerous image editors available at varying costs. For most needs, I highly recommend Pinta which is based on Paint.NET for windows.

Launcher

Spotlight is pretty good, but Alfred is even better. Launch what you want quickly and intuitively with some nice extra features thrown in.

General Development

Terminal

iTerm2 is a terminal replacement that supports tmux and panels. Very useful when working predominately in the terminal.

XCode

Xcode is a suite of development tools used to create Mac and iPhone applications. It provides a number of building blocks for other applications, such as build tools and compilers. Make sure you install the command line tools too from the preferences -> downloads dialog once xcode itself is installed.

XQuartz

XQuartz is a version of X Windows that runs on OS X. It is used by some cross platform tools.

Homebrew

Homebrew is the best package manager for OS X. XCode and XQuartz are prerequisites for some packages (aka formulae in homebrew), providing gcc and other important tools. Be sure to run brew doctor to ensure you have the right configuration.

Ruby

Ruby is a great programming language and has packages that are called gems. Some gems contain binaries and are therefore executable from the terminal.

To manage the ruby environment, it is useful to an environment manager. rbenv and ruby-build take a lightweight approach, whereas RVM is a collection of tools for ruby development.

Java

Java is not installed by default on mountain lion, but simply running java -version from the terminal will ensure it installs, prompting you as appropriate. Over time, Oracle will be taking on the responsibility of providing java packages for OS X.

Base Packages

I use the following script to populate base packages / gems / eggs / npms:

#!/bin/bash
brew install ack bash-completion curl dos2unix readline rename tmux tree wakeonlan wget zsh
brew install gist git imagemagick subversion xmlstarlet
brew install mongodb mysql postgresql redis riak sqlite
brew install rbenv ruby-build
brew install node
brew install maven
brew install python

# java stuff - this forces install on mountain lion
java -version

# ruby stuff
if [ ! -d ~/.rbenv/versions/1.9.3-p194 ] ; then
  rbenv install 1.9.3-p194
  rbenv global 1.9.3-p194
  gem install --no-rdoc --no-ri bundler yard pry whenever lunchy
  gem install --no-rdoc --no-ri rails cucumber-rails rspec-rails capistrano
  rbenv rehash
fi

# python stuff
pip install jsonpipe pygments

# npm for node
npm install -g coffee-script

zsh

zsh seems to be the way to go these days over bash. If you want to use the latest version from homebrew, you will need to register it as outlined in the info showed after installing, i.e. adding /usr/local/bin/zsh to /etc/shells. oh-my-zsh is a good collection of zsh goodness to get you started which does most of the hardwork for you, including switching your default shell.

Git GUI

There are a few good git graphical user interfaces, SourceTree, Github for Mac and GitX fork by Laullon.

Rails 3.1 / Rack App Hosting with RVM, Passenger 3, Capistrano, Git and Apache

2011-10-28T00:00:00+01:00

This guide supersedes a number of earlier guides, providing the details to get up and running with a rack based application from a basic ubuntu server VPS. It is a walkthrough of the steps taken, and it is advisable to first read the installation instructions provided by the software authors.

At the time of writing, the latest versions of the software in use are:

RVM: 1.9.0
Ruby: 1.9.2-p290
Passenger: 3.0.9
Rails: 3.1.1
Capistrano: 2.9.0

You may need to alter the versions in the steps below as some commands and paths contain explicit versions. The server used for hosting in the guide is an Ubuntu Lucid VPS, so tailor the steps to your platform accordingly. Also, substitute domainname, hostname and appname as appropriate for your environment.

RVM Multi User Install

RVM can be installed for multiple users by installing as root. This allows installations to be shared across different users, although different users can use which ever version they wish.

bash < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)

In order for a user to use the multi user RVM installation the must belong to the rvm group. Also check that they do not have a ~/.rvm directory as this takes precedence over the multi user install for that user.

Create Passenger User

It is a good idea to create a passenger user on the server to install services as so that it is isolated from other users. You can also deploy your applications using this user. The passenger user needs sudo rights for installing packages using apt-get and running the passenger install script, and needs to be a member of the rvm group to use the multi user install.

sudo adduser passenger
sudo usermod -G passenger,www-data,sudo,rvm passenger
su - passenger

You can remove the sudo right after completing the installation using the following:

sudo usermod -G passenger,www-data,rvm passenger

Generate a key pair for ssh too. On your client machine, create a key and use a string pass phrase:

ssh-keygen -v -t rsa -f passenger@domainname -C passenger@domainname

This will create a private key called passenger@domainname and a public key called passenger@domainname.pub. Keep these somewhere safe.

You can register the public key into authorized_keys to allow key based ssh authentication. Run the following on your client machine:

scp passenger@domainname.pub passenger@hostname:~/.ssh/authorized_keys

Depending on the client and your preference, you can either enter the passphrase for you private key each time you connect, or run ssh-agent and register the key using ssh-add. To register the private key into your ssh agent on your client machine run the following and enter the passphrase when prompted:

ssh-add passenger@domainname

You should now be able to ssh to the server without being challenged for the password, e.g:

ssh passenger@hostname

Passenger RVM Configuration

Note: if you previously had RVM installed as a single user you need to remove the old source line for $HOME/.rvm/scripts/rvm and move (or delete) the .rvm directory so that the system wide installation is picked up instead.

Log out and log in again as passenger to ensure the scripts work as expected. You can test RVM is available correctly by running:

type rvm | head -n1

You should then see the following output:

rvm is a function

You can view the requirements for installing the different rubies by running rvm requirements.

To install the packages (requires sudo):

sudo apt-get install build-essential openssl libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev automake libtool bison subversion

To install ruby 1.9.2:

rvm install 1.9.2
rvm use --default 1.9.2

You can can now verify the ruby version and install some gems, e.g.:

ruby -v
gem install bundler rails

Passenger

Passenger is very easy to install. RVM provides some instructions specifically for passenger.

Install the gem and run the installation script. It will tell you what dependencies you are missing and how to install them:

gem install passenger
passenger-install-apache2-module

Passenger provides some instructions on how to configure apache. I like to treat passenger as a mod for apache, allowing it to be enabled and disabled as required through the a2enmod and a2dismod commands respectively.

To enable passenger, create /etc/apache2/mods-available/passenger.load with the following contents:

LoadModule passenger_module /usr/local/rvm/gems/ruby-1.9.2-p290/gems/passenger-3.0.9/ext/apache2/mod_passenger.so

and /etc/apache2/mods-available/passenger.conf with the following contents:

PassengerRoot /usr/local/rvm/gems/ruby-1.9.2-p290/gems/passenger-3.0.9
PassengerRuby /usr/local/rvm/wrappers/ruby-1.9.2-p290/ruby

then enable the module and restart apache:

sudo a2enmod passenger
sudo /etc/init.d/apache2 restart

Apache

Each rack/rails application is made available through a virtual host directive in the apache configuration. The recommended way to do this is to create a file named after the site, e.g. domainname in /etc/apache2/sites-available and then enable and disable the site as required using the a2ensite and a2dissite commands respectively.

I prefer to put all web applications in /var/www and have users be able to create applications if they are a member of www-data:

sudo mkdir /var/www
sudo chown www-data:www-data /var/www
sudo chmod g+w /var/www

Hello World rack app

I find it useful to have a simple Hello World rack application to verify the rvm and passenger installation. The application can be run as a subdomain to allow testing and can be enabled and disabled as desired.

Create the necessary directory structure for a rack application:

cd /var/www
mkdir hw.domainname
cd hw.domainname
mkdir public
mkdir tmp

Create /var/www/hw.domainname/config.ru with the following contents:

app = proc do |env|
  [200, { "Content-Type" => "text/html" }, ["hello, world"]]
end
run app

Create /etc/apache2/sites-available/hw.domainname with the following contents:

<VirtualHost *:80>
    ServerName hw.domainname
    DocumentRoot /var/www/hw.domainname/public
    <Directory /var/www/hw.domainname/public>
        AllowOverride all
        Options -MultiViews
    </Directory>
</VirtualHost>

Enable the site, and then view it in your browser:

sudo a2ensite hw.domainname
sudo /etc/init.d/apache2 reload

This can then be disabled and reenabled if needed for debugging any issues at a later date:

sudo a2dissite hw.domainname
sudo /etc/init.d/apache2 reload

Rails app

You now need to create a virtual host for the rails app so that apache can handle the requests and pass off to passenger.

Create the necessary directory structure for a rails application (assuming its running as the domainname):

cd /var/www
mkdir domainname

Create /etc/apache2/sites-available/domainname with the following contents:

<VirtualHost *:80>
    ServerName domainname
    DocumentRoot /var/www/domainname/current/public
    <Directory /var/www/domainname/current/public>
        AllowOverride all
        Options -MultiViews
    </Directory>
</VirtualHost>

Enable the site, although there is nothing there to view until we have deployed it:

sudo a2ensite domainname
sudo /etc/init.d/apache2 reload

Capistrano

The following steps should be performed on your client machine to capify an existing rails application and deploy it.

Configuration

Make sure you have capistrano installed:

gem install capistrano

Initialise the rails application:

cd ~/dev/appname
capify .

This will generate some files, along with a sample config/deploy.rb. Replace the file contents with the following, using suitable values where applicable.

# RVM bootstrap
$:.unshift(File.expand_path('./lib', ENV['rvm_path']))
require 'rvm/capistrano'
set :rvm_ruby_string, '1.9.2-p290'

# bundler bootstrap
require 'bundler/capistrano'

# main details
set :application, "domainname"
role :web, "domainname"
role :app, "domainname"
role :db,  "domainname", :primary => true

# server details
default_run_options[:pty] = true
ssh_options[:forward_agent] = true
set :deploy_to, "/var/www/domainname"
set :deploy_via, :remote_cache
set :user, "passenger"
set :use_sudo, false

# repo details
set :scm, :git
set :scm_username, "passenger"
set :repository, "git@gitserver:domainname.git"
set :branch, "master"
set :git_enable_submodules, 1

# tasks
namespace :deploy do
  task :start, :roles => :app do
    run "touch #{current_path}/tmp/restart.txt"
  end

  task :stop, :roles => :app do
    # Do nothing.
  end

  desc "Restart Application"
  task :restart, :roles => :app do
    run "touch #{current_path}/tmp/restart.txt"
  end
end

Note: the above file supplies the rvm ruby version as 1.9.2-p290. It is advisable to be specific about the version here as if this were 1.9.2 and a newer ruby version is available you may start to receive errors in the capistrano deployment.

Deployment

The first time you deploy to the server you should run the deployment setup task:

cap deploy:setup

Thereafter you can deploy using:

cap deploy

Or if you have database migrations to run then use:

cap deploy:migrations

You may get some strange errors or failures when deploying. If you have followed the steps I have mentioned in this guide then hopefully you shouldn’t have many problems. Common problems are:

wrong permissions of /var/www
wrong permissions of passenger user
not having rvm installed for passenger user
not having the basic gems required to use capistrano on the server, simply install them as the passenger user

Testing

View your website in your browser and ensure it is loading.

Subversion Autoprops

2010-11-17T00:00:00+00:00

I have noticed in particular with Subversion that in an environment with different operating systems, you often end up with mixed line endings in files unless some measures are taken to combat this. This is a particular issue with using an svn master with git mirrors as you may see problems with git-svn.

Subversion allows metadata to be stored and versioned against any file or directory in the repository using properties. Subversion has its own namespace for properties that extend its capabilities to manage the items under source control. Some of the key properties are:

svn:mime-type - value is returned as the Content-type HTTP header on GET requests (useful when browsing the repository)
svn:keywords - provides keyword expansion such as Author, Date, Id and Revision, and ensuring that changes to these values do not cause differences
svn:eol-style - handles CR+LF issues in heterogeneous operating system environments
svn:executable - will set execution mask correctly on operating systems that support it

A key part of this is ensuring that properties are set effectively. Subversion provides a mechanism called autoprops to set default properties on files and directory based on pattern matching the filename. Autoprops does not modify existing files as you may need to deviate from the defaults in rare cases, so addition is chosen as a suitable point to consider what these values should be.

Configuring autoprops

All subversion clients support autoprops and this is usually configured in the runtime configuration area. This file is usually located at ~/.subversion/config although this may vary depending on the subversion client and the operating system.

Modify the subversion configuration file (config), making the following changes:

Ensure enable-auto-props = yes is present and not commented out
Ensure the [auto-props] section at the end of the file contains at least the following entries:

*.bat = svn:mime-type=text/plain;svn:eol-style=native;svn:executable=on
*.c = svn:mime-type=text/plain;svn:eol-style=native
*.cmd = svn:mime-type=text/plain;svn:eol-style=native;svn:executable=on
*.cpp = svn:mime-type=text/plain;svn:eol-style=native
*.css = svn:mime-type=text/css;svn:eol-style=native
*.doc = svn:mime-type=application/msword
*.dtd = svn:mime-type=text/plain;svn:eol-style=native
*.erb = svn:mime-type=text/plain;svn:eol-style=native
*.gif = svn:mime-type=image/gif
*.h = svn:mime-type=text/plain;svn:keywords=Author Date Id Revision;svn:eol-style=native
*.html = svn:mime-type=text/html;svn:eol-style=native
*.java = svn:mime-type=text/plain;svn:eol-style=native;svn:keywords=Author Date Id Revision
*.jpg = svn:mime-type=image/jpeg
*.js = svn:mime-type=text/plain;svn:eol-style=native
*.jsp = svn:mime-type=text/plain;svn:eol-style=native
*.png = svn:mime-type=image/png
*.properties = svn:mime-type=text/plain;svn:eol-style=native
*.rb = svn:mime-type=text/plain;svn:eol-style=native;svn:executable=on
*.sh = svn:mime-type=text/plain;svn:eol-style=native;svn:executable=on
*.sql = svn:mime-type=text/plain;svn:eol-style=native
*.tld = svn:mime-type=text/xml;svn:eol-style=native
*.txt = svn:mime-type=text/plain;svn:eol-style=native
*.xls = svn:mime-type=application/msexcel
*.xml = svn:mime-type=text/xml;svn:eol-style=native
*.xsd = svn:mime-type=text/xml;svn:eol-style=native
*.xsl = svn:mime-type=text/xml;svn:eol-style=native
*.yml = svn:mime-type=text/plain;svn:eol-style=native
.checkstyle = svn:mime-type=text/plain;svn:eol-style=native
.classpath = svn:mime-type=text/plain;svn:eol-style=native
.project = svn:mime-type=text/plain;svn:eol-style=native
Makefile = svn:eol-style=native

The above list is not definitive, so add any file types common to your development process.

Fixing existing autoprops

The following script is designed to run in a bash shell and requires ruby. It reads the subversion config file from ~/.subversion/config and applies the rules to files already added to subversion in the current directory and its children - effectively applying autoprops to the file as if it were a new addition.

In particular, note that:

the modification will be to the working copy and will therefore need to be committed as required
the use of dos2unix and unix2dos gives EOL consistency. Subversion may complain if you try to change the eol-style on a file with inconsistent line endings.
this can generate a lot of changes and cause some major conflicts if people have uncommitted local modifications. It is advised to do this during a quiet period.

#!/usr/bin/env ruby
require 'optparse'
require 'fileutils'

options = { :dryrun => false }
opts = OptionParser.new do |opts|
  opts.banner = "fix_svn_autoprops"

  opts.on("-n", "--dry-run", "Show what would happen but don't actually do anything") do |tf|
    options[:dryrun] = tf
  end

  opts.on("--fix-eols-windows", "Fix eol for windows if eol-style property is set") do |tf|
    options[:fixeols] = :windows
  end

  opts.on("--fix-eols-linux", "Fix eol for linux (and mac) if eol-style property is set") do |tf|
    options[:fixeols] = :linux
  end

  opts.on_tail("-h", "--help", "Show this message") do
    puts opts
    exit
  end
end
opts.parse!

autoprops = Hash.new {|h,k| h[k] = {}}
File.open(File.expand_path("~/.subversion/config"), "r") do |file|
  section = nil
  while line = file.gets
    line.chomp!.strip!
    next if line =~ /^$/
    next if line =~ /^#.*$/
    if line =~ /^\[[\w-]*\]$/
      section = line
      next
    end
    next unless section == "[auto-props]"
    # parse auto props line
    m, filename_pattern, props = *line.match(/([^\s=]+)\s*=\s*(.+)$/)
    props.split(';').each do |prop|
      propname, value = prop.split('=')
      autoprops[filename_pattern][propname] = value
    end
  end
end

autoprops.each do |filename_pattern, props|
  puts "Fixing #{filename_pattern} with #{props.inspect}"
  command = %Q{find . -name .svn -prune -o -name target -prune -o -name test-output -prune -o -name "#{filename_pattern}" -print0}
  props.each do |propname, value|
    if propname == 'svn:eol-style'
      case options[:fixeols]
      when :windows
        command << %Q{ -exec dos2unix '{}' \\;}
        command << %Q{ -exec unix2dos '{}' \\;}
      when :linux
        command << %Q{ -exec unix2dos '{}' \\;}
        command << %Q{ -exec dos2unix '{}' \\;}
      end
    end
    command << %Q{ -exec svn ps "#{propname}" "#{value}" '{}' \\;}
  end
  puts command
  system command unless options[:dryrun]
end

Ruby Rack App Hosting with RVM, Passenger 3, Capistrano, Git and Apache

2010-11-15T00:00:00+00:00

This guide supersedes a number of earlier guides, providing the details to get up and running with a rack based application from a basic ubuntu server VPS. The notable differences are using passenger 3 and system wide rvm.

At the time of writing, the latest versions of the software in use are:

RVM: 1.0.21, system wide
Ruby: 1.9.2-p0
Passenger: 3.0.0
Rails: 3.0.2
Capistrano: 2.5.19

RVM System Wide Install

rvm is installed system wide as this is now the best practice for deployments. This allows installations to be shared across different users, although different users can use which ever version they wish.

bash < <( curl -L http://bit.ly/rvm-install-system-wide )

In order for a user to use the system wide rvm installation the following must be setup:

User must belong to the rvm group - system wide installation requires the correct permissions
User must use the rvm load script in profile files - the location of the load script is different to user specific installs
User must not have ~/.rvm - a user specific install overrides a system wide one

Create Passenger User

sudo adduser passenger
sudo usermod -G passenger,www-data,sudo,rvm passenger
su - passenger

You can remove the sudo right after completing the installation using the following:

sudo usermod -G passenger,www-data,rvm passenger

Generate a key pair for ssh too. On your client machine, create a key and use a string pass phrase:

ssh-keygen -v -t rsa -f passenger@domainname -C passenger@domainname

This will create a private key called passenger@domainname and a public key called passenger@domainname.pub. Keep these somewhere safe.

You can register the public key into authorized_keys to allow key based ssh authentication. Run the following on your client machine:

scp passenger@domainname.pub passenger@hostname:~/.ssh/authorized_keys

ssh-add passenger@domainname

You should now be able to ssh to the server without being challenged for the password, e.g:

ssh passenger@hostname

Passenger RVM Configuration

Simply edit .bashrc and add the following line at the bottom:

[[ -s /usr/local/lib/rvm ]] && source /usr/local/lib/rvm

Note: if you previously had rvm installed as a single user you need to remove the old source line for $HOME/.rvm/scripts/rvm and move (or delete) the .rvm directory so that the system wide installation is picked up instead.

Log out and log in again as passenger to ensure the scripts work as expected. You can test rvm is available correctly by running:

type rvm | head -n1

You should then see the following output:

rvm is a function

You can view the notes about requirements for installing the different rubies by running rvm notes. The list of packages at the time of writing had dependency conflicts which can be circumvented by removing libreadline5-dev.

To install the packages (requires sudo):

sudo apt-get install openssl curl git-core subversion autoconf \
                     build-essential bison zlib1g zlib1g-dev \
                     libssl-dev libxml2-dev libreadline5 libreadline-dev \
                     libreadline6-dev libsqlite3-0 libsqlite3-dev sqlite3

To install ruby 1.9.2:

rvm install 1.9.2
rvm use --default 1.9.2

You can can now verify the ruby version and install some gems, e.g.:

ruby -v
gem install bundler rails

Passenger

Passenger is very easy to install. rvm provides some instructions specifically for passenger.

Install the gem and run the installation script. It will tell you what dependencies you are missing and how to install them:

gem install passenger
passenger-install-apache2-module

To enable passenger, create /etc/apache2/mods-available/passenger.load with the following contents:

LoadModule passenger_module /usr/local/rvm/gems/ruby-1.9.2-p0/gems/passenger-3.0.0/ext/apache2/mod_passenger.so

and /etc/apache2/mods-available/passenger.conf with the following contents:

PassengerRoot /usr/local/rvm/gems/ruby-1.9.2-p0/gems/passenger-3.0.0
PassengerRuby /usr/local/rvm/wrappers/ruby-1.9.2-p0/ruby

then enable the module and restart apache:

sudo a2enmod passenger
sudo /etc/init.d/apache2 restart

Apache

I prefer to put all web applications in /var/www and have users be able to create applications if they are a member of www-data:

sudo mkdir /var/www
sudo chown www-data:www-data /var/www
sudo chmod g+w /var/www

Hello World rack app

Create the necessary directory structure for a rack application:

cd /var/www
mkdir hw.domainname
cd hw.domainname
mkdir public
mkdir tmp

Create /var/www/hw.domainname/config.ru with the following contents:

app = proc do |env|
  [200, { "Content-Type" => "text/html" }, ["hello, world"]]
end
run app

Create /etc/apache2/sites-available/hw.domainname with the following contents:

<VirtualHost *:80>
    ServerName hw.domainname
    DocumentRoot /var/www/hw.domainname/public
    <Directory /var/www/hw.domainname/public>
        AllowOverride all
        Options -MultiViews
    </Directory>
</VirtualHost>

Enable the site, and then view it in your browser:

sudo a2ensite hw.domainname
sudo /etc/init.d/apache2 reload

This can then be disabled and reenabled if needed for debugging any issues at a later date:

sudo a2dissite hw.domainname
sudo /etc/init.d/apache2 reload

Rails app

You now need to create a virtual host for the rails app so that apache can handle the requests and pass off to passenger.

Create the necessary directory structure for a rails application (assuming its running as the domainname):

cd /var/www
mkdir domainname

Create /etc/apache2/sites-available/domainname with the following contents:

<VirtualHost *:80>
    ServerName domainname
    DocumentRoot /var/www/domainname/current/public
    <Directory /var/www/domainname/current/public>
        AllowOverride all
        Options -MultiViews
    </Directory>
</VirtualHost>

Enable the site, although there is nothing there to view until we have deployed it:

sudo a2ensite domainname
sudo /etc/init.d/apache2 reload

Capistrano

The following steps should be performed on your client machine to capify an existing rails application and deploy it.

Configuration

Make sure you have capistrano installed:

gem install capistrano

Initialise the rails application:

cd ~/dev/appname
capify .

This will generate some files, along with a sample config/deploy.rb. Replace the file contents with the following, using suitable values where applicable.

# RVM bootstrap
$:.unshift(File.expand_path("~/.rvm/lib"))
require 'rvm/capistrano'
set :rvm_ruby_string, '1.9.2-p0'
set :rvm_type, :system

# bundler bootstrap
require 'bundler/capistrano'

# main details
set :application, "domainname"
role :web, "domainname"
role :app, "domainname"
role :db,  "domainname", :primary => true

# server details
default_run_options[:pty] = true
ssh_options[:forward_agent] = true
set :deploy_to, "/var/www/domainname"
set :deploy_via, :remote_cache
set :user, "passenger"
set :use_sudo, false

# repo details
set :scm, :git
set :scm_username, "passenger"
set :repository, "git@gitserver:domainname.git"
set :branch, "master"
set :git_enable_submodules, 1

# tasks
namespace :deploy do
  task :start, :roles => :app do
    run "touch #{current_path}/tmp/restart.txt"
  end

  task :stop, :roles => :app do
    # Do nothing.
  end

  desc "Restart Application"
  task :restart, :roles => :app do
    run "touch #{current_path}/tmp/restart.txt"
  end

  desc "Symlink shared resources on each release - not used"
  task :symlink_shared, :roles => :app do
    #run "ln -nfs #{shared_path}/config/database.yml #{release_path}/config/database.yml"
  end
end

after 'deploy:update_code', 'deploy:symlink_shared'

Note: the above file supplies the rvm ruby version as 1.9.2-p0. It is advisable to be specific about the version here as if this were 1.9.2 and a newer ruby version is available you may start to receive errors in the capistrano deployment.

Deployment

The first time you deploy to the server you should run the deployment setup task:

cap deploy:setup

Thereafter you can deploy using:

cap deploy

Or if you have database migrations to run then use:

cap deploy:migrations

You may get some strange errors or failures when deploying. If you have followed the steps I have mentioned in this and previous articles then you shouldn’t have many problems. Common problems are:

wrong permissions of /var/www
wrong permissions of passenger user
not having rvm installed for passenger user
not having the basic gems required to use capistrano on the server, simply install them as the passenger user

Testing

View your website in your browser and ensure it is loading.

Subversion Master With Git Mirrors

2010-10-01T00:00:00+01:00

Subversion has proved itself to be a strong centralised version control system, establishing itself as the major corporate version control system. But the times are changing, people are collaborating with others around the world, often working offline while travelling, wanting something with a little more features locally or just something more performant. Distributed version control systems (DVCS) address all of these issues and have been proven by the vast numbers of open source developers using them to develop critical software. Dropping subversion and adopting a DVCS such as git is not an easy step for most corporate organisations, the tool chain is slightly different and there needs to be a change in approach to attain the increase in productivity that can be achieved.

This article looks at not replacing subversion but continuing to use it as the master source repository and use one or more git mirrors to provide git for those that are familiar or wish to try it. Having subversion as the master repository can also alleviate management worries around authoritative sources, learning curves and any lost productive during upskilling, and other barriers to change. Those users using git can interact with both the subversion master and git mirrors, committing any work to be shared with all users back to subversion as and when appropriate. To reduce any issues, commits should always be pushed back to the subversion repository, creating a circular flow of changes.

Note the use of the fetch repository. This is the recommended approach to creating a mirror as git svn clone will create a local working area which we do not want if hosting as a mirror. Therefore we have to use two repositories, a standard git bare repository for the mirror and a git svn “fetch” repository for updating the mirror.

This guide makes the following assumptions:

master subversion repository is available at http://servername/svn/reponame/ with standard conventions for trunk, tags and branches
you are able to set up access to a remote git repository, ideally using something like gitosis or gitolite or even github
you are able to create cron jobs on the server hosting the fetch repository
svn, git and git-svn are installed and available on the command line
optional: gitk is installed and available on the command line

Updates 2010-10-06:

Fixed a number of mirror issues with the approach using master. Now uses trunk and HEAD is set to refs/heads/trunk using symbolic ref.
sync script is now a standalone script allowing manual and automated (via cron) updates.

Creating a git mirror

A git mirror can be created on the git repository server of your choice.

Note: It is important to set HEAD to refs/heads/trunk. github allows this through the repository admin, and you can do this manually with gitosis or gitolite by running the following command on a bare repository:

GIT_DIR=/path/to/reponame.git git symbolic-ref HEAD refs/heads/trunk

For example, using gitolite, clone the admin repo:

git clone git@hostname:gitolite-admin

Modify the conf/gitolite.conf file to also contain:

@mirrors    = reponame

repo        @mirrors
            R       =   @all
            RW+     =   @admins

Add and commit the changes:

git add -A
git ci -m "added reponame mirror"
git push

Now ssh to the server and run the following:

GIT_DIR=/home/git/repositories/reponame.git git symbolic-ref HEAD refs/heads/trunk

The rest of the article assumes that the git mirror is available at git@hostname:reponame.git.

Setting up a fetch repo for the mirror

git svn clone will create a clone of a subversion repository including the entire history. Unfortunately, for repositories with a lot of history, this can take a long time to run as git-svn has to replay each revision during the clone.

Ideally, the fetch repo will be setup on a server so that it cron can be used to provide regular updates. If you are hosting your own mirror then it is a good idea to also put the fetch repository on the same server.

To create the fetch repo on the server as the git user from a particular revision (e.g. 12345), run the following:

ssh username@hostname
su - git
mkdir -p ~/fetchers
cd ~/fetchers
git svn clone -s --no-minimize-url -r12345:HEAD http://servername/svn/reponame/ reponame-fetch
cd reponame-fetch
git branch -m master trunk
git remote add mirror git@hostname:reponame.git
git config --unset remote.mirror.fetch
git config remote.mirror.push 'refs/remotes/*:refs/heads/*'
git push mirror

Create a script to sync the repository. Edit ~/bin/sync-reponame and add the following:

cd ~/fetchers/reponame-fetch
echo Syncing at `date`
git svn rebase
git push mirror
echo

Ensure the script can be run:

chmod +x ~/bin/sync-reponame

Create a suitable crontab by running crontab -e and entering the following:

*/5 * * * * ~/bin/sync-reponame >> ~/sync-reponame.log 2>&1

This will run a sync script every 5 minutes as the git user, appending the output to a log file. Change the timings as appropriate for your setup.

Git clone

Each user can create a clone using the git mirror and create the necessary svn references:

git clone -o mirror git@hostname:reponame.git
cd reponame
git svn init --prefix=mirror/ -s http://servername/svn/reponame/
git svn rebase

A user can keep up to date by pulling from the git mirror:

git pull --rebase

Or they can update directly from subversion:

git svn rebase

The git mirror may lag behind subversion depending on the timings of any cron jobs to sync repos, although this is not an issue usually. A user may wish to update from svn if they are waiting on a change that they need quickly for example. Also, in order to commit to subversion, a user should have the latest version so it is usual to update from subversion just prior to the subversion commit.

A user can now perform the usual workflows with git, committing any changes and when happy using git svn dcommit to commit the changes to the subversion master. For example [comments in square brackets]:

git pull --rebase               [ensure up to date with mirror]
....                            [modify a file]
git add -A                      [add any changes in files to the index]
git ci -m "updated a file"      [commit the changes in the index]
git svn rebase                  [ensure up to date with svn]
git svn dcommit                 [push changes back to subversion]

Working with branches

When adding features it is good practice to use a feature branch. The master branch can be kept up to date with subversion using git svn rebase and then the feature branch can be rebased onto master as and when you want to align with the current stream of development. When you are ready to push back to the subversion master repository, rebase the feature branch from the master branch and merge the branch using a fast forward merge onto the master branch. You can then dcommit the changes as normal. A sample workflow would be:

git checkout -b feature         [create a feature branch and checkout]
....                            [modify a file]
git add -A                      [add any changes in files to the index]
git ci -m "updated a file"      [commit the changes in the index]

occasionally rebase from mirror and rebase branch from master:

git checkout master             [switch back to the master branch]
git pull --rebase               [ensure up to date with mirror]
git checkout feature            [switch back to the feature branch]
git rebase master               [replay branch changes on top of master and fix any problems]
....                            [do some work, until you are ready to commit back to svn]
git checkout master             [switch back to the master branch]
git svn rebase                  [ensure up to date with svn]
git checkout feature            [switch back to the master branch]
git rebase master               [replay branch changes on top of master and fix any problems]
git checkout master             [switch back to the master branch]
git merge feature --ff-only     [merge feature branch ensuring fast forward merge]
git svn dcommit                 [push changes back to subversion]

It is important to be careful with merges due to the way git svn works, see CAVEATS from git svn man page.

Useful aliases

Git allows aliases to be created to reduce the amount you have to type for common commands and their options. Run the following:

git config --global alias.svnsync "pull --rebase"
git config --global alias.svnup "svn rebase"
git config --global alias.svnci "svn dcommit --rmdir"

You can then run the following commands, with an explanation of what they do:

git svnsync will do the equivalent of svn up using the git mirror
git svnup will do the equivalent of svn up using the svn server
git svnci will do the equivalent of svn ci using the svn server

Other tools

gitk (and gitx, its more beautiful counterpart for OS X) can be used to view the git repository. Its is a useful gui tool for reviewing commits. Alternatively, git log and friends will give you details on the command line.

Apache follows a similar git svn workflow.
Why you should use git pull –rebase as opposed to the default of merge.
Other articles about setting up git mirrors of svn repositories.

Git Repository Server using Gitolite

2010-09-30T00:00:00+01:00

Some time ago, I covered setting up a git repository server using gitosis. Since then, gitolite has entered the arena and has built on gitosis adding features such as branch level access control, gitweb control and improved configuration. This guide covers installing gitolite using the non-root method and simple administration.

The documentation for gitolite is pretty comprehensive and I encourage you to read the official install and admin instructions. In addition to the official instructions, I also setup and authorise the git user as this allows any server sync repositories to use the git user as a system account and be able to interact with git repositories hosted in gitolite. This is useful for fetch repositories for svn mirrors for example (the topic of a future post).

The following conventions are used and should be substituted with appropriate values for your environment:

username is the username of the person installing gitolite (i.e. you) and is assumed to have an account on the server
hostname is the hostname of the git server - add host entries as appropriate (e.g. 172.18.25.102 hostname)
git is the username of the user on the git server that will run gitolite and will be created
reponame is the name of a repository you want to host

This post assumes that you have a server following the Ubuntu Lucid VPS Base Build post.

Git Packages

The git-core package contains all the basic command line for git. As the server is running headless, there is no point in installing packages such as gitk.

sudo apt-get install git-core

ssh keys

If you already have ssh keys then use those, otherwise create some, e.g.:

cd ~
ssh-keygen -v -t rsa -C username@hostname

Gitolite

Installation

Copy your public key to the server:

scp ~/.ssh/id_rsa.pub username@hostname:username.pub

SSH to the server:

ssh username@hostname

Create the git user that gitolite will run as (i.e. git):

sudo adduser git

Copy the public key over to the git user:

sudo chown git:git username.pub
sudo mv username.pub ~git/username.pub

su - git

Generate ssh keys for git user with no password and copy the public key to your clipboard for later:

ssh-keygen -v -t rsa
cat ~/.ssh/id_rsa.pub

Install gitolite:

cd ~
git clone git://github.com/sitaramc/gitolite gitolite-source
cd gitolite-source
mkdir -p ~/bin ~/share/gitolite/conf ~/share/gitolite/hooks
src/gl-system-install ~/bin ~/share/gitolite/conf ~/share/gitolite/hooks
cd ..
gl-setup username.pub
rm -f username.pub

Administration

You can administer the configuration through git itself on the gitolite-admin repository by cloning, modifying and push changes back to the server.

On your client machine, clone the admin repo:

git clone git@hostname:gitolite-admin

Modify the conf/gitolite.conf file to contain:

@admins     = username git

repo        gitolite-admin
            RW+     =   @admins

repo        testing
            RW+     =   @all

And ensure that the keydir directory contains the following files:

git.pub
username.pub

Add and commit the changes:

git add -A
git ci -m "updated configuration"
git push

Adding repositories

As above, clone the admin repo and append the following to conf/gitolite.conf:

repo        reponame
            RW+     =   username

Add and commit the changes:

git add -A
git ci -m "updated configuration"
git push

You can now add create the project (or skip this for an existing project):

mkdir myproject
cd mypyroject
git init
# do some work, git add and commit files

Then add your server as a remote and push:

git remote add hostname git@hostname:reponame.git
git push hostname master

You can track the remote by adding suitable config:

git config branch.master.remote hostname
git config branch.master.merge refs/heads/master

Rails 3 Hosting with RVM, Passenger, Capistrano, Git and Apache

2010-08-30T00:00:00+01:00

Now that Rails 3 and Ruby 1.9.2 are released, I have compiled some of my earlier posts to create this all in one guide to installing and running Rails 3, RVM, Ruby 1.9.2, Passenger and Capistrano for hosting your rack/rails applications. It assumes that you are using apache and git.

At the time of writing, the latest versions of the software in use are:

RVM: 1.0.1
Ruby: 1.9.2-p0
Passenger: 2.2.15
Rails: 3.0.0
Capistrano: 2.5.19

Updates:

2010-09-11: Fixed path to public folder in apache virtual host to include current if deployed using capistrano
2010-09-11: Added note about using cap deploy:migrations if you have db migrations to run

Passenger User

It is a good idea to create a passenger user on the server to install rvm for so that other users do not modify it by accident. You can also deploy your applications using this user. The passenger user needs sudo rights for installing packages using apt-get and running the passenger install script.

sudo adduser passenger
sudo usermod -G passenger,www-data,sudo passenger
su - passenger

You can remove the sudo right after completing the installation using the following:

sudo usermod -G passenger,www-data passenger

Generate a key pair for ssh too. On your client machine, create a key and use a string pass phrase:

ssh-keygen -v -t rsa -f passenger@domainname -C passenger@domainname

This will create a private key called passenger@domainname and a public key called passenger@domainname.pub. Keep these somewhere safe.

You can register the public key into authorized_keys to allow key based ssh authentication. Run the following on your client machine:

scp passenger@domainname.pub passenger@hostname:~/.ssh/authorized_keys

ssh-add passenger@domainname

You should now be able to ssh to the server without being challenged for the password, e.g:

ssh passenger@hostname

RVM

rvm is installed locally for each user rather than a single install for the server. This allows different versions to be used by different users on the server. Unfortunately, only one ruby version can be used with passenger at the moment due to the way passenger hooks into apache.

Install using the preferred script:

bash < <( curl http://rvm.beginrescueend.com/releases/rvm-install-head )

There are some warnings about issues with return in .bashrc. This appears to be a red herring for Ubuntu as mentioned in this post.

Simply edit .bashrc and add the following line at the bottom:

[[ -s $HOME/.rvm/scripts/rvm ]] && source $HOME/.rvm/scripts/rvm

Log out and log in again as passenger to ensure the scripts work as expected. You can test rvm is available correctly by running:

type rvm | head -n1

You should then see the following output:

rvm is a function

To install the packages (requires sudo):

sudo apt-get install openssl curl git-core subversion autoconf \
                     build-essential bison zlib1g zlib1g-dev \
                     libssl-dev libxml2-dev libreadline5 libreadline-dev \
                     libreadline6-dev libsqlite3-0 libsqlite3-dev sqlite3

To install the rubies, first 1.8.7 as the notes indicate, followed by 1.9.2:

rvm install 1.8.7
rvm use 1.8.7
rvm install 1.9.2
rvm use --default 1.9.2

You can can now verify the ruby version and install some gems, e.g.:

ruby -v
gem install bundler rails

Passenger

Passenger is very easy to install. rvm provides some instructions specifically for passenger.

Tell rvm the desired version of ruby to use:

rvm 1.9.2 --passenger

Install the gem and run the installation script. It will tell you what dependencies you are missing and how to install them:

gem install passenger
rvmsudo /home/passenger/.rvm/gems/ruby-1.9.2-p0/bin/passenger-install-apache2-module

I had to run the following to fix the missing dependencies:

sudo apt-get install apache2-prefork-dev libapr1-dev libaprutil1-dev

Passenger provides some instructions on how to configure apache, although, due to the way rvm works, the value of PassengerRuby should be different. I like to treat passenger as a mod for apache, allowing it to be enabled and disabled as required through the a2enmod and a2dismod commands respectively.

To enable passenger, create /etc/apache2/mods-available/passenger.load with the following contents:

LoadModule passenger_module /home/passenger/.rvm/gems/ruby-1.9.2-p0/gems/passenger-2.2.15/ext/apache2/mod_passenger.so

and /etc/apache2/mods-available/passenger.conf with the following contents:

PassengerRoot /home/passenger/.rvm/gems/ruby-1.9.2-p0/gems/passenger-2.2.15
PassengerRuby /home/passenger/.rvm/bin/passenger_ruby

then enable the module and restart apache:

sudo a2enmod passenger
sudo /etc/init.d/apache2 restart

Apache

I prefer to put all web applications in /var/www and have users be able to create applications if they are a member of www-data:

sudo mkdir /var/www
sudo chown www-data:www-data /var/www
sudo chmod g+w /var/www

Hello World rack app

Create the necessary directory structure for a rack application:

cd /var/www
mkdir hw.domainname
cd hw.domainname
mkdir public
mkdir tmp

Create /var/www/hw.domainname/config.ru with the following contents:

app = proc do |env|
  [200, { "Content-Type" => "text/html" }, ["hello, world"]]
end
run app

Create /etc/apache2/sites-available/hw.domainname with the following contents:

<VirtualHost *:80>
    ServerName hw.domainname
    DocumentRoot /var/www/hw.domainname/public
    <Directory /var/www/hw.domainname/public>
        AllowOverride all
        Options -MultiViews
    </Directory>
</VirtualHost>

Enable the site, and then view it in your browser:

sudo a2ensite hw.domainname
sudo /etc/init.d/apache2 reload

This can then be disabled and reenabled if needed for debugging any issues at a later date:

sudo a2dissite hw.domainname
sudo /etc/init.d/apache2 reload

Rails app

You now need to create a virtual host for the rails app so that apache can handle the requests and pass off to passenger.

Create the necessary directory structure for a rails application (assuming its running as the domainname):

cd /var/www
mkdir domainname

Create /etc/apache2/sites-available/domainname with the following contents:

<VirtualHost *:80>
    ServerName domainname
    DocumentRoot /var/www/domainname/current/public
    <Directory /var/www/domainname/current/public>
        AllowOverride all
        Options -MultiViews
    </Directory>
</VirtualHost>

Enable the site, although there is nothing there to view until we have deployed it:

sudo a2ensite domainname
sudo /etc/init.d/apache2 reload

Capistrano

The following steps should be performed on your client machine to capify an existing rails application and deploy it.

Configuration

Make sure you have capistrano installed:

gem install capistrano

Initialise the rails application:

cd ~/dev/appname
capify .

This will generate some files, along with a sample config/deploy.rb. Replace the file contents with the following, using suitable values where applicable.

# RVM bootstrap
$:.unshift(File.expand_path("~/.rvm/lib"))
require 'rvm/capistrano'
set :rvm_ruby_string, '1.9.2-p0'
set :rvm_type, :user

# bundler bootstrap
require 'bundler/capistrano'

# main details
set :application, "domainname"
role :web, "domainname"
role :app, "domainname"
role :db,  "domainname", :primary => true

# server details
default_run_options[:pty] = true
ssh_options[:forward_agent] = true
set :deploy_to, "/var/www/domainname"
set :deploy_via, :remote_cache
set :user, "passenger"
set :use_sudo, false

# repo details
set :scm, :git
set :scm_username, "passenger"
set :repository, "git@gitserver:domainname.git"
set :branch, "master"
set :git_enable_submodules, 1

# tasks
namespace :deploy do
  task :start, :roles => :app do
    run "touch #{current_path}/tmp/restart.txt"
  end

  task :stop, :roles => :app do
    # Do nothing.
  end

  desc "Restart Application"
  task :restart, :roles => :app do
    run "touch #{current_path}/tmp/restart.txt"
  end

  desc "Symlink shared resources on each release - not used"
  task :symlink_shared, :roles => :app do
    #run "ln -nfs #{shared_path}/config/database.yml #{release_path}/config/database.yml"
  end
end

after 'deploy:update_code', 'deploy:symlink_shared'

Deployment

The first time you deploy to the server you should run the deployment setup task:

cap deploy:setup

Thereafter you can deploy using:

cap deploy

Or if you have database migrations to run then use:

cap deploy:migrations

You may get some strange errors or failures when deploying. If you have followed the steps I have mentioned in this and previous articles then you shouldn’t have many problems. Common problems are:

wrong permissions of /var/www
wrong permissions of passenger user
not having rvm installed for passenger user
not having the basic gems required to use capistrano on the server, simply install them as the passenger user

Testing

View your domain in your browser and ensure it is loading.

Sendmail on domain host

2010-08-29T00:00:00+01:00

If you followed my guide for Ubuntu Lucid VPS Base Build then /etc/hosts will have the following DNS entries and aliases:

127.0.0.1 localhost.localdomain localhost
127.0.1.1 servername.domainname alias.domainname

This is ok as the servername and alias are subdomains. Any email sent to user@domainname will be delivered as expected by SMTP using the MX entry for the domain.

However, it is possible to have an alias in the hosts file for the domain itself, e.g.:

127.0.0.1 localhost.localdomain localhost
127.0.1.1 servername.domainname alias.domainname domainname

A consequence of this is that email to user@domainname from that server will be delivered locally instead.

Of course, this may be the desired behaviour, in which case it would warrant a more complicated setup. For example; in a single server environment, instead of using aliases, a mail delivery agent (MDA) could deliver the mail to a mailbox that users can interact with accordingly. Multi-server environments may have a mail server, in which case the other servers may have sendmail configured to use it as a smart host, e.g. the following may be added to /etc/mail/sendmail.mc:

define(`SMART_HOST', `mail.domainname')

Getting back to a simple setup where email should be relayed, e.g. where email is provided by an external party such as gmail, the following lines can be added to the end of /etc/mail/sendmail.mc to effectively disable local delivery:

define(`MAIL_HUB', `domainname.')dnl
define(`LOCAL_RELAY', `domainname.')dnl

Note the dot after the domainname. This allows the local routing to be circumvented. You can also make the email addresses a little nicer by adding the following lines before the MAILER_DEFINITIONS section:

MASQUERADE_AS(`domainname')
FEATURE(`allmasquerade')

When making modifications to /etc/mail/sendmail.mc, be sure to run sendmailconfig afterwards to update the compiled configuration and restart sendmail to use the new configuration.

sshd and cron on Windows Using Cygwin

2010-08-27T00:00:00+01:00

For those working on windows who miss their linux command line friends, there is hope. Cygwin is a great tool to allow you to use your favourite linux tools on your windows machine. Installing and running cygwin is very simple, this guide presumes that you are familiar with linux and running cygwin shells. This guide focuses on running linux daemons for ssh and cron. The following instructions work on Windows 7 so your milage may vary on different versions.

Note: on Windows Vista or Windows 7, you need to run as administrator when launching the cygwin shell to allow the registry changes required in some of the steps below. You can configure cygwin to always launch with admin privileges if you wish.

Prerequisites

The following packages are required:

cygrunsrv
openssh
cron
nano or vim or another editor

Ensure that the $EDITOR environment property is set for your preferred editor, ideally in one of the profile files (e.g. .bashrc or .bash_profile).

Windows Security

As outlined in windows security in cygwin, there are 3 methods to allow the user context to switch without passwords. The most appropriate choice is to use the LSA authentication package by running the cyglsa-config script. Simply run it and answer the questions posed. The following is a typical log output:

$ cyglsa-config

Warning: Registering the Cygwin LSA authentication package requires
administrator privileges!  You also have to reboot the machine to
activate the change.

Are you sure you want to continue? (yes/no) yes

Cygwin LSA authentication package registered.

Activating Cygwin's LSA authentication package requires to reboot.
Do you want to do this immediately? (yes/no) no

Now reboot when its safe.

Alternatively, passwd -R can be used to install a given users password into the registry but this may pose a security risk in some environments.

OpenSSH

Cygwin takes all the pain out of setting up sshd by providing a setup script called ssh-host-config. Simply run it and answer the questions posed. The following is a typical log output:

$ ssh-host-config
*** Query: Overwrite existing /etc/ssh_config file? (yes/no) yes
*** Info: Creating default /etc/ssh_config file
*** Query: Overwrite existing /etc/sshd_config file? (yes/no) yes
*** Info: Creating default /etc/sshd_config file
*** Info: Privilege separation is set to yes by default since OpenSSH 3.3.
*** Info: However, this requires a non-privileged account called 'sshd'.
*** Info: For more info on privilege separation read /usr/share/doc/openssh/README.privsep.
*** Query: Should privilege separation be used? (yes/no) yes
*** Info: Note that creating a new user requires that the current account have
*** Info: Administrator privileges.  Should this script attempt to create a
*** Query: new local account 'sshd'? (yes/no) yes
*** Info: Updating /etc/sshd_config file

*** Warning: The following functions require administrator privileges!

*** Query: Do you want to install sshd as a service?
*** Query: (Say "no" if it is already installed as a service) (yes/no) yes
*** Query: Enter the value of CYGWIN for the daemon: [] ntsec tty
*** Info: On Windows Server 2003, Windows Vista, and above, the
*** Info: SYSTEM account cannot setuid to other users -- a capability
*** Info: sshd requires.  You need to have or to create a privileged
*** Info: account.  This script will help you do so.

*** Info: You appear to be running Windows 2003 Server or later.  On 2003
*** Info: and later systems, it's not possible to use the LocalSystem
*** Info: account for services that can change the user id without an
*** Info: explicit password (such as passwordless logins [e.g. public key
*** Info: authentication] via sshd).

*** Info: If you want to enable that functionality, it's required to create
*** Info: a new account with special privileges (unless a similar account
*** Info: already exists). This account is then used to run these special
*** Info: servers.

*** Info: Note that creating a new user requires that the current account
*** Info: have Administrator privileges itself.

*** Info: No privileged account could be found.

*** Info: This script plans to use 'cyg_server'.
*** Info: 'cyg_server' will only be used by registered services.
*** Query: Do you want to use a different name? (yes/no) no
*** Query: Create new privileged user account 'cyg_server'? (yes/no) yes
*** Info: Please enter a password for new user cyg_server.  Please be sure
*** Info: that this password matches the password rules given on your system.
*** Info: Entering no password will exit the configuration.
*** Query: Please enter the password:
*** Query: Reenter:

*** Info: User 'cyg_server' has been created with password 'xxxxxxxx'.
*** Info: If you change the password, please remember also to change the
*** Info: password for the installed services which use (or will soon use)
*** Info: the 'cyg_server' account.

*** Info: Also keep in mind that the user 'cyg_server' needs read permissions
*** Info: on all users' relevant files for the services running as 'cyg_server'.
*** Info: In particular, for the sshd server all users' .ssh/authorized_keys
*** Info: files must have appropriate permissions to allow public key
*** Info: authentication. (Re-)running ssh-user-config for each user will set
*** Info: these permissions corrently. [Similary restrictions apply, for
*** Info: instance, for .rhosts files if the rshd server is running, etc].

*** Info: The sshd service has been installed under the 'cyg_server'
*** Info: account.  To start the service now, call `net start sshd' or
*** Info: `cygrunsrv -S sshd'.  Otherwise, it will start automatically
*** Info: after the next reboot.

*** Info: Host configuration finished. Have fun!

As the instructions state, the sshd service can be started using either net start sshd, cygrunsrv -S sshd or even using the services management console. Test the connection by sshing to localhost. If you experience any problems try running in verbose mode, e.g:

ssh localhost
ssh username@localhost
ssh -vvv username@machinename

Windows likes to title case usernames and allows spaces, therefore make sure the username is correct if this is the case, e.g. ssh John\ Smith@localhost.

Cron

Once again, Cygwin provides a script to install cron called cron-config. Simply run it and answer the questions posed. Even if using passwd -R or cyglsa package, answer no to install the service under the cyg_server user. The following is a typical log output:

$ cron-config
Do you want to install the cron daemon as a service? (yes/no) yes
Enter the value of CYGWIN for the daemon: [ ] ntsec

You must decide under what account the cron daemon will run.
If you are the only user on this machine, the daemon can run as yourself.
   This gives access to all network drives but only allows you as user.
To run multiple users, cron must change user context without knowing
  the passwords. There are three methods to do that, as explained in
  http://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-nopasswd1
If all the cron users have executed "passwd -R" (see man passwd),
  which provides access to network drives, or if you are using the
  cyglsa package, then cron should run under the local system account.
Otherwise you need to have or to create a privileged account.
  This script will help you do so.
Do you want the cron daemon to run as yourself? (yes/no) no

Were the passwords of all cron users saved with "passwd -R", or
are you using the cyglsa package ? (yes/no) no

Attempting to find or create a privileged user.
The following accounts were found: 'cyg_server' .
This script plans to use the name cyg_server for the new user,
which will only be able to run as a service.
Do you want to use another name (not an interactive account)? (yes/no) no

Please enter the password for user 'cyg_server':
Reenter:

Running cron_diagnose ...
WARNING: Your computer does not appear to have a cron table for username.
Please generate a cron table for username using 'crontab -e'

... no problem found.

Do you want to start the cron daemon as a service now? (yes/no) yes
OK. The cron daemon is now running.

In case of problem, examine the log file for cron,
/var/log/cron.log, and the Windows event log (using /usr/bin/cronevents)
for information about the problem cron is having.

Examine also any cron.log file in the HOME directory
(or the file specified in MAILTO) and cron related files in /tmp.

If you cannot fix the problem, then report it to cygwin@cygwin.com.
Please run the script /usr/bin/cronbug and ATTACH its output
(the file cronbug.txt) to your e-mail.

WARNING: PATH may be set differently under cron than in interactive shells.
         Names such as "find" and "date" may refer to Windows programs.

I prefer to reinstall the cron service so that it is similarly named to the sshd service. The following commands will remove the service and recreate it as desired:

cygrunsrv -R cron
cygrunsrv -I cron -d "CYGWIN cron" -u cyg_server -p /usr/sbin/cron -a -n

Like any service, the cron service can be started using either net start cron, cygrunsrv -S cron or even using the services management console.

Crontab entries can be made for a given user by logging in as that user and running crontab -e. This will use the $EDITOR environment variable to determine which editor to use, and will also validate the file afterwards. Essentially, using crontab creates a file named after the user in /var/cron/tabs with the correct permissions and ownership.

For example, run crontab -e and enter the following then save:

PATH=/usr/bin:/usr/sbin:.
*/10 * * * * echo `date` > /home/username/test-crontab.txt

This will output the date every 10 minutes into the /home/username/test-crontab.txt file. Using the crontab command also informs cron to re-read the crontabs without having to restart the service. Note that it is a good idea to supply the PATH environment variable to ensure that the execution path is correct for any commands. HOME is set to the home of the user.

Cron also checks for files in the system directory /etc/cron.d. By default this does not exist in cygwin so run the following commands to set this up with the required permissions:

mkdir /etc/cron.d
chmod 754 /etc/cron.d

Additionally, any crontabs must be owned and only writable by SYSTEM. You can ensure that crontabs have the correct permissions and ownership by running the following (after creating the crontab):

chmod 644 /etc/cron.d/*
chown SYSTEM:SYSTEM /etc/cron.d/*

System crontabs differ slightly from normal crontabs in that the first argument after the time and date fields is the username of the user to execute the command. For example, create /etc/cron.d/example with the following contents:

PATH=/usr/bin:/usr/sbin:. HOME=/home/username */10 * * * * username echo date > /home/username/test-cron.d.txt

Other Considerations

Cron problems

If you are having problems then look at C:\tools\cygwin\usr\share\doc\Cygwin\cron-*.README

For example, you can install the service with some debug options for cron:

cygrunsrv -I cron -d "CYGWIN cron" -u cyg_server -p /usr/sbin/cron -a "-x sch,proc,pars,load,misc"

View the logs with something like:

cat /var/log/cron.log
cat /home/username/cron.log

There is lots of information in the man pages:

man crontab
man cron
man 5 crontab

Bash Prompt Here

This isn’t strictly part of this guide, but I think its so useful that I have to mention it. The chere package is a tool that creates a “bash prompt here” entry in windows explorer context menus. To install it simply run the following (with admin rights):

chere -i

Updates

Sometimes when updating cygwin there are modifications to major binaries that require services to be stopped and possibly kill any cygwin processes that are running to allow the update to overwrite the files. Simply restart the services after completion. If there are updates to the LSA then you will need a reboot also.

Registering users in passwd

Sometimes you need to register the user in passwd to allow ssh and cron to use the user. This can be acheived using mkpasswd.

Check if the user is already registered:

cat /etc/passwd

If the user is a domain account then run the following to add them:

mkpasswd -d domain_name -u domain_username >> /etc/passwd

If the user is a local account then run the following to add them:

mkpasswd -l -u local_username >> /etc/passwd

Rails 3, RVM, Passenger and Capistrano

2010-08-05T00:00:00+01:00

This post builds on my previous posts on Ubuntu Lucid VPS Base Build and RVM and Passenger to create a suitable capistrano config file for deploying rails 3 applications.

Assuming you have followed the previous posts, you should be able to run simple rack applications and have a passenger user setup with ssh access. You should also have a rails 3 application that you would like to deploy and ideally this should be using git for source control.

Capistrano configuration

Make sure you have capistrano install:

gem install capistrano

Initialise the rails application:

cd ~/dev/mywebsite
capify .

This will generate some files, along with a sample config/deploy.rb. Replace the file contents with the following, using suitable values where applicable.

# RVM bootstrap
$:.unshift(File.expand_path("~/.rvm/lib"))
require 'rvm/capistrano'
set :rvm_ruby_string, '1.9.2'
set :rvm_type, :user

# main details
set :application, "mywebsite"
role :web, "mywebsite.com"
role :app, "mywebsite.com"
role :db,  "mywebsite.com", :primary => true

# server details
default_run_options[:pty] = true
ssh_options[:forward_agent] = true
set :deploy_to, "/var/www/mywebsite"
set :deploy_via, :remote_cache
set :user, "passenger"
set :use_sudo, false

# repo details
set :scm, :git
set :scm_username, "passenger"
set :repository, "git@gitserver:mywebsite.git"
set :branch, "master"
set :git_enable_submodules, 1

# runtime dependencies
depend :remote, :gem, "bundler", ">=1.0.0.rc.2"

# tasks
namespace :deploy do
  task :start, :roles => :app do
    run "touch #{current_path}/tmp/restart.txt"
  end

  task :stop, :roles => :app do
    # Do nothing.
  end

  desc "Restart Application"
  task :restart, :roles => :app do
    run "touch #{current_path}/tmp/restart.txt"
  end

  desc "Symlink shared resources on each release"
  task :symlink_shared, :roles => :app do
    #run "ln -nfs #{shared_path}/config/database.yml #{release_path}/config/database.yml"
  end
end

after 'deploy:update_code', 'deploy:symlink_shared'

namespace :bundler do
  desc "Symlink bundled gems on each release"
  task :symlink_bundled_gems, :roles => :app do
    run "mkdir -p #{shared_path}/bundled_gems"
    run "ln -nfs #{shared_path}/bundled_gems #{release_path}/vendor/bundle"
  end

  desc "Install for production"
  task :install, :roles => :app do
    run "cd #{release_path} && bundle install --production"
  end

end

after 'deploy:update_code', 'bundler:symlink_bundled_gems'
after 'deploy:update_code', 'bundler:install'

Capistrano deployment

The first time you deploy to the server you should run the deployment setup task:

cap deploy:setup

Thereafter you can deploy using:

cap deploy

Or if you have database migrations to run then use:

cap deploy:migrations

You may get some strange errors or failures when deploying. If you have followed the steps I have mentioned in this and previous articles then you shouldn’t have many problems. Common problems are:

wrong permissions of /var/www
wrong permissions of passenger user
not having rvm installed for passenger users
not having the basic gems required to use capistrano on the server, simply install them as the passenger user

Apache configuration

You now need to create a virtual host for the website so that apache can handle the requests and pass off to passenger.

Create /etc/apache2/sites-available/mywebsite.com with the following contents:

<VirtualHost *:80>
    ServerName mywebsite.com
    DocumentRoot /var/www/mywebsite/public
    <Directory /var/www/mywebsite/public>
        AllowOverride all
        Options -MultiViews
    </Directory>
</VirtualHost>

Enable the site, and then view it in your browser:

sudo a2ensite mywebsite.com
sudo /etc/init.d/apache2 reload

Rack Application Hosting on Ubuntu Lucid using RVM and Passenger

2010-07-11T00:00:00+01:00

Thanks to Ruby Version Manager (rvm) and Phusion Passenger setting up a server to run rack (and therefore rails or sinatra) applications is very simple.

This post is an update to my earlier post that used Ruby Enterprise Edition directly rather than allowing a choice of ruby versions through rvm. It also assumes that you have a server following the Ubuntu Lucid VPS Base Build post.

With the impending release of Rails 3, this post will cover the installation of Ruby 1.9.2, although the steps are applicable for other rubies.

Updated 2010-08-01: Ruby 1.9.2 rc2 is now available so updated to use correct paths. Application config now sets up an example Hello World rack application.

Updated 2010-08-30: Rails 3, Ruby 1.9.2, Bundler 1.0 released - see rails 3 hosting all in one for updated instructions.

User

It is a good idea to create a passenger user to install rvm for so that other users do not modify it by accident. You can also deploy your applications using this user. The passenger user needs sudo rights for installing packages using apt-get and running the passenger install script.

sudo adduser passenger
sudo usermod -G passenger,www-data,sudo passenger
su - passenger

You can remove the sudo right after completing the installation using the following:

sudo usermod -G passenger,www-data passenger

Its also a good idea to generate a key pair for ssh too. The public key can be stored in ~passenger/.ssh/authorized_keys and if you use ssh agent forwarding you do not need to store the private key on the server. This is very useful if you use something like capistrano to handle your deployments.

RVM

Install using the preferred script:

bash < <( curl http://rvm.beginrescueend.com/releases/rvm-install-head )

There are some warnings about issues with return in .bashrc. This appears to be a red herring for Ubuntu as mentioned in this post.

Simply edit .bashrc and add the following line at the bottom:

[[ -s $HOME/.rvm/scripts/rvm ]] && source $HOME/.rvm/scripts/rvm

Log out and log in again as passenger to ensure the scripts work as expected. You can test rvm is available correctly by running:

type rvm | head -n1

You should then see the following output:

rvm is a function

To install the packages (requires sudo):

sudo apt-get install openssl curl git-core subversion autoconf \
                     build-essential bison zlib1g zlib1g-dev \
                     libssl-dev libxml2-dev libreadline5 libreadline-dev \
                     libreadline6-dev libsqlite3-0 libsqlite3-dev sqlite3

To install the rubies, first 1.8.7 as the notes indicate, followed by 1.9.2:

rvm install 1.8.7
rvm use 1.8.7
rvm install 1.9.2
rvm use --default 1.9.2

You can can now verify the ruby version and install some gems, e.g.:

ruby -v
gem install rails --pre

Passenger

Passenger is very easy to install. rvm provides some instructions specifically for passenger.

Tell rvm the desired version of ruby to use:

rvm 1.9.2 --passenger

Install the gem and run the installation script. It will tell you what dependencies you are missing and how to install them:

gem install passenger
rvmsudo /home/passenger/.rvm/gems/ruby-1.9.2-rc2/bin/passenger-install-apache2-module

I had to run the following to fix the missing dependencies:

sudo apt-get install apache2-prefork-dev libapr1-dev libaprutil1-dev

To enable passenger, create /etc/apache2/mods-available/passenger.load with the following contents:

LoadModule passenger_module /home/passenger/.rvm/gems/ruby-1.9.2-rc2/gems/passenger-2.2.15/ext/apache2/mod_passenger.so

and /etc/apache2/mods-available/passenger.conf with the following contents:

PassengerRoot /home/passenger/.rvm/gems/ruby-1.9.2-rc2/gems/passenger-2.2.15
PassengerRuby /home/passenger/.rvm/bin/passenger_ruby

then enable the module and restart apache:

sudo a2enmod passenger
sudo /etc/init.d/apache2 restart

Application config

Each rack/rails application is made available through a virtual host directive in the apache configuration. The recommended way to do this is to create a file named after the site, e.g. example.com in /etc/apache2/sites-available and then enable and disable the site as required using the a2ensite and a2dissite commands respectively.

I find it useful to have a simple Hello World rack application to verify the installation. The application can be run as a subdomain to allow testing and can be enabled and disabled as desired.

I prefer to put all web applications in /var/www and have users be able to create applications if they are a member of www-data:

sudo chown www-data:www-data /var/www
sudo chmod g+w /var/www

Now, create the necessary directory structure for an application:

cd /var/www
mkdir hw.example.com
cd hw.example.com
mkdir public
mkdir tmp

Create /var/www/hw.example.com/config.ru with the following contents:

app = proc do |env|
  [200, { "Content-Type" => "text/html" }, ["hello, world"]]
end
run app

Create /etc/apache2/sites-available/hw.example.com with the following contents:

<VirtualHost *:80>
    ServerName hw.example.com
    DocumentRoot /var/www/hw.example.com/public
    <Directory /var/www/hw.example.com/public>
        AllowOverride all
        Options -MultiViews
    </Directory>
</VirtualHost>

Enable the site, and then view it in your browser:

sudo a2ensite hw.example.com
sudo /etc/init.d/apache2 reload

This can then be disabled and reenabled if needed for debugging any issues at a later date:

sudo a2dissite hw.example.com
sudo /etc/init.d/apache2 reload

Ubuntu Lucid VPS Base Build

2010-07-10T00:00:00+01:00

This post is an update to Ubuntu Jaunty Base Server Build, but focuses on Ubuntu Server 10.04 (Lucid). Unfortunately, I was planning on just updating using do-release-upgrade, however, after what seemed like a successful upgrade the VPS never came back to life and as it was remote I could not attempt to fix it. I had planned for failure, so I had a backup of all my instructions on this blog and the data too.

The aim is to provide a basic build that is secure for living out on the internet, has a basic web server stack (i.e. LAMP: Apache HTTP Server, MySql, PHP) and any other useful basic utilities. Future posts will rely on this build when covering additional application installations.

Base install

The initial image I was given was Ubuntu Server 10.04 with root being the only interactive user (i.e. able to log on). ssh was available using passwords. Apache2 and PHP packages were installed in addition to the essentials. The following command can be run to provide information about the installation, which may be of use when asking for help on forums:

lsb_release -a

Note: It is best practice to log into the server as a non-root user and when necessary use elevated permissions through the sudo command. The standard Ubuntu installation does not expose the root user, instead you create a user during the installation and login using that user. Setting up the user is covered in a later section called Users and Sudoers. Regardless of this, all commands that require sudo have been prefixed with it as it is good practice to get use to this.

Getting up-to-date

Ubuntu uses apt as its default package manager. Run the following commands to upgrade the installed packages to the latest version.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade

For more information visit the Ubuntu Community Documentation AptGet HowTo

The server can also be configured to apply automatic updates for certain categories of updates, e.g. all, security. Alternatively, the apticron package can be used for notification of any updates that are available. See automatic updates in the server guide for more details.

Additional util packages

The following packages are also useful to have installed, so run the following command:

sudo apt-get install nano bash-completion aptitude curl

Configure hostname

Its a good idea to give your machine the correct hostname as early as possible. Some packages may use the hostname during installation and therefore if you change it later you may encounter some issues.

The simplest way to set the host name permanently is to modify (or create if not present) the /etc/hostname file:

echo servername.domainname | sudo tee /etc/hostname

The above command will take effect after a reboot. You can also change the hostname for the current boot of the VPS using the hostname command:

sudo hostname servername.domainname

You also need to update /etc/hosts with DNS entries and aliases like the following:

127.0.0.1 localhost.localdomain localhost
127.0.1.1 servername.domainname alias.domainname

The following commands should respond suitably:

uname -n
hostname -a
hostname -s
hostname -d
hostname -f
hostname

Correcting the server time

Depending on the image used, the server may be in the wrong timezone. Run the following commands to reconfigure and utilise NTP to sync the clock:

sudo dpkg-reconfigure tzdata
sudo apt-get install ntpdate

Note that it may require a change by the VPS provider to fix the host machines clock if the time is still incorrect but the server is in the correct timezone.

For more information visit the Ubuntu Community Time documentation.

Users and Sudoers

Sudoers controls who can run what commands as which users on which machines. Therefore, it is good practice to create an administrator with the appropriate permissions and use sudo as required.

Run the following command to add a user:

sudo adduser <username>

Add the user to additional groups as required, e.g.:

sudo usermod -a -G sudo,adm,www-data <username>

By adding the user to the sudo group the user should be granted full sudo priviledges. If this is not the case, or you wish to modify the priviledges, then modify /etc/sudoers accordingly using visudo:

sudo visudo

Adding the following entry would give the user the same rights as root:

username ALL=(ALL) ALL

For more information visit the Ubuntu Community Documentation Sudoers Guide.

Note: now that you have a user, it would be a good idea to log out as root and log in as the new user in line with best practices.

Email forwarding

Email is used as a primary notification mechanism for linux, however, by default, email is stored on the server. If you do not intend to set the server up as an email server (e.g. POP3 or IMAP), it is useful to forward email to an appropriate external email address. There are two main approaches to this (with sendmail which is the default MTA):

root maintains email addresses in /etc/aliases
each user maintains their email address in ~/.forward

It is possible to also combined the two approaches, by aliasing root to a given user and then the user being able to allocate the email address as they see fit.

This is covered in the man pages for aliases:

man aliases

Modifying aliases

e.g. alias root to another user, edit /etc/aliases, adding:

root:           <otheruser>

e.g. alias root to external email addresses (comma separated), edit /etc/aliases, adding:

root:           user1@example.com, user2@example.com

Note: after modifying /etc/aliases you must run newaliases to refresh the alias database for the changes to take affect, e.g.:

sudo newaliases

Creating .forward

The simplest way to do this is to log in as the user required and run the following command (replacing the email address as appropriate):

echo username@example.com > ~/.forward

Secure Shell (SSH)

Secure shell (ssh) is the defacto standard for remote shell access.

Client

You can connect to a remote machine using the following command:

ssh username@hostname

You will usually be prompted for your password.

Rather than using a password you can use key based authentication. On your client machine, create a key and use a string pass phrase:

ssh-keygen -v -t rsa

This will create a ~/.ssh directory containing id_rsa which is your private key and id_rsa.pub which is your public key. Its important to have the correct permissions, so check the output of ls -la ~/.ssh is similar for the following entries:

drwx------   5 user  group   170 22 Mar 07:04 .
-rwx------   1 user  group  1743 22 Mar 07:04 id_rsa
-rwxr--r--   1 user  group   397 22 Mar 07:04 id_rsa.pub

You can copy you public key up to a server using the following command:

scp ~/.ssh/id_rsa.pub username@hostname:~/.ssh/authorized_keys

You should now be able to ssh to the server without being challenged for a password, e.g:

ssh username@hostname

Depending on the client and your preference, you can either enter the passphrase for you private key each time you connect, or run ssh-agent and register the key using ssh-add.

Server

sshd is the ssh daemon and is responsible for providing ssh access to the server. Its important to ensure that this is secure as possible. There are a number of articles out there about changing the configuration of sshd to make it more secure, in particular I like to turn off password authentication (forcing the use of public/private key pairs) and disable root login. Of course, you should ensure that you have a suitable user able to log in using a key that you have tested before you lock yourself out permanently.

The configuration for sshd is found at /etc/ssh/sshd_config. Modify it to have the following lines:

PasswordAuthentication no
PermitRootLogin no

Restart sshd for your changes to take effect:

sudo /etc/init.d/ssh restart

Basic web server stack

The basic stack of Apache HTTP Server (A), MySQL (M) and PHP (P) is known as AMP due to the initials, or LAMP on Linux.

Apache HTTP Server and PHP were already installed and available on the VPS, however, you can make sure by running the following:

sudo apt-get install apache2 php5

As part of the MySQL installation, I like to install phpMyAdmin as this is a really useful admin application. Install mysql server and phpMyAdmin packages, entering suitable values when prompted:

sudo apt-get install mysql-server phpMyAdmin

Check connectivity by accessing the site using http://servername/phpmyadmin.

phpmyadmin could be used by hackers to brute force attack your databases as it provides a form on the web to attempt log ons. As always, you should use suitably strong passwords to ensure that this is not easily done, and also use suitable permissions for the database users to minimise external access. The best way to avoid attack is to only have phpmyadmin available when you are using it. This can be achieved by removing the symlink in /etc/apache2/conf.d and managing phpmyadmin’s availability using the a2ensite and a2dissite commands:

sudo rm /etc/apache2/conf.d/phpmyadmin.conf
sudo ln -s /etc/phpmyadmin/apache.conf /etc/apache2/sites-available/phpmyadmin

You can then enable it using:

sudo a2ensite phpmyadmin
sudo /etc/init.d/apache2 reload

And disable it using:

sudo a2dissite phpmyadmin
sudo /etc/init.d/apache2 reload

System Monitoring

Keeping an eye on utilisation

Disk space can be checked using df. Memory usage can be checked using free.

Issues

Something went wrong

The Ubuntu Server Guide should be able to help you. If not then try to post on a suitable forum or mailing list with as much detail as possible.

Favourite Mac OS-X Apps

2010-06-01T00:00:00+01:00

This guide covers applications that I like to run on the mac. I work on multiple platforms so where possible I prefer to use cross platform software.

Power Tools

I find the following collection of power tools useful:

MenuMeters provides resource monitoring on the menubar
The Unarchiver is a replacement basic unarchiver that supports more formats

Preference Panes

The following preference pane snap-ins can be very helpful:

RCDefaultApp allows default applications to be configured more easily

Browser

I use a mixture of Google Chrome and Firefox depending on what I’m doing and how well the sites work in the browsers. I’m tending to use Chrome most of the time and falling back to Firefox these days - they both have good plugin support these days.

Xmarks, formerly known as Foxmarks, is what I use to sync bookmarks between platforms and browsers. Firefox and Chrome both have plugins for this.

For Chrome I use the following plugins:

Google Dictionary
RSS Subscriptions Extension (by Google)
Xmarks Bookmarks Sync
Google Mail Checker
Google Reader Notifier
Notifier for Google Wave
Google Calendar Checker

For Firefox I use the following plugins:

Better Gmail 2 - bundle of user scripts for gmail
Better Flickr - bundle of user scripts for flickr
DownloadHelper - downloading images and video
DownThemAll! - mass downloader
FiddlerHook - tie in for fiddler (windows)
Firebug - dev tools
Fission - progress bar (mac)
FlashGot - download manager integrator
Greasemonkey - user scripts
IETab - allows sites to be viewed in firefox using IE (windows)
Refractor for Prism - allows creation of dockable apps from websites
Web Developer - dev tools
Xmarks - cross platform and browser bookmark sync
YSlow - extends Firebug to provide suggested improvements for better sites

Notifications

Growl is the notification system for the mac. Many other applications, including apple ones, utilise growl if installed.

Instant messaging

Adium is a generic chat client with support for most chat protocols that also integrates with Growl.

FTP

FTP Client

Filezilla is a free, open source, cross platform ftp client. Cyberduck is a free, open source FTP, SFTP, WebDav and cloud browser for Mac OS-X.

FTP Server

FTP is available as part of OS-X, if enabled.

Media

Codec

Whilst OS-X comes with QuickTime, it only supports a few codecs. Additional codecs can be supported via plugins. I have used the following:

flip4mac - plays WMV, pro version allows conversion to mac formats for use in iMovie
Perian - many other codecs
MPEG2 - apple’s quicktime MPEG2 playback codec (needs to be purchased)

Video Conversion

MPEG Streamclip is a conversion tool that supports the codecs installed.

Player

VLC is a really good cross platform media player with support for most formats.

If you need to play real media then there is a RealPlayer for the mac also.

Transcoder

Handbrake is a good cross platform video transcoder.

Uninstaller

AppZapper is a commercial tool for removing all traces of an installed application. I’ve not used it but may help if there is a really pesky app that needs removing.

Sync

iSync

iSync can be used to sync to your mobile if its supported. My work one (Nokia 3109 Classic - euk) isn’t supported, but there are ways to get it working

File

There are two main applications in this area; Unison is an open source project that is powerful but requires a fair amount of user knowledge, ChronoSync is a commercial application with a much richer user interface.

Dropbox

Text Editor

There are quite a number of text editors available, however, there is a lot of support for TextMate on the mac. This is one of the few applications that is not free in this guide, but I have to say that I think its one of the best I’ve used.

Bundles and Extension

TextMate comes with a large set of “Bundles” for most languages or tasks which can be modified to suit although they are great just out of the box. Open source projects often create bundles to ease adoption and use of their products and libraries, some of my favourites include:

http://github.com/aslakhellesoy/cucumber-tmbundle
http://github.com/rspec/rspec-tmbundle
http://github.com/kuroir/SCSS.tmbundle
http://github.com/glennr/uber-glory-tmbundle

These can be installed using a command prompt, for example to install the rspec bundle:

cd ~/Library/Application\ Support/TextMate/Bundles/
git clone git://github.com/rspec/rspec-tmbundle.git RSpec.tmbundle
osascript -e 'tell app "TextMate" to reload bundles'

Tips

http://blogobaggins.com/2009/03/31/waging-war-on-whitespace.html

Shortcuts

Toggle comments - ⌘ /
Drawer - ⌃ ⌥ ⌘ D
Bracket Select - ⇧ ⌘ B
Select Item - ⌃ ⌘ T
Amend Lines - ⌥ ⌘ A
Delete to EOL - ⌃ D
Next Line - ⌘ ↩
Goto file - ⌘ T
Goto Symbol - ⇧ ⌘ T

Remote Desktop

Remote Desktop for Mac allows a windows machine to be controlled from your mac.

Cisco VPN Client

Cisco VPN Client can be used to connect to a Cisco VPN.

Mac Development

XCode

Xcode is a suite of development tools used to create Mac and iPhone applications. It provides a number of building blocks for other applications, such as build tools and compilers.

Command Line

Homebrew

Homebrew is the latest package manager to hit OS-X and is gaining a large following. I was using Macports but I’ve just switched and uninstalled it.

Xcode is a prerequisite for some packages (aka formulae in homebrew), providing gcc and other important tools.

To install homebrew (as per the instructions):

sudo chown -R $USER /usr/local
curl -L http://github.com/mxcl/homebrew/tarball/master | tar xz --strip 1 -C /usr/local

I have installed the following formulae (and their dependencies):

ack
bash-completion
dos2unix
gist
git
imagemagick
ncursesw
node
rename
sl
sqlite
subversion
tree
v8
wakeonlan
wget
xmlstarlet

Macports

I’m no longer using macports but I’ve listed it here as homebrew might be too fresh for some people. Macports is a collection of common ports from unix / linux, most of which are very useful for command line use. Fink is an alternative that has a larger number of packages but seems to be less frequently updated.

Xcode is a prerequisite, providing gcc and other important tools.

When I was using macports I installed the following ports (and their dependencies):

bzip2
curl
git-core +svn +bash_completion
gzip
meld
ncurses
python_select
python26
py26-markdown
py26-pygments
ruby
rb-rubygems
sqlite3
subversion +bash_completion
unison
wget
xmlstarlet

Some handy commands:

sudo port -u install <package> : install the supplied package, uninstalling the old version
sudo port uninstall <package> : uninstall the supplied package
sudo port selfupdate : update port itself
sudo port -u upgrade installed : upgrade the installed packages
sudo port clean --all installed : cleans installation files for all installed packages
sudo port deactivate <package> @<version>+<variant> : deactivate a package
sudo port activate <package> @<version>+<variant> : activate a package
sudo port uninstall inactive : uninstall inactive packages

If you need to switch versions, or variants, you need to deactivate the current version and then activate the one you want.

Bash init scripts

.bashrc

This tends to be run for bash scripts so should contain path vars and such. This is a good place to add a bin directory to the path for housing your custom scripts.

# non-login shells
# echo "calling .bashrc"

# set PATH so it includes user's private bin if it exists
if [ -d ~/bin ] ; then
  PATH=~/bin:"${PATH}"
  export PATH
fi

# enable bash completion
if [ -f `brew --prefix`/etc/bash_completion ]; then
  . `brew --prefix`/etc/bash_completion
fi

# editor
export EDITOR=nano

# java
export JAVA14_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.4/Home
export JAVA15_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.5/Home
export JAVA16_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home
export JAVA_HOME=$JAVA16_HOME

# maven
export M2_HOME=~/tools/maven/2.2.1
export M2=$M2_HOME/bin
export PATH=$M2:$PATH
export MAVEN_OPTS="-Xms256m -Xmx512m -XX:MaxPermSize=256m"

# prompt format
PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '

# bash history niceness
export HISTIGNORE="&:ls:[bf]g:exit"

.bash_profile

This is run when logging in or creating a new bash session, so its not used as much. Its a good idea to call .bashrc to setup the environment in a DRY fashion.

# login shells
# echo "calling .bash_profile"

if [ -f ~/.bashrc ]; then
  source ~/.bashrc
fi

Visual Diff

Xcode contains an application called FileMerge but its very lightweight. DiffMerge is a free cross platform visual diff application that is worth trying out.

GitX

A gitk like repository viewer designed for OS X. brotherbard’s fork is worth looking at.

Java Development

The tools I use for java development are covered in a recent article, however, there are some specifics that are needed to get older versions of java installed if you don’t happen to be working with the latest versions.

JDKs

OS-X Leopard comes with JDK 1.4, 1.5 and 1.6 pre installed. The java tools are available on the command line (e.g. java, javac, jconsole, etc).

Note: Snow Leopard removed JDK 1.4 and 1.5, so there are some steps you can take to reinstall them:

download the last release from leopard.
use pacifist to copy the following to the same location on your machine: * /System/Library/Frameworks/JavaVM.framework/Versions/1.4 * /System/Library/Frameworks/JavaVM.framework/Versions/1.4.2 * /System/Library/Frameworks/JavaVM.framework/Versions/1.5 * /System/Library/Frameworks/JavaVM.framework/Versions/1.5.0

You can then verify by accessing the java preferences app in Applications -> Utilities -> Java Preferences.

Command Line Tips

2010-05-03T00:00:00+01:00

Bash

There are some nice shortcuts to typing everything all the time. These are called event designators, word designators and modifiers. There is plenty of info on these in the bash man pages, but some real world examples follow:

Download, expand and change directory:

wget http://rubyforge.org/frs/download.php/68719/ruby-enterprise-1.8.7-2010.01.tar.gz
tar xvf !$:t
cd !$:r:r

Repackaging:

cat myfile-1.0.1.tar.gz | gunzip | bzip2 | !#:1:s/gz/bz2

Forgot to sudo, rerun the previous event with sudo using !!:

nano /etc/hosts
sudo !!

Copying

Copy files merging and preserving permissions:

tar cf - * | ( cd /target; tar xfp -)

Find

Find is a very powerful command and can save a lot of time. Here are a few of my favourite uses:

Finding in files

I have the following script in ~/bin/find_all to find all files excluding some common metadata and generated locations:

#!/bin/bash
# has support for null separator due to usage of spaces in paths
if [ "-0" = "$1" ]; then
	print="-print0"
else
	print="-print"
fi
prune="( -name target -o -name .svn -o -name .git ) -prune"
match="-type f"
find . $prune -o $match $print

This can then be used to search through the files using grep, e.g.:

find_all | xargs grep TODO

Search and Replace

Find that execs sed on each matched file:

find . -type f -exec sed -i '' 's/FROM/TO/g' {} \;

xmlstarlet

Reformatting xml files

Also known as pretty printing, this makes use of xmlstarlet and I have this script in ~/bin/xmlpp:

#!/bin/bash
files=`find . -type d -name target -prune -o -name *.xml -print`
for file in $files
do
   mv $file $file.bak
   xml fo $file.bak > $file
   rm $file.bak
done

XML Queries

Query xml for particular elements:

curl http://feeds.feedburner.com/railscasts -o rss.xml
xml sel -t -m "/rss/channel/item" -v "title" -n rss.xml > episode-list.txt
xml sel -t -m "/rss/channel/item" -v "enclosure/@url" -n rss.xml > episode-media-urls.txt

Java Development Tools

2010-05-02T00:00:00+01:00

This article has now moved to its own page and been significantly extended.

Mac File System tips

2010-05-01T00:00:00+01:00

Home directory conventions

I like to add the following directories to my home directory:

bin - contains scripts and is put on path by .bashrc
dev - contains development work, such as project working directories
tools - contains custom tools for a particular user, e.g. eclipse, maven.

Add the following lines to your .bashrc or appropriate script to put the bin directory on the system path:

# set PATH so it includes user's private bin if it exists
if [ -d ~/bin ] ; then
  PATH=~/bin:"${PATH}"
  export PATH
fi

Multi-User setup

In order to share various files around users on a single machine it is useful to put shared files under /Users/Shared and then add symbolic links in each user’s home directory to pass through to those locations. For example:

ln -s /Users/Shared/documents ~/Documents/shared

Prevent .DS_Store file creation over network connections

This is covered by Apple knowledge base article but I’ve given the command here also (just run it in a terminal window):

defaults write com.apple.desktopservices DSDontWriteNetworkStores true

This will keep your windows friends and file system purist happier :)

Synchronisation between machines

Its good practice to ensure that operating system specific files are not synchronised. Often these files are localised cache structures or permission related so they could cause more problems if sychronised.

The following operating system files should not be transferred between machines:

.DS_Store
Thumbs.db
ehthumbs_vista.db

The following app specific files should not be transferred between machines:

ZbThumbnail.info
*.onetoc2
Picasa.ini

Startup items

Startup items for the system can be added to /Library/StartupItems or configured through the preference pane.

Upgrading to Rails 3 Beta 3

2010-04-15T00:00:00+01:00

Rails 3 Beta 3 hit the streets a few days ago and I was able to upgrade with only a few issues to sort out. This post covers the issues I had and how I overcame them.

First off I updated my rvm to the latest 1.9.2-head:

rvm install 1.9.2-head

This can be a little risky as head is constantly changing so if you have a revision of 1.9.2 that you are happy with then stick with it unless you are getting some strange problems and an update might help.

Next I installed the latest rails prerelease using:

gem install rails --pre

And updated my Gemfile to reference beta 3:

gem 'rails', '>=3.0.0.beta3'

When I tried to start up the server I got the following error:

$ rails s
/Users/kris/.rvm/gems/ruby-1.9.2-head/gems/rails-3.0.0.beta3/bin/rails:1:in `require': no such file to load -- rails/cli (LoadError)
	from /Users/kris/.rvm/gems/ruby-1.9.2-head/gems/rails-3.0.0.beta3/bin/rails:1:in `<top (required)>'
	from /Users/kris/.rvm/gems/ruby-1.9.2-head/bin/rails:19:in `load'
	from /Users/kris/.rvm/gems/ruby-1.9.2-head/bin/rails:19:in `<main>'

The error turned out to due to a change that moved the rails binary file to a different location. This meant that older versions of the gem were conflicting due to incorrect paths. A simple gem clean fixed it and the server was able to start. Note that gem clean will remove all old versions of installed gems - you may be able to remove the specific old version of rails to fix the issue.

The next error was the deprecation warning for the error_messages helper method. I strongly agree with the removal of this, but it did worry me at first as I often use it when putting together a quick sample app. Not to worry as the generators now generate the HTML into the view directly rather than using the helper so I simply generated another resource and copied and pasted the markup around. You can copy and past the following snippet if it helps, or even put it into a partial and reuse it:

<% if @comment.errors.any? %>
  <div id="error_explanation">
    <h2><%= pluralize(@comment.errors.count, "error") %> prohibited this comment from being saved:</h2>
    <ul>
    <% @comment.errors.full_messages.each do |msg| %>
      <li><%= msg %></li>
    <% end %>
    </ul>
  </div>
<% end %>

Perhaps generating the partials would have made the transition a little easier and DRYer. However, in a real app the likelihood is that you would want to create a more awesome error highlighting look ‘n’ feel and therefore this is a minimalist starting point that is still functional.

Jekyll 0.5.7

2010-03-04T00:00:00+00:00

This site is using jekyll, a static site generator. The documentation for jekyll is very good. I originally intended this to be my first post on the blog covering what I had to work around, however, I ended up fixing some of the issues and managed to get quite a few commits in and jekyll has moved from 0.5.4 through to 0.5.7 during that time.

Overall I’m quite happy with jekyll although I think it needs to move forward with a good extensions framework to make it more useful to those not running on github pages. I guess jekyll does what github needs so in some respects there isn’t any great motivation in moving it forwards officially, hence the large number of forks to customise it to peoples needs.

Google on Mac OS X

2010-02-18T00:00:00+00:00

There are a number of applications and services that I regard as important and having them readily available is vital for me. I don’t want to be responsible for administering the data or the applications. With HTML5 in varying degrees of implementation in a number of browsers, it is becoming more the case that web applications can be desktop applications too.

There are of course always good candidates for native applications and ideally they should help to faciliate web applications or integrate to the web or the cloud.

As you would expect, Google provides many services and a large majority are web orientated. This post focuses on Google services and applications and how they can be utilised on Mac OS-X. Google have a landing site for mac applications and mac code.

Browser

Given that the web is critical to the success of Google its not surprising that they created a browser too. Google Chrome has moved the browser market forward ensuring that focus is put on security, performance and reliability, making the browser more suitable than ever before for applications. Its adoption of HTML5 and the offline capabilities in the specification has made the case for web applications that can be desktop applications a reality.

Prior to HTML5, there was Google gears which provided extensions to browsers to facilitate offline web applications. As with HTML5, there is a reliance on the web application to utilise the available resources.

Now that Google Chrome supports extensions (plugins) on Mac OS-X I have made it my default browser.

Web Applications

On a regular basis I use Google Mail, Google Reader and Google Calendar. All of these applications make use of offline capabilites, either through Google gears or HTML5.

Google Mail also has other services embedded in it, such as contacts, chat and tasks.

Launcher

Google Quick Search Box (QSB) is a great search and launcher replacement for spotlight by the guy who wrote Quicksilver. I turned off the spotlight menu keyboard shortcut in the spotlight preferences so I can bind Command + Space to QSB.

Notifications

Google Notifier sits on the menubar and provides notifications for email and calendar. It also provides quick options to open emails, compose emails and add calendar entries amongst others. There are a number of extensions for Google Chrome that provide similar notifications, however, the browser must be open to see them.

Google Notifier allows plugins, and wafflesoftware have created a plugin called Google Growl to make use of Growl. Its pretty simple to set up, you just need to change the Google Notifier Preferences to not show popups or play sounds and therefore rely on the configuration in growl.

Google Earth

Google Earth is great. I’m sure there lots of great uses for it but I just love flying around and looking at interesting things.

Desktop Application Integration

Mail

Google Mail supports IMAP (and POP but that is not bidirectional), allowing good integration with pretty much any mail program with IMAP support. Using POP with Apple Mail covers the details for Apple Mail integration.

iCal

Google provide a setup program called calaboration to initialise the calDAV settings for iCal. I’m not fond of the way the calDAV integrated calendars are shown in iCal, so I’m looking for alternative solutions.

I like the look of Spanning Sync which can sync using the native iCal files as well as contacts, however, it costs money and data is passed through their servers. The licence is also linked to a single account and I want to be able to sync multiple accounts, although this can be worked around by having a master account and sharing your other calendars with it (e.g. from your work or home accounts).

Chat

Adium is a generic chat client with support for google chat that also integrates with Growl.

Using git for sync and backup to a NAS

2010-02-16T00:00:00+00:00

git can be the ideal choice for some sync and backups tasks where maintaining the history is important. Even if the history becomes less important over time, you can simply blow the repository away and recreate it from your local working copy without any history, or even use some interesting commands to do this. Hopefully soon someone will create something that uses gits concepts to do this and allow history rewriting / squashing although this does go against the immutability concepts in git.

For this example, I have a local photos repository (/Users/kris/Pictures/shared/.git) that I want to make available on the nas (mounted at /Volumes/share/):

git clone --bare --no-hardlinks /Users/kris/Pictures/shared/.git /Volumes/share/shared/photos.git
git remote add share /Volumes/share/shared/photos.git

This first does a bare clone of the local repo to the nas, then sets the remote for the local repo to the clone on the nas.

Provided the nas is mounted, the git repository can be used under normal workflows.

Git Tips

2010-02-15T00:00:00+00:00

This is a collection of git tips from various sources and tailored to my needs. The tips are known to work under git 1.7.3.

Updates 2010-10-20:

Updated git version known to work against to 1.7.3 as I’ve not tested any new features on old versions
Improved some aliases based on Mislav’s post
I no longer use core.autocrlf setting, so set to false
Covered push -u for setting up remote tracking

Read the community documentation

There is a lot of really good community documentation out there. The primary git-scm site is a good start, as well as git ready.

Useful global config

To show who you are:

git config --global user.name "Kris Brown"
git config --global user.email kris@kris.me.uk

To colourise the outputs:

git config --global color.diff auto
git config --global color.status auto
git config --global color.branch auto

To add some useful aliases:

git config --global alias.ci commit
git config --global alias.st "status -sb"
git config --global alias.qlog "log --oneline -n 10 --decorate"
git config --global alias.glog "log --oneline -n 10 --decorate --graph"
git config --global alias.co checkout
git config --global alias.svnup "svn rebase"
git config --global alias.svnci "svn dcommit --rmdir"

Github goodies:

git config --global github.user krisb
git config --global github.token $tokenhash

Handling CR+LF issues:

git config --global core.autocrlf false

Defaulting to rebase pulls to reduce merge commits:

git config branch.master.rebase true
git config --global branch.autosetuprebase always

System wide ignores (mac + windows):

git config --global core.excludesfile ~/.gitignore
echo .DS_Store >> ~/.gitignore
echo Thumbs.db >> ~/.gitignore
echo ehthumbs_vista.db >> ~/.gitignore

In git version 1.6.3, pushing no longer defaults to anything (it did default to matching previously). To set the default to current (i.e. just the current branch), modify the config:

git config --global push.default current

Project setup

Cloning an existing repository

Cloning an existing repository tracks that remote repository by default:

git clone git://github.com/$username/$project.git
cd $project

Creating a new project

Creating a new project, touching the readme and then performing the initial commit:

mkdir $project
cd $project
git init
touch README
git add README
git commit -m "first commit"

Modifying code and committing

In git, changes can be in one of 3 locations:

working tree which contains your local modifications on the file system
index or staging area which contains your local modifications on the file system and indexes changes that are staged ready for committing
repository which contains your committed modifications. This can be local or remote.

The basic work flow is to check your status, diff your changes and review them, add to the index and then commit to the repository.

<modify some files>
git status
git diff
git add -A
git commit -m "a suitably good description of the change"

There are many variations on the above, commands in git are very powerful and can be used in a variety of ways. git add -A will ensure that any deletions or new files are also indexed.

Once commits are in repositories can then be fetched, pushed, pulled, merged, rebased and many other things. Technically there is another location, the stash, think of this as a modifiable local branch.

Adding a remote

If you created a new project, or you want to add another remote to an existing project:

git remote add origin git@github.com:$username/$project.git
git push -u origin master

The -u option is optional and sets up tracking if present.

Tracking a remote

If you did not branch/clone from a repository initially or use the push -u option then you can add the tracking manually. This is also applicable if you wish to change the remote being tracked. For example, to track origin from master:

git config branch.master.remote origin
git config branch.master.merge refs/heads/master

This then allows you to pull and push without specifying the local and remote branches:

git pull
git push

git status also plays nicely with tracked remotes, showing if you are ahead or behind and by how many commits.

Patching

The following examples assume that the master is the main development branch and feature_branch is the feature branch.

Preparation

To create clean patches, the modifications should be made on a branch:

git checkout master
git branch feature_branch
git checkout feature_branch

The last 2 commands could be done in a single command using:

git checkout -b feature_branch

Updating

Before submitting a patch it is good practice to pull in the latest changes and rebase the branch to ensure the patch will work against the latest version:

git checkout master
git pull
git checkout feature_branch
git rebase master

The 1st and 2nd commands ensure that master is up to date, the 3rd and 4th commands rebase the feature branch to master. If the feature branch has been remoted already then you’ll have to merge instead, i.e. git merge master

Email style patches

This method is the recommended way of doing patches in git. The author is still credited with the changes and it is easy to commit. Run the following from the feature branch:

git format-patch master --stdout > patch.diff

The committer can then apply the patch using:

git am < patch.diff

The commiter may choose to use git apply if they wish to make alterations.

Traditional style patches

Git also allows the traditional way of patching using a diff but this does not preserve the commit history of the author. Run the following from the feature branch:

git diff > patch.diff

The committer can then apply the patch using:

patch -p1 < patch.diff

-p1 is needed rather than the usual -p0 due to the extra a/ and b/ prefixes in the git diff output. Alternatively, git diff can be supplied the --no-prefix option and patch supplied the -p0 option.

Tidying up

Checking if a branch contains commits or other branches

git branch --contains branch-to-delete
git branch --merged
git branch --no-merged

Deleting a remote branch

git push origin :branch

Finding duplicates

This is a really handy little script I use a lot of my photo archive to find and remove duplicates. Its based on the script from StackOverflow but I had to modify the regex to allow any characters in the filename as I have spaces in there. I create two scripts, one called gitdup that handles the usage comment in the second script, called gitdup_collate

#!/usr/bin/perl
# usage: git ls-tree -r HEAD | gitdup_collate

use strict;
use warnings;

my $sha1_path = {};

while (my $line = <STDIN>) {
    chomp $line;
    if ($line =~ m{ \A \d+ \s+ \w+ \s+ (\w+) \s+ (.+)}xms) {
        my $sha1 = $1;
        my $path = $2;
        push @{$sha1_path->{$sha1}}, $path;
    }
}

foreach my $sha1 (keys %$sha1_path) {
    if (scalar @{$sha1_path->{$sha1}} > 1) {
        foreach my $path (@{$sha1_path->{$sha1}}) {
            print "$sha1  $path\n";
        }
        print '-' x 40, "\n";
    }
}

Enabling sshd for Buffalo Linkstation Live (LS-CHL)

2010-01-03T00:00:00+00:00

THIS GUIDE IS FOR THE LS-CHL MODEL ONLY. ANY MODIFICATIONS ARE MADE AT YOUR OWN RISK. A MISTAKE CAN BRICK YOUR NAS

Buffalo @ nas-central.org is a good site for information on customising your linkstation live.

If unfamiliar with vi then find a Quick Guide to help you get by as vi is the main editor available to you.

Also, note that most of the commands are using BusyBox so may have limited functionality.

This guide is largely taken from Open Stock Firmware Guide with my commentary on how I overcame any problems. sshd is already available on the linkstation, it just needs switching on. ACP_Commander is the key to opening the initial access allowing you to make the necessary modifications.

Prerequesites

Java installed
Download ACP_Commander
Ensure you have read the ACP_Commander readme.

Steps

Connect using acp_commander to reset the root password and open telnet:

$ java -jar acp_commander.jar -t 192.168.xxx.xxx -o
there seems to be no existing prefs, write default values
ACP_commander out of the nas-central.org (linkstationwiki.net) project.
Used to send ACP-commands to Buffalo linkstation(R) LS-PRO.

WARNING: This is experimental software that might brick your linkstation!


Using random connID value = BE442C00xxxx
Using target:	192.168.xxx.xxx/192.168.xxx.xxx
Starting authentication procedure...
Sending Discover packet...	
Found:	NAS (/192.168.xxx.xxx) 	LS-CHL(HANZEI) (ID=00016) 	mac: 00:1D:73:A3:xx:xx	Firmware=  1.20	Key=xxxxxxx
Trying to authenticate EnOneCmd...	ACP_STATE_OK
start telnetd...	OK (ACP_STATE_OK)
Reset root pwd...	Password changed.


You can now telnet to your box as user 'root' providing no / an empty password.

Now telnet to your box with username ‘root’ and no password:

$ telnet 192.168.xxx.xxx
Trying 192.168.xxx.xxx...
Connected to 192.168.xxx.xxx.
Escape character is '^]'.

NAS login: root
No mail.

Change the password to protect the box:

# passwd
Changing password for root
Enter the new password (minimum of 5, maximum of 127 characters)
Please use a combination of upper and lower case letters and numbers.
New password: 
Re-enter new password: 
Password changed.

Ensure there is a script to control sshd and that it has the correct permissions. My linkstation already had a script at /etc/init.d/sshd.sh. If not, find a suitable script from Open Stock Firmware Guide. To ensure the correct permissions:
```
# chmod 0755 /etc/init.d/sshd.sh
```
Ensure suitable configuration for sshd in /etc/sshd_config. I had to change UsePrivilegeSeparation from the default of yes to no as I was getting an error for users other than root sshing to the box.

Backup the start up script and modify, adding telnetd between step 2 and 3 and sshd to step 3:

# cp /etc/init.d/rcS /etc/init.d/rcS.bak
# vi /etc/init.d/rcS

Before (snippet starts on line 81):

echo "** step2 **"                                                                                     
for cmd in EnablingAutoip.sh usb.sh hotplug.sh networking.sh syslog.sh network_control.sh inetd.sh errormon.sh kernelmon.sh miconmon.sh checkSysMd.sh start_data_array.sh
do                                                                                                                                                                       
        exec_sh ${cmd}                                                                                                                                                   
done                                                                                                                                                                                                                                                                                                                          

echo "** step3 **"                                                                                                                                                       
for cmd in diskmon.sh drivecheck.sh ftpd.sh atalk.sh httpd.sh smb.sh clientUtil_servd.sh bonjour.sh lsprcvd.sh daemonwatch.sh cron.sh checkconfig.sh ups.sh pwrmgr.sh
do                                                                                                                                                                           
        exec_sh ${cmd}                                                                                                                                                       
done

After:

echo "** step2 **"                                                                                     
for cmd in EnablingAutoip.sh usb.sh hotplug.sh networking.sh syslog.sh network_control.sh inetd.sh errormon.sh kernelmon.sh miconmon.sh checkSysMd.sh start_data_array.sh
do                                                                                                                                                                       
        exec_sh ${cmd}                                                                                                                                                   
done                                                                                                                                                                     

# telnetd for debug                                                                                                                                                      
/usr/sbin/telnetd                                                                                                                                                        

echo "** step3 **"                                                                                                                                                       
for cmd in diskmon.sh drivecheck.sh ftpd.sh atalk.sh httpd.sh smb.sh clientUtil_servd.sh bonjour.sh lsprcvd.sh daemonwatch.sh cron.sh checkconfig.sh ups.sh pwrmgr.sh sshd.sh
do                                                                                                                                                                           
        exec_sh ${cmd}                                                                                                                                                       
done

Rack Application Hosting on Ubuntu Jaunty

2010-01-02T00:00:00+00:00

Thanks to Phusion, setting up a server to run rack and rails applications is very simple.

Ruby

Ruby is an obvious requirement and I’ve covered doing this with standard ruby packages for Ubuntu and Ruby Enterprise Edition, the later of which is recommended for a Passenger based rack application server. You can also use rvm for user accounts.

Standard Ruby

Ubuntu comes with an old fixed version of ruby gems so you need to do the following to get the standard ruby tools installed via packages:

sudo apt-get install ruby irb ri rdoc libruby-extras rubygems ruby1.8-dev
sudo gem install rake
sudo gem install rubygems-update
cd /var/lib/gems/1.8/bin
sudo ./update_rubygems

This will give you the usual ruby commands executed from /usr/bin. You can install and switch between multiple versions of ruby if you so wish.

Ruby Enterprise Edition

Ruby Enterprise Edition is a specific build of ruby by Phusion purposefully designed to perform well in a server environment. This is optional so if you want to run on standard ruby simply skip this section.

I ran the following commands and followed the information given as detailed on the installation page:

cd /opt
wget http://rubyforge.org/frs/download.php/68719/ruby-enterprise-1.8.7-2010.01.tar.gz
tar xzvf ruby-enterprise-1.8.7-2010.01.tar.gz
sudo ./ruby-enterprise-1.8.7-2010.01/installer

Accept the defaults and at the end, the following note is given regarding using with passenger:

Ruby Enterprise Edition is successfully installed!
If want to use Phusion Passenger (http://www.modrails.com) in combination
with Ruby Enterprise Edition, then you must reinstall Phusion Passenger against
Ruby Enterprise Edition, as follows:

  /opt/ruby-enterprise-1.8.7-2010.01/bin/passenger-install-apache2-module

Make sure you don't forget to paste the Apache configuration directives that
the installer gives you.

If you ever want to uninstall Ruby Enterprise Edition, simply remove this
directory:

  /opt/ruby-enterprise-1.8.7-2010.01

If you have any questions, feel free to visit our website:

  http://www.rubyenterpriseedition.com

Enjoy Ruby Enterprise Edition, a product of Phusion (www.phusion.nl) :-)

I like to symlink the version I want to use, so run the following:

sudo ln -s ruby-enterprise-1.8.7-2010.01 ruby-enterprise

To avoid any path issues, I make the modifications outlined in section 3.3 of the documentation by modifying the path in /etc/environment, i.e.:

PATH=”/opt/ruby-enterprise/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games”

Passenger

Passenger is very easy to install. There are two sets of instructions depending on whether you want to use the standard ruby installation or ruby enterprise edition.

Using standard ruby

Install the gem and run the installation script. It will tell you what dependencies you are missing and how to install them.

sudo gem install passenger
cd /var/lib/gems/1.8/bin
sudo ./passenger-install-apache2-module

To enable passenger, create /etc/apache2/mods-available/passenger.load with the following contents:

LoadModule passenger_module /var/lib/gems/1.8/gems/passenger-2.2.9/ext/apache2/mod_passenger.so

and /etc/apache2/mods-available/passenger.conf with the following contents:

PassengerRoot /var/lib/gems/1.8/gems/passenger-2.2.9
PassengerRuby /usr/bin/ruby1.8

then enable the module and restart apache:

sudo a2enmod passenger
sudo /etc/init.d/apache2 restart

The virtual host should point the document root to the public directory of the rails app.

<VirtualHost *:80>
    ServerName www.yourhost.com
    DocumentRoot /somewhere/public    # <-- be sure to point to 'public'!
</VirtualHost>

Using ruby enterprise edition

Install the gem and run the installation script. It will tell you what dependencies you are missing and how to install them.

sudo gem install passenger
sudo /opt/ruby-enterprise/bin/passenger-install-apache2-module

To enable passenger, create /etc/apache2/mods-available/passenger.load with the following contents:

LoadModule passenger_module /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/passenger-2.2.9/ext/apache2/mod_passenger.so

and /etc/apache2/mods-available/passenger.conf with the following contents:

PassengerRoot /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/passenger-2.2.9
PassengerRuby /opt/ruby-enterprise/bin/ruby

then enable the module and restart apache:

sudo a2enmod passenger
sudo /etc/init.d/apache2 restart

The virtual host should point the document root to the public directory of the rails app.

<VirtualHost *:80>
   ServerName www.yourhost.com
   DocumentRoot /somewhere/public    # <-- be sure to point to 'public'!
   <Directory /somewhere/public>
      AllowOverride all              # <-- relax Apache security settings
      Options -MultiViews            # <-- MultiViews must be turned off
   </Directory>
</VirtualHost>

Wordpress Installation on Ubuntu Jaunty

2010-01-01T00:00:00+00:00

A community article covers installation via an official package, however, I found this to be problematic with modifications and upgrades. I’ve included instructions for both package installation and manual installation.

I’ve made the following assumptions:

you want to be able to run multiple sites (or at least have the possibility of)
site document roots exist at /var/www/<sitename>

Dependencies

Ensure any major dependencies are installed:

sudo apt-get install apache2 php5-gd mysql-server

Apache2 Virtual Hosts

Create a suitable virtual host entry for the site, e.g. /etc/apache2/sites-available/example.com

<VirtualHost *:80>
    ServerAdmin admin@example.com
    ServerName example:80
    DocumentRoot /var/www/example.com

    <Directory /var/www/example.com>
        Options FollowSymLinks
        AllowOverride Limit Options FileInfo
        DirectoryIndex index.php
    </Directory>

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn
    ErrorLog /var/log/apache2/example.com-error.log
    CustomLog /var/log/apache2/example.com-access.log combined
</VirtualHost>

Enable the site and restart apache2:

sudo a2ensite example.com
sudo /etc/init.d/apache2 restart

Installation

By Package

Install using apt-get:

sudo apt-get install wordpress

The application is installed to /usr/share/wordpress, so if you wish to use a sub url (i.e. http://example.com/wordpress) then simply symbolic link to this, e.g.:

sudo ln -s /usr/share/wordpress /var/www/example.com/wordpress

or if you want to run as the root url (i.e. http://example.com/) then use the install location directly in the DocumentRoot and Directory directives in the httpd config.

You need to have a config file for the domain name wordpress is being accessed by, and make symbolic links for any aliases too, e.g.:

sudo cp /usr/share/wordpress/wp-config-sample.php config-example.com.php
sudo ln -s config-example.com.php config-blog.example.com.php
sudo ln -s config-example.com.php config-www.example.com.php

Manually

Install using the following commands:

cd /var/www/example.com
sudo wget http://wordpress.org/latest.tar.gz
sudo tar xvf latest.tar.gz
sudo rm latest.tar.gz

This will create an instance of wordpress at /var/www/example.com/wordpress which will be serviceable at http://example.com/wordpress. To service as the root url, run the following:

sudo mv wordpress/* .
sudo rmdir wordpress

Create a default configuration from the sample config:

sudo cp wp-config-sample.php wp-config.php

Create a .htaccess file for any url mappings:

sudo touch .htaccess

Ensure the correct permissions are given:

sudo chown -R www-data:www-data .

MySQL setup

Create a suitable database, user and permissions.

CREATE DATABASE IF NOT EXISTS `wordpress`;
CREATE USER 'wordpressuser'@'localhost' IDENTIFIED BY 'wordpresspassword';
GRANT USAGE ON *.* TO 'wordpressuser'@'localhost' IDENTIFIED BY 'wordpresspassword';
GRANT ALL PRIVILEGES ON `wordpress`.* TO 'wordpressuser'@'localhost';

Use a suitable database name, username and password - not the above!

Configuration

Database credentials

Modify the configuration file to supply the credentials for the database.

Securing logins and admin site

As described in the Administration Over SSL in the Wordpress Codex, modify the configuration file by adding:

/** Enable SSL */
define('FORCE_SSL_ADMIN', true);

Git Repository Server using Gitosis

2009-12-31T00:00:00+00:00

This post assumes that you have a server following the Ubuntu Jaunty Base Server Build.

Git Packages

The git-core package contains all the basic command line for git. As the server is running headless, there is no point in installing packages such as gitk.

sudo apt-get install git-core

Gitosis

Gitosis is a useful tool for managing git repositories using ssh. Gitosis can be obtained using git:

mkdir ~/src
cd ~/src
git clone git://eagain.net/gitosis.git

README.rst contains instructions for setting up and managing gitosis.

Installation

The following outlines the steps I took.

Ensure you have python-setuptools:

sudo apt-get install python-setuptools

Install gitosis:

cd gitosis
sudo python setup.py install

Create the git user that gitosis will run as (i.e. git):

sudo adduser \
    --system \
    --shell /bin/bash \
    --gecos 'git version control' \
    --group \
    --disabled-password \
    --home /home/git \
    git

Initialise gitosis with your ssh key so you can administer the configuration (assuming you have already setup ssh via certification):

sudo -H -u git gitosis-init < /home/username/.ssh/authorized_keys

Also make sure the post-update hook script is executable:

sudo chmod 755 /home/git/repositories/gitosis-admin.git/hooks/post-update

Administration

You can administer the configuration through git itself on the gitosis-admin repository by cloning, modifying and push changes back to the server.

Clone the gitosis-admin repository:

git clone git@hostname:gitosis-admin.git

Modify gitosis.conf adding a group for the new repository:

[group groupname]
writable = reponame
members = username

Note that you can reuse a group for other repositories should you want to. Generally I use the same groupname as the reponame. members and writable are comma separated lists.

Add, commit and push your changes to the repository:

git add -A
git ci -m "added reponame"
git push

You can now add create the project (or skip this for an existing project):

mkdir myproject
cd mypyroject
git init
# do some work, git add and commit files

Then add your server as a remote and push:

git remote add serveralias git@hostname:reponame.git
git push serveralias master

You can track the remote by adding suitable config:

git config branch.master.remote serveralias
git config branch.master.merge refs/heads/master

Ubuntu Jaunty Base Server Build

2009-12-30T00:00:00+00:00

This is a guide to the basic setup of a VPS on Ubuntu Server 9.04 (Jaunty). The aim is to provide a basic build that is secure for living out on the internet, has a basic web server stack (i.e. LAMP: Apache HTTP Server, MySql, PHP) and any other useful utilities. Future posts will rely on this build when covering additional application installations.

The VPS provider I chose was CheapVPS.

Base install

The initial image I was given was Ubuntu Server 9.04 with root being the only interactive user (i.e. able to log on). ssh was available using passwords. Apache2 and PHP packages were installed in addition to the essentials. The following commands can be run to provide information about the installation, which may be of use when asking for help on forums:

nuser -a
lsb_release -a

Getting up-to-date

Ubuntu uses apt as its default package manager. Run the following commands to upgrade the installed packages to the latest version.

apt-get update
apt-get upgrade
apt-get dist-upgrade

For more information visit the Ubuntu Community Documentation AptGet HowTo

apt has the concept of manually installed packages. A manually installed package is one which is passed to the apt-get install command. These packages are assumed to be required and will not be marked as unused dependency for commands such as apt-get autoremove. It would be nice if there were a command to see the list of manually installed packages (excluding the essential packages), however, as this discussion explains, there isn’t one that works as expected.

Keeping an eye on utilisation

Disk space can be checked using df. Memory usage can be checked using free.

Additional util packages

The following packages are also useful to have installed, so run the following command:

apt-get install nano

Configure hostname

The hostname is set using the hostname command and updating entries in /etc/hosts.

To set the hostname:

hostname servername.domainname

Update /etc/hosts with DNS entries and aliases like the following:

127.0.0.1 localhost.localdomain localhost
127.0.1.1 servername.domainname alias.domainname

The following commands should respond suitably:

uname -n
hostname -a
hostname -s
hostname -d
hostname -f
hostname

I also had to create the /etc/hostname file:

echo servername.domainname > /etc/hostname

Correcting the server time

Depending on the image used, the server may be in the wrong timezone. Run the following commands to reconfigure and utilise NTP to sync the clock:

dpkg-reconfigure tzdata
apt-get install ntpdate

Note that it may require a change by the VPS provider to fix the host machines clock if the time is still incorrect but the server is in the correct timezone.

For more information visit the Ubuntu Community Time documentation.

Sudoers

Sudoers controls who can run what commands as which users on which machines. The standard Ubuntu installation does not expose the root user, instead you create an admin user during the installation and can execute commands. The VPS image did not conform to this and had root access available with no additional users. Therefore, it is good practice to create an administrator with the appropriate permissions and use sudo as required.

Run the following command to add a user:

adduser <username>

Add the user to additional groups as required, e.g.:

usermod -a -G sudo,adm,www-data <username>

Modify /etc/sudoers to allow a given user root privileges using visudo:

visudo

Adding the following entry:

<username> ALL=(ALL) ALL

For more information visit the Ubuntu Community Documentation Sudoers Guide.

Email forwarding

root maintains email addresses in /etc/aliases
each user maintains their email address in ~/.forward

It is possible to also combined the two approaches, by aliasing root to a given user and then the user being able to allocate the email address as they see fit.

This is covered in the man pages for aliases:

man aliases

Modifying aliases

e.g. alias root to another user, edit /etc/aliases, adding:

root:           <otheruser>

e.g. alias root to external email addresses (comma separated), edit /etc/aliases, adding:

root:           user1@example.com, user2@example.com

Note: after modifying /etc/aliases you must run newaliases to refresh the alias database for the changes to take affect, e.g.:

sudo newaliases

Creating .forward

The simplest way to do this is to log in as the user required and run the following command (replacing the email address as appropriate):

echo username@example.com > ~/.forward

Secure Shell (SSH)

Secure shell (ssh) is the defacto standard for remote shell access.

Client

You can connect to a remote machine using the following command:

ssh username@hostname

You will usually be prompted for your password.

Rather than using a password you can use key based authentication. On your client machine, create a key and use a string pass phrase:

ssh-keygen -v -t rsa

drwx------   5 user  group   170 22 Mar 07:04 .
-rwx------   1 user  group  1743 22 Mar 07:04 id_rsa
-rwxr--r--   1 user  group   397 22 Mar 07:04 id_rsa.pub

You can copy you public key up to a server using the following command:

scp ~/.ssh/id_rsa.pub username@hostname:~/.ssh/authorized_keys

You should now be able to ssh to the server without being challenged for a password, e.g:

ssh username@hostname

Depending on the client and your preference, you can either enter the passphrase for you private key each time you connect, or run ssh-agent and register the key using ssh-add.

Server

The configuration for sshd is found at /etc/ssh/sshd_config. Modify it to have the following lines:

PasswordAuthentication no
PermitRootLogin no

Restart sshd for your changes to take effect:

sudo /etc/init.d/ssh restart

Basic web server stack

The basic stack of Apache HTTP Server (A), MySQL (M) and PHP (P) is known as AMP due to the initials, or LAMP on Linux.

Apache HTTP Server and PHP were already installed and available on the VPS, however, you can make sure by running the following:

sudo apt-get install apache2 php5

As part of the MySQL installation, I like to install phpMyAdmin as this is a really useful admin application. Install mysql server and phpMyAdmin packages, entering suitable values when prompted:

sudo apt-get install mysql-server phpMyAdmin

To enable phpMyAdmin, symlink the apache conf as a site and enable it:

sudo ln -s /etc/phpmyadmin/apache.conf /etc/apache2/sites-available/phpmyadmin
sudo a2ensite phpmyadmin
sudo /etc/init.d/apache2 reload

Check connectivity by accessing the site using http://servername/phpmyadmin.

Note that phpmyadmin could be used by hackers to brute force attack your databases. As always, you should use suitably strong passwords to ensure that this is not easily done, and also use suitable permissions for the database users to minimise external access. I also tend to disable the site (using the a2dissite command) when its not in active use.

Issues

opening /dev/null failed (Permission denied)

This is mentioned in 63031

After a reboot the permissions on /dev/null are incorrect:

ls -l /dev/null
crw------- 1 root root 1, 3 Sep  5 14:26 /dev/null

This can be fixed by running the following:

sudo rm /dev/null
sudo mknod -m 0666 /dev/null c 1 3