The process is often repetitive. To me, it seems to be a waste to do it manually every time. It also consumes time and attention. It would be great if I could spend them on tasks that bring more values to clients.

To minimize such waste, I have written two Chef cookbooks to automate the process:

• rackbox - to provision rack-based web server (Nginx as front server, Unicorn and Passenger as upstream app servers, rbenv as ruby version manager).
• databox - to provision database server (supports MySQL and PostgreSQL).

Getting started

In this post, I will show you a step-by-step guide on how to use the cookbooks together with knife-solo to provision a remote server in 4 steps:

1. setup Chef Solo environment
2. modify config file
3. provision remote server
4. tweak Capistrano deploy.rb

A working example in also available at teohm/kitchen-example.

1. Setup Chef Solo environment

• Install Chef Solo tools on local machine.
• Download Chef cookbooks to local machine.
• Install chef-solo on remote server.

Install Chef Solo tools

Let’s create a new directory,

1
2

mkdir chef-kitchen
cd chef-kitchen

and a Gemfile.

1
2
3
4

source "https://rubygems.org"

gem "knife-solo", ">= 0.3.0pre3"
gem "berkshelf"

I recommend knife-solo >= 0.3 as it includes a few major fixes and improvements.

Now, install the ruby gems.

1

bundle install

Finally, setup a kitchen directory structure with knife-solo.

1

bundle exec knife solo init .

Download Chef cookbooks

I use Berkshelf to manage cookbooks. So we need a Berksfile,

1
2
3
4
5

site :opscode

cookbook "runit", ">= 1.1.2"  # HACK: force-use this version
cookbook "databox"
cookbook "rackbox"

(I added a hack here to force berkshelf to use runit 1.1.2 required by rackbox. Still looking for a better solution.)

We can now download cookbooks with berks install.

1

bundle exec berks install --path cookbooks/

Install chef-solo on remote server

1

bundle exec knife solo prepare testbox

In this example, testbox is a host I setup in my ~/.ssh/config:

1
2
3
4

Host testbox
  User ubuntu
  Hostname ec2-51-221-13-121.ap-southeast-1.compute.amazonaws.com
  IdentityFile ~/.ssh/testbox_ec2.pem

2. Customize config file

• Download config example
• Customize config file

Download config example

1

curl https://raw.github.com/teohm/kitchen-example/master/nodes/host.json.example --output nodes/testbox.json

Modify config file (JSON)

The config file starts with a run_list. You specify a list of cookbook recipes here. Chef will run them in the same order in this list.

It is followed by cookbook attributes. You can modify these attributes. A full reference of attributes are described in each cookbook’s README (see appbox, databox, rackbox).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

{
  "run_list": [
    "databox",
    "rackbox"
  ],
  "appbox": {
    "deploy_keys": ["ssh-rsa 5bnmu23890fghghjk"],
    "admin_keys": ["ssh-rsa 456789fghjkvbn567"]
  },
  "databox": {
    "db_root_password": "welcome!",
    "databases": {
      "mysql": [
        { "username": "app1",
          "password": "app1",
          "database_name": "app1_production" }
      ],
      "postgresql": [
        { "username": "app2",
          "password": "app2",
          "database_name": "app2_production" }
      ]
    }
  },
  "rackbox": {
    "ruby": {
      "versions": ["1.9.3-p385", "1.9.2-p320"],
      "global_version": "1.9.3-p385"
    },
    "apps": {
      "unicorn": [
        { "appname": "app1",
          "hostname": "app1.test.com" }
      ],
      "passenger": [
        { "appname": "app2",
          "hostname": "app2.test.com" }
      ]
    }
  }
}

3. Provision remote server

1

bundle exec knife solo cook testbox

It uploads the kitchen directory and runs chef-solo on the remote server. Chef-solo will then takeover and execute the run list to setup everything.

What do we get at this point?

Basically, it’s done!

We have a full-stack, rack-based server with:

• 3 user accounts (deploy, devops, apps)
• rbenv as ruby version manager
• nginx as front-server
• unicorn, passenger-standalone as upstream app servers, managed by runit
• postgresql, mysql installed and databases created
• all apps will be stored in /home/apps/

4. Tweak Capistrano deploy.rb

Now, it’s ready to deploy a Rack-based app to the remote server!

I have two example Rails apps available on Github:

• teohm/sample-app1 runs on unicorn,
• teohm/sample-app2 runs on passenger-standalone.

There are a few minor tweaks required in Capistrano deploy.rb, as listed below.

I will explain the tweaks in details next time. Meanwhile, check out the complete working examples at: app1/config/deploy.rb and app2/config/deploy.rb

Login as deploy user

1

set :user, "deploy"

Deploy to /home/apps

1

set :deploy_to, "/home/apps/#{application}"

Load rbenv in Capistrano

1

default_run_options[:shell] = '/bin/bash --login'

Run bundler with --binstubs

1
2
3

require 'bundler/capistrano'
set :bundle_flags, "--deployment --binstubs"
set :bundle_without, [:test, :development, :deploy]

Restart app with runit sv

1
2
3
4
5
6
7
8
9
10
11

namespace :deploy do
  task :start do
    run "sudo sv up app1"
  end
  task :stop do
    run "sudo sv down app1"
  end
  task :restart, :roles => :app, :except => { :no_release => true } do
    run "sudo sv restart app1"
  end
end

Feedback

If you are interested on using the cookbooks, or have an idea/feedback/question about this topic, feel free to drop me (@teohm) a message. Pull requests and issue reports are definitely welcomed!

The isolated module will still stay in the main Rails app codebase, but can be easily packaged as a Ruby gem, tested separately, and used in other related applications.

Requirements

During the refactoring, we want to:

1. Use require to load the module like a normal gem. If possible, no dependency on Rails autoload features such asrequire_dependency and autoload_paths.

2. Edit the module without restarting server during development. In other words, we need to find a way to reload the module on every request.

Reload require files

After some research, I found a working solution by Timothy Cardenas:

1
2
3
4
5
6
7

ActionDispatch::Callbacks.to_prepare do
  if Object.const_defined?("Module1")
    Object.send(:remove_const, "Module1")
  end
  $".delete_if {|s| s.include?('module1') }
  require 'module1'
end

What it does is, before each request,

1. Unload the top-level module.
2. Un-require all required files from the module.
3. Re-require the top-level module.

An extra step in Rails 3.2

In Rails 3.2, ActionDispatch::Callbacks.to_prepare has a slightly different behavior. It will run before a request only if a watchable file is modified.

You need to specify your own watchable files:

1
2

# watch all .rb files recursively under modules/module1/ dir
config.watchable_dirs['modules/module1'] = [:rb]

Introduce require_reloader gem

Before bundling the solution into a gem, I did a search on RubyGems.org and found Colin Young’s gem_reloader. It’s based on Timy’s solution as well. I forked it and started playing around.

In the end, I made some major changes to include Rails 3.2 support, some fixes and new features.

So I decided to release it as a new Ruby gem: require_reloader.

1
2
3
4
5
6

# config/environments/development.rb
YourApp::Application.configure do
  ...
  RequireReloader.watch :module1,
    path: 'modules/module1'
end

Currently it supports Rails 3, including 3.1 and 3.2.

If you are working on something similar, looking forward for your feedbacks and pull requests. It is not tested on Rails 2 yet, so pull requests are definitely welcomed!

Delimiter allows any non-alphanum

You can use any non alpha-numeric character as delimiter:

1
2
3
4

%(any alpha-numeric)
%[char can be]
%%used as%
%!delimiter\!! # escape '!' literal

Bracket pairs no need to escape

No need to escape bracket pairs, even when nested. You can escape, but will need to escape both open and close bracket.

1
2
3
4
5

%( (pa(re(nt)he)sis) ) #=> "(pa(re(nt)he)sis)"
%[ [square bracket] ]  #=> "[square bracket]"
%{ {curly bracket} }   #=> "{curly bracket}"
%< <pointy bracket> >  #=> "<pointy bracket>"
%< \<this works as well\> >  #=> "<this works as well>"

Modifiers for String, Regex, Array, Symbol, Shell command

We often use % notation to create String and Array literals. But it also supports Symbol, Regex and shell command.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

%(interpolated string (#{ "default" }))
  #=> "interpolated string (default)"
%Q(interpolated string (#{ "default" }))
  #=> "interpolated string (default)"
%q(non-interpolated string)
  #=> "non-interpolated string"
%r(#{ "interpolated" } regexp)i
  #=> /interpolated regexp/i
%w(non-interpolated\ string  separated\ by\ whitespaces)
  #=> ['non-interpolated string', 'separated by whitespaces']
%W(interpolated\ string #{ "separated by whitespaces" })
  #=> ['interpolated string', 'separated by whitespaces']
%s(non-interpolated symbol)
  #=> :'non-interpolated symbol'
%x(echo #{ "interpolated shell command" })
  #=> "interpolated shell command\n"

Here’s some % notation examples written in minitest in case you interested.

1
2
3
4
5
6

arry = input || []  # handle input == nil
arry = [input] unless arry.kind_of?(Array)  # handle single value object

arry.each do |item|
  #process item
end

In Ruby, you should use Kernel#Array to convert the variable into an array object:

1
2
3
4
5
6
7

Array(input).each do |item|
  # process item
end

# Array(nil)     # => []
# Array("foo")   # => ["foo"]
# Array([1,2,3]) # => [1,2,3]

More usage examples written in minitest.

When you have a collection of Transactions objects and want to calculate its average amount, instead of looping through the collection like this:

1
2
3
4
5

double sum = 0;
for (Transaction *transaction in transactions) {
    sum += [transaction.amount doubleValue];
}
NSNumber *avg = [NSNumber numberWithDouble:(sum / [transactions count])];

you can reduce the loop to 1 line of code, using Objective-C key-value coding:

1

NSNumber *avg = [transactions valueForKeyPath:@"@avg.amount"];

Currently, there is a fixed set of collection operators:

• Simple collection operators (@avg, @count, @sum, @max, @min)
• Object operators (@distinctUnionOfObjects, @unionOfObjects)
• Array and set operators (@distinctUnionOfArrays, @unionOfArrays)

For details, refer to more usage examples.

When profiled the app using Instruments > Allocations, we found that a lot of unused ViewController objects were not released from memory.

Retain Cycle

The codebase uses a lot of Objective-C blocks as shown below, and self is often being called within the blocks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

@interface DetailPageViewController : UIViewController
@property(nonatomic, strong) UIButton *backButton;
...
@end

@implementation DetailPageViewController
@synthesize backButton;

- (void)loadView {
  ...
  [self.doneButton onTouch:^(id sender) {
    [self doSomething];
    self.isDone = YES;
  }];
}
...
@end

In this example code, the controller object holds a strong reference to doneButton object. But because doneButton onTouch: block is calling self, now the button object holds a strong reference back to the controller object.

When object A points strongly to object B, and B points strongly to A, a retain cycle is created, and both objects cannot be released from memory.

Use Lifetime Qualifiers

Apple Developer’s guide on ARC transition suggested a few ways to break the retain cycle using different lifetime qualifiers. It is definitely a must read for iOS development.

If your app is targeted for iOS 5, you can use __weak lifetime qualifier to break the cycle:

1
2
3
4
5
6
7
8
9
10

@interface DetailPageViewController : UIViewController
- (void)loadView {
  ...
  __weak DetailPageViewController *controller = self;
  [self.doneButton onTouch:^(id sender) {
    [controller doSomething];
    controller.isDone = YES;
  }];
}
@end

Because we are using __weak qualifier, doneButton onTouch: block only has a weak reference to the controller object. Now, the controller object can be released from memory when its reference count drops to 0.

For Mac users, open System Preferences > Keyboard, choose Keyboard tab. Click button Modifier Keys.. and change Caps Lock Key: to ^ Control.

Remember, remember.. click Change All… button to apply the changes to all files. It seems obvious, but took me a while to figure this out. :)

Since then, it has become my light-weight network tunneling tool in daily work.

Install sshuttle

The installation is easy now. You can install it through Mac OSX Homebrew, or Ubuntu apt-get.

1

brew install sshuttle

I use sshuttle to..

1. Tunnel all traffic

This is the first command I learned. It forwards all TCP traffic and DNS requests to a remote SSH server.

1

sshuttle --dns -vr ssh_server 0/0

Just like ssh, you can use any server specified in ~/.ssh/config. The -v flag means verbose mode.

Besides TCP and DNS, currently sshuttle does not forward other requests such as UDP, ICMP ping etc.

2. Tunnel all traffic, but exclude some

You can exclude certain TCP traffic using -x option.

1

sshuttle --dns -vr ssh_server -x 121.9.204.0/24 -x 61.135.196.21 0/0

For instance, when I am in China, I don’t want to tunnel Youku.com traffic to a foreign server, because its movie streaming service is only available within China.

In this case, I use -x option to exclude Youku.com IP addresses.

3. Tunnel only certain traffic

To tunnel only certain TCP traffic, specify the IP addresses or IP ranges that need tunneling.

1

sshuttle -vr ssh_server 121.9.204.0/24 61.135.196.21

This command comes in handy, whenever I need to test an app feature (e.g. Netflix movie streaming) which only available in certain countries, or to bypass ISP faulty caches.

4. VPN to office network

I seldom do VPN, but all you need is the remote SSH server with -NH flags turned on.

1

sshuttle -NHvr office_ssh_server

-N flag tells sshuttle to figure out by itself the IP subnets to forward, and -H flag to scan for hostnames within remote subnets and store them temporarily in /etc/hosts.

IP addresses.. troublesome?

Well, I try not to deal with IP addresses manually. So I wrote a few sshuttle helpers (tnl, tnlbut, tnlonly, vpnto) that allow me to use domain names instead of IP addresses:

Tunnel all traffic

1

tnl

Tunnel all traffic, but exclude some

1

tnlbut youku.com weibo.com

Tunnel only certain traffic

1

tnlonly netflix.com movies.netflix.com

VPN to office network

1

vpnto office_ssh_server

The script is available on my GitHub repo. You can load it into your ~/.bashrc. To override the default tunneling SSH server in the script:

1

TNL_SERVER=user@another_server tnl

Issue

When submit a multipart form that contains Unicode parameter name e.g.

1
2
3
4

<form method="post" enctype="multipart/form-data" action="">
  <input name="Iñtërnâtiônàlizætiøn_name"
         value="Iñtërnâtiônàlizætiøn_value" />
</form>

Rails controller returns the param value "Iñtërnâtiônàlizætiøn_value" as expected.

But the param name becomes: "I\xC3\xB1t\xC3\xABrn\xC3\xA2ti\xC3\xB4n\xC3\xA0liz\xC3\xA6ti\xC3\xB8n_name".

It makes life miserable, if you are not expecting this to happen:

1
2

params["Iñtërnâtiônàlizætiøn_name"] # => nil
params["I\xC3\xB1t\xC3\xABrn\xC3\xA2ti\xC3\xB4n\xC3\xA0liz\xC3\xA6ti\xC3\xB8n_name"] # => "Iñtërnâtiônàlizætiøn_value"

What happened?

When Rack returns multipart form data to Rails, it returns:

1
2

{ "I\xC3\xB1t\xC3\xABrn\xC3\xA2ti\xC3\xB4n\xC3\xA0liz\xC3\xA6ti\xC3\xB8n_name" =>
  "I\xC3\xB1t\xC3\xABrn\xC3\xA2ti\xC3\xB4n\xC3\xA0liz\xC3\xA6ti\xC3\xB8n_value" }

However, ActionDispatch::Http::Parameters#encode_params in Rails decided to only encode parameter values, but not parameter names. As a result, we get:

1
2

{ "I\xC3\xB1t\xC3\xABrn\xC3\xA2ti\xC3\xB4n\xC3\xA0liz\xC3\xA6ti\xC3\xB8n_name" =>
  "Iñtërnâtiônàlizætiøn_value" }

Solutions?

1. Don’t use Unicode param name.
2. Patch Rails source code. I added a fix in my forked branch, and reported the issue. Hopefully it will get fixed soon in the coming release.

Install iTerm2

If you haven’t heard of iTerm, it’s a popular open source alternative to Mac OS X Terminal. Give it a try, download and install it from http://www.iterm2.com.

Fine-Tune Settings

Launch iTerm, open iTerm > Preferences or just Cmd + ,.

Open tab/pane with current working directory

Under Profiles tab, go to General subtab, set Working Directory to “Reuse previous session’s directory”.

Enable Meta key

To enable Meta key for Bash readline editing e.g. Alt + b to move to previous word, under Profiles tab, go to Keys subtab, set Left option key acts as: to “+Esc”.

Hotkey to toggle iTerm2

Under Keys tab, in Hotkey section, enable “Show/hide iTerm2 with a system-wide hotkey” and input your hotkey combination, e.g. I use Ctrl + Shift + L.

Switch pane with mouse cursor

Under Pointer, in Miscellaneous Settings section, enable “Focus follows mouse”.

Handy Shortcut Keys

Here’s a set of shortcut keys I commonly use. You can always look for other shortcut keys in the iTerm menu.

Tab navigation

• open new tab Cmd + t
• next tab Cmd + Shift + ]
• previous tab Cmd + Shift + [

Pane navigation

• split pane left-right Cmd + d
• split pane top-bottom Cmd + Shift + d
• next pane Cmd + ]
• previous pane Cmd + [

Search

• open search bar Cmd + f
• find next Cmd + g

Input to all panes

• input to all panes in current tab Cmd + Alt + i

Clear screen

• clear buffer Cmd + k
• clear lines (Bash command) Ctrl + l

Zooming / Font Resize

• toggle maximize window Cmd + Alt + =
• toggle full screen Cmd + Enter
• make font larger Cmd + +
• make font smaller Cmd + -

iTerm lovers, did I miss anything out?

Last night, I spent some time to read about “Command Line Editing” in the bash manual. The bash manual is a well-written piece of documentation. I think I should read it more often.

Well, here’s the new shortcuts I learned:

Basic moves

• Move back one character. Ctrl + b
• Move forward one character. Ctrl + f
• Delete current character. Ctrl + d
• Delete previous character. Backspace
• Undo. Ctrl + -

Moving faster

• Move to the start of line. Ctrl + a
• Move to the end of line. Ctrl + e
• Move forward a word. Meta + f (a word contains alphabets and digits, no symbols)
• Move backward a word. Meta + b
• Clear the screen. Ctrl + l

What is Meta? Meta is your Alt key, normally. For Mac OSX user, you need to enable it yourself. Open Terminal > Preferences > Settings > Keyboard, and enable Use option as meta key. Meta key, by convention, is used for operations on word.

Cut and paste (‘Kill and yank’ for old schoolers)

• Cut from cursor to the end of line. Ctrl + k
• Cut from cursor to the end of word. Meta + d
• Cut from cursor to the start of word. Meta + Backspace
• Cut from cursor to previous whitespace. Ctrl + w
• Paste the last cut text. Ctrl + y
• Loop through and paste previously cut text. Meta + y (use it after Ctrl + y)
• Loop through and paste the last argument of previous commands. Meta + .

Search the command history

• Search as you type. Ctrl + r and type the search term; Repeat Ctrl + r to loop through results.
• Search the last remembered search term. Ctrl + r twice.
• End the search at current history entry. Ctrl + j
• Cancel the search and restore original line. Ctrl + g

Need more?

• A comprehensive bash editing mode cheatsheet by Peteris Krumin (catonmat.net).
• Vim users! Do you know you can switch to Vi-style editing mode? Here: vi-style cheatsheet.
• Bash command line editing is actually handled by GNU Readline Library. So just dive into Readline manual for everything else.

The app has a simple feature: a user can add some events, then invite other users to attend the event. Its data are represented in 3 models: User, Event, and Event Guest.

Let say, we are going to add a read-only JSON web API for clients to browse data records from the app.

Problems

Model is not view

When working on a non-trivial web API, you will soon realize that, model often cannot be serialized directly in web API.

Within the same app, one API may need to render a summary view of the model, while another needs a detail view of the same model. You want to serialize a view or view object, not a model.

RABL (Ruby API Builder Language) gem is designed for this purpose.

Define once, reuse everywhere

Let say, we need to render these user attributes: id, username, email, display_name, except password.

In RABL, we can define the attribute whitelist in a RABL template.

# tryrabl/app/views/users/base.rabl
attributes :id, :username, :email, :display_name

To show individual user, we can now reuse the template through RABL extends.

# tryrabl/app/views/users/show.rabl
extends "users/base"
object @user

## JSON output:
# {
#     "user": {
#         "id": 8,
#         "username": "blaise",
#         "email": "matteo@wilkinsonhuel.name",
#         "display_name": "Ms. Noe Lowe"
#     }
# }

Here’s another example to show a list of users.

# tryrabl/app/views/users/index.rabl
extends "users/base"
collection @users

## JSON output:
# [{
#     "user": {
#         "id": 1,
#         "username": "alanna",
#         "email": "rubie@hayes.name",
#         "display_name": "Mrs. Gaylord Hoeger"
#     }
# }, {
#     "user": {
#         "id": 2,
#         "username": "jarrell.robel",
#         "email": "jarod@eichmann.com",
#         "display_name": "Oran Lebsack"
#     }
# }]

The template can be reused in nested child as well, through RABL child.

attributes :id, :title, :description, :start, :end, :location
child :creator => :creator do
  extends 'users/base'
end

## JSON output:
# {
#     "event": {
#         "id": 7,
#         "title": "Et earum sed fuga.",
#         "description": "Quis sed ..e ad.",
#         "start": "2011-05-31T08:31:45Z",
#         "end": "2011-06-01T08:31:45Z",
#         "location": "Saul Tunnel",
#         "creator": {
#             "id": 1,
#             "username": "alanna",
#             "email": "rubie@hayes.name",
#             "display_name": "Mrs. Gaylord Hoeger"
#         }
#     }
# }

Join table rendered as subclass

I notice a recurring pattern in two recent projects. For instance, in this example, from client’s point of view, Event Guest is basically a User with an additional attribute: RSVP status.

When query database, usually we need to query the join table: event_guests.

class GuestsController < ApplicationController
  def index
    @guests = EventGuest.where(:event_id => params[:event_id])
  end
end

But when rendering, the result set needs to be rendered as a list of Users. RABL allows you to do that easily, using its glue feature (a weird name though :).

# tryrabl/app/views/guests/index.rabl
collection @event_guests

# include the additional attribute
attributes :rsvp

# add child attributes to parent model
glue :user do
  extends "users/base"
end

## JSON output:
# [{
#     "event_guest": {
#         "rsvp": "PENDING",
#         "id": 3,
#         "username": "myrna_harvey",
#         "email": "shad.armstrong@littelpouros.name",
#         "display_name": "Savion Balistreri"
#     }
# }, {
#     "event_guest": {
#         "rsvp": "PENDING",
#         "id": 4,
#         "username": "adelle.nader",
#         "email": "brendon.howe@cormiergrady.info",
#         "display_name": "Edgardo Dickens"
#     }
# }]

I think I will use RABL for the next Rails web API project.

The complete Rails example code is available at github.com/teohm/tryrabl.

Movitation

After switching to Git from Subversion and Mercurial for a few months, somehow I feel that Git is fundamentally different from Subversion or Mercurial, but couldn’t really tell the differences. I often see terms like tree, parent etc. in GitHub, which I have no idea what they actually mean.

So I decided to spent some time to learn Git.

I will try to summarize and publish important stuffs I learned about Git along the way.. but here is the first entry, about Git internals, which helped me to answer how Git is different other source control tools.

Objects, References, The Index

To understand the core of Git internals, there are 3 things to we should know: objects, references, the index.

I find this model is elegant. It fits well in a small diagram, as well as in my head.

Objects

All files that you commited into a Git repository, including the commit info are stored as objects in .git/objects/.

An object is identified by a 40-character-long string – SHA1 hash of the object’s content.

There are 4 types of objects:

1. blob - stores file content.
2. tree - stores direcotry layouts and filenames.
3. commit - stores commit info and forms the Git commit graph.
4. tag - stores annotated tag.

The example will illustrate how these objects relate to each others.

References

A branch, remote branch or a tag (also called lightweight tag) in Git, is just a pointer to an object, usually a commit object.

They are stored as plain text files in .git/refs/.

Symbolic References

Git has a special kind of reference, called symbolic reference. It doesn’t point to an object directly. Instead, it points to another reference.

For instance, .git/HEAD is a symbolic reference. It points to the current branch you are working on.

The Index

The index is a staging area, stored as a binary file in .git/index.

When git add a file, Git adds the file info to the index. When git commit, Git only commits what’s listed in the index.

Examples

Let’s walkthrough a simple example, to create a Git repository, commit some files and see what happened behind the scene in .git directory.

Initialize New Repository

$ git init canai

What happened:

• Empty .git/objects/ and .git/refs/ created.
• No index file yet.
• HEAD symbolic reference created.

  $ cat .git/HEAD 
  ref: refs/heads/master
  

Add New File

$ echo "A roti canai project." >> README
$ git add README

What happened:

• Index file created.
  It has a SHA1 hash that points to a blob object.

  $ git ls-files --stage
  100644 5f89c6f016cad2d419e865df380595e39b1256db 0 README
  

• Blob object created.
  The content of README file is stored in this blob.

  # .git/objects/5f/89c6f016cad2d419e865df380595e39b1256db
  $ git cat-file blob 5f89c6
  A roti canai project.
  

First Commit

$ git commit -m'first commit'
[master (root-commit) d9976cf] first commit
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 README

What happened:

• Branch ‘master’ reference created.
  It points to the lastest commit object in ‘master’ branch.

  $ cat .git/refs/heads/master 
  d9976cfe0430557885d162927dd70186d0f521e8
  

• First commit object created.
  It points to the root tree object.

  # .git/objects/d9/976cfe0430557885d162927dd70186d0f521e8
  $ git cat-file commit d9976cf
  tree 0ff699bbafc5d17d0637bf058c924ab405b5dcfe
  author Huiming Teo <huiming@favoritemedium.com> 1306739524 +0800
  committer Huiming Teo <huiming@favoritemedium.com> 1306739524 +0800
  
  first commit
  

• Tree object created.
  This tree represents the ‘canai’ directory.

  # .git/objects/0f/f699bbafc5d17d0637bf058c924ab405b5dcfe
  $ git ls-tree 0ff699
  100644 blob 5f89c6f016cad2d419e865df380595e39b1256db  README
  

Add Modified File

$ echo "Welcome everyone." >> README
$ git add README

What happened:

• Index file updated.
  Notice it points to a new blob?

  $ git ls-files --stage
  100644 1192db4c15e019da7fc053225d09dea14bc3ac07 0 README
  

• Blob object created.
  The entire README content is stored as a new blob.

  # .git/objects/11/92db4c15e019da7fc053225d09dea14bc3ac07
  $ git cat-file blob 1192db
  A roti canai project.
  Welcome everyone.
  

Add File into Subdirectory

$ mkdir doc
$ echo "[[TBD]] manual toc" >> doc/manual.txt
$ git add doc

What happened:

• Index file updated.

  $ git ls-files --stage
  100644 1192db4c15e019da7fc053225d09dea14bc3ac07 0 README
  100644 ea283e4fb22719fad512405d41dffa050cd16f9a 0 doc/manual.txt
  

• Blob object created.

  # .git/objects/ea/283e4fb22719fad512405d41dffa050cd16f9a
  $ git cat-file blob ea283
  [[TBD]] manual toc
  

Second Commit

$ git commit -m'second commit'
[master 556eaf3] second commit
 2 files changed, 2 insertions(+), 0 deletions(-)
 create mode 100644 doc/manual.txt

What happened:

• Branch ‘master’ reference updated.
  It points to a lastest commit in this branch.

  $ cat .git/refs/heads/master 
  556eaf374886d4c07a1906b9fdcaba195292b96
  

• Second commit object created. Notice its ‘parent’ points to the first commit object. This forms a commit graph.

  $ git cat-file commit 556e
  tree 7729a8b15b747bce541a9752a8f10d57daf221b6
  parent d9976cfe0430557885d162927dd70186d0f521e8
  author Huiming Teo <huiming@favoritemedium.com> 1306743598 +0800
  committer Huiming Teo <huiming@favoritemedium.com> 1306743598 +0800
  
  second commit
  

• New root tree object created.

  $ git ls-tree 7729
  100644 blob 1192db4c15e019da7fc053225d09dea14bc3ac07  README
  040000 tree 6ff17d485bf857514f299f0bde0e2a5c932bd055  doc
  

• New subdir tree object created.

  $ git ls-tree 6ff1
  100644 blob ea283e4fb22719fad512405d41dffa050cd16f9a  manual.txt
  

Add Annotated Tag

$ git tag -a -m'this is annotated tag' v0.1 d9976

What happened:

• Tag reference created.
  It points to a tag object.

  $ cat .git/refs/tags/v0.1 
  c758f4820f02acf20bb3f6d7f6098f25ee6ed730
  

• Tag object created.

  $ git cat-file tag c758
  object d9976cfe0430557885d162927dd70186d0f521e8
  type commit
  tag v0.1
  tagger Huiming Teo <huiming@favoritemedium.com> 1306744918 +0800
  
  this is annotated tag
  

Add new (lightweight) tag

$ git tag root-commit d9976

What happened:

• Tag reference created.
  It points to a commit object.

  $ cat .git/refs/tags/root-commit 
  d9976cfe0430557885d162927dd70186d0f521e8
  

More Readings

• “Chapter 7: Internals and Plumbing” in The Git Community Book.
• “Chapter 9: Git Internals” in Pro Git.

What’s Next?

Looking for a minimal git workflow suitable for a distributed team, long-running project..

After experimenting with the latest jquery-ujs and making an embarrassing mistake, it turns out that the issue is resolved in the latest version.

(Mistake: You may notice I removed a previous post about this topic, where I mistakenly concluded the latest jquery-ujs is not working with JQuery Validation. Thanks to JangoSteve for pointing it out. The post was misleading, so I believe it’s best to remove it to avoid confusion. :-)

Get the latest jquery-ujs

There are 2 reasons to use the latest jquery-ujs:

1. it has a patch that fixes the issue (see issue #118).
2. it exposes an internal function that we may need – $.rails.handleRemote() (see more details)

Working example

The example is tested with:

• JQuery 1.6.1
• JQuery Vaidation 1.8.1
• Latest Rails UJS (commit: dad6982dc592686677e6)

When using submitHandler in JQuery Validation

If you are using JQuery Validation’s submitHandler(form) function, you need to call $.rails.handleRemote( $(form) ) function manually, so that it submits the form via XHR.

$('#submit_form').validate({

  submitHandler: function(form) {
    // .. do something before submit ..
    $.rails.handleRemote( $(form) );  // submit via xhr
    //form.submit();                  // don't use, it submits the form directly
  }

});

def create
  @comment = Comment.new(params[:comment])
  ..
end

def update
  ..
  @comment.update_attributes(params[:comment])
  ..
end

Why knowing mass assignment is important?

By default, mass assignment opens up an undesirable security hole, by allowing web clients to update any attributes they passing in, including attributes like created_at.

For details, you can read:

• Ruby On Rails Security Guide - Mass Assignment
• Rail Spikes: Is your Rails application safe?

Mass assignment methods

There are a few ActiveRecord methods that accept mass assignment:

• new
• create
• attributes=
• update_attributes

Using any of these methods, means you are now responsible for the safety of the web application.

Minimal protection

To use mass assignment safely, we want to specify exactly which attributes allowed to be updated.

1. Define attr_accessible on every model

attr_accessible defines a white-list of attributes that can be updated during mass assignment.

class Comment < ActiveRecord::Base
  attr_accessible :title, :content
end

2. Enforce attr_accessible usage

You may set the white-list default to empty. This forces you to define the whitelist explicitly on a model, before mass assignment can be used.

# config/initializer/enforce_attr_accessible.rb

ActiveRecord::Base.send(:attr_accessible, nil)

if user.id == current_user.id
  # do something
end

Use obj1 == obj2 instead

Of course, that reflects my ignorance on learning the new tool.

So, here’s a proper way to compare if two model objects are equal:

test "2 records with same #id equal to each others." do
  c1 = Comment.create
  c2 = Comment.find(c1.id)

  assert c1 == c2
  assert c2 == c1
end

What about new record?

Basically, a new model object only equals to itself:

test "New record does not equal to another new record." do
  new_record = Comment.new
  another_new = Comment.new

  assert new_record == new_record
  assert another_new == another_new
  assert new_record != another_new
end

Learning to use == operator for checking model equality is useful. Because now, we can start using other methods that depend on == operator. For instance, Array#include?.

test "When applied in Array" do
  c1 = Comment.create
  c2 = Comment.find(c1.id)

  assert [c1].include? c2
  assert [c2].include? c1
  assert [c1, c2] == [c1, c2]
end

Rails learning tests

You may notice the code examples are written as tests. This is my first Rails learning test. So the plan is to write more tests as a way to learn Rails. I will post them here occationally if interesting enough to share.

Source code: model_equality_test.rb