teohm.dev

Chef cookbooks for busy Ruby developers

2013-04-17T23:06:00+08:00

Have you ever setup a Rails production environment from scratch, by hand? If you had, I share your pain every time when a new project started.

The process is often repetitive. To me, it seems to be a waste to do it manually every time. It also consumes time and attention. It would be great if I could spend them on tasks that bring more values to clients.

To minimize such waste, I have written two Chef cookbooks to automate the process:

rackbox - to provision rack-based web server (Nginx as front server, Unicorn and Passenger as upstream app servers, rbenv as ruby version manager).
databox - to provision database server (supports MySQL and PostgreSQL).

Getting started

In this post, I will show you a step-by-step guide on how to use the cookbooks together with knife-solo to provision a remote server in 4 steps:

setup Chef Solo environment
modify config file
provision remote server
tweak Capistrano deploy.rb

A working example in also available at teohm/kitchen-example.

1. Setup Chef Solo environment

Install Chef Solo tools on local machine.
Download Chef cookbooks to local machine.
Install chef-solo on remote server.

Install Chef Solo tools

Let’s create a new directory,

mkdir chef-kitchen
cd chef-kitchen

and a Gemfile.

source "https://rubygems.org"

gem "knife-solo", ">= 0.3.0pre3"
gem "berkshelf"

I recommend knife-solo >= 0.3 as it includes a few major fixes and improvements.

Now, install the ruby gems.

bundle install

Finally, setup a kitchen directory structure with knife-solo.

bundle exec knife solo init .

Download Chef cookbooks

I use Berkshelf to manage cookbooks. So we need a Berksfile,

site :opscode

cookbook "runit", ">= 1.1.2"  # HACK: force-use this version
cookbook "databox"
cookbook "rackbox"

(I added a hack here to force berkshelf to use runit 1.1.2 required by rackbox. Still looking for a better solution.)

We can now download cookbooks with berks install.

bundle exec berks install --path cookbooks/

Install `chef-solo` on remote server

bundle exec knife solo prepare testbox

In this example, testbox is a host I setup in my ~/.ssh/config:

Host testbox
  User ubuntu
  Hostname ec2-51-221-13-121.ap-southeast-1.compute.amazonaws.com
  IdentityFile ~/.ssh/testbox_ec2.pem

2. Customize config file

Download config example
Customize config file

Download config example

curl https://raw.github.com/teohm/kitchen-example/master/nodes/host.json.example --output nodes/testbox.json

Modify config file (JSON)

The config file starts with a run_list. You specify a list of cookbook recipes here. Chef will run them in the same order in this list.

It is followed by cookbook attributes. You can modify these attributes. A full reference of attributes are described in each cookbook’s README (see appbox, databox, rackbox).

{
  "run_list": [
    "databox",
    "rackbox"
  ],
  "appbox": {
    "deploy_keys": ["ssh-rsa 5bnmu23890fghghjk"],
    "admin_keys": ["ssh-rsa 456789fghjkvbn567"]
  },
  "databox": {
    "db_root_password": "welcome!",
    "databases": {
      "mysql": [
        { "username": "app1",
          "password": "app1",
          "database_name": "app1_production" }
      ],
      "postgresql": [
        { "username": "app2",
          "password": "app2",
          "database_name": "app2_production" }
      ]
    }
  },
  "rackbox": {
    "ruby": {
      "versions": ["1.9.3-p385", "1.9.2-p320"],
      "global_version": "1.9.3-p385"
    },
    "apps": {
      "unicorn": [
        { "appname": "app1",
          "hostname": "app1.test.com" }
      ],
      "passenger": [
        { "appname": "app2",
          "hostname": "app2.test.com" }
      ]
    }
  }
}

3. Provision remote server

bundle exec knife solo cook testbox

It uploads the kitchen directory and runs chef-solo on the remote server. Chef-solo will then takeover and execute the run list to setup everything.

What do we get at this point?

Basically, it’s done!

We have a full-stack, rack-based server with:

3 user accounts (deploy, devops, apps)
rbenv as ruby version manager
nginx as front-server
unicorn, passenger-standalone as upstream app servers, managed by runit
postgresql, mysql installed and databases created
all apps will be stored in /home/apps/

4. Tweak Capistrano deploy.rb

Now, it’s ready to deploy a Rack-based app to the remote server!

I have two example Rails apps available on Github:

teohm/sample-app1 runs on unicorn,
teohm/sample-app2 runs on passenger-standalone.

There are a few minor tweaks required in Capistrano deploy.rb, as listed below.

I will explain the tweaks in details next time. Meanwhile, check out the complete working examples at: app1/config/deploy.rb and app2/config/deploy.rb

Login as `deploy` user

set :user, "deploy"

Deploy to `/home/apps`

set :deploy_to, "/home/apps/#{application}"

Load `rbenv` in Capistrano

default_run_options[:shell] = '/bin/bash --login'

Run bundler with `--binstubs`

require 'bundler/capistrano'
set :bundle_flags, "--deployment --binstubs"
set :bundle_without, [:test, :development, :deploy]

Restart app with `runit sv`

namespace :deploy do
  task :start do
    run "sudo sv up app1"
  end
  task :stop do
    run "sudo sv down app1"
  end
  task :restart, :roles => :app, :except => { :no_release => true } do
    run "sudo sv restart app1"
  end
end

Feedback

If you are interested on using the cookbooks, or have an idea/feedback/question about this topic, feel free to drop me (@teohm) a message. Pull requests and issue reports are definitely welcomed!

Reload required files in Rails

2013-01-10T12:34:00+08:00

When a Rails project grows, I often notice the need to refactor domain logic into a separate module, isolated from Rails framework.

The isolated module will still stay in the main Rails app codebase, but can be easily packaged as a Ruby gem, tested separately, and used in other related applications.

Requirements

During the refactoring, we want to:

Use require to load the module like a normal gem. If possible, no dependency on Rails autoload features such asrequire_dependency and autoload_paths.
Edit the module without restarting server during development. In other words, we need to find a way to reload the module on every request.

Reload `require` files

After some research, I found a working solution by Timothy Cardenas:

ActionDispatch::Callbacks.to_prepare do
  if Object.const_defined?("Module1")
    Object.send(:remove_const, "Module1")
  end
  $".delete_if {|s| s.include?('module1') }
  require 'module1'
end

What it does is, before each request,

Unload the top-level module.
Un-require all required files from the module.
Re-require the top-level module.

An extra step in Rails 3.2

In Rails 3.2, ActionDispatch::Callbacks.to_prepare has a slightly different behavior. It will run before a request only if a watchable file is modified.

You need to specify your own watchable files:

# watch all .rb files recursively under modules/module1/ dir
config.watchable_dirs['modules/module1'] = [:rb]

Introduce `require_reloader` gem

Before bundling the solution into a gem, I did a search on RubyGems.org and found Colin Young’s gem_reloader. It’s based on Timy’s solution as well. I forked it and started playing around.

In the end, I made some major changes to include Rails 3.2 support, some fixes and new features.

So I decided to release it as a new Ruby gem: require_reloader.

# config/environments/development.rb
YourApp::Application.configure do
  ...
  RequireReloader.watch :module1,
    path: 'modules/module1'
end

Currently it supports Rails 3, including 3.1 and 3.2.

If you are working on something similar, looking forward for your feedbacks and pull requests. It is not tested on Rails 2 yet, so pull requests are definitely welcomed!

Start Using Ruby % (Percent) Notation

2012-10-15T18:04:00+08:00

If you seldom use Ruby percent (%) notation in daily work, here’s a quick summary of what I picked up recently.

Delimiter allows any non-alphanum

You can use any non alpha-numeric character as delimiter:

%(any alpha-numeric)
%[char can be]
%%used as%
%!delimiter\!! # escape '!' literal

Bracket pairs no need to escape

No need to escape bracket pairs, even when nested. You can escape, but will need to escape both open and close bracket.

%( (pa(re(nt)he)sis) ) #=> "(pa(re(nt)he)sis)"
%[ [square bracket] ]  #=> "[square bracket]"
%{ {curly bracket} }   #=> "{curly bracket}"
%< <pointy bracket> >  #=> "<pointy bracket>"
%< \<this works as well\> >  #=> "<this works as well>"

Modifiers for String, Regex, Array, Symbol, Shell command

We often use % notation to create String and Array literals. But it also supports Symbol, Regex and shell command.

%(interpolated string (#{ "default" }))
  #=> "interpolated string (default)"
%Q(interpolated string (#{ "default" }))
  #=> "interpolated string (default)"
%q(non-interpolated string)
  #=> "non-interpolated string"
%r(#{ "interpolated" } regexp)i
  #=> /interpolated regexp/i
%w(non-interpolated\ string  separated\ by\ whitespaces)
  #=> ['non-interpolated string', 'separated by whitespaces']
%W(interpolated\ string #{ "separated by whitespaces" })
  #=> ['interpolated string', 'separated by whitespaces']
%s(non-interpolated symbol)
  #=> :'non-interpolated symbol'
%x(echo #{ "interpolated shell command" })
  #=> "interpolated shell command\n"

Here’s some % notation examples written in minitest in case you interested.

Ruby idiom to ensure variable is Array

2012-10-02T23:27:00+08:00

Stop doing this:

arry = input || []  # handle input == nil
arry = [input] unless arry.kind_of?(Array)  # handle single value object

arry.each do |item|
  #process item
end

In Ruby, you should use Kernel#Array to convert the variable into an array object:

Array(input).each do |item|
  # process item
end

# Array(nil)     # => []
# Array("foo")   # => ["foo"]
# Array([1,2,3]) # => [1,2,3]

More usage examples written in minitest.

Objective-C Collection Operators

2012-09-24T13:52:00+08:00

Calculate average - a shorter way

When you have a collection of Transactions objects and want to calculate its average amount, instead of looping through the collection like this:

double sum = 0;
for (Transaction *transaction in transactions) {
    sum += [transaction.amount doubleValue];
}
NSNumber *avg = [NSNumber numberWithDouble:(sum / [transactions count])];

you can reduce the loop to 1 line of code, using Objective-C key-value coding:

NSNumber *avg = [transactions valueForKeyPath:@"@avg.amount"];

Currently, there is a fixed set of collection operators:

Simple collection operators (@avg, @count, @sum, @max, @min)
Object operators (@distinctUnionOfObjects, @unionOfObjects)
Array and set operators (@distinctUnionOfArrays, @unionOfArrays)

For details, refer to more usage examples.

Breaking ARC retain cycle in Objective-C blocks

2012-09-03T17:50:00+08:00

In a recent client project, we noticed its iOS app often received low memory warning. The iOS app is developed with ARC-enabled (Automatic Reference Counting).

When profiled the app using Instruments > Allocations, we found that a lot of unused ViewController objects were not released from memory.

Retain Cycle

The codebase uses a lot of Objective-C blocks as shown below, and self is often being called within the blocks:

@interface DetailPageViewController : UIViewController
@property(nonatomic, strong) UIButton *backButton;
...
@end

@implementation DetailPageViewController
@synthesize backButton;

- (void)loadView {
  ...
  [self.doneButton onTouch:^(id sender) {
    [self doSomething];
    self.isDone = YES;
  }];
}
...
@end

In this example code, the controller object holds a strong reference to doneButton object. But because doneButton onTouch: block is calling self, now the button object holds a strong reference back to the controller object.

When object A points strongly to object B, and B points strongly to A, a retain cycle is created, and both objects cannot be released from memory.

Use Lifetime Qualifiers

Apple Developer’s guide on ARC transition suggested a few ways to break the retain cycle using different lifetime qualifiers. It is definitely a must read for iOS development.

If your app is targeted for iOS 5, you can use __weak lifetime qualifier to break the cycle:

@interface DetailPageViewController : UIViewController
- (void)loadView {
  ...
  __weak DetailPageViewController *controller = self;
  [self.doneButton onTouch:^(id sender) {
    [controller doSomething];
    controller.isDone = YES;
  }];
}
@end

Because we are using __weak qualifier, doneButton onTouch: block only has a weak reference to the controller object. Now, the controller object can be released from memory when its reference count drops to 0.

Mac Tips: Use Caps Lock as Control key

2012-04-08T10:20:00+08:00

After using tmux, I start using Control key a lot often. I decided to move my Control key up to the home row, by assigning it to Caps Lock.

For Mac users, open System Preferences > Keyboard, choose Keyboard tab. Click button Modifier Keys.. and change Caps Lock Key: to ^ Control.

Mac Tips: Change default application for a file type

2012-04-07T18:46:00+08:00

To switch the default application for file type, right-click the file, select Get Info and change the application in Open With section.

Remember, remember.. click Change All… button to apply the changes to all files. It seems obvious, but took me a while to figure this out. :)

Mac Tips: Turn on remote login (SSH server)

2012-04-07T15:12:00+08:00

Enable SSH server on Mac OSX is surprisingly easy. Open System Preferences > Sharing, and turn on Remote Login. You can futher restrict which user can login via SSH on the same screen.

Using sshuttle in daily work

2012-04-01T13:13:00+08:00

I was first introduced to sshuttle by Sooyoung (@5ooyoung) in Favorite Medium as a workaround to The Great Firewall in China.

Since then, it has become my light-weight network tunneling tool in daily work.

Install sshuttle

The installation is easy now. You can install it through Mac OSX Homebrew, or Ubuntu apt-get.

brew install sshuttle

I use sshuttle to..

1. Tunnel all traffic

This is the first command I learned. It forwards all TCP traffic and DNS requests to a remote SSH server.

sshuttle --dns -vr ssh_server 0/0

Just like ssh, you can use any server specified in ~/.ssh/config. The -v flag means verbose mode.

Besides TCP and DNS, currently sshuttle does not forward other requests such as UDP, ICMP ping etc.

2. Tunnel all traffic, but exclude some

You can exclude certain TCP traffic using -x option.

sshuttle --dns -vr ssh_server -x 121.9.204.0/24 -x 61.135.196.21 0/0

For instance, when I am in China, I don’t want to tunnel Youku.com traffic to a foreign server, because its movie streaming service is only available within China.

In this case, I use -x option to exclude Youku.com IP addresses.

3. Tunnel only certain traffic

To tunnel only certain TCP traffic, specify the IP addresses or IP ranges that need tunneling.

sshuttle -vr ssh_server 121.9.204.0/24 61.135.196.21

This command comes in handy, whenever I need to test an app feature (e.g. Netflix movie streaming) which only available in certain countries, or to bypass ISP faulty caches.

4. VPN to office network

I seldom do VPN, but all you need is the remote SSH server with -NH flags turned on.

sshuttle -NHvr office_ssh_server

-N flag tells sshuttle to figure out by itself the IP subnets to forward, and -H flag to scan for hostnames within remote subnets and store them temporarily in /etc/hosts.

IP addresses.. troublesome?

Well, I try not to deal with IP addresses manually. So I wrote a few sshuttle helpers (tnl, tnlbut, tnlonly, vpnto) that allow me to use domain names instead of IP addresses:

Tunnel all traffic

tnl

Tunnel all traffic, but exclude some

tnlbut youku.com weibo.com

Tunnel only certain traffic

tnlonly netflix.com movies.netflix.com

VPN to office network

vpnto office_ssh_server

The script is available on my GitHub repo. You can load it into your ~/.bashrc. To override the default tunneling SSH server in the script:

TNL_SERVER=user@another_server tnl

UTF-8 param name issue in Rails multipart form

2012-03-27T21:36:00+08:00

I first stumbled upon this issue when Yasith (@meaningful) showed me a strange bug in a Rails project. Here’s what happened:

Issue

When submit a multipart form that contains Unicode parameter name e.g.

<form method="post" enctype="multipart/form-data" action="">
  <input name="Iñtërnâtiônàlizætiøn_name"
         value="Iñtërnâtiônàlizætiøn_value" />
</form>

Rails controller returns the param value "Iñtërnâtiônàlizætiøn_value" as expected.

But the param name becomes: "I\xC3\xB1t\xC3\xABrn\xC3\xA2ti\xC3\xB4n\xC3\xA0liz\xC3\xA6ti\xC3\xB8n_name".

It makes life miserable, if you are not expecting this to happen:

params["Iñtërnâtiônàlizætiøn_name"] # => nil
params["I\xC3\xB1t\xC3\xABrn\xC3\xA2ti\xC3\xB4n\xC3\xA0liz\xC3\xA6ti\xC3\xB8n_name"] # => "Iñtërnâtiônàlizætiøn_value"

What happened?

When Rack returns multipart form data to Rails, it returns:

{ "I\xC3\xB1t\xC3\xABrn\xC3\xA2ti\xC3\xB4n\xC3\xA0liz\xC3\xA6ti\xC3\xB8n_name" =>
  "I\xC3\xB1t\xC3\xABrn\xC3\xA2ti\xC3\xB4n\xC3\xA0liz\xC3\xA6ti\xC3\xB8n_value" }

However, ActionDispatch::Http::Parameters#encode_params in Rails decided to only encode parameter values, but not parameter names. As a result, we get:

{ "I\xC3\xB1t\xC3\xABrn\xC3\xA2ti\xC3\xB4n\xC3\xA0liz\xC3\xA6ti\xC3\xB8n_name" =>
  "Iñtërnâtiônàlizætiøn_value" }

Solutions?

Don’t use Unicode param name.
Patch Rails source code. I added a fix in my forked branch, and reported the issue. Hopefully it will get fixed soon in the coming release.

Working effectively with iTerm2

2012-03-22T22:42:00+08:00

I have been using iTerm in daily work for almost a year now. Along the way, I learned a few handy settings tweaks and shortcut keys to boost my productivity in command-line environment.

Install iTerm2

If you haven’t heard of iTerm, it’s a popular open source alternative to Mac OS X Terminal. Give it a try, download and install it from http://www.iterm2.com.

Fine-Tune Settings

Launch iTerm, open iTerm > Preferences or just Cmd + ,.

Open tab/pane with current working directory

Under Profiles tab, go to General subtab, set Working Directory to “Reuse previous session’s directory”.

Enable `Meta` key

To enable Meta key for Bash readline editing e.g. Alt + b to move to previous word, under Profiles tab, go to Keys subtab, set Left option key acts as: to “+Esc”.

Hotkey to toggle iTerm2

Under Keys tab, in Hotkey section, enable “Show/hide iTerm2 with a system-wide hotkey” and input your hotkey combination, e.g. I use Ctrl + Shift + L.

Switch pane with mouse cursor

Under Pointer, in Miscellaneous Settings section, enable “Focus follows mouse”.

Handy Shortcut Keys

Here’s a set of shortcut keys I commonly use. You can always look for other shortcut keys in the iTerm menu.

Tab navigation

open new tab Cmd + t
next tab Cmd + Shift + ]
previous tab Cmd + Shift + [

Pane navigation

split pane left-right Cmd + d
split pane top-bottom Cmd + Shift + d
next pane Cmd + ]
previous pane Cmd + [

Search

open search bar Cmd + f
find next Cmd + g

Input to all panes

input to all panes in current tab Cmd + Alt + i

Clear screen

clear buffer Cmd + k
clear lines (Bash command) Ctrl + l

Zooming / Font Resize

toggle maximize window Cmd + Alt + =
toggle full screen Cmd + Enter
make font larger Cmd + +
make font smaller Cmd + -

iTerm lovers, did I miss anything out?

Shortcuts to move faster in Bash command line

2012-01-04T20:13:00+08:00

Nowadays, I spend more time in Bash shell, typing longer commands. One of my new year resolutions for this year is to stop using left/right arrow keys to move around in the command line. I learned a few shortcuts a while ago.

Last night, I spent some time to read about “Command Line Editing” in the bash manual. The bash manual is a well-written piece of documentation. I think I should read it more often.

Well, here’s the new shortcuts I learned:

Basic moves

Move back one character. Ctrl + b
Move forward one character. Ctrl + f
Delete current character. Ctrl + d
Delete previous character. Backspace
Undo. Ctrl + -

Moving faster

Move to the start of line. Ctrl + a
Move to the end of line. Ctrl + e
Move forward a word. Meta + f (a word contains alphabets and digits, no symbols)
Move backward a word. Meta + b
Clear the screen. Ctrl + l

What is Meta? Meta is your Alt key, normally. For Mac OSX user, you need to enable it yourself. Open Terminal > Preferences > Settings > Keyboard, and enable Use option as meta key. Meta key, by convention, is used for operations on word.

Cut and paste (‘Kill and yank’ for old schoolers)

Cut from cursor to the end of line. Ctrl + k
Cut from cursor to the end of word. Meta + d
Cut from cursor to the start of word. Meta + Backspace
Cut from cursor to previous whitespace. Ctrl + w
Paste the last cut text. Ctrl + y
Loop through and paste previously cut text. Meta + y (use it after Ctrl + y)
Loop through and paste the last argument of previous commands. Meta + .

Search the command history

Search as you type. Ctrl + r and type the search term; Repeat Ctrl + r to loop through results.
Search the last remembered search term. Ctrl + r twice.
End the search at current history entry. Ctrl + j
Cancel the search and restore original line. Ctrl + g

Need more?

A comprehensive bash editing mode cheatsheet by Peteris Krumin (catonmat.net).
Vim users! Do you know you can switch to Vi-style editing mode? Here: vi-style cheatsheet.
Bash command line editing is actually handled by GNU Readline Library. So just dive into Readline manual for everything else.

Using RABL in Rails JSON Web API

2011-05-31T00:00:00+08:00

Let’s use an event management app as the example.

The app has a simple feature: a user can add some events, then invite other users to attend the event. Its data are represented in 3 models: User, Event, and Event Guest.

Let say, we are going to add a read-only JSON web API for clients to browse data records from the app.

Problems

Model is not view

When working on a non-trivial web API, you will soon realize that, model often cannot be serialized directly in web API.

Within the same app, one API may need to render a summary view of the model, while another needs a detail view of the same model. You want to serialize a view or view object, not a model.

RABL (Ruby API Builder Language) gem is designed for this purpose.

Define once, reuse everywhere

Let say, we need to render these user attributes: id, username, email, display_name, except password.

In RABL, we can define the attribute whitelist in a RABL template.

# tryrabl/app/views/users/base.rabl
attributes :id, :username, :email, :display_name

To show individual user, we can now reuse the template through RABL extends.

# tryrabl/app/views/users/show.rabl
extends "users/base"
object @user

## JSON output:
# {
#     "user": {
#         "id": 8,
#         "username": "blaise",
#         "email": "matteo@wilkinsonhuel.name",
#         "display_name": "Ms. Noe Lowe"
#     }
# }

Here’s another example to show a list of users.

# tryrabl/app/views/users/index.rabl
extends "users/base"
collection @users

## JSON output:
# [{
#     "user": {
#         "id": 1,
#         "username": "alanna",
#         "email": "rubie@hayes.name",
#         "display_name": "Mrs. Gaylord Hoeger"
#     }
# }, {
#     "user": {
#         "id": 2,
#         "username": "jarrell.robel",
#         "email": "jarod@eichmann.com",
#         "display_name": "Oran Lebsack"
#     }
# }]

The template can be reused in nested child as well, through RABL child.

attributes :id, :title, :description, :start, :end, :location
child :creator => :creator do
  extends 'users/base'
end

## JSON output:
# {
#     "event": {
#         "id": 7,
#         "title": "Et earum sed fuga.",
#         "description": "Quis sed ..e ad.",
#         "start": "2011-05-31T08:31:45Z",
#         "end": "2011-06-01T08:31:45Z",
#         "location": "Saul Tunnel",
#         "creator": {
#             "id": 1,
#             "username": "alanna",
#             "email": "rubie@hayes.name",
#             "display_name": "Mrs. Gaylord Hoeger"
#         }
#     }
# }

Join table rendered as subclass

I notice a recurring pattern in two recent projects. For instance, in this example, from client’s point of view, Event Guest is basically a User with an additional attribute: RSVP status.

When query database, usually we need to query the join table: event_guests.

class GuestsController < ApplicationController
  def index
    @guests = EventGuest.where(:event_id => params[:event_id])
  end
end

But when rendering, the result set needs to be rendered as a list of Users. RABL allows you to do that easily, using its glue feature (a weird name though :).

# tryrabl/app/views/guests/index.rabl
collection @event_guests

# include the additional attribute
attributes :rsvp

# add child attributes to parent model
glue :user do
  extends "users/base"
end

## JSON output:
# [{
#     "event_guest": {
#         "rsvp": "PENDING",
#         "id": 3,
#         "username": "myrna_harvey",
#         "email": "shad.armstrong@littelpouros.name",
#         "display_name": "Savion Balistreri"
#     }
# }, {
#     "event_guest": {
#         "rsvp": "PENDING",
#         "id": 4,
#         "username": "adelle.nader",
#         "email": "brendon.howe@cormiergrady.info",
#         "display_name": "Edgardo Dickens"
#     }
# }]

I think I will use RABL for the next Rails web API project.

The complete Rails example code is available at github.com/teohm/tryrabl.

Learning Git Internals by Example

2011-05-30T00:00:00+08:00

Status: Draft.
Plan to revise this post, probably simplify it in future..

Movitation

After switching to Git from Subversion and Mercurial for a few months, somehow I feel that Git is fundamentally different from Subversion or Mercurial, but couldn’t really tell the differences. I often see terms like tree, parent etc. in GitHub, which I have no idea what they actually mean.

So I decided to spent some time to learn Git.

I will try to summarize and publish important stuffs I learned about Git along the way.. but here is the first entry, about Git internals, which helped me to answer how Git is different other source control tools.

Objects, References, The Index

To understand the core of Git internals, there are 3 things to we should know: objects, references, the index.

I find this model is elegant. It fits well in a small diagram, as well as in my head.

Objects

All files that you commited into a Git repository, including the commit info are stored as objects in .git/objects/.

An object is identified by a 40-character-long string – SHA1 hash of the object’s content.

There are 4 types of objects:

blob - stores file content.
tree - stores direcotry layouts and filenames.
commit - stores commit info and forms the Git commit graph.
tag - stores annotated tag.

The example will illustrate how these objects relate to each others.

References

A branch, remote branch or a tag (also called lightweight tag) in Git, is just a pointer to an object, usually a commit object.

They are stored as plain text files in .git/refs/.

Symbolic References

Git has a special kind of reference, called symbolic reference. It doesn’t point to an object directly. Instead, it points to another reference.

For instance, .git/HEAD is a symbolic reference. It points to the current branch you are working on.

The Index

The index is a staging area, stored as a binary file in .git/index.

When git add a file, Git adds the file info to the index. When git commit, Git only commits what’s listed in the index.

Examples

Let’s walkthrough a simple example, to create a Git repository, commit some files and see what happened behind the scene in .git directory.

Initialize New Repository

$ git init canai

What happened:

Empty .git/objects/ and .git/refs/ created.
No index file yet.

HEAD symbolic reference created.

$ cat .git/HEAD 
ref: refs/heads/master

Add New File

$ echo "A roti canai project." >> README
$ git add README

What happened:

Index file created.
It has a SHA1 hash that points to a blob object.

$ git ls-files --stage
100644 5f89c6f016cad2d419e865df380595e39b1256db 0 README

Blob object created.
The content of README file is stored in this blob.

# .git/objects/5f/89c6f016cad2d419e865df380595e39b1256db
$ git cat-file blob 5f89c6
A roti canai project.

First Commit

$ git commit -m'first commit'
[master (root-commit) d9976cf] first commit
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 README

What happened:

Branch ‘master’ reference created.
It points to the lastest commit object in ‘master’ branch.
```
$ cat .git/refs/heads/master 
d9976cfe0430557885d162927dd70186d0f521e8
```

First commit object created.
It points to the root tree object.

# .git/objects/d9/976cfe0430557885d162927dd70186d0f521e8
$ git cat-file commit d9976cf
tree 0ff699bbafc5d17d0637bf058c924ab405b5dcfe
author Huiming Teo <huiming@favoritemedium.com> 1306739524 +0800
committer Huiming Teo <huiming@favoritemedium.com> 1306739524 +0800

first commit

Tree object created.
This tree represents the ‘canai’ directory.

# .git/objects/0f/f699bbafc5d17d0637bf058c924ab405b5dcfe
$ git ls-tree 0ff699
100644 blob 5f89c6f016cad2d419e865df380595e39b1256db  README

Add Modified File

$ echo "Welcome everyone." >> README
$ git add README

What happened:

Index file updated.
Notice it points to a new blob?

$ git ls-files --stage
100644 1192db4c15e019da7fc053225d09dea14bc3ac07 0 README

Blob object created.
The entire README content is stored as a new blob.

# .git/objects/11/92db4c15e019da7fc053225d09dea14bc3ac07
$ git cat-file blob 1192db
A roti canai project.
Welcome everyone.

Add File into Subdirectory

$ mkdir doc
$ echo "[[TBD]] manual toc" >> doc/manual.txt
$ git add doc

What happened:

Index file updated.

$ git ls-files --stage
100644 1192db4c15e019da7fc053225d09dea14bc3ac07 0 README
100644 ea283e4fb22719fad512405d41dffa050cd16f9a 0 doc/manual.txt

Blob object created.

# .git/objects/ea/283e4fb22719fad512405d41dffa050cd16f9a
$ git cat-file blob ea283
[[TBD]] manual toc

Second Commit

$ git commit -m'second commit'
[master 556eaf3] second commit
 2 files changed, 2 insertions(+), 0 deletions(-)
 create mode 100644 doc/manual.txt

What happened:

Branch ‘master’ reference updated.
It points to a lastest commit in this branch.
```
$ cat .git/refs/heads/master 
556eaf374886d4c07a1906b9fdcaba195292b96
```

Second commit object created. Notice its ‘parent’ points to the first commit object. This forms a commit graph.

$ git cat-file commit 556e
tree 7729a8b15b747bce541a9752a8f10d57daf221b6
parent d9976cfe0430557885d162927dd70186d0f521e8
author Huiming Teo <huiming@favoritemedium.com> 1306743598 +0800
committer Huiming Teo <huiming@favoritemedium.com> 1306743598 +0800

second commit

New root tree object created.

$ git ls-tree 7729
100644 blob 1192db4c15e019da7fc053225d09dea14bc3ac07  README
040000 tree 6ff17d485bf857514f299f0bde0e2a5c932bd055  doc

New subdir tree object created.

$ git ls-tree 6ff1
100644 blob ea283e4fb22719fad512405d41dffa050cd16f9a  manual.txt

Add Annotated Tag

$ git tag -a -m'this is annotated tag' v0.1 d9976

What happened:

Tag reference created.
It points to a tag object.

$ cat .git/refs/tags/v0.1 
c758f4820f02acf20bb3f6d7f6098f25ee6ed730

Tag object created.

$ git cat-file tag c758
object d9976cfe0430557885d162927dd70186d0f521e8
type commit
tag v0.1
tagger Huiming Teo <huiming@favoritemedium.com> 1306744918 +0800

this is annotated tag

Add new (lightweight) tag

$ git tag root-commit d9976

What happened:

Tag reference created.
It points to a commit object.

$ cat .git/refs/tags/root-commit 
d9976cfe0430557885d162927dd70186d0f521e8

What’s Next?

Looking for a minimal git workflow suitable for a distributed team, long-running project..

Using JQuery Validation in Rails Remote Form

2011-05-25T00:00:00+08:00

In a recent project, I was trying to use JQuery Validation in an earlier version of Rails 3 remote form (jquery-ujs). They didn’t work out well in IE.

After experimenting with the latest jquery-ujs and making an embarrassing mistake, it turns out that the issue is resolved in the latest version.

(Mistake: You may notice I removed a previous post about this topic, where I mistakenly concluded the latest jquery-ujs is not working with JQuery Validation. Thanks to JangoSteve for pointing it out. The post was misleading, so I believe it’s best to remove it to avoid confusion. :-)

Get the latest `jquery-ujs`

There are 2 reasons to use the latest jquery-ujs:

it has a patch that fixes the issue (see issue #118).
it exposes an internal function that we may need – $.rails.handleRemote() (see more details)

Working example

The example is tested with:

JQuery 1.6.1
JQuery Vaidation 1.8.1
Latest Rails UJS (commit: dad6982dc592686677e6)

When using `submitHandler` in JQuery Validation

If you are using JQuery Validation’s submitHandler(form) function, you need to call $.rails.handleRemote( $(form) ) function manually, so that it submits the form via XHR.

$('#submit_form').validate({

  submitHandler: function(form) {
    // .. do something before submit ..
    $.rails.handleRemote( $(form) );  // submit via xhr
    //form.submit();                  // don't use, it submits the form directly
  }

});

ActiveRecord Mass Assignment

2011-05-09T00:00:00+08:00

Even you haven’t heard of mass assignment, it already exists in your first Rails generated scaffold code.

def create
  @comment = Comment.new(params[:comment])
  ..
end

def update
  ..
  @comment.update_attributes(params[:comment])
  ..
end

Why knowing mass assignment is important?

By default, mass assignment opens up an undesirable security hole, by allowing web clients to update any attributes they passing in, including attributes like created_at.

For details, you can read:

Mass assignment methods

There are a few ActiveRecord methods that accept mass assignment:

Using any of these methods, means you are now responsible for the safety of the web application.

Minimal protection

To use mass assignment safely, we want to specify exactly which attributes allowed to be updated.

1. Define `attr_accessible` on every model

attr_accessible defines a white-list of attributes that can be updated during mass assignment.

class Comment < ActiveRecord::Base
  attr_accessible :title, :content
end

2. Enforce `attr_accessible` usage

You may set the white-list default to empty. This forces you to define the whitelist explicitly on a model, before mass assignment can be used.

# config/initializer/enforce_attr_accessible.rb

ActiveRecord::Base.send(:attr_accessible, nil)

ActiveRecord Model Equality

2011-05-05T00:00:00+08:00

In daily web app development, we often need to check if two model objects are equal. When I first started using Rails, this is what I wrote:

if user.id == current_user.id
  # do something
end

Use obj1 == obj2 instead

Of course, that reflects my ignorance on learning the new tool.

So, here’s a proper way to compare if two model objects are equal:

test "2 records with same #id equal to each others." do
  c1 = Comment.create
  c2 = Comment.find(c1.id)

  assert c1 == c2
  assert c2 == c1
end

What about new record?

Basically, a new model object only equals to itself:

test "New record does not equal to another new record." do
  new_record = Comment.new
  another_new = Comment.new

  assert new_record == new_record
  assert another_new == another_new
  assert new_record != another_new
end

Learning to use == operator for checking model equality is useful. Because now, we can start using other methods that depend on == operator. For instance, Array#include?.

test "When applied in Array" do
  c1 = Comment.create
  c2 = Comment.find(c1.id)

  assert [c1].include? c2
  assert [c2].include? c1
  assert [c1, c2] == [c1, c2]
end

Rails learning tests

You may notice the code examples are written as tests. This is my first Rails learning test. So the plan is to write more tests as a way to learn Rails. I will post them here occationally if interesting enough to share.

Source code: model_equality_test.rb

teohm.dev

Chef cookbooks for busy Ruby developers

Getting started

1. Setup Chef Solo environment

Install Chef Solo tools

Download Chef cookbooks

Install chef-solo on remote server

2. Customize config file

Download config example

Modify config file (JSON)

3. Provision remote server

What do we get at this point?

4. Tweak Capistrano deploy.rb

Login as deploy user

Deploy to /home/apps

Load rbenv in Capistrano

Run bundler with --binstubs

Restart app with runit sv

Feedback

Reload required files in Rails

Requirements

Reload require files

An extra step in Rails 3.2

Introduce require_reloader gem

Start Using Ruby % (Percent) Notation

Delimiter allows any non-alphanum

Bracket pairs no need to escape

Modifiers for String, Regex, Array, Symbol, Shell command

Ruby idiom to ensure variable is Array

Objective-C Collection Operators

Calculate average - a shorter way

Breaking ARC retain cycle in Objective-C blocks

Retain Cycle

Use Lifetime Qualifiers

Mac Tips: Use Caps Lock as Control key

Mac Tips: Change default application for a file type

Mac Tips: Turn on remote login (SSH server)

Using sshuttle in daily work

Install sshuttle

I use sshuttle to..

1. Tunnel all traffic

2. Tunnel all traffic, but exclude some

3. Tunnel only certain traffic

4. VPN to office network

IP addresses.. troublesome?

Tunnel all traffic

Tunnel all traffic, but exclude some

Tunnel only certain traffic

VPN to office network

UTF-8 param name issue in Rails multipart form

Issue

What happened?

Solutions?

Working effectively with iTerm2

Install iTerm2

Fine-Tune Settings

Open tab/pane with current working directory

Enable Meta key

Hotkey to toggle iTerm2

Switch pane with mouse cursor

Handy Shortcut Keys

Tab navigation

Pane navigation

Search

Input to all panes

Clear screen

Zooming / Font Resize

Shortcuts to move faster in Bash command line

Basic moves

Moving faster

Cut and paste (‘Kill and yank’ for old schoolers)

Search the command history

Need more?

Using RABL in Rails JSON Web API

Problems

Model is not view

Define once, reuse everywhere

Join table rendered as subclass

Learning Git Internals by Example

Movitation

Install `chef-solo` on remote server

Login as `deploy` user

Deploy to `/home/apps`

Load `rbenv` in Capistrano

Run bundler with `--binstubs`

Restart app with `runit sv`

Reload `require` files

Introduce `require_reloader` gem

Enable `Meta` key

Get the latest `jquery-ujs`

When using `submitHandler` in JQuery Validation

1. Define `attr_accessible` on every model

2. Enforce `attr_accessible` usage