Odes of the Occult

systemd socket activation and Ruby

2012-01-21T00:00:00+01:00

For anyone who doesn't know what systemd is:

systemd is a system and service manager for Linux, compatible with SysV and LSB init scripts. systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux cgroups, supports snapshotting and restoring of the system state, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic.

www.freedesktop.org/wiki/Software/systemd

It's quite similar to Apple's launchd (used in OSX) and is fully utilizing powerful features of the latest Linux kernel. systemd is default init system in latest Fedora, openSUSE and Mandriva and is available for many other Linux distros as alternative boot solution. I hope Ubuntu's upstart team will give up soon because having systemd on Ubuntu servers would be awesome. For more info and idea behind the project I recommend reading Lennart Poettering's announcement

One of the great features of this init system is socket activation of system services. In short, services are lazily started when they're actually needed. Systemd listens on the sockets for them and starts the services on first incoming connection, passing them the listening sockets. Started services just start accepting clients on these sockets (without calling socket()+bind()+listen()).

It appears that the protocol for passing sockets to service processes is very simple. Environment variable LISTEN_PID is set to the PID of the service process and another environment variable LISTEN_FDS is set to the number of listening sockets passed. Socket descriptors start from number 3 and are sequential. For example, LISTEN_FDS with value of 2 means process should accept connections on 2 sockets with descriptors 3 and 4.

I'll show you how all this works on an example echo server written in ruby. The server will send back what it receives. Additionally it will send information telling if listening socket came from systemd or not to each new connected client.

But first we need to create the socket unit file that specifies where systemd should listen on behalf of our service. /etc/systemd/system/echo-server.socket file can look as simple as this:

[Socket]
ListenStream=8888

Next, we need service unit file that specifies what binary to start when connections start coming. /etc/systemd/system/echo-server.service file may look like this:

[Service]
ExecStart=/home/kill/.rvm/bin/ruby-1.9.2-p290 /home/kill/bin/echo-server.rb
User=kill
StandardOutput=syslog
StandardError=syslog

I have ruby 1.9.2 installed via RVM so I'm running my ruby script with RVM's wrapper specifying full paths (remember init process runs as root). I'm also setting the user on whose behalf the process should be run and I'm asking systemd to log process' stdout/stderr to syslog (simplifies debugging).

Now, the echo server (/home/kill/bin/echo-server.rb):

#!/usr/bin/env ruby

require 'socket'

SD_LISTEN_FDS_START = 3

from_systemd = false

if ENV['LISTEN_PID'].to_i == $$
  # use existing socket passed from systemd
  server_socket = Socket.for_fd(SD_LISTEN_FDS_START + 0)
  from_systemd = true
else
  # create new listening socket on port 8888
  server_socket = Socket.tcp_server_sockets(8888)
end

Socket.accept_loop(server_socket) do |client_socket, addr|
  client_socket.send("OHAI! systemd socket: #{from_systemd}\n", 0)

  while (data = client_socket.recv(1000)).size > 0
    client_socket.send(data.upcase, 0)
  end
end

Implementation is very simple, still I'm gonna explain it a little bit as it illustrates the use of systemd socket activation protocol and the fallback - normal way of creating server socket.

Like I mentioned earlier, descriptors of systemd passed sockets start with 3:

SD_LISTEN_FDS_START = 3

We check if LISTEN_PID points to our echo-server.rb process:

if ENV['LISTEN_PID'].to_i == $$

If so, we're creating new Socket instance for existing descriptor (3). Socket unit file tells systemd to listen on one port only (8888) so we can assume there's only one socket descriptor passed:

  # use existing socket passed from systemd
  server_socket = Socket.for_fd(SD_LISTEN_FDS_START + 0)

If LISTEN_PID doesn't match our process we just create TCP socket the usual way:

else
  # create new listening socket on port 8888
  server_socket = Socket.tcp_server_sockets(8888)
end

Finally, in Socket.accept_loop(server_socket) do { ... } we handle incoming clients.

Tunnelling VirtualBox guest's network traffic

2011-11-12T00:00:00+01:00

This post is quite different than my usual ones. It's not about programming at all. Usually I'm not writing about how I solve various, non-programming related problems but this time I want to share what I achieved as this may be helpful to others facing similar scenario. I couldn't find similar solution explained anywhere so I'm putting this for anyone seeking such information.

The problem

Tunnel whole network traffic of VirtualBox guest machine through remote server located in different country.

Basically what I needed was a virtual machine that was connecting to internet with London's IP address. Simple port forwarding was not an option as the machine needed to connect from many apps (not only web browser) to multiple (unknown to me) hosts and ports.

The idea

Setup SSH tunnel from my local machine (VirtualBox host) to my Linode server in London and tell VirtualBox to use this tunnel as guest VM's ethernet interface endpoint.

Seems simple. And after a few failed attempts it turned out to be really simple. You just need to know few concepts and run few commands here and there.

The solution

Below is the complete solution that worked well for me. I use following symbols for involved machines:

P - remote machine with London's IP, used as a proxy
H - local machine, VirtualBox host
G - virtual machine, VirtualBox guest, needs public IP of P

Solution is build on OpenSSH-based VPN. From waldner's post at backreference.org:

This is a poorly documented yet really useful feature of Openssh. It allows you to connect two tun/tap interfaces together, to create a layer-2 or layer-3 network between remote machines. This results in OpenVPN-like VPNs (but much simpler and, admittedly, less scalable).

With presented solution virtual machine (guest) can be running any OS: Linux, Windows, whatever.

Foreplay

You need to have root access on the machine that will work as a proxy (P). This machine needs to have eth0 interface configured with public IP address (London in my case).

The sshd running on it should allow root logins (I'll explain later why) and setting up ssh tunnels. Make sure you have following entries in /etc/ssh/sshd_config on P:

# /etc/ssh/sshd_config

PermitRootLogin yes
PermitTunnel yes

Restart sshd if needed.

Step I

First you need layer-2 (ethernet) ssh tunnel. Run following on H as root:

$ ssh -o Tunnel=ethernet -w 0:0 root@<P hostname>

replacing <P hostname> with real hostname of your proxy machine.

This will setup L2 link and create tap0 network interfaces on both machines (H and P). Because only privileged users can create network interfaces you need to run this command as root, ssh-ing to remote host as root user as well.

* 0:0 in above command specifies numbers for tap interfaces for both machines. You can safely use 0 for both of them as you probably don't have any other existing tap interfaces.

Step II

Now you need to create NAT on the proxy machine (P). This will make traffic from tap0 interface to be seen with P's public IP address (going through its eth0 interface).

Run following on P as root:

$ echo 1 > /proc/sys/net/ipv4/ip_forward
$ /usr/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
$ /usr/sbin/iptables -A FORWARD -i eth0 -o tap0 -m state --state RELATED,ESTABLISHED -j ACCEPT
$ /usr/sbin/iptables -A FORWARD -i tap0 -o eth0 -j ACCEPT

Masquerade ready. Now bring P's tap0 interface up and configure it:

$ ifconfig tap0 up
$ ifconfig tap0 192.168.0.1 netmask 255.255.255.0

* You can use different network than 192.168.0.* for NAT if P is already attached to such network.

Step III

Next step is to bring H's tap0 interface up. Run following on local machine (H) as root:

$ ifconfig tap0 up

There is no need to assign IP to this interface as it will be directly connected to virtual machine's network interface.

Step IV

Now configure virtual machine to use the tunnel.

Open VM network settings (on H), select Bridged adapter and choose tap0 as a bridged device.

Step V

Finally start virtual machine (G) and configure its network interface.

The guest machine can be either Linux or Windows (or just anything you want). Just setup the interface to be configured as below:

IP: 192.168.0.2
Netmask: 255.255.255.0
Gateway: 192.168.0.1
DNS: 8.8.8.8 / 8.8.4.4 (use Google's ones for simplicity)

To confirm that VM (G) is visible to the world with public IP of P run:

$ curl icanhazip.com

or open icanhazip.com in the browser.

Simplifying

You can put commands from steps I, II and III in shell scripts to simplify the task in case you want to run it frequently. I have made 2 bash scripts.

First on on my local machine (H):

# vm-tunnel.sh

$ ssh -o Tunnel=ethernet -w 0:0 root@<P hostname> "~/taptap.sh"
$ ifconfig tap0 up

Second one on proxy machine (P) in root's home:

# /root/taptap.sh

$ echo 1 > /proc/sys/net/ipv4/ip_forward
$ /usr/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
$ /usr/sbin/iptables -A FORWARD -i eth0 -o tap0 -m state --state RELATED,ESTABLISHED -j ACCEPT
$ /usr/sbin/iptables -A FORWARD -i tap0 -o eth0 -j ACCEPT

$ ifconfig tap0 up
$ ifconfig tap0 192.168.0.1 netmask 255.255.255.0

Thanks to these 2 scripts I can summon my VPN with one command:

$ sudo vm-tunnel.sh

My EuRuKo 2010 talk

2011-10-23T00:00:00+02:00

I was speaking at EuRuKo conference last year (2010) and I'm just putting this here for archiving purposes.

Here's a video of the talk:

And here are the (slightly updated) slides:

Coloration + terminal Vim + 256 colors

2011-10-22T00:00:00+02:00

Year has passed since last Coloration update. Time has come for new improvements. Brand new Coloration v0.3.1 focuses on Vim theme writer and includes:

Line numbering (LineNr) background matching cursor line highlight style (CursorLine)
New style for ColorColumn also matching cursor line highlight
Highlighting for terminal Vim running in 256 colors capable terminal

Let's focus on the last one.

Nowadays most terminal emulators support 256 colors. Additionally to 16 base colors everyone knows about they also support 216 colors from 6x6x6 color cube and 24 shades of grey. Look here for details. If you don't use 256 colors capable terminal then start using it now. xterm had it in 1999. gnome-terminal, konsole, iTerm2 and many more have it.

Vim colorschemes allows you to specify ctermfb and ctermbg from the range 0-255. People usually only use 0-15 in their themes in order to be compatible with 16-color-my-grandpa-uses terminals. That's so wrong! Coloration is fast-forwarding us to the future with it's updated Vim theme writer. It converts the colors used by the gui Vim version (GVim/MacVim) to their xterm256 nearest equivalents with simple approximation.

Sunburst theme converted from Textmate theme:

Twilight theme converted from Textmate theme:

Now go convert your old, dusty Textmate themes. Oh, and I've put Sunburst and Monokai themes on github (vim-sunburst, vim-monokai) for all you lazy guys.

Better Rails production box

2011-09-26T00:00:00+02:00

Here are 3 simple things to do in order to improve deployment process of your Rails application and friendliness of your production server. Nothing new here but many good practices are often overlooked even by experienced developers.

Set RAILS_ENV in .bash_profile

I see my fellow devs typing this on production server all the time:

script/rails console production

Or this:

RAILS_ENV=production rake some_task

It's obvious that on production box you want it to run in production environment, right? Set RAILS_ENV in .bash_profile and save your fingers:

# ~/.bash_profile
export RAILS_ENV="production"

This might not be the case if your hosting service is like Heroku or similar as they export RAILS_ENV for you automatically. But if you're on self-managed VPS or EC2 instance you can ease your work with this trivial setup.

Use logrotate

Create /etc/logrotate.d/rails_logs file with following content:

/var/www/*/shared/log/*.log {
  daily
  missingok
  rotate 30
  compress
  delaycompress
  copytruncate
}

That will tell logrotate to rotate log files daily, compress them, keep last 30 days and don't choke when file is missing. copytruncate is important here as it will make sure log file currently used by Rails app is not moved but truncated. That way the app can just keep on logging without reopening log file.

Don't forget about this one if you manage production box yourself. And do it when you initially setup the box, not "later". "Later" often means "when app is down due to not enough disk space". Srsly.

Use maintenance page

When deploying with long running or non-trivial (more than add column) migrations you should use maintenance page of some sort. With capistrano you can just use (before running migrations):

cap production deploy:web:disable

and (after they finish):

cap production deploy:web:enable

Default maintenance page put by capistrano is kind of ugly so you should make your own matching your site design. In order to do this prepare app/views/layouts/maintenance.html.erb and override deploy:web:disable task in config/deploy.rb:

namespace :deploy do
  namespace :web do
    task :disable, :roles => :web do
      require 'erb'
      on_rollback { run "rm #{shared_path}/system/maintenance.html" }

      reason = ENV['REASON']
      deadline = ENV['UNTIL']
      template = File.read('app/views/layouts/maintenance.html.erb')
      page = ERB.new(template).result(binding)

      put page, "#{shared_path}/system/maintenance.html", :mode => 0644
    end
  end
end

That will put your custom page in #{shared_path}/system/maintenance.html (also accessible via public/system/maintenance.html by webserver).

On the other end, you should configure webserver to respect presence of this file. Here is config snippet for Nginx:

server {
  listen 80;
  server_name example.org;

  ...

  # Maintenance page support

  set $maintenance 0;

  if (-f $document_root/system/maintenance.html) {
    set $maintenance 1;
  }

  if ($request_uri ~* (jpg|jpeg|gif|png|js|css)$) {
    set $maintenance 0;
  }

  if ($maintenance) {
    return 503;
  }

  error_page 503 @maintenance;
  location @maintenance {
    rewrite ^(.*)$ /system/maintenance.html break;
  }
}

It will make sure that when maintenance.html file is there Nginx will serve it and return 503 (Service unavailable) HTTP status. Proper response status is important here as it gives search engines the message: don't index me now please, come back later. You don't want your maintenance page get indexed, do you?

Above Nginx config also allows assets to be served as usual when maintenance mode is on - especially useful if you don't host them on separate host/subdomain and you want the page to look nice and match presence of your full site.

Formatting XML in Vim with indent command

2011-09-08T00:00:00+02:00

Today I had a need to look at XML doc fetched from Google Calendar API. I saved it to a file and opened in Vim. Unfortunately API output was generated to be consumed by machines rather than humans.

First I tried gg=G command. = is used to auto-indent selected line(s) and gg=G re-indents whole file. Usually it works great, especially for source code files. It does not reformat code, it only changes indentation. And that's good, I should be the one to control look of my code. But for XML I want it to do full reformatting. I'm not writing XML and in all of the cases when I open such docs in my editor I only want to look at well-formatted, human-readable XML.

From Vim help on equalprg setting:

External program to use for "=" command. When this option is empty the internal formatting functions are used; either 'lisp', 'cindent' or 'indentexpr'. When Vim was compiled without internal formatting, the "indent" program is used. ...

"Bingo! Let's use xmllint for that!" - I thought immediately. xmllint command comes bundled with libxml package on Unix-like systems and does really good job at producing pretty output. You can use it like this:

# shell
xmllint --format --recover foo.xml

Okee, so my first approach to reformatting XML in Vim was:

:1,$!xmllint --format --recover - 2>/dev/null

Not bad. But writing it every time (or remembering) would be painful. Let's use mentioned equalprg option:

# .vimrc
set equalprg=xmllint\ --format\ --recover\ -\ 2>/dev/null

Ok... but Y U USE XMLLINT WHEN I'M INDENTIN' MY RUBY CODE?? Ahaa! equalprg need to be set locally only for XML-type buffers. Autocommand did the trick:

# .vimrc
au FileType xml setlocal equalprg=xmllint\ --format\ --recover\ -\ 2>/dev/null

Restarted Vim, typed gg=G and said "Hell yeah!".

Bitpocket as a Dropbox alternative

2011-07-18T00:00:00+02:00

As an excuse for trying out Posterous email posting and its Autopost feature (in this case to Twitter and Identi.ca) I'll present you Bitpocket.

In short, Bitpocket is a small but smart bash script that does two-way directory synchronization resembling Dropbox sync. Simply it just uses rsync to make the actual sync. It runs rsync twice: first syncing from remote to local machine, then from local to remote machine. This way all new files that appeared on remote are fetched to local machine and all new locally created files are replicated on remote machine.

But additionally it makes sure that file deletion is properly propagated in this 2-way synchronization, which isn't possible by just running rsync twice. The problem is rsync deletes all new created files or brings back the files you deleted depending on which direction we sync in first (and when using --delete option). Bitpocket solves that by tracking names of created and deleted files between its invocations and using these lists as a source for --exclude-from rsync option when doing first, remote -> local sync.

To make sure Bitpocket behaves like I wanted it to behave I've spec'ed it. Yes, it's a bash script, not a ruby code, but who said I can't use rspec to test a bash script?

For details about usage see Bitpocket's README. I'm using it instead of Dropbox for few weeks now and it does quite good job. Give it a try.

Resque Mailer says: Put your emails to background

2011-04-05T00:00:00+02:00

It's no secret that handling HTTP requests should be quick. Very quick. Your business' "to be or not to be" may depend on how "fast" is you site. "Fast" site for end-users is when they're clicking and they don't need to switch to other tabs to kill time while waiting for page to open.

How to make your site fast?

buy more RAM (and install Oracle) - it costs $$$
lazy load what you can after initial page - complicates the code
do any processing in background - that one is quick win!

Depending on the application there will always be need for some combination of above. Best practice however, is: Webserver should not do any heavy data processing when handling request. It should just process the request and return response as soon as possible.

It means that all you can do is:

parsing params
loading resource(s)
changing state of resource(s)
utilizing background processing workers for data grinding
rendering response body from template
returning response to client

All of the above steps except last one are optional but they're all usually required in most apps. Point 4 is the important here. Developers often forget about it or are just too lazy to implement it. But it makes huge difference! Waiting 10 seconds for page load because the controller needs to finish something... it's unacceptable.

Resque to the rescue! Very reliable, framework independent, fast "Redis-backed library for creating background jobs" by Github's defunkt. I use various Ruby web-frameworks to build my webapps, the choice is not always Rails but Resque can be used with anything. Good stuff.

Anyway, what is not that common is sending e-mails asynchronously. Yeah, some people never thought about it. But sending them right from controller costs. Building mail message and sending it involves creating lots of ruby objects, rendering templates, preparing mail headers and eventually passing it to local MTA or sometimes to remote SMTP server. Sounds like a lot of time, especially when you use remote SMTP server.

But who would want to create dedicated background worker for this task? Who would want to enqueue such a job instead of doing simple Notifications.signup(@user).deliver in the controller? Not me.

Enter ResqueMailer. Originally created for Rails 2 by Nick Plante, updated by me to work with both Rails 2.x and 3.x. ResqueMailer allows you to move processing of Rails mailers out of controller to an async Resque worker with minimal fuss.

Just put the gem into your Rails project Gemfile:

gem 'resque_mailer'

and then include the Resque::Mailer module into your Mailer class:

class Notifications < ActionMailer::Base
  include Resque::Mailer

  def signup(...)
    ...
end

Instaaant gratification!

This is it. Now start your worker and enjoy faster responses. For information on ResqueMailer's optional settings see project's github repository. For details about setting up Resque and running workers go to README of Resque project.

Github's Gist and Gnome Keyring

2011-04-03T00:00:00+02:00

I like to keep my dotfiles in git repository but I never put my .gitconfig there because it included my GitHub API token, that is used for example by gist. Git allows you to get output of a system commands as values (by prepending value with exclamation mark) for settings in [alias] section of .gitconfig only but I just found out that gist script can also do this trick for github.token setting. Thanks to that we can prepare some script that gets token from different place, use it in .gitconfig, put config into repository and worry no more about publishing sensitive data.

I put the token in Gnome Keyring as it feels pretty secure. Just go to System -> Preferences -> Passwords and Encryption Keys, press Ctrl+N, choose Stored password. Select login keyring, enter "GitHub API Token" as description and your token as password. Now we need a way to get this token out of keyring in shell script. Fortunately there are python language bindings for Gnome Keyring.

Python script for retrieving passwords from keyring can look like this:

#!/usr/bin/env python

import sys
import gnomekeyring as gk

if len(sys.argv) > 2:
    ring_name = sys.argv[2]
else:
    ring_name = 'login'

for key in gk.list_item_ids_sync(ring_name):
    item = gk.item_get_info_sync(ring_name, key)
    if item.get_display_name() == sys.argv[1]:
        sys.stdout.write(item.get_secret())
        break

Simple. It prints out password with given name to standard output, without newline character (so it's easier for other scripts to use it). Let's save it to ~/bin/keyring-get-pass.py and try it:

$ python ~/bin/keyring-get-pass.py 'GitHub API Token'
my-secret-token-i-wont-show-you$

Cool. By default we get secrets from login keyring. This is for convenience as login keyring is being unlocked at system login time on Gnome (at least on Ubuntu) and it won't ask us to unlock it when running this script. If we need to get password from different keyring then its name can be passed as the second argument to the script.

Now, let's use the script in .gitconfig:

[github]
  user = sickill
  token = !python ~/bin/keyring-get-pass.py 'GitHub API Token'

If you have other solutions for avoiding publishing passwords and tokens in your dotfiles (like config templates etc) tell me, I'm eager to hear!

Open3 And The PID Of The Spawn

2010-09-18T00:00:00+02:00

Open3 is a standard way of starting subprocess from ruby if you need IO objects for stdin, stdout and/or sterr. It goes like this:

require "open3"

stdin, stdout, stderr = Open3.popen3('sort')
stdin.puts "oso de peluche"
stdin.puts "del ratón"
stdin.close
while !stdout.eof?
  puts stdout.readline
end

Simple. I needed one more thing: PID of the spawned process to be able to send it some SIGNALs. After looking at open3 docs I realized there is no way to get the PID. Several posts like this one made me believe that open4 is the only way to go.

But I didn't gave up. I didn't want to add open4 as a dependency of my code only because of this small shortcoming. After a while of googling I found open3 documentation for ruby 1.9. How it's different?

"Open stdin, stdout, and stderr streams and start external executable. In addition, a thread for waiting the started process is noticed. The thread has a thread variable :pid which is the pid of the started process."

Hurray!

stdin, stdout, stderr, wait_thr = Open3.popen3(cmd)
pid = wait_thr[:pid]  # pid of the started process.
Process.kill("USR1", pid)  # send it USR1 signal

So MRI 1.9 solves the problem by returning additional element with reference to waiting thread with :pid variable set on it. How about other ruby implementations?

JRuby doesn't return waiting thread reference from open3() call but it has open4 incorporated into its standard library in the shape of IO.popen4 method:

pid, stdin, stdout, stderr = IO.popen4(cmd)

Unfortunately, both MRI 1.8.x and Rubinius don't include this functionality in their standard lib.

So, is open4 the only reliable way? Let's see:

MRI 1.8.7:

ruby-1.8.7-p249 > open4 "cat"
 => [31373, #<IO:0xb741c8ac>, #<IO:0xb741c870>, #<IO:0xb741c7f8>]

MRI 1.9.2:

ruby-1.9.2-p0 > open4 "cat"
 => [31498, #<IO:fd 4>, #<IO:fd 5>, #<IO:fd 7>]

JRuby:

jruby-1.5.2 > open4 "cat"
NotImplementedError: fork is unsafe and disabled by default on JRuby
        from /home/kill/.rvm/gems/jruby-1.5.2/gems/open4-1.0.1/lib/open4.rb:23:in `popen4'

Rubinius:

rbx-1.0.1-20100603 > open4 "cat"
 => [31635, #<IO:0x212>, #<IO:0x214>, #<IO:0x216>]

Damn, looks like we cannot use open4 under JRuby. Fortunately, there is IO.popen4 which should be used instead. Solution that will work in all ruby implementations might look like this:

if IO.respond_to?(:popen4)
  def open4(*args)
    IO.popen4(*args)
  end
else
  require 'open4'
end

open4("cat")

Summarizing, go with open4/IO.popen4 if you want to be sure your code works on all ruby implementations.

Announcing Coloration, editor color scheme converter

2010-07-24T00:00:00+02:00

Without further ado I'm introducing Coloration - editor/IDE color scheme converter. It is an evolution of tm2jed tool and at the moment it can convert Textmate color themes (in XML plist format) to Vim, JEdit and Kate/KWrite/KDevelop color schemes. So if you are Textmate->Vim convert or you just envy Textmate users for their good looking, dark themes now you have no excuse to not try out Coloration.

Here's how Vim with Sunburst theme looks like:

If you want to give Coloration a try you have two options. Either you can use online version at coloration.sickill.net or you can install ruby gem coloration:

gem install coloration

It will give you tm2vim, tm2jedit and tm2katepart commands. Note it requires ruby 1.9.

Let me know if you find it useful. Source code is available at github.com/sickill/coloration.

Enjoy!

Thoughts on Ruby gem dependencies

2010-07-03T00:00:00+02:00

The problem

Having gem bundler is great. First, you can forget about gem version collisions (damn you activation errors!). Second, you can forget about manually installing gems on all your machines. Third, you can use git repositories and local directories as gem sources which is neat and is invaluable when working on your own gems. But bundler is a workaround. Yes, it's a workaround for poor rubygems design. Let's look at few examples of the problems I've noticed.

I'll start with dm-paperclip gem. Recently I was porting my Merb app to Rails 3 and I've encountered few problems with this little sucker. The biggest headache I had with validations. Note it was fixed recently in dm-paperclip, but makes good example of problem that still exists in relation to other gems. dm-paperclip was doing this:

...
unless defined?(DataMapper::Validate).nil?
...

What's the problem here? Let's say you have following lines in your Gemfile:

gem "dm-validations"
gem "dm-paperclip"

Looks like everything should work. Wrong! Rails 3 requires all the gems from your Gemfile by calling Bundler.require Rails.env but Bundler.require doesn't use order of gems specified in Gemfile. For me it first required dm-paperclip and then dm-validations, so when dm-paperclip was being required DataMapper::Validate wasn't defined yet. I heard that it may change in future and Bundler will respect the order, to some degree of course. Does this problem lay in bundler or in dm-paperclip? In none of them. Bundler doesn't know about "optional" dependencies of dm-paperclip and dm-paperclip has no way to tell us/bundler it can benefit from some optional lib.

Anyway, this is good example why explicit requiring of gems seems better in this scenario. By adding require "this" and require "that" in places where you actually need it you're breaking DRY principle (you already have it in Gemfile) but you can have perfect control of the order. This way you can solve above dm-paperclip problem. And by using explicit requires you can move your app away from bundler (to i.e. rip or Isolate) if you want to and you don't need to add these requires because they're already in the app. But why would you need to think about gem require order in your Rails app? You shouldn't think about it. And you don't always know what are optional libraries for the gems you're using.

Let's look at second issue. Multiple markdown processing libraries. There's BlueCloth, Maruku, Kramdown, RDiscount, rpeg-markdown and some more. All of them are doing the same thing but the difference is mainly in performance. BlueCloth and Maruku are pure ruby libraries, RDiscount and rpeg-markdown are bindings to fast C libraries. But they all have the same interface:

BlueCloth.new(markdown_string).to_html          # bluecloth
Maruku.new(markdown_string).to_html             # maruku
Kramdown::Document.new(markdown_string).to_html # kramdown
RDiscount.new(markdown_string).to_html          # rdiscount
Markdown.new(markdown_string).to_html           # rpeg-markdown

Now, if gem A depends on Maruku and gem B depends on RDiscount I'll have two markdown libraries with identical interface required in my app. I know that RDiscount is faster because it's C and I'd like all of my app's functionalities to use it but gem A will use slower Maruku. Sad but true.

Third rubygems issue is something in the middle between dm-paperclip+bundler problem and markdown problem. Devise, "Flexible authentication solution for Rails with Warden" supports many ORMs (ActiveRecord, DataMapper, Mongoid). Now, all code related to integration with mentioned ORMs is included in devise itself. Devise checks which ORM is available at runtime (like in dm-paperclip example) and requires appropriate file choosing from many available alternatives (like in markdown example). It's far from being perfect. How this could be improved?

It'd be probably better to put integration code in separate gems. This way people involved in development of ORMs could work on integration without access to Devise repository. Yeah, I know, it's not a big deal on github, just fork and send pull request. But it worked really well for dm-rails. DataMapper guys were more interested in getting DM work under Rails 3 than Rails developers and they definitely knew more about DM specifics. They provide the gem and everyone is happy. But if there were many gems providing this functionality for Devise then user would be responsible for installing it. There's better solution on my mind though, read on.

Possible solutions

To solve first mentioned issue let's just add optional dependencies to rubygems. It could work in following way.

Gem author specifies list of optional dependencies:

Gem::Specification.new do |s|
  ...
  s.optional_dependency "nokogiri", "for tidying output"
  s.optional_dependency "bar", "for bar support"
end

You are installing the gem:

$ gem install foo
Installed foo.
Optional dependencies for foo:
- nokogiri (for tidying output)
- bar (for bar support)

Or you could install it with optional deps:

$ gem install foo --with-optional-deps
Installed nokogiri.
Installed bar.
Installed foo.

Benefits of having explicit optional deps would be:

user could see what are the optionals and decide if he wants them
automatic tools like bundler could be configured to install optional deps, either for all gems or for specific ones
bundler could require optional gems before the one which "optionally" depends on them, solving require order issues

To solve second and third issue we can add "provider gems". What's that? Decent Linux package managers like Archlinux's pacman allows you to specify that the package provides the same functionality and the same interface as the other one. In rubygems it could look like this:

Gem authors of all mentioned markdown processors could specify that their gem provides "markdown":

Gem::Specification.new do |s|
  ...
  s.provides "markdown"
  ...
end

Additionally they would need to provide unified interface, in this case simple Markdown = RDiscount should do the trick.

Now, you as gem user could use it like this:

require "markdown"
Markdown.new(markdown_string).to_html

This way gem A and B can depend on "markdown", and you decide which one you want to install. gem install A or bundle install in this case could show you this:

$ gem install A
Gem "A" depends on "markdown". Please select provider gem from following alternatives:
1. BlueCloth
2. Maruku
3. ....

Of course having two "markdown providers" with the same interface (note that require "markdown" is also part of the interface) would be impossible in the same gem environment but this can easily be solved by using Bundler, Isolate or awesome rvm's gemsets. Only problem I see here is when one gem depends on "maruku", not on "markdown", another gem depends on "rdiscount", and your app depends on both of these gems... yuck.

"Provider gems" can be easily applied to Devise's case:

Gem::Specification.new do |s|
  s.name "devise"
  s.dependency "devise-orm-proxy"
  ...
end

Gem::Specification.new do |s|
  s.name "devise-orm-mongoid"
  s.provides "devise-orm-proxy"
  ...
end

Gem::Specification.new do |s|
  s.name "devise-orm-dm"
  s.provides "devise-orm-proxy"
  ...
end

Another example why "provider gems" can be good idea is extlib / AS collision. For example dm-paperclip depends on extlib because it needs inflector. But you can't easily use it in Rails 3 app at the moment because AS+extlib = "UsersesController" :) dm-core used to use extlib for inflections and has been recently converted to AS for some reasons.

"Provider gems" can solve also this problem. Look at this:

Gem::Specification.new do |s|
  s.name "dm-core"
  s.dependency "inflector"
  ...
end

Gem::Specification.new do |s|
  s.name "dm-paperclip"
  s.dependency "inflector"
  ...
end

Gem::Specification.new do |s|
  s.name "extlib"
  s.provides "..."
  s.provides "inflector"
  s.provides "..."
end

Gem::Specification.new do |s|
  s.name "activesupport"
  s.provides "..."
  s.provides "inflector"
  s.provides "..."
end

Conclusion

I realize that these proposed solutions are not ideal and there are some edge cases which need some more thought but maybe it will become a good start for further discussion about the problem. Rubygems code is on github now so we can fork it and improve it!

Quick update on Open File Fast 1.0.1

2010-06-09T00:00:00+02:00

This is really short update on OpenFileFast. Recently new JEdit (4.3 final with new ProjectViewer plugin) and Netbeans (6.9RC2) came out and they’ve both changed some APIs. The changes were not backwards compatible and OFF stopped working for them. Fortunately the fix was simple and I’ve prepared updated builds of the plugin for JEdit and NB.

Here are the links for 1.0.1 update:

Netbeans NBM package: off-netbeans-1.0.1.nbm"
JEdit jar file: off-jedit-1.0.1.jar"

Open File Fast reached 1.0

2010-01-17T00:00:00+01:00

After almost a year of development and several beta versions Open File Fast finally became stable and mature enough to bump its version to 1.0. There’s not much new in this release except several bugfixes and improved look of search results list.

Now it looks like this:

Notice that entered characters are now displayed in bold indicating where they were found in each filename. File path is now right aligned and has lighter color. It doesn’t come into way anymore when looking at results and lets you localize the file you seek even faster.

Netbeans version of plugin was updated to work in Netbeans 6.8, JEdit’s one works now in final JEdit 4.3 version (with ProjecViewer 2.9.1+).

Netbeans NBM package: off-netbeans-1.0.nbm"
JEdit jar file: off-jedit-1.0.jar"

Quick and dirty hack for RSpec's before(:all)

2009-11-23T00:00:00+01:00

RSpec is generally nice testing framework. It supports before and after hooks which can be invoked before/after each test case or all test cases. before(:all) is a little confusing though. It runs your "before" block before all "describes" and "contexts", also nested ones. Here's example:

describe User do
  before(:all) { puts "preparing for war" }
  it "should foo" do
    ...
  end
  context "active" do
    it "should bar" do
      ...
    end
  end
  context "inactive" do
    it "should baz" do
      ...
    end
  end
end

I would expect "preparing for war" to show up once. But before(:all) was called thrice. First, for top level "describe", then two times for "contexts". There were some suggestions to change this behaviour or to add and option to skip call for nested groups but nothing has changed recently. People are even trying some crazy hacks like this.

What I needed was to wipe database for every model spec and every request spec because factory generated records made my build unstable (due to uniqueness validations). After trying few things I ended up with using before(:all) with some condition. First, I've added before_top_level_group method to Spec::Runner::Configuration and saved it in spec/before_top_level_group.rb:

$_groups = []

class Spec::Runner::Configuration
  def before_top_level_group
    before(:all) do
      top_level_group = self.class.to_s[/^.+ExampleGroup::([^:]+)/, 1]
      unless $_groups.any? { |g| top_level_group == g }
        $_groups << top_level_group
        yield
      end
    end
  end
end

Then in spec_helper I used it like this:

require 'before_top_level_group'

Spec::Runner.configure do |config|
  config.before_top_level_group do
    # re-migrate db for each top-level group (it usually equals one *_spec.rb file)
    DataMapper.auto_migrate!
  end
end

Voila! I know it's a dirty hack but it works for me and I'll be using it until RSpec is patched or I switch my testing framework to something else (Bacon looks nice).

More Rack::Shell goodies for all Rack worshippers

2009-11-19T00:00:00+01:00

Rack::Shell got a lot of attention lately and I received some feature requests/ideas from great ruby hackers. Daniel Neighman, currently working on Pancake, pointed me towards Bryan Helmkamp's rack-test. This awesome piece of code is now used by racksh to simulate HTTP requests to your Rack application. Yay!

% racksh
Rack::Shell v0.9.4 started in development environment.
>> $rack.get "/"
=> #<Rack::MockResponse:0xb68fa7bc @body="<html>...", @headers={"Content-Type"=>"text/html", "Content-Length"=>"1812"}, @status=200, ...

Check out README for details. Here are just few examples what's possible:

$rack.get "/", {}, { 'REMOTE_ADDR' => '123.45.67.89' }
$rack.header "User-Agent", "Firefox"
$rack.post "/users", :user => { :name => "Jola", :email => "jola@misi.ak" }

Now you can build and test Sinatra apps in single racksh session like this.

Another nice thing in new version is support for session setup through config files. Rack::Shell supports configuration file .rackshrc which is loaded from two places during startup: user's home dir and application directory (in this order). You can put any ruby code in it, but it's purpose is to setup your session, ie. setting headers which will be used for all $rack.get/post/... requests.

For example to set user agent to Firefox and re-migrate db if loaded environment is test put following in .rackshrc:

# .rackshrc

$rack.header "User-Agent", "Firefox"
DataMapper.auto_migrate! if $rack.env == "test"

You can also make requests in config file:

# .rackshrc

$rack.put "/signin", :login => "jola", :password => "misiacz"

This will ensure you are always logged in when you start racksh :)

Full documentation and sources are on github, gems for all versions on gemcutter.org. Enjoy!

Rails-like console for any Rack based ruby web app

2009-11-15T00:00:00+01:00

I always miss script/console from Rails while developing my Sinatra apps, especially ones built with DataMapper where I need to auto-migrate my db. Sinatra doesn't come with any comparable solution as it's not a full framework, but rather library for creating simple web apps. Recently I tried Heroku platform and their "heroku console" command inspired me to create something similar - racksh aka Rack::Shell.

racksh is a console for Rack based ruby web applications. It's like Rails' script/console or Merb's merb -i, but for any app built on Rack. You can use it to load application environment for Rails, Merb, Sinatra, Camping, Ramaze or your own framework provided there is config.ru file in app's root directory.

It's purpose is to allow developer to introspect his application and/or make some initial setup, ie. running mentioned DataMapper.auto_migrate!. It's mainly aimed at apps that don't have similar facility (like Sinatra) but can be used without problems with Merb or Rails apps.

How it works? It loads whole application environment like Rack web server, but it doesn't run the app. Simply, methods like use or run which are normally invoked on Rack::Builder instance are being stubbed.

Instalation is as easy as:

gem install racksh -s http://gemcutter.org

Then to open console run following inside rack application directory (containing config.ru file):

racksh

To specify location of config.ru set CONFIG_RU env variable:

CONFIG_RU=~/projects/foobar/config.ru racksh

Executing ruby code inside application environment and printing results is also supported:

racksh Order.all
racksh "Order.first :created_at => Date.today"

Default Rack environment is set to development but it can be changed by setting RACK_ENV env variable:

RACK_ENV=production racksh

Now I don't need to create some kind of console.rb for my new Rack app, I just use racksh. Enjoy!

UPDATE: Read here for more info.

Subdomain shared cookies in Merb's specs

2009-11-13T00:00:00+01:00

I’m currently working on Merb app which makes use of subdomains for user accounts. In order to have user authenticated on both base domain and subdomains I set cookie domain like this in init.rb:

 Merb::Config.use do |c|
    c[:default_cookie_domain] = ".mydomain.com"
  end

Dot at the start of domain name specifies that the cookie will be set for “mydomain.com” and all its subdomains. This works perfectly in browser(s) but I’ve encountered a problem in my request specs. After authenticating at “mydomain.com” app redirects to “username.mydomain.com”. But next request to either “mydomain.com” or “username.mydomain.com” shows that session is no longer authenticated. After some reading through merb-core sources I’ve found that Merb::Test::Cookie can’t properly handle cookies with domain set to ‘.foo.com’ (with dot at the start).

This is because Merb::Test::Cookie#valid? makes some regexp check comparing current request domain with domain from cookie, which fails for cookie domain set to ‘.foo.com’. I’ve created a fix and spec for this scenario. Patch can be found in this commit.

~~Until it gets merged and released with new Merb version (I hope in 1.1)~~ It was already merged into Merb master branch, but until the 1.1 release the simplest workaround is to monkey patch Merb::Test::Cookie class, ie. in spec_helper.rb like this:

class Merb::Test::Cookie
  def valid?(uri)
    domain_ = domain.start_with?('.') ? domain[1..-1] : domain
    uri_path = uri.path.blank? ? "/" : uri.path
    uri.host =~ Regexp.new("#{Regexp.escape(domain_)}$") &&
    uri_path =~ Regexp.new("^#{Regexp.escape(path)}")
  end
end

Finish Him completes them all!

2009-11-02T00:00:00+01:00

Once again extending JEdit’s feature set. This time word completion. JEdit already has two word completion mechanisms: built-in one and TextAutocomplete plugin. However none of them provides as smooth completion as Netbeans or Textmate’s ones. First I was thinking about creating some BeanShell macro too improve my completion experience but it looked like to much logic for just a macro. So I ended up with new plugin, and this time written in… Scala (finally an opportunity to start learning Scala on real project).

It’s called “Finish Him!” and following comparison shows how it’s different from existing solutions for JEdit.

JEdit’s built in word completion:

shows popup when triggered (-1)
doesn’t sort suggestions by distance from the caret (-1)
searches for suggestions in other (visible) buffers (+1)
Total: -1

TextAutocomplete:

shows popup automatically (-1)
doesn’t sort suggestions by distance from the caret (-1)
doesn’t search for suggestions in other (visible) buffers (-1)
Total: -3

Finish Him!:

doesn’t show popup when triggered, just inserts next suggestion (+1)
sorts suggestions by distance from the caret (+1)
searches for suggestions in other visible buffers (+1)
Total: 3

Finish Him! kills other solutions with score of 3. I know that these things may look like details but after using Netbeans word completion you know that it’s the simplest, fastest and most “correct” (by proper suggestions sorting) word completion. If you’re using JEdit then check it out, it’s now available on Plugin Central so it can be installed directly from JEdit’s plugin manager. For sources look here.

Coderack, a Rack middleware contest launched

2009-10-15T00:00:00+02:00

You’ve probably heard about Rack, right? Few months ago during the conversation with my teammate about potential and possibilities of Rack, and Rack middleware we’ve came up with an idea of creating some coding contest. We wanted to encourage Ruby developers to explore the power of Rack because we believe it’s the best thing since sliced bread. And what better way than to hold a contest? And now we can proudly say that it happened! We’ve just launched it and got really positive feedback about it so far.

But what is this CodeRack thingie all about? Like I’ve mentioned first goal is to show Rack’s power and encourage its use in Ruby web apps. The secondary goal of the contest is to generate a set of open source solutions that will solve real problems and inspire others. Every entry will be released under the MIT open source license.

Programmers are encouraged to submit contest entries that will be judged based on the cleverness of the application and the elegance of the code. Entries can be submitted at coderack.org until midnight CEST November 15th. Finalists are scheduled to be announced on the 1st of December and public voting will run for one month. The final winners will be announced on the 5th of January.

The first round of the contest will be judged by an elite panel of judges including Ben Bangert (Pylons framework), Chris Wanstrath and PJ Hyett of GitHub, Joshua Peek of 37Signals, Yehuda Katz of Engine Yard (also Rails core team member), Ryan Tomayko of Heroku, Core Rails team member Matt Aimonetti, and the Rails Envy team of Gregg Pollack and Jason Seifer. Once the finalists have been selected by the panel, the public will vote for the top prize winners.

Prizes have been donated by Bytemark Hosting, GitHub, Jetbrains, Mindmeister, Freelance Total, Heroku, Rackspace Hosting, Peepcode, BDDCasts, and Zenbe Shareflow. The top prize includes a dedicated quad core server package and is valued at over $3000. Every entrant will receive a credit from bddcasts.com and $30 credit from Heroku! All finalists will receive a package including Zenbe Shareflow subscriptions, a RubyMine license from JetBrains, and five credits from bddcasts.com. Details of all of the prize packages will soon be available on the contest website.

So, want to become a Rackstar? It’s easy, create some interesting code and submit it to the contest. We’ve already collected over 45 entries in just few days. Oh, and I’ve almost forgot, CodeRack site itself is using middlewares, one of them being Ryan Tomayko’s awesome Rack::Cache.