Jesse Storimer

Passing Open Files Between Processes with UNIX Sockets

2012-05-07T00:00:00-04:00

I want to share with you today a neat little technique I learned involving UNIX sockets.

In the land of Unix everything is a file. This is faithfully mirrored in Ruby with the IO class. The IO class models any so-called files on a Unix system. This includes stuff like File, TCPSocket, UDPSocket, all of which are subclasses of IO. Since these classes all share a common parent they also share a common API, as they should.

Here's a quick example:

require 'socket'

# open 3 different IO objects
file        = File.open('/etc/passwd', 'r')
tcp_socket  = TCPSocket.new('google.com', 80)
pipe        = IO.pipe

# write some data to the tcp socket and close
# the writing stream afterwards
tcp_socket.write("GET / HTTP/1.1\r\n")
tcp_socket.close_write

# write some data to the pipe and close
# the writable IO afterwards
pipe[1].write('some test data')
pipe[1].close_write

# read from all the IO objects in the
# same way. Yay for polymorphism and
# duck typing!
[file, tcp_socket, pipe[0]].each do |io|
  puts io.read
  io.close
end

This example shows how different subclasses of IO can be treated in a similar way. But, each subclass also implements some of its own methods. In some cases it overwrites the parent implementation, but some of the subclasses can do things that others can't. Enter UNIXSocket.

UNIX Sockets are Local

UNIX sockets have (almost) the exact same API as TCP sockets. The defining difference is that while TCP sockets can connect any two endpoints on the same network, UNIX sockets can only connect to other UNIX sockets on the same physical domain, ie. on the same machine. As such, the kernel can take some shortcuts and ensure that UNIX sockets are much faster than TCP sockets because they skip the entire networking layer.

Since UNIX Sockets are local and fast they make a great candidate for IPC. Besides sharing plain text data between processes you can also share open file descriptors between unrelated processes using UNIX Sockets. The most important word in the last sentence is unrelated.

Related Processes Share Open Files By Default

When I say related, I'm talking about a hierarchichal relationship. One process is related to another when it has a parent/child or parent/grandchild, etc., relationship.

I've written previously about how open files are shared between parent/child processes when forking. Related processes share file descriptors through the semnatics of fork(2). Even if the child process decides to exec(2) and become a totally different process it can still access any shared file descriptors.

But what if you want to pass a file descriptor to a totally unrelated process? You can! Using UNIX sockets.

Sharing File Descriptors Between Unrelated Processes

Let's imagine that I have a Ruby daemon process that listens on a UNIX socket. It listens on that socket to receive open TCP connections. When it gets one it will respond in its own special way. I could distribute this daemon as a rubygem, at which point developers could write clients that interact with it by sending open connections over the UNIX socket. By 'open' I mean that the client would accept a new connection, then pass it to the daemon over a UNIX socket and the daemon would write a response and close the connection. Fun!

The daemon could look like this:

require 'socket'

# This sockets listens for local connections.
listener = UNIXServer.new('/tmp/pingback.sock')
client = listener.accept

loop do
  # This is where an open connection is received.
  connection = client.recv_io

  # An exciting response. Echo!
  connection.write(connection.read)
  connection.close
end

Stick that code into a file and run it. Then it can be interacted with using a client like:

require 'socket'

# This socket listens for connections from the outside world.
listener = TCPServer.new('0.0.0.0', 4481)

# This socket connects up to the local daemon.
receiver = UNIXSocket.new('/tmp/pingback.sock')

Socket.accept_loop(listener) do |connection, _|
  # When it gets a new connection it passes it right to the
  # daemon process.
  receiver.send_io(connection)

  # Passing the connection to the daemon made a copy of it,
  # so now there are two open connections. If we didn't close
  # it here then the connection would remain open even after
  # it was closed by the daemon.
  connection.close
end

If you run both the daemon and the client you can test drive the two pieces using netcat:

$ echo boofar | nc localhost 4481
    boofar

The response is pretty unexciting, it's just an echo server. But realize what's happening: you're initiating a TCP connection to the client which passes that connection to the daemon, and it's the daemon that actually writes the response back to you.

Note that these two processes are started up independently and are unrelated. There's no forking semantics at play here.

Can't handle multiple clients atm.

Conclusion

So the important methods here were UNIXServer#send_io and UNIXServer#recv_io. The first sends an open file, the second receives one.

You could use this to pass open files between any two processes on your system. And you don't need to restrict yourself to two Ruby processes. Thanks to the fact that most modern languages support UNIX sockets you could write a client to interact with the Ruby daemon in Python, or even in C. I'll leave that as an exercise for the reader :)

4 Months of ebook Sales

2012-04-20T00:00:00-04:00

Four months ago, to the day, I launched an ebook about Unix programming for Ruby developers. Here I'll share how it came to be and some sales numbers.

Backstory

I've been interested in writing an ebook for a while. The first time I tried I didn't finish because I couldn't stay motivated. This time around I was able to stay motivated because I put up a fake sales page to test the idea before I even started writing. This is the same technique that Tim Ferriss talks about in the 4HWW. The validation this provided was what I needed to see it through.

Just as I was ready to start writing the book I left for a wilderness retreat. My family and I spent 2 and 1/2 weeks in the wilderness with no electricity or electronics of any kind. I expected to feel frustrated that I couldn't get started on the book, but it was the total opposite. Spending time away from everyday distractions allowed me to do some productive planning for the book, as well as have a great time. By the time I went home I had a good outline of what I wanted to write about.

Over the next few months I spent many late nights writing. I didn't take time off from my day job or have anyone working with me. I didn't work on it every night but usually several nights per week for ~2 hours.. After about 2 and 1/2 months of writing I had something I was pretty happy with. I basically wrote the book that I had wanted to read a year or two earlier, before I went ahead and learned the hard way.

Launch

I did zero promotion before launching the book. Literally no one knew about it besides me, my wife, and a few friends. The book is fully self-published so there was no publisher involved, no copyeditor, no technical editor, just me. I'm really grateful to the few people that I reached out to who provided valuable advice and reviews. Without their interaction I wouldn't have known what I had. If I were to do this again I would certainly offer a pre-launch mailing list to let people follow along as I worked on the book.

I was very nervous about putting something out there with my name on it and asking people to pay money for it. Once the book was 'done' I deliberated for a month on whether or not I would release it. After putting together a web site for the book (with the help of Copywriting for Geeks) and showing it to one more person who gave really positive feedback I decided to bite the bullet and see what would happen. I treated the whole thing as an experiment and had no idea what to expect.

Someone close to me asked what I would consider to be a successful outcome after launching the book. At first I said that if I can create something, put it online, and get even one person to pay for it then that's a huge accomplishment. After some thought I said that if the revenue were $1,000 in the first month (about 1 sale per day) I would consider it a success. To my surprise revenue was just over $1,000 on launch day alone. The only launch announcement I made was via Twitter.

Numbers

Total Revenue: $17,992 USD
Total # of Sales: 746

Click the image to see a high res version. This chart is straight from SumAll. Keep in mind that these numbers do not include processing fees or expenses, but discounts have been deducted.

The data points are all daily, so you can see there are highs and lows. The initial spike is simply my twitter announcement and friendly folks retweeting. The spike on day 3 happened when the book was linked in Ruby Weekly. The other two big spikes on the chart were the result of discount promotions I ran. All of the other smallish spikes are invariably two things: 1) Someone recommends the book on twitter, or 2) I publish content on my blog. You'll notice fewer sales in the last few weeks, certainly due in part to the fact that I haven't published to my blog.

I'm ecstatic with the results so far, but I have no idea how they compare. A pair of design ebook authors recently shared their launch numbers which were much higher. But they're in a different space selling to a different audience. I still haven't a clue how my results compare to other (e)books for programmers.

Aftermath

Anyone who has ever launched their own product will tell that they didn't just kick their feet back and watch the money come in. There's lots to do. Between answering emails, producing content to market the book, and adding new content to the book itself, I've continued to invest late night hours into the project at least a few nights per week.

On that note, there is a new chapter for the book as of today. If you've already purchased the book then you should already have gotten the email. If you haven't purchased the book then you can read the new chapter for free. If you learn something then I encourage you to check out the book.

Inspiration

So why am I sharing this info instead of just keeping quiet? I was always inspired when people who launched their own products shared their story. I promised myself that if I ever got anything off the ground I would do the same. This is me delivering on that promise and hoping that I might inspire just one person down the line to build that thing they really want to build.

Introduction to IPC in Ruby

2012-04-19T00:00:00-04:00

This is a sample chapter from my book Working With Unix Processes. Learning how stuff works under the hood is one of the most important things you can do as a developer. If this article pushes you in that direction then I encourage you to learn more about the book.

Up until now we've looked at related processes that share memory and share open resources. But what about communicating information between multiple processes?

This is part of a whole field of study called Inter-process communication (IPC for short). There are many different ways to do IPC but I'm going to cover two commonly useful method: pipes and socket pairs.

Our First Pipe

A pipe is a uni-directional stream of data. In other words you can open a pipe, one process can 'claim' one end of it and another process can 'claim' the other end. Then data can be passed along the pipe but only in one direction. So if one process 'claims' the position of reader, rather than writer, it will not be able to write to the pipe. And vice versa.

Before we involve multiple processes let's just look at how to create a pipe and what we get from that:

reader, writer = IO.pipe #=> [#<IO:fd 5>, #<IO:fd 6>]

IO.pipe returns an array with two elements, both of which are IO objects. Ruby's amazing IO class is the superclass to File, TCPSocket, UDPSocket, and others. As such, all of these resources have a common interface.

The IO objects returned from IO.pipe can be thought of something like anonymous files. You can basically treat them the same way you would a File. You can call #read, #write, #close, etc. But this object won't respond to #path and won't have a location on the filesystem.

Still holding back from bringing in multiple processes let's demonstrate communication with a pipe:

reader, writer = IO.pipe
writer.write("Into the pipe I go...")
writer.close
puts reader.read

outputs

Into the pipe I go...

Pretty simple right? Notice that I had to close the writer after I wrote to the pipe? That's because when the reader calls IO#read it will continue trying to read data until it sees an EOF (aka. end-of-file marker). This tells the reader that no more data will be available for reading.

So long as the writer is still open the reader might see more data, so it waits. By closing the writer before reading it puts an EOF on the pipe so the reader stops reading after it gets the initial data. If you skip closing the writer then the reader will block and continue trying to read indefinitely.

Pipes Are One-Way Only

reader, writer = IO.pipe
reader.write("Trying to get the reader to write something")

outputs

 >> reader.write("Trying to get the reader to write something")
IOError: not opened for writing
        from (irb):2:in `write'
        from (irb):2

The IO objects returned by IO.pipe can only be used for uni-directional communication. So the reader can only read and the writer can only write.

Now let's introduce processes into the mix.

In the chapter on forking I described how open resources are shared, or copied, when a process forks a child. Pipes are considered a resource, they get their own file descriptors and everything, so they are shared with child processes.

Here's a simple example of using a pipe to communicate between a parent and child process. The child indicates to the parent that it has finished an iteration of work by writing to the pipe:

reader, writer = IO.pipe

fork do
  reader.close
  
  10.times do
    # heavy lifting
    writer.puts "Another one bites the dust"
  end
end

writer.close
while message = reader.gets
  $stdout.puts message
end

outupts Another one bites the dust ten times.

Notice that, like above, the unused ends of the pipe are closed so as not to interfere with EOF being sent. There's actually one more layer when considering EOF now that two processes are involved. Since the file descriptors were copied there's now 4 instances floating around. Since only two of them will be used to communicate the other 2 instances must be closed. Hence the extra instances of closing.

Since the ends of the pipe are IO objects we can call any IO methods on them, not just #read and #write. In this example I use #puts and #gets to read and write a String delimited with a newline. I actually used those here to simplify one aspect of pipes: pipes hold a stream of data.

Streams vs. Messages

When I say stream I mean that when writing and reading data to a pipe there's no concept of beginning and end. When working with an IO stream, like pipes or TCP sockets, you write your data to the stream followed by some protocol-specific delimiter. For example, HTTP uses a series of newlines to specify delimit the headers from the body.

Then when reading data from that IO stream you read it in one chunk at a time, stopping when you come across the delimiter. That's why I used #puts and #gets in the last example: it used a newline as the delimiter for me.

As you may have guessed it's possible to communicate via messages instead of streams. We can't do it with pipe, but we can do it with Unix sockets. Without going into too much detail, Unix sockets are a type of socket that can only communicate on the same physical machine. As such it's much faster than TCP sockets and is a great fit for IPC.

Here's an example where we create a pair of Unix sockets that can communicate via messages:

require 'socket'
Socket.pair(:UNIX, :DGRAM, 0) #=> [#<Socket:fd 15>, #<Socket:fd 16>]

This creates a pair of UNIX sockets these sockets that are already connected up to each other. These sockets communicate using datagrams, rather than a stream. In this way you write a whole message to one of the sockets and read a whole message from the other socket. No delimiters required.

Here's a slightly more complex version of the pipe example where the child process actually waits for the parent to tell it what to work on, then it reports back to the parent once it's finished the work:

require 'socket'

child_socket, parent_socket = Socket.pair(:UNIX, :DGRAM, 0)
maxlen = 1000

fork do
  parent_socket.close
  
  10.times do
    instruction = child_socket.recv(maxlen)
    child_socket.send("#{instruction} accomplished!", 0)
  end 
end 
child_socket.close

5.times do
  parent_socket.send("Heavy lifting", 0)
end 
5.times do
  parent_socket.send("Feather lifting", 0)
end 

10.times do
  $stdout.puts parent_socket.recv(maxlen)
end

So whereas pipes provide uni-directional communication, a socket pair provides bi-directional communication. The parent socket can both read and write to the child socket, and vice versa.

Remote IPC?

IPC implies communication between processes running on the same machine. If you're interested in scaling up from one machine to many machines while still doing something resembling IPC there are a few things to look into. The first one would simply be to communicate via TCP sockets. This option would require more boilerplate code than the others for a non-trivial system. Other plausible solutions would be RPC (remote procedure call), a messaging system like ZeroMQ, or the general body of distributed systems.

In the Real World

Both pipes and socket pairs are useful abstractions for communicating between processes. They're fast and easy. They're often used as a communication channel instead of a more brute force approach such as a shared database or log file.

As for which method to use: it depends on your needs. Keep in mind that pipes are uni-directional and socket pairs are bi-directional when weighing your decision.

For a more in-depth example have a look at the Spyglass Master class in the included Spyglass project. It uses a more involved example of the code you saw above where many child processes communicate over a single pipe with their parent process.

System Calls

Ruby's IO.pipe maps to pipe(2), Socket.pair maps to socketpair(2). Socket.recv maps to recv(2) and Socket.send maps to send(2).

Daemon Processes in Ruby

2012-04-19T00:00:00-04:00

Daemon processes are processes that run in the background, rather than under the control of a user at a terminal. Common examples of daemon processes are things like web servers, or database servers which will always be running in the background in order to serve requests.

Daemon processes are also at the core of your operating system. There are many processes that are constantly running in the background that keep your system functioning normally. These are things like the window server on a GUI system, printing services or audio services so that your speakers are always ready to play that annoying 'ding' notification.

The First Process

There is one daemon process in particular that has special significance for your operating system. We talked in a previous chapter about every process having a parent process. Can that be true for all processes? What about the very first process on the system?

This is a classic who-created-the-creator kind of problem, and it has a simple answer. When the kernel is bootstrapped it spawns a process called the init process. This process has a ppid of 0 and is the 'grandparent of all processes'. It's the first one and it has no ancestor. Its pid is 1.

Creating Your First Daemon Process

What do we need to get started? Not much. Any process can be made into a daemon process.

Let's look to the rack project for an example here. Rack ships with a rackup command to serve applications using different rack supported web servers. Web servers are a great example of a process that will never end; so long as your application is active you'll need a server listening for connections.

The rackup command includes an option to daemonize the server and run it in the background. Let's have a look at what that does.

Diving into Rack

def daemonize_app
  if RUBY_VERSION < "1.9"
    exit if fork
    Process.setsid
    exit if fork
    Dir.chdir "/" 
    STDIN.reopen "/dev/null"
    STDOUT.reopen "/dev/null", "a" 
    STDERR.reopen "/dev/null", "a" 
  else
    Process.daemon
  end 
end

Lots going on here. Let's first jump to the else block. Ruby 1.9.x ships with a method called Process.daemon that will daemonize the current process! How convenient!

But don't you want to know how it works under the hood? I knew ya did! The truth is that if you look at the MRI source for Process.daemon and stumble through the C code it ends up doing the exact same thing that Rack does in the if block above.

So let's continue using that as an example. We'll break down the code line by line.

Daemonizing a Process, Step by Step

exit if fork

This line of code makes intelligent use of the return value of the fork method. Recall from the forking chapter that fork returns twice, once in the parent process and once in the child process. In the parent process it returns the child's pid and in the child process it returns nil.

As always, the return value will be truth-y for the parent and false-y for the child. This means that the parent process will exit, and as we know, orphaned child processes carry on as normal.

If a process is orphaned then what happens when you ask for Process.ppid?

This is where knowledge of the init process becomes relevant. The ppid of orphaned processes is always 1. This is the only process that the kernel can be sure is active at all times.

This step is imperative when creating a daemon because it causes the terminal that invoked this script to think the command is done, returning control to the terminal and taking it out of the equation.

Process.setsid

Calling Process.setsid does three things:

The process becomes a session leader of a new session
The process becomes the process group leader of a new process group
The process has no controlling terminal

To understand exactly what effect these three things have we need to step out of the context of our Rack example for a moment and look a little deeper.

Process Groups and Session Groups

Process groups and session groups are all about job control. By 'job control' I'm referring to the way that processes are handled by the terminal.

We begin with process groups.

Each and every process belongs to a group, and each group has a unique integer id. A process group is just a collection of related processes, typically a parent process and its children. However you can also group your processes arbitrarily by setting their group id using Process.setpgrp(new_group_id).

Have a look at the output from the following snippet.

puts .getpgrp
puts Process.pid

If you ran that code in an irb session then those two values will be equal. Typically the process group id will be the same as the pid of the process group leader. The process group leader is the 'originating' process of a terminal command. ie. If you start an irb process at the terminal it will become the group leader of a new process group. Any child processes that it creates will be made part of the same process group.

Try out the following example to see that process groups are inherited.

puts Process.pid
puts Process.getpgrp

fork { 
  puts Process.pidputs Process.getpgrp
}

You can see that although the child process gets a unique pid it inherits the group id from its parent. So these two processes are part of the same group.

You'll recall that we looked previously at Orphaned Processes. In that section I said that child processes are not given special treatment by the kernel. Exit a parent process and the child will continue on. This is the behaviour when a parent process exits, but the behaviour is a bit different when the parent process is being controlled by a terminal and is killed by a signal.

Consider for a moment: a Ruby script that shells out to a long-running shell command, eg. a long backup script. What happens if you kill the Ruby script with a Ctrl-C?

If you try this out you'll notice that the long-running backup script is not orphaned, it does not continue on when its parent is killed. We haven't set up any code to forward the signal from the parent to the child, so how is this done?

The terminal receives the signal and forwards it on to any process in the foreground process group. In this case, both the Ruby script and the long-running shell command would part of the same process group, so they would both be killed by the same signal.

And then session groups...

A session group is one level of abstraction higher up, a collection of process groups. Consider the following shell command:

git log | grep shipped | less

In this case each command will get its own process group, since each may be creating child processes but none is a child process of another. Even though these commands are not part of the same process group one Ctrl-C will kill them all.

These commands are part of the same session group. Each invocation from the shell gets its own session group. An invocation may be a single command or a string of commands joined by pipes.

Like in the above example, a session group may be attached to a terminal. It might also not be attached to any terminal, as in the case of a daemon.

Again, your terminal handles session groups in a special way: sending a signal to the session leader will forward that signal to all the process groups in that session, which will forward it to all the processes in those process groups. Turtles all the way down :)

There is a system call for retrieving the current session group id, getsid(2), but Ruby's core library has no interface to it. Using Process.setsid will return the id of the new sesssion group it creates, you can store that if you need it.

So, getting back to our Rack example, in the first line a child process was forked and the parent exited. The originating terminal recognized the exit and returned control to the user, but the forked process still has the inherited group id and session id from its parent. At the moment this forked process is neither a session leader nor a group leader.

So the terminal still has a link to our forked process, if it were to send a signal to its session group the forked process would receive it, but we want to be fully detached from a terminal.

Process.setsid will make this forked process the leader of a new process group and a new session group. Note that Process.setsid will fail in a process that is already a process group leader, it can only be run from child processes.

This new session group does not have a controlling terminal, but technically one could be assigned.

exit if fork

The forked process that had just become a process group and session group leader forks again and then exits.

This newly forked process is no longer a process group leader nor a session leader. Since the previous session leader had no controlling terminal, and this process is not a session leader, it's guaranteed that this process can never have a controlling terminal. Terminals can only be assigned to session leaders.

This dance ensures that our process is now fully detached from a controlling terminal and will run to its completion.

Dir.chdir "/"

This changes the current working directory to the root directory for the system. This isn't strictly necessary but it's an extra step to ensure that current working directory of the daemon doesn't disappear during it's execution.

This avoids problems where the directory that the daemon was started from gets deleted or unmounted for any reason.

STDIN.reopen "/dev/null"
STDOUT.reopen "/dev/null", "a" 
STDERR.reopen "/dev/null", "a"

This sets all of the standard streams to go to /dev/null, a.k.a. to be ignored. Since the daemon is no longer attached to a terminal session these are of no use anyway. They can't simply be closed because some programs expect them to always be available. Redirecting them to /dev/null ensures that they're still available to the program but have no effect.

In the Real World

As mentioned, the rackup command ships with a command line option for daemonizing the process. Same goes with any of the popular Ruby web servers.

If you want to dig in to more internals of daemon processes you should look at the daemons rubygem.

If you think you want to create a daemon process you should ask yourself one basic question: Does this process need to stay responsive forever?

If the answer is no then you probably want to look at a cron job or background job system. If the answer is yes, then you probably have a good candidate for a daemon process.

System Calls

Ruby's Process.setsid maps to setsid(2), Process.getpgrp maps to getpgrp(2). Other system calls mentioned in this chapter were covered in detail in previous chapters.

How to Give a Good Tech Talk

2012-04-12T00:00:00-04:00

So I'm giving a talk next week. My first.

It's an internal talk at Shopify, and I want it to be good. But I have no idea what I'm doing, I've never done this before.

So what did I do? I turned to the internet and found a bunch of great resources. While preparing the talk this week I've been looking over them and getting some great advice about how to go about giving and preparing talks. Without further ado here's the list:

p.s. I've personally consumed all of these resources and definitely recommend them.

Free

The secret structure of great talks - Really enlightening TED video. See the structure underlying both 'I Have a Dream' and the 2007 iPhone launch announcement.
HOWTO Give a Good Presentation - Good tips to keep in mind.
Slide Design for Developers - Make your slides have an impact but support what you're saying, rather than sayint it for you.
Improve Your Technical Slides - Nine easy tips that will help you communicate better at your next conference.

Paid

Speaking for Hackers - An in-progress ebook of tips. Covers end-to-end from proposing to planning to speaking.
The Art of Speaking - Watch a respected speaker prepare a talk from scratch in 90 mins. Lots of great advice.

Exemplary Talks

How Github uses Github to Build Github - Great talk, great slides. Follows the pattern revealed in the TED talk.
Confident Code - The example code used in this one is particularly compelling. Also conforms to the pattern from the TED talk.

Let me know if I'm missing any other great resources. I'll continue to update this post as I come across more.

Put Your Inbox in the Upstairs Bathroom

2012-03-22T00:00:00-04:00

My house has two bathrooms.

One is upstairs and one is downstairs. In order to understand the story you need to know that the bedroom and main bathroom are upstairs. Downstairs is just a half bath, as they say. I work from home and spend most of my day downstairs. When I have to use the bathroom, I use the downstairs one; no surprise there.

A few weeks ago a peculiar thing happened: my wife took our toothbrushes, which usually reside in the upstairs bathroom, and put them in the downstairs bathroom. I can usually manage to brush my teeth twice a day: once in the morning after I get up and once at night before I go to bed. I'd really like to brush my teeth after every meal. I'm currently eating tons of fresh fruit and vegetables which really takes its toll on teeth, so I really should be brushing several times per day.

Alas, when the toothbrush is upstairs it's out of sight and out of mind. As such, I rarely venture upstairs in the middle of the day simply to brush my teeth and return downstairs.

However, when my toothbrush was brought downstairs I found myself brushing my teeth far more often. I was reminded of its presence every time I went to the bathroom and found it far easier to take an extra few minutes to brush my teeth several times per day.

So going forward I'm going to keep my toothbrush in the downstairs bathroom. My email inbox, however, will be moved to the upstairs bathroom.

For the longest time I kept my inbox in the downstairs bathroom. I'll stop mincing words and say that I checked my inbox too often. Why is this a problem? Checking email is a really great way for me to break my concentration, reduce my motivation, and totally switch contexts.

So why was I checking so often? There are certainly psychological factors, wanting to respond quickly to coworkers, procrastination, but one of the biggest factors was that checking my inbox was just too plain easy.

I was using a nice GUI email client. It wanted to send me growl notifications whenever I got a new message and show me how many unread messages I had on its Dock icon. Needless to say, I turned off these features. However, this made no difference, it was still too easy to check. Anytime I was programming I could quickly press Cmd-Tab and just have a peek at my inbox. This quickly became a reflex action. I was checking my inbox every minute or so as I waited for a test run. It was literally optimized down to a sub-second operation.

So how did I remedy this? I now read my email using mutt(1).

There are a few things that make this better. The main thing is when I want to check my inbox I perform the following steps:

Open a new terminal session.
Launch mutt.
Watch in detail as mutt makes the initial connection, authenticates, and pulls down new messages.
Read email.
Close mutt.

I've found this technique very effective in clobbering my addiction to checking my inbox. Why is it so effective? My mail is never just sitting there waiting for me. If I want to check my inbox it takes about 15 seconds of my time from having the idea to seeing my new mail. Believe it or not, this is often enough to prevent me from even getting started with checking my inbox. A sub-second operation became a 15-second operation.

My email inbox used to be even closer than the downstairs bathroom. It was akin to keeping my toothbrush proppped up on my shoulder in case I felt like I had an extra few seconds to brush my teeth. Now my inbox is at a safe distance. About 15 seconds away. About the same distance as my upstairs bathroom.

A Unix Shell in Ruby - Part 4: Pipes

2012-03-18T00:00:00-04:00

In this article we'll investigate a peculiarity. It will reveal an impostor and eventually lead to implementing shell pipelines. You may want to read part 1, part 2, or part 3.

First, the Peculiarity

Observe these interactions with the shirt shell:

$ ./shirt 
$ ls
LICENSE README shirt
$ ls | grep README
README
$ ls > output.txt
$ cat output.txt && echo 'win'
LICENSE
README
output.txt
shirt
win

These are typical shell interactions, except that shirt hasn't implemented these features yet. There's no code in the project to handle features such as pipes, output redirection, conditional execution, etc. These things are basically syntactic sugar that the shell provides on top of the system capabilities, so it's up to shirt to implement it. So where does this peculiarity come from?

If you look at the shirt source it's not too hard to figure out. There are only two code paths that a command can take at the moment, either it's a builtin command or it gets sent to exec. None of the commands we used in the example session above were builtins so the strange behaviour must be coming from exec.

Before I unearth the mystery I want to make clear the expected behaviour of exec and then I'll do the big reveal.

The Base Case

In part 1 I explained the semantics of exec(2). It transforms the current process into a new process based on the command given as input.

That makes perfect sense when you do something like exec('ls'). The current process becomes an instance of ls. But what about something like exec('ls | grep README')? Based on my description this method invocation doesn't really make sense. There are actually two programs specified in that command joined by a pipe. The current process can't become two other processes, so there must be something else at play here.

Before we look at the source for a Ruby VM I want to show what is happening with the processes visually. This is output from a tool called pstree(1). It prints a textual representation of the process hierarchy clearly showing child processes of a given process, for instance^1.

The following shirt session:

$ ./shirt
$ ls
LICENSE README shirt

yields this from pstree(1):

-+= 24853 jessestorimer ruby ./shirt
 \--- 31837 jessestorimer ls

This is exactly what we expected. We have a shirt process with one child, an ls process. This comes about because we created a child process with fork, then used exec to transform that to an ls process. So far so good.

An Impostor in the Lineage

The following shirt session:

$ ./shirt
$ ls | grep README
README

yields this from pstree(1):

-+= 57816 jessestorimer ruby ./shirt
 \-+- 58082 jessestorimer sh -c ls | grep README
   |--- 58083 jessestorimer ls
   \--- 58084 jessestorimer grep README

That one looks different. The first child of shirt is sh! sh, often referred to as /bin/sh is the Bourne shell, predecessor to Bash (a.k.a Bourne-Again SHell). It's the inspiration for the syntax and functionality of most modern shells and is available on most any Unix system. But let's go back and understand the pstree(1) output.

So the first child of shirt is sh. Then sh has two children in ls and grep. So that's how all of the commands I showed above are working: exec is actually sending that input to a different shell. This is certainly against the spirit of our project here and we want to implement everything that would otherwise be implemented by a shell like sh.

The Big Reveal

Let's look at the Rubinius source code for exec and see where it decides to use a subshell:

 1     # kernel/common/process.rb
 2 
 3     def exec(cmd, *args)
 4       if args.empty? and cmd.kind_of? String
 5         raise Errno::ENOENT if cmd.empty?
 6         if /([*?{}\[\]<>()~&|$;'`"\n\s]|[^\w-])/o.match(cmd)
 7           Process.perform_exec "/bin/sh", ["sh", "-c", cmd]
 8         else
 9           Process.perform_exec cmd, [cmd]
10         end
11       else
12         if cmd.kind_of? Array
13           prog = cmd[0]
14           name = cmd[1]
15         else
16           name = prog = cmd
17         end
18 
19         argv = [name]
20         args.each do |arg|
21           argv << arg.to_s
22         end
23 
24         Process.perform_exec prog, argv
25       end
26     end
27     module_function :exec

The most relevant lines for our purposes are #4 and #6. On line #4 it enters the first if block if it's given only a String as an argument. You can see that line #6 has a gnarly regex which, if matched, will cause exec to actually create an instance of /bin/sh and pass the string along to it. How can we prevent this from happening in shirt?

The simplest way is: never pass a string of input directly in to exec. You can see that, in that case, we'd enter the outer else block where there's no chance of shelling out to a different shell. The only exception to the rule is if you pass a 'plain' string to exec. This is something like exec('ls'). In that case it won't match the gnarly regex and will hit line #9 which does an exec on that command without a subshell.

So, the first argument given to exec should be the name of the program to be executed, while the rest of the arguments will be treated as a list of the command-line arguments to that program. In Ruby we know this as ARGV.

Here's the simple change we can make to shirt so that it won't end up invoking the Bourne shell:

  ...
  pid = fork {
    exec command, *arguments
  }
  ...

Previously we had just passed the line into exec here. The line being a string of input from the user.

Aftermath

This is great, now we can be sure that shirt isn't leaning on /bin/sh for features. The downside is that the stuff we were doing at the beginning doesn't work any more. Let's try a pipeline:

$ ./shirt
$ ls | grep README
ls: grep: No such file or directory
ls: |: No such file or directory
README

That's some pretty strange output. In this case ls is being sent to exec as the command to execute and the rest of the line is being treated as the argument list for ls. Since ls expects its arguments to be names of files or directories it's telling us that its first two arguments don't exist on the filesystem. Notice that it successfully printed the last entry? That's because it's in a directory that actually contains a README file, so that one does work correctly.

So shirt needs pipes. Time to start plumbing.

Shell Pipelines Demystified

I think shell pipelines are misunderstood. For the longest time I had no idea how they worked. More than once I read someone casually tossing up something like:

Yeah, the shell just uses the pipe(2) system call to hook up the stdout from one process to the stdin of the other.

Oh?

I had some idea what stdin and stdout was, but I didn't know anything about a pipe(2) system call. It was certainly harder to understand than the simplicity of using pipes in my shell.

The reason I'm telling you this is because I want you to understand that there's no mystery behind pipes in the shell. Going forward I'm going to refer to pipes in the shell as 'shell pipelines' or just 'pipelines'. When I say 'pipe' I'm going to be referring to the programmatic concept of the pipe, which is used to implement shell pipelines but can also be used for a myriad of other things.

The Shell Pipeline

First things first, we know we're going to need a 'pipe'. What does that look like? Let's ask irb.

>> IO.pipe
=> [#<IO:fd 5>, #<IO:fd 6>]

In Ruby, IO.pipe will create a pipe using the pipe(2) system call. It returns an array with two IO objects. These IO objects are a bit like anonymous files. They can read from them and written to like files, but they have no location on the filesystem.

Something else about the pipe: Although it returns two IO objects the pipe only allows for a uni-directional stream of data. So the first IO object in that array is read-only, the second IO object is write-only. In this way data can travel only one way along the pipe.

So, step one in creating a shell pipeline is to create the pipe. Let's represent this with a diagram. This is a shirt process:

So our shell now has a pipe. Let's say, for example, that we're going to be creating the following pipeline: ls | grep README. There are two processes there so shirt must fork two child processes.

Remember that fork creates an exact copy of the calling process. So the child processes got their own copy of the pipe that the parent process created.

Eventually the left child will become ls and the right will become grep. But first we need to set up the pipe.

We're going to tell the left child (the one that will become ls) that it should redirect its $stdout to the write-only IO object. In this way anything that ls writes to its $stdout will fill up the pipe and wait there for the grep process to read it.

Similarly we'll tell the right child (the one that will become grep) that it should read its $stdin from the read-only IO object. This will be the receiving end of whatever the ls process writes to the pipe. That might look something like this:

There's a bit of housecleaning to do still but we'll look at that when we step through the code.

Now the first child process does its exec and becomes ls. The second process does the same and becomes grep. Just to give you a visual:

That's a rough overview of how a shell pipeline works from the view of processes. Now let's dive into the implementation in shirt.

The Ending, Then the Beginning

I'm going to start by pasting the code shirt has after implementing pipelines. Then we'll step through the changes line by line.

  ...
    commands = split_on_pipes(line)

    placeholder_in = $stdin
    placeholder_out = $stdout
    pipe = []

    commands.each_with_index do |command, index|
      program, *arguments = Shellwords.shellsplit(command)

      if builtin?(program)
        call_builtin(program, *arguments)

      else
        if index+1 < commands.size
          pipe = IO.pipe
          placeholder_out = pipe.last
        else
          placeholder_out = $stdout
        end

        spawn_program(program, *arguments, placeholder_out, placeholder_in)

        placeholder_out.close unless placeholder_out == $stdout
        placeholder_in.close unless placeholder_in == $stdin
        placeholder_in = pipe.first
      end
    end

    Process.waitall
  end

The first notable change is this line, after we get a line of input from $stdin:

commands = split_on_pipes(line)

...and here's the implementation:

def split_on_pipes(line)
  line.scan( /([^"'|]+)|["']([^"']+)["']/ ).flatten.compact
end

This method takes a line of input and splits it using pipes as a delimiter. It's a bit hard to grok but the gist of it is that it matches one or more characters that are not a quote or a pipe or it matches a quote, followed by some non-quote characters, followed by another quote. Full credit goes to http://stackoverflow.com/a/4970136/1124616. It returns an array of command strings.

The reason we dance around the quotes and go to that trouble is that it should be possible to include a pipe character inside a quoted string, to pass as an argument to a program. If we were just to use a naive line.split('|') then we'd end up splitting on pipe characters that were passed inside quotes as program arguments.

    placeholder_in = $stdin
    placeholder_out = $stdout
    pipe = []

This is simply some boilerplate setup. We'll talk about these variables further into the source.

    commands.each_with_index do |command, index|
      program, *arguments = Shellwords.shellsplit(command)

      if builtin?(program)
        call_builtin(program, *arguments)

Now that we can have multiple commands running at one time we need to process each one individually, hence the loop.

There's something notable here about the handling of builtins: it hasn't changed. Even though we'll be supporting pipes in the shell the builtin commands will not be able to use pipes in the same way that subprocesses do. I hope the reason why becomes clear as we look at the next two blocks of code, but I'll be sure to remind you when the time comes.

      if index+1 < commands.size
        pipe = IO.pipe
        placeholder_out = pipe.last
      else
        placeholder_out = $stdout
      end

This is some setup before we actually spawn a new process for the current command. This particular piece of code is responsible for setting up the future $stdout of the current command.

The else block takes care of the case where the current command is the last one in the pipeline. In that case its $stdout should be set to $stdout of the shell. In this way the last command will print its output back to the terminal.

The if block takes care of the other commands in the pipeline. It creates a new pipe and assigns the future $stdout of the current command to the write-only IO object that it returned.

Then we pass the program, arguments, and the placeholder stdin and stdout to the spawn_program method.

        spawn_program(program, *arguments, placeholder_out, placeholder_in)

Here's the implementation:

 def spawn_program(program, *arguments, placeholder_out, placeholder_in)
  fork {
    unless placeholder_out == $stdout
      $stdout.reopen(placeholder_out)
      placeholder_out.close
    end

    unless placeholder_in == $stdin
      $stdin.reopen(placeholder_in)
      placeholder_in.close
    end

    exec program, *arguments
  }
end

The fork + exec parts of this code are the same before, but there's new code too. There's a block of code to reopen $stdout to point to the stdout placeholder we assigned earlier, this would be a write-only end of a pipe. If there is no placeholder stdout (ie. this is the last command in a pipeline) then we don't modify $stdout.

We use IO#reopen to change $stdout to point to a different IO object. We could have instead passed an instance of File to have $stdout write to a file.

We give the same treatment to $stdin below that.

One very important point about this code: the placeholder IO objects are closed after being passed to #reopen. Why? We need a quick review of what happens with IO#reopen. So $stdout.reopen(placeholder_out) will reassociate $stdout with the file descriptor that's pointed at by placeholder_out. So now there are two references to the same file descriptor. Once we do the exec and ls finishes writing to its $stdout it will close its $stdout. This signals the process listening on the read-only end of the pipe that there is no more data to be sent so it can stop listening. But, if we didn't close placeholder_out then the process listening on the read-only end of the pipe could never stop listening for more data because there is still an open reference to the write-only end of the pipe that might be written to. So, make sure to close references to IO objects you're not going to use.

In the end the program gets passed to exec and the modified $stdout and $stdin are shared with the executed program.

This is a good time to bring up builtins again. The reason that builtins can't participate in pipelines in this way is because they don't have their own $stdin and $stdout to redirect. They share these with shirt since they're called in the context of the shell itself. So we'll have to hook them up in a different way, we'll save that for a later article.

Lastly:

        placeholder_out.close unless placeholder_out == $stdout
        placeholder_in.close unless placeholder_in == $stdin
        placeholder_in = pipe.first
      end
    end

    Process.waitall

The first two lines here are related precisely to what I explained in the last section. The parent process also holds references to the IO objects from the pipes. In order for the child processes to communicate properly the parent process needs to close its references to those IO objects.

The next line assigns the future $stdin for the next process to the read-only IO object returned from the pipe generated for the current command. Putting it at the end of the loop ensures that every command in the pipeline gets its $stdin reassociated, except for the first command through the loop. This makes perfect sense because the first command can't take input from a command that came before it, because there isn't one.

The last line here tells the current process to wait for all child processes to exit. Since we're now spawning multiple processes at one time we have to wait for all of them to finish before we can return control to the terminal. Currently shirt doesn't inspect the return value from that call to check for failures, etc. but it'll need to do that in the future.

Whew!

If you're still reading then I hope you learned a thing or two and now understand how simple shell pipelines are. The more I look at the implementation of a Unix shell the more I realize how many system primitives were originally created for the purpose of implementing a shell.

Next time around we'll implement another feature that was being handled by a subshell: io redirection.

If you enjoy learning about this stuff then check out Working With Unix Processes because it's all about Unix programming for the Rubyist.

The source for this article is available on github.

¹ I got this data by first installing pstree through homebrew. Then I started up a shirt instance with ./shirt. I used ps(1) to get the pid of the shirt process. Then I fed that into pstree as pstree <pid> to get the output you see.

eBooks for Programmers Should Include a TXT

2012-03-13T00:00:00-04:00

If I'm not in the terminal then I'm not doing real work.

Recently I had to refer to my book while I was working on a project. I work in a full-screen terminal and don't like to leave it. The world outside my terminal is filled with distractions, bright colours, and menus. My textual environment is where my real work happens.

So to consult my book I opened the PDF version and kept switching context back and forth between the PDF and the terminal. Have you tried copying and pasting code from a PDF lately? Not perfect.

I had the same experience when I was reading Objects on Rails. As I was reading the text I wanted to follow along with the code examples. I was annoyed with switching back and forth between my graphical reader and my programming environment.

A few days later I read this. It's a free eBook about Linux, available in several formats including TXT. Brilliant!

This made perfect sense to me. As a programmer I'm used to working in the terminal. If I'm going to be referring to a technical book while I'm working then, of course, I want to be able to read the book from my terminal.

Back to my original issue: I converted both my book and my copy of Objects on Rails to TXT files¹ and set this up:

$ alias wwup='less /path/to/workingwithunixprocesses.txt'
$ alias oor='less different//path/to/objectsonrails.txt'

Now when I'm working in a terminal I can just open a new session and type wwup or oor to read the books.

The best part about this is that the book is now in the realm of the powerful text processing tools offered in the terminal. There's simple things like piping the book text into grep(1) to search for something, but the most useful tool is certainly less(1). Notice that I loaded the text with less(1) as part of the alias I created.

less(1) is so useful for this kind of stuff. Vim users will be right at home with its commands. You can scroll the text up and down line-by-line using j and k respectively. Use Ctrl-d and Ctrl-u to scroll the text up and down page-by-page. Besides the basic movement commands the most useful feature for reading is incremental search. Press / and start typing to begin a search. Any matches in the text will be highlighted, at this point you can use n to see the next match and N to see the previous match.

It's also really easy to play with some code while keeping the text open, the exact thing I wanted to do when reading through Objects on Rails. You can shell out from less(1) by prefixing a command with a !. So you can use !irb to open an irb session right from less(1). Copy and paste code that you're reading right into the irb session for easy tinkering. If I'm working on something more incremental I just split my shell horizontally, reading the book on one side and coding on the other side.

I think I'll definitely start doing this for programming books that I'm reading. Especially for ones that are useful to keep around as references.

Working With Unix Processes now ships with a TXT version alongside the other formats. The book will be available at 40% off until this Friday because I think this is fun and I want you to have it.

Happy Spring!

¹ I used pdftotext(1) (part of xpdf) to convert PDF files to TXT files. I used html2text to convert HTML files to TXT files.

A Unix Shell in Ruby - Part 3: A Login Shell and the PATH

2012-03-10T00:00:00-05:00

First, A Word of Warning

In this article I'm going to set up shirt as the default login shell for my system to show what will happen when it's treated like every other shell. Be wary of doing this yourself. I definitely encourage you to follow along at home, but make sure you read through the article first so you know what to expect and don't end up with a broken shell :)

Changing the Login Shell

Each user can specify which shell they want to use when opening a new terminal or logging in to the sytem.

Changing the login shell can be done with the chsh(1) command. On OSX it can also be changed through the GUI. Let's change my login shell to shirt.

$ sudo chsh -s /path/to/shirt `whoami`

If I do that and launch a new terminal it dies immediately :/ So it appears that something didn't work.

With a bit of digging I found that this is due to the 'shebang' line at the beginning of the program. It tells the system to use the version of ruby specified by env(1). The env(1) command has everything to do with environment variables. Let's talk about those.

Environment Variables

Environment variables are a process-generic method of sharing data. I say process-generic because most any programming language has a way of reading and writing environment variables. I'll be referring to the collection of current environment variables as 'the environment'.

We'll talk more about the environment in a bit, what we care about right now is how come shirt died when we booted it as a login shell?

One of the most important environment variables is called PATH. The PATH is an ordered list of directories specifying where the shell and the system calls should look for programs to execute. For example, a common PATH looks like this:

$ echo $PATH
/usr/local/bin:/usr/bin:/bin

Each directory is separated by a colon. This PATH tells the system to first look for programs in /usr/local/bin, then, if it finds nothing there, look in /usr/bin, then /bin. In this way you can have a custom install of vim in /usr/local/bin that would be favoured over the system default vim in /usr/bin.

How the PATH Broke Our Login Shell

When a login shell boots it has no customization. The system default, when booting a new login shell, is to set the PATH to the following:

/usr/bin:/bin:

So when shirt boots in this way the shebang line looks in the environment to find ruby. With this PATH the only version it will find is the system default at /usr/bin/ruby. Since I'm on OS X the system default is MRI 1.8.7. So shirt is being booted with this ruby.

In my everyday shell I have the rbenv shims at the start of my PATH so that when I run the ruby command it's always handled by rbenv.

As a responsible Ruby developer I have my global rbenv version set to MRI 1.9.3. Let's simulate what's happening with the login shell by trying to boot shirt with the system ruby:

$ /usr/bin/ruby shirt
shirt:5: syntax error, unexpected '=', expecting '|'
  'cd' => lambda { |dir = ENV["HOME"]| Dir.chdir(dir) },
                         ^
shirt:5: syntax error, unexpected '}', expecting $end
  'cd' => lambda { |dir = ENV["HOME"]| Dir.chdir(dir) },

Aha. So we're using some syntax that is only supported in Ruby 1.9. I'm OK with that, but that means we'll need to have 1.9 installed as the system default in order for shirt to be a login shell. Rather than actually building Ruby and replacing the version at /usr/bin/ruby I'm just going to move that one out of the way for now and symlink in my rbenv ruby.

$ mv /usr/bin/ruby /usr/bin/ruby.orig
$ ln -s `rbenv which ruby` /usr/bin/ruby

Now if I open a new shell it works!

Back on the Path

If you actually try to use this as your login shell you'll probably be frustrated, there's many things that still aren't implemented. Let's continue working with the PATH.

Hypothetically, the first thing I need to do in this shell session is work with redis-cli. I installed redis through homebrew, so that program was installed into /usr/local/bin. If I try running redis-cli from inside the shirt login shell I get an error saying that it couldn't be found.

I can use the env(1) command to inspect the current environment variables and I see that my PATH, as expected, doesn't include /usr/local/bin. Hence it can't find my redis-cli. So we need a way to set environment variables.

Since this will need to change the state of the shell itself, we'll implement it as a builtin.

Like Bash?

How does bash handle this? It uses a command called export(1). Here's how we'd add a custom value to the PATH in bash.

$ export PATH=/usr/local/bin:$PATH

export(1) is telling bash to set the variable PATH to the previous value of PATH (specified by $PATH) prepended with /usr/local/bin.

Personally, I think export is a little unclear as far as names go, so we'll use set to specify changes to environment variables shell-wide.

Setting Environment Variables

Here's the implementation for shirt:

BUILTINS = {
  ...
  'set' => lambda { |args|
    key, value = args.split('=')
    ENV[key] = value
  }
}

Notice how we simply set a key on the ENV constant? ENV is the Ruby interface to environment variables for the current process.

It's a pretty naive approach, but it works. When using the set command it simply splits the first argument on the '=' character. The first part is assumed to be the variable name, the second part is considered to be its value.

Now issue the following commands to change and inspect the PATH for our shell:

-> set PATH=/usr/local/bin:/usr/bin:/bin
-> env

Notice that I specified the PATH in full, rather than referring to the existing value ($PATH) as we did with bash. shirt doesn't yet have a way to substite variables (or substitute other things a la backticks) into the command string. We'll tackle that problem all at once in another article.

Now when I try to run my redis-cli from /usr/local/bin it works just fine!

Connecting the Dots

This comes full circle when we take into account that 1) a child process inherits the environment of its parent, and 2) the environment is preserved through an exec.

That's how come shirt was able to work previously and find programs in /usr/local/bin. I had added /usr/local/bin to my PATH as part of my .bashrc file, so when I would exec or just launch an instance of shirt it would inherit that environment, along with the customized PATH.

Similarly, when shirt forks a process to exec a command, the environment (and subsequently the PATH) is passed all the way down. This is how exec finds which program it should become when you pass it a program that's not at an absolute path.

'Local' Environment Variables

The set command we implemented can change environment variables shell-wide, but what about when you want to change an environment variable just for the invocation of one command? This is very common (as you'll see from the examples below) and we can actually do it without changing shirt at all.

# Here's how bash does it
$ RAILS_ENV=staging rails server
$ VERBOSE=1 make install

# Here's how shirt does it (also works in bash)
-> env RAILS_ENV=staging rails server
-> env VERBOSE=1 make install

The env(1) command is available on most systems and fits this purpose exactly. bash has that nice shorthand, but I don't feel the need to add that extra parsing code right now when the env(1) command does it so nicely.

More Environment Variables

Now that we have a facility for changing environment variables we can add some more customization to our shell. The prompt, for instance.

...
ENV['PROMPT'] = '-> '

loop do
  $stdout.print ENV['PROMPT']
...

Now the prompt can be customized simply by setting the PROMPT environment variable. It's still primitive in that you can't dynamically use commands in the prompt line, but that will be added later when we have a more general way of evaluating command strings.

A Refactor

Before we end this article I want to do some refactoring: I'd like to keep everything in one file for now so the source is easy to read and easy to follow, but the problem is that we now have a bunch of boilerplate code before the actual meat of the program.

We first define all of our builtins, as well as setting up a default environment variable, before we actually show the logic of the program. Let's reverse those two. Here's shirt after the refactor.

The notable changes are that the program logic moved to the top of the file into a method called main. This method gets invoked on the last line of the file.

#!/usr/bin/env ruby
require 'shellwords'

def main
  loop do
    $stdout.print ENV['PROMPT']
    line = $stdin.gets.strip
    command, *arguments = Shellwords.shellsplit(line)

    if BUILTINS[command]
      BUILTINS[command].call(*arguments)
    else
      pid = fork {
        exec line
      }

      Process.wait pid
    end
  end
end

BUILTINS = {
  'cd' => lambda { |dir = ENV["HOME"]| Dir.chdir(dir) },
  'exit' => lambda { |code = 0| exit(code.to_i) },
  'exec' => lambda { |*command| exec *command },
  'set' => lambda { |args|
    key, value = args.split('=')
    ENV[key] = value
  }
}

ENV['PROMPT'] = '-> '

main

As always, the source is on Github.

In the next article we'll finish up with some search path stuff, then I'll show some pretty peculiar behaviour which will lead to implementing pipes in Ruby! Don't miss it!

A Unix Shell in Ruby - Part 2: Builtins

2012-02-21T00:00:00-05:00

Wecome to the second article in this series. Last time we implemented a verrrry basic shell that could just run most basic commands. This time around we're going to look at how it's broken and how we can fix it.

First up, we'll need a few builtins.

A Note on Launching

Last time I explained the semantics of exec: a call to exec will replace the current process with another. This time, when we launch shirt we're going to do so with the exec(1) utility. So we'll launch our shell like this:

$ exec ./shirt

When we do this from bash, our bash process becomes a shirt process. This way we can ensure that we're not running as a descendant of a bash shell and we can minimize the dependence on bash and its features.

Onward.

Even cd doesn't work!

First, we'll try the cd command:

$ exec ./shirt
$ pwd
/Users/jessestorimer/projects/shirt
$ cd /usr/local
$ pwd
/Users/jessestorimer/projects/shirt

Even though cd appears to have worked it didn't actually change our current directory. If we look back at the code from last lesson it becomes obvious why this is the case.

Backstory

When we tell shirt to run a command it does it in two steps: fork + exec. The call to fork creates a new child process. The call to exec transforms that new process into a cd process.

Although the child process knows the process ID of its parent, it can have no direct effect on the parents' state. So then, we shouldn't be surprised that the parent process (our shirt process) is unaffected when one of its children changes directories.

Just to make this crystal clear: the cd is taking effect on the child process, but that change is not reflected in the parent process. So once the child has finished and exited it's as though the cd never happened.

We'll get around this by making cd a 'builtin' command.

A Note About Builtins

Every shell has some built in commands. These are commands which are inherently different from typical utilies in that they change the state of the shell and run in the shell process itself.

For instance, a utility like cat simply prints text to the terminal. It does not change the state of the shell itself. The same goes for common commands like grep, tail, and curl. But there are certain commands that need to change the state of the shell, and cd is one of them.

Implementing cd

OK, so we can't implement cd in the same way that our other commands are handled. We'll have to add a special handling to our shell for builtin commands.

Before that, let's do a slight refactor of our shell from last time to have a more consistent run loop that we can build upon:

#!/usr/bin/env ruby

loop do
  $stdout.print '-> '
  line = $stdin.gets.strip

  pid = fork {
    exec line
  }

  Process.wait pid
end

In this implementation we've reduced a bit of duplication when printing the prompt. We start by entering an endless loop, printing the prompt, getting a line of input using gets, removing leading and trailing whitespace with strip, and then processing that input as before.

Here's our first pass at implementing cd:

#!/usr/bin/env ruby
require 'shellwords'

BUILTINS = {'cd' => lambda { |dir| Dir.chdir(dir) }}

loop do
  $stdout.print '-> '
  line = $stdin.gets.strip
  command, *arguments = Shellwords.shellsplit(line)

  if BUILTINS[command]
    BUILTINS[command].call(*arguments)
  else
    pid = fork {
      exec line
    }

    Process.wait pid
  end
end

Perfect, now cd works as expected. Here's the explanation.

The BUILTINS constant stores a hash denoting the builtin commands we support. The key specifies the builtin command and the value is a lambda which encapsulates the behaviour. For cd we use Ruby to change the directory without first creating a child process. In this way, the current directory of the shell process itself will be changed.

After getting the input we split the input line into a command and its arguments. This is done with Shellwords, which is part of the Ruby standard library. Why is it used, you ask? Shellwords will split the string the same way that bash does. The documentation for Shellwords.shellsplit has a great example of this.

The last addition is a conditional. If the command is one of the builtin functions then we should use that instead of the fork+exec method.

Just Try to Leave, I Dare You!

Let's implement another builtin, exit. exit doesn't exist as an executable anywhere on your system, it's always implemented as a builtin. Here's the relevant code for our implementation of exit:

BUILTINS = {
  'cd' => lambda { |dir| Dir.chdir(dir) },
  'exit' => lambda { |code = 0| exit(code.to_i) }
}

Now we can do a session like:

$ ls
shirt
$ pwd
/Users/jessestorimer/projects/shirt
$ cd /usr/local
$ pwd
/usr/local
$ exit

Most of the time you'll probably exit without passing an exit code, but it can be useful sometimes.

Change me!

We'll implement one more builtin, exec. This will behave the same way that bash' exec does, it will transform the shell process into another process. Useful for scripting or when you want the shell to become another program leaving no trace. Here's the implemenation:

BUILTINS = {
  'cd' => lambda { |dir| Dir.chdir(dir) },
  'exit' => lambda { |code = 0| exit(code.to_i) },
  'exec' => lambda { |*command| exec *command }
}

You get the idea.

Before You Go

We're implementing builtins by splitting a string and matching it to an array of predefined commands. In reality a shell like bash implements a real language and would do this work with a real parser. We're taking shortcuts for the purposes of moving quickly and learning.

OK, now you can leave

That's it for our first look at builtins. This shell is getting more and more functional :) You can get the latest version of shirt on Github.

If you want to see what other builtin commands are part of a shell like bash use man builtin.

Next time we'll look at handling environment variables, specifically the PATH. I won't be able to post again next week, but look for the next article two weeks from now.

For a more thorough introduction to Unix programming basics check out Working With Unix Processes.

A Unix Shell in Ruby

2012-02-16T00:00:00-05:00

This is the first article in a series where I'll implement a Unix shell in pure Ruby code.

Why a shell?

A shell is the quintessential example of a Unix program. It hits all of the interesting points that a Unix system is capable of.

It has to read input from STDIN, spawn processes requested from users, send signals to processes based on input, connect processes together with pipes, and provide a scripting environment for programmers to get work done.

What's a shell?

It's possible that you're using a shell all the time without knowing where the shell begins and where it ends. Or the fact that there are many different shells available to you. The default shell on most Unix systems is called bash. bash was written in the eighties as a replacement for its predecessor: the Bourne shell. Hence, bash stands for 'Bourne Again SHell'.

bash is a program like any other. It gets treated the same way as would grep, cat, or tail. So what makes a shell different? When you launch a shell it enters an endless loop of reading input and processing it. It reads the commands you type and launches programs based on that.

You can see this firsthand if you open a command line and try typing bash. Chances are you are already in an instance of a bash shell, but since it's a program like any other you can launch it again. Now you're in a shell in a shell. Inception!? Go ahead and exit the 'subshell' to get back to your original bash session.

Prior Art

There are many shells available for Unix systems. Besides bash, the most popular alternative is zsh. There are lots of good reasons to give zsh a try, enumerated here.

The Plan

We'll start out building a shell that resembles bash for familiarity. Along the way we might choose to make design decisions that deviate from what bash does, but that's the fun of writing your own! You can do it your way.

shirt

Our shell will be called shirt. This stands for 'SHell in Ruby? Totally!'. Here's our first implementation:

#!/usr/bin/env ruby

$stdin.each_line do |line|
  pid = fork {
    exec line
  }

  Process.wait pid
end

Stick that in a file, make it executable with chmod +x shirt and then run it using ./shirt. You'll be presented with a blank line, but try typing in a shell command like ls or ruby -v. This shell is still missing key features but the absolute basics are working.

So what's going on here?

Taking the Hard Way

The truth is that we could have implemented this in a much simpler way. We could replace the whole fork, exec, and wait bit with one call to system(). We're goint to avoid this method for two reasons:

We're here to learn, so we're going to use the lowest level building blocks available to us to get an idea of what's really possible here. This implementation is closer to bash or other system shells.
This is more flexible. We won't get to it much in this article but doing it this way will actually provide us with more flexibility later when we want to implement things like pipes.

I'll explain things one piece at a time.

Getting Input

$stdin.each_line do |line|

This line will put the program in an endless loop. It will read from standard input until it gets a newline and then pass that line of text into the block.

If this is your first introduction to $stdin it's fairly simple. Every process has a $stdin file hooked up by default. This is where processes look to read input. This input might be coming from a keyboard or from a pipe, either can be read just the same. In this case it causes the program to block until it gets an entire line of input from the user.

Executing a Command

  pid = fork {
    exec line
  }

There are two important concepts here: fork and exec.

Calling fork creates an new process that's an exact copy of the current process. It gets a copy of everything in memory and all open file descriptors. Certain housekeeping data is copied over like process group id, session group id, etc., and the new process gets its own process id, but for all intents and purposes it's an exact copy. So fork gets us a new process where we will execute the command that came in from $stdin.

Notice that we pass a block to fork and store its return value. The block is passed to the newly created process. It runs that block and then exits. In the parent process the process id (pid) of the newly created process is returned from fork. More on that pid in the next section.

Calling exec transforms the current process into another one specified by the file passed as input. An example should clear this up, calling exec('ls') will transform the current Ruby process into an ls process. The transformation happens immediately so any code that you have after the exec will never be reached. exec is a no-looking-back kind of thing.

Try out exec('ls') in irb if you never have before. Notice that you get the output of ls and then you're back to the terminal. The irb process is gone, replaced entirely by ls. So when ls exited you're back to the shell.

The pattern we're using here, fork and exec is a common way to spawn commands. It's the method used by Ruby constructs like Kernel#system and Kernel#`. If we were to skip the fork and just call exec then the shirt process itself would be transformed and it would no longer be able to respond to input. Calling fork creates a new child process to be transformed into the necessary command.

Waiting Patiently

  Process.wait pid

The last piece of the puzzle is Process.wait. Calling fork will return immediately. It creates a child process then returns context to the parent process. If we didn't wait for the child process to finish its work then we'd end up spawning multiple processes in parallel. In that case their input and output would be interleaved all over each other and it would be unclear which output is coming from which command. This keeps things usable and gives us a synchronous shell.

Process.wait is a blocking call that won't return until the process represented by the passed-in pid has exited. Using this ensures that our shell will only run one command at a time.

Some Chrome

Before we end this first article let's give our shell a bit of chrome so we can recognize the prompt.

#!/usr/bin/env ruby

$stdout.print '-> '

$stdin.each_line do |line|
  pid = fork {
    exec line
  }

  Process.wait pid
  $stdout.print '-> '
end

Now when you run the shell you get a simple prompt before each command so you can tell which lines are output and which lines are input.

Just the Start

There's still lots of ways to break the shell. Did you try using cd? How about going back in history?

There's still lots of missing features, and since the shell is actually running inside another instance of a shell it's getting some features for free.

Next time I'll look at launching shirt outside the context of another shell and implemeting some built-in commands.

Grab the source on Github.

For a more thorough, Ruby-centric introduction to Unix programming basics check out Working With Unix Processes.

11 Resources for Learning Unix Programming

2012-02-10T00:00:00-05:00

Recently I got an email from a reader of WWUP asking how I learned about all this stuff. Here's what I told him.

I personally get a lot from reading source code and from piecing together different resources. I have very little experience with C programming, and learned everything I know about Unix programming w/o having to write C code. Besides the resources below I learn a lot from experimenting with little bits of code and writing my own software.

Here's a brain dump of some stuff that helped me:

http://unicorn.bogomips.org/. Lots of good info in the included files like DESIGN, PHILOSOPHY, etc.
Any other piece of software by Eric Wong (author of Unicorn) http://rubygems.org/profiles/normalperson
https://www.destroyallsoftware.com/
http://tomayko.com/writings/unicorn-is-unix
https://github.com/TwP/servolux
I'm also a frequent reader of manpages
https://github.com/defunkt/resque
http://pic.plover.com/UnixReader/
Anything by Russ Cox (http://swtch.com/~rsc/)
http://cat-v.org/
The Linux Programming Interface http://nostarch.com/tlpi
The Art of Unix Programming http://www.faqs.org/docs/artu/
USP Ruby mailing list http://bogomips.org/usp.ruby/README

The truth is that there are a lot of really great free resources out there, but they're not cohesive and most are written for C programmers.

I tend to steer clear of the thick reference books and go instead for books that give me a look into how smart people think about programming. There's lots of good thought in the history of Unix and its programmers.

Want a cohesive guide to Unix programming written specifically for non-C programmers? Have a look at Working With Unix Processes, a brief book for precisely that.

I'm sure there are other resources available for learning this stuff. What am I missing?

Pairing Is Not Codereview

2012-01-30T00:00:00-05:00

I previously posted about the importance of formal code review and its benefits. Though I shared why the code review process is important I didn't make it clear how that fits into a software development process alongside TDD, pairing, etc.

I replied thusly to @brynary's tweet:

I want to expand on that.

Pairing

Others have written about pair programming and I don't really have anything to add. I think pairing is a fine practice in certain situations.

In my experience it is most useful at the beginning of the software development process. When you're looking to the horizon for a solution, unsure of which direction to take. Once a course has been chosen then a single developer can push the code in that direction.

Code Review

Code review doesn't even relate orthogonally to pairing. Code review is the last step in the software development process.

Code review is one developer saying to another:

"Hey, I've found a problem, wrote some tests, implemented a solution, tested the solution locally and in the staging environment and now I think it's ready to go. Do you see any reason why it shouldn't be deployed?"

If there is anything that indicates the change shouldn't go online then this is the time to bring it up. This is not the time for sweeping design changes.

What Code Review is For

The biggest benefit of code review is fresh eyes on the code. Ideally, eyes that weren't involved in the design and implementation of the code. Fresh eyes are the best bet to find edge cases that the original author may have missed.

You should have at least one reviewer on each review who is familiar with the (official or unofficial) style guide of the project. This is the stage where your style guide is enforced. Did the author forget to use an idiom common to the project? Was some code re-implemented that could have been included from a module elsewhere?

Making Suggestions

When reviewing code it can be tempting to point out all the ways that the author is 'wrong'. In other words, all the things that you would have done differently. Remove ego from the equation as much as possible, your way isn't necessarily 'right'. After that err on the side of being verbose, keeping in mind that not every comment needs to be addressed.

Not all code review comments need to be answered with modifications to the code. Not every suggestion needs to be implemented. Anything that prevents the code from working as advertised needs to be addressed. Everything else is at the discretion of the author.

The point of this kind of comment is basically to say 'gotcha!'. If the author decided to do a database query from a view helper then you should remind them that we put database queries in the controller and reserve the view helpers for extracted view logic. This doesn't mean that the entire changeset should be delayed to ensure that this convention be followed. The most important thing is that code gets shipped.

The author can always follow up later to fix the regression. The subtle point of this kind of comment is to make the author think twice the next time they break convention. It's to remind them that they have to be accountable to their peers. Their peers are watching, and caring!

Waiting On Others

It seems at first glance like code review will slow down a development cycle. We have one developer responsible for implementing a feature from start to finish. Before deploying the code he hands off responsibility to a reviewer to '+1' the whole thing. What if the reviewer is busy, takes days to respond? Talk about buzzkill.

If code review is going to be part of your development process then developers need to understand its gravity. Developers should be aware that when a code review is assigned to them they are the last thing holding up a changeset from being deployed. There's no excuse for letting this go on longer than it has to.

Unless something prevents it: code reviews should be performed the same day they're received. If you can't do it then forward it on to someone else. The point isn't to enforce rigid structure into your process where none is needed. The point is to respect your co-workers. The way to do that is to help them get their stuff deployed without delay.

Conclusion

As I mentioned, code review has had a huge impact on my development as a programmer. Having my code critiqued by other developers always sheds new light and I've learned a ton from the suggestions of others.

My biggest takeaway from code review was a different perspective on code. Code review helped me understand that I'm not just implementing features for end users, I'm implementing APIs for my peers. They may very well be the ones who have to understand and maintain my code in the future.

No matter how important it seems to ship something quick to make an end user happy the guy performing maintenance must be considered. Code review ensures that that guy gets a voice before your code ever sees the light of day.

I'm Jesse Storimer. Follow me on Twitter.

The Better To RTFM

2012-01-26T00:00:00-05:00

Unix manpages are great if you already know what you're looking for. But what if you don't?

If you've read prior art on manpages then you know that the manpages are separated into sections: section 1 is for general commands, section 2 is for system calls, etc. If you know exactly what command or system call you're looking for then it's easy to look it up: man grep finds the grep documentation. Here's some tips to dig a little deeper.

1. Each manpage section has its own intro page.

The intro pages provide an overview about each section. Try man intro for an example. For something of more interest try man 2 intro. Believe it or not that's probably one of the most concise and to the point introductions to systems programming that I've seen. Unless you knew you were looking for that it's kind of hard to find.

Contrary to man man, which provides information about how to use the man command, intro pages provide general info about the documentation itself. Very useful.

2. You can search the manpages.

There are two commands for this purpose: whatis(1) and apropos(1). The whatis command searches for exact matches in the manpage database. The apropos command searches for partial matches. Want to know which variants of grep you have documentation for?

$ whatis grep

This will return a few manpages with the whole word 'grep' in the name.

$ apropos grep

This will return more manpages than before, this time including some that have grep as a substring. This includes stuff like zgrep(1), pgrep(1), etc.

The good news is that you don't need to remember any extra commands to search the manpages: just use man -k search-string. man -k is equivalent to apropos(1).

$ man -k grep

3. Some interesting manpages you probably didn't know about.

intro(2). signal(7). environ(7). style(9).

Go forth and RTFM.

I'm Jesse Storimer. Follow me on Twitter.

For a more comprehensive introduction to systems programming try my book: Working With Unix Processes.

Exit, Exit! Abort, Raise…Get Me Outta Here!

2012-01-19T00:00:00-05:00

I guest-posted over on rubysource.com about exit codes, why they're important, and how to use them from your Ruby programs. It's worth a read.

I'm Jesse Storimer. Follow me on Twitter.

Get my eBook, Working With Unix Processes, and learn even more about Unix programming, all from the comfort of Ruby.

The Hungry Programmer

2012-01-09T00:00:00-05:00

Programming is a lot like eating.

Tonight I was hungry. I wasn't at home. I couldn't go to the kitchen for a snack. I looked around and saw only McDonald's across the street.

Then I was struck with the same dilemma that I face whenever I leave the comfort of my own home for any decent spread of time: Do I eat crappy food now and satisfy that hunger? Or do I stay hungry for a little longer and eat a healthy meal back at home?

As I pondered this dilemma I couldn't help but notice how much it relates to code quality. But more about food first.

I can only speak for myself, but I do not consider a McDonald's restaurant capable of providing me with a healthy meal. As an aspiring raw fooder, or even as a person who cares what I eat, McDonald's food is far from what I would consider a healthy meal. But, like a lot of you I'm sure, I grew up eating McDonald's without questioning its health benefits. I was hungry and McDonald's was food.

Now that I understand the short and long-term effects of McDonald's on my body I avoid it all costs. The same goes for pre-processed foods in general. This has been a journey for me. Deciding to change your eating habits is more than a one-day process and healthy eating is a continuum.

This'll all relate to programming very soon :)

Healthy Eating is a Continuum?

Eating at McDonald's is not a healthy meal. However eating a Big Mac with a side salad is marginally healthier than eating a Big Mac with fries. Eating a tuna wrap from Starbucks is marginally healthier than any meal containing a Big Mac. Eating a bowl of soup from your local boutique coffee shop is marginally healthier than a tuna wrap from Starbucks. Eating a homemade meal with fresh ingredients is healthier than any of these meals.

There are two takeaways from our look at food:

Everything is relative. You can almost always do better and almost always do worse. Eating a Big Mac does not equal death, but refraining from food for an entire day in anticipation of a homemade meal is obviously not healthy either.
Every meal is anew. It's very easy to get into traps like "Well, I had a Big Mac yesterday so I might as well have another one today". That kind of thinking is not logical or helpful. Every decision to eat is a new decision. Even if you had a Big Mac for breakfast today, having a home-made meal for lunch is a step up on that continuum.

Code as Food

Much like food, code quality is a continuum.

At a very basic level we can say that a library that ships with no tests and no documentation is of lower quality than the same library with tests and documentation.

But between the obvious is a lot of murky water. Everyone has their own ideas about what is 'healthy' code. Some people are very aware of their programming habits and work on improving them. Some people are doing the same crap they've been doing for years without a thought to whether or not it's 'healthy'.

Making Healthy Choices

Treat code like food.

Just like you have to eat, you have to ship code. But how you do that has both short- and long-term effects.

If you take the McDonald's approach and ship shit then you satisfy that need in the short-term. But you'll feel the effects in the long-term. Your code will be harder to maintain and need more attention later. It won't have a long and healthy life.

If you take the home-made meal approach then you leave that need unsatisfied for a little while. The short-term effect is that you stay hungry, but it's for a good reason: the long-term effects will be positive. Your code will be easier to maintain, bugs will be easier to fix, and it will be easier to introduce change. Your code will have a long and healthy life.

Decisions

Writing healthy code requires the same thing that healthy eating does: recognition and commitment. To go from eating poorly to eating well requires a recognition that there's something that needs improvement. To act on that recognition requires commitment to your decision.

To improve your programming habits requires the same: recognition that something needs to be improved and the commitment to do so.

As always, your choices must be pragmatic. You have to eat something, and your code has to ship sometime. In lieu of finding a healthy meal McDonald's will have to do. I don't know the constraints of your software project, and sometimes you'll have to ship shit. Just remember that healthy programming is a continuum. Get a side salad with that Big Mac instead of fries.

There is no 'healthiest' meal. You can spend all day preparing the healthiest meal you can imagine but it does no good unless it's in your stomach. Code quality is important, but it's nothing unless you ship it.

The surest path to healthier code is incremental improvements. Program a little bit healthier every day rather than discarding the old habits in one go and expecting them to be replaced by new, healthy ones.

I'm Jesse Storimer. Follow me on Twitter.

Get my eBook, Working With Unix Processes, and become a Unix guru without any C programming.

When to use STDERR instead of STDOUT

2011-12-29T00:00:00-05:00

stdout and stderr both quack the same, but are used for very different purposes.

Every process is initialized with three open file descriptors, stdin, stdout, and stderr. stdin is an abstraction for accepting input (from the keyboard or from pipes) and stdout is an abstraction for giving output (to a file, to a pipe, to a console).

That's a very simplified explanation but true nonetheless. Those three file descriptors are collectively called 'The Standard Streams'.

Where does `stderr` come from?

It's fairly straightforward to understand why stdin and stdout exist, however stderr seems like the odd one out. Why do we need another stream for errors?

This is a quote from Doug McIllroy, inventor of Unix pipes, explaining how stderr came to be. 'v6' is referring to a version of specific version of the original Unix operating system that was released in 1975.

All programs placed diagnostics on the standard output. This had always caused trouble when the output was redirected into a ﬁle, but became intolerable when the output was sent to an unsuspecting process. Nevertheless, unwilling to violate the simplicity of the standard-input-standard-output model, people tolerated this state of affairs through v6. Shortly thereafter Dennis Ritchie cut the Gordian knot by introducing the standard error ﬁle. That was not quite enough. With pipelines diagnostics could come from any of several programs running simultaneously. Diagnostics needed to identify themselves.Doug McIllroy, "A Research UNIX Reader: Annotated Excerpts from the Programmer’s Manual, 1971-1986"

Why do we need `stderr`?

So there was a time when stderr didn't exist. As McIllroy mentions, in that time people were unwilling to violate the simplicity of the standard-input-standard-output model to add something like stderr. Let's have a look at the grep command to see a real example of why stderr is needed.

$ grep hosts /private/etc/*
grep: /private/etc/AFP.conf: Permission denied
grep: /private/etc/aliases.db: Permission denied
/private/etc/amavisd.conf:# from internal hosts to a dedicated TCP port (such as 10026) for filtering
/private/etc/auto_master:/net          -hosts      -nobrowse,hidefromfinder,nosuid
grep: /private/etc/kcpassword: Permission denied
grep: /private/etc/krb5.keytab: Permission denied
grep: /private/etc/master.passwd: Permission denied

From this command I've got some diagnostic messages (permission errors) mixed with real matched output. When writing to a console both stdout and stderr will be printed, this is useful so that I can see the errors as they happen. But what if I was redirecting the output to a file to send to somebody else, or save for later?

$ grep hosts /private/etc/* > results.txt
grep: /private/etc/AFP.conf: Permission denied
grep: /private/etc/aliases.db: Permission denied
grep: /private/etc/kcpassword: Permission denied
grep: /private/etc/krb5.keytab: Permission denied
grep: /private/etc/master.passwd: Permission denied

$ cat results.txt
/private/etc/amavisd.conf:# from internal hosts to a dedicated TCP port (such as 10026) for filtering
/private/etc/auto_master:/net          -hosts      -nobrowse,hidefromfinder,nosuid

That's actually just what I wanted: I saw the errors printed to the console but all of the matched output was sent to the file. In the time of Unix v6 both the errors AND the output would have been sent to the file, which I would have needed to clean up manually. Woohoo for stderr.

Redirecting `stderr`

You can also achieve the opposite, send stderr to a file and print stdout on the console by redirecting a specific file descriptor number. For stderr that's 2.

$ grep hosts /private/etc/* 2> error.log
/private/etc/amavisd.conf:# from internal hosts to a dedicated TCP port (such as 10026) for filtering
/private/etc/auto_master:/net          -hosts      -nobrowse,hidefromfinder,nosuid

$ cat error.log
grep: /private/etc/AFP.conf: Permission denied
grep: /private/etc/aliases.db: Permission denied
grep: /private/etc/kcpassword: Permission denied
grep: /private/etc/krb5.keytab: Permission denied
grep: /private/etc/master.passwd: Permission denied

And one more example, redirecting both stderr and stdout to a file, to preserve exactly what you'd see on the console.

$ grep hosts /private/etc/* &> results.txt
$ cat results.txt
grep: /private/etc/AFP.conf: Permission denied
grep: /private/etc/aliases.db: Permission denied
/private/etc/amavisd.conf:# from internal hosts to a dedicated TCP port (such as 10026) for filtering
/private/etc/auto_master:/net          -hosts      -nobrowse,hidefromfinder,nosuid
grep: /private/etc/kcpassword: Permission denied
grep: /private/etc/krb5.keytab: Permission denied
grep: /private/etc/master.passwd: Permission denied

Identify Yourself!

Notice how the error messages from grep are prepended with 'grep:'?

grep: /private/etc/master.passwd: Permission denied

This is what McIllroy was talking about when he said "With pipelines diagnostics could come from any of several programs running simultaneously. Diagnostics needed to identify themselves." Consider a pipeline like this:

grep hosts /private/etc/* | awk '{print($1)}' | xargs cat
grep: /private/etc/AFP.conf: Permission denied
grep: /private/etc/aliases.db: Permission denied
grep: /private/etc/kcpassword: Permission denied
grep: /private/etc/krb5.keytab: Permission denied
grep: /private/etc/master.passwd: Permission denied
cat: /private/etc/amavisd.conf:#: No such file or directory
cat: /private/etc/auto_master:/net: No such file or directory

Were it not for the prefixed command name in front of the error we'd have no idea which command was printing the error. For all we know every error could have been coming from awk. Thankfully most commands support this convention today so we can figure out where we're going wrong.

`stderr` and Pipes

When you pipe the output of one command to another command only the stdout is passed over the pipe. Again, this is just what we want, but was not the case back in the Unix v6 days. Have a look at this grep example:

$ grep hosts /private/etc/* | grep 'Permission denied' > results.txt
grep: /private/etc/AFP.conf: Permission denied
grep: /private/etc/aliases.db: Permission denied
grep: /private/etc/kcpassword: Permission denied
grep: /private/etc/krb5.keytab: Permission denied
grep: /private/etc/master.passwd: Permission denied

$ cat results.txt

So the errors were printed to the console but they're not present in the output. Highlighted in this case by the fact that we searched for them with the second grep command but they don't exist in the results file.

If you need to pass stderr over a pipe along with stdout you can do that by redirecting the stderr stream (file descriptor #2) to stdout (file descriptor #1), like so:

$ grep hosts /private/etc/* 2>&1 | grep 'Permission denied' > results.txt

$ cat results.txt 
grep: /private/etc/AFP.conf: Permission denied
grep: /private/etc/aliases.db: Permission denied
grep: /private/etc/kcpassword: Permission denied
grep: /private/etc/krb5.keytab: Permission denied
grep: /private/etc/master.passwd: Permission denied

In this case there was nothing printed to the console, but the errors were matched and sent to the results file.

I probably don't want your help docs

It's pretty obvious that any error messages should be printed to stderr because I probably don't want them as input to other programs, and if I really do then I can use redirection to get it. But McIllroy specifically says that stderr is for diagnostic messages, why didn't he just say errors?

stderr is also the place where you print stuff like help docs when an invalid argument is passed to a command. In this case its help docs should be printed, but not be passed on to the next program.

$ cat Gemfile | grep -k gem | awk '{print($2)}' | xargs gem install
grep: invalid option -- k
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.

Thankfully the help docs for grep are printed to stderr so I don't try to install them as gems in this case.

But if I'm invoking a command specifically to read its help docs, then it should be printed to stdout. Take the following example:

$ git --help | grep pull

In this case I certainly want the help docs; I asked for them explicitly with the --help option so I could grep for the specific option I was looking for. When asking explicitly for help docs they should be printed to stdout.

Unix Reader

The Unix Reader is a fascinating read from someone who was there and contributing in the early days of Unix. This document is a good place to start if you enjoy reading about Unix history and origins, or just like to hear some hacker stories.

Leave a comment if you know of other stuff that's printed to stderr besides errors and usage documentation.

Like what you just read? Check out my eBook: Working With Unix Processes. The first crash course in Unix programming just for Rubyists.

The rails command and exec(2)

2011-12-20T00:00:00-05:00

TL;DR: The sole job of the rails command that ships with the rails gem is to exec ./script/rails in your Rails project. Combine that with Aaron's PSA and that can be considerable overhead. You should use ./script/rails instead of rails.

Ah, the rails command. Chances are that if you're writing Ruby code: you're using the rails command all the time.

The rails command is just a thin wrapper around exec(2). If you don't know what exec(2) is or why the rails command uses it then keep reading.

From the top

## File: rails/bin/rails
#!/usr/bin/env ruby

begin
  require "rails/cli"
rescue LoadError
  railties_path = File.expand_path('../../railties/lib', __FILE__)
  $:.unshift(railties_path)
  require "rails/cli"
end

This is the contents of bin/rails from the rails gem. This is the code that gets invoked when you use the rails binary.

You can see that all it really does is require rails/cli from railties. Let's look there next.

## File: railties/lib/rails/cli.rb
require 'rbconfig'
require 'rails/script_rails_loader'

# If we are inside a Rails application this method performs an exec and thus
# the rest of this script is not run.
Rails::ScriptRailsLoader.exec_script_rails!

railties_path = File.expand_path('../../lib', __FILE__)
$:.unshift(railties_path) if File.directory?(railties_path) && !$:.include?(railties_path)

require 'rails/ruby_version_check'
Signal.trap("INT") { puts; exit }

require 'rails/commands/application'

The first line of interest is the one after the requires. The comment mentions that this method may or may not call exec. If it does then the rest of the script won't run.

It's time to step back and look at what exec(2) does. First, exec(2) is a system call: a direct request to the kernel asking for some function that only the kernel can do.

exec(2) transforms the current process into another process.

puts 'Hi from Ruby'
exec 'ls'
puts 'Bye from Ruby'

If you execute that block of code you'll notice that the last line after the exec is never reached. The call to exec actually transformed the Ruby process into an ls process. You'll see the output from ls right after the output from the first line.

In this way exec(2) is a bit like jumping out of a plane. Your process fully commits to becoming another process and there's no way to go back and resume execution. The ls process has no idea that it used to be a Ruby process, and is not programmed to transform itself to another process.

This looks a bit like Kernel#system but works differently. If you shell out using something like system your Ruby process will resume execution after the shell command is finished.

puts 'Hi from Ruby'
system 'ls'
puts 'Bye from Ruby'

In this example the last line will be executed.

This is because Kernel#system uses a combination of fork(2) (create a new process) and exec(2) (transform that process) rather than transforming the current process.

The example above was a bit contrived. Keep reading to see how the rails command uses it.

Down another level.

## File: railties/lib/rails/script_rails_loader.rb
require 'pathname'

module Rails
  module ScriptRailsLoader
    RUBY = File.join(*RbConfig::CONFIG.values_at("bindir", "ruby_install_name")) + RbConfig::CONFIG["EXEEXT"]
    SCRIPT_RAILS = File.join('script', 'rails')

    def self.exec_script_rails!
      cwd = Dir.pwd
      return unless in_rails_application? || in_rails_application_subdirectory?
      exec RUBY, SCRIPT_RAILS, *ARGV if in_rails_application?
      Dir.chdir("..") do
        # Recurse in a chdir block: if the search fails we want to be sure
        # the application is generated in the original working directory.
        exec_script_rails! unless cwd == Dir.pwd
      end 
    rescue SystemCallError
      # could not chdir, no problem just return
    end 

    def self.in_rails_application?
      File.exists?(SCRIPT_RAILS)
    end 

    def self.in_rails_application_subdirectory?(path = Pathname.new(Dir.pwd))
      File.exists?(File.join(path, SCRIPT_RAILS)) || !path.root? && in_rails_application_subdirectory?(path.parent)    
    end
  end   
end

This class is a bit noisy. We'll take it one line at a time.

RUBY = File.join(*RbConfig::CONFIG.values_at("bindir", "ruby_install_name")) + RbConfig::CONFIG["EXEEXT"]
SCRIPT_RAILS = File.join('script', 'rails')

The class sets up some constants for itself. The first is notable as it gets the full path to the Ruby executable for this system. By running it through File.join and adding the extension suffix it is compatible with both Unix and Windows filesystems.

def self.exec_script_rails!
  cwd = Dir.pwd
  return unless in_rails_application? || in_rails_application_subdirectory?
  exec RUBY, SCRIPT_RAILS, *ARGV if in_rails_application?
  Dir.chdir("..") do
    # Recurse in a chdir block: if the search fails we want to be sure
    # the application is generated in the original working directory.
    exec_script_rails! unless cwd == Dir.pwd
  end   
rescue SystemCallError
  # could not chdir, no problem just returnend
end
  
def self.in_rails_application?
  File.exists?(SCRIPT_RAILS)
end 

def self.in_rails_application_subdirectory?(path = Pathname.new(Dir.pwd))
  File.exists?(File.join(path, SCRIPT_RAILS)) || !path.root? && in_rails_application_subdirectory?(path.parent)    
end

The main method here is exec_script_rails!. Notice that it returns unless already in a Rails app directory or subdirectory. In that case exec won't be called and the command will take a different path.

The methods in_rails_application_subdirectory? and in_rails_application? recursively search the filesystem looking for script/rails.

Assuming that we're somewhere in a Rails app directory the following block of code is executed.

  exec RUBY, SCRIPT_RAILS, *ARGV if in_rails_application?
  Dir.chdir("..") do
    # Recurse in a chdir block: if the search fails we want to be sure
    # the application is generated in the original working directory.
    exec_script_rails! unless cwd == Dir.pwd
  end
rescue SystemCallError
  # could not chdir, no problem just return

When already in the root of a Rails app then the exec is done immediately, passing any of the arguments that were passed in on the command line. Remember from our talk about exec that if this happens then none of the rest of the code afterwards will be executed.

The other code path covers the case where the rails command is being used from a Rails app subdirectory. It continues to Dir.chdir to the parent directory until it gets to the root of the Rails app, at which point it does the same call to exec.

The comment about chdir refers to the fact that even though the pwd is being changed with Dir.chdir it will be reset to the original working directory when the block returns.

So...

So, besides being used à la rails new to bootstrap a new app, the whole point of the rails command is just to find ./script/rails and 'become' it. You can do the job of the rails command yourself by just using ./script/rails directly.

rails console
# same as
./script/rails console

Using ./script/rails directly avoids only a little bit of code, but it may provide more of a boost than you think given Aaron Patterson's recent PSA regarding rubygems.

When you use the rails command you pay the penalty of loading every gemspec on your system. Even worse is using bundle exec rails. Since both bundle and rails are rubygems binstubs you pay that penalty twice!

By using ./script/rails you can avoid paying that penalty and execute just a little bit less code from Rails startup. And, as we all know: The best code is no code at all.

Understanding how your tools work is critical to improving your workflow, as well as properly debugging your production apps, find out what you need to know about Unix programming in Working With Unix Processes.

On Colorized Output

2011-12-16T00:00:00-05:00

Colorized output should be configurable.

This is a followup to my last post.

In that post I said:

When you are piping output to another program you should always send plain, unformatted text. Unix utilities expect to deal with plain text.

A redditor commented on that post to this effect: (paraphrasing is mine)

Modern pagers and other programs can handle colorized output just fine, so colorized output from your scripts should be configurable.

If you're redirecting to a file, or some other program that doesn't understand escaped color sequences, then colorized output can be turned off, otherwise it'll probably work just fine.

Good advice, and he's absolutely right.

I started from the same script used in the last post and removed the check for $stdout.tty? so that colorized output is always on. Now to experiment.

$ hilong Gemfile | less
$ hilong Gemfile | grep gem
$ hilong Gemfile | more
$ hilong Gemfile > out.put

Your results may vary depending on your system and your configuration, but for me the first two experiments worked just fine and the second two didn't.

So grep and less handled the color codes fine, but more didn't. Redirecting to a file obviously just preserved the raw text, which won't be interpreted by text editors and the like when it's opened later.

So there needs to be some way to turn colorized output on...or off...

Sensible Default?

When in doubt, follow suit. Tools like grep and less don't colorize output by default. But ack, for instance, does colorize by default. So what should hilong do?

Given that the 'highlighting' is basically the point of this tool I think that colorized output should be enabled by default. For most modern tools colorized output will not cause a problem, disabling colorized output will be a special case.

But there needs to be an option for that.

Parsing command-line options

Ruby's standard library has a class just for parsing command-line options. It's called optparse. Using the script from the last blog post, here's how to use optparse to accept a --no-color option.

maximum_line_length = 80
colorized_output = true

require 'optparse'
OptionParser.new do |options|
  options.on("--no-color", "Disable colorized output") do |color|
    colorized_output = color
  end 
end.parse!

# Keep reading lines of input as long as they're coming.
while input = ARGF.gets
  input.each_line do |line|
    # Construct a string that begins with the length of this line
    # and ends with the content. The trailing newline is #chop'ped 
    # off of the content so we can control where the newline occurs.
    # The string are joined with a tab character so that indentation
    # is preserved.
    output_line = [line.size, line.chop].join("\t")

    # If the line is long and our $stdout is not being piped then we'll
    # colorize this line.
    if colorized_output && line.size > maximum_line_length
      # Turn the output to red starting at the first character.
      output_line.insert(0, red)

Something of interest is that colorized_output is assigned to true at the beginning, and then is assigned the value of the parsed option if that was passed in. Nowhere is it specified that --no-color should set colorized_output to false, it's simply assigned the value passed by optparse.

optparse encourages best practices by recognizing certain conventions. eg. --[no-]thing for boolean options, --arg value for required options.

So optparse was able to figure that --no-color is a boolean and that it should be negative (false) considering it's prefixed with a --no. Rad.

Notice that the check for $stdout.tty? has been removed. It no longer matters. Output is colorized unless disabled with the command-line option.

Now hilong can be used with that option to disable colorized output.

$ hilong --no-color Gemfile
$ cat Gemfile Gemfile.lock | hilong --no-color | more

Perfect. And thanks to optparse there's also some nice documentation showing which options are supported.

$ hilong -h
Usage: hilong [options]
        --no-color                   Disable colorized output

Not bad, but it's easy to make that look better.

require 'optparse'
OptionParser.new do |options|
  # This banner is the first line of the help documentation.
  options.banner = "Usage: hilong [options] [files]\n" \
    "Show character count for each line of input and highlight long lines."

  # Separator just adds a new line with the specified text.
  options.separator ""
  options.separator "Specific options:"

  options.on("--no-color", "Disable colorized output") do |color|
    colorized_output = color
  end

  # on_tail says that this option should appear at the bottom of
  # the options list.
  options.on_tail("-h", "--help", "You're looking at it!") do
    $stderr.puts options
    exit 1
  end
end.parse!

Now the help docs look like this:

$ hilong -h
Usage: hilong [options] [files]
Show character count for each line of input and highlight long lines.

Specific options:
        --no-color                   Disable colorized output
    -h, --help                       You're looking at it!

Much better.

Moar options

I want to show one more case with optparse. Currently the script assumes that long lines are 80 chars or more. It's generally accepted that 80 chars is a long line but some some devs push that number to 120, or something else entirely. The maximum line length should be configurable.

  end

  options.on("-m", "--max NUM", Integer, "Maximum line length") do |max|
    maximum_line_length = max
  end
end.parse!

Again optparse is making a few assumptions about the options, but it does a pretty good job of it. It's set to accept an Integer along with the -m or --max option. The result is assigned to the previously existing local variable and is handled by the rest of the script.

Now it can be used like so:

$ cat Gemfile* | hilong --max 20
$ hilong --m 140 Gemfile
$ hilong --max=180 Gemfile

Try to pass something besides an Integer with the max option and optparse rightfully complains.

Full source of the hilong script is at https://gist.github.com/1465437.

The ruby-core documentation for OptionParser contains a useful example if you want to know more.

Writing Ruby Scripts That Respect Pipelines

2011-12-12T00:00:00-05:00

Pipes are the most powerful concept on the command line.

With pipes you can string together small, simple commands into bigger, more useful pipelines. This is the secret sauce that goes along with the Unix philosophy of "Do one thing, and do it well". Take this as an example:

# Look for any lines mentioning 'user' in the current git diff
# and display them one page at a time using less(1).
$ git diff | grep user | less

A simple utility

Here's a Ruby script I'm calling hilong.

It's a small, simple utility that takes a file as an argument and prints each line prepended with its length. Lines longer than the maximum length (80 chars by default) are highlighted in red. Here's the simplest version of our script:

#!/usr/bin/env ruby

# A file is passed in as an argument
input = File.open(ARGV[0])

# escaped bash color codes
red = "\e[31m"
reset_color = "\e[0m"

maximum_line_length = 80

# For each line of the input...
input.each_line do |line|
  # Construct a string that begins with the length of this line
  # and ends with the content. The trailing newline is #chop'ped 
  # off of the content so we can control where the newline occurs.
  # The strings are joined with a tab character so that indentation
  # is preserved.
  output_line = [line.length, line.chop].join("\t")

  if line.length > maximum_line_length
    # Turn the output to red starting at the first character.
    output_line.insert(0, red)
    # Reset the text color back to what it was at the end of the
    # line.
    output_line.insert(-1, reset_color)
  end

  $stdout.puts output_line
end

And here's how it works:

 $ hilong Gemfile
17    source :rubygems
1
13    gem 'jekyll'
82    gem 'liquid', '2.2.2'         # The comment on this line is long-winded, not sure why... 
16    gem 'RedCloth'

This works pretty well. It takes a file as input and puts the modified version onto its $stdout. Let's see how it fares when we combine it with other utilities.

Introducing Pipes

Let's start by trying to pipe the output of this utility to another utility:

# Only show 'gem' lines.
$ hilong Gemfile | grep gem

This works nicely. Since we put our output on $stdout grep(1) can read it and do the proper filtering. It even preserves our color codes!

Let's try another one:

# View the output one page at a time.
$ hilong Gemfile | more

Eeee. Now we get some ugly escape codes in our output. It seems that more(1) doesn't know what to do with the bash escaped colors that we included so it just includes it as part of the output. The same things happens if you redirect the output to a file.

When you are piping output to another program you should always send plain, unformatted text. Unix utilities expect to deal with plain text.

Is a tty?

So we can't include our color codes if our output is being piped to another program, but we want to include the color codes if our output is being displayed in a terminal. How do we tell?

Ruby's IO#isatty method (aliased as IO#tty?) will tell you whether or not the IO in question is attached to a terminal. Calling it on $stdout, for instance, when it's being piped will return false.

We'll rewrite our script to make use of this, I've highlighted the relevant part below:

  # If the line is long and our $stdout is not being piped then we'll
  # colorize this line.
  if $stdout.tty? && line.size > maximum_line_length
    # Turn the output to red starting at the first character.
    output_line.insert(0, red)
    # Reset the text color back to what it was at the end of the
    # line.
    output_line.insert(-1, reset_color)
  end 

  $stdout.puts output_line

Now if we try piping our output to more(1) again, or to a file, we get nice plain text.

Incoming!

Most Unix utilties also respond to pipes coming from the other direction, as input. Let's see how our utility responds when we pipe in some input:

$ cat Gemfile Gemfile.lock | hilong 
/Users/jessestorimer/projects/hilong/hilong:4:in `initialize': can't convert nil into String (TypeError)
from /Users/jessestorimer/projects/hilong/hilong:4:in `open'
    from /Users/jessestorimer/projects/hilong/hilong:4:in `<main>'

Right now our utility is only written to handle input given as a filename passed in via ARGV. How can we make it accept raw data from a pipe?

Ruby has a wonderful facility for this called ARGF. ARGF provides a consistent interface for raw data coming in via a pipe, and filenames passed via ARGV. Let's rewrite our script to take advantage of it:

# Read input from files or pipes.
-input = File.open(ARGV[0])
+input = ARGF.read

Wonderful! If something is passed in on ARGV then ARGF will assume that it's filenames and call IO#read on them sequentially. If ARGV is empty then it reads from $stdin to get data passed in via pipe.

Unix utilities will ignore standard input if filenames are given.

Now let's look at all the ways we can use our new utility.

$ hilong Gemfile
$ hilong Gemfile | more
$ hilong Gemfile > output.txt
$ hilong Gemfile Gemfile.lock
$ cat Gemfile* | hilong
$ cat Gemfile | hilong - Gemfile.lock
$ hilong < Gemfile

One more case...

With only a few small changes we were able to get our utility to respect pipelines like any other Unix utility would. But there's one more case I want to demonstrate. What if we're getting input from a pipe coming from a command such as tail -f where the input never stops coming?

$ tail -f log/test.log | hilong

If you give this a try and append to the log file you'll notice that our utility seems to be supressing the output. We're not seeing anything being printed.

This is due to the fact that we're using ARGF#read. #read will block until it receives EOF, but the tail utility will never send EOF because it always has more data. So the first time our utility is invoked with some data it simply blocks and never returns. So we need to change the way we're reading from ARGF.

We'll read from ARGF one line at a time using #each_line. #each_line will, duh, read each line in succession. So anytime a newline is encountered the String is passed into the block.

Here are the required changes:

# Keep reading lines of input as long as they're coming.
ARGF.each_line do |line|
  # Construct a string that begins with the length of this line
  # and ends with the content. The trailing newline is #chop'ped 
  # off of the content so we can control where the newline occurs.
  # The string are joined with a tab character so that indentation
  # is preserved.
  output_line = [line.size, line.chop].join("\t")

And that'll do it! Now our utility can handle the slew of input methods I showed above, plus handle continuous data from a pipe.

The full, finished source for the hilong utility is at https://gist.github.com/1465437/c75c872957329fd4034bedaf11c4de70635992e4.

If you can think of a better way to accomplish any of this or there's another use case that I missed let me know in the comments.

Read the followup post: On Colorized Output where the colorized output becomes configurable.

You Should be Doing Formal Code Review

2010-09-30T00:00:00-04:00

The two practices that have contributed most greatly to my development as a programmer are TDD and formal code review.

Much like TDD, you should be doing formal code review. This isn’t for QA purposes, or because developers are untrustworthy in general. Much like TDD, the benefits aren’t just the outputs of the behaviour, the real benefit is that formal code review changes the way that you think about code.

From a managerial standpoint the most obvious benefit is that two heads are better than one. Much like pair programming, it’s best to have more than one developer look at every line of code that makes it into your product.

From a developer’s standpoint formal code review often looks like a nuisance. If your current workflow is to write tests, write code, and then ship it, formal code review introduces a brand new blocker into that simple chain of events. Now you need to wait for other developers to comment on your code, you need to resolve any comments they make, and make sure that everyone else is as happy with your code as you are.

Let me tell you, it’s well worth it.

Here are some reasons why:

It helps you disidentify with your code.

By disidentify I mean that it helps you think of your code as outside yourself. This may seem obvious but I know a lot of programmers who don’t make the distinction between code and self. The real point of this is that in order to produce the best code possible you need to be able to admit to yourself that what you already have may not be perfect.

Let’s face it, developers are often overly confident in their work, and telling them that something is done wrong can be taken as a personal attack. If you get used to letting other people look at, and critque, your code then disidentification becomes a necessity. This also goes vice versa, you need to be able to talk about the code of your peers without worrying about them taking your critiques as a personal attack. The goal here is to ensure that the best code possible makes it into your final release.

It makes you avoid shortcuts.

This one may not be true for everyone, but it’s true for me. Before ever doing formal code review I would often choose suboptimal solutions in order to get changes online fast and get on to the next thing. There was a good chance that other developers wouldn’t read my code, and of course the end users of the software wouldn’t know the difference.

But when you know, for a fact, that your peers will be reviewing every line of code that you write you will ensure your code is as good as it can be. It’s pretty motivating to think that your peers will be looking at, and commenting on, all of the little shortcuts that you took in your latest patch.

Really, it’s about being accountable to your fellow developers. Like I said, when you’re writing code for your end users they won’t know how pretty it is, how fast it could be, etc. But when you think of your end users as your fellow developers, motivation increases to ensure that code is readable, extensible, and optimized.

It encourages you to review your own code.

When making changes and pushing them directly to a production branch, there’s often little motivation to review your own code once you get all the tests passing. “That seems a good enough indicator that things are working, on to the next thing!”

Again, when you know, for a fact, that your peers are going to review this patch, you’ll be motivated to give it a thorough review yourself before anyone else does. I’ve found that this step is crucial in the development process, much like the refactoring step of TDD, during this step you will always find ways to improve on what you’ve already got.

Reading other people’s code helps you to improve, and helps you improve others.

This tip is often given by elite programmers: Read more code. Formal code review facilitates that. It’s not a matter of some programmers being better than others so that the n00bs can learn from the hax0rz, it’s the point that different developers work on different things and solve problems in different ways.

I’ve found that programmers can learn most when they are exposed to the programming styles of other programmers. This also helps to create a funnel effect in terms of your Coding style. Any organization will have an explicit (or implicit) set of Coding style guidelines. Formal code review will ensure that any code that is not in accordance with style guidelines gets noticed. Similarly, any code that introduces new, better, styles will also get noticed!

It makes for better code.

This is a no brainer. Formal code review is kind of like delayed pair programming. I don’t need to reiterate the benefits of pair programming here, see posts by Obie Fernandez.

Formal Code Review Tools

OK, so now that you’re convinced you need to be doing formal code review, you need tools to support this. If your team is small enough you might think that simply emailing around review requests with Github links would be good enough, but I assure that there is a better way, especially if you have a team size > 2.

Rietveld

The tool that I am most familiar with is called Rietveld. The CLI and UI are less than perfect, but the overall workflow is great. You can see an example project running on App Engine.

My only complaint about Rivetld is that it was built for SVN and has poor integration with Git. See Gerrit.

Gerrit

Gerrit I am less familiar, I’ve only tried the demo. From what I can tell the UI is even more complicated, but the software itself seems cleverer and knows more about Git. I can’t speak to the workflow.

Github

With Pull Requests 2.0 Github is becoming a viable platform for formal code reviews. My opinion is that their tools aren’t quite ready yet but they may be in the future. Missing features: inline commenting on diffs, ability to shorten the list of reviewers (you don’t always want everyone on the project reviewing changes), ability for individual reviewers to mark changesets as resolved, and a Merge! buton. With a few more features, Github really could be the best of the bunch.

Ending

There is often talk in the Ruby community of deploying code dozens of times per day, supporting Agile development or Extreme programming, realeasing early and often, etc. You hear less talk about code quality and maintainability, which are the two main benefits of formal code review.

For the reasons mentioned above, code making it into your project will be of higher quality and higher maintainability. The second point, maintainability, is especially important as the size of your development team grows. When the size of a development team gets above 3, and especially as it enters the double digits, code begins to move very fast. A lot of code will be committed to the project on any given day. Development at that pace is hard to track, and quality/maintainability is left up to the programmer writing the code.

By implementing formal code review you can spread some of that workload. Make sure that a new feature going into your project gets a thorough review by more than just the programmer who wrote it. Like me, you’ll come to enjoy reading and reviewing code almost as much as you enjoy writing it.

Why the Shopify Platform is a Big Deal for Developers

2010-02-09T00:00:00-05:00

Where I give three reasons why the Shopify platform is a big deal for developers and two reasons why it hasn’t caught on (yet). In doing so I will compare the Shopify platform to other, somewhat similar, platforms, namely the Facebook developers program and the Apple App Store.

If you are not familiar with the Shopify Platform, here is a good introduction from @tobi: The Shopify platform.

Disclaimer: I work at Shopify.

Unique market

The Shopify platform provides a unique market that hasn’t ever been more easily exposed. Let me explain with some examples.

Facebook: The Facebook developers program gives an application developer access to a massive market. I forget the current numbers but Facebook has something like 300 million users, each one a potential user of your application. However, Facebook is a free service. It provides a lot of value at no charge. The challenge for a Facebook application is trying to convince its users that it is providing a service valuable enough to warrant payment, even if that compensation is not very high, it can be subsisted with the sheer volume of users for your application.

Apple: Apple’s App Store provides access to a large market of paying customers. iPhone users are paying customers (albeit of some terrible service provider) and had to pay for the iPhone, and there are a lot of iPhone users, again I don’t have numbers, but those things are everywhere. Apple has proved that people are willing to pay for iPhone applications. However, and I’ll bet this is true if you ask any iPhone user, most App Store purchases are made on a whim. An application provides a novel concept and warrants a one-time $1 purchase. However if that application instead required recurring billing, I have a feeling that most purchases wouldn’t last past the first month.

Shopify: The Shopify platform, at first glance, offers a less lucrative market. There are orders of magnitude fewer potential users. Shopify currently boasts approximately 5,000 active sellers, this is your potential user base. However, unlike Facebook, every Shopify user is a paying customer and most are paying significant monthly fees. And unlike iPhone users, Shopify users are typically using this service to power their business and personal income. Shopify has gone to the trouble of filtering out users who are willing to pay for online services, and they are offering developers, through the Shopify platform, the chance to sell their online service to these users.

Unlike the App Store, Shopify users are unlikely to purchase a novel application on a whim. Rather, they will be willing to pay good money for an application that will drive their business. If you come up with an application that lets them process their orders faster, thereby providing better customer service, or increase sales then it seems likely that users will be willing to pay, and continue to pay every month, for a service providing that.

Simplicity of getting started

It’s very simple to get started with the Shopify platform. Especially if you are a Ruby on Rails developer. There is a Rails plugin that you can install that gets you up and running very fast. The plugin handles the Shopify platform authentication scheme for you, it also provides a simple ActiveResource API to access Shopify stores.

I don’t want to show any code examples because the power of the API is that its implemented as with a REST interface as XML over HTTP, so can be accessed from any programming language: Ruby, Python, PHP, Objective-C, etc. You can check out the API docs to see some sample HTTP requests/responses and there is also a screencast showing how to get up and running with a Rails app.

A real strength of the API is the simple, yet flexible, Billing system. The Apple App Store has very simple pricing, you fill in a form that lists your price, they go to the trouble of collecting the money, storing/charging the credit cards, and so on, then you get 70% of that. The Shopify Billing API is similar, except you create the charges for your application programatically, in your code, rather than in a form field.

This opens up whole new billing avenues for your application. Rather than charging every customer the same amount when they install the app, you have a lot of choices. Want to provide a beta period? Trial accounts? Charges based on the number of products that a merchant has? Based on how much the merchant is paying Shopify? Multiple subscription plans? This is all possible, and even simple, with the Shopify Billing API. Just like Apple, Shopify handles the storage of credit cards, collection of money, and you get 80% of your charges. The Shopify Billing API even provides a little more sugar than that, things such as upgrading/downgrading subscription plans is built into the API and the logic for pro-rating plan changes is handled by Shopify!

Ease of marketing

I won’t spend too much time talking about this point. Shopify provides the same kind of marketing as Facebook and Apple. Your application will be listed for free in the Shopify App Store, which is integrated into the admin area of every Shopify customer. Your application may also be featured in the App Store, on the Shopify blog, etc. The Shopify App Store has support for reviews and categories to make your apps easier to find.

Problems

So why hasn’t the Shopify App Store taken off yet? Why isn’t everyone jumping on that wagon?

Problem: Shopify Developers aren’t Shopify Users

This is a problem that Facebook and Apple don’t have. Undoubtedly, Every Facebook developer is a Facebook user, they know how the software works and they know how people are using it. The same is true of the iPhone, anyone developing apps for the iPhone is an iPhone user. They have one of these devices and they know the ins and outs of the device, the expected behaviour of the interface, how much apps typically cost, and so on.

Conversely, very few software developers also happen to be Shopify merchants. Sure, there must be some crossover, but most software developers won’t know what it’s like to run an online store. This is a stumbling block for developers to get started with the Shopify platform. For example, an iPhone user, in the process of using their iPhone, may find a problem that an app could solve, so they can go and write this app, put it on the App Store, and make a buck. However, since most developers aren’t also going to be users of Shopify, this process isn’t really possible.

I was lucky enough to attend FutureRuby this past July and had several conversations with developers about this very topic. They had heard of the Shopify platform and could see the benefits, however they didn’t know what kinds of apps Shopify users were looking for. These developers weren’t e-commerce merchants themselves, so they didn’t know where to begin when thinking about contributing to the Shopify platform.

A possible solution to this problem might be having Shopify app developers work closely with specific merchants to build an app that that a particular merchant might find useful and, most likely, other merchants would find useful. Any developers interested in this approach should probably start by looking around the Shopify forums.

Problem: Lack of a Killer App to Lead the Way

The Apple App Store has done a great job in this area. Stories abound of small developer shops, usually a team of 2-3, pulling in a hundred thousand, or even a million, dollars per month from their iPhone apps. Then there is the story of iWik, the wikipedia browser, selling 50,000 copies in its first week after just a few days of development.

The Shopify App Store isn’t without its success stories however. Fetch, an application helping you sell digital goods, has been going strong for 3 years now with many happy customers. There is also the story of MNDCreative releasing an iPhone app that integrates with Shopify. That company was acquired by Shopify and is now leading mobile development there.

So the Shopify platform already has several success stories. None of them are yet big enough to really inspire others to jump on the bandwagon, but I think it’s just a matter of time.

delayed_paperclip

2010-01-30T00:00:00-05:00

Delayed_paperclip is a rubygem that works with Paperclip and delayed_job. It allows you to easily process Paperclip attachments (such as resizing images) in the background with delayed_job.

Installation

Install the gem:

sudo gem install delayed_paperclip

Add it to your environment.rb:

config.gem 'delayed_paperclip'

Or, even better, to your Gemfile:

source "http://gemcutter.org"
gem 'delayed_paperclip'

Dependencies: Paperclip && delayed_job

Source

http://github.com/jstorimer/delayed_paperclip

Usage

Make sure that you have DJ up and running.

In your model:

  class User < ActiveRecord::Base
    has_attached_file :avatar, :styles => { :medium => "300x300>", :thumb => "100x100>" }
    
    process_in_background :avatar
  end

Use your Paperclip attachment just like always in controllers and views.

Motivation

Every resource that talks about background processing seems to mention typical tasks that should be handled in the background. Image resizing is always one of these tasks.

It makes perfect sense because you want the user experience to be fast, they shouldn’t be waiting while you resize images and upload them to s3. DJ should pick up and process the job almost immediately anyway, so its win-win.

I have used Paperclip on a number of projects now and have never processed my images in the background (I’m sure I’m not the only one). There were a few guides out there with instructions on how to add the right callbacks and methods to your model to resize your images in the background, but I wanted something simpler, so I wrote this gem.

In the Wild

The project that I wrote this gem has a form that allows users to upload screenshots and dynamically add more screenshot upload fields. I did some benchmarks with this form with and without using this gem.

Specs: The project is running on Heroku and using s3 for file storage. I tested the gem by uploading 6 screenshots, each has 3 different styles.

Without using this gem, it took the server 8.5 seconds to process the form and images. Let’s think about the operations for a second here. There were 6 screenshots, each of those had to be resized to three different sizes. Then each of those image styles, plus the original, were uploaded to s3, that makes for 6 × 4 = 24 s3 uploads, all while the user is waiting for the form to submit.

With this gem the server tool only 1.2 seconds to process the form and images. Let’s look at the operations again. There were again 6 screenshots, this time there was no resizing going on. Then each of the original images was uploaded to s3, so a total of 6 uploads to s3. Meanwhile we can present the next page to the user 7 seconds faster, in this case, 1/8 of the time.

Using Mutt with Gmail on OSX

2010-01-19T00:00:00-05:00

What?

Mutt is a command-line email client whose motto is “All mail clients suck. This one just sucks less”. Basically it brings GMail to your Terminal. You can find more information on their site.

Why?

1. Lately I’ve had a bit of an obsession with command line tools. So the fact that mutt gives me command line access to my mail piqued my interest immediately :)

2. I’ve always wanted tab-completion while writing emails, they are a repetitive medium. But GMail never offered that. With mutt I set TextMate to be my editor so I now write email messages in TextMate, which does have tab(escape) completion.

Installation

Install mutt from your favourite package management system.

# Homebrew
brew install mutt

# MacPorts
sudo port install mutt

Customization

You will need to create a ~/.muttrc with your specific mutt configuration. This is the minimum required to get mutt working.

# ~/.muttrc
set imap_user = 'user@gmail.com'
set imap_pass = 'password'
set spoolfile = imaps://imap.gmail.com:993/INBOX
set folder = imaps://imap.gmail.com:993
set postponed=”imaps://imap.gmail.com/[Gmail]/Drafts”
set message_cachedir=”~/.mutt/cache/bodies”
set certificate_file=~/.mutt/certificates

You need to make sure that the ~/.mutt directories actually exist in order to mitigate funny certificate errors. If you want to know what other options are available you can check out my .muttrc or the mutt docs.

Opening URLs

Since mutt is a command line tool, you can’t click links in your emails. You will need urlview in order to open URLs in emails. You will need to install urlview from source and apply a patch for Snow Leopard.

wget ftp://ftp.guug.de/pub/mutt/contrib/urlview-0.9.tar.gz
tar xvzf urlview-0.9.tar.gz
cd urlview-0.9
./configure
# apply the patch found here http://blog.joelesler.net/2009/12/getting-urlview-to-work-on-snow-leopard.html
make
make install

To open URLs in your default browser you need to create a ~/.urlview.

# ~/.urlview
COMMAND open %s

This tells mutt to use the open command on URLs, %s is a placeholder for the URL. Now, when reading an email in mutt, press <Ctrl>-b to see a list of links in the message and use your arrow keys and enter to select one.

HTML emails

You may quickly notice that by default mutt prints out the source of HTML emails, not very useful. It is true that everyone should be providing plain text email for command line lovers like us, but alas, some people are ignorant of this fact. In order to view HTML emails with mutt just add this line to your ~/.muttrc.

# ~/.muttrc
auto_view text/html

Cheating

I created a cheat sheet that has some common commands that you might use with mutt to manage your GMail. You can see the sheet online or access from the command line with cheat: cheat mutt_gmail.

I’m still a beginning mutt user (about 1 week), so I haven’t yet explored all the nuances of the software. Particularly excited about looking into keyboard macros.

Speed Up Your Paperclip Tests

2010-01-05T00:00:00-05:00

We all love Paperclip. One of the reasons why is that it makes it painless to resize images. Which is great when you are working in development or on production, but image resizing can really slow down your automated tests.

For example, I was working on a project that used factories instead of fixtures. Every time a factory object was created, which happened multiple times in most tests, Paperclip had to resize its images.

I added this little snippet to my test_helper.rb and saw my total testing time go from 100 seconds to 30 seconds. Meaning that 2/3 of the time spent running tests was actually spent resizing images!

# test/test_helper.rb

class ActiveSupport::TestCase
  # ...
end

# replace the class name with one appropriate for your app
class Screenshot
  before_post_process do |image|
    false # halts processing
  end
end

I’m sure that there are some automated testing purists out there who will cry that this method no longer tests the entire stack. It’s possible to not have ImageMagick installed on your system and still have passing tests. My answer to that is that any good testing strategy will involve an element of human testing which should catch this kind of problem.

Explanations

2009-12-30T00:00:00-05:00

“We teachers – perhaps all human beings – are in the grip of an astonishing delusion. We think that we can take a picture, a structure, a working model of something, constructed in our minds out of long experience and familiarity, and by turning that model into a string of words, transplant it whole into the mind of someone else.

Perhaps once in a thousand times, when the explanation is extraordinary good, and the listener extraordinary experienced and skillful at turning word strings into non-verbal reality, and when the explainer and listener share in common many of the experiences being talked about, the process may work, and some real meaning may be communicated. Most of the time, explaining does not increase understanding, and may even lessen it.”

— John Holt, (1923-1985) American Educator, How Children Learn

Jekyll on Heroku

2009-12-29T00:00:00-05:00

This blog is published using Jekyll. Jekyll is a static site generator, so you write your posts and pages on your local machine, generate your static HTML site and then publish it using a service that serves static files. Like GitHub Pages or Slicehost.

But neither of those services are free, and my blog has nowhere near enough traffic that it should cost me money, so I wanted to use the free Heroku plan. This is the method I have been using to publish my blog for the past month.

Basically Heroku treats my blog like its a Sinatra app. When you hit the root of the application it serves the index.html from the _site/ folder, and then the links from that page take the user where they want to go.

Simplicity

I recently noticed that there is a project called rack-jekyll that is supposed to make it easy to publish your Jekyll site with Rack. I tried out this project and it failed miserably when trying to display my blog.

The method here is much simpler. The nice thing about the fact that this is a Sinatra app is that you have some built in niceties. Want custom 404 pages? Sinatra has that. Need to demonstrate some dynamically generated code? Just set up another route and go right ahead.

Local Development

Use the jekyll --auto --server command to develop locally.

Deploying

To deploy your jekyll site to Heroku:

Make sure that your _site/ folder is not in the .gitignore file.
Add the two files from the gist above.
Commit changes.
heroku create (first time around)
git push heroku master

Notes on git pushing

2009-12-27T00:00:00-05:00

Pushing all local branches

Executing

git push

will update all local branches that are tracking remote branches. For example, if you have set up your local master branch to track the remote origin master branch, then executing git push will update that branch. More on tracking remote branches here

Pushing the current local branch

Say you are working in your local master branch and want to push your changes, but don’t want to update all branches. You can run the git push command, passing in the remote and the name of the branch you want to update:

git push origin master

Pushing to a named remote branch

So, what if have changes in your master branch that you want to push to the remote experimental branch? You could checkout your experimental branch, cherry pick the changes and then use the method above to push those changes, but that’s no fun.

You can actually specify that you want to push your changes to some branch with a different name than the one you are working on with something like this:

git push <remote> <from>:<to>

In our example we would fill in the blanks like so:

git push origin master:experimental

This tells git to push the changes from your local master branch to the remote experimental branch. And we didn’t even need to write /refs/heads anywhere ;)

Deleting remote branches

I discovered a while ago the way to delete remote branches. For example, if we wanted to delete our remote experimental branch we would issue this command:

git push origin :experimental

At first I was thinking ‘Wow! That looks really dumb and makes no sense.’ But, if you look at the above section it makes perfect sense. Think of it this way:

git push origin <nothing>:experimental

It’s as if we are pushing nothing into the experimental branch. The same way we pushed the master branch into the experimental branch above.

Essential Rails Plugins for your Inner DBA

2009-12-13T00:00:00-05:00

I like to think that I’m not bad at writing SQL. When it comes to writing complex queries, joinging tables, grouping columns, sorting, I’m quite comfortable.

However I never seem to give any thought to the possible size of the tables I am working with. Are there hundreds of records? Hundreds of thousands? Millions? This is usually not an issue for local development, however once your app is out in the wild, this is a big issue.

Recently, I have been using two awesome Rails plugins to remind me to not forget these things.

Query Reviewer

http://github.com/dsboulder/query_reviewer

This plugin is great. Basically it adds a js widget to all your pages that runs the EXPLAIN command on each of your queries and lets you know if any warnings were generated.

Just install the plugin and it will be working by default in your development environment. For each problem query, the Query reviewer widget will tell you what the problem is, ie. ‘No index was used. Performed full table scan’, ‘A temporary table had to be used’, etc.

Here is an example where you need to fix something, adding an index in this case.

Here’s where you hope to get to:

Bullet

http://github.com/flyerhzm/bullet

This library detects N+1 queries and unused eager loading. Again, something that you probably wouldn’t notice in a development environment but can become very important in a production environment.

For me this is even more important because I am beginning to understand SQL indexes and where they should be used, but eager loading still escapes me at most junctions.

For Bullet, you need to add some config code to your development environment. By default it logs problems to the Rails log, as well as giving you a js alert. You can even get this thing working with Growl, if you’re into that kind of thing.

Here is an example of its output, its pretty smart with its suggestions:

Conclusion

These plugins are a great way to catch database optimizations that you might have missed. However, these suggestions should be taken with a grain of salt. Much like code metrics, they can only go so far. You know your app better than these tools and in some cases it may be the right decision to ignore them.

Installing rb-gsl on Snow Leopard

2009-10-18T00:00:00-04:00

I wanted to try experimenting with SVD recommenders in Ruby (via this fantastic blog post http://www.igvita.com/2007/01/15/svd-recommendation-system-in-ruby/) and was having a hell of a time trying to install Ruby’s linalg. When I saw that there was also an example using rb-gsl, I was thrilled because rb-gsl is available from MacPorts. No problem, right? Wrong!

Currently if you try to install rb-gsl form MacPorts (on Snow Leopard) you will get a build error. If you download the latest release of rb-gsl (1.10.3 at the time of this writing) you will get the same build error. It appears to have some code that doesn’t work on Snow Leopard. Here is how I got it installed.

If you haven’t already, install MacPorts for Snow Leopard (http://forums.macrumors.com/showthread.php?t=720035)
Install the GSL library
$ sudo port install gsl plotutils
Because of the incompatibility of the current rb-gsl and Snow Leopard, you will have to install from the latest svn checkout.
$ svn checkout svn://rubyforge.org/var/svn/rb-gsl
$ cd rb-gsl/trunk/rb-gsl
$ sudo ruby setup.rb config --prefix=/usr/local
Then we need to edit two lines in the generated ext/Makefile
Open up the file for editing and change these two lines to look like this:
DLDFLAGS = -L.
LDSHARED = $(CC) -pipe -bundle -undefined dynamic_lookup
Then continue with the build and you should be on your way
$ sudo ruby setup.rb setup
$ sudo ruby setup.rb install

Enjoy!

If you are installing this lib to test out the example from above then you may want to check out my comment on that article.

Sources:
http://rubyforge.org/forum/forum.php?thread_id=19389&forum_id=1225
http://rubyforge.org/forum/forum.php?thread_id=45613&forum_id=1225

Sinatra and Shopify: A Classy shopifyapp

2009-06-20T00:00:00-04:00

This week I wrote up a basic sinatra app that allows you to work with the Shopify API. It’s basically just a port of the shopify_app rails plugin to sinatra.

Why?

I have been having fun working on Shopify apps lately, but I needed a simpler way than always generating a new rails app, new controllers, etc., for every little idea.

Rails is great, but most of the Shopify apps that I have come up with are one page apps, maybe two or three pages. The point is, I don’t need a controller for that, I don’t need the 63 files or however many the rails generator generates for me.

Sinatra is most awesome when you are working on a small app with just a few routes. You can clone this app and use it as a starting point for a shopify_app. All of the authentication logic/routes are in the lib folder, so they don’t pollute the code of your app.

So in app.rb, you just have to worry about the functionality of your app.

Usage

The fastest way to get this thing running:

git clone git://github.com/jstorimer/sinatra-shopify
cd sinatra-shopify
heroku create (make sure that you remember the name of your app as you will need that later)
visit the Shopify Partners area
Sign up for an account if you don’t already have one
Log in to your Shopify partners account
Create a new application.
Make sure that the Callback URL is set to http://yourapp.heroku.com/login/finalize
In the lib/sinatra/shopify.yml file replace API_KEY and SECRET with the Api Key and Secret that you just got from the Shopify Partners area
git push heroku master
heroku open

The current implementation for authentication is to add the authorize! method at the top of any routes that you want to be authenticated, like so:

Gimme the code!

http://github.com/jstorimer/sinatra-shopify

Disclaimer

I am by no means a sinatra expert, this is really my first experience with sinatra. So if you see anything that could be done better, the source is on Github, fork away.

Shopify API Extensions

2009-06-15T00:00:00-04:00

Excited by the release of the Shopify platform, I have been playing a lot with the Shopify API and building some apps for the new App Store.

The Shopify API is super easy to work with, thanks to the fact that it provides an ActiveResource interface, and of course I write all of my apps in Ruby. In the course of playing with the API, I have needed some specific helpers not provided by the Shopify API, more specifically, things that ActiveResource doesn’t do well. So I added a few helpers to the Shopify API that others might find useful.

Base class

The official shopify_app plugin uses ActiveResource::Base when setting the site, username, password, etc. This can potentially cause problems if you are working with multiple ActiveResource API’s in your application. I have added a ShopifyAPI::Base class so that there is never any spillover of username’s and password’s between different ActiveResource API’s.

Batched find

Can be used like:

This is especially useful if you need to do something to all products, or all orders, or whatever, in a background task. There is no easy way to ask for all of a particular object using ActiveResource, so this method just asks for a batch at a time, and lets you work with each batch until there is no more.

By default, it uses a batch size of 100, but you can override that by passing in a limit param, like so:

This will work in the same but will fetch things in smaller batches instead of 100 at a time.

Count

ActiveResource does not provide a nice way to ask for a count of objects, nor does the Shopify API. Now you can do something like:

to get a count of all the orders for a particular shop. Once again this is actually finding all orders in batches and just recording the total number of orders returned.

Word of warning

Both of these methods involve fetching all of a particular type of object through the Shopify API. Since some shops may have hundreds or thousands of orders, products, etc. it is not wise to do this in a user-facing screen. These things should be reserved for background processes only.

Check out my fork of the shopify_app plugin on github to get these goodies.

Readability

2009-06-10T00:00:00-04:00

I decided to update my blog’s template after using Readability for a few days. I liked the setup that Readability provided so much that when coming to my blog, I noticed that it was very hard to read. The font was small and grey and on a white background, which I thought was OK.

Through using Readability I have discovered that it’s easier (for me at least) to read things when the background is off-white, this kind of manilla color is best for my eyes.

So you should check out Readability, it’s an awesome way to read websites, especially useful for those sites that insist on using 9-point grey font that’s impossible to read :P

I’m currently using the Novel style with medium font and a wide margin.

I’d love to hear what settings work best for any of you out there.

Transformers and Git

2009-05-30T00:00:00-04:00

Today, I was watching some of the Special Features from the live-action Transformers movie, very interesting stuff.

I noticed that whenever they would show clips from the movie there would be some text at the bottom of the screen, always a timestamp followed by a string of text.

Here is one such message: “additional paint and suppression sweetening…”

Looks like a version control system to me, and looks like those animators have the same problem as I do, trying to come up with good commit messages.

UML vs. Rails Simplicity

2009-03-24T00:00:00-04:00

Yesterday I was working through a conceptually difficult relationship between my Rails models. After sitting at the keyboard and pulling my hair for a few minutes I went to the whiteboard where I could lay things out and get a better overview of the situation.

Having taken a course on Software Engineering at University, my first instinct was to use UML and pictures to represent the situation I wanted and transcribe it to code later. But since I'm using Rails it was actually easier to just write the code. I just wrote something like this:

instead of

YIKES!

It's funny that UML exists as a pseudo-language on top of other programming languages, so that people, technical and non-technical, can talk about these things and you can have a shorthand for defining relations between models etc. without having to actually write code.

I love how Rails circumvented a lot of that by just making the framework so simple. The way I talk about Rails (and Ruby) code at a high level is the same way that I actually implement it in code.

Posted via email from Jesse’s posterous

Ruby on a Plane

2009-03-12T00:00:00-04:00

Lately it has seemed that every time I turn around, I hear about another great Ruby project that was started on the plane to .

I have been keeping a list of the ones I have come across so far, either in conversation, blog posts or podcast. If you know of more, please leave them in the comments.

I realize that great projects also happen off of planes, but I can’t ignore how often I have seen this lately :[]

Skating to Work

2009-01-25T00:00:00-05:00

Ahh Canada. Where else could someone skate to work?

I consider myself lucky, not only do I live in Ottawa, home of the worlds largest skating rink, but I also live near the middle of the Rideau Canal and work near one end of the Rideau Canal.

In the mornings and evenings I tie up my skates and go for a 15 min. skate on the Canal. It’s a great way to start and end the day.

And then there are ‘those’ days. ↑

Skating on the Canal is a great way to get some exercise and see some of the downtown from a different angle. If you happen to be in Ottawa in the winter, it’s definitely checking out.

Sometimes in the morning there is no one else out there but me, and it’s so freeing to be skating out there in downtown Ottawa with no people around and no cars around, it’s awesome.

Rework Your Git History

2009-01-16T00:00:00-05:00

Git is an awesome tool. Recently I have been discovering more and more cool things I can do with git. Here is one such thing.

I was recently working in a development branch and had made more commits than were necessary. I had been committing every time I got a small piece of new functionality to work and I had about 12 commits that should have been only 3 commits. Enter git rebase --interactive:



git rebase -i

This is awesome. It gives you an interactive way to reorganize your commit history since the one specified in <commit>. It opens up your $EDITOR with the list of commits, something like this:



pick c9205f4 Ok, so spawn doesn’t work exactly right yet.

pick c770395 Added Errno::ENOTDIR to the list of errors when rmdiring

pick 49221b6 There’s no reason to load anything but test for the database.yml

pick bc1af0c Removed rake/gempackagetask and the rubyforge rake task

pick 8a43c13 Added requirement for thoughtbot-shoulda >=2.0.0 in test_helper

pick 06f9b99 #47: Multipage PDFs fixed by Pete Deffendol

pick 279b757 Updated the rake task to take STI into consideration.

 

# Rebase 701abb0..279b757 onto 701abb0

#

# Commands:

#  pick = use commit

#  edit = use commit, but stop for amending

#  squash = use commit, but meld into previous commit

#

# If you remove a line here THAT COMMIT WILL BE LOST.

# However, if you remove everything, the rebase will be aborted.

#

You can rewrite pick to change the behaviour.

If you want to combine a commit with the one above, just change it from pick to squash.

If you want to leave a commit as it is, just leave it at pick.

If you want to change a commit, change it to edit. Edit is awesome, you can change which files are included in the commit and even update old commit messages.

You can even change the order of commits by just rearranging the lines in the file. But be careful, if you delete a line, then that commit will be deleted.

When you exit the text editor, it will start rebasing your commits. If there is a merge conflict, or when a commit has been marked as edit, then the rebase stops. You make your changes and amend your commit with git commit --amend. Then just git rebase --continue to continue on with the rebase!

What the heck is JSONP?

2009-01-12T00:00:00-05:00

Recently I stumbled across something called JSONP. I read that it was a hack that allowed for cross domain AJAX requests, which of course piqued my interest, but I had a really hard time finding out how it worked. After spending a few days experimenting and reading blog posts, I think I have a grasp on it.

So here is my attempt at a pop-up book explanation:

What is JSON?

JSON stands for JavaScript Object Notation. It is simple a format for representing data, much like XML. Here is some data represented in XML:

and the same data represented in JSON:

What is JSONP?

JSONP stands for JSON with Padding. It is basically a clever hack that allows for cross domain AJAX requests, but it requires the participation of the remote server you are requesting data from. It works something like this:

You request a JSON document from a remote server and specify a callback parameter. This would look something like http://search.twitter.com/trends.json?callback=handle_data.
This is the part that requires participation from the remote server. If they support JSONP, they will wrap the resulting JSON data in a JS function call, returning a script that looks something like this:

How can I use this?

If you define a callback method in your document and then use a script tag to include a JSONP call, you can request and handle that JSON data, all from JS. Like so:

Yay! Cross domain JS request.

If this went right over your head, as it did for me the first few times, here is a recap:

You included a remote script (from Twitter in this case) that made a call to a JS function (whose name you specified with the callback parameter) with the JSON data that you wanted as an argument to that function.

This is pretty neat, but it does have drawbacks:

Danger: You are allowing the remote server to execute arbitrary JS on your page. This is generally a bad idea and you have no way of verifying what that server is embedding in your document. If you trust the server then this technique works just fine :|
This technique requires that the remote server implement some logic to allow you to do this. This is becoming more prevalent however. Thus far I have discovered that Twitter and Flickr both support this. Anyone find others?

As a result of this stuff, I did a fun experiment. It grabs the latest trends from Twitter and searches for images tagged with the same at Flickr, all in JS! The source is on Github.

Environmentally Friendly Image Search

2009-01-07T00:00:00-05:00

Recently I was looking for some stock images on Google Image Search and wasn’t finding what I wanted. So I searched for other image search engines and found Picsearch.

Not only did it give me better results but they are apparently the first search engine to go carbon neutral.

Here is their 4 step program:

1. Picsearch plants 1000 new trees for every gigawatt-hour consumed.
2. Picsearch buys and holds in trust 2 acres of old growth forest for every gigawatt-hour consumed.
3. Picsearch only uses electricity that is carbon free and follows the highest standard of production.
4. Picsearch buys carbon credits equivalent to all energy that Picsearch consumes.

So next time Google Images isn’t finding what you are after, try a friendly alternative.

Posted via email from Jesse's posterous

The Beginning

2008-12-31T00:00:00-05:00

Whaddya know? My first blog post.

This blog is coming to you live from Shopify. The best in e-commerce and evidently a solid blogging platform as well.

I'm really liking the ~~Vostok theme~~ I went with for this blog, not bad eh.

Anyway, stay tuned as I hope to be bringing more actual content to this blog in the near future. I am currently really interested in Rails and new technologies that make web development easier and faster. I hope to be posting my findings on this blog.

Posted via email from Jesse’s posterous

Jesse Storimer

Passing Open Files Between Processes with UNIX Sockets

UNIX Sockets are Local

Related Processes Share Open Files By Default

Sharing File Descriptors Between Unrelated Processes

Conclusion

4 Months of ebook Sales

Backstory

Launch

Numbers

Total Revenue: $17,992 USD

Total # of Sales: 746

Aftermath

Inspiration

Introduction to IPC in Ruby

Our First Pipe

Pipes Are One-Way Only

Sharing Pipes

Streams vs. Messages

Remote IPC?

In the Real World

System Calls

Daemon Processes in Ruby

The First Process

Creating Your First Daemon Process

Diving into Rack

Daemonizing a Process, Step by Step

Process Groups and Session Groups

In the Real World

System Calls

How to Give a Good Tech Talk

Free

Paid

Exemplary Talks

Put Your Inbox in the Upstairs Bathroom

A Unix Shell in Ruby - Part 4: Pipes

First, the Peculiarity

The Base Case

An Impostor in the Lineage

The Big Reveal

Aftermath

Shell Pipelines Demystified

The Shell Pipeline

The Ending, Then the Beginning

Whew!

eBooks for Programmers Should Include a TXT

A Unix Shell in Ruby - Part 3: A Login Shell and the PATH

First, A Word of Warning

Changing the Login Shell

Environment Variables

How the PATH Broke Our Login Shell

Back on the Path

Like Bash?

Setting Environment Variables

Connecting the Dots

'Local' Environment Variables

More Environment Variables

A Refactor

A Unix Shell in Ruby - Part 2: Builtins

A Note on Launching

Even cd doesn't work!

Backstory

A Note About Builtins

Implementing cd

Just Try to Leave, I Dare You!

Change me!

Before You Go

OK, now you can leave

A Unix Shell in Ruby

Why a shell?

What's a shell?

Prior Art

The Plan

shirt

Taking the Hard Way

Getting Input

Executing a Command

Waiting Patiently

Some Chrome

Just the Start

Where does `stderr` come from?

Why do we need `stderr`?

Redirecting `stderr`

`stderr` and Pipes