Odes of the Occult

Semi-eager realization of lazy sequences in ClojureScript

2017-04-21T00:00:00+02:00

asciinema player represents "frames" as a lazy sequence of screen states.

It starts by fetching asciicast JSON file, and extracts STDOUT print events from it. They form a vector like this:

(def stdout-events [
  [1.2 "hell"]
  [0.1 "o"]
  [6.2 " "]
  [1.0 "world"]
  [2.0 "!"]
  ...
])

To get a sequence of "frames" we run reductions function over it, with blank terminal (vt) as initial value:

(defn reduce-vt [[_ vt] [curr-time str]]
  [curr-time (vt/feed-str vt str)])

(let [vt (vt/make-vt width height)]
  (reductions reduce-vt [0 vt] stdout-events))

The details of the above code are not really important. The important thing is that the result of reductions is a lazy sequence. The player consumes this sequence as the playback progresses, sleeping proper amount of time (the number in stdout-events items) between screen updates.

When the sequence is consumed, reduce-vt function interprets text with control sequences (str arg in reduce-vt) and applies screen transformations to terminal model (vt). Usually this is not computationally intensive, but can be pretty heavy for more colorful, animated recordings. In such cases the animation stutters.

Also, when you skip forward to further position in the recording the player needs to consume all items from this lazy sequence up to the point you requested. This often requires lots of work, because computation for all not realized items adds up. In some cases it's visible as UI lag - player doesn't respond to user input because it's busy interpreting tens of kilobytes of text.

Knowing that the sequence will eventually be consumed in full, why not pass it through doall right before the beginning of the playback to generate all screen states upfront? That would certainly help with stuttering and made seeking blazing fast, but it would block the UI thread for couple of seconds or more. Not quite acceptable solution.

Would be great if we could semi-eagerly realize the lazy sequence piece by piece when the browser is idle during playback (e.g. when the player waits for time to pass before displaying next screen contents).

Cooperative scheduling to the rescue! requestIdleCallback, a younger sibling of requestAnimationFrame, allows code execution to be scheduled when the browser has nothing to do. Perfect fit for the problem.

Here's a requestIdleCallback based version of Clojure's dorun:

(defn dorun-when-idle [coll]
  (when-let [ric (aget js/window "requestIdleCallback")]
    (letfn [(make-cb [coll]
              (fn []
                (when (seq coll)
                  (ric (make-cb (rest coll))))))]
      (ric (make-cb coll)))))

(dorun-when-idle my-lazy-seq)

It goes through all sequence items, one item per every requestIdleCallback invocation. Clojure(Script)'s data structures are immutable, but lazy sequences cache their items internally. This allows us to walk through the sequence in one place, realizing it, while consuming it in another place, at slower pace.

The above function reminds me of little-known seque function, which realizes lazy sequence in a separate JVM thread, trying to stay ahead of the consumption. dorun-when-idle is pretty hungry, as it goes to the end of the sequence regardles of the consumption pace, however I think it should be possible to implement seque in ClojureScript through requestIdleCallback with the same semantics as in Clojure.

Anyway, this greatly improved animation smoothness and seeking responsiveness in asciinema player under major browsers, and it's already running on asciinema.org.

core.async in the browser is sweet

2015-10-12T00:00:00+02:00

Recently I've built the new version of asciinema player using ClojureScript, Reagent and core.async. The experience was so pleasurable that the player, which was so far my least favorite part of asciinema (I never fully enjoyed building frontends in JavaScript), became my most favorite part of the whole project.

ClojureScript tooling is much, much better now than it was 2 or even 1 year ago, and with Figwheel interactive development experience went to "eleven". If you wanted to get your feet wet with ClojureScript there was never better time to do that (try lein new figwheel hello-world).

When re-building the player I found several neat use cases for core.async which you may find useful in your projects. Below I picked two of them, both accidentally related to time.

Let's start with requiring few pieces from Google Closure and core.async libraries which we'll need later:

(ns demo.core
  (:require-macros [cljs.core.async.macros :refer [go go-loop]])
  (:require [goog.dom :as dom]
            [goog.events :as events]
            [cljs.core.async :refer [put! chan <! >! timeout close!]]))

asciinema player models the animation as a sequence of frames, where each frame is a tuple of delay and data (screen diff). For each frame it waits for the number of seconds specified by the delay and then it updates the terminal by applying the screen diff to the current content of the screen.

Let's simplify the problem (and animation frame model): each frame is a tuple of delay and text to print.

(def frames [
  [1.2 "hell"]
  [0.1 "o"]
  [6.2 " "]
  [1.0 "world"]
  [2.0 "!"]
  ...
])

There's no blocking sleep function in JavaScript, so the usual way of printing each frame's text would be to use setTimeout with a function that prints the text and then schedules next printing with setTimeout again. You can imagine it's not very pretty. But the biggest problem with this solution is that the ceremony with setTimeout completely obscures the intent of the code.

How about this:

(let [data-chan (coll->chan frames)]
  (go-loop []
    (when-let [text (<! data-chan)]
      (print text)
      (recur))))

We loop over the data received from data-chan channel, printing text, breaking the loop when the channel is closed (when-let). It looks like an old school loop that iterates over the elements of a collection, printing them. The difference is that it's not a collection but core.async channel, so the values arrive "when they're ready".

So, when are they ready? Here's coll->chan function used above to convert a sequence of frames to a channel:

(defn coll->chan [coll]
  (let [ch (chan)]
    (go
      (loop [coll coll]
        (when-let [[delay data] (first coll)]
          (<! (timeout (* 1000 delay)))
          (>! ch data)
          (recur (rest coll))))
      (close! ch))
    ch))

This is similar to core.async's to-chan, which turns a collection into a channel and emits all collection values immediately. Here, we know that each collection value is a tuple containing delay and data to emit, so instead of emitting all values as quickly as possible coll->chan sleeps for necessary time before emitting data for a given frame.

In asciinema player coll->chan function is used not only for obtaining a channel of frames, but also for obtaining a channel of cursor blinks ((coll->chan (cycle [[0.5 false] [0.5 true]]))) and a channel of progress bar status updates ((coll->chan (repeat [0.3 true]))). It turned out to be a great abstraction for these time related problems.

The other problem for which core.async happened to be a great fit was detecting user activity. In fullscreen mode I wanted to show the title bar and progress bar when user moves the mouse but hide them 2 seconds after the last mouse move.

We can generalize this problem to: convert arbitrary input channel into an output channel, where output channel emits true after new values start showing up on input, then emits false after no new values show up on input for the specified amount of time, then again waits for new values on input and emits true and so on and so on.

Here's how we can do that:

(defn activity-chan [input msec]
  (let [out (chan)]
    (go-loop []
      ;; wait for activity on input channel
      (<! input)
      (>! out true)

      ;; wait for inactivity on input channel
      (loop []
        (let [t (timeout msec)
              [_ c] (alts! [input t])]
          (when (= c input)
            (recur))))
      (>! out false)

      (recur))
    out))

This function creates and returns an output channel and sends true/false to it according to the above specification.

Note, this assumes the input channel will never close (which may be true in case where you send DOM events onto input channel). If it is closed at some point then... infinite loop (how to make it channel-close-proof is an exercise for the reader).

With activity-chan function in hand, we can detect and print changes to user activity, defined as mouse movement here, with this piece of code:

(let [dom-element (dom/getElement "player")
      mouse-moves (chan)
      mouse-activity (activity-chan mouse-moves 2000)]
  (events/listen dom-element "mousemove" #(put! mouse-moves %))
  (go-loop []
    (print (<! mouse-activity))
    (recur)))

All "mousemove" events get sent to mouse-moves channel, which is feeding mouse-activity channel, from which we read (and print) in a loop. It prints true, false, true, false, true and so on.

These are are just two of many use cases for core.async in the browser. core.async is a really great tool for solving async problems and modeling process communication, and as you can see it can help with abstracting many UI problems, making the code concise and intent revealing.

Deep Dive Into ROM... With Clojure

2015-07-13T00:00:00+02:00

Recently I read the slides of Deep Dive Into ROM, the latest talk by Piotr Solnica. It's a great talk about how ROM is built and what are the main principles behind its implementation. While reading the Ruby code presented on the slides, code that is very functional in nature, with focus on immutability, I started realizing how close all of this was to a code one could write in Clojure. Let's look at some examples.

On slide 9 we can see the following Ruby code:

class Thing
  attr_reader :value

  def initialize(value)
    @value = value
  end

  def with(value)
    self.class.new(value)
  end
end

thing = Thing.new(1)
other = thing.with(2)

Here Piotr is implementing an immutable object, with handy with method for getting a modified version of the object. The original thing object is never modified.

One of the many great things about Clojure is its immutable, persistent data structures. Let's look at the snippets below:

(conj '(1 2 3) 4) ; => '(4 1 2 3)
(conj [1 2 3] 4) ; => [1 2 3 4]
(conj {:a 1, :b 2} {:a 5}) ; => {:b 2, :a 5}
(conj #{:a :b} :c) ; => #{:c :b :a}

conj is a function which adds a new element to a given collection. Here we add new elements to a list, vector, map and set. The input collection stays unmodified and you get a new one, which shares its structure (previous elements) with the original one.

Thing class example from Ruby could be as simple as this in Clojure:

(def thing {:value 1})
(def other (conj thing {:value 2}))

Let's jump to the slide 11 of the presentation, containing the following code:

class AddTo
  attr_reader :value

  def initialize(value)
    @value = value
  end

  def call(other)
    other + value
  end
end

add_to = AddTo.new(1)

add_to.call(1) # => 2
add_to.call(2) # => 3

This looks a lot like partial function application in functional languages. In Clojure we can use partial function to achieve similar effect:

(defn add-to [value]
  (partial + value))

(def add-to-one (add-to 1))

(add-to-one 1) ; => 2
(add-to-one 2) ; => 3

Note, that + here isn't any special language keyword (operator). It is the name of the function, that returns a sum of numbers. Thanks to this, we are able to pass it as a function to other higher-order functions like partial.

Now let's look at slides 15 & 16, where we can see a usage of a class which requires some collaborators and configuration:

class StringTransformer
  attr_reader :executor, :operations

  def initialize(executor, operations)
    @executor = executor
    @operations = operations
  end

  def call(input)
    operations.reduce(input) { |a, e|
      executor.call(a, e)
    }
  end
end

executor = -> str, meth { str.send(meth) }
operations = [:upcase]

upcaser = StringTransformer.new(executor, operations)

upcaser.call('hello world') # => "HELLO WORLD"

This is how a Clojure equivalent could look like:

(defn string-transformer [executor operations]
  (fn [input]
    (reduce executor input operations)))

(defn executor [str f]
  (f str))

(def operations [clojure.string/upper-case])

(def upcaser (string-transformer executor operations))

(upcaser "hello world") ; => "HELLO WORLD"

string-transformer function takes required collaborators (executor) and configuration (operations) and returns a new function that takes one argument and executes reduce operation with it.

Slide 18 shows how proc currying can be used in Ruby:

add = proc { |i, j| i + j }

add_to_one = add.curry.call(1)

add_to_one.call(2) # => 3

While Clojure doesn't have auto-curried functions (it prefers variadic functions) this example can be approximated with partial application:

(defn add [i j] (+ i j))

(def add-to-one (partial add 1))

(add-to-one 2) ; => 3

Or, even shorter:

(def add-to-one (partial + 1))

(add-to-one 2) ; => 3

On slide 21 Piotr shows the same interface (.[]) for executing a default operation (method) on objects of different types:

add = proc { |i, j| i + j }

add[1, 2] # WEIRD?

hash = { a: 1 }
hash[:a] # less weird, right?

arr = [:a, :b]
arr[0] # less weird, right?

add[1, 2] doesn't feel natural in Ruby but it's consistent with hash and array element access (the latter two feel very natural). Interestingly, Clojure's syntax doesn't distinguish between calling a function and accessing an element in a collection:

(defn add [i j]
  (+ i j))

(add 1 2)

(def hash {:a 1})
(hash :a) ; => 1
(:a hash) ; => 1

(def arr [:a :b])
(arr 0) ; => :a

As you can see maps, vectors (also lists and sets) are functions too! They implement IFn interface and thus are callable.

But wait, what about (:a hash) invocation? Keyword (called symbol in Ruby) also implements IFn and can be used to lookup itself in a given map. This is really useful when mapping on a collection of maps because you can pass a keyword as a mapping function to map:

(map :name [{:name "Jane" :age 30} {:name "John" :age 31}]) ; => ("Jane" "John")

Fast-forward to slide 33, we see procs, procs and even more procs:

all_users = [
  { name: 'Jane', email: 'jane@doe.org' },
  { name: 'John', email: 'john@doe.org' }
]

User = Class.new(OpenStruct)

find_by_name = proc { |arr, name|
  arr.select { |el| el[:name] == name }
}

map_to_users = proc { |arr, name|
  arr.map { |el| User.new(el) }
}

users_by_name = proc { |arr, name|
  map_to_users[find_by_name[all_users, name]]
}

users_by_name[all_users, 'Jane'] # => [#<User name="Jane", email="jane@doe.org">]

So, functions, functions and even more functions in Clojure, right?

(def all-users [{:name "Jane", :email "jane@doe.org" } {:name "John", :email "john@doe.org"}])

(defrecord User [name email])

(defn find-by-name [arr name]
  (filter #(= name (:name %)) arr))

(defn map-to-users [arr]
  (map #(->User (:name %) (:email %)) arr))

(defn users-by-name [arr name]
  (map-to-users (find-by-name arr name)))

(users-by-name all-users "Jane") ; => (User{:email "jane@doe.org", :name "Jane"})

Note that usage of Clojure's record is a bit artificial here but I left it to match the intent of the above Ruby code.

There's one more interesting (and highly functional) piece of code we can look at. Slides 45-47 show this:

class Users < ROM::Relation
  forward :select

  def by_name(name)
    select { |user| user[:name] == name }
  end
end

class Tasks < ROM::Relation
  forward :select

  def for_users(users)
    user_names = users.map { |user| user[:name] }
    select { |task| user_names.include?(task[:user]) }
  end
end

user_dataset = [
  { name: 'Jane', email: 'jane@doe.org' },
  { name: 'John', email: 'john@doe.org' }
]

task_dataset = [
  { user: 'Jane', title: 'Task One' },
  { user: 'John', title: 'Task Two' }
]

users = Users.new(user_dataset).to_lazy
tasks = Tasks.new(task_dataset).to_lazy

user_tasks = users.by_name >> tasks.for_users

user_tasks.call('Jane').to_a # => [{:user=>"Jane", :title=>"Task One"}]

The above code defines 2 relation classes, instantiates them with actual datasets and combines them into a pipeline. The resulting pipeline can later be called to get the result.

This is no different from function composition in functional programming. Clojure has comp function we can use:

(def all-users [{:name "Jane", :email "jane@doe.org" } {:name "John", :email "john@doe.org"}])
(def all-tasks [{:user "Jane", :title "Task One"} {:user "John", :title "Task Two"}])    

(defn users-by-name [users name]
  (filter #(= name (:name %)) users))

(defn tasks-for-users [tasks users]
  (let [user-names (map :name users)]
    (filter #(some #{(:user %)} user-names) tasks)))

(def my-users-by-name (partial users-by-name all-users))
(def my-tasks-for-users (partial tasks-for-users all-tasks))

(def user-tasks (comp my-tasks-for-users my-users-by-name))

(user-tasks "Jane") ; => ({:title "Task One", :user "Jane"})

Here my-users-by-name and my-tasks-for-users functions are a result of calling partial over users-by-name and tasks-for-users respectively. They're parametrized with all-users and all-tasks, and can be seen as equivalents of users.by_name and tasks.for_users from the Ruby example.

To translate Piotr's users.by_name >> tasks.for_users to Clojure we used comp to compose these 2 functions into a new one, which when called calls the first one with the result of the second one. Note that comp executes the functions in reversed order (unlike Piotr's Ruby version which uses >> operator to nicely visualize the data flow).

To sum up, if you feel that your code looks weird, then maybe it's time to look at other programming languages (not necessarily Clojure). There's a good chance, that you will find one, that suits your current way of thinking, allowing you to express your favourite problems in a simpler, more concise way.

My 2015 Programming Languages Tool Belt

2015-03-25T00:00:00+01:00

A friend of mine approached me recently with a question: "What are your current thoughts on go-to platforms for building web apps? What would you choose for a web app? Ie. app that both should work as API, admin console, some end user facing stuff, etc".

Because this is a topic I've been recently researching I find a reply to this question a fantastic way to express my current point of view. Let me first set the background and tell you where I'm coming from.

I started programming in AMOS programming language, back in the days when Amiga 600 was a masterpiece of engineering, that every kid desired. Then I used Pascal, C, x86 assembly, C++, PHP, Java, C#, Python, JavaScript. Then, 8 years ago I started using Ruby as my primary language. I loved it for many years. I don't love it any more, but I still appreciate some of its aspects.

The change to my relationship with Ruby correlates with my interest in programming languages in general, and getting familiar with the ones that showed up in the last ~decade in particular.

Let's come back to my friend's question. Even though he asked about "go-to platform for web apps" I replied with a list of languages I would consider using for building software in general these days - my programming languages tool belt. The list includes both the ones I'm familiar with and the ones I'm looking at from a close distance. It doesn't include my pre-Ruby languages though, as I either don't enjoy them anymore or I simply find them problematic these days.

Here's what I replied to him (edited):

Ruby/Rails. These days I find Ruby not really well suited to anything that's more complicated than a simple CRUD app or a reasonably simple API. I've been using it professionally for 8 years, and I've seen too many Ruby (Rails in particular) apps grow and become a mess which is extremely hard to work with and reason about. And that includes projects built and maintained by experienced software engineers too. That's in part because of the "Rails Way", which leads to achieving results quickly, and in part because of the Ruby language itself. In last few years I've seen many Ruby/Rails developers looking for solutions to build bigger things with this stack. But it requires a hell lot of a discipline and experience from the whole team. Also, by not following the "Rails Way" you loose ability to benefit from the Rails ecosystem and its plethora of ready made gems. The other part is Ruby, which is a very unpredictable and unreliable language. No proper namespaces/imports so everything is essentially global. Everything is mutable during runtime. Good luck chasing issues in your app after adding a new gem to your Gemfile which happens to monkey patch something etc. And last but not least dynamic typing doesn't "scale" in Ruby (does it scale anywhere else?). After writing several things in statically typed languages I don't trust most of Ruby code (even mine!). Every change brings anxiety and requires additional tests that check things that you wouldn't have to check if you used a language with static typing. I could go on about Ruby/Rails for hours :) But my opinion these days is that this stack is nice to build something simple very quickly. Sustainability and reliability is not its biggest strength. Job market visibility is pretty high though so it's worth having it in your skill set if you're deep into "web" apps.

Go (aka Golang). Given a fair amount of lines of code I wrote in Go I think I got to a point where I finally see its use cases clearly. It was a perfect match for git-archive-daemon and gitorious-proto (ssh and http repository access). It brought safety and speed to this critical piece of Gitorious backend. It really shines when you do systems and/or networking stuff. On the other side, it kinda sucks when it comes to modeling domain logic. It's neither OO (structs with behavior but that's it), nor functional (has closures, has functions as first class citizens, but no functional constructs), and while being statically typed it doesn't have generics which makes you write the same boring, imperative code for anything that is slightly higher level. Concurrency built into the language in the shape of channels and Goroutines makes it a fantastic fit for building robust, high-scale, multi-core, networked apps though. It's great for command line apps too, because these don't usually have lots of business logic, most often they deal with files, streams and network, and it's awesome to have a single self-sufficient binary for distribution. I'm not sure if I would build a whole "web app" (API, client facing pages, admin part) in Go. Probably not. I'd probably limit the Go part to API, but only provided that the business logic isn't very rich in that case. I haven't written an actual "web app" in Go yet, but I have a strong feeling that it is too low-level for that. I can also see risks when it comes to finding libraries that relate to other stuff than systems/networking. That's probably one of the reasons why I would be cautious when considering it for a typical web app. One thing I'd like to point out here is its learning curve. Go is one of the most easy to learn languages out there due to its explicitness, simplicity and thin syntax surface.

Clojure. It's a language I now use in place of Ruby (when I can). It solves many of the problems I have with Ruby. It has proper namespaces/imports, immutable data structures and trivial syntax (it's pretty much only an AST after all). These 3 alone put it in a more favorable position than Ruby when it comes to building something slightly bigger. It's very pragmatic - it's highly functional but it allows you to use atoms, refs and other state-keeping constructs when you really need them. It also allows you to access IO with no fuss (spit/slurp functions for example), contrary to other functional languages like Haskell, where IO is pushed to I/O Monad. This makes it a really approachable functional language. Thanks to macros it supports pattern matching (core.match), gradual typing (core.typed) and sane async programming, similar to Go (core.async). So I see Clojure as a "much better Ruby, which runs on JVM" (and I think being on JVM is a plus here). Clojure is dynamically typed just like Ruby (which also has many functional constructs), hence the comparison here. Oh, it also has ClojureScript/Om which brings sanity to building rich browser UIs \o/. One downside of Clojure is its low position on job market. It's getting better with every year though.

Haskell. I haven't used Haskell on a real-world project yet but it seems to have all the ingredients I'm looking for in a language today: functional, immutability, serious type system, powerful pattern matching, and failure handling with Maybe type. Btw, I talked to a guy on LambdaDays conf recently who owns a consulting company that began as a Rails shop and now they're moving towards building their web apps in Haskell. 6 out of 25 Ruby devs switched to full time Haskell already, and more to come. Interesting, isn't it?

Scala. This one got my attention recently again. It has more or less the same qualities I find in Haskell: functional (mostly), immutability (mostly), static typing with generics, Option (Maybe) type, powerful pattern matching. I wrote some small piece of code in Scala some time ago, but I did it in a very imperative, and not really idiomatic style back then. I'm inclined to try it out again on some side project, there are several things which put me off a bit though. It's very close to Java, so it's close to XML/SOAP/Enterprise ecosystem and type of work/projects. It also has like a half-dozen types meaning "nothing": Null, null, Nil, Nothing, None, and Unit. O_o. This may be nitpicking but it doesn't show it as a consistent and simple language IMHO. And its "syntax surface" feels to be very wide, which is something I try to avoid these days. From HR POV it's definitely not as "dramatic" as Haskell or Clojure.

Rust. I looked at it recently and I find it to be a way better designed language than Go (if we compare languages advertised as "systems language"). Functional constructs built-in, immutability, static typing with generics, Option type, powerful pattern matching, and no exception handling (solved by Option). <3 <3 <3. However, there's one thing I miss in Rust: garbage collector. I understand why it's not there. They (Mozilla) wanted systems language with predictable performance, that would replace C++. And I think it delivers on that promise. But that makes it a bit too complicated (memory is managed for you but with your help) and not well suited for building web apps. I guess I could have skipped Rust in this already long list, but I love discussing programming languages recently, sorry. Note that I compared Rust to Go in the first sentence of this paragraph, but it's only because both are called "systems" languages. In reality they seem to have different use-cases and excel in different niches.

Elixir. It's on my list of languages to have a closer look at. It builds on Erlang/OTP and as Erlang it is best suited for building scalable, fault-tolerant distributed systems. It's dynamic, functional, has pattern matching and includes several Ruby-inspired features like modules/mixins and similar syntax. Phoenix seems to be its most popular web framework. If I had to compare it to other language from this list it would probably stand somewhere near statically typed Scala with AKKA toolkit/runtime. Feel free to bash me for this comparison.

So, this list didn't really give a direct answer to my friend's question. But I hope it is still useful for him and anyone else looking for a great tool to fit for the next job. There's no single programming language that excels in everything and it's important to understand that languages are just tools. Sometimes I'm choosing multiple languages for different components of a single system (Gitorious' frontend was a Rails app, its backend was built in Go and bash). Sometimes I'm building UI as a HTML5/JS/ClojureScript app, fully decoupled from the backend. It all varies from case to case.

I hope to have lots of opportunities to try more of these "tools" in the future, finding out how well they fit specific applications, and adding the proven ones to my programming languages tool belt.

I'd love to discuss this topic more so I'm looking forward to your opinion on this!

Rails is half of your application

2014-03-13T00:00:00+01:00

Only a week ago I was still standing strong behind the "Rails is not your application" statement. It felt natural to me and I haven't bothered to think deeper about it. Since then I discussed this topic with several people and I read Domain Logic in Rails by Reed Law which mentions DHH's and Uncle Bob's differing approaches to building applications. A torrent of thoughts hit me then and I realized the truth lays somewhere in between. I started analyzing the subject deeper and this article is my attempt to explain what your application actually is and where it stands in relation to "Rails application" label.

Let's first find out what different types of code we put in the application. What does a Rails app directory contain? When you run rails new you get several directories created for you - your Rails app structure. Three most interesting directories are app, lib and config. You put your controllers to app/controller, views to app/views. You configure your routes in config/routes.rb. The business logic goes to app/models, app/services and to some other directories under app or lib. Many people still put it into controllers though.

We can group entities contained in the mentioned directories in two groups: things that can be used outside of Rails application, and things that cannot. Your domain logic, services and models fall into the first group. Controllers, views, helpers and routes fall into the second group. And no, putting entities from the second group into a Rails engine (and making a gem out of it) is not the way of re-use I'm talking about - this engine can't be used in let's say an iOS app, right? Sounds obvious, but bear with me.

What this means is that you can reuse business logic (backend) but you can't reuse the UI (frontend). You don't actually want to reuse the frontend. How likely it is that you will need your routes, controllers and views in another app? Unlikely. You just build another UI, be it a mobile, html5/js, or a desktop app. So Rails is your frontend. In fact it's a business logic delivery mechanism. You have chosen Rails to be on the frontline, to serve the most user facing duty of your business.

Let's say you actually want to build another application to use the existing business logic. We know it's possible, so you extract the logic into a gem or a separate ruby application that is an API (Sinatra or rails-api). When extracting your business logic into an API you still need to access it - thin adapter(s) used from the controllers can do the job. Now your original app becomes only the UI. A "UI application", "frontend application" or just "one of the applications using your API/gem". As a matter of fact the code that is now left in app/* means nothing outside of Rails app. Think about it. Also, what stands after run in your config.ru file? Rails::Application. It's your own flavor of it (you inherit from it) but it's still a Rails::Application. When you start a server Rails boots and your code in app/* and config/* just fills some slots in the Rails app. Not the other way around! Your code doesn't use Rails - Rails uses your code. Can we say then that Rails is your application? It may be more fair to say that Rails is your application runtime. It runs your application and provides many facilities to it. Anyway, if you don't like that idea then there are some nice new Ruby web frameworks emerging (Lotus, Pakyow).

We can look at it from a slightly different perspective. In reality most Rails apps serve both purposes. They provide the UI and domain logic at the same time, keeping both under the same source code tree. The number of Rails apps that don't implement any business logic themselves is pretty damn low. Regardless of where the business logic technically sits, we have the Rails dependent part (UI) and the Rails independent part (logic). Can we say that Rails dependent part, together with Rails itself is one half of your application and the logic part the second half? I think we can. Some developers believe that only the part of the code that is below the controllers (services and models) is "the application". Is it really the case? What's worth your "application" without the UI? What do you think?

In the next post I'm going to find out where the exact line between those parts goes (hint hint: controller) and how we can pull this line to our side to maximize benefits (where should the authorization go? where should the loading/building of models happen? Why ActiveSupport can invisibly make your domain logic Rails dependent?)

Rumors of Ruby's demise are exaggerated but...

2014-02-24T00:00:00+01:00

Rumor's of Ruby's demise are exaggerated indeed. Ruby isn't going anywhere. And that's good because it's a fantastic language for some things. For some things.

I believe main purpose of Ruby will end at running rails apps (like it is now), mostly small-to-medium apps or just frontends for some larger systems.

The truth is the lack of built in, modern concurrency support in Ruby makes it less fitting for many things. Especially the things that touch the ground of networking (long persistent connections, not necessarily http) and coordination of concurrent processes (processes in the sense of a unit of work, not OS).

Yes, you can make your Ruby code work fine at a larger scale, by optimizing it here and there, using some techniques known to highly skilled Ruby programmers, but that takes lots of effort. It's possible but just not elegant.

Let me give you an example:

I wanted to implement live streaming for Asciinema, you know, "live session sharing", for remote pairing etc, without the tmux/ssh hassle. That means (multiple) persistent connections to server from both producers and consumers (watchers). How do you do that in Ruby? You have some options but they all have some serious cons. I recently spent ~month learning Go, just for fun. And after a month I have written a simple (~130 loc) server that streams terminal session to both a browser and to a terminal based client. I can't say the code I wrote is perfect but when working on it it was really pleasurable experience and it was straightforward to achieve what I wanted.

Of course I considered building this in Ruby in the first place. But every time I started thinking about a possible solution I was finding that it's neither straightforward nor pleasurable. Ruby doesn't like persistent connections. It's awesome at serving some quick response but when you want to handle multiple, long-running clients you have to either: 1) use JRuby + thread synchronization primitives, 2) Eventmachine, 3) Celluloid.

I don't want to synchronize threads (1) nor write evented code (2). Celluloid is using fairly decent concurrency model (agents) and is closest to give me the expected productivity/pleasure ratio. But still, I can write the same amount of lines of code in Go and handle 100x more simultaneous clients using much less memory.

Having said that I'm not saying it's impossible to build more complex, distributed systems with Ruby. It's pretty much possible but you have to try harder to achieve the end result.

You may say I'm wrong here. But it is what worked for me much better and finally allowed me to write clean streaming code and solved the problem I wanted to solve without giving me the pain while thinking of it. And I'll still be happily building Asciinema website in Ruby. Right tool for the job.

YMMV.

Vimdiff support in Coloration

2012-06-09T00:00:00+02:00

Coloration just got support for Vimdiff. Now all generated Vim color schemes will include DiffAdd/DiffDelete/DiffChange/DiffText syntax groups with red, green and blue based on Tango palette.

Here's how GitHub color scheme looks in vimdiff:

And Monokai's diff mode:

Extract variable in Vim

2012-06-07T00:00:00+02:00

Given you are working on following code in Vim (cursor position is |):

for strawberry in |Strawbery.where(size: 'xxl')
  puts "Oh look, a #{strawberry.size} strawberry!"

You want to extract Strawbery.where... into a variable. Recently I've learned that typing <C-a> in insert mode inserts the characters that were typed previously in insert mode. Let's use it to extract a variable:

Cstrawberries<ESC>O<C-a> = <ESC>p

After using above key combo you end up with this:

strawberries = Strawbery.where(size: 'xxl')|
for strawberry in strawberries
  puts "Oh look, a #{strawberry.size} strawberry!"

Profit.

Scratch dir

2012-05-04T00:00:00+02:00

"AUTOMATE ALL THE THINGS!" one would say. "Time is money" other would say. "Don't make me think" another smart one would say. "Computers are our slaves and we should put as much work as we can on their circuits" I say.

I'm constantly looking for automation opportunities in my daily working habits. Today I automated one thing that was very simple but at the same time boring and fully repeatable: creating temporary "scratch directory" and cd'ing to it.

Yesterday it was like this:

I need a scratch dir...
I'll create it in.... ~/tmp/ or /tmp/ ... let it be /tmp/..
$ mkdir /tmp/ - stop, hmm, I need a name.. q then
$ mkdir /tmp/q
mkdir: cannot create directory '/tmp/q': File exists
"Damn", qq then
$ mkdir /tmp/qq
$ cd !$
Hooray! Wait, but what I need this dir for?

Today I created short shell function for zsh/bash:

function new-scratch {
  cur_dir="$HOME/scratch"
  new_dir="$HOME/tmp/scratch-`date +'%s'`"
  mkdir -p $new_dir
  ln -nfs $new_dir $cur_dir
  cd $cur_dir
  echo "New scratch dir ready for grinding ;>"
}

Thanks to above function I land in a brand new scratch dir. This dir is located in ~/tmp/ and it has unique name scratch-{TIMESTAMP}. Additionally there's symlink ~/scratch pointing to it (always to the latest scratch dir) that is handy when you need to open another terminal/shell instance in this dir.

So today when I need a scratch dir:

~/code % new-scratch
New scratch dir ready for grinding ;>
~/scratch %

One would ask: why shell function instead of shell script? Because shell scripts can't change current working directory of current (parent to them) shell. Shell functions run inside the current shell and can alter cwd.

systemd socket activation and Ruby

2012-01-21T00:00:00+01:00

For anyone who doesn't know what systemd is:

systemd is a system and service manager for Linux, compatible with SysV and LSB init scripts. systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux cgroups, supports snapshotting and restoring of the system state, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic.

www.freedesktop.org/wiki/Software/systemd

It's quite similar to Apple's launchd (used in OSX) and is fully utilizing powerful features of the latest Linux kernel. systemd is default init system in latest Fedora, openSUSE and Mandriva and is available for many other Linux distros as alternative boot solution. I hope Ubuntu's upstart team will give up soon because having systemd on Ubuntu servers would be awesome. For more info and idea behind the project I recommend reading Lennart Poettering's announcement

One of the great features of this init system is socket activation of system services. In short, services are lazily started when they're actually needed. Systemd listens on the sockets for them and starts the services on first incoming connection, passing them the listening sockets. Started services just start accepting clients on these sockets (without calling socket()+bind()+listen()).

It appears that the protocol for passing sockets to service processes is very simple. Environment variable LISTEN_PID is set to the PID of the service process and another environment variable LISTEN_FDS is set to the number of listening sockets passed. Socket descriptors start from number 3 and are sequential. For example, LISTEN_FDS with value of 2 means process should accept connections on 2 sockets with descriptors 3 and 4.

I'll show you how all this works on an example echo server written in ruby. The server will send back what it receives. Additionally it will send information telling if listening socket came from systemd or not to each new connected client.

But first we need to create the socket unit file that specifies where systemd should listen on behalf of our service. /etc/systemd/system/echo-server.socket file can look as simple as this:

[Socket]
ListenStream=8888

Next, we need service unit file that specifies what binary to start when connections start coming. /etc/systemd/system/echo-server.service file may look like this:

[Service]
ExecStart=/home/kill/.rvm/bin/ruby-1.9.2-p290 /home/kill/bin/echo-server.rb
User=kill
StandardOutput=syslog
StandardError=syslog

I have ruby 1.9.2 installed via RVM so I'm running my ruby script with RVM's wrapper specifying full paths (remember init process runs as root). I'm also setting the user on whose behalf the process should be run and I'm asking systemd to log process' stdout/stderr to syslog (simplifies debugging).

Now, the echo server (/home/kill/bin/echo-server.rb):

#!/usr/bin/env ruby

require 'socket'

SD_LISTEN_FDS_START = 3

from_systemd = false

if ENV['LISTEN_PID'].to_i == $$
  # use existing socket passed from systemd
  server_socket = Socket.for_fd(SD_LISTEN_FDS_START + 0)
  from_systemd = true
else
  # create new listening socket on port 8888
  server_socket = Socket.tcp_server_sockets(8888)
end

Socket.accept_loop(server_socket) do |client_socket, addr|
  client_socket.send("OHAI! systemd socket: #{from_systemd}\n", 0)

  while (data = client_socket.recv(1000)).size > 0
    client_socket.send(data.upcase, 0)
  end
end

Implementation is very simple, still I'm gonna explain it a little bit as it illustrates the use of systemd socket activation protocol and the fallback - normal way of creating server socket.

Like I mentioned earlier, descriptors of systemd passed sockets start with 3:

SD_LISTEN_FDS_START = 3

We check if LISTEN_PID points to our echo-server.rb process:

if ENV['LISTEN_PID'].to_i == $$

If so, we're creating new Socket instance for existing descriptor (3). Socket unit file tells systemd to listen on one port only (8888) so we can assume there's only one socket descriptor passed:

  # use existing socket passed from systemd
  server_socket = Socket.for_fd(SD_LISTEN_FDS_START + 0)

If LISTEN_PID doesn't match our process we just create TCP socket the usual way:

else
  # create new listening socket on port 8888
  server_socket = Socket.tcp_server_sockets(8888)
end

Finally, in Socket.accept_loop(server_socket) do { ... } we handle incoming clients.

Tunnelling VirtualBox guest's network traffic

2011-11-12T00:00:00+01:00

This post is quite different than my usual ones. It's not about programming at all. Usually I'm not writing about how I solve various, non-programming related problems but this time I want to share what I achieved as this may be helpful to others facing similar scenario. I couldn't find similar solution explained anywhere so I'm putting this for anyone seeking such information.

The problem

Tunnel whole network traffic of VirtualBox guest machine through remote server located in different country.

Basically what I needed was a virtual machine that was connecting to internet with London's IP address. Simple port forwarding was not an option as the machine needed to connect from many apps (not only web browser) to multiple (unknown to me) hosts and ports.

The idea

Setup SSH tunnel from my local machine (VirtualBox host) to my Linode server in London and tell VirtualBox to use this tunnel as guest VM's ethernet interface endpoint.

Seems simple. And after a few failed attempts it turned out to be really simple. You just need to know few concepts and run few commands here and there.

The solution

Below is the complete solution that worked well for me. I use following symbols for involved machines:

P - remote machine with London's IP, used as a proxy
H - local machine, VirtualBox host
G - virtual machine, VirtualBox guest, needs public IP of P

Solution is build on OpenSSH-based VPN. From waldner's post at backreference.org:

This is a poorly documented yet really useful feature of Openssh. It allows you to connect two tun/tap interfaces together, to create a layer-2 or layer-3 network between remote machines. This results in OpenVPN-like VPNs (but much simpler and, admittedly, less scalable).

With presented solution virtual machine (guest) can be running any OS: Linux, Windows, whatever.

Foreplay

You need to have root access on the machine that will work as a proxy (P). This machine needs to have eth0 interface configured with public IP address (London in my case).

The sshd running on it should allow root logins (I'll explain later why) and setting up ssh tunnels. Make sure you have following entries in /etc/ssh/sshd_config on P:

# /etc/ssh/sshd_config

PermitRootLogin yes
PermitTunnel yes

Restart sshd if needed.

Step I

First you need layer-2 (ethernet) ssh tunnel. Run following on H as root:

$ ssh -o Tunnel=ethernet -w 0:0 root@<P hostname>

replacing <P hostname> with real hostname of your proxy machine.

This will setup L2 link and create tap0 network interfaces on both machines (H and P). Because only privileged users can create network interfaces you need to run this command as root, ssh-ing to remote host as root user as well.

* 0:0 in above command specifies numbers for _tap interfaces for both machines. You can safely use 0 for both of them as you probably don't have any other existing tap interfaces._

Step II

Now you need to create NAT on the proxy machine (P). This will make traffic from tap0 interface to be seen with P's public IP address (going through its eth0 interface).

Run following on P as root:

$ echo 1 > /proc/sys/net/ipv4/ip_forward
$ /usr/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
$ /usr/sbin/iptables -A FORWARD -i eth0 -o tap0 -m state --state RELATED,ESTABLISHED -j ACCEPT
$ /usr/sbin/iptables -A FORWARD -i tap0 -o eth0 -j ACCEPT

Masquerade ready. Now bring P's tap0 interface up and configure it:

$ ifconfig tap0 up
$ ifconfig tap0 192.168.0.1 netmask 255.255.255.0

* You can use different network than 192.168.0.* for NAT if P is already attached to such network.

Step III

Next step is to bring H's tap0 interface up. Run following on local machine (H) as root:

$ ifconfig tap0 up

There is no need to assign IP to this interface as it will be directly connected to virtual machine's network interface.

Step IV

Now configure virtual machine to use the tunnel.

Open VM network settings (on H), select Bridged adapter and choose tap0 as a bridged device.

Step V

Finally start virtual machine (G) and configure its network interface.

The guest machine can be either Linux or Windows (or just anything you want). Just setup the interface to be configured as below:

IP: 192.168.0.2
Netmask: 255.255.255.0
Gateway: 192.168.0.1
DNS: 8.8.8.8 / 8.8.4.4 (use Google's ones for simplicity)

To confirm that VM (G) is visible to the world with public IP of P run:

$ curl icanhazip.com

or open icanhazip.com in the browser.

Simplifying

You can put commands from steps I, II and III in shell scripts to simplify the task in case you want to run it frequently. I have made 2 bash scripts.

First on on my local machine (H):

# vm-tunnel.sh

ssh -o Tunnel=ethernet -w 0:0 -f root@<P hostname> "~/taptap.sh"
ifconfig tap0 up

Second one on proxy machine (P) in root's home:

# /root/taptap.sh

echo 1 > /proc/sys/net/ipv4/ip_forward
/usr/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
/usr/sbin/iptables -A FORWARD -i eth0 -o tap0 -m state --state RELATED,ESTABLISHED -j ACCEPT
/usr/sbin/iptables -A FORWARD -i tap0 -o eth0 -j ACCEPT

ifconfig tap0 up
ifconfig tap0 192.168.0.1 netmask 255.255.255.0

Thanks to these 2 scripts I can summon my VPN with one command:

$ sudo vm-tunnel.sh

My EuRuKo 2010 talk

2011-10-23T00:00:00+02:00

I was speaking at EuRuKo conference last year (2010) and I'm just putting this here for archiving purposes.

Here's a video of the talk:

And here are the (slightly updated) slides:

Coloration + terminal Vim + 256 colors

2011-10-22T00:00:00+02:00

Year has passed since last Coloration update. Time has come for new improvements. Brand new Coloration v0.3.1 focuses on Vim theme writer and includes:

Line numbering (LineNr) background matching cursor line highlight style (CursorLine)
New style for ColorColumn also matching cursor line highlight
Highlighting for terminal Vim running in 256 colors capable terminal

Let's focus on the last one.

Nowadays most terminal emulators support 256 colors. Additionally to 16 base colors everyone knows about they also support 216 colors from 6x6x6 color cube and 24 shades of grey. Look here for details. If you don't use 256 colors capable terminal then start using it now. xterm had it in 1999. gnome-terminal, konsole, iTerm2 and many more have it.

Vim colorschemes allows you to specify ctermfb and ctermbg from the range 0-255. People usually only use 0-15 in their themes in order to be compatible with 16-color-my-grandpa-uses terminals. That's so wrong! Coloration is fast-forwarding us to the future with it's updated Vim theme writer. It converts the colors used by the gui Vim version (GVim/MacVim) to their xterm256 nearest equivalents with simple approximation.

Sunburst theme converted from Textmate theme:

Twilight theme converted from Textmate theme:

Now go convert your old, dusty Textmate themes. Oh, and I've put Sunburst and Monokai themes on github (vim-sunburst, vim-monokai) for all you lazy guys.

Better Rails production box

2011-09-26T00:00:00+02:00

Here are 3 simple things to do in order to improve deployment process of your Rails application and friendliness of your production server. Nothing new here but many good practices are often overlooked even by experienced developers.

Set RAILS_ENV in .bash_profile

I see my fellow devs typing this on production server all the time:

script/rails console production

Or this:

RAILS_ENV=production rake some_task

It's obvious that on production box you want it to run in production environment, right? Set RAILS\ENV_ in .bash\profile_ and save your fingers:

# ~/.bash_profile
export RAILS_ENV="production"

This might not be the case if your hosting service is like Heroku or similar as they export RAILS\ENV_ for you automatically. But if you're on self-managed VPS or EC2 instance you can ease your work with this trivial setup.

Use logrotate

Create /etc/logrotate.d/rails\logs_ file with following content:

/var/www/*/shared/log/*.log {
  daily
  missingok
  rotate 30
  compress
  delaycompress
  copytruncate
}

That will tell logrotate to rotate log files daily, compress them, keep last 30 days and don't choke when file is missing. copytruncate is important here as it will make sure log file currently used by Rails app is not moved but truncated. That way the app can just keep on logging without reopening log file.

Don't forget about this one if you manage production box yourself. And do it when you initially setup the box, not "later". "Later" often means "when app is down due to not enough disk space". Srsly.

Use maintenance page

When deploying with long running or non-trivial (more than add column) migrations you should use maintenance page of some sort. With capistrano you can just use (before running migrations):

cap production deploy:web:disable

and (after they finish):

cap production deploy:web:enable

Default maintenance page put by capistrano is kind of ugly so you should make your own matching your site design. In order to do this prepare app/views/layouts/maintenance.html.erb and override deploy:web:disable task in config/deploy.rb:

namespace :deploy do
  namespace :web do
    task :disable, :roles => :web do
      require 'erb'
      on_rollback { run "rm #{shared_path}/system/maintenance.html" }

      reason = ENV['REASON']
      deadline = ENV['UNTIL']
      template = File.read('app/views/layouts/maintenance.html.erb')
      page = ERB.new(template).result(binding)

      put page, "#{shared_path}/system/maintenance.html", :mode => 0644
    end
  end
end

That will put your custom page in #{shared\path}/system/maintenance.html_ (also accessible via public/system/maintenance.html by webserver).

On the other end, you should configure webserver to respect presence of this file. Here is config snippet for Nginx:

server {
  listen 80;
  server_name example.org;

  ...

  # Maintenance page support

  set $maintenance 0;

  if (-f $document_root/system/maintenance.html) {
    set $maintenance 1;
  }

  if ($request_uri ~* (jpg|jpeg|gif|png|js|css)$) {
    set $maintenance 0;
  }

  if ($maintenance) {
    return 503;
  }

  error_page 503 @maintenance;
  location @maintenance {
    rewrite ^(.*)$ /system/maintenance.html break;
  }
}

It will make sure that when maintenance.html file is there Nginx will serve it and return 503 (Service unavailable) HTTP status. Proper response status is important here as it gives search engines the message: don't index me now please, come back later. You don't want your maintenance page get indexed, do you?

Above Nginx config also allows assets to be served as usual when maintenance mode is on - especially useful if you don't host them on separate host/subdomain and you want the page to look nice and match presence of your full site.

Formatting XML in Vim with indent command

2011-09-08T00:00:00+02:00

Today I had a need to look at XML doc fetched from Google Calendar API. I saved it to a file and opened in Vim. Unfortunately API output was generated to be consumed by machines rather than humans.

First I tried gg=G command. = is used to auto-indent selected line(s) and gg=G re-indents whole file. Usually it works great, especially for source code files. It does not reformat code, it only changes indentation. And that's good, I should be the one to control look of my code. But for XML I want it to do full reformatting. I'm not writing XML and in all of the cases when I open such docs in my editor I only want to look at well-formatted, human-readable XML.

From Vim help on equalprg setting:

External program to use for "=" command. When this option is empty the internal formatting functions are used; either 'lisp', 'cindent' or 'indentexpr'. When Vim was compiled without internal formatting, the "indent" program is used. ...

"Bingo! Let's use xmllint for that!" - I thought immediately. xmllint command comes bundled with libxml package on Unix-like systems and does really good job at producing pretty output. You can use it like this:

# shell
xmllint --format --recover foo.xml

Okee, so my first approach to reformatting XML in Vim was:

:1,$!xmllint --format --recover - 2>/dev/null

Not bad. But writing it every time (or remembering) would be painful. Let's use mentioned equalprg option:

# .vimrc
set equalprg=xmllint\ --format\ --recover\ -\ 2>/dev/null

Ok... but Y U USE XMLLINT WHEN I'M INDENTIN' MY RUBY CODE?? Ahaa! equalprg need to be set locally only for XML-type buffers. Autocommand did the trick:

# .vimrc
au FileType xml setlocal equalprg=xmllint\ --format\ --recover\ -\ 2>/dev/null

Restarted Vim, typed gg=G and said "Hell yeah!".

Bitpocket as a Dropbox alternative

2011-07-18T00:00:00+02:00

As an excuse for trying out Posterous email posting and its Autopost feature (in this case to Twitter and Identi.ca) I'll present you Bitpocket.

In short, Bitpocket is a small but smart bash script that does two-way directory synchronization resembling Dropbox sync. Simply it just uses rsync to make the actual sync. It runs rsync twice: first syncing from remote to local machine, then from local to remote machine. This way all new files that appeared on remote are fetched to local machine and all new locally created files are replicated on remote machine.

But additionally it makes sure that file deletion is properly propagated in this 2-way synchronization, which isn't possible by just running rsync twice. The problem is rsync deletes all new created files or brings back the files you deleted depending on which direction we sync in first (and when using --delete option). Bitpocket solves that by tracking names of created and deleted files between its invocations and using these lists as a source for --exclude-from rsync option when doing first, remote -> local sync.

To make sure Bitpocket behaves like I wanted it to behave I've spec'ed it. Yes, it's a bash script, not a ruby code, but who said I can't use rspec to test a bash script?

For details about usage see Bitpocket's README. I'm using it instead of Dropbox for few weeks now and it does quite good job. Give it a try.

Resque Mailer says: Put your emails to background

2011-04-05T00:00:00+02:00

It's no secret that handling HTTP requests should be quick. Very quick. Your business' "to be or not to be" may depend on how "fast" is you site. "Fast" site for end-users is when they're clicking and they don't need to switch to other tabs to kill time while waiting for page to open.

How to make your site fast?

buy more RAM (and install Oracle) - it costs $$$
lazy load what you can after initial page - complicates the code
do any processing in background - that one is quick win!

Depending on the application there will always be need for some combination of above. Best practice however, is: Webserver should not do any heavy data processing when handling request. It should just process the request and return response as soon as possible.

It means that all you can do is:

parsing params
loading resource(s)
changing state of resource(s)
utilizing background processing workers for data grinding
rendering response body from template
returning response to client

All of the above steps except last one are optional but they're all usually required in most apps. Point 4 is the important here. Developers often forget about it or are just too lazy to implement it. But it makes huge difference! Waiting 10 seconds for page load because the controller needs to finish something... it's unacceptable.

Resque to the rescue! Very reliable, framework independent, fast "Redis-backed library for creating background jobs" by Github's defunkt. I use various Ruby web-frameworks to build my webapps, the choice is not always Rails but Resque can be used with anything. Good stuff.

Anyway, what is not that common is sending e-mails asynchronously. Yeah, some people never thought about it. But sending them right from controller costs. Building mail message and sending it involves creating lots of ruby objects, rendering templates, preparing mail headers and eventually passing it to local MTA or sometimes to remote SMTP server. Sounds like a lot of time, especially when you use remote SMTP server.

But who would want to create dedicated background worker for this task? Who would want to enqueue such a job instead of doing simple Notifications.signup(@user).deliver in the controller? Not me.

Enter ResqueMailer. Originally created for Rails 2 by Nick Plante, updated by me to work with both Rails 2.x and 3.x. ResqueMailer allows you to move processing of Rails mailers out of controller to an async Resque worker with minimal fuss.

Just put the gem into your Rails project Gemfile:

gem 'resque_mailer'

and then include the Resque::Mailer module into your Mailer class:

class Notifications < ActionMailer::Base
  include Resque::Mailer

  def signup(...)
    ...
end

Instaaant gratification!

This is it. Now start your worker and enjoy faster responses. For information on ResqueMailer's optional settings see project's github repository. For details about setting up Resque and running workers go to README of Resque project.

Github's Gist and Gnome Keyring

2011-04-03T00:00:00+02:00

I like to keep my dotfiles in git repository but I never put my .gitconfig there because it included my GitHub API token, that is used for example by gist. Git allows you to get output of a system commands as values (by prepending value with exclamation mark) for settings in [alias] section of .gitconfig only but I just found out that gist script can also do this trick for github.token setting. Thanks to that we can prepare some script that gets token from different place, use it in .gitconfig, put config into repository and worry no more about publishing sensitive data.

I put the token in Gnome Keyring as it feels pretty secure. Just go to System -> Preferences -> Passwords and Encryption Keys, press Ctrl+N, choose Stored password. Select login keyring, enter "GitHub API Token" as description and your token as password. Now we need a way to get this token out of keyring in shell script. Fortunately there are python language bindings for Gnome Keyring.

Python script for retrieving passwords from keyring can look like this:

#!/usr/bin/env python

import sys
import gnomekeyring as gk

if len(sys.argv) > 2:
    ring_name = sys.argv[2]
else:
    ring_name = 'login'

for key in gk.list_item_ids_sync(ring_name):
    item = gk.item_get_info_sync(ring_name, key)
    if item.get_display_name() == sys.argv[1]:
        sys.stdout.write(item.get_secret())
        break

Simple. It prints out password with given name to standard output, without newline character (so it's easier for other scripts to use it). Let's save it to ~/bin/keyring-get-pass.py and try it:

$ python ~/bin/keyring-get-pass.py 'GitHub API Token'
my-secret-token-i-wont-show-you$

Cool. By default we get secrets from login keyring. This is for convenience as login keyring is being unlocked at system login time on Gnome (at least on Ubuntu) and it won't ask us to unlock it when running this script. If we need to get password from different keyring then its name can be passed as the second argument to the script.

Now, let's use the script in .gitconfig:

[github]
  user = sickill
  token = !python ~/bin/keyring-get-pass.py 'GitHub API Token'

If you have other solutions for avoiding publishing passwords and tokens in your dotfiles (like config templates etc) tell me, I'm eager to hear!

Open3 And The PID Of The Spawn

2010-09-18T00:00:00+02:00

Open3 is a standard way of starting subprocess from ruby if you need IO objects for stdin, stdout and/or sterr. It goes like this:

require "open3"

stdin, stdout, stderr = Open3.popen3('sort')
stdin.puts "oso de peluche"
stdin.puts "del ratón"
stdin.close
while !stdout.eof?
  puts stdout.readline
end

Simple. I needed one more thing: PID of the spawned process to be able to send it some SIGNALs. After looking at open3 docs I realized there is no way to get the PID. Several posts like this one made me believe that open4 is the only way to go.

But I didn't gave up. I didn't want to add open4 as a dependency of my code only because of this small shortcoming. After a while of googling I found open3 documentation for ruby 1.9. How it's different?

"Open stdin, stdout, and stderr streams and start external executable. In addition, a thread for waiting the started process is noticed. The thread has a thread variable :pid which is the pid of the started process."

Hurray!

stdin, stdout, stderr, wait_thr = Open3.popen3(cmd)
pid = wait_thr[:pid]  # pid of the started process.
Process.kill("USR1", pid)  # send it USR1 signal

So MRI 1.9 solves the problem by returning additional element with reference to waiting thread with :pid variable set on it. How about other ruby implementations?

JRuby doesn't return waiting thread reference from open3() call but it has open4 incorporated into its standard library in the shape of IO.popen4 method:

pid, stdin, stdout, stderr = IO.popen4(cmd)

Unfortunately, both MRI 1.8.x and Rubinius don't include this functionality in their standard lib.

So, is open4 the only reliable way? Let's see:

MRI 1.8.7:

ruby-1.8.7-p249 > open4 "cat"
 => [31373, #<IO:0xb741c8ac>, #<IO:0xb741c870>, #<IO:0xb741c7f8>]

MRI 1.9.2:

ruby-1.9.2-p0 > open4 "cat"
 => [31498, #<IO:fd 4>, #<IO:fd 5>, #<IO:fd 7>]

JRuby:

jruby-1.5.2 > open4 "cat"
NotImplementedError: fork is unsafe and disabled by default on JRuby
        from /home/kill/.rvm/gems/jruby-1.5.2/gems/open4-1.0.1/lib/open4.rb:23:in `popen4'

Rubinius:

rbx-1.0.1-20100603 > open4 "cat"
 => [31635, #<IO:0x212>, #<IO:0x214>, #<IO:0x216>]

Damn, looks like we cannot use open4 under JRuby. Fortunately, there is IO.popen4 which should be used instead. Solution that will work in all ruby implementations might look like this:

if IO.respond_to?(:popen4)
  def open4(*args)
    IO.popen4(*args)
  end
else
  require 'open4'
end

open4("cat")

Summarizing, go with open4/IO.popen4 if you want to be sure your code works on all ruby implementations.

Announcing Coloration, editor color scheme converter

2010-07-24T00:00:00+02:00

Without further ado I'm introducing Coloration - editor/IDE color scheme converter. It is an evolution of tm2jed tool and at the moment it can convert Textmate color themes (in XML plist format) to Vim, JEdit and Kate/KWrite/KDevelop color schemes. So if you are Textmate->Vim convert or you just envy Textmate users for their good looking, dark themes now you have no excuse to not try out Coloration.

Here's how Vim with Sunburst theme looks like:

If you want to give Coloration a try you have two options. Either you can use online version at coloration.sickill.net or you can install ruby gem coloration:

gem install coloration

It will give you tm2vim, tm2jedit and tm2katepart commands. Note it requires ruby 1.9.

Let me know if you find it useful. Source code is available at github.com/sickill/coloration.

Enjoy!