Chained Experiments

Process supervision

2012-02-05T17:00:00Z

If you ever had to administer some servers you necessarily stumbled upon this problem:
How can I make sure all my applications are running and restarted on crash ?

There are three different schools I know of for doing this:

My applications never crashs

let's call this one the silly man, you are sure your application will never ever crash and so a basic init script will work for you but if something fails you will probably never know until a client calls your support.

In this category we have the init system used on many linux/bsd hosts, they vary in execution but the idea stays the same: you have a script, when called with start as argument it starts the application and when called with stop it stops it.
This approach works well if you are sure the applications will behave well no matter what and cannot crash (or have their how supervisor mechanism).

Polling (by PIDs)

The second way of doing it is to have a dedicated application watching process on a regular basis using their pids, for me the problem with this approach is the lag between the time the process crash and the time the external supervisor notices that the application is not running.

Some seconds can look ridiculous but it is not even with a low traffic system especially with persistant connections.

In this category we have (These are the main one I know, I am sure there is a lot more):

monit
god (ruby application)
bluepill (ruby too)

The first one is a C application and works more or less but I dropped it the first day I noticed it did not restart a process as I requested it to.

The other two have an incredibly horrible syntax to configure what you want to monitor so I never really tried any of them but I am not really fond of the way those three do things anyway.

Subprocesses

Daemontools

The last way of supervising processes is to spawn them as child of the supervisor, I used daemontools until now and it works really well. If one of your application crash daemontools is notified right away and can restart the process without delay.
The things I always missed with daemontools is the ability to control (with something else than command line) and be notified of what actually happens and the lack of resources usage monitoring (if you want to to restart a process taking 100% for too much time).

Daemontools have another particularity: it supposes your daemon will not fork in the background and output its logs to stdout so they can be piped to a dedicated process writing and rotating them on disk.

I really love to consider a daemon that way for many reasons:

since you do not need to go in the background you can run your application in production the same way as in development plus you don't need to invest time figuring out how to go in the background.
no need for any specialized logger class/object, just write on stdout and you are good to go, the default process for handling logs with daemontools can add the timestamp in front of the lines you write on stdout so you can even remove that from your code.
no need to write a pid file anywhere, this is not that the task is hard but it is one less thing you have to worry about.
the daemontools logger process can guarantee you that the space taken by the log files will never exceed what you allow (X files of Y bytes) unlike syslog (newsyslog to be precise) for example which only do checks on a regular basis and cannot prevent an application to flood your disk (at least not the syslog installed by default on most distributions).

Supervisor

In this category I only had daemontools for a long time but I recently stumbled on another interesting alternative: supervisor.

it does most if not all the things described above but add some nice ones:

an XLM-RPC interface allowing full control over supervisor from another process.
you can register notification process which will be notified of any state change for your applications (start, stop, restart).
you can use the registration functionality to register specialized process monitoring the resources used by each application and act on it with the XML-RPC interface
process have more than one state, if a process crash on start it will be put aside and it will try to restart it later instead of burning the cpu like daemontools does.

I am just in testing phases for supervisord currently but I have high hopes for it.

If you have other interesting supervision application I would gladly hear about them if they have interesting options to offer.

On fibers and threads

2011-09-25T09:00:00Z

Now that ruby 1.9 is gaining more attention and more and more people are saying goodbye to ruby 1.8 to welcome ruby 1.9.2 (don't even try 1.9.1 the latest patch release does not even work correctly...) they rediscover the new concurrency tool 1.9 gave us: the Fiber.

So what is it all about and why can it be useful ?

Threads

First let's start with some facts about ruby 1.9 threads:

Ruby threads are now mapped 1:1 with a native os threads, it may sounds great but not that much because of the evil Global Interpreter Lock which guarantee only one thread will ever have the hand so no real concurrency (in short the garbage collector was not designed to handle multithreading so the GIL allows it to work).

The problem is that while you not gain real concurrency you still have to deal with the pain associated with threads in any language, specifically your thread can be put to sleep anytime and another thread will start/continue its job.

Here is an example of such behavior:

require 'thread'

MUTEX = Mutex.new

def msg(str)
  MUTEX.synchronize { puts str }
end

th1 = Thread.new do
  100.times {|n| msg "[Thread 1] Tick #{n}" }
end


th2 = Thread.new do
  100.times {|n| msg "[Thread 2] Tick #{n}" }
end


th1.join
th2.join

Just paste this code in a text file an execute it with ruby, what you will see depends on many factors but the sure thing is that you will see tangled lines starting with Thread 1 and Thread 2 which means that none of thread did is job in one go, here is the result on my computer (I only kept the context changes):

[Thread 1] Tick 0
[...]
[Thread 1] Tick 67
[Thread 2] Tick 0
[...]
[Thread 2] Tick 99
[Thread 1] Tick 70
[...]
[Thread 1] Tick 99

This example just use the standard input (and have to use a mutex to keep both threads to write at the same time) but imagine what it can be with concurrent database queries...

I did not dug really far in ruby sources to see how threads are implemented so this explanation my be slightly off but my understanding so far is that a ruby thread is given a fixed amount of time to do its work, if in the given amount it is not finished the scheduler suspends it and wakeup another thread waiting in the queue, it goes on and on until all threads are done or the program is stopped.

Fibers

So now that we saw what Threads had to offer let's talk about their counterpart: the Fiber, first I want to say I have no idea how fibers are implemented but it does not change the validity of the following.

You can think of fibers as threads without automatic scheduling, when a fiber is running nothing else is, this is a core concept and a really important one indeed but we will talk more about this later,
let's see an example before that:

def msg(str)
  puts str
end

fb1 = Fiber.new do
  100.times {|n| msg "[Fiber 1] Tick #{n}" }
end

fb2 = Fiber.new do
  100.times {|n| msg "[Fiber 2] Tick #{n}" }
end

fb1.resume
fb2.resume

What I did was just convert the thread example from before to fibers, now the result is really different and most importantly does not depends on external factors like threads do, when executing this example here is what you will get on any machine:

[Fiber 1] Tick 0
[...]
[Fiber 1] Tick 99
[Fiber 2] Tick 0
[...]
[Fiber 2] Tick 99

When we called resume on the first fiber it started its job and completed it, then the next line was executed and the second fiber was started and finished its job before the program completed, no magic, no random, just what you wrote !

Now there are things you should know about fibers, the power you have on them comes at a price:
You need to be sure than none of your fiber will block the whole application, if you are making a server doing heavy computation in ruby on each request you can forget fibers as only one client will be served at once but if you are working on a server working with I/O typically the network and one or more databases then you are good to go (with proper 1.9 drivers).

There is also some calls to avoid in the standard library, the first to come to mind is "sleep",
here is an example of the problem with it:

require 'fiber'

fb1 = Fiber.new do
  puts "[Fiber 1] Started"
  sleep 2
  puts "[Fiber 1] Completed"
end

fb2 = Fiber.new do
  10.times {|n| puts "[Fiber 2] tick #{n}" }
end

fb1.resume
fb2.resume

This program will sleep 2 seconds and then output 10 times a message, we will see in the next section how EventMachine can help us with this specific case, here is the output for this one:

[Fiber 1] Started
[Fiber 1] Completed
[Fiber 2] tick 0
[...]
[Fiber 2] tick 9

EventMachine

Short introduction

If you already know what EventMachine is you can jump to the next section.

When you write a standard ruby application your program once executed will run from the start to the end in a linear way, now this is where EventMachine slightly change things: when you execute an EventMachine based ruby application you are no longer in charge of the "main loop" which is now C code, instead you register for events and do action based on those events.
A simple example would be a console program: when you are waiting for user input the application is doing nothing and that would be where the EventMachine main loop is, then when a user type something a callback in your application is called with the text, after you handled it the EventMachine main loop takes back control.

How can it help us

Fibers power are best put to use in an asynchronous environment, in ruby we have the great EventMachine but working in asynchronous mode is a real pain, here is an example from the em-http-request gem:

require 'eventmachine'
require 'em-http-request'

EM.run do
  http = EM::HttpRequest.new('http://google.com/').get(
      :query => {'keyname' => 'value'}
    )

  http.errback { p 'Uh oh'; EM.stop }
  http.callback do
    p http.response_header.status
    p http.response_header
    p http.response

    EM.stop
  end

  puts "Done."
end

The callback and errback block you provide will be executed respectively when a result or an error arrives and the execution will continue so "Done." will be displayed before any of the two blocks.
We can use Threads/Fibers to transform any asynchronous code path to match the synchronous code path we are used to work with, here is an example:

require 'fiber'
require 'eventmachine'
require 'em-http-request'

EM.run do
  Fiber.new do
    fb = Fiber.current
    http = EM::HttpRequest.new('http://google.com/').get(
        :query => {'keyname' => 'value'}
      )

    http.errback  { fb.resume }
    http.callback { fb.resume }

    # suspend the fiber
    Fiber.yield

    if http.error
      puts 'An error occured, damn !'
    else
      p http.response_header.status
      p http.response_header
      p http.response
    end

    EM::stop()
  end.resume
end

See what we did ? The program execution is now linear in the fiber, this code may run a little slower than the asynchronous code I never did any serious benchmarks but the program is so much easier to write this way that you will see immediate gains right away !

Note that the same can be done with threads

require 'eventmachine'
require 'em-http-request'

EM.run do
  Thread.new do
    th = Thread.current
    http = EM::HttpRequest.new('http://google.com/').get(
        :query => {'keyname' => 'value'}
      )

    http.errback  { th.wakeup }
    http.callback { th.wakeup }

    # suspend the thread until explicit
    # wakeup
    sleep

    if http.error
      puts 'An error occured, damn !'
    else
      p http.response_header.status
      p http.response_header
      p http.response
    end

    EM::stop()
  end
end

Fiber have another limitation though, they only have access to a limited stack size (4Kb in 1.9.2), while you may use fibers without even noticing there is a catch: rails 3.1 which hit stable release recently can hit the boundaries of a fiber stack (see my Rails 3.1 test application) resulting in a stack overflow error and the sad thing is that there nothing you can really do about this if you are not running your own server with a modified ruby interpreter extending the stack.

A last example to show you how to do a sleep in an EventMachine fibered application:

require 'fiber'
require 'eventmachine'

def em_sleep(n)
  fb = Fiber.current
  EM::add_timer(n){ fb.resume }
  Fiber.yield
end

EM::run do
  fb1 = Fiber.new do
    puts "[Fiber 1] Started"
    em_sleep 2
    puts "[Fiber 1] Completed"
    EM::stop()
  end

  fb2 = Fiber.new do
    10.times {|n| puts "[Fiber 2] tick #{n}" }
  end

  fb1.resume
  fb2.resume
end

And here is the output:

[Fiber 1] Started
[Fiber 2] tick 0
[...]
[Fiber 2] tick 9
[Fiber 1] Completed

One last thing to know about fibers is that you can only one running as I already said but this limitation is per thread so you could have more fibers running but I see little use fot this case since you get get back to the same problems you would have with threads.

My experience in this field

I designed and implemented the core and network library of my current company's ruby servers, these servers form a telecommunication platform and interface themselves with an heavy client on user's computers as well as an Asterisk server, only one out of now 4/5 application servers is a Ruby On Rails application the others are headless EventMachine servers communicating with each other.

When development started ruby 1.9 was not yet released and so I started the work on 1.8 with threads and then later switched to 1.9 + fibers to get away from the thread concurrency hell, our productivity raised by a great factor and the overall performances too (faster queries) but ruby 1.9 by itself is faster than 1.8 so both implementations cannot be fairly compared.

Interesting links

Building a Rack Middleware with sinatra

2011-08-12T12:00:00Z

I recently had the need to separate some logic from my main web application, after thinking about it for a while (months actually) I finally got an illumination: what about rack ?
After a quick test aside of my project I confirmed that building a rack middleware is really easy ( even easier that I thought ), this article is a quick tutorial to build a simple middleware.

What is rack ?

Rack is a library used in the Ruby to standardize the interactions between the different frameworks we have and the web server themselves, thanks to Rack we can now build part of an application in sinatra, another in rails, etc...
Rack is a really nice piece of software but I doubt many out there really used it directly, I knew it for quite a long time now but never really felt the need to dig in it.

You can learn more about Rack rack

Our middleware

The middleware will simply serve an url with sinatra, I consider this simple enough to get anyone interested started, based on this you can do nearly anything.
The middleware will also accept parameters since I had to do some search to found out how to do it.

require 'rack'
require 'sinatra/base'

class AboutApp < Sinatra::Base
  def initialize(app, opts)
    @name = opts.delete(:name)
  end

  get '/about' do
    "Hello, my name is #{@str} !"
  end
end

And here is a config.ru file showing how to use it:

require 'about_app'

# this class will act as our main application
class MyApp < Sinatra::Base
  get '/test' do
    "Test it yourself !"
  end
end


use AboutApp, :name => "Julien"
run MyApp

And that is really all you need, now you can run your application with any rack compliant web server which basically means you can run it on every web server supporting ruby ;)

For example:

$ unicorn

Doing things differently: Erlang

2011-06-04T12:00:00Z

Since I discovered RabbitMQ I kept an eye on Erlang but never looked past what is at first look an horrible syntax. I recently watched some presentations and read more about it to the point where I decided to take a dive: I started reading tutorials and even bought two books, with that I was ready to start.

I was never a fan of writing useless test applications which will end up in the trash right when you finish it, what I do instead is that I always have a folder full of what would be best described as small tests and the way I use this is every time I am not 100% sure of how to do something or I do not know how to do it at all I start a new file (or reopen an existing one) and start a minimal application which is usually 5 to 10 lines.

I do this in ruby a lot and the nature of the language itself really makes it easy, see for yourself:

create a new document in textmate (Apple + N)
write code
run it (Apple + R)

While it is a little harder in Erlang I still use the same process which do wonders for me.

So, aside from these tests I still needed a real project to work on, luckily I had one in stock: I wanted to build a monitoring server similar to what collectd does but I wanted to try a different approach and I kept this ideas for months since I had troubles deciding which language I wanted to write it in.
I will not speak about this application here but the project is already started and the basic parts are working (the project is on my github account and is named collector if you want to check it out).

The goal here is not to make another tutorial, there are enough already on the web, what I will do now is give you an overview of erlang features and you knowns, maybe makes you want to know more about it.

The way Erlang is built is really different from what I used until now, the main difference is obviously the language itself but there are concepts built into Erlang which are too a great change, the syntax is similar to prolog and knowing prolog surely helped me understand the erlang syntax faster.

We will now see some of the specific features of Erlang.

Processes

In Erlang everything is run in what is called a process, this was really disturbing at first because the "process" term in Erlang have really nothing to do with system processes and unlike them you can have thousands or even millions of Erlang processes running judging by some articles/presentations I saw.
I did not dug that too much but Erlang process seems to be code blocks distributed on a poll of threads.

You use them like you would use threads in C/Java/Ruby except they are not threads.

Actor model

Each Erlang process has only one way to communicate with the others (that is not entirely true but let's assume it is for now) and that is by passing a message to another process.
That is a powerful model which is part of the language, I will speak more about it later.

Virtual Machine

That is the first time I see what we call a virtual machine looks like an actual virtual machine ! In Erlang your can run more than one application on a given vm or even connect to it to run commands on a live system.

Each running vm is called a node, you can connect multiple nodes to create a network which nearly acts like a single computer making concurrent programming part of the language (both for multiple cpu cores and multiple machines) and not something you have to specifically think about.

Immutability

That is another disturbing things when you first try to understand the language: once a "variable" (they are not really variable since they cannot change) is assigned a value you cannot assign another value to it. This really looks like a limitation at first but when used with pattern matching it is in fact really powerful.

fail() ->
  A = 2,
  A = A + 4. % error !

work() ->
  A = 2,
  B = A + 4. % it works

Pattern matching

It is quite hard to wrap your mind around this one when not used to it but once you start understanding it you can really appreciates it.

connect(A) ->
  {ok, Result} = mod:do_something(A),
  Result + 2.

This piece of code looks pretty simple but what happens behind is that the function do_something has to return a tuple (similar to a list but used to group things together, sort of a C structure or an object) and the first element has to be "ok" (which is an atom here, they are similar to the ruby symbols).

This means that if the function does not return a tuple or if the tuple returned does not has ok as its first value this code will raise an error and crash.

As A C/Java/Ruby developer it is frightening when you realize that the process can crash anywhere and our reflex would be to test if an error occurred but this is not how it works in Erlang (C is especially horrible with the requirement to test the return value of EACH function you call which is a huge waste of time).

Error Handling strategy

In Erlang all your work is separated in processes, as we saw there may be a lot of them each doing their work and optionally dying when finished, this is how you create a process in Erlang:

start() ->
  spawn(func() ->
    do_something()
  end).

In the previous code the function do_something() will be executed in a separate process from the start function and the start function will end. But there is another way to start a process !
If instead of using spawn you use spawn_link the process you create will be linked to the current one and if an error occurs in the spawned process a message will be sent to this process telling it that the process exited with the reason as an argument (which may just be that it finished its work but it may also be that an error occured).

The way you handle errors in erlang is exactly that, instead of trying to handle everything that could go wrong with your code you just write for the better and separate your code in different logical parts, then you add a supervisor which is simply a process monitoring the other processes and which will restart them on error (if you ask it to).

That is something I really do appreciate, each time I write C code it really makes me sad having to test everything... Not only it slows me down when writing code but it create a huge amount of code when in fact your code do not do that much.

Let's take a real world use case: You want to build a TCP Server, here is what you might write in C
(there is more function calls in C but that is not the point here):

void start( int port )
{
  int s, l, client;

  s = socket(...);
  if( s > 0 ){
    l = listen(s, 5);
    if( l != 0 ){
      client = accept(s, ...);
      if( client != -1 ){
        send(s, "Hello !");
      }
      else {
        // handle error
      }
    }
    else {
      // handle the error
    }
  }
  else {
    // handle the error
  }
}

And in Erlang:

start(Port) ->
  {ok, Listen} = gen_tcp:listen(Port, [])
  {ok, Socket} = gen_tcp:accept(Listen),
  gen_tcp:send(Socket, "Hello !").

In a real application the start function will be ran under a supervisor, if the listen of accept call fails the process will die and will be restarted ! Coding this way is really nice, you can concentrate on your real goal instead of keeping in mind the worst possible things that could happen.

I do not say you can completely ignore them but I found than when coding this way you can put them aside.

Distributed computing

The last thing I want to mention is how easy an application can be distributed across cpu cores or even physical machines, the vm knowns how to uses all the cpu on the server running it and you can link multiple virtual machines together to form a network of nodes (each node being an erlang vm running on a different computer) in which a process can be spawned on any node.

Since spawn_link works from a node to another too you can have supervisors monitoring process on multiple nodes allowing you to easily failover case where a process is spawned on node2 to replace the one on node2 you just lost because the machine is down.

This was just was an overview of the features of this language, if you want to lean more about it here is some resources which helped me learning it:

Presentation from Joe Armstrong, one of Erlang creators

The two books I read after finding good reviews about them:
Programming Erlang written by Joe Armstrong
Erlang from O'reilly

Both books obviously have common parts but the second cover things that were only mentionned in the first one, If you should only buy one I consider Programming Erlang.

And lastly here is a nice website with tutorials:
learn you some Erlang

Ruby Application Monitoring

2011-05-19T20:30:00Z

When any of your applications starts to really fly by themselves you are bound to reach a point where you want/need to know what happen inside it and translate that into graphs you can show and monitor to check the health of the system.

Such a system include many parts that must work together to bring you the data you want and/or want to show and that is where I faced my first wall.

What are those parts ?

Producer

Your applications can produce data you want to store and view immediately or later, a general purpose probe can also extract useful metrics from a physical server like the cpu usage, the load, memory, etc...

Aggregator / Router

This node is the one receiving the data and deciding what to do with them, whether it will be stored on a local disk in rrd or send on the network to another application and whether to use the data as is or to transform them before.

Storage Engine

I decided to separate this one but most of the time it will be a subpart of the Aggregator / Router. The storage will determine how and where your data will be stored, most of the time it will be rrd since it works and there are not real alternatives out there.
While the rrd format itself is nice I am not really too fond of the rrd library itself, the api is really ugly and always force me to do things I do not want to do whenever I need to work with it.

Graph drawer

That is the last part and will surely be the most important since whatever energy/dedication you put in setting up the other parts it is that one anyone will face and that is also one of my biggest problem...

Choosing the right parts

There are some beast out there that are heavily used when you start speaking about monitoring you are nearly forced to hit them (or get it by them), the name I hear the most are: Nagios, Munin, Cacti, MRTG, SmokePing.

There a multiple problems for me with those systems, the first one is that some are more or less focused on a specific need (like SmokePing) and making them play together can be really "fun", the kind of fun nobody wants.
The other problem I faced is that they tend to want control over the whole monitoring system, they want you to use their producers module, their graph drawer, their router and if you want to step aside and change one part your are doomed.
It may have changed since last time I did a check but I highly doubt it.

I worked with some of those in the past but never really liked them so when I faced again the need to collect metrics from different applications/languages as well as server's metrics I tried to find something else.

The SmokePing case

Before I started working on a monitoring solution for a work project I am currently on there was previously SmokePing set up, I may be heavily biased on it because of my first impression but what first impression it made !

The first things I noticed when looking at it was that the server part was slow as hell and was taking a more than noticeable part of the cpu (I honestly do not remember how much but it was certainly not invisible on the machine speaking of resources) so I checked its producer counterpart which is on an embedded system (Soekris hardware) and here too the daemon was not invisible at all !

The last thing that crushed that beast for me is that loading a page with graphs on it was slow too and caused cpu spikes, perfect !

It is perfectly possible that smokeping was badly installed and would work far better in normal conditions (that said I am not too sure about the producer part) but seriously what can you do with such bright first impression ?

Enter Collectd

While looking for a decent Aggregator I came upon collectd which I never saw before and does a really great job at dispatching metrics.

Collectd is a modular application with modules falling in two categories:

readers: these plugins receive data from somewhere or extract it (ex: network, shell command, /proc file, etc...)
writers: these plugins will output the data somewhere (ex: network again, rrdtool, rrdcached, database, etc...)

Collectd do a really nice job but some parts are hard to use and not really designed for usability, the best example I have is the filter system, Collectd comes with a really powerful filter subsystem which lets you update/add/delete fields of the collectd "messages" or the whole messages.

Sadly this filter subsystem has one of the worst configuration interface I have seen until now, see for yourself (and this one is a simple one):

<Rule "ping_server1">
  <Match "regex">
    Plugin "^ping$"
    TypeInstance "^1.2.3.4$"
  </Match>
  <Target "replace">
    TypeInstance "1.2.3.4" "server1"
  </Target>
</Rule>

This one is pretty straight forward but all this text is just to replace a string with another...

In another language this could become:

if p.type_instance == "1.2.3.4"
  p.type_instance = "server1"
end

Just imagine what a complex flow would look with the xml like syntax :/

While I have some issues with collectd I am currently using it on production servers as well as embedded systems and it works well.

The producers

Sending your own counters to collectd is not really hard, the network plugin of collectd use a clearly defined and simple binary format served over UDP and many libraries exist out there for different languages to generate those packets and send them to collectd.

Serving graphs

The only part left is the user interface used to navigate and generate the graphs from all these data we collected and that is where the real problem arise... Since many existing systems want control over the whole data collection chains you cannot easily integrate data from somewhere else.

My quest was to find a tool doing a really simple thing (at least I thought it was): draw graphs from rrd file and serve them to the clients while staying nearly invisible on the server resource wise.

Do you know one ? Because I could really use some names here.

I ended up coding my own modest graph drawer but that is just a small part of what I need. WHat I have now is a sinatra application which uses the client itself to compute the graphs with some javascript around the excellent jquery library: flot.

Since the clients are drawing the graphs the server is really not doing much, only extracting data from the rrd and formatting them in json which is really nice.

Current State

I have a working stack for now, I am not completly happy with it but at least I have control over the parts and except for the grapher I am just using existing application/library so I did not have to write any code.

I am still searching for a better solutions and working on some projects which may help but that will be for another time :)