Unlimited Novelty

Moving to svbtle

noreply@blogger.com (Tony) — Wed, 19 Dec 2012 07:56:00 +0000

In case you haven't noticed already, I've started blogging on the svbtle network. You can find my new blog at:

http://tonyarcieri.com/

Debunking the Node.js Gish Gallop

noreply@blogger.com (Tony) — Fri, 10 Aug 2012 17:18:00 +0000

A programmer who once a Ruby on Rails enthusiast switches to Node.js and thinks it's awesome, then proceeds to write a blog post about why Node is the bee's knees and Rails is crap. Attention is drawn to the changing nature of web design, from web pages with server-generated HTML to single-page JS-heavy apps written using Backbone, Ember, etc. Stop me if you think that you've heard this one before...

This is an argument I keep hearing over and over, and as far as I'm concerned it's nothing but a Gish Gallop of completely specious arguments, but I really worry... I worry because I keep hearing it over and over, and the fact that I keep hearing it over and over makes me worry that people are actually believing it. I don't know why I keep hearing it over and over. I'm not sure if people are running into problems, reading some of the prevailing "wisdom", and coming to the same conclusion or what. This really makes me sad, because whenever I read the posts like this, I do feel my previous passion for these same ideas, but for me that was half a lifetime ago, and my opinions have changed. I have been down these roads, over mountains, blazed my own trails, and then realized how stupid I was...

How do you defeat the Gish Gallop? I don't really enjoy doing this, but as far as I can tell there is no other way: we must go through the arguments one by one and show why they are completely ludicrous. So here we go...

In case you were confused, Rails is AWESOME for JSON APIs and single page applications

I love client-heavy HTML5/JS apps. I don't want every page on the web to be one, but there are many applications that can benefit a ton from keeping all of their state in the browser. In general: if you can do something without having to go across the network to do it, you will provide a better user experience, bar none.

The primary thing these applications crave are awesome JSON APIs (and Websockets... stay tuned). So why should you use Rails for a JSON API? Isn't Rails designed for HTML/JS pages? What benefit does Rails give you for building JSON APIs? And isn't Rails really slow?

Well no, I've been through this before. If you are building API-only applications with a single-page HTML5/JS frontend, you should definitely check out Rails::API. Rails::API completely eliminates any ActionView-centrism you may be worried about in Rails, and gives you awesome tools for building JSON APIs, like ActiveModel::Serializers. But that alone can't express what Rails brings to the table, so here as list of features Rails provides which are useful for JSON APIs, courtesy the Rails::API README:

Handled at the middleware layer:

Reloading: Rails applications support transparent reloading. This works even if your application gets big and restarting the server for every request becomes non-viable.
Development Mode: Rails application come with smart defaults for development, making development pleasant without compromising production-time performance.
Test Mode: Ditto test mode.
Logging: Rails applications log every request, with a level of verbosity appropriate for the current mode. Rails logs in development include information about the request environment, database queries, and basic performance information.
Security: Rails detects and thwarts IP spoofing attacks and handles cryptographic signatures in a timing attack aware way. Don't know what an IP spoofing attack or a timing attack is? Exactly.
Parameter Parsing: Want to specify your parameters as JSON instead of as a URL-encoded String? No problem. Rails will decode the JSON for you and make it available in params. Want to use nested URL-encoded params? That works too.
Conditional GETs: Rails handles conditional GET, (ETag and Last-Modified), processing request headers and returning the correct response headers and status code. All you need to do is use the stale? check in your controller, and Rails will handle all of the HTTP details for you.
Caching: If you use dirty? with public cache control, Rails will automatically cache your responses. You can easily configure the cache store.
HEAD requests: Rails will transparently convert HEAD requests into GET requests, and return just the headers on the way out. This makes HEAD work reliably in all Rails APIs.

Handled at the ActionPack layer:

Resourceful Routing: If you're building a RESTful JSON API, you want to be using the Rails router. Clean and conventional mapping from HTTP to controllers means not having to spend time thinking about how to model your API in terms of HTTP.
URL Generation: The flip side of routing is URL generation. A good API based on HTTP includes URLs (see the GitHub gist APIfor an example).
Header and Redirection Responses: head :no_content and redirect_to user_url(current_user) come in handy. Sure, you could manually add the response headers, but why?
Caching: Rails provides page, action and fragment caching. Fragment caching is especially helpful when building up a nested JSON object.
Basic, Digest and Token Authentication: Rails comes with out-of-the-box support for three kinds of HTTP authentication.
Instrumentation: Rails 3.0 added an instrumentation API that will trigger registered handlers for a variety of events, such as action processing, sending a file or data, redirection, and database queries. The payload of each event comes with relevant information (for the action processing event, the payload includes the controller, action, params, request format, request method and the request's full path).
Generators: This may be passé for advanced Rails users, but it can be nice to generate a resource and get your model, controller, test stubs, and routes created for you in a single command.
Plugins: Many third-party libraries come with support for Rails that reduces or eliminates the cost of setting up and gluing together the library and the web framework. This includes things like overriding default generators, adding rake tasks, and honoring Rails choices (like the logger and cache backend).

Rails has an unquestionably awesome feature set even if applied exclusively to JSON APIs, and this guy is taking it completely for granted:

"So your Rails server becomes an API, and your web site, like the iOS app, is the client. It's a clean separation of responsibilies, but given what Rails was designed to do, it's like having a horse rider climb on top of an elephant."

The design of Rails, as of Rails 1.2, provided clean abstractions for using the same code to provide server-generated HTML views and "REST" APIs in multiple serialization formats. This was a big deal at the time, and "the time" was 2 years before Node even existed. Fast forward 4 years and Rails 3 has been rewritten with an emphasis on modularization, allowing you to strip out the components you don't use and build lightweight stacks with only the things you need. Rails::API provides convention over configuration for a lightweight JSON-oriented stack.

But let me back up a little bit...

"The view in MVC is not just HTML and CSS; it's the presentation logic, and the presentation logic needs structure. With this need, client-side frameworks like Backbone, Spine, and Ember have come into the picture."

So I hear this guy Yehuda Katz worked on both Ember and Rails. You may have heard of Ember, it just won Throne of JS's framework of choice (Backbone won in the "library" category). But appeal to authority aside, what does using Ember and Rails in combination actually get you?

A problem I am certain you have run into is the manual nature of serializing JSON. Exactly how should you translate from a domain object into a JSON representation? What if the client wants to avoid repeat requests by eagerly loading other domain objects which are associated with the one you want to retrieve and including them in the JSON result? And wouldn't it be great if there were a single canonical representation for all of this that a standardized domain object abstraction running in the browser could automatically consume for us, so we don't have to manually write a bunch of JSON serialization and deserialization logic for everything in our system?

Can we put JSON on Rails? Yes we can: it's called ActiveModel::Serializers and Ember Data. All that glue code you've been writing over and over for serializing and unserializing JSON? Stop that. Seriously. You have better things to do than deal with the idiosyncrasies of whether you should wrap a particular array in an object or return a literal string or number as opposed to an object for future proofing. You are wasting your time with this minutiae and chances are the ActiveModel::Serializers representation is better than the one you are using. Let's take a look at why.

The defining characteristics of the ActiveModel::Serializers JSON representation is that it explicitly avoids nesting objects within objects, instead preferring to keep the resulting structure flat and using IDs to correlate the relationships between data in the structure. Here is an example of a "post" object which includes comments and tags, taken from the ActiveModel::Serializers README:

{
  "post": {
    "id": 1,
    "title": "New post",
    "body": "A body!",
    "comments": [ 1, 2 ]
  },
  "comments": [
    { "id": 1, "body": "what a dumb post", "tags": [ 1, 2 ] },
    { "id": 2, "body": "i liked it", "tags": [ 1, 3 ] },
  ],
  "tags": [
    { "id": 1, "name": "short" },
    { "id": 2, "name": "whiny" },
    { "id": 3, "name": "happy" }
  ]
}

There are multiple nested relationships in this document: the post has many comments, and comments have many tags. And yet we don't see duplication of comment or tag objects. We don't have to worry about which version of a repeated object is canonical, because there are no repeated objects. Objects within the resulting document are deduplicated and referred to symbolically by their ID. Using this JSON structure we can represent arbitrarily nested relationships between objects in the most efficient manner possible and completely avoid any problems with inconsistencies between duplicated versions of objects present in the document. This representation of JSON just makes sense, and perhaps you too have standardized upon it. Better yet, if you use this representation, then with very little effort on your part Ember Data can automatically consume it.

If you use Ember and Rails, you can abstract away JSON and save yourself the headache of writing custom serialization code. I'm going to say: score one for Rails and single page applications. Maybe you have some Node thing that can do that too, I don't know, but seriously, if you think Rails is bad for JSON APIs, you don't know Rails.

Moving right along, let's continue slogging through the Gish Gallop.

Node has nonblocking async I/O and Rails doesn't so Rails is slow!!!

Where to start with this one. Hmm, let's start here:

"When I think of Ruby and Rails' performance, I think of Ilya Grigorik."

Let me start by saying that Ilya is an awesome guy who has done a very thorough and nuanced survey of the many facets of Ruby performance over time. Taking any single thing he's said out of context and treating it like gospel is probably doing a disservice to Ilya. That said, let's see what thing Ilya said that this guy chose to single out and present out of context. Quoth Ilya:

"There is nothing about node that can't be reproduced in Ruby or Python (EventMachine and Twisted), but the fact that the framework forces you to think and use the right components in place (fully async & non-blocking) is exactly why it is currently grabbing the mindshare of the early adopters. Rubyists, Pythonistas, and others can ignore this trend at their own peril. Moving forward, end-to-end performance and scalability of any framework will only become more important."

So this is a line I hear out of Ryan Dahl a lot too. It's a line I used to believe.

Folks, I've been doing this stuff for awhile. I first discovered synchronous I/O multiplexing when I was about 15, which for me was half a lifetime ago, and since then I've been building network servers using this approach. I've built my own abstraction layers across select/poll/epoll/kqueue. I wrapped libev for Ruby in Rev/Cool.io and nio4r, the latter of which is a cross-platform abstraction for Java NIO on JRuby. I cannot express to you how much work I've invested in doing things the evented non-blocking way.

I don't think non-blocking I/O is a good fit for web applications that talk HTTP, although I think it can be a good fit for Websocket applications. I will get to my reasons later. But first, let's continue digging through the Gish Gallop:

"Ilya mentioned the framework/ecosystem that I now consider to be the threat to Rails: Node.js [...] The biggest thing I noticed was the difference in performance. It consumed less memory than Ruby, and it served more requests per second than Sinatra or even Rack."

I have a huge pet peeve, and that's when people talk about performance without numbers. I tried it and it was faster. I tried it and it was slower. If you really want to make a point about the performance of a particular thing, can you at least pretend you're using science?

I hate to do this, but I think I have to destroy your god. Let's see how Ilya's software stacks up to mine on a crappy "hello world" web server benchmark. First, the numbers for my web server Reel:

# httperf --num-conns=50 --num-calls=1000

Ruby Version        Throughput    Latency
------------        ----------    -------
JRuby HEAD          5650 reqs/s   (0.2 ms/req)
Ruby 1.9.3          5263 reqs/s   (0.2 ms/req)
JRuby 1.6.7         4303 reqs/s   (0.2 ms/req)
rbx HEAD            2288 reqs/s   (0.4 ms/req)

Let's compare to Ilya's web server Goliath, as well as Thin and Node.js:

Web Server          Throughput    Latency
----------          ----------    -------
Goliath (0.9.4)     2058 reqs/s   (0.5 ms/req)
Thin    (1.2.11)    7502 reqs/s   (0.1 ms/req)
Node.js (0.6.5)     11735 reqs/s  (0.1 ms/req)

All of these servers, including mine, are using non-blocking evented I/O. Is that remotely relevant? No. That's just a coincidence.

My web server is faster than Ilya's. So by Gish Gallop logic, Ilya must be wrong about everything. There must be no reason to use Ilya's web server. Let's write everything in Node since it won the benchmark.

There's a huge problem here: Goliath does things that Reel, Thin, and Node's HTTP server don't do. The reason it's slower isn't because Ilya sucks and is clueless about performance. The reason is that Goliath has features which these other web servers don't, which makes it an apples to oranges comparison. (I guess scumbag me for putting them all in a big list on the Reel web page)

The same can be said of Rails: it probably isn't ever going to have better latency through the entire stack than any Node.js framework, but the latency of the Rails stack is probably going to be a lot less than your application logic, and that's still going to be a drop in the bucket compared to the network latency to a given user.

Celluloid solves every single problem you're whining about better than Node

Node has a lot of problems, and I'm not just talking about the audience it attracts. Let me start by saying this: many of the things I have built in Celluloid are based off of technologies originally developed for Node. My web server Reel uses the Node HTTP parser, and it's quite likely that the next iteration of nio4r I develop will be based off of libuv.

All that said, let me start with Node's fundamental problem: callback-driven I/O. Celluloid::IO is one of many systems, including Erlang and Go, that demonstrate that "nonblocking" and "evented" I/O are orthogonal to callbacks. Celluloid uses Ruby's coroutine mechanism to provide a synchronous I/O API on top of an underlying nonblocking system. However, where systems like Node force you to use nonblocking I/O for everything, Celluloid lets you mix and match blocking and nonblocking I/O as your needs demand.

If you have ever worked in a language like C(++) or Java, you probably know an amazing property of sockets: you can mix and match blocking and nonblocking I/O, even over the lifecycle of a single socket. Perhaps you will handle incoming sockets in a nonblocking manner at first, but if they make a complex request, you might change the socket to a blocking mode and hand it off to a worker thread.

Celluloid::IO makes this handoff completely transparent: simply by giving the socket to another Ruby thread which isn't a Celluloid::IO actor, it will automatically switch from nonblocking to blocking mode completely transparently.

But let's talk about Node's real fundamental problem, one that is extremely difficult to solve in any callback-driven system: flow control. Unfortunately the Node.js community has adopted the phrase "flow control" to mean "building abstractions around managing callbacks", however the phrase "flow control" has a very specific definition relating to the rates at which data is transmitted between systems.

In general, callback-driven systems can't manage flow control effectively. The most notable pathological case is the producer-consumer problem, whereby a slow consumer might force a system like Node to unboundedly buffer data from an unchecked producer. There's a clear and simple solution to this problem: make all I/O synchronous. Using coroutines that provide blocking-style APIs, you can easily compose producer/consumer problems in a manner that doesn't result in unbounded writes to a buffer, because simply by virtue of a virtual blocking API, the rate at which data is transfered from producer to consumer is kept in check.

But what about WebSockets?

Ruby has had some pretty awesome albeit overlooked and therefore stagnant solutions for WebSockets for awhile, like Cramp. I've been working on web-based push technologies for half a decade now, and explored a multitude of solutions including Comet, XMPP/BOSH, RabbitMQ long polling, and my own XHR long polling systems which I originally built around *gasp* threads nearly 3 years ago at this point.

Well, I'm quite happy to say that Reel now supports WebSockets. I certainly don't want to say that my recent spike is anywhere as mature as WebSockets in Node or their surrounding ecosystem. Instead, I think the API that Reel provides for WebSocks is simply better by design. If you managed to catch tenderlove's recent blog post on streaming live data, you may understand that all previous APIs you may have encountered in both systems like Rails or Node for streaming data were really obscuring the one API that truly makes sense for this use case: a socket.

WebSockets are in many ways similar to 0MQ sockets (which are used in DCell via Celluloid::ZMQ). WebSockets provide a framing mechanism which provides a message-based transport instead of the typical stream-based transport provided by TCP. That said, when processing message sequences, callbacks become extremely problematic, because you must reconstruct the state of the current request from the point of each incoming message. Callbacks work well for e.g. a chat protocol where there is no state relationship between messages, but as soon as there is you are effectively stuck building a finite state machine to manage the processing of each incoming message.

This is madness. There's a much better and much more straightforward solution to this problem: just use the goddamn stack. In order to do so, you need to provide a "blocking" API, but this isn't orthogonal to using nonblocking I/O. Celluloid::IO, Go, and Erlang all let you build concurrent, multithreaded, and potentially multicore systems on top of coroutines spread across multiple native threads.

That said, native threads are cheap nowadays and they're only getting cheaper. On most Ruby VMs a native thread will cost you about 20kB of RAM. If you want you can just build blocking I/O systems completely out of native threads without using any sort of evented I/O, and these systems can scale up to tens of thousands of connections.

Don't believe the hype

Node provides a limited subset of what Ruby can do, and it can be done better with Ruby. Node does not have a web framework of the same caliber as Rails. Node doesn't have threads, which in Ruby will spare you from Node's callback soup. Finally, there's the elephant in the room: JavaScript is a terrible, terrible programming language compared to Ruby. We're forced to use JavaScript in the browser, but on the server, we can choose the best language for the job.

Ruby on Rails remains the best-in-class web framework, and while there are arguments to be made against it, the ones I hear coming out of confused Node.js detractors do not hold water.

Ruby is faster than Python, PHP, and Perl

noreply@blogger.com (Tony) — Wed, 06 Jun 2012 16:01:00 +0000

There's a pervasive myth that Ruby is slow, and moreover, that it's the slowest language in popular use. Everyone knows Ruby is slow. Right? Who would possibly disagree that Ruby is slow? Here's an example IRC discussion on freenode's #postgres which happened just yesterday:

16:57 sobel: i can't find it now, but arstechnica benched all the popular dynamic languages
16:58 sobel: using C/C++ as the standard (1.0) they ranked other languages as multiples of C/C++ performance
16:58 sobel: java was a 2
16:58 sobel: twice as slow as C. and it was the winner, head and shoulders above the rest
16:58 sobel: i think Erlang placed somewhat favorably
16:59 sobel: python/perl were near the middle, at something like 25-35x slower than C
16:59 sobel: Ruby: a full 72x slower than C

Ruby loses against invisible Ars Technica benchmarks in the sky with unknown URLs! 72x slower than C! Over twice as slow as Python and Perl!

Fortunately, we don't have to rely on some one-off benchmark Ars Technica may or may not have done at some indeterminate point in time whose URL cannot be located offhand, because there's a site with a fairly decent reputation which provides ongoing benchmarks across multiple programming languages using implementations submitted by fans of said language. It's been around for awhile and is relatively well-trusted.

That site is the Programming Languages Shootout, and unlike the alleged Ars Technica benchmark, you can actually visit their web site at shootout.alioth.debian.org. What do they have to say about programming language performance?

According to this benchmark suite, JRuby is 34.5 times slower than (not C, not C++, but) Fortran. Ruby 1.9 (MRI/YARV) is 43.80 times slower than (not C, not C++, but) Fortran. Both JRuby and Ruby 1.9 beat Python, PHP, and Perl by a considerable margin. The nearest competitor is Python 3, at 47.93 times slower than (not C, not C++, but) Fortran. By the way, did I mention that the fastest language on their benchmark is... not C... not C++, but Fortran? (nothing personal sobel, but unsubstantiated hearsay is bad!)

Yes, that's right folks: according to the Programming Languages Shootout, Python, PHP, and Perl are all slower than Ruby. Did you think Ruby was slower than Python? Guess what, you're wrong. Ruby used to be one of the slowest popular languages, but that has changed. Ruby performance has advanced considerably over the years, so if you're still repeating some offhand information you may or may not have gotten from Ars Technica at some point but can't find the link to as your metric of Ruby performance, you may want to try again, and find modern, relevant information you can actually get a link to.

There are many future VM improvements in the pipe for Ruby, Python, and PHP (and I guess Perl users might continue dreaming of a Parrot-powered Perl 6). Rubyists can look forward to the upcoming JRuby 1.7 release which features InvokeDynamic support and allows for Java-speed method dispatch... in Ruby. InvokeDynamic is a game changer for the JVM in general, and it's a game changer for Ruby, because InvokeDynamic makes JRuby dispatch potentially as fast as Java.

Python users can look forward to PyPy, which is posting some incredibly impressive numbers, especially around numerical computing. Python users can also look forward to resumed work on Jython, which is adding InvokeDynamic support which can potentially make Python as fast as Java. Finally, PHP users can look forward to the HipHop VM developed at Facebook, which will provide much improved performance for PHP. These are all great projects, but none of them are really ready for general consumption yet (including JRuby 1.7).

All that said, the Programming Language Shootout doesn't include any of these unreleased development versions in the benchmarks you see when you visit their site. They show the numbers for the latest production releases, and those numbers show Ruby is faster than Python, PHP, and Perl.

The game has changed: you just weren't paying attention.

Last but not least, if you've seen some benchmark somewhere, even if you have an eidetic memory and remember but the numbers were, but can't even dredge up a link to it, please, please, don't quote said benchmark, even if you have an eidetic memory and remember what the numbers were.

For benchmarks to be remotely scientific, they must be both reproducible and falsifiable, and hopefully in addition to both those things they have a good methodology. If you can't even dredge up a link to the benchmark in question, please don't go quoting numbers off the top of your head to people who might be influenced by them.

Let's advance computer science beyond the state of witch doctors telling people to bleed themselves with leeches because at some point someone said they might make you feel better maybe.

Introducing DCell: actor-based distributed objects for Ruby

noreply@blogger.com (Tony) — Wed, 18 Apr 2012 05:50:00 +0000

DCell, which is short for Distributed Celluloid (and pronounced like the batteries you used to jam into boom boxes and RC cars) is an actor-based distributed object oriented programming framework for Ruby. Celluloid is an actor library I wrote for Ruby which exposes concurrent Ruby objects that "quack" just like any other Ruby object. DCell takes the asynchronous messaging protocol from Celluloid and exposes it to distributed networks of Ruby interpreters by marshaling Celluloid's messages as strings and sending them to other nodes over 0MQ.

Before I talk about DCell I'd like to talk a little bit about the history behind distributed objects in general and the ideas that DCell draws upon.

A Brief History of Distributed Objects

Nowadays you don't hear people talking much about distributed objects. However, once upon a time, distributed objects were big business. They used to be one of Steve Jobs' passions during his time at NeXT. In the mid-90's, Steve Jobs phased out NeXT's hardware division and began repositioning the company as, among other things, the "largest object supplier in the world." NeXT turned its focus exclusively to its software, namely the WebObjects framework for building dynamic web sites. It would also ship the Enterprise Objects Framework, which Steve described as allowing you to "make NeXTSTEP objects, and with no programming, have persistent and coherent storage with SQL databases" (sound a little bit like ActiveRecord a decade before ActiveRecord, anyone?)

WebObjects was built on a technology called Portable Distributed Objects, which allowed Objective C applications developed on any platform to seamlessly interoperate over computer networks and across platforms. If you haven't seen it, watch Steve Jobs 1995 presentation The Future of Objects (although you may want to skip directly to where Steve begins talking about distributed objects). Even now, some 16 years later, these demos still seem futuristic. Steve loved the simplicity of distributed objects: "Objects can message objects transparently that live on other machines over the network, and you don't have to worry about the networking gunk, and you don't have to worry about finding them, and you don't have to worry about anything. It's just as if you messaged an object that's right next door."

So why is it some of Steve's demos seem futuristic and slightly beyond our grasp even a decade and a half later, and building multi-tier client/server (i.e. web) applications is a challenge typically involving building lots of client and server wrapper code instead of transparently invoking distributed objects? Unfortunately, Steve's rosy future of distributed objects never really came to pass. NeXT's technology was commercial, only available in Objective C (at a time when C++ reigned supreme and no one used Objective C), and couldn't beat the open web and the standards that would emerge for developing web APIs. Meanwhile, the industry standards for distributed objects would be mired in debacles of their own.

In the before time, in the long long ago, in the days before JSON, XML, and REST, when Tim Berners-Lee had just started hosting the world's first web site on his personal NeXTstation, C++ was the lingua franca of the day and people were just getting started making C++ programs communicate over networks. The Object Management Group (with the now-unfortunate acronym OMG) hashed out the necessary wire protocols, object representations, interface definitions, and a profusion of other standards needed to allow C++ programs to invoke remote objects over TCP/IP networks. The result was CORBA.

CORBA is a technology that drowned in its own complexity, however in the early '90s it was the Enterprise: Serious Business way to allow applications to communicate over networks. Once you waded through the myriad complexity of the various CORBA technologies, you were left with objects you could invoke over the network in a manner somewhat reminiscent of the way you would invoke an object locally.

However, soon the web would take off and HTTP would soon become the de facto protocol programs used to communicate. Unfortunately, HTTP is a terrible protocol for implementing distributed objects. Where HTTP exposes a predefined set of verbs you can use to manipulate resources which have a number of possible representations, objects don't have representations: they are just objects, and you talk to them with messages. This didn't stop people from trying to build a distributed object protocol on top of HTTP though. The result was SOAP, a protocol which abandoned CORBA's "orbs" for web servers, its Interface Definition Language for WSDL, its Common Data Representation for XML, and its Inter-Orb Object Protocol for HTTP.

This was something of a step in the right direction: SOAP actually was a "Simple" Object Access Protocol when compared to CORBA, and SOAP would soon vanquish CORBA for Enterprise: Serious Business applications. While SOAP would gain a few fans, particularly in the Java and .NET communities, who saw the value of being able to expose objects to the network with a few point-and-clicks which generated gobs of XML, SOAP would soon join CORBA's ranks as a reviled technology.

SOAP's complexity comes first and foremost being a committee standard that would succumb to "too many cooks" syndrome in the same way CORBA did. It also suffered from trying to be a cross-language protocol that needed to deal with static type systems, requiring tools which could read WSDL definitions and spit out volumes of generated boilerplate code for interacting with remote services. Beyond that, SOAP suffered from an impedance mismatch with HTTP by largely ignoring the features HTTP provides, using it as little more than a transport wrapper for shoving blobs of XML across the network. The XML contained the actual messages to be sent to remote objects or the responses coming from a remote method invocation, while anything done at the HTTP level itself was just boilerplate.

REST to the rescue?

For all its complexity, it was easy to lose sight of what SOAP was actually trying to do. Rather than painlessly interacting with remote objects, SOAP left us wondering why things were so slow, staring at WSDL errors wondering what went wrong, and picking through gobs and gobs of XML trying to debug problems. REST, which eschewed distributed objects and favored the paradigms of HTTP, would be SOAP's coup de grâce. SOAP is now relegated to a handfull of legacy enterprise applications whereas the rest of the open web has almost universally embraced REST.

So REST triumphed and web APIs reign supreme. Distributed objects are little more than a footnote in history. And we're still left wondering why putting together the sorts of demos Steve Jobs was showing with Portable Distributed Objects in 1995 is so hard.

Using REST makes sense when exposing services for third parties to use. However, if you control both the client and server, and have Ruby frontends talking to Ruby services, REST begins to look like a bunch of gunk that's getting in the way:

Implementing domain objects, REST services, and REST clients becomes work duplicated in 3 places across systems using this sort of architecture. Wouldn't it be a lot simpler if the frontend web application could simply consume remote services as if they were just objects?

This sort of simplicity is the goal of distributed objects.

Distributed Objects in Ruby

Ruby has its own foray into the world of distributed objects: DRb, or Distributed Ruby. DRb exposes Ruby objects to the network, each uniquely identified by a URI. Clients can ask for a DRb object by URI and get a DRbObject back. These DRbObjects act as proxies, intercepting calls and sending them over the network, serializing them with Ruby's Marshal, where they're handled by a remote DRbServer. DRbServer uses a thread-per-connection model, allowing it to concurrently process several requests. When a DRb connection handler receives a request, it looks up the requested object, invokes a method on it, then serializes the response and sends it back to the caller.

For the most part, DRb has lingered in obscurity, save until recently with the publication of the dRuby book which has stirred up a modicum of interest. Where Steve Jobs thought PDO was "by far the easiest way to build multi-tier client/server applications because of [its] completely transparent distributed object model," Rubyists don't turn to DRb to build multi-tier applications, but instead typically rely on REST, building out APIs for what is, in the end, distributed communication between Ruby objects.

Why is it then that DRb isn't the go-to tool people use for building multi-tiered web applications in Ruby? It's easy to say that DRb failed because people are used to thinking in terms of HTTP and can better understand the semantics of the system when using HTTP, especially when it comes to areas like load balancing and caching. Separating services with HTTP also opens up the door to reimplementing those services in a different language in the future. However, even with future-proofing for a rewrite in another language out of the picture, I think most Rubyists would still choose to use REST APIs instead of DRb, and I think that's a defensible position.

While DRb does a great job of trying to make access to remote objects as transparent as possible, it has a number of flaws. DRb is inherently multithreaded but doesn't give the user any sort of tools or framework to manage concurrent access to objects. This means building DRb applications immediately exposes you to all the complexities of multithreaded programming whether you're aware of it or not, and Rubyists seem generally uncomfortable with building thread-safe programs. While DRb allows you to talk to in-process objects the same way you'd talk to out-of-process objects, but it doesn't make it natural to build a program that way.

Beyond Objects: The Power of Distributed Erlang

While CORBA and SOAP are reviled for their complexity, there's another distributed system which is beloved for its high level of abstraction: Distributed Erlang. Erlang is, if anything, a highly opinionated language whose central design goal is to build robust self-healing systems you never need to shut down. When it comes to distribution, Erlang's goal is to make it as transparent as possible. Erlang is a dynamic language which insists you express everything within a scant number of core types. This makes serializing state in Erlang so you can ship it across the wire extremely simple and fast.

However, the real strength of Erlang is the Actor Model, which can be more or less summarized as follows:

Actors communicate by sending messages
Every actor has a mailbox with a unique address. If you have an actor's address, you can send it messages
Actors can create other actors, and when they do they know their address so they can send the newly-created actors messages
Actors can pass addresses (or handles) to themselves or other actors in messages

Erlang uses this method within individual VMs as the basis of its concurrency model. Erlang actors (a.k.a. processes) all run concurrently and communicate with messages. However, Erlang also supports distribution using the exact same primitives it uses for concurrency. It doesn't matter which type of actor you're talking to in Erlang, they "quack" the same, and thus Erlang has you model your problem in a way that provides both concurrency and distribution using the same abstraction.

Distributed Erlang offers several features aimed at building robust distributed systems. The underlying messaging protocol is asynchronous, allowing many more messaging patterns than traditional RPC systems (e.g. HTTP) which use a request/response pattern that keeps the client and server in lockstep. Some examples of these patterns are round robin (distributing messages across N actors), scatter/gather (distributing computation across N actors and gathering the results), and publish/subscribe (allowing actors to register interest in a topic, then informing them of events related to that topic).

In addition, Erlang processes can link to each other and receive events whenever a remote actor exits (i.e. if it crashes). This allows you to build robust systems that can detect errors and take action accordingly. Erlang emphasizes a "fail early" philosophy where actors are encouraged not to try to handle errors but instead crash and restart in a clean state. Linking allows groups of interdependent actors to be taken down en masse, with all of them restarting in a clean state afterward. Exit events can also be handled, which is useful in distributed system for things like leader election.

DCell provides all of these features. When you create an actor with Celluloid, a proxy object to the actor is returned. This proxy lets you use the method protocol to communicate with an actor using messages. DCell implements special marshalling behavior for these proxy objects, allowing you to pass them around between nodes in a DCell system and invoke methods on remote actors in the exact same way you would with local actors.

Unlike DRb, DCell also exposes asynchronous behaviors, such as executing method calls in the background, and also using futures to schedule method invocation in advance then waiting for the result later. DCell also lets distributed actors to link to each other and be informed when a remote actor terminates.

I'm certainly not the first to imitate Erlang's approach to distribution. It's been seen in many other (distributed) actor frameworks, including the Akka framework in Scala and the Jobim framework in Clojure.

Bringing Erlang's ideas over to Ruby

I have a long history of projects that try to cross-pollenate Ruby and Erlang. My first attempt was Revactor, my previous attempt at an actor library which provided a very raw and low-level API which is almost identical to the Rubinius Actor API. Revactor modeled each actor as a Fiber and thus provided no true concurrency. Another of my projects, Reia, tried to bring a more friendly syntax and OO semantics to Erlang.

With Celluloid I've come full circle, trying to implement Erlang's ideas on Ruby again. Only this time, Celluloid makes working with actors easy and intuitive by embracing the uniform access principle and allowing you to build concurrent systems that you can talk to just like any other Ruby object. Celluloid also provides asynchronous calls (what Erlang would call a "cast") where a method is invoked on the receiver but the caller doesn't wait for a response. In addition to that Celluloid provides futures, which allow you to kick off a method on a remote actor and obtain the value returned from the call at some point in the future.

In addition Celluloid embraces many of Erlang's ideas about fault tolerance, including a "crash early" philosophy. Celluloid lets you link groups of interdependent actors together so if any one fails you can crash an entire group. Supervisors and supervision trees automatically restart actors in a clean state whenever they crash.

Celluloid does all of this using an asynchronous message protocol. Actors communicate with other actors by sending asynchronous messages. A message might say an actor has crashed, or another actor is requesting a method should be invoked, or that a method invocation is complete and the response is a given value. All of the heavy lifting for building robust, fault-tolerant systems is baked into Celluloid.

When programs are factored this way, adding distribution is easy. DCell takes the existing primitives Celluloid has built up for building concurrent programs and exposes them onto the network. DCell itself acts as little more than a message router, and the majority of the work in adding fault tolerance is still handled by Celluloid.

Getting Started with DCell

To understand how DCell works we need to look at how a cluster is organized. This is an example of a cluster with 5 nodes:

In this picture the green nodes represent individual Ruby VMs. The links between the nodes are shown in black or gray to illustrate actively connected or potentially connected nodes. DCell makes connections between nodes lazily as actors request them. This means DCell clusters are potentially fully connected networks where each of the nodes is (or can be) directly connected to every other node in the cluster. DCell doesn't implement any sort of routing system or overlay network, and instead depends on all nodes being directly accessible to each other over TCP.

DCell uses 0MQ to manage transporting data over the network. 0MQ supports several different messaging patterns, and in the future DCell may use more of them, but for the time being DCell uses PUSH/PULL sockets exclusively. This works well because Celluloid's messaging system is asynchronous by design: each node has a PULL socket that represents the node's mailbox, and the other nodes on the network have a PUSH socket to send that node messages.

To configure an individual DCell node, we need to give it a node ID and a 0MQ address to bind to. Node IDs look like domain names, and 0MQ addresses look like URLs that start with tcp:

To create a cluster, we need to start another Ruby VM and connect it to the first VM. Once you have a cluster of multiple nodes, you can bootstrap additional nodes into the cluster by pointing them at any node, and all nodes will gossip about the newly added node:

Once you are connected to another node, you can browse the available nodes using DCell::Node.all:

To invoke a node on a particular service, obtain a handle to its node object, then look up an individual actor by the name it's registered under. By default, all nodes run a basic information service which you can use to experiment with DCell:

To implement your own DCell service, all you have to do is create a Celluloid actor and register it on your node. See the info service source code for an example.

DCell Explorer

DCell also includes a simple web UI for visualizing the state of a particular DCell cluster. To launch the web UI, run:

Then go to http://localhost:8000/ (provided you used the same host/port). You should see the following:

This is the basic dashboard that DCell's web UI provides. You can see connected nodes, their connection state, and if they're available browse the info service and see various information about them.

We're just getting started...

DCell is only about half a year old, and still relatively immature, but already it's seen a great number of contributors and a lot of attention. Some of the most exciting developments are still on the horizon, including paxos-powered multicall support which will let you call a quorum of nodes, along with generalized group membership support with leadership election, and generalized pub/sub.

All that said, I'd like to think that DCell is the most powerful distributed systems framework available for Ruby today, and I would love to see the remaining bugs ironed out and missing features added.

That can only happen if people are using DCell, finding bugs, and reporting missing features. This may sound a little bit scary, but if you're considering building a nontrivial distributed system in Ruby, DCell is a great place to start.

Don't use bcrypt

noreply@blogger.com (Tony) — Mon, 19 Mar 2012 16:13:00 +0000

(Edit: Some numbers for you people who like numbers)

If you're already using bcrypt, relax, you're fine, probably. However, if you're looking for a key derivation function (or in bcrypt's case, password encryption function) for a new project, bcrypt is probably not the best one you can pick. In fact, there are two algorithms which are each better in a different way than bcrypt, and also widely available across many platforms.

I write this post because I've noticed a sort of "JUST USE BCRYPT" cargo cult (thanks Coda Hale!) This is absolutely the wrong attitude to have about cryptography. Even though people who know much more about cryptography than I do have done an amazing job packaging these ciphers into easy-to-use libraries, use of cryptography is not something you undertake lightly. Please know what you're doing when you're using it, or else it isn't going to help you.

The first cipher I'd suggest you consider besides bcrypt is PBKDF2. It's ubiquitous and time-tested with an academic pedigree from RSA Labs, you know, the guys who invented much of the cryptographic ecosystem we use today. Like bcrypt, PBKDF2 has an adjustable work factor. Unlike bcrypt, PBKDF2 has been the subject of intense research and still remains the best conservative choice.

There has been considerably less research into the soundness of bcrypt as a key derivation function as compared to PBKDF2, and simply for that reason alone bcrypt is much more of an unknown as to what future attacks may be discovered against it. bcrypt has a higher theoretical-safety-to-compute-time factor than PBKDF2, but that won't help you if an attack is discovered which mitigates bcrypt's computational complexity. Such attacks have been found in the past against ciphers like 3DES. Where 3DES uses a 168-bit key, various attacks have reduced that key size's effectiveness to 80-bits.

PBKDF2 is used by WPA, popular password safes like 1Password and LastPass, and full-disk encryption tools like TrueCrypt and FileVault. While I often poke fun at Lamer News as a Sinatra antipattern, I have to applaud antirez on his choice of PBKDF2 when he got bombarded with a "just use bcrypt!" attack (although bro, antirez, there's a PBKDF2 gem you can use, you don't have to vendor it)

The second cipher to consider is scrypt. Not only does scrypt give you more theoretical safety than bcrypt per unit compute time, but it also allows you to configure the amount of space in memory needed to compute the result. Where algorithms like PBKDF2 and bcrypt work in-place in memory, scrypt is a "memory-hard" algorithm, and thus makes a brute-force attacker pay penalties both in CPU and in memory. While scrypt's cryptographic soundness, like bcrypt's, is poorly researched, from a pure algorithmic perspective it's superior on all fronts.

The next time you need to pick a key derivation function, please, don't use bcrypt.

Announcing Lightrail: Lightweight Rails stack for HTML5/JS applications

noreply@blogger.com (Tony) — Thu, 08 Mar 2012 17:20:00 +0000

There's been a lot of debate lately surrounding Rails suitability for the server stack underlying modern HTML5/JS applications. Having used Rails for some four years for this purpose, and worked with a number of Rails core members, this is a problem I think the Ruby community has solved wonderfully, but yet some are confused as to what solutions are available or the way forward.

Rails 3 provided enormous advances in terms of letting you specialize what Rails provides to the problem at hand. However, to a certain extent this goes against the Rails mantra of "convention over configuration". While Rails 3 provides ample opportunities for configuration, as Rubyists, we shouldn't have to configure anything, right?

I'm a huge believer in both Rails' suitability as a backend for modern client-heavy HTML5/JS applications, and someone experienced in building such applications. Rails is, was, and continues to be a game-changer for modern web development. ActionController::Metal provides the bare minimum needed to build apps which don't need the complete set of HTML-generating abstractions provided by ActionView, but need more tools than a more minimalistic framework like Sinatra makes available. Between Sinatra and Rails lies an unaddressed middle ground, one where, in theory, you should be able to build an ActionController::Metal stack appropriate to your needs, but maybe this is too daunting a task.

For you, the JSON API builder, who wants more than Sinatra but less than Rails... I have what you desire. Introducing Lightrail:

What is Lightrail? Lightrail is Strobe's ActionController::Metal stack for HTML5 applications, originally used to provide the backend APIs for Strobecorp.com and its frontend HTML5/JS application authored with SproutCore (which has been superseded by Ember.js).

Lightrail contains everything you need to build lightweight applications on the Rails stack which serve only JSON, and furthermore, contains an innovative system for building JSON APIs around your objects. Rather than adding a fat #to_json method on your models, or using a template to construct your JSON, Lightrail's allows you to map the JSON serializations of your objects to specific wrappers that know how to serialize specific objects. Like using #to_json, this makes it easy to recursively serialize nested objects to JSON without having to use ActionView voodoo like invoking other renderers. Besides that, it still separates the concerns of what your domain objects are and how they serialize to JSON. If you've tried to build JSON APIs in Rails and found the existing mechanisms for JSON serialization lacking, please try out Lightrail::Wrapper and let me know what you think.

Lightrail is something of an experiment. I didn't write it, but it's software I believe in so much I'd like to support it and see if people are interested in it. Rather than competing with Rails, Lightrail takes the latest, greatest Rails stack and reconfigures it for lightweight applications that provide a JSON API exclusively. Lightrails builds upon all of the modularity that Rails 3 brings to the table, and simply and easily delivers a lightweight stack which is still suitable for complex applications.

Please let me know if Lightrail seems like a good idea to you and if you'd like to help support it. As I have my hands in an awful lot of other open source projects, Lightrail isn't the sort of thing I can support full time. However, if you have some time to spare and ideas to contribute, I am definitely looking for people to help maintain and improve this project.

If you're interested in using Lightrail or helping out with its development, sign up for the mailing list. Just send any message to lightrail@librelist.com to join.

Why critics of Rails have it all wrong (and Ruby's bright multicore future)

noreply@blogger.com (Tony) — Mon, 05 Mar 2012 16:52:00 +0000

Edit: Contrary to what I said here, José Valim is not stepping down from Rails core, he is merely on sabbatical. My bad.

Lately I've been getting the feeling the Ruby community has gotten a bit emo. The enthusiasm surrounding how easy Ruby makes it to write clean, concise, well-tested web applications quickly is fading. Rails has become merely a day job for many. Whatever hype surrounded Rails at its inception has died down into people who are just getting work done.

Meanwhile, Node.js is the new hotness, and many in the Node community have sought to build Node up by bringing Ruby and Rails down. I know that once upon a time Ruby enthusiasts were doing this sort of thing to Java. However, the tables have turned, and where Ruby used to be the mudslinging hype-monkey, it's now become the whipping boy and Node.js the new provocateur.

The sad thing is many of these people are former or current Rubyists who have taken a liking to Node and build it up by spreading blatant untruths about Ruby. I won't go as far as to call them liars, but at the very least, they are extremely misinformed, ignorant of the state of the Ruby ecosystem, and pushing their own agendas.

Jeremy Ashkenas, the creator of CoffeeScript, recently trashed Rails 3 and claimed "Node.js won":

The idea that Rails 3 was a major step backward was recently reiterated by both Giles Bowkett and Matt Aimonetti. Both of them painted building ActionController::Metal applications as some sort of byzantine, impossible task which can only be accomplished by a Rails core member. Are people actually building lightweight Rails applications using the newfound modularity of Rails 3?

Jose Valim, (now former) Rails core member, published a small, simple gist illustrating how to build barebones apps on ActionController::Metal (one of the most forked gists I've ever seen) which is further documented in his book Crafting Rails Applications. In just 50 lines of code you can strip Rails down to its core, making it ideal for use in modern client-heavy HTML5 applications. The funny thing about this gist is that while the idea of a 50 line Rails app seems pretty impressive, the basis of that gist is what Rails 3 puts into your config/boot.rb, environment.rb, and application.rb, just combined into a single file. Did I just blow your mind? Sadly, all the (in my opinion completely undeserved) bad press seems to have made Jose emo as well, and he has stepped down from Rails to pursue his Elixir language.

ActionController::Metal-based applications (along with apps written in Clojure) were the basis of our backend at Strobe, where we sought to ease the pains of people building modern client-heavy HTML5/JS applications with frameworks including SproutCore/Ember, Backbone, and Spine. ActionController::Metal provided a great, fully-featured, mature, and modular platform for us to build applications on top of, and Strobe's ActionController::Metal stack for client-heavy HTML5/JS applications is available on Github. The apps we built with the Strobe ActionController::Metal stack talked only JSON and our frontend was an HTML5/JS application written with SproutCore.

Before Strobe, I worked at a company building rich HTML/JS applications for digital television. Our backend was written in Rails. Our frontends were Flash and HTML/JS applications, the latter of which were single-page client-heavy HTML/JS apps that were packaged in .zip files and installed onto digital televisions and set top boxes, a sort of weird hybrid of web technologies and installable applications. Our Rails application didn't have any views, but provided only a JSON API for the HTML/JS frontend to consume.

Rails was great for this, because it provided the sort of high level abstractions we needed in order to be productive, ensure our application was well-tested, and above all else provided the necessary foundation for clean, maintainable code. I was doing this in 2008, and even then this was already considered old hat in the Rails community. In case you're not paying attention, that's one year before Node even existed.

Modern HTML5/JS apps depend on beautiful, consistent RESTful JSON APIs. This is a great way to develop rich interactive applications, because it separates the concerns of what the backend business logic is doing from the view layer entirely. Separate teams, each specialized in their role, can develop the frontend and backend independently, the frontend people concerned with creating a great user experience, and the backend people concerned with building a great API the frontend application can consume.

Rails is great for JSON APIs.

And yet this meme persists, that somehow Rails is actually bad at JSON APIs. Those who propagate this meme insist that Rails has lost its edge, and that only Node understands the needs of these sorts of modern client-heavy web applications. Giles recently attempted to argue this position:

Giles recently blogged about this issue at length. Let's look at what he has to say about ActionController::Metal and the new level of modularity and clean design that Rails 3 brings to the table:

So Jose wrote a great book about the incredible power of Rails 3's new modular APIs... but... but... but what?

WARD CUNNINGHAM BITCHES. TWEETS > BOOKS. NODE WINS. QED.

Hurrrrrrrr? Ward Cunningham is a cool guy and his concept of a Wiki was a transformative technology for the web, but what the fuck does that have to do with Rails 3's new modular APIs or Jose's book? I think that's what people in logical debate circles call a "non-sequitur".

Perhaps there's still a cogent argument to be had here. Let's dig deeper:

Okay, so the problem is there's not a damn simple way to do websockets. OH WAIT, THERE IS:

Cramp is an awesome, easy-to-use websockets/server-sent events framework (with socket.io support) which runs on Rainbows or Thin, and Thin is a great web server. According to my benchmarks it's approximately the same speed as Node's web server:

Web Server            Throughput  Latency
----------            ----------  -------
Thin    (1.2.11)      8747 reqs/s (7.3 ms/req)
Node.js (0.6.5)       9023 reqs/s (7.1 ms/req)

Yes folks, Node isn't significantly faster than Ruby at web server performance. They're about the same.

Giles also bemoans bundler, because typing "bundle exec" represents ceremony, and using any of the myriad solutions to avoid typing "bundle exec", such as bundler binstubs or rvm gemsets, represents configuration which violates the Rails mantra of "convention over configuration", and how npm is that much easier. I'm sure we would all love to not have to add a one line .rvmrc file to each project to avoid typing "bundle exec", but uhh Giles, bro, mountain out of a molehill much?

Meanwhile, let's check out how convention over configuration is going in the JavaScript world:

But enough about Giles... what kinds of awesome, modern HTML5 applications are people using Rails to build?

I think one of the best examples of this sort of application is Travis CI. Travis is an open source distributed build system with an Ember-based frontend and a Rails backend. Travis's interface shows, in real time, the state of all builds across the entire (distributed) system, allows you to interactively explore the history, see the distributed build matrix completing jobs in realtime, and even have it stream the console output of builds in progress directly to your browser as they complete. It's an amazing, modern client-heavy HTML5/JS application, and it's built on Rails.

Who else is using Ruby/Rails for their frontend? Oh, just Twitter, LivingSocial, Groupon, Heroku, EngineYard, Github, Square, Zendesk, Shopify, Yammer, Braintree, Boundary, Stripe, Parse, Simple, and of course let's not forget 37signals. Rails is the technology underlying the frontend web stack of many huge businesses. Many of these companies have client-heavy HTML5/JS applications which consume a JSON API coming out of Rails. Many of them have APIs that are routinely cited as archetypical RESTful JSON APIs. Many of them have top notch engineering teams that choose the best tools for the job and use many languages for many different tasks. Many of them were founded "post-Node" and had the opportunity to choose Node as their frontend web technology, and while they may use Node in some capacity, their main consumer-facing sites are written with Rails.

Node is three years old now. Where are the Node.js success stories? Who's built a brand on top of Node? Nodejitsu? Hubot? Is Node anything more than a pyramid scheme or a platform for Campfire bots? Where Rails selling points eschewed performance and instead focused on clear code, rapid development, extensive testing, and quick time-to-market, Node's selling points seem to universally revolve around its insanely fast, destroy the internet fast performance (benchmarks not provided). Meanwhile code quality is de-emphasized and large Node programs degrade into incomprehensible, byzantine structures of callbacks and flow-control libraries, instead of being written in sequential code, you know, the code you can read:

What about Ruby in general? What advancements in the Ruby ecosystem are worth getting excited about?

JRuby is maturing into a high-performance Ruby implementation which taps the JVM's advanced features including the HotSpot compiler, multiple pluggable garbage collectors, and parallel multithreading which makes it suitable for multicore applications. One thing I think sets JRuby apart is that it's the most mature language on the JVM which didn't start there. Other projects to implement non-JVM languages on top of the JVM, such as Rhino and Jython, have languished, while JRuby keeps going strong.

The most exciting development in JRuby is Java 7's new InvokeDynamic feature. The Java Virtual Machine was originally designed for the statically-typed Java language, but has its roots in dynamic languages, namely Smalltalk. With InvokeDynamic, the JVM has come full circle and now natively supports dynamic languages like Ruby. InvokeDynamic provides the necessary information to the JVM's HotSpot compiler to generate clean native code whenever Ruby methods are called, in addition to many other potential optimizations. So how much faster will InvokeDynamic make Ruby?

Rubinius, a clean-room Ruby virtual machine based on the Smalltalk-80 architecture, is also a very exciting prospect for the Ruby community as it matures and reaches production quality. It features an LLVM-based JIT compiler, parallel thread execution, and advanced garbage collection, also making it suitable for multicore applications. Beyond being an awesome Ruby implementation, Rubinius has evolved into a true polyglot platform and now features multiple Rubinius-specific language implementations including Fancy and Atomy.

MacRuby also eliminated the GIL from their implementation and now supports parallel thread execution along with an LLVM-based JIT compiler.

There are no less than three Ruby implementations which now support thread-level parallelism and thus multicore CPUs. This is especially relevant in a time when computing is undergoing a sort of phase transition from single-threaded sequential applications to massively multithreaded concurrent applications and distributed systems made out of these multithreaded applications.

It wasn't too long ago that having even four CPU cores in your home computer seemed like a lot, and now 16-core commodity AMD CPUs are available. The future is multicore, and if your programming language doesn't have a multicore strategy, its usefulness is vanishing. Following Moore's Law, the number of cores in a CPU is set to explode exponentially. Is your programming language prepared?

Thanks to JRuby and Rubinius, Ruby can take advantage of multicore CPUs. This still leaves the small matter that multithreaded programming is, uhh, hard. Fortunately I have some ideas about that.

Celluloid is an actor-based concurrent object system that tries to pick up on the concurrent object research that was hot in the mid-90's but died shortly after the web gained popularity. In the '90s concurrent objects were ahead of their time, but with the advent of massively multicore CPUs I believe it's an area of computer science research that's worth reviving.

Celluloid packages up Ruby's core concurrency features into a simple, easy-to-use package that doesn't require any modifications to the language. Where many functional languages solve the issues surrounding concurrency with immutable state, Celluloid solves it with encapsulation (more information is available on the Celluloid github page).

Celluloid takes advantage of many of the features of Ruby, including parallel threads, fibers (coroutines), method_missing (proxy objects), and duck typing. There aren't many other languages with this particular mix of features. Python probably comes the closest, aside from multicore execution due to its GIL. Jython supports parallel thread execution thanks to the JVM but seems abandoned. For what it's worth, Python once had a concurrent object system quite similar to Celluloid back in the '90s called ATOM, unfortunately the source code has been lost.

Ruby is by far the best language available today to implement a system like Celluloid, and that alone makes me excited to be a Rubyist. Where Node.js gives you a hammer, the single-threaded event loop, Celluloid gives you a large toolbox and provides a singular framework of interoperable components which can be used to build arbitrary hybrids of concurrent multithreaded applications, event-based nonblocking applications (that are callback-free!), and distributed systems.

Ruby is a language which can survive the massively multicore future. Whether Node will stick around remains to be seen.

"Can you solve this problem for me on the whiteboard?"

noreply@blogger.com (Tony) — Tue, 31 Jan 2012 23:30:00 +0000

Jim is a great chef. He's too modest to say that about himself, but he's worked either as head chef or assistant head chef at a number of restaurants. Everywhere he's worked he's been dependent and reliable, prepared great food, worked well with the other chefs, and is generally a fun guy to have in the kitchen. Unfortunately, due to the poor economy and some bad decisions by management, Jim's restaurant is about to close, so Jim is out of work and looking for a new job.

There's a new restaurant opening, a fancy place with many well-to-do investors. In Jim's world, chefs are hard to find, so Jim assumes he's a shoo-in for the job. Jim arrives at the interview at a Mexican restaurant, which feels like a great fit for Jim because Mexican food is his specialty. Jim calls up the restaurant on the phone and chats with the manager about a chef position, and the manager likes what he hears enough to schedule a job interview for Jim.

Jim arrives at the interview and talks to the manager a bit. Things seem to be going well, Jim is in his element at a Mexican restaurant. The initial meeting goes well: Jim talks his job history, how much he cares about having a fresh house salsa, and how good his Baja sauce is. "Look up the Yelp reviews of my Baja sauce!" remarks Jim. "It's the #1 reason people came to the last restaurant I worked at." The manager smiles and nods, and informs Jim he looks great on paper, however the remainder of the interview will be conducted by all of the other chefs in the kitchen. "Awesome!" Jim thinks, "I have a rapport with other chefs. This should go smoothly."

The first chef walks in, sits down at the table, and coldly stares at Jim's resume. "Can you write down a recipe for me?" he asks Jim, "There's a whiteboard over there, can you write down your preferred recipe for crème brûlée?"

Jim is a bit dumbfounded, both by the request and being asked to demonstrate his cooking ability on a whiteboard. "I'm sorry," he says, "I don't know how to make crème brûlée. I thought this was a Mexican restaurant. Would you like to know my favorite recipe for Flan?"

"No, that won't do," the assistant chef says. "Please write down how you would prepare crème brûlée"

Jim is a bit taken aback, first because he's a specialist in Mexican food, and second because instead of being asked to cook, he's being asked to write stuff on a whiteboard. "I honestly don't know how to make crème brûlée," Jim says. "Perhaps you could let me google the recipe and I could actually try to prepare it for you, instead of just demonstrating a rote ability to memorize recipes and write them down on a whiteboard."

"No, that won't do," says the interviewer, who jots down "lack of confectionary skills" in his notes. "Can you at least attempt to write down how you would prepare crème brûlée?"

Jim feels embarrassed and lost. He's being asked to do something he would never have to do in a professional capacity, and worse, rather than actually doing it, he's being asked to describe how he would do it on a whiteboard. Perhaps this is a test of Jim's ability to think on his feet, but given the position he's being asked to interview for and the question he's been presented, it's certainly an unfair one. Jim picks up the black marker and thinks hard about what the possible ingredients of crème brûlée would be.

"Well," says Jim, "I'll need cream." Jim pulls the cap off the marker and attempts to write "1. Cream", however the marker is dry and the whiteboard is on wheels that roll back when Jim attempts to write. Jim only succeeds in making a long, barely perceptible mark on the whiteboard. Having made a messy mark on the whitebard, Jim looks for an eraser but there isn't one.

"Yes," says the interviewer sarcastically, rolling his eyes, "obviously you need cream for crème brûlée. Try a different marker." Jim picks up the red marker and tries to write with that to the same result, it's dried out and won't work. Frustrated, Jim puts it down and tries the green marker, which works fine, however the board swivels vertically as he tries to write. Jim grabs the board in the upper right corner and finally manages to jot down "1. Cream"

"Okay, we have the most obvious ingredient down," says the assistant chef. "Can you think of any other ingredients that would go into crème brûlée?"

"Sugar," says Jim. The assistant chef nods, and Jim writes down sugar. "What else?"

"Milk," says Jim, and he begins to write it down before he comes to the realization that the cream and milk are redundant. Jim doesn't often cook with cream. The interviewer shakes his head in exasperation and pinches the bridge of his nose as Jim looks dumbfounded. "It's not milk brûlée," he says. Unfortunately, there's no eraser, so Jim tries to erase "3. Milk" with his hand, smearing green ink all over the board and his hand before asking "do you have an eraser?" The interviewer looks around unenthusiastically before shrugging no. Jim continues smearing the marker's ink across the surface of the board with his fingertips in a desperate attempt to compensate for the absence of an eraser.

"Can you think of any other ingredients that might go in crème brûlée?" asks the interviewer, clearly bored.

"Eggs?" asks Jim. The interviewer nods. Jim writes down "eggs". "What else?" the interviewer asks. Jim stares at what he's written down: cream, sugar, eggs. "Well," says Jim, "I assume some kind of flavoring. Chocolate perhaps?"

"Wrong," says the interviewer. "Please write vanilla." Jim looks confused for a second and jots down vanilla as asked. The interviewer jots down "trouble with basic recipes" before asking "What other ingredients can you think of?"

Jim stares at the ingredients so far: cream, sugar, eggs, vanilla. "Perhaps some water?" Jim guesses. The interviewer nods, and Jim writes down water. "Now, what are you missing?" asks the interviewer.

Jim stares at the list: cream, sugar, eggs, vanilla, water. Those seem like they should be the basic ingredients, and the interviewer rejected additional flavoring that wasn't vanilla. Jim is stupefied... he can't think of anything else. Taking a stab in the dark, Jim suggests "Salt?"

The assistant chef does a facepalm and sighs, before looking up at Jim and stating the obvious solution: "the units. Your recipe is lacking units." The ambiguity of the interviewer's question has caught Jim off guard, especially when he professed no idea of what the recipe was to being with, and worse, he has absolutely no idea what the units should be. He stares at the whiteboard for awhile before asking "how much crème brûlée are we making?"

"That's up to you," says the interviewer, "how many servings would you like to prepare?"

Jim has absolutely no clue. He's not a confectioner, but he doesn't want to completely bomb the interview, so he ventures a guess. "I'd like to prepare 2 servings. Let's try a cup of cream, a teaspoon of vanilla, two tablespoons of sugar, 4 eggs, and a cup of water."

"Those aren't the right proportions," say the interviewer. "You should use a quart of cream, two quarts water, a teaspoon of vanilla extract, a cup of sugar, and six eggs to produce six servings. Let's move on to the recipe. Can you write it down on the whiteboard for me?"

Now Jim is completely lost. The ingredients of a recipe he has no clue about are something he can guess at, but how is he supposed to guess the recipe itself? He takes his best shot.

"Break the eggs into a bowl and whisk them with the cream and sugar," guesses Jim.

"Wrong," says the interviewer.

"Whisk them with the cream and vanilla?" asks Jim.

"Still wrong," says the interviewer, "but you were closer the first time."

"Do you want me to keep guessing?" asks Jim. The interviewer sighs, writes down "completely incompetent", stands up, and says "Thank you for your time. I'll go get the next person."

Jim stands by the whiteboard and feels confused and out of place. He wonders what crème brûlée has to do with preparing Mexican food. He sits down at the table and googles for crème brûlée on his phone, quickly scanning over the recipe and thinking "that doesn't look too hard at all, I could probably make a great crème brûlée if I had a little practice." The recipe for crème brûlée is in fact quite similar to Flan, and Jim can make great Flan, but unfortunately, the interviewer won't even know as he hasn't asked Jim to cook anything. The next interviewer comes into the room.

He sits down at the table and scans over Jim's résumé, making a few grunts after scrutinizing various items. "You didn't go to culinary school?"

"No," says Jim, "but I've loved cooking since I was a little kid. I used to cook dinner with my mom every night. I've been working professionally as a chef all my life, and I can prepare great food. Why don't you just take me to the kitchen and let me show you?"

"That won't be necessary," says the interviewer. "Now, can you please write on the whiteboard how you would prepare a cheese danish?" Unfortunately, Jim is not a pastry chef either.

. . .

The manager has returned to conclude the interview. "Well Jim," he says, "we've discussed the issue, and we don't think you'd be a good fit here."

At this point Jim is entirely expecting this response. Jim is most comfortable in a kitchen, preparing food hands on. He feels out of place trying to explain the theoretical act of preparing food with a whiteboard. Jim loves food so much that whenever he went out for a smoke break with his fellow chefs, he continued to talk about food even when they were on break. Unfortunately, during the interview he didn't get the opportunity to discuss food in this sort of context. Instead he was asked only pointed questions about food items he didn't know how to prepare.

"I see," says Jim. "Can I ask you one question before I go?"

"Okay," says the manager.

"Throughout this interview," Jim asked, "I was asked about preparing confections and pastries, but not once was I asked about preparing Mexican food. I thought this was a Mexican restaurant. Do you serve confections and pastries here?"

"No, we don't prepare confections or pastries," said the manager, "however we're all classically-trained pastry chefs. Some of the people you talked to are actually pretty new to Mexican food. But they've all gone through culinary school and have impeccable cooking skills because of it."

"Have you considered asking your candidates to actually cook instead of explain how they would theoretically prepare something on a whiteboard?"

"It's a lot easier for us to just use the whiteboard," he says, "and we want candidates who are as knowledgable about the theory of cooking as the act of cooking."

Jim is extremely frustrated. It's not that he isn't knowledgable about the theory of cooking, but he hasn't memorized the recipe to every foodstuff on earth. Confectionaries and pastries are two areas that Jim knows very little about.

. . .

Jim arrives for another interview at another popular Mexican restaurant. On his way in he notes the health inspector's grade on the certificate displayed on the window: a 100%! Jim doesn't think he's actually seen a 100% score before. Jim walks in and the manager is actually there to greet him for his interview. Jim's actually pretty close with the manager, having seen him around at various farmers markets, concerts, and other events, and Jim wonders why he went to see those crazy pastry chefs before coming here.

"Hey Jim," the manager says, "I ate at your restaurant a few times. The food there was delicious!"

"Did you try the baja sauce?" asks Jim, "because I made that myself."

"Yes!" exclaims the manager, "the Baja sauce was so orgasmically delicious! Now I hope you don't mind, but we have a little test prepared. Come with me, please."

The manager leads Jim into their state of the art kitchen. It's hopping on a busy night, with people everywhere preparing the various menu items the restaurant has to offer. The order management system is fully automated using LCD displays which are mounted on the ceiling, tracking which items have been ordered, prepared, and served. The kitchen looks extremely clean and modern and the workflow seems highly efficient. The manager continues leading Jim around and shows him a prep area in the back of the kitchen which is unused. "You can work here," he says, "come with me and you can get your ingredients."

The manager continues leading Jim back to their refrigerator, where Jim notices an LCD display showing a realtime graph of the refrigerator's temperature, with bars for "too hot" and "too cold". Jim also notes in the visible history the temperature has remained within the guidelines the display is showing with very little alteration.

The manager pulls the latch to the door on the refrigerator and Jim feels a whoosh of cold air. Inside Jim finds a cornucopia of ingredients. Jim grasps some cilantro and inhales it, and the smell is deliciously fresh. Jim darts about the refrigerator taking inventory, and discovers all the requisite ingredients are in place to concoct his own trademark Baja sauce.

"I know you can make awesome food," says the manager, "but you need to convince the owners you're a good chef. You have an hour," says the manager, "Your goal is to make delicious Mexican food."

. . .

58 minutes later the two co-owners of the restaurant have arrived along with the manager and have come to the back of the kitchen where Jim has been spending his time. He introduces himself and shows them the food he's prepared.

Jim has prepared some Baja fish tacos made of battered and fried red snapper, topped with Jim's own Baja sauce freshly made on-the-spot using only ingredients from the restaurant's well-stocked refrigerator. "I'm sorry it took so long," Jim says, "but really I spent 40 minutes making the sauce, and 10 minutes actually making the tacos"

The owners and the manager each grab one of Jim's tacos and bite in. They're unbelievably delicious, and it's all thanks to Jim's baja sauce. In his moment of triumph, Jim thought back to the first restaurant where he interviewed, and wondered why they were so caught up on their pastry-making ways. Clearly it takes a different kind of chef to make fish tacos than to make pastries, and perhaps Jim wasn't cut out for being a pastry chef. But when it came to making Mexican food, Jim was in his element. It seemed really weird that former pastry chefs-turned-owners of a Mexican restaurant would expect him to be a competent pastry chef, but perhaps that's what they're used to.

. . .

If you haven't already seen through the thinly-veiled allegory, I'm describing an interview process that based on my experience has become incredibly common in the Silicon Valley. I'm not going to name names, first and foremost because I've signed NDAs, but to those of you who have a rigorously whiteboard-driven interview process, I can't comprehend what you're doing. At the very least, if you're asking me to write on a whiteboard, make sure you have good markers and good erasers. That said...

Programmers use computers. It's what we do and where we spend our time. If you can't at least give me a text editor and the toolchain for the language(s) you're interested in me using, you're wasting both our time. While I'm not afraid of using a whiteboard to help illustrate general problems, if you're asking me to write code on a whiteboard and judging me based on that, you're taking me out of my element and I'm not giving you a representative picture of who I am or how I code.

Don't get me wrong, whiteboards can be a great interview tool. Some of the best questions I've been asked have been presented to me on a whiteboard, but a whiteboard was used to explain the concept, the interviewer wrote down what I said and used that to help frame a solution to the problem. It was a very different situation than "write code for me on the whiteboard."

Want to do better? Give a programmer a computer. Programmers like computers. Install common editors like vim, Emacs, and TextMate and let someone choose what they're most familiar with. Better yet, give them Internet access, or even let them use their own laptop. If you're looking over their shoulder the entire time, they can't "cheat" on the interview, and maybe you'll learn something new about their workflow and how they develop software. Who knows, maybe they have a better programming workflow than you which you can only discover by watching how they work on their computer. Limiting potential programming hires to a medium like a whiteboard is a degrading experience, and one that doesn't give you an indication of a person's potential.

Last but not least, treat your potential hires like people, because they are people. Take the time to get to know your potential hires before their interview. If they've developed open source software and gained some notoriety for it, that should be a major factor in your decision, more so than what you can distill from a cursory whiteboard interview. Bottom line, if you're interviewing someone for a software engineering position, and they have a Github, and you haven't spent at least 10 minutes familiarizing yourself with what's on their Github account before you even talk to them, you're doing yourself and your company a disservice.

My worst experience (still not naming names, but you know who you are) was a company specializing in Ruby who, not to beat around the bush, was the inspiration for this whole blog post. My first in-person conversation with someone technical at the company was a nonstop wall of coding questions on a whiteboard, with no preliminary discussion of what kind of people we are or what wavelength we're on. The entire interview was conducted in an interrogation of "solve my problems or I won't give you the job." This style is completely degrading to the person being interviewed. It's computer science trivia where the prize is a job. I'm sorry, but winning a computer science trivia contest isn't a good way to gauge potential employees.

Call it sour grapes if you want, but if you're the company I interviewed with and you're reading this, and remember who I am, and remember interviewing me, I think you missed out. And I think it's your fault, which is bad because I wanted to believe in your company. I hope you're not surprised if you never heard a word back from me.

My attitude is if I'm a good Ruby programmer, and you're trying to hire me when the supply for Ruby programmers is low and demand is high, that before you even talk to me you've spent at least 10 minutes Googling for my name, looking at my code, and figuring out who I am, rather than spending an hour subjecting me to a series of ad hoc programming questions in areas I may or may not specialize in. That 10 minutes of Google will tell you a lot more than asking me to come in and scribble stuff on a whiteboard.

I think this process has left me a bit more discerning about the companies I'll actually interview with. When you're trying to hire talented developers in a scarce market, please do your due diligence and don't insult somebody skilled by asking them to do a degrading whiteboard interview instead of looking at code they have freely available on the Internet or just looking over their shoulder as they code on a computer, preferably their own, at least the first time you get to know them. You may even learn something.

Goodbye 2011!

noreply@blogger.com (Tony) — Sun, 01 Jan 2012 00:23:00 +0000

I don't usually write a year-end retrospective blog post but I thought 2011 was a pretty interesting year with many noteworthy events we should all take notice of and keep in mind. The events were of a sort that several science fiction authors, particularly of the cyberpunk variety, have been writing about since the '80s, and now it feels like their visions are materializing.

Let's start with the Japan earthquake disaster and Fukushima. I'm something of a Japanophile, so it was very saddening to see the devastation wrought by the earthquake and the ensuing tsunami. The Fukushima nuclear disaster was very sad to see for many reasons. First because of all the people it hurt/killed, particularly all the volunteers who selflessly exposed themselves to radiation trying to keep the nuclear reaction occurring in the broken remains of the reactor's core under control. I'm also saddened by the devastating effect this will have on public opinion about nuclear power, at a time when liquid thorium nuclear reactors seem like a promising near-term solution to the world's energy problems. The Fukushima disaster reminded me of the Miyazaki-animated music video for Chage and Aska's "On Your Mark," which takes place in the shadow of an entombed nuclear reactor.

To me at least, "Wikileaks" was the story of the year. Bradley Manning managed to sneak confidential diplomatic cables out of a secured government facility by burning them onto a CD-RW previously filled with Lady Gaga. Wikileaks began disclosing them, Julian Assange rose to celebrity because of it, and Freenet failed to fulfill its goal of being a useful platform for exactly this use case. I included the photo above which I took on the way to work one day (complete with tacky Instagram filters), which really makes me remark, "why not 'Free Bradley Manning?'"

The collateral damage the Wikileaks disclosure had on the world as a whole was absolutely remarkable. I think the disclosure of the diplomatic cables is the catalyst for the Arab Spring, in which dictators fell like dominoes. As terrible as the atrocities that occurred during these struggles were, I can't help but feel that some positive progress was made in the world, even when the future seems incredibly uncertain such as in Egypt.

BitCoin certainly caught my attention. The world now has a workable p2p currency scheme using crypto-coins as in the Cryptonomicon (except sans the gold backing). It's anyone's guess as to where it will go at this point, but it's a remarkable demonstration of the power p2p systems have to solve large-scale general problems in theoretical terms. Whether BitCoin proves to be a practical real-world currency remains to be seen. Critics have equated the value of the currency to the hype surrounding it, comparing it to things like beanie babies.

Both Anonymous and LulzSec made this year extra surreal. If nothing else, 2011 will be remembered as a year that many institutions started paying a hell of a lot more attention to infosec. It's a lesson Sony will certainly remember after the entire PlayStation Network was hacked and taken down. I thoroughly enjoyed the techno-anarchist spin that LulzSec originally put on their hacks, claiming quite ostensibly that they did it "for the lulz," and was somewhat disappointed to see them turn political after that. Still, the entire experience of having them in the world certainly felt like something out of a cyberpunk novel.

Steve Jobs died. It's really hard to imagine the scope of the effect the guy had on my life. Love him or hate him, Steve Jobs created the company that made the first home computer I ever used. WindowMaker, the window manager I used for nearly a decade before switching to OS X, was inspired by NeXTstep. At the end of 2011, I have a computer, tablet, and phone all running evolutions of NeXT's OS. The software I'm writing, DCell, also follows in NeXT's footsteps. I have a half-written blog post about that I hope to get published soon.

The IPv4 address space was exhausted. It seems like IPv6 will actually become a reality soon, forced by the increasing costs of obtaining IPv4 addresses when they're scarce due to the technology's limitations.

I saw an entire string of announcements in cancer and AIDS research this year that sounded extremely promising. The announcements spanned everything from using HIV to cure cancer to a vaccine for HIV comparable to the polio vaccine. The ability for modern medicine to cure disease never ceases to astound me and I hope that we will see bona fide generalized cures for both cancer and AIDS quite soon.

One of the craziest things I saw this year was a NOVA special on immortality (available on Netflix, btw). They had footage of human organs grown in vitro, first by taking an organ from a donor (which could be an animal), using a special chemical to wash away everything but the cartilage, then using someone's own stem cells to grow a replacement organ from the cartilaginous skeleton of a particular organ. The special showed lungs that breathe and a heart that beats. The future of organ replacement will be growing you a brand-new replacement from your own cells, using the cartilage structure from a dead animal (or potentially in the future constructing the cartilage from a 3D blueprint).

The Large Hadron Collider supposedly found the Higgs boson. We'll see what future experiments say, but it seems like the standard model has been validated and the particles we expected to be there exist.

Oh yes, we got Osama. AMERICA! Fuck yea!

Occupy Wall Street emerged, almost a liberal reaction to the Tea Party, a movement whose anger wasn't centered directly on the government, but on the financial institutions who made out like bandits with TARP money. No major financial reform has taken place, and the institutions which were too big to fail are still too big to fail, and getting back to the same tricks which tanked our economy in the first place. I'm not sure when the US government is going to eventually make the decision to try to better regulate Wall Street, but I can't see it happening before the 2012 election.

Speaking of 2012, happy 2012 everyone! The Mayans and Terrence McKenna believed that the world is going to end on December 21st. Let's hope humanity has many, many more years ahead of it :)

As for myself, I had quite the crazy year. I went to Switzerland and got back home to find my basement flooded. I lost two jobs and got two new jobs after half a decade of job security. I created Celluloid, stopped working on Reia (guess I should update my blog header). I moved to San Francisco, where I met Jose Valim, the scumbag who started working on a language just like Reia (just kidding Jose) who threw it away and started working on a much cooler version of the same language. I also met Chris Bertels, Steve Dekorte, Yehuda Katz, Carl Lerche, Ben Reesman, Tom Dale, Colin Campbell, Majd Taby, and Slava Pestov. I stuck it to Erlang as a language and Node as a community. And I happened to have about 4 months of time spent unemployed where I practically worked on Celluloid, DCell, and surrounding projects full-time. It was pretty nice.

Happy new year everyone!

Node.js has jumped the shark

noreply@blogger.com (Tony) — Tue, 04 Oct 2011 16:03:00 +0000

I've been trying to understand this whole Node.js thing that's been going on for the past few days. It's taught me a lot about how the Node.js community tries to reason about problems, and perhaps what's the real problem with Node.

It began with a relatively short rant by Ryan Dahl. Ryan talk's about how he's mad at complex software, which is at least an arguable position. I liked his rant, but Ted Dziuba did not.

Ted Dziuba wrote a brilliant troll post which lambasted Ryan and the concept of Node.js in general. I'm a huge fan of Ted's previous trolling, especially his parody TechCrunch site uncov, which trolled TechCrunch before TechCrunch became a parody of itself. Except in the case of uncov, it would be a few years before TechCrunch became a parody itself. With Node.js, the reaction was almost instantaneous.

Ted's argument centered around the problem of doing non-trivial computation within the event loop of a system like Node, and how that would affect the performance of the system. To illustrate this point, he gave an intentionally deoptimized Fibonacci function, along with the boldface caveat "Yes, I know there's a closed form solution. Shouldn't you be in front of a mirror somewhere, figuring out how to introduce yourself to her?"

Reading through the lines of Ted's trolling, his point is "I'm not specifically talking about Fibonacci. I'm talking about non-trivial computation as a general problem in Node." He specifically chose a slow algorithm to illustrate his point. However, what was the Node community's reaction?

One reaction was to go out and implement Ted's intentionally-retarded Fibonacci algorithm in a bunch of different languages and benchmark how fast the VM could optimize away a crappy algorithm. Based on several benchmarks, V8 actually does a comparatively good way of optimizing away the retarded algorithm compared to a lot of other languages. Cool story bro!

In the Reddit comments for this story I first truly gazed into the abyss of Node.js insanity. Node provides a way to schedule an anonymous function to run the next time it's event loop iterates using a function called nextTick. EventMachine (ugh) users may be familiar with this same pattern with EM.next_tick (which needs to die for most use cases, but that's the subject of another blog post)

Redditor "headzoo" suggests that we provide a "non-blocking solution" by unrolling the Fibonacci loop and execute one iteration of the Node.js event loop, calculating one iteration for each I/O multiplexing system call Node is making. When Ted Dziuba asked "Have you taken epoll into your heart?," I think he had no idea of the depths Node people actually have.

I don't really get what's going on in Node people's heads here. I think they seriously believe that throwing computation into the middle of an event loop is a good idea, because Node's event loop is FAST!!#$#! funroll-loops + nextTick = LUDICROUS SPEED!!

This crap is absolutely insane. For each iteration of the loop, Node is making an expensive system call. It's also defeating all of the mechanisms that V8's impressive JIT would use to optimize this problem, as substantiated by the previously mentioned Nodian's roflbenchmark. Clearly this is a very bad idea. But wait, it gets worse!

It wouldn't take long before another Node.js fan put together a Github project for this technique, which also added an "optimization": memoization! In addition to unrolling each iteration of the Fibonacci function to run within the event loop, it now caches the nth result. Now let's throw ApacheBench at it and see how fast Node can serve the precomputed result where n = 100.

Holy balls, it does 5390.22 requests per second serving a memoized value when I ask ApacheBench to request the same memoized value over and over!!#$! TAKE THAT TED DZIUBA! Is it any good? One word: yes. Did we miss the point that this isn't about Fibonacci? One word: yes. (Side note: I second raganwald in adding "Is it any good? Yes." to the top of all my READMEs)

Memoizing each member of the Fibonacci sequence makes this algorithm O(n) in memory. For shits and grins I installed Node-fib and asked it for the 1 millionth Fibonacci number. Node ground away (on a single core of my quad core i7 MacBook) for about a minute before running out of memory, crashing the entire VM.

Okay, so mistakenly implementing an algorithm this pathologically bad when you're trying to prove a point to someone who's just trolling you is pretty bad, right? But it gets worse.

After pointing this out, a member of the Node.js community (post now deleted) suggested I might have an obsolete version of Node with a 1GB heap limit (because Node uses a VM targeted at client-side JavaScript where 1GB heaps are an uncommon use case, but hey, let's throw it on the server!) and that I recompile without the 1GB restriction so that this retarded algorithm can continue eating up all my system RAM. He says this as if this is a good idea instead of, I dunno, using a better algorithm.

Clearly the next logical steps are to store previously computed Fibonacci numbers in MongoDB. Or you could build a system with Socket.IO which caches previously computed Fibonacci numbers client-side using a distributed hash table. Then you'd be well on your way to a multimillion dollar cloud-based Fibonacci number service. Eventually you'll probably want to build in an OpenMP C++ native extension to calculate Fibonacci numbers across multiple CPU cores. That way you'll have a truly roflscale Fibonacci web service.

Object Oriented Concurrency: A Deep Dive into Celluloid (Part 1)

noreply@blogger.com (Tony) — Wed, 28 Sep 2011 07:05:00 +0000

There's a quote from Alan Kay I've often cited as a source of inspiration in my various projects, from Revactor to Reia to Celluloid:

"I thought of objects being like biological cells and/or individual computers on a network, only able to communicate with messages"--Alan Kay, creator of Smalltalk, on the meaning of "object oriented programming"

This conception of objects likens them to things in the world which are very much concurrent. Every cell in your body is running at the same time, and cells communicate with each other using chemical signals. Computers on a network are all running at the same time, sending each other messages via network protocols.

In Alan Kay's conception of object oriented programming, it should be a natural model for concurrency. However, most languages that describe themselves as "object oriented" today bolt on concurrency as an afterthought, just a thin abstraction layer around the concurrency primitives provided by the operating system, such as threads, conditions, and mutexes. The hardware and OS people went through the herculean effort of abstracting hardware-level concurrency into threads, and from there, for the most part the programming language people haven't done much, at least in mainstream object oriented languages.

The exception lies in the functional language domain, in languages like Haskell, Clojure, and Erlang. Erlang, in part inspired by Alan Kay's language Smalltalk, has abstracted concurrency in such as way that users of the language are able to reason about it intuitively, something which can't be said for most users of threads. The key to Erlang's success, its secret sauce, lies in its message-based concurrency system, a system inspired by Smalltalk.

Smalltalk is generally considered to be the forerunner of all object oriented programming languages, but when it comes to taking to heart Alan Kay's conception of a message-based system, Erlang beats the mainstream object oriented languages hands down. Erlang has also incorporated the idea of communicating sequential processes, first seen in languages like Occam. By combining the ideas of Smalltalk with those of communicating sequential processes, Erlang created a system where concurrency is baked into the core of how the language operates, enabling fully isolated processes that talk to each other with asynchronous messages.

If this idea works so well in Erlang, can it be added to other languages in a way that makes sense?

From Actors to Concurrent Objects

Given Erlang was vicariously inspired by Smalltalk, an object oriented language, is there any reason why Erlang's ideas can't be applied to present-day object oriented languages to give them concurrent objects? The way Alan Kay describes objects they should inherently be concurrent, but the way we're used to working with objects is inherently sequential.

A whole slew of projects, including one of mine, have tried to implement Erlang's concurrency model in Ruby. None of them have been popular. In my opinion, this is because these projects (including my own project Revactor) have stuck far too close to Erlang, trying to write an Erlang DSL in Ruby instead of trying to reimplement these concepts in a Ruby-like manner. This makes trying to learn them pretty confusing, to the point where the best way to learn them is to first learn Erlang, then after you understand how Erlang works, you can translate in your head from Erlang into these Erlangy Ruby DSLs. Seems bad...

I read a blog post recently about how in the Smalltalk objects that end in "-er" (e.g. Manager, Controller, Builder, Worker, Analyzer) are "bad." For a long time I've thought much the opposite, that there are fundamentally two different types of objects: those which are inanimate and represent objects in the world, and those which are animate and active and manipulate the inanimate objects. I see the programming world as consisting of two very distinct things: actors, and the objects which they act upon. It's not that objects that end in -er are inherently bad, it's simply that OOP languages (including Smalltalk) have never contained the proper primitive for modeling their behavior.

Celluloid is a new kind of concurrency library. It allows you to model concurrent (i.e. thread-backed) actors in a system but still interact with them as if they were plain old Ruby objects. All you need to do to create concurrent objects in Celluloid is to include the Celluloid module in any Ruby class:

That's it! Just by doing "include Celluloid" all instances of this class you create will not be normal Ruby objects, but instead fully concurrent Celluloid-powered objects. These objects will continue to function just like any other object in Ruby. When you call methods on them they return values. If a method raises an exception, the method call aborts and the exception will continue to propagate up the stack.

I want to emphasize this because I think it's incredibly important: users of concurrent objects should not need to be aware that they're concurrent. They should work as closely as possible to any other plain old sequential objects.

Concurrent Objects in Celluloid

I'm going to help you understand how Celluloid works by first starting from a diagram of normal Ruby object interactions and building up from there. Let's start with a diagram of one method calling another:

Hopefully this diagram is easy to understand. Calls come in, responses go out, never a miscommunication! A call consists of the name of a method to invoke and arguments. The response consists of a value to return, or an exception to be raised. Object oriented dogma would have you think of a call as the caller sending a message to the receiver, which in turn sends a response back.

Now let's look at a concurrent object as implemented by Celluloid:

A quick overview: the caller and proxy objects function like normal Ruby caller and receiver objects in the previous diagram. However, the actor circle (i.e. the giant olive-looking thing) represents Celluloid's encapsulation of concurrency. Each actor runs in a separate thread, and exchange messages with proxy objects through their mailboxes, which are thread-safe message queues. Let's dive in!

All communication between regular Ruby objects and Celluloid's actors passes through a proxy object which intercepts the calls. This proxy is responsible for the central illusion that makes Celluloid's concurrent objects as easy to use as any other object, despite them being off in a distant actor somewhere that you must communicate with via messages through thread-safe mailboxes. The proxy object abstracts that all away, and gives you the familiar object-dot-method syntax that you know and love.

The proxy object translates the method call from a normal invocation into a Celluloid::SyncCall object. This object contains the method to be invoked and the parameters. This message is sent from the proxy to the recipient actor by way of its mailbox (which is an object of the Celluloid::Mailbox class).

Mailboxes work like thread-safe message queues, allowing several pending method invocations that the actor can process one-at-a-time. After a method has been invoked on the receiver object, the return value is sent back to the proxy in the form of a Celluloid::Response object. This value is then returned from the original method call made to the proxy.

This gives us the same behavior as a normal method call. However, normal Ruby objects aren't concurrent. To gain concurrency, we need to go beyond the normal method call pattern.

Adding Concurrency: Async Calls and Futures

Celluloid provides two methods for breaking out of the typical request/response call pattern:

Asynchronous calls: send a Celluloid::Call to a recipient actor, but doesn't wait for a response. The method call to the proxy returns immediately. To make an async call, add "!" to the end of any method.
Futures: allow the value a method returns to be obtained later, while that method computes in the background. To make a future call to an actor, use the #future(:method_name, arg1, arg2... &block) syntax. This returns a Celluloid::Future object, whose value can be obtained by calling Celluloid::Future#value.

Let's look at these both to see how they actually work. First an asynchronous call, as we'd do it from Ruby code:

Even though ScarlettJohansson doesn't have an #oops! method, things are continuing to work as if you made two calls to the normal ScarlettJohansson#oops method, but no value is returned. Here's a diagram of what happens when you make an asynchronous call:

Asynchronous calls do not wait for responses, but forward a request for a method to execute, immediately returning back to the caller. No mechanism is provided to obtain a response to this call; async calls are a strictly fire-and-forget type of affair.

What if we want to invoke a method asynchronously but obtain the value the method returns at some point in the future? Enter futures!

Let's imagine we have a small army of Scarlett Johansson clones leaking photos on the Internet. Each one of them takes a little while drinking booze and fumbling around with their phone before they're actually able to post a nude photo. How can we tell them all to start doing this then reaping the results instantaneously when they're done? Let's use Celluloid::Actor#future to accomplish this task:

Our Scarlett Johanssons are now leaking photos to the Internet in parallel, and the wall clock time needed to have all of them get drunk and fumble around with their phone in order to post their nude photos is now barely more than the time each individual Scarlett took to post a couple photos. We're able to get the return value from ScarlettJohansson#oops after asynchronously requesting it be computed. How does this work exactly?

Celluloid implements futures through the Celluloid::Future object. Futures represent the result of a method which may or may not have finished executing before its result is requested. To obtain the result of a given future, call Celluloid::Future#value, but you need to be aware that Celluloid::Future#value will block until the method the future is associated with has finished executing.

Obtaining the value of a pending future is, like always, synchronized through a Celluloid::Mailbox object, which is used to provide a thread-safe interaction point for receiving the result of a method call if it hasn't already completed.

More to come!

I'd originally hoped for this blog post to contain a lot more, but honestly it's been sitting around half-finished for so long I thought I'd break it apart into a series.

In Part II, I'd like to cover what makes Celluloid truly unique (to my knowledge) among similar systems: "reentrant" method dispatch. This post will cover problems in existing RPC systems, including Erlang's gen_server, and discuss why Celluloid provides a more natural object abstraction. It will also explain why and how Celluloid uses Fibers to solve this problem, and the additional features that can be built upon Celluloid's reentrant method dispatch.

Part III (if I get that far) will provide a more in-depth look into some of the Erlangier features of Celluloid such as linking and supervision, and also discuss the roadmap for integrating some additional Erlang features into the Celluloid framework such as OTP-style applications and distributed actor systems. Stay tuned!

Erlang's biggest missing feature: globals

noreply@blogger.com (Tony) — Tue, 09 Aug 2011 15:37:00 +0000

There's one problem with Erlang I didn't touch upon in my previous rant, whose symptom can be found in some Erlang code called mochiglobal.erl. To understand what mochiglobal does we can read the description found in the source code's header:

Abuse module constant pools as a "read-only shared heap"

Erlang has a global centralized "code server" which maps module names to the corresponding BEAM bytecode containing all of the functions in a given module (including a limited number of previous versions, so it has some MVCC properties). The code server stores the state which represents the functions of a given module in a separate shared heap. The code server has pretty decent semantics for changing its contents, which all focus on the specific case of managing the actual Erlang code behind a particular application and allowing hot deploys.

What mochiglobal does is flip the old Lisp aphorism of "code is data" on its head. As the Erlang "code server" is exclusively focused on managing Erlang modules, if you want to use it to store a global value, you must convert it into Erlang code, compile it, and put it into the code server. This is exactly what mochiglobal does.

It's not as if mochiglobal is some arcane hack. It's used by some prominent Erlang applications, including Riak. Global shared state, something the Erlang VM has specifically been designed to exclude, is necessary in some real-world problems that are otherwise highly conducive to the Erlang VM. How many prominent Erlang applications have "mochiglobal" or some other "mochi*" code in them to provide features which really should be included in OTP?

Erlang could've done better here by providing some mechanism for managing global shared state. The Erlang code server is just one specific case where global shared state is needed, and that's the only one the language designers tackled.

A common Erlang reaction to the suggestion that the Erlang VM should support shared state is "how do you tolerate faults?" I don't think tolerating faults is particularly hard. We can look at one model for how to tolerate faults in this case: the Erlang code server itself. If you crash the Erlang code server, it brings down the entire VM. This probably isn't the best solution, but it's one solution. Let's do better.

Another solution would be to treat the global state as a sort of database, or perhaps more specifically a key-value store. This store can contain the code which implements a particular module in addition to shared global values. Programs could modify its contents in atomic sections, which are committed when they complete or aborted and rolled back if an exception is raised. I don't know about you, but it's beginning to sound a lot like STM.

I think shared-nothing systems like actors and STM are two ways of managing state which together build a more complete picture of how you can build general purpose programs. Code should just be another kind of global state, and global state needs some kind of concurrency story, which STM addresses. Like peanut butter and chocolate, a language which combined actor-like ideas with STM ideas has a good way to manage not just code, but global shared data.

The Trouble with Erlang (or Erlang is a ghetto)

noreply@blogger.com (Tony) — Tue, 26 Jul 2011 15:01:00 +0000

This is a blog post I have been meaning to write for quite some time. I lament doing so because I've made a considerable time investment into the Erlang infrastructure and really love some of its ideas. Erlang has done a great and still unique job of synthesizing a number of concepts in a very interesting way. But after using the platform in various capacities for some 4 years now, there are some glaring issues I think need to be called out.

Records suck and there's no struct/map data structure

Erlang has a feature called "records" which uses the preprocessor to give you something akin to a struct or map, i.e. a way to access named fields of a particular object/term within the system. As far as I can tell, there's pretty much universal agreement within the community that this is a huge limitation, and several proposals have been made to remedy the problem. The requested feature has typically been referred to as a "frame", and several proposals for implementing frames have been floating around for several years. Yet no action has been taken on the problem.

So why doesn't Erlang have frames? While Erlang is an open source project, its implementation and release cycle are managed by Ericsson, the company that created it, and Ericsson just doesn't seem to care. I'm not sure what Ericsson's priorities are when it comes to adding features to Erlang, but in my opinion they're doing a worse job of engaging the community than Oracle has been doing with Java. I hate Oracle as a company, but so far it feels like they've actually done a fairly good job managing Java development and moving Java forward. I can't say that at all with Ericsson, and frames are the quintessential example of this.

Erlang sucks at managing memory

Once upon a time I looked upon BEAM's design as the future pattern all virtual machines would follow. I strongly encourage you to read that post before taking issue with anything I have to say in regard to this matter. I have completely reversed my opinion since the time I write that post.

The other night I tweeted "If you're looking for a language that gets multicore concurrency right, look at how Azul implemented Java on their Vega architecture" and I definitely stand by that. Azul is a company that got a lot of smart hardware and software people together and had them work on designing a custom system which would scale to hundreds of CPU cores (up to 768 of them), heaps that topped 500 GB (up to 768GB), and had the GC pause only 10-20ms at a time. The realtime performance characteristics Azul managed to eek out of their system lead them to often describe their GC as "pauseless".

Where Azul was scaling up to 768 CPUs in 2007, Erlang was crapping out around 15 CPUs in 2009. For everything Erlang had to say about the importance of immutability and messaging in concurrent systems, and despite Joe Armstrong's promise that "your Erlang program should just run N times faster on an N core processor," it turns out that on the Erlang VM the N core processor promise had an upper bound of around 15.

Why is this? Erlang implements its own memory allocator and can't take advantage of libraries like tcmalloc to provide better multithreaded heap management. I can't fault a language VM like BEAM for doing this save for the fact that what Erlang provides is relatively crappy.

Erlang has done a fairly decent job given the constraints it was working within. Erlang wanted to provide a soft realtime system, and managed to create one that works on commodity architectures, unlike the Azul Vega appliances which require custom hardware. However, Azul has managed to port their version of the JVM to x86 hardware with their Zing Architecture, which wraps the JVM in a separate runtime container which uses software transactional memory to replace the hardware transactional memory found on the Vega appliances. It's higher overhead but provides similar guarantees. Java also provides the RTSJ specification for building realtime systems in Java.

Both Zing and RTSJ demonstrate that Erlang's approach to building a realtime garbage collected system, using separate heaps per process, isn't necessary to still provide realtime characteristics. Erlang's approach of using separate heaps is nonstandard and comparatively hard to optimize because most other systems are using a shared heap model. Azul's Vega architecture shows that shared heaps can scale up to hundreds of CPU cores and hundreds of gigabytes of heap while still providing realtime characteristics. Even more exciting is that AMD's Fusion architecture, which they're implementing in conjunction with ARM, provides read and write barriers at the hardware level necessary to provide a system like Azul using commodity hardware.

However, I think everything I just said is moot for the majority of applications. People building messaging systems want the best performance possible but don't typically have software realtime constraints. The Erlang VM's approach to soft realtime made a design decision which hampers its messaging speed, namely the use of separate heaps, which requires messages be copied from one heap to another. This means the Erlang VM does not provide zero-copy messaging. Every time you send a message from one Erlang process to another, some amount of data must be copied.

Erlang has partly mitigated this problem by providing a separate shared heap for binaries, which are the Erlang type for arbitrary blobs of binary data. This means if you ensure the majority of data you move around doesn't contain anything of significant size except binaries, perhaps this won't be a problem. However, if you're moving large collections of numbers around (Erlang's strings-as-lists-of-integers come to mind), messaging will be comparatively slow compared to a zero copy system.

Various solutions to this have been proposed for BEAM, such as switching from a shared-nothing heap to a shared heap or a hybrid heap (where message-passed objects are copied once), however the Erlang garbage collector is not suitable for managing shared/hybrid heaps and would need to be rewritten for the task, and nobody has managed to get the shared/hybrid heaps working with Erlang's SMP scheduler, or rewritten the garbage collector to be more suitable to the task of managing a shared/hybrid heap.

A potential solution to this? Erjang, an implementation of Erlang on the JVM, provides zero copy messaging using the Kilim library for lightweight threads.

JIT? What JIT?

Erlang has a "JIT" compiler called HiPE, which is mostly hype. I put JIT in quotes because HiPE is mostly an Erlang-to-native-code compiler with a limited set of backends which does a pretty bad job of optimizing and can't use runtime profiling information to improve the quality of the native code it generates in the way JIT compilers like HotSpot are able to. Calling HiPE a just-in-time compiler is a stretch as it is for most part an ahead-of-time native compiler for Erlang. The quality of native code produced by HiPE can be so poor that it's often outperformed by the userland bytecode interpreter implemented in BEAM.

HiPE can perform a very limited set of optimizations. In particular, Erlang code is factored into modules, and HiPE's inliner is unable to inline natie code across modules. This is due to HiPE's lack of a deoptimizer (a.k.a. deopt), or a way to translate JITed code back into bytecode, which is necessary in general but particularly necessary in Erlang for cases like hot code swapping. Deopt support is a feature of many JIT compilers in languages more popular than Erlang, most notably the HotSpot compiler on the JVM. Google's V8 virtual machine for JavaScript added deoptimization support as part of their "Crankshaft" compilation infrastructure.

Erlang isn't general purpose

Erlang hates state. It especially hates shared state. The only facility provided by the language for dealing with shared state in Erlang is called "Erlang Term Storage" and provides a Judy array that several Erlang processes can talk to. The semantics of ETS are fairly awkward and using it directly is difficult. Erlang has a baked-in database called Mnesia which is built on ETS. Mnesia's performance characteristics aren't great but it provides a friendlier face for ETS. These are the only solutions to shared state baked into the language.

What should you do if you want to deal with a shared-state concurrency program in Erlang? The general advice is: don't. Erlang isn't designed for solving shared-state concurrency problems. If you encounter a shared state concurrency problem while developing your Erlang program, sorry, you picked the wrong language. Perhaps you should move along... and Clojure offers you some great ways to tackle shared state concurrency problems.

The syntax is atrocious

I think this one goes without saying. That said...

Let me come at this from a different angle than you're probably expecting: I've recently started working with Clojure, and I have to say, I really think Erlang would've been a lot better off with a Lisp-like syntax than a Prolog-inspired syntax. To-date Erlang is the only popular language with a Prolog inspired syntax and all of the awkward tokens and gramatical constructions make me wish it just had a simple Lispy syntax. This has been implemented in Robert Virding's Lisp Flavoured Erlang, which is very cool and worth checking out.

That opinion might come as a surprise, because the main project I was developing in Erlang was Reia, a Ruby-like syntax and runtime for Erlang. I've discontinued this project, for many reasons, one of which is because it's been surpassed in features and documentation by a similar project, José Valim's Elixir. After years of working on Reia, I've really grown to believe I'd rather spend my time working on a language which incorporates Erlang's ideas, but on the JVM with mutable state.

The Erlang cargo cult would love to hang me out to dry for even saying that... so let me address it right now.

Immutable state sucks and isn't necessary for Erlang-Style Concurrency

Immutable state languages force object creation whenever anything changes. This can be partially mitigated by persistent data structures, which are able to share bits and pieces of each other because they're immutable. This works, for example, when attempting to create a sublist that consists of the last N elements of a list. But what if you want the first N elements? You have to make a new list. What if you want elements M..N? You have to make a new list.

In mutable state languages, performance problems can often be mitigated by mutating local (i.e. non-shared) state instead of creating new objects. To give an example from the Ruby language, combining two strings with the + operator, which creates a new string from two old ones, is significantly slower than combining two strings with the concatenating >> operator, which modifies the original string. Mutating state rather than creating new objects means there's fewer objects for the garbage collector to clean up and helps keep your program in-cache on inner loops. If you've seen Cliff Click's crash course on modern hardware, you're probably familiar with the idea that latency from cache misses is quickly becoming the dominating factor in today's software performance. Too much object creation blows the cache.

Cliff Click also covered Actors, the underpinning of Erlang's concurrency model, in his Concurrency Revolution from a Hardware Perspective talk at JavaOne. One takeaway from this is that actors should provide a safe system for mutable state, because all mutable state is confined to actors which only communicate using messages. Actors should facilitate a shared-nothing system where concurrent state mutations are impossible because no two actors share state and rely on messages for all synchronization and state exchange.

The Kilim library for Java provides a fast zero-copy messaging system for Java which still enables mutable state. In Kilim, when one actor sends a message, it loses visibility of the object it sends, and it becomes the responsibility of the recipient. If both actors need a copy of the message, the sender can make a copy of an object before it's sent to the recipient. Again, Erlang doesn't provide zero-copy (except for binaries) so Kilim's worst case is actually Erlang's best case.

The limitations of concurrent objects in Reia were solved using mutable state in my Celluloid concurrent object library for Ruby, but that deserves a blog post in and of itself.

Single assignment is just as problematic as destructive assignment

Erlang doesn't allow destructive assignments of variables, instead variables can only be assigned once. Single assignment is often trotted out as a panacea for the woes of mistakenly rebinding a variable then using it later expecting you had the original value. However, let me show you a real-world case that has happened to me on several occasions which wouldn't be an error in a language with destructive assignment and pattern matching (e.g. Reia).

There exists a complimentary case of mistaken variable usage to the afforementioned problem with destructive assignment. In single-assignment programs, it involves mistakenly using the same variable name twice excepting the variable to be unbound the second time:

The first pattern matching expression binds the Foo variable to something. In the second case, we've mistakenly forgot Foo was already bound. What's the result?

exception error: no match of right hand side...

We get no compiler warning in this case. This is the type of error you only encounter at runtime. It can lay undetected in your codebase, unless you're writing tests. Know what other problem writing tests solves? Mistaken destructive assignments.

Single assignment is often trotted out by the Erlang cargo cult as having something to do with Erlang's concurrency model. This couldn't be more mistaken. Reia compiled destructive assignments into Static Single Assignment (SSA) form. This form provides versioned variables in the same manner as most Erlang programmers end up doing manually. Furthermore, SSA is functional programming. While it may not jive with the general idealism of functional programming, the two forms (SSA and continuation passing style) have been formally proven identical.

The standard library is inconsistent, ugly, and riddled with legacy

Should module names in the standard library be plural, like "lists"? Or should they be singular, like "string"? Should we count from 1, as in most of the functions found in things like the lists module, or should we count from 0 like the functions found in the array module? How do I get the length of a list? Is it lists:length/1? No, it's erlang:length/1. How do I get the Nth element of the tuple? Should I look in the tuple module? Wait, there is no tuple module! Instead it's erlang:element/2. How about the length of a tuple? It's erlang:tuple_size/1. Why is the length of a list just "length" whereas the length of a tuple is "tuple_size"? Wouldn't "list_length" be more consistent, as it calls out it works on lists?

When we call erlang:now() to get the current time, it returns {1311,657039,366306}. What the hell does that mean? It's a tuple with three elements. How could time possible need three elements? A quick look at the documentation reveals that this tuple takes the form {Megaseconds, Seconds, Microseconds}. Separating out Microseconds makes sense... Erlang has no native decimal type so using a float would lose precision. But why split apart Megaseconds and Seconds?

Once upon a time Erlang didn't support integers large enough to store the combination of Megaseconds and Seconds, so they were split apart. The result is a meaningless jumble of three numbers, which you have to run through the confusingly named calendar:now_to_local_time/1 function to get a human meaningful result, which doesn't tell you what time it is now, but instead takes the tuple that erlang:now/0 returns as an argument and will spit back meaningful {Year, Month, Day} and {Hour, Minute, Second} tuples.

Legacy in the grammar

Try to use "query" as an atom in Erlang, e.g. {query, "SELECT * FROM foobar"}. What happens?

syntax error before: ','

This is because 'query' is a reserved word which was reserved for Mnemosyne queries. Never heard of Mnemosyne? That's because it's an archaic way of querying Erlang's built-in database, Mnesia, and has been replaced with Query List Comprehensions (QLC). However, it remains around for backwards compatibility.

You can't use "query" as a function name. You can't tag a tuple with "query". You can't do anything with "query" except invoke a deprecated legacy API which no one uses anymore.

Strings-as-lists suck

Erlang provides two ways of representing strings. One is as lists of integers, which is the traditional way that most of the library functions support. Another is binaries. Erlang has no way of differentiating lists of integers that represent strings from lists of integers that are actually lists of integers. If you send a list of integers in a message to another process, the entire list of integers is copied every time. On 64-bit platforms, every integer takes up 64-bits.

The obvious solution here is to use binaries instead of lists of integers. Binaries are more compact and exist in a separate heap so they aren't copied each time they're sent in a message. The Erlang ecosystem seems to be gradually transitioning towards using binaries rather than strings. However, much of the tooling and string functions are designed to work with list-based strings. To leverage these functions, you have to convert a binary to a list before working with it. This just feels like unnecessary pain.

The abstract concept of lists as strings isn't inherently flawed. In many ways it does make sense to think of strings as lists of characters. Lists as strings would probably make a lot more sense if Erlang had a native character type distinct from integers which was more compact and could avoid being copied each time a string is sent in a message like a binary. Perhaps in such a system it'd be possible to avoid transcoding strings read off the wire or completely transforming them to a different representation, which is costly, inefficient, and often times unnecessary (yes, this is a problem with Java too).

There's no "let"

Want a local binding in Erlang? Perhaps you've used let for this in a Lisp. What happens when you try to do this in Erlang? Even attempting to use "let" in Erlang just yields: syntax error before: 'let'

Once upon a time Erlang was supposed to get let bindings, and the "let" keyword was set aside for this purpose. But much like frames, it never happened. Instead, let is now an unimplemented reserved word which just breaks your programs.

There's no "nil"

In Clojure, I can write the following: (if false :youll-never-know). This implicitly returns "nil" because the condition was false. What's the equivalent Erlang?

Erlang forces you to specify a clause that always matches regardless of whether you care about the result or not. If no clause matches, you get the amazingly fun "badmatch" exception. In cases where you don't care about the result, you're still forced to add a nonsense clause which returns a void value just to prevent the runtime from raising an exception.

Where do I go from here?

Again, I want to emphasize that I have a great deal of respect for Erlang conceptually. But at this point I'd like to take what I've learned and go elsewhere with it. One direction I've gone is the Celluloid concurrent object library for Ruby. You can read more about it in the original blog post I wrote about Celluloid, which is a bit out-of-date at this point. I have a forthcoming blog post which should dive a bit deeper into Celluloid's guts and how it can do things which aren't possible in Erlang.

As you've probably guess from the references sprinkled throughout this post, I'm learning Clojure. I'm a fan of the JVM and Clojure provides a great functional language for leveraging the JVM's features. I think the sort of things that I'd be writing in Erlang I'll try writing in Clojure instead. Clojure has elegant Lisp syntax. Clojure has maps. Clojure has powerful facilities for dealing with concurrent shared state problems. Clojure has great semantics for safely managing mutable state in a concurrent environment. Clojure has real strings. Clojure has let. Clojure has nil. Clojure runs on the JVM and can leverage the considerable facilities of the HotSpot JIT and JVM garbage collectors.

I'd also like to try my hand at creating a JVM language, especially with the impeding release of Java 7 this Thursday. Java 7 brings with it InvokeDynamic, a fast way to dispatch methods in dynamic languages, and considerably eases the difficulty of implementing dynamic languages on the JVM. Stay tuned for more details on this.

Why I'm stopping work on Reia

noreply@blogger.com (Tony) — Wed, 29 Jun 2011 20:40:00 +0000

Some of you may have seen Elixir, a Ruby-inspired language for the Erlang VM, created by
José Valim. Whenever I see posts on HN or proggit about Elixir, I check out the comments, and almost every time someone asks about Reia, specifically how the two languages compare.

Reia and Elixir are extremely similar. If you check out their respective codebases, there's an almost bizarro world similarity between them. The project layout is almost identical, as is the build process and associated scripts to drive various parts of the language like the compiler and REPL. My impression of the code after looking over it was we were using largely the same mechanisms, and where they differed, José's solutions were something I had been thinking of changing, or something entirely new I hadn't thought of which seemed pretty cool.

At this point, in all respects Elixir is farther along than Reia and a better implementation. I think it's pretty bad to have two competing niche languages with nearly identical features.

So I'll be discontinuing work on Reia. The codebase and the web site will remain up on Github for however long Github wants to host them. Celluloid is Reia's spiritual successor. The main feature I always wanted out of Reia was a concurrent object system, and Celluloid implements that on top of Ruby.

Introducing Celluloid: a concurrent object framework for Ruby

noreply@blogger.com (Tony) — Wed, 11 May 2011 21:17:00 +0000

I've spend a lot of time recently working on a new concurrency library for Ruby called Celluloid. In short, its goal is to make working with threads as easy as working with Ruby objects in most cases, while still remaining loaded with all sorts of power user features for the edge cases. It's heavily inspired by Erlang and the actor model (hence "celluloid") and represents my best effort to port as many of these concepts over while making them more Ruby-like. This is the first in what I hope will be a series of blog posts describing various concurrency problems you might encounter in Ruby and how Celluloid can help solve them.

If you're already sold on using threads in Ruby, feel free to skip the next section of this post. However, as threads have remained a perpetual pariah in Ruby as a language, I feel some explanation is in order as to why you might actually consider using them.

Ruby and Threads

Rubyists generally don't like threads. There's plenty of good reasons to dislike threads: they're error prone for end users and the original implementation of threads in the Matz Ruby Interpreter was pretty crappy and broken in multiple ways. Even with the latest YARV interpreter found in Ruby 1.9, a global lock prevents multiple threads from running concurrently.

On the flip side, if you need multicore concurrency Ruby processes are cheap and there are some pretty good libraries like DRb for allowing Ruby VMs to work together. But even then most people are using Ruby to write stateless webapps that store all state in a database, so you can just run multiple Ruby VMs which all have the same application loaded to leverage multiple CPUs in a machine.

I used to be in the thread-hater camp, having cut my teeth on multithreaded C programs which left a bitter taste in my mouth, but recently I've changed my tune. This is mainly due to the great work of the JRuby and Rubinius teams to add true multicore concurrency to their Ruby implementations. JRuby has supported multicore concurrency via threads for awhile, and Rubinius is adding it in their hydra branch. With these Ruby interpreters, you can run one virtual machine per host and the threads you create will be automatically load balanced among all available CPU cores.

This has immediate benefits for things like Rails applications which enable thread safe mode. Rails will automatically create a new thread per request, allowing one VM to service multiple requests simultaneously. On interpreters like JRuby and Rubinius Hydra, this means you can run just a single VM per host and your application will utilize all available CPU cores. All the memory overhead of loading multiple copies of your application is mitigated, and as an added benefit you can take advantage of the better garbage collection these VMs (particularly the JVM) offer.

There is a catch: libraries can't share state across threads without using some form of thread synchronization. This is often trotted out as a persistent worry of those who prefer to run their Rails apps in the standard single threaded mode. Those gem authors, who knows what they're doing? Maybe they're using global variables! People don't think about this sort of stuff in Ruby, so shouldn't we just assume that 100% of Ruby libraries aren't thread safe per default?

The truth, at least for things like Rails apps, is that the general way they operate typically eschews thread safety issues. Ruby as a language favors object creation over mutating existing objects, and webapps generally create a new set of objects per request and don't provide mechanisms for sharing state between connections due to their stateless nature. In general, webapps are stateless and don't do things which will share state between threads.

If you do intend to go thread safe on your Rails app, you should certainly do your due diligence for auditing the libraries you use for unsafe usage of global and class variables, but in general I think the worries about running Rails apps in multithreaded mode are overblown. Ruby has much better semantics for promoting thread safety than other languages that have made the leap from single threaded to multithreaded (e.g. C/C++), and those languages have managed to make the transition with countless applications running in a multithreaded mode.

In the two years I've been deploying thread safe Rails applications, I've encountered exactly one thread safety bug, and that was in a library that originally claimed to have a specific thread safe mode but removed it from later releases and I unfortunately didn't catch that they had done so. The fix was simple: just create a thread-specific instance of an object I needed rather than sharing one across all threads. I won't say finding the bug was easy peasy, but all in all I don't think one bug was a bad price to pay for all the benefits of moving to a multithreaded deployment.

Concurrent Objects: How do they work?

Celluloid's concurrent objects work like a combination of normal Ruby objects and threads. You can call methods on them just like normal Ruby objects. To create a concurrent object, declare a class that includes the Celluloid::Actor module:

Then call the spawn method on the class:

This creates a new concurrent Charlie Sheen object. Calling the current_status method on it returns the normal value we'd expect from a method call. If an exception is raised, it will likewise be raised in the scope of the caller. But behind the scenes, all these things are happening in a separate thread.

Let's say things aren't going so well for Charlie. Instead of winning, Charlie is fired:

How can we help Charlie win again?

Calling Sheen#win! here does something special: it executes the method asynchronously. Adding a ! to the end of any method name sends a message to a concurrent object to execute a method, but doesn't wait for a response and thus will always return nil. You can think of this like signaling an object to do something in the background, and perhaps you'll check on the result later using normal synchronous method calls.

Using a ! to call a method asynchronously follows the Ruby convention of predicate methods with a bang on the end being "dangerous." While there are certain dangers of asynchronous methods (namely in how errors are handled), providing thread safe access to instance variables is not one of them. Charlie is running in his own thread, but there's no need to synchronize access to his private variables. This is where Celluloid's secret sauce comes in.

Charlie maintains a queue of pending method calls and executes them one at a time in sequence. Celluloid uses and asynchronous messaging layer that you can communicate with using normal Ruby method call syntax. However, when you call a method on a concurrent object in Celluloid, the "message" you send is quite literal and takes the form of a request object which waits for a response object (instances of Celluloid::Call and Celluloid::Response respectively).

This approach is largely inspired by the gen_server abstraction within the Erlang/OTP framework. For you Erlang nerds who might be worried Celluloid tries to jam everything into gen_server-shaped boxes, let me say right away that isn't the case, but you will have to wait for a further installment of my blog to find out why.

Celluloid by Example: Parallel Map

Let's start with a simple, practical, real-world example. If you're interested in digging deeper into the guts of Celluloid before starting this, I'd suggest you check out the README. That said, let's start with a simple problem: how can we implement a parallel map? That is to say, how can we reimplement Enumerable#map such that all of the map operations are performed in parallel rather than sequentially?

As this is a contrived and relatively simple problem, I'll go ahead and share with you how you might do it using Ruby threads as opposed to using Celluloid:

This version alone winds up being all you need to accomplish simple parallel map operations in Ruby. Here are some examples of using it from irb:

This pmap implementation behaves just like we'd expect map to. It returns the value of the block for each element if everything succeeds correctly, and raises an exception if anything goes wrong along the way.

Now I'd like to show you how to refactor this code to fit into the concurrent object pattern Celluloid uses. Let's start by trying to represent this same code using an object to perform the computation:

To turn this into a concurrent object, we first need to include Celluloid::Actor. To achieve concurrency, we need to make the method that performs the computation callable asynchronously. The initialize method is called synchronously by spawn (in case something goes wrong during initialization), so we'll need to create a separate method that actually calls the given block:

After that we can rewrite Enumerable#pmap using this class:

This creates a new Mapper actor for each element and calls Mapper#run asynchronously on each of them. After every one of them is executing they're iterated again, this time checking the return value. Since actors can only process one method call at a time, the call to Mapper#value will block until Mapper#run has completed, even though Mapper#run was called asynchronously.

This approach of allowing a value to be computed in the background and then only blocking when the value is requested is called a future. You've now seen how to implement a future, but it's also baked directly into Celluloid itself. Here's how to implement Enumerable#pmap using Celluloid::Futures:

Like Mapper, Celluloid::Future takes arguments, passes them to a block, then runs that block in the background asynchronously. Only when the value is requested does it block the current thread.

Now that we have a pmap function, what can we do with it? How about we compare the time it takes to do a bit of Google Image Search screen scraping for different color images for a particular search term using regular map vs. pmap?

The performance metrics vary across Ruby implementations, but in general, the parallel version goes approximately 3.5x faster, and the Celluloid version is 5-10ms slower than the version written using raw Ruby threads.

While this example is fairly trivial, in the next installment of this blog I'd like to demonstrate how to write a Sinatra-based chat server similar to node_chat.

What does this mean for Revactor?

In 2008 I wrote another actor library called Revactor, based on Fibers and evented I/O. While those ideas have grown increasingly popular, Revactor never did. I attribute this largely to Revactor's API, which was a fairly literal translation of Erlang's APIs into Ruby form with too little focus on putting an easy and accessible face on things. If you saw Mike Perham's recent article on actors in Rubinius (Revactor used a mostly identical API, as did MenTaLguY's Omnibus library), the code can be a little daunting, to the point you might need to learn a little Erlang just to figure out how to use it.

Celluloid is the official successor to Revactor. While Celluloid is primarily based on thread-based actors, it's been designed from the ground up with the idea of eventually incorporating event-based actors as well which can interoperate with an event library like EventMachine or cool.io. I know I originally dissed Scala for having both thread-based and event-based actors, but short of an Erlang-like process abstraction, it's not a bad compromise.

What about Reia?

One of the projects I've been working on the longest is Reia, a Ruby-like programming language for the Erlang VM. Today happens to be Reia's third birthday, and I do have to say after three years it's not where I thought it would be. It's generally usable but still missing some big features I feel are needed to make it a "complete" language. The main thing I feel is missing from Reia is a concurrent object model, and you can think of Celluloid as being an in-Ruby prototype of how it would work. I started to work on this in the legacy branch of Reia, but felt like it was too big of a project to tackle until I had fleshed out some of the other fundamentals of the language.

After I feel comfortable with how Celluloid is working I would like to try reimplementing in Reia. After that I think Reia will evolve into a truly useful language which bridges the gap between object oriented languages and Erlang-style concurrency.

I think Celluloid has the potential to be a truly useful library in Ruby on its own, however. It provides most of the underpinnings needed for more Erlang-like concurrent applications without having to switch to a different language.

Reia: Now with a PEG parser

noreply@blogger.com (Tony) — Tue, 05 Apr 2011 17:38:00 +0000

I haven't blogged about Reia in so long. Mea culpa! It's about this time that people start to think "I haven't heard about Reia in awhile. It must be dead." No, Reia is very much alive my friends, and making some pretty interesting progress... I hope to be able to announce some important new features around Reia's 3rd birthday. Stay tuned for that, but first a preview of things to come...

An experimental branch of Reia is now available which uses Neotoma to generate its parser thanks to some pretty impressive contributions by Graeme Defty. Neotoma is a Parsing Expression Grammar (PEG) based parser generator for Erlang, inspired by Packrat. Neotoma's author Sean Cribbs has also released Neotoma 1.5 with contributions by Graeme Defty which uses Erlang binaries internally and is substantially faster than the previous version.

Why does a PEG matter? PEGs make short work of problems that are extremely complex to handle when you attempt to subdivide the problem of parsing into separate scanning and parsing steps, such as how a traditional lex/yacc (or in Reia's case, leex/yecc) parser would operate. If you've ever attempted to look at the source code of the Ruby parser, you'll find some strange and confusing feedback between the lexer and the parser.

What's this for? Well the foremost reason is: interpolated strings. Ruby allows you to embed arbitrary Ruby code inside of any string using the #{...} syntax. This is a really great feature and one I managed to half-assed implement in Reia because I strongly believe in its awesomeness. However, Reia's implementation is a bit brittle and has a lot of implementations. Just recently I switched to the latest greatest version of leex which ships with the Erlang runtime and had to make some rather arcane changes to some code that uses leex internals, just to continue to support a partially functioning string interpolation mechanism.

Now, toss in some more fun complexities: the awesome /.../ regex literal syntax. We all love it, but why doesn't every language have it? What you may not realize is that this syntax is ambiguous in certain cases with the / and /= operators like it is in other languages like JavaScript. Now, for added fun, toss in the fact that regular expressions can interpolate just like strings, and you're beginning to see the makings of a gramatical nightmare.

All of these things are easy with a PEG. PEGs blend tokenization with parsing and allow comprehension of a much wider range of languages than is possible without pulling your hair out using lex/yacc-like tools, and much more than that, it's a very natural process with PEGs. PEGs are also by nature right-recursive, something that works quite well in immutable state languages like Erlang that have to build their syntax trees from right-to-left due to the nature of immutable singly linked lists. These lists only let you append elements on the left.

This sounds well and good, but unfortunately I have some bad news. Even after a few rounds of performance tuning, using an experimental branch of Neotoma that uses binaries internally instead of lists, the PEG parser is still about an order of magnitude slower than the leex/yecc version. Talking to Neotoma's author Sean Cribbs, it sounds like Neotoma might be doing an excessive amount of unnecessary copying internally. If PEGs pique your interest and you know a bit of Erlang you might take a peek at Neotoma and see if you can find some potential performance optimizations.

I would still like to merge the peg branch of Reia, and the PEG grammar fixes a lot of quirks in the present yecc grammar, but I'm still holding out until the performance is improved to closer to the leex/yecc speeds.

Distributed systems and dynamic typing

noreply@blogger.com (Tony) — Tue, 22 Mar 2011 16:29:00 +0000

There's a very good blog post by Robert Harper, a CMU CS professor, called "Dynamic languages are static languages." It's very enlightening and I strongly encourage you to read it. If I understand it correctly, which I freely admit I may not, the general idea is that dynamic languages can be thought of as static languages that have a single all-encompassing type. In that regard, dynamic languages are a proper subset of static languages. If you think I misinterpreted his post and I'm confused, please flame me in the comments.

In a previous post, called "Parallelism is not concurrency", he opines on a pet peeve of mine, namely how the terms parallelism and concurrency are nonchalantly and incorrectly interchanged. Parallelism applies to deterministic operations that operate on similar data in similar time. Some examples of parallel operations include rendering of 3D scenes on GPUs, or encoding/decoding blocks of compressed video. Concurrency, on the other hand, refers to the nondeterministic manner in which distributed systems operate, particularly ones where CPUs are separated over an unreliable network which can lose connectivity at any time.

In a third blog post, "Teaching FP to freshmen", Robert says he'll be teaching Standard ML to freshmen, and further argues he won't be teaching Object Oriented Programming because it's "both anti-modular and anti-parallel by its very nature, and hence unsuitable for a modern CS curriculum."

Three very interesting blog posts by the same person, clearly a well-educated computer science professor who knows more than I do. I mean, I like programming languages and I'm working on my own dynamically typed functional programming language, but Robert Harper wrote a book, one I certainly couldn't write. So who am I to judge?

Parallelism and Concurrency are Both Important

What I see missing from Robert Harper's writing is any attention paid to concurrency. He pays intense attention to parallelism, recognizes parallelism as important for the future, and strongly advocates functional languages as a great way to address the problems of parallelism. I have absolutely no bone to pick with him here and wish him great luck with his courses at CMU which address this problem domain.

However, my interests primarily lie in the realm of concurrent computing, and particularly in the area of distributed computing. In the area of distributed computing I find dynamic languages particularly important and think Robert Harper's article on static vs. dynamic languages omits some of the advantages of dynamic languages which make them work better in distributed systems.

This is a weighty subject, and one in which I don't think my own ideas alone make a particularly cogent argument. For some opening remarks, I will defer to Joe Armstrong, creator of the Erlang programming language, and his opinion from the book "Coders at Work" by Peter Seibel. As a bit of context to this quote, Joe is discussing the potential advantages that a static type system might confer upon Erlang. But then he gets to the disadvantages...

...the static type people say, "Well, we really rather like the benefits of dynamic types when we're marshaling data structures." We can't send an arbitrary program down a wire and reconstruct it at the other end because we need to know the type. And we have—Cardelli called it a system that's permanently inconsistent. We have systems that are growing and changing all the time, where the parts may be temporarily inconsistent. And as I change the code in a system, it's not atomic. Some of the nodes change, others don't. They talk to each other—at certain times they're consistent. At other times—when we go over a communication boundary—do we trust that the boundary is correct?

Type Systems and the CAP Theorem

There are two particularly sticky problems when it comes to the use of type in distributed systems. The first is the issue of serialization, or marshaling, of particular states. One way or another this is a solvable problem, both for statically typed and dynamically typed languages. I really don't want to delve too deep into this issue as it distracts from the larger point I'm trying to make, but in general, I think this is an easier problem to solve in dynamic type systems. I would also like to note that serialization formats which vomit their types all over the protocol and the codebase are outgrowths of static type systems. I'm looking at you, CORBA, SOAP, Protocol Buffers, and Thrift. On the flip side, systems which choose a minimal, semi-universal type system, such as JSON, BSON, BERT, and Msgpack, are all outgrowths of dynamic type systems. If I have a point to make here, I think it's that these systems are outgrowths of two very different ways of thinking about the same problem.

Marshaling is still a very important topic in distributed systems. Erlang largely abstracts this problem away from the programmer, allowing distributed nodes to exchange Erlang "terms" between processes on distributed nodes in the exact same way one would exchange messages between two processes located on the same host. The process of serializing that data, transmitting it over the network, receiving it via TCP, decoding it, and sending it to the appropriate Erlang process, is completely transparent to the end user. This is an extraordinarily powerful abstraction.

While statically typed languages can attempt to marshal data in a minimalistic JSON-like type system, this typically isn't the case. Instead, statically typed languages generally seem to prefer to vomit their types all over the protocol. The boilerplate code needed to marshal/unmarshal particular types can be generated automatically by a declaration of the types and methods of a particular protocol, such as the WSDL files used by SOAP. Again, users of statically typed languages could reduce the state of particular entities in their system to one which could fit into a minimalistic type system, but for static languages this is still a manual process, or one which requires manual code generation. In a language like Erlang which is built from the ground up to be distributed, dynamic, and operate around a minimalistic type system, serialization and deserialization can happen completely automatically.

Why is this important in a distributed system? Because, to paraphrase Joe Armstrong, distributed systems are messy. Imagine an Erlang-like distributed system that's statically typed. In order for such a system to work effectively, all nodes in the system must have the exact same code loaded and therefore have a universal consensus on what the types in the system are. This has huge implications on the theoretical properties on such a system. In order for a distributed system to agree on the nature of all types, it must be consistent.

However, if you're familiar with the CAP theorem, you may recognize the inherent problem this may cause. The CAP theorem gives you three options: a consistent fault-tolerant system, a highly available fault-tolerant system, or a consistent highly available system which breaks at the first sign of a fault. Only two of these options provide the consistency needed to ensure universal agreement on the types in the system such that automatic marshaling/unmarshaling of static types is even possible. In a distributed system, you either must give up universal agreement on the types, or sacrifice availability.

To quote Joe again, distributed systems are composed of parts which are "growing and changing all the time" with "parts may be temporarily inconsistent." While there aren't any guarantees that distributed systems built around dynamic type systems will work, inconsistent statically typed systems with disagreements about types are guaranteed not to work. Dynamic systems not only provide the possibility that your system may continue to function with different versions of the code loaded at the same time, but the ability for the programmer to plan for this contingency and offer ways to mitigate it. It's possible this may result in errors, but it may work, whereas incompatible type definitions are universally guaranteed to create errors. In a distributed environment, dynamic type systems provide extensibility, whereas static type systems actively seek to preclude it.

Something I've seen time and time again in systems like SOAP and even Thrift and Protocol buffers is attempts by programmers to escape the constraints of the type system, which almost universally fall into proprietary ways to store key/value pairs. One SOAP API I'm working with now provides "Maps" with "Entry" tags that have a key attribute and an associated value. Another implementation provides an array of "NameValuePair" objects. These solutions seem ugly, but in my opinion, their intentions are not. These are people seeking to extend running systems without the need to completely redefine the protocol. That's very much a practical concern.

Distributed Applications Must Be Flexible

In order for distributed programming to work effectively, nodes need to be able to call functions on each other without the need for programmers to write custom marshaling/demarshaling code for each type. The marshaled data needs to work extensibly, so that nodes running different versions of the code can still talk to each other in a forwards and backwards compatible manner.

Protocols will change over time. Older versions of the code need to work with a newer protocol, and vice versa, older versions of the protocol need to work with newer code. Nodes should be upgraded as practicality dictates. Perhaps your system administrator begins an upgrade and you lose access to a datacenter, because a janitor at your upstream ISP spilled a bucket of mopwater all over their datacenter's raised floor and caused a huge electrical disaster. Now your distributed application is running two different versions of the code, and it's suffered a network partition. Does this mean it should just break when the network partition is fixed?

Erlang has shown us it's possible to recover from these kinds of conditions. Even when we can't change code atomically across the entirety of a distributed application, it should still continue to operate without breaking. Distributed systems should be able to grow and change all the time without rigid boundaries.

Relativity for Programmers (A.K.A. Oh CAP!!!)

noreply@blogger.com (Tony) — Wed, 16 Mar 2011 06:03:00 +0000

When you first start to learn physics you typically learn Newtonian Mechanics. Objects have inertia. Force is mass times acceleration. For every action there is an equal and opposite reaction. These all seem straightforward.

Albert Einstein turned the world on its head. Everything is relative. Time is relative. Simultaneity is relative. There is no universal clock.

The same thing happened in the computer science world, not with a bang or the attention that was given relativistic physics, but with a small crowd paying attention. CAP Theorem has the same implications for computer scientists as relativity had for physicists, but no one is really paying attention.

Computer scientists may dream of a fully consistent database distributed across a network, capable of two-phase commit and typical consistency guarantees. Not many think about the implications of such a database.

Imagine you had your database split across two datacenters. Let's call them Foo and Bar. While at one point in time, Foo and Bar could talk, now they can't. Perhaps a clumsy janitor spilled the bucket for his mop, and it destroyed all of your routing hardware between your datacenters. Some customers are accessing Foo and some are accessing Bar. If they want to perform a read, they'll get data that's possibly stale. If they want to perform a write, what happens?

CAP Theorem

CAP Theorem is possibly the most important property of distributed systems that you can possibly understand. It's been motherfucking proven by computer scientists, bitches! So what does it state? The core idea is that consistency which is both instantaneous and global is impossible, except on a single node. As soon as you have more than node storing data for you, you either have to sacrifice availability or consistency. Much like how you can only infer the state of the universe from old light beams hitting your light cone in relativistic physics, distributed systems can only infer the state of other nodes via the protocols they communicate with. These protocols can fall into one of three different forms per the CAP theorem:

CP: we have masters you have to listen to, bitch slaves. You can try to change the cluster state but all changes have to go through the masters, and if your master is split, you can't modify any state you bitch submissive node
AP: sure, we'll accept reads and writes. We'll try to eventually resolve any conflicts in the event of a netsplit. If we're cool we'll let you specify your own conflict resolution mechanism in the event of conflicting writes, otherwise we'll just do last writer wins.
CA: the network broke? Whoa. I was totally not expecting that. I was just happily doing bongrips until you barged in. I'm so high I don't even know what's going on.

If you're writing a datastore you consider to be "CA", you're doing it wrong. Fortunately, most datastores that claim to have "CA" properties are actually CP. That is to say, in the event of a network partition, they sacrifice availability. Reads to the cluster hopefully work, but writes shouldn't. Writes should error or time out for a consistent system. Any system that functions this way is in fact "CP". That is to say: it provides consistency, but in the event of a netsplit, you lose availability, at least for writes. Ideally these systems still provide stale reads.

Calling your system CA is an admission of the fact that if a partition separation occurs, and due to writes each partition has a separate view of the world, when the netsplit resolves your partitions have no mechanism for resolving conflicting states between partitions and thus your datastore is just fucked. Short of a system administrator picking which view of the world they consider to be the best and blowing away the state on the other nodes, the system state is going to be inconsistent and irresolvable.

So what are the alternatives which are partition tolerant and don't break in the event that the janitor spills mopwater on the router that routes packets between your datacenters?

Option #1: sacrifice consistency for availability

This option lets clients read and write from any partition. When connectivity between partitions is restored, there's a plan for resolving all of the writes into a consistent world state. The simplest plan is last writer wins, which is used by Cassandra. Whoever was the last one to alter the cluster state provided the definitive cluster state, simply by virtue of writing last.

In an eventual consistency model, clients may have inconsistent views of the world. This is unfortunately a fact of life. State may unexpectedly change by a particular client's view, because it's being modified on a different partition and those partitions managed to sync up. The road ahead of you may be a beach, but after syncing up with another part of the world, it may turn into a cactus. Such is life.

Option #2: sacrifice availability for consistency

Say you have a master/slave data store, and someone writes to a slave. Wat do? Slave can't talk to master. What's the most obvious option? Error.

This is a "CP" system. Slaves can serve stale state, but if you want to change the state, you have to talk to the master for that state. State can be partitioned/shareded/durr between multiple masters, but if the master for the particular state a slave wants to modify isn't available, sorry, that's a paddlin', or rather an error.

Option #3: highly available, and consistent at the partition level, netsplit fixed wat???

The only other option, the "CA" option, is that each partition pretends that writes to that partition are consistent, but when the netsplits are resolved, there's no consistent way to resolve that state. That is to say, if netsplits occur, each partition is highly available and allows writes, but as soon as the netsplit resolves, the cluster state is broke and irreconcilable.

In a quick survey of NoSQL datastores, Membase, MongoDB, and the hypothetical Redis Cluster all claim to be "CA". Are they really? Or do they just sacrifice availability in the event of a netsplit? If that's true, they're actually "CP", not "CA".

You really don't want to sacrifice partition tolerance. In fact, some might go as far as to say you CAN'T sacrifice partition tolerance. If you think your system is "CA" you probably understand the the CAP theorem and its implications.

CAP is the law, and AP or CP are the only reasonable options

If you're running an app on a distributed network, things are going to go wrong. You might idealistically claim that you can provide both consistency and availability, sacrificing partition tolerance. You can't.

Can you pick "CA"? Sure, if you're content with losing all your data if the network had any errors. That's probably not what you mean. If you're building software for a fault-tolerant world, it's probably either going to be:

Consistent but unavailable during netsplits (CP)
"Eventually" consistent but available during netsplits (AP)

The only other option is "broken after netsplits". The best option is probably AP: this is the approach was pioneered by the Amazon Dynamo system which powered Amazon's shopping cart, and by copycat open source systems like Riak and Cassandra, not that Riak and Cassandra aren't awesome too.

If you're not a fan AP of broken "CA", you're a fan of CP, which puts you in the same league as this guy:

How to make a cheap CI traffic light

noreply@blogger.com (Tony) — Fri, 15 Oct 2010 03:02:00 +0000

We use continuous integration (CI) testing where I work, using Hudson as our CI server. I saw this recent article on the social impacts of making the CI testing status more visible and how that encourages people to fix things that are broken. One way to improve the visibility beyond just having a monitor displaying the CI server status page is to add a large traffic light displaying the status of your CI server. This is an idea I first heard of being used at Digg and have later seen at companies like Github. Below is the Hudson status setup we have at my workplace, showing both the traffic light and the monitor displaying the build status page:

Green means everything built successfully. Yellow means a build is in progress. Red means something is broken. It's a very effective way to let everyone in the office know the status of your builds and encourage people to fix them!

Believe it or not, adding a traffic light to your CI setup is simple, relatively easy, and cheap! Also, if you're like me occasionally breaking into electronics and soldering things is fun! Here's what you'll need to build your own CI traffic light:

Lava Lite Traffic Light from Amazon ($19.99)
A computer running Linux with a parallel port
4-conductor wire (one for each lamp + ground)
A DB-25 connector
Soldering supplies (soldering iron, solder, desolder braid, etc)

I bought the Lava Lite Traffic Light based on some reviews from people who used it as a Toastmasters timing light (to let people know when their time speaking is up):

By default it blinks in what the box claims is a "random" pattern but is actually a fixed pattern of red, yellow, green, red, yellow, green. But after opening up the traffic light itself, which only requires a few screws, I found out it was extremely easy to modify.

By default it's controlled by a single IC which is mounted on its own daughterboard, soldered to a small control board which contains the requisite transistors and resistors for powering the lamps and switching them on and off. The first thing you'll need to do is unscrew the control board and desolder the daughterboard from it. Desolder braid is your friend here.

The next step will be following the traces from the lamps back to where the daughterboard is mounted. This is fairly straightforward as the components involved in switching the lamps on and off are in threes. Looking at the control board you should very quickly be able to identify these components. There are three larger blue resistors which limit the current going to the lamps. Don't touch those! There are three transistors which are used to turn the lamps on and off. And finally there are three smaller brown resistors... follow the traces from these resistors to where they used to attach to the daughterboard. These are at TTL voltage, which is good, because so is a parallel port. What we'll be doing is soldering wires onto these three connectors and wiring them directly into a parallel port:

I'm afraid I don't have any fancy schematics for how exactly to do this, but it shouldn't be too hard! Here's a photo of where I soldered onto the control board. You can see the slot in the lower right corner where the daughterboard for the control IC used to be. I used a green wire for the green light, red for red, white for yellow, and black for ground. The circuits that go to ground are large and well labeled, so it should be fairly straightforward to find the points to solder onto:

If you've soldered correctly, you should be able to short any of the lamp wires (or a combination thereof) with the ground wire and they should turn on.

On the other side of your wire, you'll be wanting to solder on a DB25 connector. Wire the three wires which control the lamps to data lines 0, 1, and 2, and the ground to any of the many ground pins available on a parallel port. Here's a picture of mine:

And here's a handy dandy parallel port pinout diagram:

You'll be wanting to solder onto D0, D2, D2, and probably pin 18 (ground). I put green on D0, yellow on D1, and red on D2.

Next, you'll need some software to control the traffic light over the parallel port. To control our traffic light, I wrote a simple UDP server in C that receives 1-byte packets and writes them out to the parallel port. I originally wrote the server in Ruby but garbage collection was hanging up some fun light shows I was trying to put on, so I rewrote it in C for moar realtime. Here is the C source code to the traffic light server:

This uses the /dev/port character device on Linux, so you'll either need to run this server as root or a user specifically configured to have permissions to /dev/port. One thing I soon discovered was that the parallel port is stateful and will remain in the last state you set it to. This means you don't constantly have to write to the parallel port to keep a lamp on. Instead you can just set a state and the parallel port will remain in that state. Another thing I discovered that I still can't explain is that for some reason I needed to write the same byte twice to get the light to show the appropriate state. I don't know why, but it works!

Next, you'll need a client for your traffic light server. Here's a simple one I wrote in Ruby:

Just wait until this script gets passed around the office. Believe me from experience, everyone will go nuts for the first few days playing with the traffic light. Setting the light to an individual color is all, but for a true acid test here's a script I wrote which calls the above one and will make your traffic light rave balls!

But we didn't build this traffic light just to make it rave balls, did we? No, back to business, we need to get it to show the status of the Hudson build server. Here are some examples you can go off of which we're using now. It's a hacked together collection of scripts, to be sure, but it gets the job done:

Just set your Hudson server address (and username and password, if you protect it with HTTP basic auth like we do) and you're ready to go.

With all those scripts combined and a quick cron job to periodically poll Hudson's status, the traffic light will automatically display your build status and let everyone in your office know how diligent you've been about keeping your tests passing.

Now, get to work and build your own traffic light!

Multithreaded Rails is generally better than Async Rails, and Rainbows is cooler than Node

noreply@blogger.com (Tony) — Tue, 10 Aug 2010 17:18:00 +0000

Once upon a time Rails was single threaded and could only process one request at a time. This meant for each concurrent request you wanted to process with Rails, you needed to run an entirely separate Ruby VM instance. This was not a good state of affairs, especially in cases where your Rails application was blocking on I/O when talking to a database or other external service. An entire instance of your application sat there useless as it was waiting for I/O to complete.

The lack of multithreading in Rails would lead Ezra Zygmuntowicz to write Merb, a thread-safe web framework for Ruby which certainly borrowed conceptually from Rails and would go on to serve as the core for the upcoming Rails 3.0 release. In the meantime, the earlier Rails 2.x branch would get support for a thread safe mode as well. This meant that web applications written in Ruby could process multiple requests using a single VM instance: while one thread was blocking on a response from a database or other service, the web application could continue processing other requests in other threads.

Even better, while Ruby originally started out with a "green threads" implementation which executed threads in userspace and could not provide multicore concurrency, newer, more modern Ruby implementations emerged which provided true native multithreading. JRuby and IronRuby, implementations of Ruby on the JVM and .NET CLR respectively, provided truly concurrent native multithreading while still maintaining Ruby's original threading API. Rubinius, a clean-room implementation of a Ruby VM based on the Smalltalk 80 architecture, has started to take steps to remove its global lock and allow concurrent multithreading as well.

With a multithreaded web framework like Merb, recent versions of Rails 2.x, or Rails 3.0, in conjunction with a Ruby VM that supports concurrent multithreading, you now need to only run one VM instance with a copy of your web application and it can utilize all available CPU cores in a server, providing true concurrent computation of Ruby code. No longer do you need a "pack of Mongrels" to serve your Rails application. Instead, you can just run a single VM and it will utilize all available system resources. This has enormous benefits in terms of ease-of-deployment, monitoring, and memory usage.

Ruby on Rails has finally grown up and works just like web applications in other more popular languages. You can run just one copy of any Ruby VM that supports native multithreading and utilize all available server resources. Rails deployment is no longer a hack. It Just Works.

But Wait, Threads Are Bad, And Async Is The New Hotness!

Threads have typically had a rather mired reputation in the programming world. Threads utilize shared state by default and don't exactly provide the greatest mechanisms for synchronizing bits of shared state. They're a leaky abstraction, and without eternal vigilance on the part of an entire development team and an excellent understanding of what's happening when you use thread synchronization mechanisms, sharing state between threads is error-prone and often difficult to debug.

The "threads are bad" cargo cult has often lead people to pursue "interesting" solutions to various concurrency problems in order to avoid using threads. Event-based concurrent I/O became an incredibly popular solution for writing network servers, an approach seen in libraries like libevent, libev, Python's Twisted, and in the Ruby world EventMachine and my own event library, Rev. This scheme uses a callback-driven approach, often with a central reactor core, dispatching incoming I/O asynchronously to various handlers. For strictly I/O-bound applications, things like static file web servers, proxies, and protocol transformers, this approach is pretty much the best game in town.

Node.js, a pretty awesome I/O layer for Google's V8 JavaScript interpreter, is something of the new hotness. It's certainly opened up the evented I/O approach to a new audience, and for I/O-bound tasks it provides a way to script in a dynamic language while remaining quite fast. But as others have noted, Node is a bit overhyped. If you write your server in Node, will it scale? It really depends on the exact nature of the problem. I'll get into that in a bit.

Ilya Grigorik recently presented at RailsConf and OSCON about em-synchrony, a set of "drivers" for EventMachine which facilitate various types of network I/O which present a synchronous interface but use Fibers to perform I/O asynchronously in the background. He had some rather impressive things to share there, including Rails running on top of EventMachine, dispatching requests concurrently using fibers instead of threads. This approach won't provide you the computational concurrency that truly multithreaded Rails as in JRuby and IronRuby (and Rubinius soon!), but it will provide you wicked fast I/O performance... at a price.

The New Contract

Programmers generally live in a synchronous world. We call functions which return values. That's the status quo. Some languages go so far as to make this the only possible option. Evented frameworks do not work like this. Evented frameworks turn the world upside down. For example, in Ruby, where you might ordinarily write something like:

response = connection.request params

In async land, you first have to initiate the request:

begin_request params

Then define a callback in order to receive the response:

def on_response(response)
...
end

Rather than calling functions, you initiate side effects which will eventually call one of a number of callbacks. Exceptions no longer work. The context is lost between callbacks; you always start from just your arguments and have to figure out exactly what you were up to before, which generally necessitates breaking anything complex down into a finite state machine, instead of say, an imperative list of I/O commands to perform. It's a very different approach from the status quo.

The em-synchrony approach promises to save you from this by wrapping up all that ugly callback driven stuff with Fibers. I've been down that road and I no longer recommend it. In January 2008 I wrote Revactor, a Erlang-like implementation of the Actor Model for Ruby 1.9, using Fibers as the underlying concurrency primitive. It's the first case known to me of someone using this approach, and significantly more powerful than any of the other available frameworks. Where em-synchrony makes you write Fiber-specific code for each network driver, Revactor exposed an incomplete duck type of Ruby's own TCPSocket, which means that patching drivers becomes significantly easier as you don't need asynchronous drivers to begin with.

However, for the most part I stopped maintaining Revactor, largely because I began to think the entire approach is flawed. The problem is frameworks like Revactor and em-synchrony impose a new contract on you: evented I/O only! You aren't allowed to use anything that does any kind of blocking I/O in your system anywhere, or you will hang the entire event loop. This approach works great for something like Node.js, where the entire system was written from the ground-up to be asynchronous, in a language which has a heritage of being asynchronous to begin with.

Not so in Ruby. There are tons and tons of libraries that do synchronous I/O. If you choose to use async Rails, you can't use any library which hasn't specifically been patched with em-synchrony-like async-to-Fiber thunks. Since most libraries haven't been patched with this code, you're cutting yourself off from the overwhelming majority of I/O libraries available. This problem is compounded by the fact that the only type of applications which will benefit from the async approach more than the multithreaded approach are ones that do a lot of I/O.

This is a problem you have to be eternally vigilant about what libraries you use and make absolutely sure nothing ever blocks ever. Hmm, is this beginning to sound like it may actually be as problematic as threads? And one more thing: exceptions. Dealing with exceptions in an asynchronous environment is very difficult, since control is inverted and exceptions don't work in callback mode. Instead, for exceptions to work properly, all of the "Fibered" em-synchrony-like drivers must catch, pass along, and rethrow exceptions. This is left as an exercise to the driver writer.

Threads are Good

Threads are bad when they have to share data. But when you have a web server handling multiple requests concurrently with threads, they really don't need to share any data at all. When threads don't share any data, multithreading is completely transparent to the end user. There are a few gotchas in multithreaded Rails, such as some foibles with the initial code loading, but after you get multithreaded Rails going, you won't even notice the difference from using a single thread. So what cases would Async Rails be better than multithreaded Rails for? I/O bound cases. For many people the idea of an I/O bound application draws up the canonical Rails use case: a database-bound app.

"Most Rails apps are database bound!" says the Rails cargo cult, but in my experience, useful webapps do things. That said, Async Rails will have its main benefits over multithreaded apps in scenarios where the application is primarily I/O bound, and a webapp which is little more than a proxy between a user and the database (your typical CRUD app) seems like an ideal use case.

What does the typical breakdown of time spent in various parts of your Rails app look like? The conventional wisdom would say this:

But even this is deceiving, because models generally do things in addition to querying your database. So really, we need a breakdown of database access time. Evented applications benefit from being bound on I/O with little computation, so for an Async Rails app this is the ideal use case:

Here our application does negligible computation in the models, views, and controllers, and instead spends all its time making database queries. This time can involve writing out requests, waiting while the database does its business, and consuming the response.

This picture is still a bit vague. What exactly is going on during all that time spent doing database stuff? Let's examine my own personal picture of a typical "read" case:

For non-trivial read cases, your app is probably spending a little bit of time doing I/O to make the REQuest, most of its time waiting for the database QueRY to do its magic, and then spending some time reading out the response.

But a key point here: your app is spending quite a bit of time doing nothing but waiting between the request and the response. Async Rails doesn't benefit you here. It removes some of the overhead for using threads to manage an idle connection, but most kernels are pretty good about managing a lot of sleepy threads which are waiting to be awoken nowadays.

So even in this case, things aren't going to be much better over multithreaded apps, because your Rails app isn't actually spending a lot of time doing I/O, it's spending most of it's time waiting for the database to respond. However, let's examine a more typical use case of Rails:

Here our app is actually doing stuff! It's actually spending a significant amount of time computing, with some time spent doing I/O and a decent chunk spent just blocking until an external service responds. For this case, the multithreaded model benefits you best: all your existing Ruby tools will Just Work (provided they don't share state unsafely), and best of all, when running multithreaded on JRuby or IronRuby (or Rubinius soon!) you can run a single VM image, reduce RAM usage by sharing code between threads, and leverage the entire hardware stack in the way the CPU manufactures intended.

Why You Should Use JRuby

JRuby provides native multithreading along with one of the most compatible alternative Ruby implementations out there, lets you leverage the power of the JVM, which includes a great ecosystem of tools like VisualVM, a mature underlying implementation, some of the best performance available in the Ruby world, a diverse selection of garbage collectors, a significantly more mature ecosystem of available libraries (provided you want to wrap them via the pretty nifty Java Interface), and the potential to deploy your application without any native dependencies whatsoever. JRuby can also precompile all of your Ruby code into an obfuscated Java-like form, allowing you to ship enterprise versions to customers you're worried might steal your source code. Best of all, when using JRuby you also get to use the incredibly badass database drivers available for JDBC, and get things like master/slave splits and failover handled completely transparently by JDBC. Truly concurrent request handling and awesome database drivers: on JRuby, it Just Works.

Why not use IronRuby? IronRuby also gives you native multithreading, but while JRuby has 3 full time developers working on it, IronRuby only has one. I don't want to say that IronRuby is dying, but in my opinion JRuby is a much better bet. Also, the JVM probably does a better job supporting the platforms of interest for running Rails applications, namely Linux.

Is Async Rails Useful? Kinda.

All that said, are there use cases Async Rails is good for? Sure! If your app is truly I/O bound, doing things like request proxying or a relatively minor amount of computation as compared to I/O (regex scraping comes to mind), Async Rails is awesome. So long as you don't "starve" the event loop doing too much computation, it could work out for you.

I'd really be curious about what kinds of Rails apps people are writing that are extremely I/O heavy though. To me, I/O bound use cases are the sorts of things people look at using Node for. In those cases, I would definitely recommend you check out Rainbows instead of Async Rails or Node. More on that later...

Why I Don't Like EventMachine, And Why You Should Use Rev (and Revactor) Instead

em-synchrony is built on EventMachine. EventMachine is a project I've been using and have contributed to since 2006. I really can't say I'm a fan. Rather than using Ruby's native I/O primitives, EventMachine reinvents everything. The reason for this is because its original author, Francis "Garbagecat" Cianfrocca, had his own libev(ent)-like library, called "EventMachine", which was written in C++. It did all of its own I/O internally, and rather than trying to map that onto Ruby I/O primitives, Francis just slapped a very poorly written Ruby API onto it, foregoing any compatibility with how Ruby does I/O. There's been a lot of work and refactoring since, but even so, it's not exactly the greatest codebase to work with.

While this may have been remedied since last I used EventMachine, a key part of the evented I/O contract is missing: a "write completion" callback indicating that EventMachine has emptied the write buffer for a particular connection. This has lead to many bugs in cases like when proxying from a fast writer to a slow reader, the entire message to be proxied is taken into memory. There are all sorts of special workarounds for common use cases, but that doesn't excuse this feature being missing from EventMachine's I/O model.

It's for these reasons that I wrote Rev, a Node-like evented I/O binding built on libev. Rev uses all of Ruby's own native I/O primitives, including Ruby's OpenSSL library. Rev sought to minimize the amount of native code in the implementation, with as much written in Ruby as possible. For this reason Rev is slower than EventMachine, however the only limiting factor is developer motivation to benchmark and rewrite the most important parts of Rev in C instead of Ruby. Rev was written from the ground up to perform well on Ruby 1.9, then subsequently backported to Ruby 1.8.

Rev implements a complete I/O contract including a write completion event which is used by Revactor's Revactor::TCP::Socket class to expose an incomplete duck type of Ruby's TCPSocket. This should make monkeypatching existing libraries to use Revactor-style concurrency much easier. Rather than doing all the em-synchrony-style Fiber thunking and exception shuffling yourself, it's solved once by Revactor::TCP::Socket, and you just pretend you're doing normal synchronous I/O.

Revactor comes with all sorts of goodies that people seem to ask for often. Its original application was for a web spider, which in early 2008 was sucking down and scanning regexes on over 30Mbps of data using four processes running on a quad core Xeon 2GHz. I'm sure it was, at the time, the fastest concurrent HTTP fetcher ever written in Ruby. Perhaps a bit poorly documented, this HTTP fetcher is part of the Revactor standard library, and exposes an easy-to-use synchronous API which scatters HTTP requests to a pool of actors and gathers them back to the caller, exposing simple callback-driven response handling. I hear people talking about how awesome that sort of thing is in Node, and I say to them: why not do it in Ruby?

Why Rainbows Is Cooler Than Node

Both Rev and Revactor-style concurrency are provided by Eric Wong's excellent Rainbows HTTP server. Rainbows lets you build apps which handle the same types of use cases as Node, except rather than having to write everything in upside async down world in JavaScript, using Revactor you can write normal synchronous Ruby code and have everything be evented underneath. Existing synchronous libraries for Ruby can be patched instead of rewritten or monkeypatched with gobs of Fiber/exception thunking methods.

Why write in asynchronous upside down world when you can write things synchronously? Why write in JavaScript when you can write in Ruby? Props to everyone who has worked on solutions to this problem, and to Ilya for taking it to the mainstream, but in general, I think Rev and Revactor provide a better model for this sort of problem.

Why I Stopped Development on Rev and Revactor: Reia

A little over two years ago I practically stopped development on Rev and Revactor. Ever since discovering Erlang I thought of it as a language with great semantics but a very ugly face. I started making little text files prototyping a language with Ruby-like syntax that could be translated into Erlang. At the time I had outgrown my roots as an I/O obsessed programmer and got very interested in programming languages, how they work, and had a deep desire to make my own.

The result was Reia, a Ruby-like language which runs on top of the Erlang VM. I've been working on it for over two years and it's close to being ready! It's got blocks! It's got Ruby-like syntax! Everything is an (immutable) object! All of the core types are self-hosted in Reia. It's got a teensy standard library. Exceptions are kind of working. I'd say it's about 75% of the way to its initial release. Soon you'll be able to write CouchDB views with it.

Erlang's model provides the best compromise for writing apps which do a lot of I/O but also do a lot of computation as well. Erlang has an "evented" I/O server which talks to a worker pool, using a novel interpretation of the Actor model. Where the original Actor model was based on continuations and continuation passing, making it vulnerable to the same "stop the world" scenarios if anything blocks anywhere, Erlang chose to make its actors preemptive, more like threads but much faster because they run in userspace and don't need to make a lot of system calls.

Reia pursues Erlang's policy of immutable state systemwide. You cannot mutate state, period. This makes sharing state a lot easier, since you can share a piece of state knowing no other process can corrupt it. Erlang uses a model very similar to Unix: shared-nothing processes which communicate by sending "messages" (or in the case of Unix, primitive text streams). For more information on how Erlang is the evolution of the Unix model, check out my other blog post How To Properly Utilize Modern Computers, which spells out a lot of the same concepts I've discussed in this post more abstractly. Proper utilization of modern computers is exactly what Reia seeks to do well.

Reia has been my labor of love for over two years. I'm sorry if Rev and Revactor have gone neglected, but it seems I may have just simply been ahead of my time with them, and only now is Ruby community interest in asynchronous programming piqued by things like Node and em-synchrony. I invite you to check out Rev, Revactor, and Reia, as well as fork them on Github and start contributing if you have any interest in doing advanced asynchronous programming on Ruby 1.9.

Reia: Pluggable Parsers

noreply@blogger.com (Tony) — Wed, 30 Jun 2010 05:22:00 +0000

One stand-out quality of the Ruby community is a fascination with obtaining and manipulating Ruby parse trees. Such a fascination exists in many languages, but it's particularly weird in Ruby because until Ruby 1.9 there was no first-class way to obtain a Ruby parse tree. People went spelunking with C code into Ruby's internals, ripping the parse tree right out and exposing it back to the Ruby environment. Eventually Ruby parsers were implemented in Ruby itself in various projects. Yet it remains that while Ruby as a language seems to attract parse tree tinkerers, the language itself does not provide first-class ways to satisfy their needs.

I firmly believe that being able to obtain a parse tree for the programming language you're using is important and should be a first-class language feature. To that end, Reia supports a String#parse method:

>> "2+2".parse()
=> [(:binary_op,1,:'+',(:integer,1,2),(:integer,1,2))]

This parses the "2+2" string as Reia source code. The result might remind you a little bit of Lisp: it's a Reia parse tree. Right now there aren't immediate uses for Reia parse trees, but I'd soon like to add an interface for compiling/executing them. Erlang supports a feature called "parse transforms" which allow on-the-fly transformations of Erlang syntax. I'd also like to add such a feature to Reia.

If String#parse were just used to parse Reia source code it'd be a bit of a waste. However, it can be used for more than just that. For example, parsing JSON (as of tonight):

>> '{"foo": [1,2,3], "bar": [4,5,6]}'.parse(:json)

=> {"foo"=>[1,2,3],"bar"=>[4,5,6]}

After some recent problems dealing with JSON libraries in Ruby, I really felt JSON parsing should be part of the standard library. With this syntax, it almost feels like JSON parsing is part of the core language. Rubyists generally implement that sort of thing by monkeypatching the core types. Reia lets anyone define their own String#parse method by defining special module names, with no modifications to the core types required (which Reia doesn't let you do anyway).

To better understand how this works, let's take a look at how Reia implements String#parse:

def parse(format)
  "#{format.to_s().capitalize()}Parser".to_module().parse(self)
end

Given a format of :foobar, String#parse will capitalize the argument into "Foobar", then look for a "FoobarParser" module to parse itself with. This means anyone can add a parser to the language just by defining a module name that ends with "Parser" and has a parse method which accepts a string as an argument.

In short, anyone can add a parser to the language which can be called with a simple, elegant syntax. No monkeypatching required.

How to Properly Utilize Modern Computers

noreply@blogger.com (Tony) — Tue, 29 Jun 2010 06:21:00 +0000

Holy abstract concept, Batman! Computers are complicated things, especially modern networked ones filled with multiple CPU cores, and anyone professing to know a singular way to utilize them is truly a madman, or a genius, or a little bit of both... but before we can talk about modern computers we must first talk about computers as they used to be.

While this may look prehistoric, it actually happened at Burning Man

Long ago, programming languages were crap and programming was hard. Ken Thompson and Dennis Richie reinvented the way we think about computers by designing not only a new operating system but a new programming language to write that operating system in. It was an ambitious effort that has forever shaped modern computing. Some people don't appreciate it and wax philosophical about hypothetically superior solutions. Those people are retards. Unix rules. Get over it.

No, this isn't quite as good as having sex

Unix had a brilliant underlying philosophy: do one thing and do it well; use multiple processes to solve problems; use text streams as your interface. The simplicity of the Unix model had a beautiful elegance to it and made it very easy to leverage host resources in a scalable manner. Instead of writing big clunky monolithic applications, write several small programs that use text streams to talk to each other. Then if your host just happens to have multiple processors, the kernel can handle the task of farming out multiple jobs to multiple CPUs.

Shells and scripting languages were created to provide the interface and glue to the underlying system utilities. Users could easily queue up a series of tools to analyze and digest text streams as they saw fit. The interesting thing about this approach is that often times users of these sorts of utilities were performing pure functional programming. Each utility acts as a function which accepts its input over a pipe and produces output which it sends over a pipe.

A pearl, not to be confused with Perl. Perl is not a gem.

Perl fit into this ecosystem beautifully. Perl was focused at making short scripts which work on text streams, while providing easy conversion back and forth between text streams and numbers, since often times the text stream processing you want to do in Unix involves some kind of math. Perl is an extremely expressive language which allowed people to write far more powerful scripts than anything that had been seen in previous Unix scripting languages. It was powerful, expressive, and whimsical. Unfortunately, its whimsy would also be its demise.

Perl's approach didn't scale well to large applications. The level of abstraction it provided was targeted at writing short scripts within the multiprocess Unix environment. However, the tide was turning against the entire Unix philosophy. Monolithic applications and application environments were soon to become the norm.

OHAI!!!

Java tried to abstract away the underlying operating system. It was not easy to write Java programs that fit into the traditional Unix philosophy. Java strongly prefers you talk directly to things in Java Land, and because of that they reinvented standard Unix tools like cron as Quartz. Rather that using the traditional Unix shared-nothing process model to leverage multiple CPUs, Java wants you to use threads. If you write your entire application this way, you can deploy an application by running a single instance of the Java Virtual Machine and giving it all your system memory. With a single instance of the JVM you can theoretically utilize all of your available system resources.

Java still got a lot of things wrong. Threads are one problem (I'll get into that later). Another is handling application upgrades. Some environments tried to support hot code swapping, but this usually ended up leaking memory. In general, the recommended approach for upgrading a Java application is going to be starting and stopping the JVM. If you happen to be running a network server, such as, say, a web server, this means you have to wait for all clients to disconnect, or you have to shut down without completing their requests. Depending on the nature of your network protocol, clients may continue to remain connected indefinitely, so upgrades for those types of services typically means mandatory outages.

EWWWWW!!!!

Unfortunately, both the Unix model and the multithreaded model have warts.

Unix doesn't exactly provide the greatest set of tools for managing multiple processes. The interprocess signaling model used to manage processes left an awful lot to be desired. The pipe mechanism used for interprocess communication is rather primitive. Requiring everything be serialized to text streams incurs a lot of overhead, especially when you write several programs in the same language and can use more efficient data structures than text to communicate data.

In that regard, there are a lot of incentives towards moving to something like Java for concurrent programming. However, threads have warts too.

The semantics are just plain confusing and the possibility error is huge. There are a set of best practices which mostly come down to the overriding concern: don't share state between threads. As long as you never share state between threads there is never any concern over data corruption in concurrent programs. However, many multithreaded programs share state all over the place, using a collection of highly error-prone synchronization mechanisms to try to keep everything kosher. However, if you happen to forget to synchronize access to any given piece of shared state, you're screwed, you've just encountered a threading bug. Sharing state between threads requires extreme vigilance on the part of the programmer, and also intimate knowledge about how threads work and their possible caveats.

Beyond all this, threads are managed by the kernel, and talking to the kernel has high overhead. A truly amazing feat would be to soup up the Unix model and build your system using lots of shared-nothing processes that communicate using messages and mailboxes rather than primitive text streams. This is exactly the approach that was taken by Erlang.

I AM ERLANG!!!

Erlang took the whole Unix philosophy to the next level. Erlang process work like Unix processes, except they use mailboxes and messages instead of pipes. Unlike threads, Erlang processes run in userspace, which makes them relatively fast. You can create new Erlang processes a lot faster than you can create threads. The Erlang VM can run one kernel thread per CPU on your system and load balance processes. Code can be hot-swapped at runtime in a well-defined manner with extremely consistent semantics. The entire language philosophy emphasizes the creation of distributed, fault-tolerant, self-healing programs which are able to not only leverage an entire computer, but leverage an entire network of connected computers, using a philosophy which is similar to but an improvement on the Unix approach.

In Erlang, all state is immutable. This completely eliminates the problems of sharing state between threads. Due to the way the language is designed it is simply not possible. This opens up possibilities for Erlang language implementers to safely share state across threads, since the data can't be mutated. Unfortunately attempts at using this approach in the present Erlang virtual machine have not yet lead to significant performance benefits.

Erlang has its own warts. For everything it gets right semantically, it is still an aesthetically ugly language. Very few would describe Erlang code as beautiful. Despite claims that the semantics, and not the syntax are the barrier to learning Erlang, the main excuse I've heard from people who have avoided Erlang is that they don't like the syntax.

Clojure's logo is so awesome!

Clojure offers a different approach to leveraging modern multiprocessor computers. It provides shared state that threads can work on transactionally, an approach called Software Transactional Memory (STM), which works kind of like a database. When you aren't inside a transaction, all state is immutable, which means all state within the language is inherently "thread safe".

Because it's built for the JVM, Clojure is able to take advantage of all the previous effort put into an efficient native threads implementation for the Java programming language. While this is great for utilizing multicore systems, it's still centered around the notion of shared state. Distributing your program to multiple computers requires a conceptually different approach than you would ordinarily use to distribute a problem to multiple CPUs.

Beyond that, Clojure uses Lisp syntax. While some people enjoy writing raw syntax trees because its "homoiconic" nature (not to be confused with Madonna or house music) means they can work all sorts of wonderful wizardry with macros, history has shown that in general most people are not really big fans of that sort of syntax. Lisp has a lot of parens for a reason: because in most other languages those parens are implicit.

"You don't need flying cars. But you'll need a different kind of software"

So what's the answer? How do we "properly" utilize modern computers? I don't have the answer, only an opinion.

The Unix model was great. It just lacked a few features to really carry it over to distributed systems. That said, I really like the idea of running a single VM like the JVM per host, and letting it consume all available system resources running a single application.

Erlang lets you do this, except it provides a Unix-like process model with many of the warts excised. Erlang has excellent process management, and lets you interact with processes on remote nodes the same way you'd interact with the local system. Erlang replaced the lousy pipe-based model of interprocess communication with messages, mailboxes, and even filters that allow you to selectively receive from your mailbox.

Erlang provides lightweight, shared-nothing userspace processes and a great way for them to communicate, as well as a scheduler that can dynamically load balance them between native threads and thus host CPUs. Among many programming experts I've talked to there's a general consensus that having some sort of userspace concurrency context, be it a coroutine or a userspace thread, is a very handy construct to have. Erlang, perhaps more than any other language out there, has wrapped up userspace concurrency contexts into a very neat little package.

I still feel Erlang's main drawback is its syntax, and I have a few ideas about that. I think my language Reia brings with it the expressivity of a scripting language like Perl or Ruby combined with the awesome semantics of Erlang which allow it to easily utilize networks of multicore computers. Reia can support the monolithic one-process-per-application approach so associated with Java while allowing developers to write multiprocess applications internally. Reia is scripting evolved.

Dear Twitter: fix your fucking shit, seriously

noreply@blogger.com (Tony) — Sun, 20 Jun 2010 08:40:00 +0000

UPDATE (3/2012): Hi there. This post still seems to get a lot of traffic, but I'd like for you to know I've changed my opinion. At the time I wrote this, it was immediately after my first RailsConf where I was depending on using Twitter in order to be able to get in contact with people, and at the time, their service was somewhat lousy.

Since then, Twitter has done an amazing job of shoring up their infrastructure and making it robust. That said, this post no longer reflects my opinion of Twitter. I continue to use Twitter every day and it's still my personal favorite social network. Please take the post below with a grain of salt and recognize that it's an artifact of its time. I'm leaving the original text for your consideration, but please recognize the context.

I use Twitter every day. Every single fucking day. So when Twitter goes down, it affects me. And lately, Twitter has been down every single fucking day. It's not like they're unaware of it. Twitter Unavailable. High Error Rate on Twitter.com. Temporarily Missing Tweets. High Error Rate on Twitter.com. Site Availability Issues Due to Failed Enhancementof Our Timeline Cache. Working on Incorrect Tweet Counts. Bursts of Elevated Errors. Bursts of Errors. Site-Wide Availability Issues. High Error Rate on Twitter.com. Site Availability Issues. More Site Availability Issues. And Even More Site Availability Issues! And all of those within the past two weeks.

Twitter, it isn't hard to conclude your site is fucking broken.

I use Twitter because of the community of people. From a technology perspective, Twitter is markedly inferior to Facebook and Google Buzz, which not only manage to stay up a lot more than Twitter, but also support basic features like threaded conversations. I use your site because of the community, and exclusively because of the community. I know the community of Ruby programmers likes Twitter, and I'm not going to get them to move. So I'm stuck with Twitter.

From a technological perspective, Twitter is lagging lagging behind... way behind. Facebook has uptime, an order of magnitude more traffic, and threaded conversations! Google Buzz has uptime, and threaded conversations too! Twitter does not have threaded conversations, and is broken all the time. I understand Twitter hired the Twitoaster guy to add threaded conversations. Before you add that, can you please make sure your site isn't broken all the time?

Seriously, I want to like Twitter. I use Twitter all the time. I am a fucking Twitter whore. But seriously Twitter, you are the only site whose 503 Service Temporarily Unavailable page is known by name. Stephen Colbert is even namedropping it. While I'm a systems architect, I don't want to give you architecture advice. You're a high traffic site and I can't intimately know your pain points like you do. You know your pain points. So fucking fix them. Facebook works consistently with an order of magnitude more traffic. Google Buzz works consistently. So why the fucking fuck doesn't Twitter work consistently?

Twitter, you fucking fail.. Fix your fucking shit. Seriously.

Reia: Everything Is An Object

noreply@blogger.com (Tony) — Sat, 19 Jun 2010 23:49:00 +0000

I recently added support for immutable objects to Reia. Immutable objects work in a similar manner to objects in languages like Ruby, except once created they cannot be changed. You can set instance variables inside of the "initialize" method (the constructor), but once they've been set, they cannot be altered. If you want to make any changes, you'll have to create a new object.

Now I've gone one step further: all of Reia's core types are now immutable objects. This means they are defined in Reia using the same syntax as user-defined immutable objects. And since Reia looks a lot like Ruby, that means their implementation should be easy to understand for anyone who is familiar with Ruby. Neat, huh?

When I originally started working on Reia, I drank Erlang-creator Joe Armstrong's kool-aid about object oriented programming. I wanted to map OOP directly on to the Erlang asynchronous messaging model, and proceeded along that path. When you sent a message to an object, I wanted that message to be literal, not some hand-wavey concept which was implemented as little more than function calls.

However, this meant concurrency came into play whenever you wanted to encapsulate some particular piece of state into an object. And if the state of that object never changed, not only was this needlessly complex, it was a total waste! Furthermore, the core types behaved as if they were "objects" when really they weren't... they pretended to work in a similar manner, but they were special blessed immutable objects. People asked me if they could implement their own immutable objects, and sadly my answer was no.

Encapsulating immutable states has always been a pain point for Erlang. The canonical approach, Erlang records, are a goofy and oft reviled preprocessor construct with an unwieldy syntax. Later Erlang added an "unsupported" feature called paramaterized modules, which feel like half-assed immutable objects. There are very few fans of either of these features.

The typical OOP thinking is that objects provide a great tool for encapsulating state. So why do Erlang programmers have to use things like records or parameterized modules instead of objects? Let's look at Joe Armstrong's reasoning:

Consider "time". In an OO language "time" has to be an object. But in a non OO language a "time" is a instance of a data type. For example, in Erlang there are lots of different varieties of time, these can be clearly and unambiguously specified using type declarations

Okay, great! So if we try to get the current time in Erlang, what does it give us?

Eshell V5.7.3 (abort with ^G)

1> erlang:now().

{1276,989504,651041}

Aiee! What the crap is that? In order to even begin to figure it out, we have to consult the "type declaration":

now() -> {MegaSecs, Secs, MicroSecs}

Okay, this is beginning to make more sense, after consulting the documentation. What we've received is a tuple, which splits the current time up into megaseconds, seconds, and microseconds since January 1st, 1970. Microseconds are split off so we don't lose precision by trying to store the value as a float, which makes sense. However, megaseconds and seconds were split up because at the time the erlang:now()function was written, Erlang did not have support for bignums. In other words, the type declaration is tainted with legacy.

So what if we have erlang:now() output, how do we, say, convert that into a human meaningful representation of the time, instead of the number of seconds since January 1st, 1970? Can you guess? Probably not...

1> calendar:now_to_local_time(erlang:now()).

{{2010,6,19},{17,34,40}}

Of course, the calendar module! I'm sure that's exactly where you were thinking of looking, right? No, not really. The response is decently comprehensible, if you know what time it is. However, this doesn't seem like a particularly good solution to me. I guess Joe Armstrong likes his functions and type declarations. I don't. So how does Reia do it?

>> Time()

=> #<Time 2010/6/19 17:36:36>

Look at that. Time is an object! An immutable object in this case. Thanks to the fact that there are functions bound to the time object's identity, it automatically knows how to display itself in a human-meaningful manner. Because the identity of this particular piece of state is coupled to functions which automatically know how to act on it, we don't have to do a lot of digging to figure out how to make it human meaningful. It just happens automatically.

In the end, I'm a fan of objects and Joe Armstrong is a fan of traditional functional programming principles. They're both solutions to the same problem, but my argument is Erlang doesn't have a good solution to the problem of user-defined types and coupling the identity of states to corresponding functions designed to act on them. In the case of the latter, Joe Armstrong thinks it's a bad thing whereas I consider it a basic, essential feature of a programming language. As for the former, Erlang has given us records and parameterized modules, neither of which are a good solution.

I recently learned (courtesy the JRuby developers) that when Matz created Ruby, he took the core functions of Perl and mapped them all onto objects. Every single core function in Perl has a corresponding Ruby method which belongs to a particular class. I am now doing the same thing with Erlang, mapping all of the core functions Erlang provides onto Reia objects.

If Ruby is object-oriented Perl, then Reia is object-oriented Erlang.

Reia: Immutable objects at last!

noreply@blogger.com (Tony) — Sun, 06 Jun 2010 03:34:00 +0000

When I started creating Reia, I was originally skeptical of the "everything is an object" concept seen in languages like Smalltalk and Ruby. This was partially inspired by Erlang creator Joe Armstrong's hatred of object oriented programming. However, the more I used Erlang and increasingly saw Erlang's solutions for state encapsulation like records and parameterized modules as increasingly inadequate, the more I looked to objects to provide a solution.

Moments ago I committed the remaining code needed to support instance variables within Reia's immutable objects. You can view an example of Reia's immutable objects in action. If you're familiar with Ruby, you'll hopefully have little trouble reading this code:

>> class Foo; def initialize(value); @value = value; end; def value; @value; end; end
=> Foo
>> obj = Foo(42)
=> #<Foo @value=42>
>> obj.value()
=> 42

One caveat: Reia uses Python-style class instantiation syntax, i.e. Foo(42). Rubyists should read this as Foo.new(42).

So what are immutable objects, exactly? They work much like the traditional objects you're used to using in Ruby. Howevever, unlike Ruby, Reia is an immutable state language. This means once you create an object, you cannot modify it. The constructor method, which borrows the name "initialize" from Ruby, has the special ability to bind instance variables within a particular object, however once that initialize method completes and the object is created, no changes can be made. If you want to modify the instance variables, you'll have to create a new object.

Reia will eventually support objects whose state is allowed to change. These will take the form of concurrent objects, which is the original design goal I had in mind with Reia. Mutable objects take the form of Erlang processes, and more specifically Erlang/OTP gen_servers, which do not share any state with other concurrent objects and communicate only with Erlang messages. Going forward, my goal is to make all of the Reia built-in types into immutable objects, allowing user-defined immutable objects, and also allowing concurrent objects whose state can change (albeit in a purely functional manner).

If you've been following me so far, I hope you can sense how concurrency affects Reia's state management model:

Sequential objects have immutable instance variables
Concurrent objects will have mutable instance variables

This is similar to the state management compromise Rich Hickey chose in the Clojure language. In Clojure, by default all state is immutable. However, Clojure employs Software Transactional Memory for concurrency, and inside of Clojure's STM transactions (i.e. where concurrency enters the picture) state becomes mutable.

There's still a lot to be implemented in Reia's object model. I intend to support polymorphism through simple class-based inheritance, and the code needed to support that is partially in place. I'd like to support Ruby-style mix-ins. Once these features are all in place, I intend to completely rewrite the core types, reimplementing them all as immutable objects.

All that said, if you're interested in Reia and would like to start hacking on it, Reia could really use a standard library of objects, and the requisite code is now in place to facilitate that. I would encourage anyone with an interest in Reia to clone it on github and start implementing the standard library features you will like. The standard library needs all sorts of things, particularly wrappers for things like files, sockets, ETS tables, and other things which are already provided by the Erlang standard library.

Don't worry too much about making mistakes. Just send me a pull request and I'll incorporate your code, review it, and make changes where I see issues. I'd like to prevent the standard library from snowballing into the monster the Ruby standard library is presently, so if you have a feature you'd like to see incorporated, ping me on it (through github is fine) and I'll let you know if I think it should be incorporated.

I'm actively trying to recruit the open source community to build Reia's standard library, so if you're interested, start hacking!