If you’re thinking about trying Ruby 1.9 but think it will be a hassle to install and will break all your (non-rails) apps, read on. I use Ruby 1.9 almost exclusively and have very few issues.

New Parser, Stronger Code

That said, there are still some significant improvements in the 0.2.3 release. Plenty of stability and performance fixes were added, and a completely new parser for Ruby 1.9 was implemented based on the new Ripper parser that comes with the 1.9 standard library. Those who have been using YARD know that it has its own homegrown parser built from the ground up. While I’m impressed that I was even able to write a Ruby parser that works 80% of the time, it was quite lacking in both speed and robustness. Although Nathan Weizenbaum (of Haml fame) contributed some great refactoring and quality improvement patches to the parser, it still has some pretty obvious limitations.

The new 1.9-only parser will be the face of YARD looking forward, especially as people will start migrating to Ruby 1.9 themselves. The 1.8.x parser is still supported and maintained as far as bug fixes go, though new functionality probably should not be expected.

If you’re not yet convinced that Ruby 1.9 is worth the trouble for your project, consider these benchmark results of the new ripper parser parsing the YARD codebase twice:

yard (master)$ ruby -v
ruby 1.9.1p0 (2009-01-30 revision 21907) [i386-darwin9.6.0]
yard (master)$ ruby benchmarks/ripper_parser.rb
Rehearsal -------------------------------------------------
rip-parser      1.930000   0.020000   1.950000 (  2.004864)
yard-parser     4.700000   0.020000   4.720000 (  4.712828)
---------------------------------------- total: 6.670000sec

                    user     system      total        real
rip-parser      1.760000   0.000000   1.760000 (  1.763015)
yard-parser     4.720000   0.020000   4.740000 (  4.755574)

The comparison here is between the new parser used when you run YARD in Ruby 1.9 versus my old pure-ruby implementation. The difference is significant; a speed improvement of nearly 2.7x. Realize again that these are both running in 1.9, so we’re not even comparing the speed benefit you get from using YARV over MRI. That result is actually:

yard (master)$ ruby -v
ruby 1.8.7 (2009-01-28 patchlevel 99) [i686-darwin9.6.0]
yard (master)$ ruby benchmarks/ripper_parser.rb
Rehearsal -------------------------------------------------
yard-parser     7.820000   0.030000   7.850000 (  7.859455)
---------------------------------------- total: 7.850000sec

                    user     system      total        real
yard-parser     8.070000   0.020000   8.090000 (  8.114952)

Note that I had to modify the benchmark script to ignore the ripper parser here.

So, in Ruby 1.8.x, the fastest YARD can parse its own codebase twice is roughly 8 seconds. Compare this to running YARD in Ruby 1.9.1 (previous results), which can parse nearly over 4 times faster. This becomes significant on large projects like Rails, which takes 11 seconds to parse all of code in 1.9, but roughly 65 seconds under Ruby 1.8.7. 65 seconds is a long time to wait to parse code; and that doesn’t even count HTML generation, which takes an even more significant amount of time.

Plenty of New Documentation

I’ll be the first to admit, for a documentation tool, YARD is surely lacking in documentation. I take this seriously, but at the same time, I’m only one person and there’s a lot to write about. I made it a point this time around to spend this entire last week brushing up documentation to make YARD easier to jump into. YARD now has a great Getting Started guide which should cover the really basic aspects of how to get around. More importantly, as a developer tool, there is also a hefty Technical Overview guide covering all of the major components of the project and how a developer could take advantage of them. In this release alone I’ve written roughly 7,948 new words of documentation, and I’ve barely scratched the surface. YARD has tremendous potential and tons of use cases, so writing about it all will take time, but it will happen.

Looking Forward

So in the end, this release was very important. The parser was something that was on the table for a long time, dating back to my first roadmap to 1.0 post last year. Now that it’s complete, we can start looking ahead. As more of the groundwork is properly laid out, it becomes easier to move forward. The next release will be focusing on drastically improving the quality of the HTML templates, so expect bigger changes in the coming months.

By the way, if you didn’t notice, YARD got a new site face-lift, so check out the site.

I’m referring specifically to my blog, which was becoming a complete mess of ugliness and broken HTML. I decided to completely gut it, remove a bunch of useless crap and focus on making the content clean and easy to read. This is pretty much a warm-up exercise for the coming week of writing docs and putting out the new YARD site, but it’s something I should have done long ago. Hopefully this will give me some new incentive to write more (about YARD?)

It’s not a “blog” anymore

You may have noticed that comments are gone. Good. Comments are often useful, but to be honest, like many other “bloggers”, I’ve discovered they often don’t add much to the conversation (I rarely get comments on my blog anyway, so styling all that HTML is a waste of my time). A lot of people will say a site is “not a blog” if it does not allow a two-way conversation between the author and the reader. Fine, then I guess this is no longer a blog. My intention for writing articles is not to inherently start a conversation or get feedback. I write because I want to share, not because I need validation or care that you think I’m a dumbass. Don’t get me wrong, I still like when people point out errors and omissions or insert points of view, so if you notice anything send me an email or twitter me instead, we can have the discussion there. Better yet, reply with your own article. This keeps the conversation sane, civil and keeps the signal-to-noise ratio down. I’ve noticed this trend from other blogs I visit, and I’m starting to think the future of blogging will look a lot more like magazine articles than threaded conversations. Dialog is good, but it should be controlled so that it remains on topic and relevant.

I mentioned Twitter is a good way to reach me. I’ve actually been using it a lot for the past 6 months; a lot more than this blog. It’s no surprise that microblogging is way easier than writing long articles, so for the past few months my Twitter account has probably been a better source of blog content than my blog (I guess this is probably why people like Tumblr). Let’s be honest, I’m probably not going to start blogging much more than I already do, but I can keep up with Twitter pretty well. Because of this, I’ve incorporated my Twitter feed into my blog and it’s prominently displayed on the front page and in the sidebar on others. I didn’t just add it as a widget to fill whitespace. Instead, expect that feed to be a major source of content for this site. Though you should probably just follow me.

Anyway, enough about me. Hope you like the new design.

First off, if you haven’t already, you’ll probably want to pull an updated version of TextMate’s support files from the svn repository to get some necessary 1.9 compatibility fixes.

$ cd ~/Library/Application\ Support/TextMate/
$ svn co http://macromates.com/svn/Bundles/trunk/Support

You’ll need to clone my Git.tmbundle repository into the Bundles directory to get most of the changes I made to his bundle to deal with the basic Ruby 1.9 issues.

After that, you should get most of the basic functionality of the bundle, but you’ll probably notice that if you try to compare branches, revisions, or any of the more fun stuff, you’ll get the following error:

ArgumentError: An object in the argument tree could not be converted
    from (irb):17:in `to_plist'
    from (irb):17
    from /usr/local/bin/irb:12:in `<main>'

This is because TextMate ships with a native plist library in its support directory (the one we checked out above) that isn’t quite Ruby 1.9 compatible. We’re lucky though, because an updated version of the library, osx-plist (written by Kevin Ballard) is on GitHub, so let’s pull and build it.

$ cd ~/Library/Application\ Support/TextMate/Support/lib/osx
$ git clone git://github.com/kballard/osx-plist.git
$ cd osx-plist/ext/plist
$ ruby extconf.rb && make
$ cp plist.bundle ../../../

Assuming you have TextMate using Ruby 1.9, you should now be able to compare revisions and branches, and do pretty much anything with the awesome Git.tmbundle!

Most blogs and articles I’ve seen that try to explain continuations get it wrong. They bore you with low level details that confuse you in the first place, or scare you by starting with the premise that “it’s complicated, but I’ll try to make you understand”. I’m not going to do either, I’m actually going to explain continuations in two words, and I want you not to laugh, cry, or vomit on yourself when you read them:

GLORIFIED GOTOs

Oh no, I just said a dirty word, didn’t I. You may have your own irrational fears of the goto statement, but you probably know what they are and how they work:

#include <stdio.h>
int main() {
  int i = 0;
  printf("%d\n", i);
  goto some_label;
  i = 1;
  some_label:
  printf("%d\n", i);
  return 0;
}

The above snippet of working C shows how we can jump to different points in a function. It prints 0 twice instead of 0 followed by 1, as you would expect. Language evolution has mostly deprecated this form of programming through method calls and, more intricately, exception handling. These new practices are really just syntactic sugar on the old goto concept. Hopefully all those who scoff at any mention of goto will one day sit down and realize that they’re a central part of how computers work, but I’m getting off topic. The point is, whether you use gotos or not, they’re extremely simple to understand. In fact, they’re exactly as simple as continuations; let’s re-write the above using those “scary” guys:

def main
  i = 0
  callcc do |label| # callcc gives us ‘label’, a continuation object
    puts i
    label.call # this is our "goto" statement
    i = 1      # we skip this completely
  end          # this is where our "label" would otherwise sit
  puts i
end

To those in the know, this doesn’t really show off the full-extent of callcc’s powers (that comes later), but you can look at "label.call" as the equivalent of "goto label", and the end of the callcc block is where the label would be. That’s basically how continuations work.

If you’re having trouble parsing the above: the little trick to understanding the code is simply realizing that the block that callcc yields is executed immediately, once, and only once. It’s simply there so you can do any one-time initialization with the continuation you’re creating. The end of the block is where any subsequent calls to the yielded continuation object will go, not the block itself.

NOW YOU KNOW.

So what, is it really that simple? Continuations are another way to write gotos? The answer is: pretty much. There are actually only two functional difference that makes them even more powerful than the above C-style gotos, but one that makes them a little less-so.

More Powerful, How?

1. They’re not local to your method. Simply put, we can’t do the following in C:

// a() is called by main()
void a() {
  printf("hello world\n");
  label1:
  printf("then you say...\n");
  b();
}
void b() {
  printf("then I say...\n");
  goto label1;
}

This should print "hello world" followed by "then you say", "then I say", in a never-ending loop. Problem is, it won’t compile. Without getting into the nitty-gritty of why C can’t do this, I’ll just simply show how we can actually get away with it using continuations:

def a
  puts "hello world"
  callcc {|cc| $label1 = cc } # pretend this says "label1:"
  puts "then you say..."
  b
end

def b
  puts "then I say"
  $label1.call # pretend this says "goto label1"
end

As the comments indicate, this is almost a one-to-one translation of the C code above. Our continuation object is set to a global variable, essentially making it a “global label” that can be goto’d by running the method #call (picture it as “label.goto” instead of #call).

2. They maintain stack-frame state. That’s the reason why labels in C are local to a function, because you can’t just jump between functions without switching stack frames. Actually, there are tricks in C to do just that (namely longjmp and setjmp to jump to arbitrary addresses and continue execution which is actually how continuations are done in C), but we’re focusing on Ruby here, right? Basically, the continuation object yielded from callcc (the {|cc| thing}) contains a snapshot of our stack frame at the point it was yielded at (remember, the block is only yielded that first time). This allows us to jump between methods in a class, between classes, and jump back from a nested call when we are arbitrarily deep in a stack frame.

Anyone Up For A Quick Real World Example?

The last situation is the one with some real world usage. Imagine we’re writing a web framework with a Rails-style before_filter and we want the ability to halt the execution during a filter and jump all the way back up to the router code to find the next matching route. Sinatra (sort-of) does this with the `pass` method, and though it probably doesn’t use continuations to achieve the result, the concept here is a perfect example of wanting to jump back through our stack frames arbitrarily. We could be 3 method calls deep, or we could be 20 method calls deep, but we need to “go to” a specific point in our program execution. Usually, people implement this with try/catch style exception handling (RouteAbortError maybe), but continuations may be a little cleaner depending on the scenario (we might have a method handling the exception 3 method calls up, but we want to jump 7 method calls up to the initial point in code).

Less Powerful, How?

There’s one thing you may have noticed from the initial example, it’s that jumping forward looks different from the example where we jumped backwards. The limitation is that continuations can’t actually jump forwards, at least not between method calls. The reason is that in C, labels are compiled into the program statically, but in Ruby, the continuation objects are created at runtime. This means that we’d need to execute that callcc call in our future method to generate the continuation, but we can’t run code that hasn’t yet been run. In short, continuations are great for jumping back in time, not quite so for jumping forwards.

Final Correction

So I initially called continuations “glorified gotos”. Hopefully this got you to understand the simple concept behind their use; but now I should fess up and modify the definition just a little bit to be slightly more accurate. Instead of glorified gotos, think of continuations as:

GLORIFIED, STATEFUL, BACKWARDS JUMPING, GOTOs

Now that you know what the fuss is about, you can decide to hate continuations all you want.

invalid multibyte char (US-ASCII)

- OR -

CompatibilityError: incompatible encoding regexp match (Windows-31J regexp with UTF-8 string)

The errors themselves are relatively self-explanatory. Ruby 1.9 is far more Unicode aware than 1.8, and this error happens when have some Unicode (usually UTF-8) in one of your files. What you have to do to fix these, however, is not always as straightforward.

Well, after some time pulling my hair out, I’ve figure out that the solutions to these issues are actually quite simple.

Invalid multibyte char (encoding here)

If you get this issue, add the following to the top of each exploding file (below the shebang if there is one):

# encoding: utf-8

Note: You can also use "coding:" or even the Emacs style -*- encoding: utf-8 -*- but I like the simple term, ‘encoding’. Also note that you might need to replace ‘utf-8′ wth your specific encoding if it’s something else.

This should resolve the issue.

Basically, Ruby by default assumes that every file is encoded as US-ASCII, and so when it reads UTF-8 (or any Unicode) it freaks out because it is beyond the 7-bit encoding. You have to tell it that the file is encoded as utf-8 by listing it as we did at the top of the file. Yes, it’s a little anal, but I’m sure we’ll get used to it.

Incompatible encoding regexp match

This issue is a little bit hairier. I had this problem with the BlueCloth 1.0.0 gem (on line 972 of bluecloth.rb). It turns out that there are a few switches that turn on specific encodings, and for some reason BlueCloth turns on the //s switch which enables the SJIS encoding (maybe John Gruber wanted //m for multiline?). Ruby 1.8 didn’t mind having this on, but 1.9 freaks out. Moral of the story, when you see this error, check your Regexp switches.

1. Hash entries are Ordered!

h = { cat: 1, zebra: 2, dog: 3 }
h.delete(:zebra)
h[:monkey] = 4
h.each {|key, val| puts "#{key}: #{val}" }

# Guaranteed to print:
# cat: 1
# dog: 3
# monkey: 4

I’ve been praying for this for who knows how long. Everyone knows hashes don’t maintain order internally, but much of Ruby development makes use of Hashes as dictionaries because of the simplistic syntax. The problem is that dictionaries, unlike straight hash objects, usually require order to be maintained. There are tons of places in YARD where I had to resort to nested Arrays (eg. [[key, val], ...]) to maintain order of a set of associated objects… I’ve even written countless OrderedHash classes to solve this problem. This is definitely the biggest feature in 1.9 to me.

PS. You might have noticed the Pythonish {a: 1} syntax— yep, that’s new too, but it’s not quite cool enough to make it’s own item in our list but it’s really convenient to type.

2. Symbols are now Intuitively Comparable

YARD calls #to_s on about a million things. Last I checked, Rails does too. This is because internally we try to keep everything stored as a Symbol for obvious efficiency. The problem is that when writing API’s we’re usually allowing developers to pass in a String or a Symbol, which means comparison is always annoying. It looks like the Ruby devs looked at a lot of this code, puked all over their keyboards, and cleaned up both their keyboards and the code to deal with this. We can now do things like:

:hello =~ /e/      # => 1
:hello === "hello" # => true
:hello[1] == "e"   # => true

3. Enumerations on a Hash return a Hash

For the same reasons as above, this is awesome. Performing a #select on a hash was always a pain in Ruby 1.8 because you would end up with an Array… totally not a Hash. This inconsistency has been dealt with and we can now do:

{a:1,b:2,c:3}.reject {|k,v| k == :b }
=> {:a=>1, :c=>3}

4. Proper Unicode Support

Let’s not get too far into this list without giving credit for the awesome work Ruby 1.9 has done in fixing the Unicode support. In short, it is now possible to do things like:

"Hi!".encode("utf-16be") # => "\x00H\x00i\x00!"
File.read('test.txt', encoding: 'utf-8').encoding

We also get Enumerations like #each_char that iterate properly over such strings. There’s plenty more goodies regarding Unicode, but I still don’t know all of the details. Interestingly enough, I ran into a Ruby Unicode problem the other day, so I think I’ll be revisiting the problem with these new tools soon and write about what happened.

5. Regexps get Look-ahead/Look-behind

Probably the most powerful feature of regular expressions have been missing from Ruby for the longest time. I can’t remember any specific times when I hit this limitation, but I imagine it will simplify a lot of hacky code attempting to work around the feature omission in 1.8.

6. Object #tap

This is an easy one because anyone could have implemented it, but it’s a nice way to silently hook into a method call chain. The Ruby 1.9 docs give a cool example of this, but I think a better example is when dealing with method calls that aren’t chainable like Hash#delete:

# In 1.8 we need to do this because #delete returns nil:
h = {:a => 1, :b => 2, :c => 3}
h.delete(:b)
h.each {|k,v| puts k }

# In 1.9 we can do this in one line:
{a:1,b:2,c:3}.tap {|o| o.delete(:b) }.each {|k,v| puts k }

7. Fibers are lighter Threads

Still have yet to dig into this one, but Dave Thomas covers the new Fiber class. In short, they’re basically lambdas encapsulated in threads— but of course it’s not the implementation but the way you use these guys that makes them quite elegant. I can think of a few things right off the top of my head that involve the quick creation of threads from inline storable procedures.

8. New Hash Key Syntax

UPDATE: Okay, I wasn’t going to put this in the list until I realized how the syntax can be applied to method calls. Consider the following 1.8 code:

def open(filename, opts = {}) end

open('test.txt', :access => :read, :close => true)

The same code in 1.9 can be called as:

def open(filename, opts = {}) end

open('test.txt', access: :read, close: true)

That is cool.