Way back around the start of April this year I drafted a proposal for a talk I’d begun preparing on Celluloid and JRuby. Mostly gleaned from my findings in applying both technologies to an innovative Global Dev scammer detection solution due to launch shortly on our White Label Dating platform.

With some trepidation I then submitted my proposal to the innovative EuRuKo 2013 CfP. EuRuKo have this year chosen to employ the same CfP system as pioneered by the UK’s own Ruby Manor conference - Vestibule.

Vestibule is an open source application for fielding anonymous conference proposals. The main premise being that anonymous proposals encourage inclusion, and a level playing field for potentially lesser-known speakers. Once a proposal is submitted, the community can help shape and refine the synopsis and points of discussion, all the while preserving anonymity of the originator.

EuRuKo’s process was phased into two rounds with progression through the process based on public voting, number of views, number of comments, social network mentions and refinements made throughout the process. You can see their thorough explanation here.

I was pleased to learn my proposal had made it through the first stage of over 90 initial submissions and into the shortlist of 30 proposals. Subsequently after a further voting round I learned my proposal had been ranked sixth by the community thus cementing my spot at EuRuko 2013! To think I’ll be sharing a stage with Matz - the creator of Ruby, is seriously scary and exciting in equal measure.

Hopefully I’ll see you in Athens, and I hope to both educate and entertain with my talk entitled: Achieving zomgscale with Celluloid and JRuby

Thanks go out to Global Dev for sponsoring my attendance and making this possible.

If you’re a Ruby engineer, regardless if you’re an accomplished speaker or just a happy tinkerer, you could join us. We’re hiring and have openings in both our Windsor and London offices. We will also happily consider remote positions for the right candidates. So get in touch!

Last weekend the Global Personals development team dispatched Barry, Tim, Mat, Steve and me to the Scottish Ruby Conference. Set in the absolutely stunning Crieff Hydro hotel complex - located in Perthshire in the Scottish highlands. Unfortunately, whilst we didn’t get the opportunity to visit much of the surrounding area we did manage to take in plenty of inspiration from the fantastic talks given throughout the conference.

Between the five of us we chose to strategically divide our time and attendance between almost all of the talks given during the two days and three simultaneous conference tracks. I also managed to squeeze in an off-the-cuff lightning format talk on Celluloid - a concurrent object oriented programming framework for ruby that lasted around five minutes and was well received. Also serving as sleight practice for my upcoming talk on Celluloid at Euruko 2013 - the European ruby conference in Athens.

I’ll briefly summarise the talks we attended during the conference, in roughly the order they were given:

Day One

Keynote - Rachel Meyers on “Growing Programming”

The conference kicked off in style with a personalised view of growing one’s skills as a programmer, and how we can shape the community to enable this. I can imagine sales figures for the SICP book peaked shortly after.

James Edward Gray II on “The Patterns We All Need to Know”

This talk presented a neat summation of the most regularly encountered patterns in the wild. It was great to see their application in idiomatic ruby, although unfortunately the slides featuring code were all but unreadable from a row or two back. Thankfully the slides were published shortly after.

André Arko on “Deathmatch: Bundler vs. Rubygems.org”

An interesting glance at the evolution of Bundler and Rubygems.org. Many of us will remember the early days of bundler and how much time was spent Fetching source index. This talk covered the incremental updates that Bundler and Rubygems.org received and the consequential performance gains.

Natalia Buckley on “Thinking through making”

Natalia spends her time making. Making things. Making time for play. Making mistakes. It’s through this process of making that she learns. The most important thing I took away was to just get on and make. What you build doesn’t have to be perfect; it doesn’t even have to work. But through the process of making you learn, and the next time you’ll make something better.

Steve Klabnik on “Building awesome APIs with rails-api”

A talk on the rails-api gem. A slimmed-down flavour of rails designed for serving performant and maintainable APIs. Rather than providing any specific guidance for building sane APIs this talk covered tools, gems and standards available to API developers.

Tekin Suleyman on “Bootstrapping a Startup, a Developer’s Tale”

3 years ago, Tekin had an idea. He spoke to a few people and decided it was worthwhile building. Over the course of the next 3 years he built crowd.fm as a side project, spending hundreds of hours on it. After all that work and time, he’s come to the conclusion that he’s built a solution to a problem that didn’t exist. A really interesting yet cautionary tale of what it takes to attempt to bootstrap a side project into a business.

Kevin Rutherford on “Telling, not asking”

This was Kevin’s take on the Tell, don’t ask principle and how it is commonly misunderstood and almost impossible to achieve in a modern rails codebase. Kevin detailed his approach, constructed of a simple message bus and listeners responding to application events. A pattern that is prevalent in more enterprisey languages but not so commonly found in ruby. Kevin had a very unique speaking style that was dry and ever-so humorous. An enjoyable and enlightening talk.

Chris Kelly on “Down the rb_newobj() Rabbit Hole”

This talk had the potential to be quite a heavy subject to digest. Thankfully Chris guided us all the way down through MRI’s allocation implementation in a clear and concise style. In particular I found Chris’s slides to be very well presented. A thoroughly informative talk and one I personally found very thought provoking.

Erin O’Brien on “From burning desire to burning out: How to prevent the fizzle”

One of my favourite talks of the conference, Erin’s discussion of burn-out in developers was really eye-opening. She discussed the common signs of burn-out, and ways to attempt to handle it. With a background in psychology, she lent weight to her talk by backing everything up with research, but presented in an incredibly effective and accessible manner. Definitely worth the time to watch it when the video has been posted.

Jesse Toth on “Service Oriented Harmony”

Jesse presented a talk on her findings in the field when applying SOA architecture. It was an entertaining talk and resonated with much of our thinking at Global Personals since we apply SOA liberally. I do feel that Jesse somewhat side-stepped the principals and practices of testing, which to my mind is likely the most difficult concept to grasp in this field. However, this didn’t detract from the rest of the talk. I also learned shortly after that this was Jesse’s first time speaking publicly, which I would never have suspected.

Day Two

Mike Moore on “MiniTest & Rails: Total BFFs”

Mike’s talk on MiniTest 5 and integration with rails was very entertaining. I did feel it was pitched too high for me personally but it served as a great introduction for those not so familiar with the MiniTest framework. Bonus points must also be given for featuring My Little Ponies.

Kerri Miller on “Failure for Fun and Profit!”

Kerri is someone who enjoys being a novice, and in this talk she ran us through the reasons why and how. Being a novice means you are constantly learning; learning means that you will generally be failing more often than not. She encouraged everyone to redefine failure from the big, nasty thing we generally think it is, but instead a necessary state before success. A really inspiring talk, with a bunch of awesome quotes, including “If you’re always achieving what you try, you’re only trying the achievable,” and “Good ideas come from experience; experience comes from bad ideas.”

Jim Weirich on “Given - Designing a Language”

Having never seen Jim speak in person prior to this event I was very much looking forward to his talk on the Given testing DSL. I felt completely engrossed in Jim’s inimitable and highly conversational speaking style. Given is packed full of beautiful, idiomatic ruby tricks and I learned a few new ideas I can see myself applying to my own code in the future. I was especially impressed with the clean approach to re-raising errors and preserving their original backtraces.

Chris Parsons on “Leading software teams well”

The difference between management and leadership at first may seem like nothing more than an alternative word. Chris explained just how different the two things were: just because you’re a manager doesn’t make you a leader and, conversley, anyone can be a leader, no matter how junior. A lot to think about in how to lead teams, including snippets such as the perils of job titles: in Chris’ experience they occasionally stopped people working on something, because it “wasn’t in their job title.”

Franck Verrot on “Be Wrong, but try things”

Franck’s talk was an eye-opener for me. I am a stickler for doing things The Right Way, and often agonise over a particular approach to a problem rather than experimenting with failure in the trial and error fashion. The talk was full of practical examples of failure leading to new understanding, particularly in the study of biology. An inspirational talk.

Mathias Meyer on “Cooking Infrastructure with Chef”

Mathias gave a high-level discussion on the benefits of infrastructure automation tools, focussing on Chef. He walked through the setup of the pieces of a basic infrastructure including application and database servers, and the deployment of these pieces into different environments.

Austin Vance on “Ruby the Arduino Way”

Austin’s talk was a fun and light-hearted look at the arduino kit and how simple it can be to interface with ruby. Austin’s creation was a t-shirt firing cannon. Very well received and scheduled cleverly - occurring just prior to the closing keynote.

Closing Keynote - Evan Henshaw-Plath on “Ruby History”

Evan’s talk detailed ruby’s progression from little-known and experimental, to the fully-fledged and immensely popular language it is today. The talk was humurous and very slick in it’s presentation. I felt it was the perfect end to a fantastic two days of conferencing.

Closing thoughts

As ever, much of the valuable discussion was performed in the hallways. I met up with some old colleagues and acquaintances and so many rubyists of whom I was internet-familiar but had yet to meet in person. The setting was perfect, the catering fantastic and conference as a whole felt very slick. Credit must go to the organisers, and I will definitely return next year, although hopefully as a speaker.

Thanks go to Global Personals for sponsoring our attendance to the conference. Did I mention we’re hiring? ;)

Introduction

On Friday a couple of us attended jQuery UK 2013 in Oxford. It’s only the second time the conference has been held, and from what I could gather it was far larger this time around. The day started really well, with signposts and “white rabbit” paw prints leading the way to the venue right from the station. The theme of the conference was Alice in Wonderland (hence the rabbit prints) and the sponsor hall came complete with a giant jabberwocky.

After coffee and pastries it was soon time to get started. Everyone piled in to the main room and started tweeting dubious jokes as soon as they realised there was a live Twitter wall on the right of the stage. The lack of wifi was a hot topic, but it wasn’t much of an issue. John Wards of White October, the conference organisers, kicked things off with a short introduction and then handed over to Brendan Eich, who, being the creator of JavaScript, understandably received a great reception.

Brendan Eich

Brendan’s talk focussed mainly on features of the upcoming ECMAScript 6, and the future ES7. There was a lot of interesting content, although not a huge amount of new information for people who keep up to date with es-discuss regularly.

A couple of demos of the Unreal engine, compiled to JavaScript, running in the browser with WebGL got an understandably excited reaction. The power of Emscripten and asm.js looks absolutely amazing and is definitely something to keep an eye on over the next year or so. You can read more about that on the Mozilla blog, and I highly recommend you watch the very impressive videos.

Brendan also showed off his bright orange mobile phone and mentioned that the first developer preview Firefox OS devices will be available soon from Geeksphone.

Next up was Richard D. Worth, executive director of the jQuery foundation.

Richard D. Worth

Richard’s talk, “jQuery 1.9 and 2.0 - Present and Future”, gave us some great insights into the direction of the jQuery project. He started with some impressive statistics, pointing out that 91.2% of the websites that use JavaScript use jQuery. That’s a huge number, far higher than I would have guessed.

He also confirmed the dropping of support for Internet Explorer 8 and below in jQuery 2.0, and made it clear that the team will continue to maintain the 1.9 branch, guaranteeing an API match between 1.x and 2.x as of 1.10. There was also confirmation of the new modular build system for 2.0, which means you are free to customise your copy of jQuery to suit your needs. For example, if you’re only targeting newer environments with querySelectorAll support, you can leave out Sizzle completely.

After Richard there was time for a short break before moving on to Remy Sharp.

Remy Sharp

For a jQuery conference, Remy’s talk about why not to use jQuery sounded like it might be a little out of place, but actually turned out great. He explained why we shouldn’t rely upon jQuery for everything when native JavaScript and DOM methods can often achieve the same effect in the same amount of code.

A lot of his points are even more valid with the release of jQuery 2.0. If you’re only supporting modern environments there is no reason to not start using, for example, native ES5 array methods such as Array.prototype.forEach instead of jQuery.each, and CSS3 animations instead of jQuery UI.

He introduced a new open source project, min.js, aimed at providing a barebones jQuery-compatible API that simply wraps native DOM methods. It could be a useful tool for anyone aiming to strip jQuery out of projects that only need to work in newer browsers.

Next to take to the stage was Adam J. Sontag.

Adam J. Sontag

Having never seen a talk by Adam before, this one was definitely one of my highlights. He’s a hugely entertaining speaker. The talk was pretty much contradictory to Remy’s, explaining that “jQuery is a Swiss Army knife and that’s OK”. He pointed out that jQuery was his “gateway drug” to JavaScript, something which a lot of people in the room could relate to, and a very good reason to stop criticising people who use it. He finished up with a brilliant rebuttal of the common “jQuery is too big” argument. “jQuery is gigantic… you know what else is gigantic? JPEGs!”.

After Adam’s talk there was time for Doug Neiner before lunch.

Doug Neiner

I was looking forward to Doug’s talk in particular. It was all about finite state machines in JavaScript with Machina.js. Unfortunately for Doug he seemed to be suffering from a bit of a sore throat and was quite squeaky throughout.

He gave a good introduction to finite state machines in JavaScript with an excellent analogy to the transmission in a car, and demonstrated how the code to model this could quickly get messy. He then introduced Machina and showed how it can make the previous example much cleaner, before moving on to a real-world online/offline state example.

Overall, Doug’s talk was excellent and packed with decent example code. There’s too much of it to talk about here so I highly recommend you take a look through his slides.

Lunch

Next up was a break for lunch, with loads of free food, Red Bull, and a giant cake provided by White October for their 10th anniversary. The live Twitter wall was back online, so cue many more poor jokes. Tweets containing script tags seemed to be a favourite, just to make sure the organisers were properly sanitising user input!

By this point, the one song that had been playing on repeat during all the intervals was definitely starting to become annoying!

The lunch break also featured a set of short “rising stars” talks, by upcoming speakers. Unfortunately, I didn’t get a chance to see any of these but by all accounts they were pretty good.

Garann Means

After the break, Garann Means talked about developing full-stack applications in JavaScript with Node, jQuery, websockets and MVC, with a focus on events.

This talk was interesting to us since we have been taking a similar approach with a lot of our JavaScript tools and libraries recently here at Global Personals.

Following Garann was Ilya Grigorik, Web Performance Engineer at Google.

Ilya Grigorik

For me, Ilya’s talk was the highlight of the day. The title, “Wait, Chrome DevTools can do THAT?” didn’t suggest anything particularly exciting, but I learned so much from it. Again, there’s so much to cover that I can’t fit it in here, but watch out for a follow-up post when we’ve spent a bit of time playing around with all the new Chrome DevTools features.

He started with a reminder of Chrome Canary, and by the look of Twitter since this was a good idea since there were a surprising number of people unaware of it. There was also a really useful link to Code School lessons on DevTools.

Highlights of the talk included custom themes and panels, remote mobile debugging and HAR exports. I can’t recommend this talk highly enough, so check out the slides. After another short break, John Bender was up next.

John Bender

John is a member of the jQuery mobile team and the co-creator of Vagrant. His talk was about improving the speed of DOM manipulation with jQuery by applying ideas from category theory.

The talk was fairly maths-heavy but relatively easy to follow and included some really interesting ideas. I, along with many of the other attendees, had never really considered the application of mathematical models to something like jQuery and the DOM.

John also introduced Wield, a small library developed out of many of the ideas presented in his talk, designed to abstract the collections and DOM manipulation APIs of jQuery and offer improved performance. It’s definitely worth checking out.

After that we moved on to Joe Pettersson, interaction designer and UI engineer.

Joe Pettersson

Joe talked about testing, with a big focus on legacy versions of Internet Explorer. We were reminded of the IE VMs now available from Microsoft and showed how you can use tools such as Jenkins and Selenium to automate browser testing across virtual machines communicating with a central hub.

There were some really useful ideas in this talk, especially since this is something we are currently looking at implementing here. Joe’s sentiment of wanting to “stab the JScript engine with a fork” was shared by many. This talk was the penultimate session of the day and we moved on to Jason Scott, a jQuery mobile team member at BlackBerry.

Jason Scott

In the final talk of the day, Jason explained the virtues of building an experience for your users rather than “just another framework”. He showed us how BlackBerry leverage the power of jQuery mobile to create better user experiences, including a demo of a custom jQuery mobile theme designed to mimic the native BlackBerry 10 interface.

After party

And with that we were done! Everyone moved back into the sponsor hall for jBeery, a jQuery-inspired beer festival featuring various rebranded local ales and more good food. This was an ingenious idea and it seems like everyone had a brilliant evening. Who wouldn’t when there’s free beer all night? I stayed around for most of the night and chatted to some really interesting people.

Overall, I thoroughly enjoyed jQuery UK 2013 and would like to thank White October and John Wards for putting on such a brilliant event. Let’s hope for more of the same next year.

Keyword arguments

def wrap(string, before: "<", after: ">")
  "#{before}#{string}#{after}" # no need to retrieve options from a hash
end

# optional
wrap("foo")                                  #=> "<foo>"
# one or the other
wrap("foo", before: "#<")                    #=> "#<foo>"
wrap("foo", after: "]")                      #=> "<foo]"
# order not important
wrap("foo", after: "]", before: "[")         #=> "[foo]"

One of the nice things about this compared to hash-as-fake-keyword-args is that it errors when you make a mistake.

begin
  wrap("foo", befoer: "#<")
rescue ArgumentError => e
  e.message                                  #=> "unknown keyword: befoer"
end

You can use double splat to capture all keyword arguments, just like a single splat to capture all regular arguments. You can also use the double splat to unpack a hash to keyword arguments.

# arguments
def capture(**opts)
  opts
end
capture(foo: "bar")                          #=> {:foo=>"bar"}

# keys must be symbols
opts = {:before => "(", :after => ")"}
wrap("foo", **opts)                          #=> "(foo)"

The old hash style syntax is still accepted for keyword arguments, so you can update your method definitions without having to fix all the callers.

wrap("foo", :before => "{", :after => "}")   #=> "{foo}"

If you’re writing a library that needs to be compatible across ruby 2.0 and 1.9 the old hash-as-fake-keyword-args trick still works, and can be called as if it was using keyword arguments

def wrap(string, opts={})
  before = opts[:before] || "<"
  after = opts[:after] || ">"
  "#{before}#{string}#{after}"
end

wrap("foo", before: "[", after: "]")         #=> "[foo]"

%i and %I symbol array literal

%i{an array of symbols}   #=> [:an, :array, :of, :symbols]

and with interpolation

%I{#{1 + 1}x #{2 + 2}x}   #=> [:"2x", :"4x"]

Refinements

Refinements is a neat idea, but the original implementation came with a weird edge cases and possible performance penalties, so what we get with Ruby 2.0.0 is a rather scaled back, and slightly less useful version of the original.

You create a refinement to a class, name spaced inside a module

module NumberQuery
  refine String do
    def number?
      match(/\A[1-9][0-9]*\z/) ? true : false
    end
  end
end

this refinement isn’t visible by default

"123".respond_to?(:number?)   #=> false

once you declare that you are using the module with the refinement, then it becomes visible.

using NumberQuery
"123".number?                 #=> true

however using is only available at the top level, and only applies to lines after it in the same source file it’s declared. You also get a warning that refinements are experimental if you’re running with warnings enabled.

Module#prepend

Module gains #prepend as a compliment to #include, it works just like include, but it inserts the module in to the inheritance chain as if it were a subclass rather than a superclass.

Object
superclass
included module
class
prepended module

This takes over from Rails’ #alias_method_chain or the trick of aliasing a method to a new name before redefining it and calling the original.

class Foo
  def do_stuff
    puts "doing stuff"
  end
end

module Wrapper
  def do_stuff
    puts "before stuff"
    super
    puts "after stuff"
  end
end

class Foo
  prepend Wrapper
end

Foo.new.do_stuff

outputs:

before stuff
doing stuff
after stuff

there’s also ::prepended and ::prepend_features method that work like ::included and ::append_features

Unbound methods from a module can be bound to any object

This one might sound like gibberish, or some minor change to something you’ll never use, but it’s actually a really great new feature.

You can get ahold of a method object from any class or module, with instance_method

Object.instance_method(:to_s)   #=> #<UnboundMethod: Object(Kernel)#to_s>

However, this method isn’t bound to anything, it has no self, and can’t be called. To call it you have to bind the method to an object, but methods have to be bound to an object of the same class as the one the method was taken from.

But now we can take a method from a module and bind it to any object

module Bar
  def bar
    "bar"
  end
  def baz
    "baz"
  end
end

Bar.instance_method(:bar).bind(Object.new).call   #=> "bar"

This means define_method also accepts unbound methods for modules, which will let us implement a selective include

module Kernel
  def from(mod, include: [])
    raise TypeError, "argument must be a module" unless Module === mod
    include.each do |name, original|
      define_method(name, mod.instance_method(original || name))
    end
  end
end

class Foo
  from Bar, include: {:qux => :bar}
end

f = Foo.new
f.qux                 #=> "bar
f.respond_to?(:baz)   #=> false

const_get understands namespaces

class Foo
  module Bar
    Baz = 1
  end
end

Object.const_get("Foo::Bar::Baz")   #=> 1

#to_h as standard for ‘convert to hash’

Hash, along with ENV, nil, Struct, and OpenStruct all get a #to_h method which returns a hash

{:foo => "bar"}.to_h               #=> {:foo=>"bar"}
nil.to_h                           #=> {}
Struct.new(:foo).new("bar").to_h   #=> {:foo=>"bar"}
require "ostruct"
open = OpenStruct.new
open.foo = "bar"
open.to_h                          #=> {:foo=>"bar"}

There is also a Hash() method, like Array(), that delegates to #to_h

Hash({:foo => "bar"})              #=> {:foo=>"bar"}
Hash(nil)                          #=> {}

Array#bsearch and Range#bsearch

Array and Range get a binary search method with #bsearch. This has two modes of working, find-minimum and find-any. Both modes take a block, and the array will have to be sorted with regards to this block.

In find-minimum mode it will return the first element greater than or equal to a chosen value. To use this mode you supply a block that returns true when the supplied element is greater than or equal to the chosen value, and false otherwise.

array = [2, 4, 8, 16, 32]
array.bsearch {|x| x >= 4}       #=> 4
array.bsearch {|x| x >= 7}       #=> 8
array.bsearch {|x| x >= 9}       #=> 16
array.bsearch {|x| x >= 0}       #=> 2
array.bsearch {|x| x >= 33}      #=> nil

In find any mode you need to supply a block that returns a positive number if the supplied element is less than your chosen value, a negative number if it is greater, and 0 if it is the chosen value.

array.bsearch {|x| 4 <=> x}      #=> 4
array.bsearch {|x| 7 <=> x}      #=> nil

Your block can return 0 for a range of values, in which case any of them may be chosen.

array = [0, 4, 7, 10, 12]
array.map {|x| 1 - x / 4 }       #=> [1, 0, 0, -1, -2]
array.bsearch {|x| 1 - x / 4 }   #=> 4 or 7

Enumerable#lazy

#lazy called on any Enumerable (Array, Hash, File, Range, etc) will return a lazy enumerator, that doesn’t perform any calculations till it is forced to. In addition, elements of the enumerable will be sent though the whole chain one by one, rather than evaluating the entire enumerable at each step. In some cases this will result in less work.

You can deal with infinite collections

[1,2,3].lazy.cycle.map {|x| x * 10}.take(5).to_a   #=> [10, 20, 30, 10, 20]

Or avoid consuming large resources, when you know you’ll only need a little

File.open(__FILE__).lazy.each.map(&:chomp).reject(&:empty?).take(3).force

This example only reads as many lines as it takes to find 3 that aren’t empty.

#lazy does incur a performance penalty, so you’ll need to make sure you’re only using it when it makes sense.

There’s also a Enumerator::Lazy class to create your own lazy enumerators. The example above could be written as:

def populated_lines(file, &block)
  Enumerator::Lazy.new(file) do |yielder, line|
    string = line.chomp
    yielder << string unless string.empty?
  end.each(&block) # evals block, or returns enum if nil, like stdlib
end

populated_lines(File.open(__FILE__)).take(3).force

Lazy Enumerator#size and Range#size

Enumerator#size will return the size of an enumerator, without evaluating the whole Enumerator. Range benefits from this too.

array = [1,2,3,4]
array.cycle(4).size    #=> 16
array.cycle.size       #=> Infinity
# nil is returned if the size can't be calculated
array.find.size        #=> nil
# Range too
(1..10).size           #=> 10

To enable this in Enumerators returned from your own code, Enumerator.new now accepts an argument to calculate the size.

This can be a value

enum = Enumerator.new(3) do |yielder|
  yielder << "a"
  yielder << "b"
  yielder << "c"
end

enum.size              #=> 3

or an object responding to #call

def square_times(num, &block)
  Enumerator.new(-> {num ** 2}) do |yielder|
    (num ** 2).times {|i| yielder << i}
  end.each(&block)
end

square_times(6).size     #=> 36

#to_enum (and it’s alias #enum_for) also now take a block to calculate the size, so the above could be written like so:

def square_times(num)
  return to_enum(:square_times) {num ** 2} unless block_given?
  (num ** 2).times do |i|
    yield i
  end
end

square_times(6).size     #=> 36

Rubygems Gemfile support

Rubygems can now use your Gemfile (or Isolate, or gem.deps.rb) to install gems and load activation information.

You can install the gems listed in your Gemfile (and their dependancies) by specifying the --file (or -g) option. This only uses the Gemfile, not Gemfile.lock, so only versions specified in the Gemfile will be respected.

gem install --file Gemfile

To load activation information (the version of the gems to use) specify the RUBYGEMS_GEMDEPS environment variable. The value for this should be the path to your Gemfile, but you can use - to have Rubygems auto-detect this.

Like install, this only uses the Gemfile, ignoring Gemfile.lock, so it’s less strict than Bundler. It also only activates the specified versions, you’ll still need to require the gems in your code.

It does have the advantage that as it’s built in, there’s no need for something similar to bundle exec or Bundlers binstubs, just run your app like normal.

export RUBYGEMS_GEMDEPS=-
# start your app

There is only basic support for the Gemfile format, it doesn’t understand the gemspec declaration for example, but it’s a handy feature that will hopefully evolve and become a bit more robust, and it’s very useful if Bundler isn’t working out for you.

A rough approximation of bundle install --path vendor/bundle can be had with

gem install --file Gemfile --install-dir vendor/gem

export GEM_HOME=vendor/gem
export RUBYGEMS_GEMDEPS=-
# start your app

RDoc markdown support

RDoc now understands markdown, to run rdoc with markdown formatting set the markup option

rdoc --markup markdown

This can be saved in your project with a .doc_options file so you don’t need to repeat it every time

rdoc --markup markdown --write-options

Use warn like puts

warn now works just like puts, taking multiple arguments, or an array, and outputting them, but on stderr rather than stdout

warn "foo", "bar"
warn ["foo", "bar"]

Logger compatible syslog interface

Now it’s super easy to switch between logging to a file descriptor and to syslog, no more picking one and then writing to that interface. You can even log to stdout in development, and syslog in production, without having to update all your logging calls.

if ENV["RACK_ENV"] == "production"
  require "syslog"
  logger = Syslog::Logger.new("my_app")
else
  require "logger"
  logger = Logger.new(STDOUT)
end

logger.debug("about to do stuff")
begin
  do_stuff
rescue => e
  logger.error(e.message)
end
logger.info("stuff done")

TracePoint

TracePoint is a new object oriented version of the old Kernel#set_trace_func. It allows you to trace the execution of your Ruby code, this can be really handy when debugging code that isn’t terribly straight forward.

# set up our tracer, but don't enable it yet
events = %i{call return b_call b_return raise}
trace = TracePoint.new(*events) do |tp|
  p [tp.event, tp.event == :raise ? tp.raised_exception.class : tp.method_id, tp.path, tp.lineno]
end

def twice
  result = []
  result << yield
  result << yield
  result
end

def check_idempotence(&block)
  a, b = twice(&block)
  raise "expected #{a} to equal #{b}" unless a == b
  true
end

trace.enable
a = 1
begin
  check_idempotence {a += 1}
rescue
end
trace.disable

outputs:

[:call, :check_idempotence, "/Users/mat/Dropbox/ruby-2.0.0.rb", 841]
[:call, :twice, "/Users/mat/Dropbox/ruby-2.0.0.rb", 834]
[:b_call, nil, "/Users/mat/Dropbox/ruby-2.0.0.rb", 850]
[:b_return, nil, "/Users/mat/Dropbox/ruby-2.0.0.rb", 850]
[:b_call, nil, "/Users/mat/Dropbox/ruby-2.0.0.rb", 850]
[:b_return, nil, "/Users/mat/Dropbox/ruby-2.0.0.rb", 850]
[:return, :twice, "/Users/mat/Dropbox/ruby-2.0.0.rb", 839]
[:raise, #<RuntimeError: expected 2 to equal 3>, "/Users/mat/Dropbox/ruby-2.0.0.rb", 843]
[:return, :check_idempotence, "/Users/mat/Dropbox/ruby-2.0.0.rb", 843]

Asynchronous Thread interrupt handling

Ruby threads can be killed or have an exception raised in them by another. This isn’t a terribly safe feature as the killing thread doesn’t know what the thread being killed is doing, you could end up killing a thread in the middle of some important resource allocation or deallocation. We now have a feature to deal with this more safely.

As an example, the stdlib timeout library works by spawning a second thread, which waits for the specified amount of time, and then raises and exception in the original thread.

Say we had a connection pool library that could cope fine with you failing to check back in a connection, but would fail if checking out or checking in a connection was interrupted. You’re writing a method to get a connection, make a request on it, then return it, and you suspect that users of this method may wrap it in a timeout.

def request(details)
  result = nil
  # Block will defer given exceptions if they are raised in this thread by
  # another thread till the end of the block. Exceptions are not rescured or
  # ignored, but handled later.
  Thread.handle_interrupt(Timeout::ExitException => :never) do
    # no danger of timeout interrupting checkout
    connection = connection_pool.checkout
    # if checkout took too long, handle the interrupt immediately, effectively
    # raising the pending exception here
    if Thread.pending_interrupt?
      Thread.handle_interrupt(Timeout::ExitException => :immediate)
    end
    # allow interrupts during IO (or C extension call)
    Thread.handle_interrupt(Timeout::ExitException => :on_blocking) do
      result = connection.request(details)
    end
    # no danger of timeout interrupting checkin
    connection_pool.checkin(connection)
  end
end

This method can safely be wrapped in a timeout, and the connection will always be completely checked out, and if it completes the request, will always be completes checked in. If the timeout happens during checkin, it won’t interrupt the checkin, but it will still be raised at the end of the method.

This is a slightly contrived example, but it covers the main points of this great new feature.

Garbage collection improvements

There are a few improvements to the garbage collector in Ruby 2.0, the main one making Ruby play nicer with Copy-on-Write. This means applications that fork multiple processes, like a Rails app running on Unicorn, will use less memory.

The GC::Profiler class also gets a ::raw_data method, to return the raw data of the profile as an array of hashes, rather than a string, making it easier to log this data with say, statsd.

GC::Profiler.enable # turn on the profiler
GC.start # force a GC run, so there will be some stats
GC::Profiler.raw_data
#=> [{:GC_TIME=>0.0012150000000000008, :GC_INVOKE_TIME=>0.036716,
#   :HEAP_USE_SIZE=>435920, :HEAP_TOTAL_SIZE=>700040,
#   :HEAP_TOTAL_OBJECTS=>17501, :GC_IS_MARKED=>0}]

ObjectSpace.reachable_objects_from

This method returns all the objects directly reachable from the given object.

require "objspace"

Response = Struct.new(:code, :header, :body)
res = Response.new(200, {"Content-Length" => "12"}, "Hello world!")

ObjectSpace.reachable_objects_from(res)
#=> [Response, {"Content-Length"=>"12"}, "Hello world!"]

You can combine this with ObjectSpace.memsize_of to get an idea of the memory size of an object and all the objects it references; very handy for debugging memory leaks

def memsize_of_all_reachable_objects_from(obj)
  memsize = 0
  seen = {}.tap(&:compare_by_identity)
  to_do = [obj]
  while obj = to_do.shift
    ObjectSpace.reachable_objects_from(obj).each do |o|
      next if seen.key?(o) || Module === o
      seen[o] = true
      memsize += ObjectSpace.memsize_of(o)
      to_do << o
    end
  end
  memsize
end

memsize_of_all_reachable_objects_from(res)   #=> 192

Optimised backtrace

Backtrace strings are now only created on demand, from a light weight collection of object, rather than with each exception. You can get ahold of these objects with caller_locations or Thread#backtrace_locations. Curiously they aren’t available from an Exception.

def foo
  bar
end

def bar
  caller_locations
end

locations = foo
locations.map(&:label)   #=> ["foo", "<main>"]
locations.first.class    #=> Thread::Backtrace::Location

caller also now accepts a limit as well as an offset, or a range

def bar
  caller(2, 1)
end
foo   #=> ["/Users/mat/Dropbox/ruby-2.0.0.rb:361:in `<main>'"]

def bar
  caller(2..2)
end
foo   #=> ["/Users/mat/Dropbox/ruby-2.0.0.rb:368:in `<main>'"]

Zlib streaming support

Zlib gets support for streaming decompression and improved support for streaming compression.

When decompressing files it might be reasonable to buffer the whole compressed file in to memory, but the uncompressed data may be many hundreds of MB. Zlib::Inflate#inflate now accepts a block that will get chunks of data as the file in uncompressed, this can process the uncompressed data without needing to buffer all of it in memory at once.

require "zlib"

inflater = Zlib::Inflate.new(Zlib::MAX_WBITS + 32)
File.open("app.log", "w") do |out|
  inflater.inflate(File.read("app.log.gz")) {|chunk| out << chunk}
end
inflater.close

In a similar fashion Zlib::Deflate#deflate also accepts a block, as you feed data in to #deflate the block will only be called when enough data has accumulated to warrant efficient compression.

deflater = Zlib::Deflate.new
File.open("app.log.gz", "w") do |out|
  File.foreach("app.log", 4 * 1024) do |chunk|
    deflater.deflate(chunk) {|part| out << part}
  end
  deflater.finish {|part| out << part}
end

Multithreaded Zlib processing

Zlib no longer holds the Global Interpreter Lock while compressing/uncompressing data, allowing parallel processing on gzip, zlib and deflate streams. This also means your application can continue to respond while Zlib works in the background.

require "zlib"

# processes 4 files in parallel, using 4 cores if required
threads = %W{a.txt.gz b.txt.gz c.txt.gz d.txt.gz}.map do |path|
  Thread.new do
    inflater = Zlib::Inflate.new(Zlib::MAX_WBITS + 32)
    File.open(path.chomp(File.extname(path)), "w") do |out|
      inflater.inflate(File.read(path)) {|chunk| out << chunk}
    end
    inflater.close
  end
end
# do other stuff here while Zlib works in other threads
threads.map(&:join)

Default UTF-8 encoding

You can now use useful characters outside of US-ASCII without magic encoding comments or inscrutable escape sequences

currency = "€"   #=> "€"

Binary string shortcut

String#b as a shortcut to get an ASCII-8BIT (aka binary) copy of a string

s = "foo"
s.encoding     #=> #<Encoding:UTF-8>
s.b.encoding   #=> #<Encoding:ASCII-8BIT>

String#lines, #chars, etc return an Array

lines, chars, codepoints, bytes now all return arrays rather then enumerators

s = "foo\nbar"
s.lines        #=> ["foo\n", "bar"]
s.chars        #=> ["f", "o", "o", "\n", "b", "a", "r"]
s.codepoints   #=> [102, 111, 111, 10, 98, 97, 114]
s.bytes        #=> [102, 111, 111, 10, 98, 97, 114]

These still accept a block for backwards compatibility, but you should use #each_line etc for that use case

The similarly named methods on IO, ARGF, StringIO, and Zlib::GzipReader still return enumerators, but are deprecated, use the each_* versions.

__dir__

Returns the file path to the executing file, like __FILE__, but without the file name, useful for things like

YAML.load_file(File.join(__dir__, "config.yml"))

__callee__ returns the method as called

__callee__ is back to returning the name of the method as called, not as defined with an aliased method. This can actually be useful.

def do_request(method, path, headers={}, body=nil)
  "#{method.upcase} #{path}"
end

def get(path, headers={})
  do_request(__callee__, path, headers)
end
alias head get

get("/test")    #=> "GET /test"
head("/test")   #=> "HEAD /test"

RegExp engine is changed to Onigmo

Onigmo is a fork of the Oniguruma regexp engine used by 1.9, with a few more features. The new features seem Perl-inspired, this seems to be a good refrence.

(?(cond)yes|no)

if cond is matched, then match against yes, if cond is false match against no cond references a match either by group number or name, or is a look-ahead/behind

example only matches a trailing cap if there is a leading cap

regexp = /^([A-Z])?[a-z]+(?(1)[A-Z]|[a-z])$/

regexp =~ "foo"   #=> 0
regexp =~ "foO"   #=> nil
regexp =~ "FoO"   #=> 0

Hash#default_proc= now accepts nil

No longer will you have to call hash.default = nil to clear hash.default_proc: now your code will actually make sense!

hash = {}
hash.default_proc = Proc.new {|h,k| h[k] = []}
hash[:foo] << "bar"
hash[:foo]                                       #=> ["bar"]
hash.default_proc = nil
hash[:baz]                                       #=> nil

Array#values_at returns nil for each value that is out-of-range

Previously #values_at would behave in an unexpected way when given a range, and you’d only get a single nil for all out-of-range indexes, now you get one for each

[2,4,6,8,10].values_at(3..7)   #=> [8, 10, nil, nil, nil]

Option for File.fnmatch? to expand braces

If for some reason you ever find yourself needing to do shell filename glob matches in Ruby you’ll be happy to know you can now use pattens like {foo,bar}

# 3rd argument enables the brace expansion
File.fnmatch?("{foo,bar}", "foo", File::FNM_EXTGLOB)   #=> true
File.fnmatch?("{foo,bar}", "foo")                      #=> false
# or together multiple options old-school C style
casefold_extglob = File::FNM_CASEFOLD | File::FNM_EXTGLOB
File.fnmatch?("{foo,bar}", "BAR", casefold_extglob)    #=> true

Shellwords calls #to_s on arguments

Shellwords#shellescape and #shelljoin will now call #to_s on the arguments, this is particularly useful with Pathname.

require "pathname"
require "shellwords"

path = Pathname.new("~/Library/Application Support/").expand_path
Shellwords.shellescape(path)
\#=> "/Users/mat/Library/Application\\ Support"

Shellwords.join(Pathname.glob("/Applications/A*"))
\#=> "/Applications/App\\ Store.app /Applications/Automator.app"

system and exec now close non-standard file descriptors by default

when using exec all open files/sockets, other than STDIN, STDOUT and STDERR will be closed for the new process. This was previously an option with exec(cmd, close_others: true) but it’s now the default.

Protected methods treated like private for #respond_to?

protected methods are now hidden from #respond_to? unless true is passed as a second argument, just like private methods.

class Foo
  protected
  def bar
    "baz"
  end
end

f = Foo.new
f.respond_to?(:bar)         #=> false
f.respond_to?(:bar, true)   #=> true

#inspect no longer calls #to_s

Under Ruby 1.9 #inspect gained the odd behaviour of delegating to #to_s if a custom #to_s method had been defined, this has been removed

class Foo
  def to_s
    "foo"
  end
end

Foo.new.inspect   #=> "#<Foo:0x007fb4a2887328>"

LoadError#path

Load error now has a #path method to retrieve the path of the file that couldn’t be loaded. That was already in the message, but now it’s more easily accessible to code

begin
  require_relative "foo"
rescue LoadError => e
  e.message   #=> "cannot load such file -- /Users/mat/Dropbox/foo"
  e.path      #=> "/Users/mat/Dropbox/foo"
end

Process.getsid

getsid returns the processes session ID. This only works on unix/linux systems

Process.getsid   #=> 240

Signal.signame

A signame method has been added to get the name for a signal number

Signal.signame(9)   #=> "KILL"

Error on trapping signals used internally

Signal.trap now raises an ArgumentError if you try and trap :SEGV, :BUS, :ILL, :FPE, or :VTALRM. These are used internally by Ruby, so you wouldn’t be able to trap them anyway.

True thread local variables

As of Ruby 1.9 Thread#[], #[]=, #keys and #key? would get/set fiber local variables, Thread now gets the methods #thread_variable_get, #thread_variable_set, #thread_variables, #thread_variable? as equivalents that are thread local

Fiber local:

b = nil

a = Fiber.new do
  Thread.current[:foo] = 1
  b.transfer
  Thread.current[:foo]
end

b = Fiber.new do
  Thread.current[:foo] = 2
  a.transfer
end

p a.resume   #=> 1

Thread local:

b = nil

a = Fiber.new do
  Thread.current.thread_variable_set(:foo, 1)
  b.transfer
  Thread.current.thread_variable_get(:foo)
end

b = Fiber.new do
  Thread.current.thread_variable_set(:foo, 2)
  a.transfer
end

p a.resume   #=> 2

Better error on joining current/main thread

If you attempt to call #join or #value on the current or main thread you now get a ThreadError raised, which inherits from StandardError, rather than ‘fatal’ which inherits from Exception.

begin
  Thread.current.join
rescue => e
  e   #=> #<ThreadError: Target thread must not be current thread>
end

Mutex changes

I can’t think of a particularly interesting example for this, but you can now check if the current thread owns a mutex.

require "thread"

lock = Mutex.new
lock.lock
lock.owned?                      #=> true
Thread.new {lock.owned?}.value   #=> false

Also affecting Mutex, methods that change the state of the mutex are no longer allowed in signal handlers: #lock, #unlock, #try_lock, #synchronize, and #sleep.

And apparently #sleep may wake up early, so you’ll need to double check the correct amount of time has passed if precise timings are important.

sleep_time = 0.1
start = Time.now
lock.sleep(sleep_time)
elapsed = Time.now - start
lock.sleep(sleep_time - elapsed) if elapsed < sleep_time

Custom thread and fiber stack sizes

The following environment variables can be set to alter the stack sizes used by threads and fibers. Ruby only checks these as your program starts up.

• RUBY_THREAD_VM_STACK_SIZE: vm stack size used at thread creation. default: 128KB (32bit CPU) or 256KB (64bit CPU).
• RUBY_THREAD_MACHINE_STACK_SIZE: machine stack size used at thread creation. default: 512KB or 1024KB.
• RUBY_FIBER_VM_STACK_SIZE: vm stack size used at fiber creation. default: 64KB or 128KB.
• RUBY_FIBER_MACHINE_STACK_SIZE: machine stack size used at fiber creation. default: 256KB or 256KB.

You can get the defaults with

RubyVM::DEFAULT_PARAMS   #=> {:thread_vm_stack_size=>1048576,
                              :thread_machine_stack_size=>1048576,
                              :fiber_vm_stack_size=>131072,
                              :fiber_machine_stack_size=>524288}
                              

Stricter Fiber#transfer

A fiber that has been transferred to now must be transferred back to, instead of cheating and using resume.

require "fiber"

f2 = nil

f1 = Fiber.new do
  puts "a"
  f2.transfer
  puts "c"
end

f2 = Fiber.new do
  puts "b"
  f1.transfer # under 1.9 this could have been a #resume
end

f1.resume

RubyVM::InstructionSequence

RubyVM::InstructionSequence isn’t new, but it gains a couple of features, and even more helpfully, detailed documentation.

You can now get the instruction sequence an existing method

class Foo
  def add(x, y)
    x + y
  end
end

instructions = RubyVM::InstructionSequence.of(Foo.instance_method(:add))

and when you have that instruction sequence you can get some details about where it was defined

instructions.path            #=> "/Users/mat/Dropbox/ruby-2.0.0.rb"
instructions.absolute_path   #=> "/Users/mat/Dropbox/ruby-2.0.0.rb"
instructions.label           #=> "add"
instructions.base_label      #=> "add"
instructions.first_lineno    #=> 654

ObjectSpace::WeakMap

This class is mainly intended to be part of WeakRef’s implementation, so you should probably use that (require "weakref"). It holds a weak reference to the objects stored, which means they may be garbage collected.

map = ObjectSpace::WeakMap.new
# keys can't be immediate values (numbers, symbols), and you must use the
# exact same object, not just one that is equal.
key = Object.new

map[key] = "foo"
map[key]                #=> "foo"
# force a garbage collection run
sleep(0.1) and GC.start 
map[key]                #=> nil

Top level define_method

define_method can now be used at the top level; it doesn’t have to be inside a class or module

Dir["config/*.yml"].each do |path|
  %r{config/(?<name>.*)\.yml\z} =~ path
  define_method(:"#{name}_config") {YAML.load_file(path)}
end

No warning for unused variables starting with _

This method will generate warnings that the family, port, and host variables are unused.

def get_ip(sock)
  family, port, host, address = sock.peeraddr
  address
end

Using underscores will stop the warnings, but lose the self-documenting nature of the code

def get_ip(sock)
  _, _, _, address = sock.peeraddr
  address
end

As of Ruby 2.0.0 we can get the best of both worlds by starting the variables with an _

def get_ip(sock)
  _family, _port, _host, address = sock.peeraddr
  address
end

Proc#== and #eql? removed

Under Ruby 1.9.3 Procs with the same body and binding were equal, but you’d only get procs like this when you’d cloned one from another. This has now been removed, which is no great loss as it wasn’t really very useful.

proc = Proc.new {puts "foo"}

proc == proc.clone   #=> false

ARGF#each_codepoint

ARGF (which is a concatination of the files supplied on the command line) gets an #each_codepoint method like IO.

count = 0
ARGF.each_codepoint {|c| count += 1 if c > 127}
puts "there are #{count} non-ascii chacters in the given files"

Time#to_s

The encoding of the string returned from Time#to_s changes from ASCII-8BIT (aka binary) to US-ASCII.

Time.now.to_s.encoding   #=> #<Encoding:US-ASCII>

Random parameter of Array#shuffle! and #sample now called with max arg

This is a small change, that will probably have no affect on you at all, but now when supplying a random param to the #shuffle!, #shuffle, and #sample methods on Array it’s now expected to take a max argument, and return an integer between 0 and max, rather than a float between 0 and 1.

array = [1, 3, 5, 7, 9]
randgen = Object.new
def randgen.rand(max)
  max                           #=> 4
  1
end
array.sample(random: randgen)   #=> 3

CGI HTML5 tag builder

CGI from the stdlib gets an HTML5 mode for its tag builder interface.

require "cgi"
cgi = CGI.new("html5")
html = cgi.html do
  cgi.head do
    cgi.title {"test"}
  end +
  cgi.body do
    cgi.header {cgi.h1 {"example"}} +
    cgi.p {"lorem ipsum"}
  end
end
puts html

The old #header method (to send the HTTP header) is now called #http_header, although as long as you’re not in HTML5 mode it’s aliased as #header for backwards compatibility.

CSV::dump and ::load removed

CSV::dump and CSV::load have been removed. They allowed you to dump/load an array of Ruby objects to a CSV file, and have them serialised and deserialised. They’ve been removed as they were unsafe.

Iconv removed

Iconv has been removed, in preference of String#encode.

Where previously you might have written something like:

require "iconv"
Iconv.conv("ISO-8859-1", "UTF8", "Résumé")   #=> "R\xE9sum\xE9"

You’d now write

"Résumé".encode(Encoding::ISO_8859_1)        #=> "R\xE9sum\xE9"

Syck removed

The Syck YAML parser has been removed in favour of Psych (libyaml bindings), and Ruby now comes bundled with libyaml. The YAML interface in Ruby stays the same, so this shouldn’t have and affect on your code.

io/console

io/console isn’t new, but the documentation is so now you can actually figure out how to use it. The Ruby 2.0.0 NEWS file claims the IO#cooked and #cooked! methods are new, but they seem to be available in 1.9.3.

require "io/console"
IO.console.raw!
# console in now in raw mode, disabling line editing and echoing
IO.console.cooked!
# back in cooked mode, line editing works like normal

#raw! and #raw get two new arguments, min and time.

IO.console.raw!(min: 5) # reading from console buffers for 5 chars
IO.console.raw!(min: 5, time: 1) # read after 1 second if buffer not full

io/wait

io/wait adds a #wait_writeable method that will block till an IO can be written to. #wait gets renamed to #wait_readable, and there’s a #wait alias for backwards compatibility.

require "io/wait"
timeout = 1
STDOUT.wait_writable(timeout)   #=> #<IO:<STDOUT>>

Net::HTTP performance improvements

Net::HTTP now automatically requests and decompresses gzip and deflate compression by default. This should play very nicely with the new non-GIL-blocking Zlib.

SSL sessions are also now reused, cutting down on time spent negotiating connections.

Net::HTTP can specify the host/port to connect from

If for some reason you need to specify the local host/port to connect from, along with the host/port to connect to, you now can

http = Net::HTTP.new(remote_host, remote_port)
http.local_host = local_host
http.local_port = local_port
http.start do
  # ...
end

OpenStruct can act like a hash

OpenStruct gains #[], #[]= and #each_pair methods so it can be used like a hash.

require "ostruct"

o = OpenStruct.new
o.foo = "test"
o[:foo]               #=> "test"
o[:bar] = "example"
o.bar                 #=> example

It also gains #hash and #eql? methods, which are used internally by Hash to check equality. These allow it to play better as a hash key, with equal objects acting as the same key.

Timeout support in Resolv

Resolv now supports custom timeouts

require "resolv"

resolver = Resolv::DNS.new
resolver.timeouts = 1 # 1 second
resolver.getaddress("globaldev.co.uk").to_s   #=> "204.232.175.78"

It will also take an array, and work its way through the timeouts, retrying after each. You could implement exponential back-off with

resolver.timeouts = [1].tap {|a| 5.times {a.push(a.last * 2)}}

And that’s it…

Please let me know if there’s anything missing or incorrect here. Also, don’t forget that we’re hiring…

A year ago today we officially opened the company’s London office, just opposite Waterloo Station. Starting off with six developers and engineers who were previously working in our Windsor office, we’ve now more than doubled in size.

A key reason for opening the office was making it easier to attract London-based staff or those who would find it difficult to get to our headquarters in Windsor. With easy access from across the South-East, we now have staff travelling in from across London, Essex, Sussex and Surrey.

In May 2012, following Global Personals’ acquisition of Woowise, we were joined by Marcus, James, Gordon and Jeff, swelling our numbers to thirteen. Matching the facilities in Windsor, the team enjoy Friday bacon sandwiches, a fridge with cans of soft drinks and of course the Xbox 360 for post-work games.

It’s a great location with excellent views across London and we expect to keep adding to our numbers in London throughout 2013 and beyond.

So happy first birthday to us! Here’s to the next 12 months in London.

What are source maps?

Obviously, our JavaScript library is served in the form of a single, minified file on production servers. During development, it’s a different story, with the various scripts that make up that library all included individually, in their original uncompressed form. This means it’s easy to identify any JavaScript issues during development, because most browser development tools will indicate at least a file name and line number when they encounter an error. But if we experience a live issue, that debugging becomes a whole lot harder. Have a look at this screenshot:

How are you meant to know where the error actually occurred? This is where source maps come in. A source map is effectively a technique to map a minified file back to its original, uncompressed state. For an excellent introduction to JavaScript source maps, I recommend that you read the HTML5 Rocks article on the subject. What we want to achieve is this:

Notice how much easier it is to see the error. The clever thing is that the browser has actually downloaded the minified file, so the average user will be completely unaffected by the addition of a source map. The browser will only download the source map and unminified file when you ask it to (currently, that’s done by opening the developer tools of Chrome or Safari and enabling source maps).

Generating our source map

The first step was to integrate the generation of our source map into our build process. We use Grunt to produce production builds of our JavaScript library, and we use the Google Closure Compiler for minification. Fortuantely, the Closure Compiler can also generate source maps. There is already an excellent Grunt plugin that adds Google Closure support, so we use that too. All that remains is to add a section to the Grunt script that uses Closure to produce a source map:

 
grunt.initConfig({
    "closure-compiler": {
        sourcemap: {
            js: "build/library.js",
            jsOutputFile: "build/library.min.js",
            options: {
                create_source_map: "build/library.min.js.map",
                source_map_format: "V3"             
            }        
        }     
    } 
});

That works well so now any time we run our build script, we get a source map file in the output directory. Unfortunately, the Closure Compiler is not yet capable of adding the reference to the source map to the minified script which means we add it ourselves. The current source maps specification gives the following format. It’s just a normal JavaScript comment so browsers that don’t support source maps will completely ignore it:

//@ sourceMappingURL=library.min.js.map 

Adding this comment manually after each build would be annoying. Luckily, it’s very easy to extend Grunt so we’ve written a simple plugin to handle it for us. By including that plugin in our build script, all we have to do is add another small section to the Grunt config:

grunt.initConfig({
    "closure-compiler": {
        // Closure compiler configuration
    },
    "append-sourcemapping": {
        main: {
            files: {
                "build/library.min.js": "library.min.js.map"
            }
        }
    }
});

Fighting with Internet Explorer

It was never going to be that simple. Turns out that there’s a problem using source maps in all versions of Internet Explorer under certain circumstances. If you’ve ever seen or used conditional JavaScript compilation in IE, you may have recognised the format of that source map include comment earlier.

If a library with a source mapping comment is included in a page after conditional compilation has been turned on (it’s turned on with a comment containing the string @cc_on @), you’re going to end up with JavaScript errors. Internet Explorer will attempt to parse your source map comment as part of the program, and will almost certainly fail because the source map URL (library.min.js.map in the example above) will not be a valid identifier.

For more details of the issue, see this jQuery bug report. That ticket also details a very simple workaround. By simply wrapping the source mapping comment in a multiline comment, Internet Explorer will not attempt to parse it, even when conditional compilation is active. All we had to do was make a little change to our Grunt plugin to get the comment output looking like this:

/*
//@ sourceMappingURL=library.min.js.map
*/

And that’s all there is to it. We now have working source maps in WebKit browsers, and Firefox support is supposed to be in the works. Over the next few months, we should see more and more chatter about this emerging technique so get a head start and implement source maps in your JavaScript libraries as soon as possible.

Each server has multiple instances of memcached running on different ports. There are multiple sets for both the “data” and “view” segments. Each of the new servers has 64GB RAM.

Migration Challenges

The main challenges when planning to migrate the data is mitigating the impact on the platform. memcached is a core element. It’s used to store HTML fragment caches and various parts of both site and member data.

There is a tight line between getting the migration completed versus keeping the application as quick as possible during the whole procedure.

Approaches

There are a few approaches which are briefly outlined below:

• Big Bang. Update all the configuration files and simply push that code live. This has a huge hit on performance as all the new instances are cold, it is however the quickest option.
• Migration instance by instance. Each instance would be moved and relevant configuration files pushed live. This way only 1 or X instances is “cold”. This limits impact, but, also takes time as only 1 instance can be done at any one time.
• Write to both. The application code could be altered to make all writes to both the existing set of instances and the new. This isn’t great as we’d need a full QA cycle to validate the code works. It’s also something we’d have to implement then pull back out down the line.
• Mirror traffic. Similar to the above, but, this time lower down the stack. Essentially duplicate all the TCP level traffic so it warms in parallel and keeps both instances in sync meaning new writes, deletes occur on both existing and new sets.

Solution

In the end the best solution which minimised the impact to the platform was to mirror all the traffic to the existing instances across to the new instances. tcpcopy helped us to get this done. As well as migrating data, tcpcopy could be used to load test, stress test and benchmark.

The plan involved running the tcpcopy for 24 hours, see how full the new instances reached then decided if another 24 hours should elapse before the code changes to point at new instances are deployed.

There would be some data “missing” however all freqently used data keys, core and shared information will all exist and as such mitigate the performance concerns.

tcpcopy

Installing tcpcopy is pretty straight forward. Assuming all requirements are in place:

wget --no-check-certificate \
  https://github.com/downloads/wangbin579/tcpcopy/tcpcopy-0.6.5.tar.gz && \
  tar xvfz tcpcopy-0.6.5.tar.gz && cd tcpcopy-0.6.5 && \
  ./configure && make && make install

Once that’s installed on your target server (new cache servers) you’ll need to setup the iptables rules. Note there are a couple of extra’s here to ensure that connections from the localhost and from the office IP were not queued to allowing monitoring.

modprobe ip_queue;
iptables -A OUTPUT -o lo -j ACCEPT;
iptables -A OUTPUT -d 1.2.3.4/32 -j ACCEPT;
iptables -A OUTPUT -p tcp --match multiport --sports 11211:11212 -j QUEUE;

You can then fire up the intercepter, the -d option backgrounds the process as a daemon.

intercept -d

Next you’ll need to head over to the source servers. In our case this is the existing cache servers.

tcpcopy -d -x 11211-192.168.2.184:11211
tcpcopy -d -x 11212-192.168.2.184:11212

The format of the copy is localport-targetip:targetport. You can tweak with more options as well, see the manual for full details. You can also include logging if needed by using -l /path/to/log.

Monitoring

In order to see how the copies were going 3 monitoring methods were used.

Direct monitoring via telnet

[root@server ~] telnet 0 11211
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
stats
STAT pid 1234
STAT uptime 1234
STAT time 111111111
...

Ruby key dumper

This small Ruby key dumper allows you to dump out a selection of the keys. It’s not great and memcached doesn’t allow a full dump of keys in anyway. It allows you to see what’s there and validate it’s content though.

Overall instance stats from memcache-top

This is a perl program available on google code which allows you monitor a given set of instances, much like regular top.

Recently I’ve spent a lot of time playing around with JSLint and JSHint, both for my work on jslinterrors.com and in preparation for the use of JSLint to enforce a coding style. During this investigation, I came across a potentially nasty little bug in both tools.

A bit of background

In JavaScript (as in most languages) you cannot use a reserved word as an identifier. For example, the following will fail with a syntax error:

var default = 10;

In the most widely supported iteration of the language, ECMAScript 3, it is also illegal to use a reserved word as an object property identifier:

var myObject = {
    default: 10
};

However, in the most recent version of the language specification ECMAScript 5, which now enjoys widespread support across the major browsers, as well as in Node.js, a change has been made to allow the use of reserved words as object property identifers. This means that the above example is actually valid and will not cause a syntax error in any modern browser.

The details of the change can be seen if we delve into the ECMAScript spec. First, here’s the relevant snippets from the old ES3 version:

The important part from that snippet is the Identifier production of PropertyName. If we look at the grammar for identifiers, we see that reserved words are not allowed:

If we now look at the same section from the ES5 spec, we can see the slight change to the grammar for PropertyName:

It’s a small change, but a significant one. The Identifer production has been replaced with IdentifierName, which is effectively the same as Identifier but without the restriction on reserved words. It’s this change that means in ES5-supporting environments we are able to use reserved words as object property identifiers.

How do JSLint and JSHint fit into this?

Both JSLint and JSHint will warn you about the use of a reserved word as an identifier by default. If you are writing code that will only run in ES5-supporting environments, you can set an option (the es5 option) that surpresses this warning:

/*jslint es5: true */
var myObject = {
    default: 10 // Valid JavaScript, does not cause a JSLint/JSHint error
};

This is useful, expected behaviour. However, when the es5 option is set, JSLint and JSHint will also allow you to use a reserved word as a variable or function identifier, as well as an object property identifier:

/*jslint es5: true */
var default = 10; // Invalid JavaScript, does not cause a JSLint/JSHint error

This is incorrect and will actually throw a syntax error in all environments, regardless of ES5 support. This is a problem, especially if you rely on JSLint/JSHint to give confidence in your code (perhaps as part of your build process), as you could easily end up releasing broken code.

Tracking down the cause of the bug

Following through the JSLint source led me to the property_name function, which is called when the parser encounters a object property identifier. The first thing it does is call another function, optional_identifier, passing through true as its lone argument. This is the only call to optional_identifier that ever passes through an argument and the function signature for optional_identifier does not contain any formal parameters.

At this point, I am guessing that when JSLint was written this was a slight oversight and the author simply forgot to handle this specific case within the optional_identifier function. A quick look at the implementation of the function reveals that no arguments are considered, and the error in question is raised if the identifier is a reserved word and the es5 option does not evaluate to true:

function optional_identifier() {
    if (next_token.identifier) {
        advance();
        if (option.safe && banned[token.string]) {
            warn('adsafe_a', token);
        } else if (token.reserved && !option.es5) {
            warn('expected_identifier_a_reserved', token); // This line raises the error in question
        }
        return token.string;
    }
}

We can make a simple change to this function to take into account the argument passed in by the call mentioned earlier and ensure JSLint continues to raise this error when it encounters a reserved word as a variable or function identifier:

function optional_identifier(prop) { // Added a formal parameter
    if (next_token.identifier) {
        advance();
        if (option.safe && banned[token.string]) {
            warn('adsafe_a', token);
        } else if (token.reserved && (!prop || !option.es5)) { // Handle that parameter
            warn('expected_identifier_a_reserved', token);
        }
        return token.string;
    }
}

The fix is pretty much the same for JSHint, since it started life as a straight fork of JSLint and much of the internals remain unchanged. I opened a pull request in JSHint on GitHub, which has since been merged in, meaning this is no longer an issue in JSHint. The problem remains in JSLint for now, but we hope to get a patch submitted to it in the near future.

As you can probably imagine, the White Label Dating platform contains a lot of JavaScript. It’s been written by many people over many years, and when you have numerous people working on the same code you’re always likely to end up with a bit of a mess. The code may work as intended. It may even follow a lot of best practices and be easily extensible. But it almost certainly won’t be nice to look at, or easy to read, and that can often lead to problems with maintenance in the future.

In an attempt to make our JavaScript easier on the eye, we have started using Douglas Crockford’s JSLint tool to enforce a consistent style throughout. I’m sure you have your own opinions on JSLint, and I’m well aware that many developers (myself included) think that some of the rules that JSLint wants you to follow are verging on ridiculous (for example, forbidding the use of the ++ increment operator). However, by fine-tuning the various options available to you, and putting up with the occasional annoyance, you can use JSLint to your advantage and end up with a JavaScript code base that is much easier to maintain.

Configuring JSLint

We decided to configure JSLint on a per-file basis. This way, every file can have its own options which allows us a bit of flexibility. If you want to allow the use of the increment and decrement operators in a file, and you want common browser properties like document to appear predefined (JSLint will throw undefined errors if they don’t), you can do so with a jslint directive at the top of the code:

/*jslint plusplus: true, browser: true */

There are some JSLint options that allow you to get away with sloppy code formatting (such as the white option, which will allow “messy white space”). Unfortunately, there is no way to enforce that this option is always turned on, so we have to simply make developers aware that such options must not be set in any files. If they were, it would kind of defeat the point.

Enforcing the use of JSLint

Now that we’ve defined the standards for JavaScript development, we need a way to enforce them. It’s very easy to simply paste your JavaScript into the online tool at on the JSLint website, but that’s a bit too much work. Instead, we wanted to automate the process, and we have achieved this with a Git pre-commit hook, which prevents anyone from committing JavaScript that doesn’t pass JSLint when it should.

There’s a couple of things that are required for our hook to work. The first is Node.js (along with NPM, the Node Package Manager). Once you’ve got those up and running, you can globally install node-jslint which will add the executable to your path:

npm install -g jslint

Now we can run JSLint from the terminal, so we can get on with our Git hook. It will run JSLint against all JavaScript files staged for commit, and even provide colour-coded feedback. It needs to be placed in repository/.git/hooks/pre-commit:

#!/bin/sh

files=$(git diff --cached --name-only --diff-filter=ACM | grep ".js$")
if [ "$files" = "" ]; then 
    exit 0 
fi

pass=true

echo "\nValidating JavaScript:\n"

for file in ${files}; do
    result=$(jslint ${file} | grep "${file} is OK")
    if [ "$result" != "" ]; then
        echo "\t\033[32mJSLint Passed: ${file}\033[0m"
    else
        echo "\t\033[31mJSLint Failed: ${file}\033[0m"
        pass=false
    fi
done

echo "\nJavaScript validation complete\n"

if ! $pass; then
    echo "\033[41mCOMMIT FAILED:\033[0m Your commit contains files that should pass JSLint but do not. Please fix the JSLint errors and try again.\n"
    exit 1
else
    echo "\033[42mCOMMIT SUCCEEDED\033[0m\n"
fi

Keeping track of our hook

One of the problems with Git hooks is that they are not part of the repository itself (since they live within the .git directory). We don’t expect to be changing our hook regularly, but it would be nice to have the hook itself under version control. We achieved this by simply finding a suitable location in the repository, and creating the hook script there. When anyone sets up their development environment, a single command can be run to activate the hook, by symlinking to the hook file from the Git hooks directory:

ln -s /path/to/repo/pre-commit-jslint /path/to/repo/.git/hooks/pre-commit

In the near future we would like to get this step integrated into a more general “development environment setup” script, so that new starters or anyone setting up a clean dev environment can kick off one script and come back later to a fully working local copy of the app.

A number of reasons have been cited, all of which stem from claims on Twitter that the speaker lineup lacks racial and gender diversity and in turn, according to the team at BritRuby, this has resulted in one or more of the sponsors pulling out.

globaldev were proud to have been announced as one of the sponsors for BritRuby less than two weeks ago. We were delighted to put our name behind a large-scale Ruby event where the intention was to help improve the Ruby developer community throughout the UK which is no mean feat given the immense disparity between the significant number of companies seeking talented developers within this niche and the unfortunate rarity of high quality Ruby developers on the job market.

The cancellation of BritRuby is a shame and we want to state categorically that globaldev backed BritRuby 100%. We want nothing more than to see the community become more diverse in every shape and form and we sincerely believe that events of the scale of BritRuby are the perfect platform to make more and more people aware of the huge demand in this particular area of development.