Ruby and/or Rails

Double Shot #1558

Mike Gunderloy — Wed, 30 Sep 2015 11:40:12 +0000

Ultimate Guide for Logging - Crowdsourced resource hosted by Loggly but with info on a bunch of different tools.
Gitcolony - Hosted or on-premises git solution with an emphasis on code reviews.
rails_stuff - A medley of little modules and helpers for Rails.

Double Shot #1557

Mike Gunderloy — Tue, 29 Sep 2015 11:27:12 +0000

Nomad and Otto - New resource manager/scheduler and orchestration tool from Hashicorp.
chasing the shiny and new - Thoughts on the trend in developers of always needing to be moving to the latest language.
Monitor and analyzer your Passenger apps today with Union Station - Phusion has built a custom app monitoring solution specifically for their own server.

Double Shot #1556

Mike Gunderloy — Mon, 28 Sep 2015 11:05:33 +0000

Zulip - Open source group chat from Dropbox.
webXray - Run-it-yourself analyzer to see which web pages are pinging which trackers.
What's new in Rails 5? - Another rundown on the major features that are coming down the pike.

Generating Word Search Puzzles

Sat, 26 Sep 2015 06:00:00 +0000

A walk-through of the author's word search puzzle generator — 7-minute read

Double Shot #1555

Mike Gunderloy — Mon, 21 Sep 2015 11:47:53 +0000

Learn Enough Command Line to Be Dangerous - Draft ebook from Michael Hartl for you Rails developers.
A Curl Cheat Sheet - Condensed list of the most common command line options for curl.

Default Scopes are an Anti-Pattern

Sat, 19 Sep 2015 06:00:00 +0000

ActiveRecord's default_scope feature is exposed as an anti-pattern, with examples showing two common ways it is abused. It is proposed that explicit scopes are a superior solution — 5-minute read

Double Shot #1554

Mike Gunderloy — Fri, 18 Sep 2015 11:53:22 +0000

flex mock revival: flex mock 2.0 released! - One of the late Jim Weirich's gems has a new maintainer and a new version.
mpssh - Tool for executing commands on multiple ssh servers in parallel.
Not Very Good MRI Heap Analyzer - A way to get a look at the heap dump from a Rails application.

Double Shot #1553

Mike Gunderloy — Thu, 17 Sep 2015 11:05:59 +0000

StackProf: The Holy Grail of Rails Profiling - How to get some insight into production code without bollixing it up.
htty, the HTTP TTY - A more or less fun way to interact with web servers from the command line.
OpenSeadragon - Pure JS image viewer offering support for high-resolution zoomable images.

Double Shot #1552

Mike Gunderloy — Mon, 14 Sep 2015 11:40:42 +0000

Crab - Command-line SQL interface to your file system.
HTTP is obsolete. It's time for the distributed, permanent web - Neocities is implementing IPFS for distributed web sites.

Little Things: Heredocs

Sat, 12 Sep 2015 06:00:00 +0000

Another minor-but-useful feature of Ruby—heredocs—is demonstrated — 4-minute read

Double Shot #1551

Mike Gunderloy — Thu, 10 Sep 2015 12:31:09 +0000

httpbin - Online HTTP request/response service for testing a wide variety of endpoints in your client code.
flexbox in 5 minutes - Learn the ins and outs of this new CSS feature in an interactive tour.

Double Shot #1550

Mike Gunderloy — Wed, 09 Sep 2015 11:35:10 +0000

Rails Composer with Rails Tutorials - Kickstarter trying to fund further development on this "Rails generator on steroids".
Keeping Your Logs from Becoming an Unreadable Mess - Using tagged logging in Rails.
You've got Mail: Action Mailer + Letter_Opener - A gem to open mails inside browser tabs during development.

Little Things: Hashes & Procs

Sat, 05 Sep 2015 06:00:00 +0000

The interchangability of hashes and arity-1 procs is demonstrated as one of the endearing little features of Ruby — 3-minute read

Double Shot #1549

Mike Gunderloy — Fri, 04 Sep 2015 11:30:55 +0000

Debug Rails Faster with Quiet Assets & Quieter Logs - How to make Rails be a bit less verbose in its logging.
Announcing RubyMotion 4.0: free Starter edition, cross-platform games, watchOS 2.0 - Ruby for iOS continues to improve.

Ideas are Cheap

Sat, 29 Aug 2015 06:00:00 +0000

Paralyzed by a fear of ruining a great idea, the author set himself a challenge to prove that he had far more ideas to draw on — 3-minute read

Reducing a Number to Its Sign

Wed, 05 Aug 2015 06:00:00 +0000

A simple technique is presented for extracting the sign of a number — 3-minute read

Writing a Klondike Puzzle Solver

Tue, 04 Aug 2015 06:00:00 +0000

In which the author presents a Ruby program that solves Sam Lloyd's famous "Back from the Klondike" puzzle — 13-minute read

Flipper: Insanely Easy Feature Flipping

John Nunemaker — Mon, 03 Aug 2015 12:00:00 +0000

Cross posted from JohnNunemaker.com as it seems relevant here too.

                                   __
                               _.-~  )
                    _..--~~~~,'   ,-/     _
                 .-'. . . .'   ,-','    ,' )
               ,'. . . _   ,--~,-'__..-'  ,'
             ,'. . .  (@)' ---~~~~      ,'
            /. . . . '~~             ,-'
           /. . . . .             ,-'
          ; . . . .  - .        ,'
         : . . . .       _     /
        . . . . .          `-.:
       . . . ./  - .          )
      .  . . |  _____..---.._/ _____
~---~~~~----~~~~             ~~

Nearly three years ago, I started work on Flipper. Even though there were other feature flipping libraries out there at the time, most notably rollout, I decided to whip up my own. Repeating others is, after all, one of the better ways to level up your game.

My main issue with rollout was that it was inflexible. You couldn’t change the ways in which a feature was enabled (ie: adding percentage of time rollout). You had to use redis. The list goes on. I poked around and couldn’t find anything like what I was looking for and I was in the mood to create, so I started flipper.

Most of the work was done off and on over the course of a few weeks. At the time, I was working on traffic graphs for GitHub and I wanted a way to turn features on/off in a flexible way.

Naming is hard

Flipper started as a simple ripoff of rollout with the primary difference being the use of adapters for storage instead of forcing redis. I struggled through awkward terminology and messy code for a while, until a great conversation with Brandon Keepers led me to the lingo flipper uses today: Actor, Feature and Gate (thanks Brandon!)

An actor is the thing trying to do something. It can be anything. On GitHub, the actor can be a user, organization or even a repository. Actors must respond to flipper_id. If you plan on using multiple types of actors, you can namespace the flipper_id with the type (ie: “User:6”, “Organization:12”, or “Repository: 2”).

A feature is something that you want to control enabled-ness for. On SpeakerDeck, I have a feature for search. With the click of a button, I can disable search if it is causing issues. On GitHub, we do thousands of feature checks per second across nearly 30 features (at the time of this writing) in different states of enabled-ness. If I told you what they were for I would have to kill you.

A gate determines if a feature is enabled for an actor. There are currently five gates — boolean, actor, group, % of actors and % of time. Amongst these you can rollout a new feature or control an existing one in whatever way you desire.

The Gates

The boolean gate allows completely enabling or disabling a feature. Think of it as a short cut to turning a feature fully on or fully off quickly. Enabling the boolean gate means the feature is on all the time for everyone. Disabling the boolean gate clears all enabled gates so the feature is completely off. Think of disable like a reset.

flipper = Flipper.new(adapter)
flipper[:search].enable # turn on
flipper[:search].disable # turn off
flipper[:search].enabled? # check

The actor gate allows enabling a feature for one or more specific actors. If you wanted to enable a new feature for one of your friends, you could use this gate.


flipper = Flipper.new(adapter)

flipper[:search].enable_actor user # turn on for actor
flipper[:search].enabled? user # true

flipper[:search].disable_actor user # turn off for actor
flipper[:search].enabled? user # false

The group gate allows enabling a feature for one or more groups. A group is a named block of code that returns true or false for a given actor. You could have a group for everyone in your company, or only engineering, or perhaps all users in the US or Europe. Anything your heart can imagine can be converted to a group and the entire group can be enabled at once.


Flipper.register(:admins) do |actor|
  actor.respond_to?(:admin?) &amp;&amp; actor.admin?
end

flipper = Flipper.new(adapter)

flipper[:search].enable_group :admins # turn on for admins
flipper[:search].disable_group :admins # turn off for admins

person = Person.find(params[:id])
flipper[:search].enabled? person # check if enabled, returns true if person.admin? is true

The percentage of actors gate allows slowly enabling a feature for a percentage of actors. As long as you continue to increase the percentage, an actor will consistently remain enabled. This allows for careful rollouts of a feature to everyone without overwhelming the system as a whole.


flipper = Flipper.new(adapter)

# turn search on for 10 percent of users in the system
flipper[:search].enable_percentage_of_actors 10

# checks if actor's flipper_id is in the enabled percentage by hashing
# user.flipper_id.to_s to ensure enabled distribution is smooth
flipper[:search].enabled? user

# turn search off for percentage of actors, other gates could retur true still
flipper[:search].disable_percentage_of_actors # sets to 0

The percentage of time gate allows enabling a feature for a random percentage of time. This is great for dark shipping and load testing. We actually used somehing similar to this to launch traffic graphs. We wanted to be positive that we could stand up to real traffic, so we performed ajax requests behind the scenes based on a percentage of time to the new feature. This allowed us to crank up the traffic, hit a bottleneck, kill the traffic, fix the bottlneck and repeat.


flipper = Flipper.new(adapter)


# turn on logging for 5 percent of the time
# could be on during one request and off the next
# could even be on first time in request and off second time
flipper[:logging].enable_percentage_of_time 5

# turn off logging for percentage of time
flipper[:logging].disable_percentage_of_time # sets to 0

All the gates are fully documented in the flipper repo as well.

Adapters

The adapter pattern is used to store which gates gates are enabled for a given feature. This means you can store flipper’s information however you desire. At the time of this writing, several adapters already exist, such as in memory, pstore, mongo, redis, cassandra, and active record. If one of those doesn’t tickle your fancy, creating a new adapter is really easy. The API for an adapter is this:

features – Get the set of known features.
add(feature) – Add a feature to the set of known features.
remove(feature) – Remove a feature from the set of known features.
clear(feature) – Clear all gate values for a feature.
get(feature) – Get all gate values for a feature.
enable(feature, gate, thing) – Enable a gate for a thing.
disable(feature, gate, thing) – Disable a gate for a thing.

At GitHub, we actually use a SQL adapter fronted by memcache for performance reasons.

Instrumentation

Flipper is wired to be instrumented out of the box, using ActiveSupport::Notifications API (though AS::Notifs are not specifically required). I even included automatic statsd instrumentation for those that are already using statsd.

require "flipper/instrumentation/statsd"
statsd = Statsd.new # or whatever your statsd instance is
Flipper::Instrumentation::StatsdSubscriber.client = statsd

If statsd doesn’t work for you, you can easily customize wherever you want to instrument to (ie: InfluxDB, New Relic, etc.).

Performance

Flipper was built based on my time working on Words with Friends and to be used at GitHub, so you can rest easy that it was built with performance in mind. The adapter API is intentionally made to allow for fetching all gate values for a feature in one network call and there is even (optional) built in memoization of adapter calls, including a Rack middleware which enables memoizing the fetching a feature for the duration of a request.

I’ve also thought about making it easy to allow for batch loading of features, though I haven’t needed this yet on any site I’ve worked on, so for now it remains a thought rather than an implementation.

Web UI

As a cherry on top, I’ve also created a rack middleware web UI for controlling flipper, which can be protected by any authentication you need. Below are a couple screenshots (at the time of this writing).

List of features

Viewing individual feature

All the gates can be manipulated to enable features however you would like through the click of a button or the clack of a keyboard.

Conclusion

Flipper is ready for the prime time. As I said earlier, we are now using it on GitHub.com for thousands of feature checks every second. The API changed a bit in 0.7, but is pretty stable now. Drop it in your next project and give it a try. If you do, please let me know (email or issue on the repo) as I love to know how people are using things I’ve worked on.

Writing a Simple Recursive Descent Parser

Thu, 30 Jul 2015 06:00:00 +0000

A simple implementation of a field-based query string, with binary operations, using a recursive descent parser — 6-minute read

tar.gz in Ruby

Thu, 23 Jul 2015 06:00:00 +0000

A method is described for reading and writing tar and gzip files, using only the Ruby standard library — 5-minute read

Mazes for Programmers

Wed, 08 Jul 2015 06:00:00 +0000

The announcement of a completed project with expressions of relief and disbelief, and a brief animation to celebrate the occassion. The author looks forward to other projects to come. — 1-minute read

A week with a Rails Security Strategy (and page update)

Tue, 05 May 2015 14:01:05 +0000

Two things, a page update and my new article “A week with a Rails Security Strategy“.

Page update: This site hasn’t been very active recently, I’m working to make it more useful. Enter your e-mail in the header to notify you when it’s ready.

I published a new article “A week with a Rails Security Strategy” on my site:

You’re a busy person. Security is not a very visible feature in most applications, and so sometimes it’s not a priority. Of course you know it’s important, but how can you fit it into your busy schedule?

The answer may be in the power of habits and a Rails security strategy…

Review: Redis Applied Design Patterns

Obie Fernandez — Mon, 01 Dec 2014 15:56:59 +0000

Redis is one of the most popular NoSQL databases, and for good reason. Even though you wouldn't want to use it as your primary datastore, Redis is a high-performance complement for caching frequently used data and running certain types of calculations such as leaderboards. I'm far from a master of Redis, but my presentations about Redis are some of my most popular online content, getting thousands of views every month.

In the Packt Publishing book, Redis Applied Design Patterns, author Arun Chinnachamy provides a concise guide to Redis for experienced developers wondering what Redis can do for them. The book opens with a crash-course 3-chapter primer on NoSQL for those coming from a background in relational databases. After that, the rest of the book focuses features unique to REDIS within the context of application use cases such as caching and auto-suggest features.

The initial chapters feel a bit rushed and amateur, featuring statements like this one that are likely to provoke anger or amusement in experienced developers (bold emphasis mine):

The main advantage of NoSQL is that there is no concept of normalization. This is the reason why you get more performance from a NoSQL database when seen against a normalized SQL database. There is a trade-off in that you are sacrificing data consistency in the NoSQL database, but the benefits achieved in doing so are higher.

Despite that, I'm still a fan of having a central point of reference for each major technology that I employ in my stack, which is one of the main reasons why I maintain that my Rails Way series is still valuable after all these years. Otherwise you're reduced to bouncing around a myriad of online sources, an effort that can be frustrating and overly time consuming. This case is no different, the Packt book is a good central point of reference to Redis, although I would advise ignoring the more subjective parts of the book and making use of it purely as a reference tool.

Overall the author employs a concise, direct style that makes it easy to read and digest. It is also chock full of pointers to external resources, making it a good all-around guide to have around when working with Redis on a day-to-day basis. It's only $11 for the ebook version, which makes it easy to recommend.

Raptor: A Forthcoming Ruby Web Server for Faster App Deployment

Peter Cooper — Tue, 21 Oct 2014 13:20:59 +0000

Raptor bills itself as a new Ruby "app server” and it claims to blow everything else out of the water performance-wise (by between 2-4x!) whether that’s Unicorn, Puma, Passenger, or even TorqueBox on JRuby. The bad news for now is there’s no source or repo yet and only a handful of people (including me) have been given a sneak peek, although a public beta is promised on November 25th.

The history of Ruby webapp deployment

The deployment of Ruby (and therefore Rails) webapps was a painful mess for years, a state I lamented 7 years ago in No True ‘mod_ruby’ is Damaging Ruby’s Viability on the Web. Thankfully, shortly thereafter a number of projects came out to make life easier, the most famous being Phusion Passenger (then known as mod_rails) in April 2008.

Things have continued to improve gradually over the years, with Passenger getting consistently better, and new approaches such as those offered by Unicorn and Puma, using JRuby, as well as proxying through Nginx, coming into the picture.

Enter Raptor

Raptor, a new entry to the burgeoning world of Ruby Web servers, boasts some compelling features. "Visibility" is cited as a key feature so that you can look ‘into’ your app and analyze its performance as easily as possible using a JSON API (so building your own tools around the API should be simple). Raptor also uses the HTTP parser from Node which itself was derived from Nginx’s HTTP parser; both are renowned for their speed and stability. Raptor boasts a zero-copy, concurrent, evented architecture which makes it efficient memory and IO-wise - so even if you have slow clients or a slow network, these won’t bring your app server to a stuttering standstill.

Another feature that jumped out at me is integrated caching. Raptor doesn’t rely on an external services like memcached or Redis at all, but is truly internal and optimized specifically for Web workloads. If you’ve never set up caching before, this could provide a big boost as with Raptor it’ll be available “out of the box”.

The initial results seem promising. Fabio Akita has already shared some early benchmark results which broadly mirror my own experience (disclaimer: as someone with rather little experience and authority in benchmarking, my benchmarks are oriented around Raptor’s own benchmarking suite) but, as always, YMMV and such benchmarks are often criticized.

The waiting game..

The team behind Raptor promise they’ll be releasing some interesting blog posts soon about the technology behind it, including how the cache is implemented and has been optimized, how the zero-copy system works and how it’ll benefit your code, and similar things. So keep an eye on rubyraptor.org, especially around November 25th.

Ruby’s Unary Operators and How to Redefine Their Functionality

Peter Cooper — Thu, 16 Oct 2014 05:05:16 +0000

In math, a unary operation is an operation with a single input. In Ruby, a unary operator is an operator which only takes a single 'argument' in the form of a receiver. For example, the - on -5 or ! on !true.

In contrast, a binary operator, such as in 2 + 3, deals with two arguments. Here, 2 and 3 (which become one receiver and one argument in a method call to +).

Ruby only has a handful of unary operators, and while it's common to redefine binary operators like + or [] to give your objects some added syntactic sugar, unary operators are less commonly redefined. In my experience, many Rubyists aren't aware that unary operators can be redefined and.. technically you can't "redefine an operator" but Ruby's operators frequently use specially named methods behind the scenes, and as you'll know.. redefining a method is easy in Ruby!

A Quick Example with -@

Let's ease into things with the - unary operator. The - unary operator is not the same thing as the - binary operator (where a binary operator has two operants). By default, the - unary operator is used as notation for a negative number, as in -25, whereas the - binary operator performs subtraction, as in 50 - 25. While they look similar, these are different concepts, different operators, and resolve to different methods in Ruby.

Using the - unary operator on a string in irb:

> -"this is a test"
NoMethodError: undefined method `-@' for "this is a test":String

The String class doesn't have unary - defined but irb gives us a clue on where to go. Due to the conflict between the unary and binary versions of -, the unary version's method has a suffix of @. This helps us come up with a solution:

str = "This is my STRING!"

def str.-@
  downcase
end

p str     # => "This is my STRING!"
p -str    # => "this is my string!"

We've defined the unary - operator by defining its associated -@ method to translate its receiving object to lower case.

Some Other Operators: +@, ~, ! (and not)

Let's try a larger example where we subclass String and add our own versions of several other easily overridden unary operators:

class MagicString < String
  def +@
    upcase
  end

  def -@
    downcase
  end

  def !
    swapcase
  end

  def ~
    # Do a ROT13 transformation - http://en.wikipedia.org/wiki/ROT13
    tr 'A-Za-z', 'N-ZA-Mn-za-m'
  end
end

str = MagicString.new("This is my string!")
p +str         # => "THIS IS MY STRING!"
p !str         # => "tHIS IS MY STRING!"
p (not str)    # => "tHIS IS MY STRING!"
p ~str         # => "Guvf vf zl fgevat!"
p +~str        # => "GUVF VF ZL FGEVAT!"
p !(~str)      # => "gUVF VF ZL FGEVAT!"

This time we've not only redefined -/-@, but the + unary operator (using the +@ method), ! and not (using the ! method), and ~.

I'm not going to explain the example in full because it's as simple as I could get it while still being more illustrative than reams of text. Note what operation each unary operator is performing and see how that relates to what is called and what results in the output.

Special Cases: & and *

& and * are also unary operators in Ruby, but they're special cases, bordering on 'mysterious syntax magic.' What do they do?

& and to_proc

Reg Braithwaite's The unary ampersand in Ruby post gives a great explanation of &, but in short & can turn objects into procs/blocks by calling the to_proc method upon the object. For example:

p ['hello', 'world'].map(&:reverse)  # => ["olleh", "dlrow"]

Enumerable#map usually takes a block instead of an argument, but & calls Symbol#to_proc and generates a special proc object for the reverse method. This proc becomes the block for the map and thereby reverses the strings in the array.

You could, therefore, 'override' the & unary operator (not to be confused by the equivalent binary operator!) by defining to_proc on an object, with the only restriction being that you must return a Proc object for things to behave. You'll see an example of this later on.

* and splatting

There's a lot of magic to splatting but in short, * can be considered to be a unary operator that will 'explode' an array or an object that implements to_a and returns an array.

To override the unary * (and not the binary * - as in 20 * 32), then, you can define a to_a method and return an array. The array you return, however, will face further consequences thanks to *'s typical behavior!

A Full Example

We've reached the end of our quick tour through Ruby's unary operators, so I wanted to provide an example that shows how to override (or partially override) them that should stand as its own documentation:

class MagicString < String
  def +@
    upcase
  end

  def -@
    downcase
  end

  def ~
    # Do a ROT13 transformation - http://en.wikipedia.org/wiki/ROT13
    tr 'A-Za-z', 'N-ZA-Mn-za-m'
  end

  def to_proc
    Proc.new { self }
  end

  def to_a
    [self.reverse]
  end

 def !
   swapcase
 end
end

str = MagicString.new("This is my string!")
p +str                   # => "THIS IS MY STRING!"
p ~str                   # => "Guvf vf zl fgevat!"
p +~str                  # => "GUVF VF ZL FGEVAT!"
p %w{a b}.map &str       # => ["This is my string!", "This is my string!"]
p *str                   # => "!gnirts ym si sihT"

p !str                   # => "tHIS IS MY STRING!"
p (not str)              # => "tHIS IS MY STRING!"
p !(~str)                # => "gUVF VF ZL FGEVAT!"

It's almost a cheat sheet of unary operators :-)

A Further Example: The TestRocket

TestRocket is a tiny testing library I built for fun a few years ago. It leans heavily on unary operators. For example, you can write tests like this:

+-> { Die.new(2) }
--> { raise }
+-> { 2 + 2 == 4 }

# These two tests will deliberately fail
+-> { raise }
--> { true }

# A 'pending' test
~-> { "this is a pending test" }

# A description
!-> { "use this for descriptive output and to separate your test parts" }

The -> { } sections are just Ruby 1.9+ style 'stabby lambdas' but, with assistance from Christoph Grabo, I added unary methods to them so that you can prefix +, -, ~, or ! to get different behaviors.

Hopefully you can come up with some more useful application for unary methods on your own objects ;-)

Africa Calling

Obie Fernandez — Tue, 14 Oct 2014 19:09:54 +0000

Reposting from my Medium blog

Africa Calling

Do. Or Do Not. There is No Try.

Obie Fernandez — Sun, 14 Sep 2014 18:16:18 +0000

A reminder that I come back to time and again.

Demo of QuickMVP

Obie Fernandez — Fri, 12 Sep 2014 14:45:47 +0000

The video features my partners at Javelin, Grace and Trevor, presenting QuickMVP to the NY Tech Meetup, one of the most prestigious in the world. I don't think our idea particular complicated or anything, but I'm proud of our execution on this one.

QuickMVP is a landing page builder + easy creation of Google Ads to drive early adopter traffic + tool suite to be able to analyze results of your experiments in a way that's compatible with lean startup best practices. This video is actually one of the best demos of the software out there. Learn why this product is driving more and more recurring revenue for us every month.

The Future of Consulting

Obie Fernandez — Tue, 05 Aug 2014 18:12:48 +0000

I am doing an online (live) panel discussion with special guests a little less than an hour from now.

https://www.airpair.com/airconf2014/keynote/future-of-consulting

Here is a brief list of topics that we hope to cover:

- state of software consulting today. static? growing?

- what are the external forces shaping change in the industry?

- opportunities of ongoing engagement outside traditional project-based models

- should firms be looking to unbundle services?

- or additional bundling and/or co-promotion of educational and othe rservices (ala Thoughtbot)

- what is the proper balance of generalists vs. specialists on staff? what are hiring challenges?

- how does a firm stay “top of mind” at clients between projects?

- deployment of proprietary tools make sense? shared IP?

- pricing: time vs. value based?

- new competitors and business models? real threat?

Feel free to add questions for the panelists on the talk page's chat section here. To access the chat, you must be registered for AirConf and RSVP for the panel. If you're a little late to the show (starts 3pm ET) don't worry, you can just hit play on the Youtube player and watch it from the beginning. We'll try to address any questions/comments posted before the scheduled end time around 4pm ET.

How to Write and Publish a Technical Book (and make lots of money)

Obie Fernandez — Tue, 29 Apr 2014 00:56:23 +0000

Over the course of the last few years, the tech publishing market has changed. Authors are in a much better position than ever before.

If you are serious about success as a technical author, here are your up-to-date instructions. As long as your content is solid and valuable to your audience, I believe this is how you make the most money. You'll also reap serious long-term benefits.

First of all, are you talking to a publisher about a book deal already? Maybe you have a book proposal and/or outline in hand already?

Good. That's a good starting place, but it's time to get in the driver's seat. Put the discussion with the publisher on hold. Be nice and tell them that you're interested in doing a book with them later in the year. Promise to get back to them once you make progress on your manuscript.

Many publishers will balk at this change of events, but don't worry about it. What I'm going to teach you is how to build leverage. You're going to need as much of it as possible later on when you go back to them.

Now, I'm assuming that you have a title and concept worked out.

At this point you should also have a first draft of a book outline. Go to leanpub.com and create an author account. As part of signing up you'll have to create a book. Enter your title and description. Follow the instructions for creating and uploading a cover image. It doesn't have to be fancy.

Done? Publish your leanpub landing page and announce the book project to your social network. You want to start generating interest from your audience of early adopters as soon as possible. You also want the search engines to pick up your new landing page and start building a foundation for traffic to flow its way.

Time to start writing the book. (I can give you advice on how to be an effective writer, but that's a separate topic. Make sure to subscribe to the mailing list below to find out more.) While you are writing, your leanpub landing page collects email addresses of interested parties. Many of them will freely tell you how much they're willing to pay for the book when it's available. That's very valuable information.

Time to publish.

Publish your incomplete book on leanpub once you have 3 chapters written. Doesn't matter if 3 chapters is only 10% of your book. If you wait longer than that you are losing income and valuable feedback from early adopters. The great news is that at that point, you start earning 90% royalties even though the book is far from complete. I promise that the income and interest from your early buyers will keep you motivated to make progress.

Keep writing.

This middle phase will take between 3 and 6 months for most people, perhaps more for difficult topics. Keep publishing every time you finish a new chapter or have significant amounts of new content. Keep making money throughout this time.

Back to the negotiating table.

When you get to about 75-80% finished then you want to contact traditional publishers like Wiley or Pearson and/or whoever you initially spoke to about a book in the first place. Tell them that you're interested in their best offer. The best thing that can happen at this point is a bidding war over who will publish your title. (You shouldn't bother with Pragmatic Programmers. Want to know why I say that? Subscribe to the mailing list below.)

Cash Advances

If you're a first time author with a popular topic, traditional publishers should be willing to advance you at least $5k USD. That is money that you'll never have to return, which is why publishers seem stingy with advances. It's all risk to them. But since your book is almost done, you've gone a long way towards mitigating that risk for them. That should loosen their pursestrings somewhat.

Half of the advance will be payable on signing the book deal. The other half will be payable on delivery of final manuscript. Consider the promise of that money your incentive to make a final push and finish the book.

Royalties

Royalties for print should start at 18% of net revenues to the publisher. (Expect that figure to be around $10-20, so you're only making a few dollars on each sale. That's just the way it works, don't fret about it.) The real money comes later, when you open up new business opportunites. But that's another topic.

Royalties are not a single number. You'll also negotiate a multi-tiered ramp up to 25% royalties as your sale numbers grow and hit stretch targets. Selling 10 thousand copies of a print tech book these days is a solid success and should be compensated accordingly.

What about ebooks?

You're going to negotiate hard for a 25% royalty on ebooks, but be willing to accept a little less. Don't get greedy and blow it.

Profit!

Once you negotiate a deal you're happy with, sign it and finish your book. Submit the manuscript and get the balance of your advance. Plan on discontinuing Leanpub sales once the officially published version is available on Amazon and in stores.

You'll be cashing royalty checks every six months. Plus you get many other intangible benefits. Traditional publishers can open the door to other writing opportunities in periodicals, speaking engagements and even video appearances. There's also the credentializing effect! Every other schmuck these days has an ebook. But traditional print still provides a strong foundation for launching successful consulting careers.

Are you an aspiring author? Interested in reading more in-depth content about how to succeed like I have?

Subscribe to my mailing list

Email Address

Glad to announce that my latest book, The…

Obie Fernandez — Tue, 08 Apr 2014 22:41:55 +0000

Glad to announce that my latest book, The Lean Enterprise, is now widely available in print and Kindle editions on Amazon. Getting lots of great reviews already!

Of Late

John Nunemaker — Mon, 24 Feb 2014 19:00:00 +0000

A lot has changed over the years. I now do a lot more than just rails and having railstips as my domain seems to mentally put me in a corner.

As such, I have revived johnnunemaker.com. While I may still post a rails topic here once in a while, I’ll be posting a lot more varied topics over there.

In fact, I just published my first post of any length, titled Analytics at GitHub. Head on over and give it a read.

TechPeaks: Startup Life in the Italian Alps

Obie Fernandez — Tue, 07 Jan 2014 14:00:54 +0000

Last summer (2013) I was honored with an invitation to serve as a mentor for the inaugural group of entrepreneurs welcomed into the TechPeaks accelerator program. It opened my eyes to what's possible when a progressive local government achieves effective cultivation of innovation and startups. Quite an experiment the Italians pulled off last year, and the results have encouraged them to do it again this year, with notable evolutions to their program that I believe make it an even better opportunity for aspiring entrepreneurs.

Here's one of the coolest aspects of it, that really sets this program apart from traditional accelerators such as YC and TechStars: You don't even need to have a startup idea or be part of a team to qualify!

There are two ways to apply to the program: 1) with an idea, team optional or 2) as an individual technologist without an idea. If you get accepted without an idea, the program functions as a matchmaker, with the expectation that you will band together with other people in the program on their ideas, or come up with new ideas to pursue. Out of a total of 50 participant slots available, a maximum of 20 are reserved for these individual explorers.

If I didn't have a family and my own startup at the moment, you better believe I'd be knocking down their door to accept me as an explorer.

All participants are relocated to Trento, a college town with some of the most gorgeous vistas I've ever seen, nestled in between vineyards in the Italian alps. It's a relatively quick train trip from Milan and a fantastic place to live. Your accomodations and living expenses are covered by the program, which has a duration of four months, from March 25, 2014 until July 25, 2014. Teams that make progress are eligible for hundreds of thousands of euros in startup grants and additional follow-on funding if they continue operations in the Trento area.

The TechPeaks 2014 application deadline is January 20th. I encourage you to check it out.

Robert Williams, man to watch in 2014

Obie Fernandez — Wed, 01 Jan 2014 18:23:02 +0000

I recently interviewed Robert Williams, founder of Workshop. Over the course of the last year Robert went from full-time employment, to successful freelancing, to launching a successful online business helping other freelancers to succeed using his methods.

What is your background?

My background is in design. I went to an expensive art school - which I now regret, because 90% of the education I use on a daily basis is stuff I learned online (mostly for free).

I worked at a few design agencies after college, then in the marketing department for a large nursery - but those jobs left me unfulfilled and wanting to start my own business. I got that opportunity after I got fired from my job only a few months after receiving a raise, to boot.

I decided that instead of going to interviews I’d look for freelance work full-time. This was an epiphany for me, because I truly became grateful for having been let go from my 9-5 job. I spent my first week literally laying out on the beach everyday - and the freedom was awesome. That was the point where I knew I probably wouldn’t ever hold a 9-5 job again.

How did you manage your freelance work?

Starting my own freelance design shop has been one of the most educational experiences in my life. I realized I had to stand out from the average designer online. This simply wasn’t happening on sites like dribbble, twitter, and behance. On those sites you’re competing with thousands of the world’s best designers and you don’t get to highlight your specific skill.

I decided to delete my portfolio website, and focus on contacting leads 1-on-1 in order to a) track the amount of people who were replying to me and asking to see my portfolio and b) have more control over how I was presenting myself and my services.

This had pretty great results, I was able to make $30k in my first six months freelancing (without a portfolio website). This fascinated me and I began to study and collect evergreen principles to freelancing. The fact was that most places online tell you to post on twitter and dribbble and ‘get your name out there,’ but for someone just starting out this isn’t the right move.

Okay, so what is the right move for a new freelancer?

I created an optimized system for finding the best leads online quickly, and made templates for stuff like emailing prospects, onboarding new clients, getting world-class feedback and more. Optimizing my freelance business has been one of my biggest passions in the past year.

Recently, I started a new project focused on helping other freelancers do the same; Workshop. I send the web’s best leads to a private list of freelancers every day. I also send all of my secrets and techniques to building my freelance business to members, which has been an awesome experience.

How did you decide to get into this particular business?

I got the idea from desperately wanting this service for my own business. As freelancers, we charge anywhere from $50-$1k per hour, and if you factor in the time is takes to look for work - you’re spending a fortune on finding work. I wanted a service that automated this process for me, but that was high quality. A few other people were doing like a auto-generated lists of leads, but the quality was really low, and not worth the money. It was something that I needed to do for my business anyway so I really liked the idea of charging for this service.

Then, about 5 weeks ago, I was listening to a podcast interview with Ruben Gamez from bidsketch. He said one thing in particular that really resonated with, ‘a lot of people are listening to podcasts nowadays but not many are actually doing something actionable as a result.’

I decided then and there to pursue this idea - and try to grind it out. I put together a quick landing page, using techniques from online courses I’ve taken in the past year including, earn1k, recurring revenue for consultants, 30x500, email marketing for startups, and a few others.

I was also a member of JFDI, a private community of bootstrappers, who really helped me talk through my landing page and iterate quickly by providing me an extra set of eyes (or ten). This community support was integral in keeping my confidence up when people started to question my authority and service. I literally had one person sign up on the first day. I spent like 5 hours finding the 5 most awesome leads I could, and sent them over to him. Luckily over the next few days I got some traction and things began to take off.

What differentiates you from potential competitors?

What I’m doing differently, is 100% hand-curating every single lead I send. I aggregate thousands of job leads everyday and painstakingly comb through them by hand. I have strict, and vigorous criteria for finding the best leads.

The projects are all stuff I would apply to. Nothing that would have a budget of less than $1000 and I make sure every lead is remote, and open to freelancers everywhere. Also, I try to make sure and do a little extra legwork to include things like the company website and contact email if possible.

The result is a list of 5-10 leads that have all the relevant info for a busy freelancer, and none of the fluff. 90% of the work is done for you. All you have to do is reply to these 5-10 leads and you will have a constant stream of people entering your prospect cycle.

Members no longer have to worry about job boards which were really designed for full-time work anyway, and instead reply to everything from their inbox. I also offer to monitor any job boards members want me to, so you can literally use me as a virtual assistant.

The difference between me and a virtual assistant you would find on places like odesk, is I’m an experienced freelancer myself. No one is going to be able to decipher the great leads from the riff-raff as good as you do, unless they’re a high-paid consultant themselves, which I am.

What have been your biggest mental obstacles?

Subconsciously it was wanting to see success immediately. I heard people like 37signals and Amy Hoy talk about growing slowly, and totally agreed, but it wasn’t until I actually started this project that I realized I’ve been way too impatient in the past.

90% of growing a great business is just grinding it out. Sure, you need luck, timing, and help from others, but if you’re not there everyday it doesn’t matter. You can look at this as a negative or a positive. It’s up to you. You can be impatient and give up on all of your ideas… or you can decide to do something regardless of how long it takes. But make sure you charge from the beginning, otherwise you won’t know how you’re doing.

To me, it’s one of the greatest strengths, because I don’t have control over whether someone with a million followers tweets out a link to my website, but I do have control over whether I put in effort each and every day. I’m okay with growing slowly because I’m building, learning, and I want to be in it for the long-term. I also want to create the absolute best service for freelancers and help them build an awesome business for themselves.

Thanks Robert!

Are you a freelancer? Check out Workshop for yourself.

Haml's little-known list_of helper

Obie Fernandez — Wed, 20 Nov 2013 11:59:00 +0000

TIL that given an Enumerable object and a block, the list_of method will iterate and yield the results of the block into sequential <li> elements.

Haml

%ul
= list_of [1, 2, 3] do |item|
Number #{item}

HTML

<ul>
<li>Number 1</li>
<li>Number 2</li>
<li>Number 3</li>
</ul>

The list_of method also optionally takes a hash of options applied to the output li tags as attributes.

Interested in more information like this? You can download my book The Rails 4 Way at Leanpub today. Final print edition on Addison Wesley coming early next year.

Installing Powerline on OS X + homebrew

Sun, 27 Oct 2013 08:29:12 +0000

I’ve always wanted to get that fancy Powerline status bar and prompt. Chevrons and git branch icons in the terminal just sounds so… defying:

I’d failed to get it working properly before because of a combination of outdated Powerline docs and confusion between OS X’s system python/vim and homebrew’s python/vim. Some of my former colleagues detailed the pains and hours lost getting Powerline setup, so I gave up.

Anyway, I got a new Mac for work recently and decided to try again, and it was easier than I thought. You (including you, future Chu Yeow) can use this guide if you’re using homebrew to install vim and python (and everything else like zsh and tmux).

Installing Powerline on OS X

Install python with homebrew: brew install python
Install vim with homebrew: brew install vim --env-std --override-system-vim. You must install vim after python so that it’ll compile with homebrew’s python.
Install powerline with pip: pip install https://github.com/Lokaltog/powerline/tarball/develop. Powerline should get installed to /usr/local/lib/python2.7/site-packages/powerline.
open /usr/local/lib/python2.7/site-packages/powerline – you should see the files required for integration with zsh, vim, tmux, etc. in the bindings directory. All you need to do now is to include these bindings in your zsh, vim and tmux config files – it’s that simple!

Add Powerline to vim by adding these lines to ~/.vimrc:

source /usr/local/lib/python2.7/site-packages/powerline/bindings/vim/plugin/powerline.vim
set laststatus=2

Add Powerline to tmux by adding this to ~/.tmux.conf:

source /usr/local/lib/python2.7/site-packages/powerline/bindings/tmux/powerline.conf

Add Powerline to zsh by adding this to ~/.zshrc:
```
source /usr/local/lib/python2.7/site-packages/powerline/bindings/zsh/powerline.zsh
```
If you use oh-my-zsh, be sure to source the Powerline bindings after oh-my-zsh’s.
Grab a patched font from https://github.com/Lokaltog/powerline-fonts (I use Inconsolata), and install it (just download, double-click the font file, and Install). If you don’t use any of these fonts, you’ll have to patch your favorite font yourself (have fun with that!)

That’s all you need! Go crazy and start tweaking your Powerline colorscheme.

This Month in Ruby: PeepCode Acquired, Rails 3.2.14, And More

Peter Cooper — Thu, 25 Jul 2013 16:25:44 +0000

Welcome to a roundup of Ruby news, articles, videos, and more, for July 2013 cobbled together from my e-mail newsletter, Ruby Weekly.

Highlights include: PeepCode acquired by Pluralsight, Practicing Ruby archives made public, Rails 3.2.14, and an interesting interview with Matz.

Featured

The First Four Volumes of Practicing Ruby, Now Available Online
Practicing Ruby is a high quality, paid Ruby journal run by Gregory Brown, but he's made archives of over 60 articles available to the public. There's a ton of stuff to enjoy here.

PeepCode Acquired by Pluralsight
Ruby and Rails screencasting pioneer Geoffrey Grosenbach has announced he has sold Peepcode to Pluralsight, a large online training provider.

The Plan for RSpec 3
RSpec 2.0 was released in October 2010 and RSpec 2.14 will be the last RSpec 2 feature release. Work on RSpec 3 has begun and Myron Marston shares what's going to be involved.

Rails 3.2.14 RC1 and RC2 Released
A variety of bug fixes for Rails 3.2 arrived in 3.2.14 RC1 with one minor regression fixed in RC2. Final due soon.

Rails 3.2.14 Released

The Future of Computing - An Interview with Matz
Last year, Ruby's creator Yukihiro 'Matz' Matsumoto released a book called The Future of Computing (only in Japanese, I believe) and did an interview with a Chinese publisher. Fred Wu has translated it into English.

RSpec 2.14 Released
Myron Marston unveils the last 2.x feature release of the popular spec framework and announces work is well underway for the future RSpec 3. 2.14 includes a new feature called 'spies' which is shown off here.

Functional Programming and Ruby
At GoRuCo 2013, Pat Shaughnessy gave a 40 minute talk comparing Haskell (a functional language) to Ruby and looked at how to implement common functional patterns in Ruby. Well explained and backed by good slides.

Reading

Streaming with Rails 4
Saurabh Bhatia looks at Rails 4's support for live streaming (the ability to send partial requests out to the client on the fly).

Reading the Ruby Source to Understand Rails Idiosyncrasies
I'm not sure you always need to dig quite so deep but Eno Compton takes an interesting journey through MRI's source code to see the difference between Range#cover? and Range#include?

Speed Up Heroku Deploys
Alex MacCaw was suffering from slow deploys to Heroku but he found a workaround.

Managing Ruby Environments on OS X: Getting Started with rbenv

Shoes 4 – A Progress Report
Shoes was a graphical toolkit put together by Why the Lucky Stiff that made it simple to create GUI apps in Ruby. Since Why disappeared, others have picked up work on it, and Shoes 4 is set to be a complete rewrite.

Put Yourself on Rails with A Push of A Button
A technique for quickly bringing up a workspace for doing Rails work (including terminals, a Rails console, a Rails server, etc.)

Automatically Scaling Heroku Workers

Multitenancy with Rails: An E-book by Ryan Bigg
Ryan Bigg, of Rails in Action fame, is writing an e-book about building a multi-tenanted Rails app.

Incremental Redesign with Rails
Lars Klevan shows how to use prepend_view_path to make in-progress redesigns on a production codebase simpler.

How to Declutter Your 'lib' Directory
If you have an established Rails project, its 'lib' folder might be getting a little full. Kuba Suder looks at ways to clean it up and put things elsewhere.

Design Patterns: The Template Method Pattern
An introductory Ruby-oriented look at arguably the simplest design pattern.

Object Oriented Rails: Writing Better Controllers
Damien Le Berrigaud of Pivotal Labs tries to avoid stubs and mocks and leans on dependency injection to test his controllers' code.

Vimscript And You
HashRocket's Jonathan Jackson demonstrates how you can use RSpec against Vim to aid in the development of a Vim plugin with Vimscript.

MotionPhrase: Next Level Localization for RubyMotion Applications
PhraseApp is a translation management tool for producing multilingual Web sites, Rails apps, etc, but it also works for localizing RubyMotion apps too, as demonstrated here.

Ruby's Eigenclasses Demystified
Andrea Singh looks at Ruby's quirky 'eigenclasses' (a.k.a. metaclasses) and explains things in both code and diagrams. Dates from 2011 but worth revisiting.

The Self-Pipe Trick Explained
Jesse Storimer shows off a cute Unix trick/technique in Ruby.

Practical RSpec Wrapping
Why would you want to use around hooks in RSpec? Dru Riley explains.

Authentication with Warden, Devise-Less

Can I Use ST3 Yet? Migrating to Sublime Text 3

Using PostgreSQL's 'hstore' in A Rails Application on Engine Yard Cloud
If you want to take advantage of schemaless features without abandoning your relational database, using 'hstore' within Postgres is a great option. Here's an introduction on using the hstore PostgreSQL extension in a Rails app.

Implementing Subdomain and Custom Domain Support in Rails
A look at how one development team implement subdomain and custom domain features in their Rails app.

An Introduction to Vagrant for Rails Developers

On Distributing Command Line Applications: Why I Switched from Ruby to Go

Events

dotRB: The Largest Ruby Conference in France (October 18, Paris)
Following on from a successful 'dotJS' JavaScript event comes dotRB. Announced speakers so far include Steve Klabnik, Konstantin Haase, and Brian Ford.

Nickel City Ruby Conference - September 20-21, 2013 - Buffalo, NY

Watching

11 Talks from La Conf Paris
Some big names to enjoy here including Yehuda Katz, Amy Hoy, Sandi Metz, and Steve Klabnik.

Deathmatch: Bundler vs Rubygems.org
At GoRuCo 2013, Andre Arko told the story of the quest to make 'bundle install' faster.

How to Set Up RSpec
A well produced 6 minute screencast.

To Know A Garbage Collector
Mike Bernstein discusses his experiments with MRI Ruby's garbage collector, his investigations into other languages and the influence of their GC implementations, the history of the subject, and more.

The Ruby Rogues Discuss 'Elixir' with José Valim

Kata and Analysis with Jim Weirich
From RubyConf India 2013 comes a live coding session by the inimitable Jim Weirich where he walks through the popular 'roman numeral' conversion kata using TDD along the way.

Aaron 'tenderlove' Patterson's RubyConf India 2013 Keynote
An hour with Ruby and Rails core contributor Aaron 'tenderlove' Patterson covering esoteric Ruby stuff and Postgres to career advice and cats. Warning: The audio is rather poor here and cuts out entirely for the second half so don't waste your time if this will drive you crazy.

5 Minutes of EuRuKo 2013
European Ruby conference (EuRuKo) took place in Athens last month and Clemens Helm has put together a 5 minute collection of clips and insights from the event. Includes Matz, Xavier Noria, Benjamin Smith, Pat Shaughnessy and Steve Klabnik.

Nokogiri: History and Future
Nokogiri is the most popular way to parse and process XML in Ruby and at GoRuCo 2013, Mike Dalessio gave a short 11 minute talk on the origins of the project, how to determine if it suits you, and looks at some of the tooling around it.

The Ruby Rogues Discuss 'DevOps'

Daniel Huckstep on Ruby's Standard Library and Go (the language)

Libraries, Code and Tools

Upton: A Web Scraping Framework
A Ruby-based web-scraping framework that abstracts away the common parts of web scraping so developers can concentrate on the unique parts of their project.

LanguageFilter: Detect and Optionally Filter Multiple Categories of Language
Wave goodbye to sex, hatred, profanity, violence, etc, in your app.

Lita: A Ruby Chat Bot with Redis-based Storage
Can be twisted to work with any chat service and extended with plugins.

Funkify: Haskell-style Partial Application and Composition for Ruby Methods

Lexeme: A Simple Lexical Analyzer Written in Ruby

Missingly: A DSL for Handling method_missing Hooks

Q: A Lightweight Promises Implementation in Ruby

Pkgr: Make A Debian Package Out of A Rails App in 5 Minutes
A high-level tool that turns Rails applications into native Debian packages.

Flynn - Open Source Platform As A Service, Powered by Docker
Flynn is an as-yet-unreleased Heroku-inspired system to simplify deploying and maintaining apps. Instead of using complex config management systems, Flynn allows self-serve management of containerized deployments. The creator is currently trying to raise money to work on the project.

Rumm: A Command Line Interface and API to Rackspace Cloud

New in RubyMotion: Blocks Rewrite, Retain Cycle Detection, Better Crash Reporting

Jobs

Software Craftsperson at Bendyworks
If you're the type of person who learns new languages as a matter of course, contributes to open source for fun, and ships code with a calm and collected professionalism: you seem like our kind of developer. Join our world-class team in Madison, Wisconsin.

Senior backend- / API-developer at Rabble (Stockholm, Sweden)
Tired of bullshit ads? Help us develop Sweden's leading app for mobile offers, where customers and businesses meet on equal terms! Join us in the heart of Stockholm to play with geospatial data and Ruby API's all day long!

Ruby on Rails developer at SupaDupa (London, UK)
We're looking for an experienced Ruby on Rails developer to join the small team behind SupaDupa.me, an e-commerce platform aimed at creatives. Excited about the challenge of working on the full stack, from front-end dev to system administration? Get in touch!

Ruby Programmer: IT and System Automation
Want to change the future of education? We are trying to build an awesome team that enjoys challenges and results. Interested? Come work with us in beautiful Switzerland.

RoR/Agile Developer at Simon & Schuster [New York, New York]

Ruby on Rails Developer at WegoWise [Cambridge, Massachusetts]

Ruby Developers at HouseTrip (London, UK)
Want to work with a 18-person team of passionate Ruby developers who love good code and care for their product in central London? We are currently hiring. Ranked by Wired Magazine the number two start-up in London (2012), HouseTrip is Europe’s largest holiday rental booking website!

Get your job listing in a future issue of Ruby Weekly

Last but not least..

ruby -run -e httpd . -p5000
Run a local HTTP server with a single line of Ruby. Just one character longer than the classic python -m SimpleHTTPServer but more obviously flexible (plus, it's Ruby ;-)).

Does the GIL Make Your Ruby Code Thread-Safe?

Jesse Storimer — Wed, 19 Jun 2013 14:19:34 +0000

This is a guest post by Jesse Storimer. He teaches the Unix fu workshop, an online class for Ruby developers looking to do some awesome systems hacking in Ruby and boost their confidence when it comes to their server stack. Spots are limited, so check it out the class while there's still room. He's also the esteemed author of Working with Unix Processes, Working with TCP Sockets and Working with Ruby Threads.

There are some misconceptions in the Ruby community about this question surrounding MRI's GIL. If you only take one thing away from this article today, let it be this: The GIL does not make your Ruby code thread-safe.

But you shouldn't take my word for it.

This series started off just trying to understand what the GIL is at a technical level. Part 1 explained how race conditions could occur in the C code that's used to implement MRI. Yet, the GIL seemed to eliminate that risk, at least for the Array#<< method we looked at.

Part 2 confirmed that the GIL did, in fact, make MRI's native C method implementations atomic. In other words, these native implementations were free from race conditions. These guarantees only applied to MRI's native C functions, not to the Ruby that your write. So we were left with a lingering question:

Does the GIL provide any guarantee that your Ruby code will be thread-safe?

I've already answered that question. Now I want to make sure that the misconception doesn't go any further.

Race conditions redux

Race conditions exist when some piece of data is shared between multiple threads, and those threads attempt to act on that data at the same time. When this happens without some kind of synchronization, like locking, your program can start to do unexpected things and data can be lost.

Let's step back and recap how such a race condition can occur. We'll use the following Ruby code example for this section:

class Sheep
  def initialize
    @shorn = false
  end

  def shorn?
    @shorn
  end

  def shear!
    puts "shearing..."
    @shorn = true
  end
end

This class definition should be nothing new. A Sheep is not shorn when initialized. The shear! method performs the shearing and marks the sheep as shorn.

sheep = Sheep.new

5.times.map do
  Thread.new do
    unless sheep.shorn?
      sheep.shear!
    end
  end
end.each(&:join)

The bit of code creates a new sheep and spawns 5 threads. Each thread races to check if the sheep has been shorn? If not, it invokes the shear! method.

Here's the result I see from running this on MRI 2.0 several times.

$ ruby check_then_set.rb
shearing...
$ ruby check_then_set.rb
shearing...
shearing...
$ ruby check_then_set.rb
shearing...
shearing...

Sometimes the same sheep is being shorn twice!

If you were under the impression that the GIL made your code 'just work' in the presence of multiple threads, this should dispel that. The GIL can make no such guarantee. Notice that the first time running the file, the expected result was produced. In subsequent runs, unexpected output was produced. If you continued running the example, you'll see still different variations.

These unexpected results are due to a race condition in your Ruby code. It's actually a common enough race condition that there's a name to describe this pattern: a check-then-set race condition. In a check-then-set race condition, two or more threads check a value, then set some state based on that value. With nothing to provide atomicity, it's possible that two threads race past the 'check' phase, then both perform the 'set' phase.

Recognizing race conditions

Before we look at how to fix this, first I want you to understand how to recognize this. I owe @brixen for introducing to me the terminology of interleavings in the context of concurrency. It's really helpful.

Remember that a context switch can occur on any line of your code. When switching from one thread to another, imagine your program being chopped up into a set of discrete blocks. This sequential set of blocks is a set of interleavings.

At one end of the spectrum, it's possible that there's a context switch after every line of code! This set of interleavings would have each line of code interleaved. At the other end of the spectrum, it's possible that there are no context switches during the body of the thread. This set of interleavings would have all the code in its original order for each thread. In between these two ends, there are lots of ways that your program can be chopped up and interleaved.

Some of these interleavings are OK. Not every line of code introduces a race condition. But imagining your programs as a set of possible interleavings can help you recognize when race conditions do occur. I'll use a series of diagrams to show this code may be interleaved by two Ruby threads.

Just to make the diagrams simpler, I replaced the shear! method call with the code from the body of the method.

Consider this diagram the legend for the ones to follow; the code highlighted in red is the set of interleavings from Thread A, the code highlighted in blue is the set of interleavings from Thread B.

Now let's see how this code could be interleaved by simulating context switches. The simplest case is if neither thread is interrupted during the course of executing this code. This would result in no race conditions and would produce the expected output for us. That might look like this:

Now I've organized the diagram so you can see the sequential ordering of events. Remember that the GIL locks around the execution of Ruby code, so two threads can't truly run in parallel. The ordering of events here is sequential, starting at the top and working down.

In this interleaving, Thread A did all of its work, then the thread scheduler triggered a context switch to Thread B. Since Thread A had already done the shearing and updated the shorn variable, Thread B didn't have anything to do.

But it isn't always this simple. Remember that the thread scheduler could trigger a context switch at any point in this block of code. This time we just got lucky.

Let's look at a more nefarious example, one that would produce unexpected output for us.

In this possible interleaving, the context switch occurs right at a point that can cause trouble. Thread A checks the condition and starts shearing. Then the thread scheduler schedules a context switch and Thread B takes over. Even though Thread A already performed the shearing, it didn't get a chance to update the shorn attribute yet, so Thread B doesn't know about it.

Thread B checks the condition for itself, finds it to be false, and shears the sheep again. Once it finishes, Thread A is scheduled again and finishes execution. Even though Thread B set shorn = true when it ran through the code, Thread A does it again because it picks up exactly where it left off.

A sheep getting shorn twice may not seem like much to care about, but replace sheep with invoice, and shearing with collecting payment; we would have some unhappy customers!

I'll share one more example to illustrate the non-deterministic nature of things here.

This just adds more context switches, so each thread progresses a little bit at a time, but keeps switching back and forth. Let your mind take this to its logical conclusion, it's possible for a context switch to happen on any line of the program. The interleaving that occurs can also be different each time the code is executed, so it may produce the expected result on one iteration, and an unexpected result the next time around.

This is really a great way to think about race conditions. When you're writing multi-threaded code, you want to be thinking about how the program might be chopped up and interleaved, and the effects of various interleavings. If it seems that some interleavings will lead to incorrect results, you should re-think your approach to the problem or introduce synchronization with Mutex.

This is terrible!

At this point it seems fitting to tell you that you can make this code example thread-safe by introducing synchronization with Mutex. It's true, you can do that. But I intentionally cooked up this example to prove a point; it's terrible code. You shouldn't write code like this in a multi-threaded environment.

Whenever you have multiple threads sharing a reference to an object, and making modifications to it, you're going to run into trouble unless you have some kind of locking in place to prevent a context switch from happening in the middle of the modification.

However, this particular race condition is easily solvable without explicitly using locks in your code. Here's one solution using Queue:

require 'thread'

class Sheep
  # ...
end

sheep = Sheep.new
sheep_queue = Queue.new
sheep_queue << sheep

5.times.map do
  Thread.new do
    begin
      sheep = sheep_queue.pop(true)

      sheep.shear!
    rescue ThreadError
      # raised by Queue#pop in the threads
      # that don't pop the sheep
    end
  end
end.each(&:join)

I left out the Sheep implementation because it's the same. Now, instead of each thread sharing the sheep object and racing to shear it, the Queue provides the synchronization.

If you run this against MRI, or any of the other truly parallel Ruby implementations, it will produce the expected result every time. We've eliminated the race condition in this code. Even though all the threads may call the Queue#pop method at more-or-less the same time, it uses a Mutex internally to ensure that only one thread can receive the sheep.

Once this single thread receives the sheep, the race condition disappears. With just one thread, there's no one else to race with!

The reason I suggest using Queue instead of a lock is that its simpler to use a Queue correctly. Locks are notoriously easy to get wrong. They bring new concerns like deadlocking and performance degradations when used incorrectly. Using a data structure is like depending on an abstraction. It wraps up the tricky stuff in a more restrictive, but simpler API.

Lazy initialization

I'll just quickly point out that lazy initialization is another form of the the check-then-set race condition. The ||= operator effectively expands to

@logger ||= Logger.new

# expands to 

if @logger == nil
  @logger = Logger.new
end

@logger

Look at the expanded version and imagine where the interleavings could occur. With multiple threads and no synchronization, it's definitely possible for that @logger to be initialized twice. Again, initializing a Logger twice may not be a problem in this case, but I have seen bugs like this in the wild that do cause issues.

Reflections

I want to leave you with some lessons at the end of all this.

4 out of 5 dentists agree that multi-threaded programming is hard to get right.

At the end of the day, all that the GIL guarantees is that MRI's native C implementations of Ruby methods will be executed atomically (but even this has caveats). This behaviour can sometimes help us as Ruby developers, but the GIL is really there for the protection of MRI internals, not as a dependable API for Ruby developers.

So the GIL doesn't 'solve' thread-safety issues. As I said, getting multi-threaded programming right is hard, but we solve hard problems every day. One way that we work with hard problems is with good abstractions.

For example, when I need to do an HTTP request in my code, I need to use a socket. But I don't usually use a socket directly, that would be cumbersome and error-prone. Instead, I use an abstraction. An HTTP client provides a more restrictive, simpler API that hides the socket interactions and associated edge cases from me.

If multi-threaded programming is hard to get right, maybe you shouldn't be doing it directly.

"If you add a thread to your application, you've probably added five new bugs in doing so."

- Mike Perham

We're seeing more and more abstractions around threads. An approach that's catching on in the Ruby community is the Actor model of concurrency, with the most popular implementation being Celluloid. Celluloid provides a great abstraction that marries concurrency primitives to Ruby's object model. Celluloid can't guarantee that your code will be thread-safe or free from race conditions, but it wraps up best practices. I encourage you give Celluloid a try.

These problems that we're talking about aren't specific to Ruby or MRI. This is the reality of programming in a multi-core world. The number of cores on our devices is only going up, and MRI is still figuring out its answer to this situation. Despite its guarantees, the GIL's restriction on parallel execution seems like the wrong direction. This is part of MRI's growing pains. Other implementations, like JRuby and Rubinius, are running truly parallel with no GIL.

We're seeing lots of new languages that have concurrency abstractions built-in at the core of the language. The Ruby language doesn't have any of this, at least not yet. Another benefit of relying on abstraction is that the abstractions can improve their implementation, while your code remains unchanged. For example, if the implementation of Queue switched from relying on locks to using lock-free synchronization, your code would reap the benefits without any modification.

For the time being, Ruby developers should educate themselves about these issues! Learn about concurrency. Be aware of race conditions. Thinking about code as interleavings can help you reason about race conditions.

I'll leave off with a quote that's influencing much of the work being done in the larger field of concurrency today:

Don't communicate by sharing state; share state by communicating.

Using data structures for synchronization supports this; the Actor model supports this. This idea is at the core of the concurrency model of languages like Go, Erlang, and others.

Ruby needs to look to what's working in other languages and embrace it. As a Ruby developer, you can do this today by trying out and supporting some of these alternative approaches. With more people on board, these approaches could become the new standard for Ruby.

Thanks to Brian Shirai for reviewing a draft of this.

Let Nunes Do It

John Nunemaker — Thu, 18 Apr 2013 19:20:00 +0000

In a moment of either genius or delirium I decided to name my newest project after myself. Why? Well, here is the story whether you want to know or not.

Why Nunes?

Naming is always the hardest part of a project. Originally, it was named Railsd. The idea of the gem is automatically subscribe to all of the valuable Rails instrumentation events and send them to statsd in a sane way, thus Railsd was born.

After working on it a bit, I realized that the project was just an easy way to send Rails instrumentation events to any service that supports counters and timers. With a few tweaks, I made Railsd support InstrumentalApp, a favorite service of mine, in addition to Statsd.

Thus came the dilemma. No longer did the (already terrible) name Railsd make sense. As I sat and thought about what to name it, I remembered joking one time about naming a project after myself, so that every time anyone used it they had no choice but to think about me. Thus Nunes was born.

Lest you think that I just wanted to name it Nunes only so that you think of me, here is a bit more detail. Personally, I attempt to instrument everything I can. Be it code, the steps I take, or the calories I consume, I want to know what is going on. I have also noticed that which is automatically instrumented is the easiest to instrument.

I love tracking data so deeply that I want to instrument your code. Really, I do. I want to clone your repo, inject a whole bunch of instrumentation and deploy it to production, so you can know exactly what is going on. I want to sit over your shoulder and look at the graphs with you. Ooooooh, aren’t those some pretty graphs!

But I don’t work for you, or with you, so that would be weird.

Instead, I give you Nunes. I give you Nunes as a reminder that I want to instrument everything and you should too. I give you Nunes so that instrumenting is so easy that you will feel foolish not using it, at least a start. Go ahead, the first metric is free! Yep, I want you to have that first hit and get addicted, like me.

Using Nunes

I love instrumenting things. Nunes loves instrumenting things. To get started, just add Nunes to your gemfile:

# be sure to think of me when you do :)
gem "nunes"

Once you have nunes in your bundle (be sure to think of bundling me up with a big hug), you just need to tell nunes to subscribe to all the fancy events and provide him with somewhere to send all the glorious metrics:

# yep, think of me here too
require 'nunes'

# for statsd
statsd = Statsd.new(...)
Nunes.subscribe(statsd) # ooh, ooh, think of me!

# for instrumental
I = Instrument::Agent.new(...)
Nunes.subscribe(I) # one moooore tiiiime!

With just those couple of lines, you get a whole lot of goodness. Out of the box, Nunes will subscribe to the following Rails instrumentation events:

process_action.action_controller
render_template.action_view
render_partial.action_view
deliver.action_mailer
receive.action_mailer
sql.active_record
cache_read.active_support
cache_generate.active_support
cache_fetch_hit.active_support
cache_write.active_support
cache_delete.active_support
cache_exist?.active_support

Thanks to all the wonderful information those events provide, you will instantly get some of these counter metrics:

action_controller.status.200
action_controller.format.html
action_controller.exception.RuntimeError – where RuntimeError is the class of any exceptions that occur while processing a controller’s action.
active_support.cache_hit
active_support.cache_miss

And these timer metrics:

action_controller.runtime
action_controller.view_runtime
action_controller.db_runtime
action_controller.posts.index.runtime – where posts is the controller and index is the action
action_view.app.views.posts.index.html.erb – where app.views.posts.index.html.erb is the path of the view file
action_view.app.views.posts._post.html.erb – I can even do partials! woot woot!
action_mailer.deliver.post_mailer – where post_mailer is the name of the mailer
action_mailer.receive.post_mailer – where post_mailer is the name of the mailer
active_record.sql
active_record.sql.select – also supported are insert, update, delete, transaction_begin and transaction_commit
active_support.cache_read
active_support.cache_generate
active_support.cache_fetch
active_support.cache_fetch_hit
active_support.cache_write
active_support.cache_delete
active_support.cache_exist

But Wait, There is More!

In addition to doing all that work for you out of the box, Nunes will also help you wrap your own code with instrumentation. I know, I know, sounds too good to be true.


class User < ActiveRecord::Base
  extend Nunes::Instrumentable # OH HAI IT IS ME, NUNES

  # wrap save and instrument the timing of it
  instrument_method_time :save
end

This will instrument the timing of the User instance method save. What that means is when you do this:

# the nerve of me to name a user nunes
user = User.new(name: "NUNES!")
user.save

An event named instrument_method_time.nunes will be generated, which in turn is subscribed to and sent to whatever you used to send instrumentation to (statsd, instrumental, etc.). The metric name will default to “class.method”. For the example above, the metric name would be user.save. No fear, you can customize this.

class User < ActiveRecord::Base
  extend Nunes::Instrumentable # never

  # wrap save and instrument the timing of it
  instrument_method_time :save, 'crazy_town.save'
end

Passing a string as the second argument sets the name of the metric. You can also customize the name using a Hash as the second argument.

class User < ActiveRecord::Base
  extend Nunes::Instrumentable # gonna

  # wrap save and instrument the timing of it
  instrument_method_time :save, name: 'crazy_town.save'
end

In addition to name, you can also pass a payload that will get sent along with the generated event.


class User < ActiveRecord::Base
  extend Nunes::Instrumentable # give nunes up

  # wrap save and instrument the timing of it
  instrument_method_time :save, payload: {pay: "loading"}
end

If you subscribe to the event on your own, say to log some things, you’ll get a key named :pay with a value of "loading" in the event’s payload. Pretty neat, eh?

Conclusion

I hope you find Nunes useful and that each time you use it, you think of me and how much I want to instrument your code for you, but am not able to. Go forth and instrument!

P.S. If you have ideas for Nunes, create an issue and start some chatter. Let’s make Nunes even better!

Using duplicity for full server backup on Ubuntu 12.04

Sat, 13 Apr 2013 12:23:00 +0000

duplicity is a great full featured backup tool, providing "encrypted bandwidth-efficient backups using the rsync algorithm." I’m choosing it for my backup needs primarily for the follow reasons:

Supports a fleet of backup server types, and even a chroot’d SFTP server will work
Works with Linux, FreeBSD, and OpenBSD, out of the box
Does not require root access on the backup server

To perform full server backups using duplicity on your Ubuntu 12.04 system, simply perform the following.

Install duplicity:
```
sudo apt-get install duplicity
```
Perform backup (using rsync method):
```
$ sudo -H duplicity --no-encryption \
  --exclude /mnt --exclude /tmp --exclude /proc --exclude /sys \
  / rsync://<user>@<host>//<backup-directory>
```
You’ll need to replace <user>, <host> and <backup-directory> with your backup username, host, and directory, respectively. See the "Full Example" below for more info.

Output should look something like:
```
Import of duplicity.backends.sshbackend Failed: No module named paramiko
Import of duplicity.backends.giobackend Failed: No module named gio
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
```
If you’d like your backups to be encrypted, simply remove the –no-encryption option and provide a passphrase when prompted. You’ll need this same passphrase to restore from a backup. It is a symmetric key. Keep the key very safe; if you lose it, your backups will be useless.

Schedule regular backups

The next step is to set this up in cron, so your backups are done regularly. The following entries in /etc/crontab will perform a full backup once a week, Sunday at midnight, then incremental backups the other 6 days of the week (also at midnight). Additionally, we want to clean up backups that are too old, so we add an additional entry to keep only the latest two full backups (plus incrementals).

0 0 * * sun     root    duplicity full --no-encryption --exclude /mnt --exclude /tmp --exclude /proc --exclude /sys / rsync://<user>@<host>//<backup-directory>
0 0 * * 1-6     root    duplicity --no-encryption --exclude /mnt --exclude /tmp --exclude /proc --exclude /sys / rsync://<user>@<host>//<backup-directory>
0 8 * * sun     root    duplicity remove-all-but-n-full 2 --force rsync://<user>@<host>//<backup-directory>

Full Example for an ARP Networks VPS

At ARP Networks, we provide raw backup space to customers for 10 cents per GB. The example below will illustrate how to do a full backup of your Ubuntu or Debian VPS. Assuming your backup username is johndoe, simply perform the following steps.

Add root SSH public key to backup server

Create an SSH key for root as follows:
```
sudo -H ssh-keygen -t rsa -b 2048
```
Accept the defaults and choose a good passphrase.

See the Portal, click on the Backup Space service, and then click the link to submit an SSH key.
Install duplicity:
```
sudo apt-get install duplicity
```
Perform backup (using rsync method):
```
$ sudo -H duplicity --no-encryption \
  --exclude /mnt --exclude /tmp --exclude /proc --exclude /sys \
  / rsync://johndoe@backup01.cust.arpnetworks.com//home/johndoe
```
backup01.cust.arpnetworks.com is one of our backup servers. If the Portal lists a different server for you, use that one.

Additionally, as mentioned above, if you want your backups to be encrypted, remove the –no-encryption option.

Schedule regular backups

Put the following in your /etc/crontab

0 0 * * sun     root    duplicity full --no-encryption --exclude /mnt --exclude /tmp --exclude /proc --exclude /sys / rsync://johndoe@backup01.cust.arpnetworks.com//home/johndoe
0 0 * * 1-6     root    duplicity --no-encryption --exclude /mnt --exclude /tmp --exclude /proc --exclude /sys / rsync://johndoe@backup01.cust.arpnetworks.com//home/johndoe
0 8 * * sun     root    duplicity remove-all-but-n-full 2 --force rsync://johndoe@backup01.cust.arpnetworks.com//home/johndoe

Note: Please adjust the backup time to your own preference, otherwise many backups may fire off at the exact same time, slowing down the host.

…and that’s all there is to it!

This Week in Ruby: Matz on Ruby 2.0, Numerous Conference CFPs, Tenderlove on define_method

Peter Cooper — Thu, 07 Mar 2013 12:44:43 +0000

Welcome to this week’s roundup of Ruby news, articles, videos, and more, cobbled together from my e-mail newsletter, Ruby Weekly. Sorry these roundups have been missing for a couple of months, I've been focusing very heavily on the e-mail newsletters which are continuing to grow like crazy! :-) I hope to get back into blogging more soon.

Matz on Ruby 2.0
Matz spoke about Ruby 2.0 ('the happiest release ever') for 30 minutes at the Heroku Waza event a week ago and the video is already available to watch. He stresses that "Ruby 1.8 will die soon" and encourages everyone to upgrade.

Dynamic Method Definitions
Aaron 'tenderlove' Patterson says that "depending on your app, using define_method is faster on boot, consumes less memory, and probably doesn’t signigicantly impact performance" compared to eval-based techniques. (And he has the numbers to prove it.)

Steel City Ruby Conference 2013 CFP Now Open
Steel City Ruby takes places in Pittsburgh, PA on August 16-17 and the CFP is now open if you want to submit a talk. The Burlington Ruby Conference has a CFP open too, as does RubyConf India.

RubyGems 2.0.1 Released: A Bug-Fix Release

Reading

Inspecting Rails 4 using Ruby 2.0 and TracePoint
Matt Aimonetti shows off a practical use for Ruby 2.0's TracePoint execution tracing functionality.

Visualizing Memory Leaks in Ruby 1.9
Conrad Irwin on some clever work to extend ObjectSpace with a new find_references method to perform better analysis on object and memory usage on Ruby 1.9.

Parsing TOML in Ruby with Parslet
Recently, GitHub founder Tom Preston-Werner created an interesting INI-influenced 'TOML' format. In this series of posts, Nathan Witmer looks at what's involved in building a parser for TOML using the Parslet PEG parser construction library.

Introducing Ress: A System for Building Mobile Optimized Rails Apps
Matthew Robertson introduces his new system for building mobile-optimized Rails applications using semantic, media query-based device detection and server side component optimization.

Ruby 2.0 Walkthrough: The Best Bits
Some slides from my yet-to-be-released 'Ruby 2.0 Walkthrough' that quickly skim through what I consider to be the 'best bits' (and not just the headline features).

A Practical Guide to Using Signed Ruby Gems - Part 1: Bundler

Rails + Ember.js
An introduction to the open source Ember.js JavaScript app framework for Rails developers.

Effective Rails, Part 2: Hiding ActiveRecord

Watching and Listening

Sinatra in SIX Lines: How to Do Crazy Stuff with Ruby
A talk by Konstantin Haase at Ruby Australia.

Libraries and Code

Phusion Passenger 4.0 Release Candidate 4
Leading Rack-based app deployment tool Passenger gets yet another step closer to the 4.0 release.

time-lord: A Human DSL for Time Expressions
A gem that gives you more human like expressions for time and space math. Get fun like 1.hour.ago.to_range and 200.minutes.ago.to_words

identity_cache: Opt-in Read-through ActiveRecord Caching, From Shopify
IdentityCache lets you specify how you want to cache your model objects, at the model level, and adds a number of convenience methods for accessing those objects through the cache. Uses Memcached as the backend cache store.

neg 1.1.0: A Small PEG Parser Library
"One could say it’s a small brother of Parslet."

Gridhook: A Rails Engine to Provide An Endpoint for SendGrid Webhooks

Gem Attack: Show Post-Initialization Notifications on Your Released Gems

Jobs

Web Application Developer for Big Nerd Ranch
Seeking smart, kind folks who want to make the world a little better through developing, training and writing about cutting-edge code.

JS / Ruby Developer at ReplayPoker (Full-Time, Remote)
Looking for a challenge? Our company is looking for a top-notch junior to mid level developer to join our small team and make a big difference!

Senior Ruby/Rails Developer at HotelTonight [San Francisco, California]

Software Engineer at Goodreads [San Francisco, California]

Talented Foodie Rails Developer Wanted at America's Test Kitchen [Brookline, Massachusetts]

Applications Developer at Central Intelligence Agency [District of Columbia]

Senior Ruby on Rails Developer at SponsorPay [Berlin, Germany]

Software Engineering /Academic Applications Developer at Dartmouth College [Hanover, New Hampshire]

Software Engineer at Nextpoint [Madison, Wisconsin]

Senior Back End Software Engineer at The Washington Post - Service Alley [District of Columbia]

Ruby Developer at HouseTrip [London, United Kingdom]

Applications Developer at Merchants Bonding Company [Des Moines, Iowa]

Last but not least..

RTanque: A Robot Programming Game for Rubyists
Players program the 'brain' of a tank and then send their tank into battle with other bots. Based upon the Java project 'Robocode.'

How to add a new partition on OpenBSD

Tue, 05 Mar 2013 11:30:00 +0000

Following the same theme as my last post, instead of claiming space added to the end of a volume by extending an existing partition (technically, a "slice" in OpenBSD), I will show how to add a new partition using this space.

I find it convenient to create the new slice as the "d" slice and then put /home onto it.

So, let’s get started.

Boot into the system’s rescue disk:
```
> boot bsd.rd
```
Then select (S)hell at the prompt.

Claim extra space

Re-initialize the MBR, using the entire disk.

# fdisk -i wd0
Do you wish to write new MBR and partition table? [n] y
Writing MBR at offset 0.
#

Add a new slice for /home (the "d" slice)

This system has only the root slice and swap. We will add a new slice, using all the unused space at the end. In the following example, I have a 120GB volume, where 20GB is used by root and swap, and the rest unallocated.

# disklabel -E wd0
Label editor (enter '?' for help at any prompt)
> p
OpenBSD area: 64-251658225; size: 251658161; free: 209728575
#                size           offset  fstype [fsize bsize  cpg]
  a:         38812976               64  4.2BSD   2048 16384 38128
  b:          3116610         38813040    swap
  c:        251658240                0  unused
> a
partition: [d]
offset: [41929650]
size: [209728575]
FS type: [4.2BSD]
Rounding offset to bsize (32 sectors): 41929664
Rounding size to bsize (32 sectors): 209728544
> p
OpenBSD area: 64-251658225; size: 251658161; free: 31
#                size           offset  fstype [fsize bsize  cpg]
  a:         38812976               64  4.2BSD   2048 16384 38128
  b:          3116610         38813040    swap
  c:        251658240                0  unused
  d:        209728544         41929664  4.2BSD   2048 16384    1
> w
> q
No label changes.
#

Create a filesystem on the new slice:

# newfs /dev/rwd0d
/dev/rwd0d: 102406.5MB in 209728544 sectors of 512 bytes
506 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
super-block backups (for fsck -b #) at:
 32, 414688, 829344, 1244000, 1658656, 2073312, 2487968, 2902624, 3317280,
 3731936, 4146592, 4561248, 4975904, 5390560, 5805216, 6219872, 6634528,
 7049184, 7463840, 7878496, 8293152, 8707808, 9122464, 9537120, 9951776,
 ...
 [snip]
 ...
 193229728, 193644384, 194059040, 194473696, 194888352, 195303008, 195717664,
 196132320, 196546976, 196961632, 197376288, 197790944, 198205600, 198620256,
 199034912, 199449568, 199864224, 200278880, 200693536, 201108192, 201522848,
 201937504, 202352160, 202766816, 203181472, 203596128, 204010784, 204425440,
 204840096, 205254752, 205669408, 206084064, 206498720, 206913376, 207328032,
 207742688, 208157344, 208572000, 208986656, 209401312,
#

Migrate /home

Here, we mount the root filesystem (which contains the old /home), and also the new filesystem, and finally migrate the data from old /home to new space.
```
# mount /dev/wd0a /mnt
# mount /dev/wd0d /mnt2
# (cd /mnt/home; tar cf - .) | (cd /mnt2; tar xpf -)
# rm -rf /mnt/home
# mkdir /mnt/home
```
Note, the tar command above is a nice platform independent way of copying everything from one directory to another, preserving everything (permissions, etc…). The cp command is different on Linux and the *BSDs, yet tar, used in the above fasion, is identical on Linux, FreeBSD, NetBSD, and OpenBSD. I thank Todd Fries of Free Daemon Consulting for teaching me that one.
Mount on boot

The following command will add the new /home to your /etc/fstab so it is automatically mounted upon boot.
```
# echo "/dev/wd0d /home ffs rw,softdep 0 1" >> /mnt/etc/fstab
```
Clean up:
```
# umount /mnt*
# reboot
```
Now enjoy the extra space!

How to resize an OpenBSD root partition

Mon, 04 Mar 2013 10:58:00 +0000

At ARP Networks, sometimes we have to resize an OpenBSD root partition to satsify a customer requirement, template requirement, or something else along those lines.

I’ve sometimes seen customers struggle to do this themselves, so the following is an example of how I resized a root partition from 5 GB to 9 GB, leaving 1 GB for swap. The underlying volume was already expanded from 5 GB to 10 GB, to give us room to do this.

So, let’s get to it.

Boot into the system’s rescue disk:
```
> boot bsd.rd
```
Then select (S)hell at the prompt.

Resize:

Grab the growfs(8) program from the root partition. This isn’t included with the rescue disk, for some reason.

# mount /dev/wd0a /mnt
# cp /mnt/sbin/growfs .
# umount /mnt

Re-initialize the MBR, using the entire disk.

# fdisk -i wd0
Do you wish to write new MBR and partition table? [n] y
Writing MBR at offset 0.
#

Change the size of the slices.

This system has only the root partition and swap. The idea is to remove the swap partition, then extend root, and finally put the swap partition back.

# disklabel -E wd0
# Inside MBR partition 3: type A6 start 63 size 20964762
Treating sectors 63-20964825 as the OpenBSD portion of the disk.
You can use the 'b' command to change this.

Initial label editor (enter '?' for help at any prompt)
> p
OpenBSD area: 63-20964825; size: 20964762; free: 9446220
#                size           offset  fstype [fsize bsize  cpg]
  a:         10490382               63  4.2BSD   2048 16384    1
  b:          1028160         10490445    swap
  c:         20971520                0  unused
> d
partition to delete: [] b
> p
OpenBSD area: 63-20964825; size: 20964762; free: 10474380
#                size           offset  fstype [fsize bsize  cpg]
  a:         10490382               63  4.2BSD   2048 16384    1
  c:         20971520                0  unused
> c
partition to change size: [] a
Partition a is currently 10490382 sectors in size, and can have a maximum
size of 20964762 sectors.
size: [10490382] 9G
Rounding to cylinder: 18876312
> a b
offset: [18876375]
size: [2088450]
FS type: [swap]
> p
OpenBSD area: 63-20964825; size: 20964762; free: 0
#                size           offset  fstype [fsize bsize  cpg]
  a:         18876312               63  4.2BSD   2048 16384    1
  b:          2088450         18876375    swap
  c:         20971520                0  unused
> w
> q
No label changes.
#

Grow the filesystem using growfs(8)

# growfs /dev/wd0a
We strongly recommend you to make a backup before growing the Filesystem

 Did you backup your data (Yes/No) ? Yes
new file systemsize is: 4719078 frags
Warning: 216792 sector(s) cannot be allocated.
growfs: 9111.1MB (18659520 sectors) block size 16384, fragment size 2048
        using 45 cylinder groups of 202.47MB, 12958 blks, 25984 inodes.
super-block backups (for fsck -b #) at:
 10781088, 11195744, 11610400, 12025056, 12439712, 12854368, 13269024, 13683680, 14098336, 14512992, 14927648, 15342304, 15756960, 16171616, 16586272, 17000928, 17415584, 17830240,
 18244896
#

Give it a rinse.

# fsck /dev/wd0a
** /dev/rwd0a
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
100205 files, 418854 used, 4172217 free (97 frags, 521515 blocks, 0.0% fragmentation)

MARK FILE SYSTEM CLEAN? [Fyn?] y


***** FILE SYSTEM WAS MODIFIED *****
#

And that’s pretty much it. Everything is done within OpenBSD’s rescue disk kernel bsd.rd, no need to have a separate CD-ROM / USB rescue disk utility.

Having a Javascript moment with Ruby, WTF is going on here...

Sat, 02 Mar 2013 02:55:00 +0000

WTF is going on here?

(rdb:1) sum
244.4
(rdb:1) amount
244.4
(rdb:1) sum != amount
true
(rdb:1) sum.to_f
244.4
(rdb:1) amount.to_f
244.4
(rdb:1) sum.to_f != amount.to_f
true
(rdb:1) sum == amount
false
(rdb:1) sum.to_f == amount.to_f
false
(rdb:1)

First guess is some type conversion problem…

Still investigating.

Using the quota command within a chroot environment

Fri, 01 Mar 2013 11:50:00 +0000

I was recently using Jailkit on Ubuntu to set up a chroot environment for users of a backup storage server. I wanted to give the users the ability to check their quotas with the regular quota command.

I go about telling Jailkit that the chroot directory will need the quota binary. This should be all that is needed, or so I thought.

Upon checking a quota as a chroot’d user, it barfs the following error,

$ quota
quota: Cannot open quotafile /home/aquota.user: Permission denied
quota: Quota file not found or has wrong format.
$

From what I can tell, this is a very misleading error message. The permissions of aquota.user are always 0600, readable / writable only by root. This can’t be changed, not even as root. Witness.

-rw------- 1 root root 7168 2013-02-27 22:32 /home/bkusers/home/aquota.user

Using a bit of strace and some luck, I find what the quota command is really looking for is /proc/sys/fs/quota.

It simply does a stat() system call on that directory within proc, so if you simply do,

$ sudo mkdir -p /home/bkusers/proc/sys/fs/quota

then the quota command works like a charm within the chroot environment.

$ quota -s
Disk quotas for user testuser (uid 1007):
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
       /dev/md3      16       0   3907M               4       0       0
$

My chroot directory in these examples was obviously /home/bkusers, modify accordingly for your needs.

The 'Yes' men

Sun, 17 Feb 2013 10:47:00 +0000

Let me turn a corner and cover a non-technical topic here for once…

Several years ago, I had a job where I had to ask myself the following question, “Do I do what I think is right, or do I do what will make my new boss happy?”

Now, this was not a moral question, it was simply about how to do some measurements / benchmarking and systems stuff; pretty low key, no big deal.

I stuck to my guns and at some point shortly after, I didn’t have that job anymore. ;)

In my case, there really wasn’t any big life altering changes to myself or anyone else regardless of which decision I made. But in some cases, this can really go the wrong way…

Let me illustrate by quoting something from my reading last night; the book was Amy Goodman’s (Host of Democracy Now!) ”Breaking the Sound Barrier”, and within was a quote from Grammy Award-winning soul singer John Legend when he gave the commencement address at the University of Pennsylvania,

Too often, in business and in government, people are rewarded for having the answer that the person they report to wants them to have: “Yes, sir. We can provide mortgages to people who have no down payment and can’t afford the monthly payments.”… “Yes, ma’am. I can write a legal brief to justify torture.”
John Legend Commencement Address at the University of Pennsylvania 2009

Stick to your guns, fuck your boss.

Migrated from Mephisto to Octopress

Tue, 12 Feb 2013 10:01:00 +0000

I’m proud to say I’ve dusted off this blog and migrated from Mephisto to Octopress. Now I can actually post new content (the old Mephisto system got so dated and went to dependency hell that I was hardly able to do anything besides keep the static cached files up).

I’m going to try to go through all the posts and cleanup / fix any loose ends. So far the import went pretty smooth and I’ve only burned a few hours (instead of a whole weekend).

I was unable to successfully import the comments, so those will be gone forever. Most were spammy anyway, and any comments posted at least within the last year or two use Disqus, and those are still around.

I’d like to get my usual theme working again, but alas, that is very low priority right now. Perhaps a new look is warranted anyway. However, at least, I can move away from the default theme. Gotta read more docs…

Let me know if anything is out of the ordinary.

Thanks!

Testing Octopress...

Tue, 12 Feb 2013 08:38:00 +0000

Testing 1 2 3…

is this thing on?

An Instrumented Library in ~30 Lines

John Nunemaker — Wed, 23 Jan 2013 20:00:00 +0000

The Full ~30 Lines

For the first time ever, I am going to lead with the end of the story. Here is the full ~30 lines that I will break down in detail during the rest of this post.

require 'forwardable'

module Foo
  module Instrumenters
    class Noop
      def self.instrument(name, payload = {})
        yield payload if block_given?
      end
    end
  end

  class Client
    extend Forwardable

    def_delegator :@instrumenter, :instrument

    def initialize(options = {})
      # some other setup for the client ...
      @instrumenter = options[:instrumenter] || Instrumenters::Noop
    end

    def execute(args = {})
      instrument('client_execute.foo', args: args) { |payload|
        result = # do some work...
        payload[:result] = result
        result
      }
    end
  end
end

client = Foo::Client.new({
  instrumenter: ActiveSupport::Notifications,
})

client.execute(...) # I AM INSTRUMENTED!!!

The Dark Side

A while back, statsd grabbed a hold of the universe. It swept in like an elf on a unicorn and we all started keeping track of stuff that previously was a pain to keep track of.

Like any wave of awesomeness, it came with a dark side that was felt, but mostly overlooked. Dark side? Statsd? Graphite? You must be crazy! Nope, not me, definitely not crazy this one. Not. At. All.

What did we all start doing in order to inject our measuring? Yep, we started opening up classes in horrible ways and creating hooks into libraries that sometimes change rapidly. Many times, updating a library would cause a break in the stats reporting and require effort to update the hooks.

The Ideal

Now that the wild west is settling a bit, I think some have started to reflect on that wave of awesomeness and realized something.

I no longer want to inject my own instrumentation into your library. Instead, I want to tell your library where it should send the instrumentation.

The great thing is that ActiveSupport::Notifications is pretty spiffy in this regard. By simply allowing your library to talk to an “instrumenter” that responds to instrument with an event name, optional payload, and optional block, you can make all your library’s users really happy.

The great part is:

You do not have to force your users to use active support. They simply need some kind of instrumenter that responds in similar fashion.
They no longer have to monkey patch to get metrics.
You can point them in the right direction as to what is valuable to instrument in your library, since really you know it best.

There are a few good examples of libraries (faraday, excon, etc.) doing this, but I haven’t seen a great post yet, so here is my attempt to point you in what I feel is the right direction.

The Interface

First, like I said above, we do not want to force requiring active support. Rather than require a library, it is always better to require an interface.

The interface that we will require is the one used by active support, but an adapter interface could be created for any instrumenter that we want to support. Here is what it looks like:

instrumenter.instrument(name, payload) { |payload|
  # do some code here that should be instrumented
  # we expect payload to be yielded so that additional 
  # payload entries can be included during the 
  # computation inside the block
}

Second, we have two options.

Either have an instrumenter or not. If so, then call instrument on the instrumenter. If not, then do not call instrument.
The option, which I prefer, is to have a default instrumenter that does nothing. Aptly, I call this the noop instrumenter.

The Implementation

Let’s pretend our library is named foo, therefore it will be namespaced with the module Foo. I typically namespace the instrumenters in a module as well. Knowing this, our noop instrumenter would look like this:

module Foo
  module Instrumenters
    class Noop
      def self.instrument(name, payload = {})
        yield payload if block_given?
      end
    end
  end
end

As you can see, all this instrumenter does is yield the payload if a block is given. As I mentioned before, we yield payload so that the computation inside the block can add entries to the payload, such as the result.

Now that we have a default instrumenter, how can we use it? Well, let’s imagine that we have a Client class in foo that is the main entry point for the gem.

module Foo
  class Client
    def initialize(options = {})
      # some other setup for the client ...
      @instrumenter = options[:instrumenter] || Instrumenters::Noop
    end
  end
end

This code simply allows people to pass in the instrumenter that they would like to use through the initialization options. Also, by default if no instrumenter is provided, we use are noop version that just yields the block and moves on.

Note: the use of || instead of #fetch is intentional. It prevents a nil instrumenter from being passed in. There are other ways around this, but I have found using the noop instrumenter in place of nil, better than complaining about nil.

Now that we have an :instrumenter option, someone can quite easily pass in the instrumenter that they would like to use.

client = Foo::Client.new({
  :instrumenter => ActiveSupport::Notifications,
})

Boom! Just like that we’ve allowed people to inject active support notifications, or whatever instrumenter they want into our library. Anyone else getting excited?

Once we have that, we can start instrumenting the valuable parts. Typically what I do is I setup delegation of the instrument to the instrumenter using ruby’s forwardable library:

require 'forwardable'

module Foo
  class Client
    extend Forwardable

    # forward instrument in this class to @instrumenter, for those unfamilier
    # with forwardable.
    def_delegator :@instrumenter, :instrument

    def initialize(options = {})
      # some other setup for the client ...
      @instrumenter = options[:instrumenter] || Instrumenters::Noop
    end
  end
end

Now we can use the instrument method directly anywhere in our client instance. For example, let’s say that client has a method named execute that we would like to instrument.

module Foo
  class Client
    def execute(args = {})
      instrument('client_execute.foo', args: args) { |payload|
        result = # do some work...
        payload[:result] = result
        result
      }
    end
  end
end

With just a tiny wrap of the instrument method, the users of our library can do a ridiculous amount of instrumentation. For one, note that we pass the args and the result along with the payload. This means our users can create a log subscriber and log each method call with timing, argument, and result information. Incredibly valuable!

They can also create a metrics subscriber that sends the timing information to instrumental, metriks, statsd, or whatever.

The Bonus

You can even provide log subscribers and metric subscribers in your library, which means instrumentation for your users is simply a require away. For example, here is the log subscriber I added to cassanity.

require 'securerandom'
require 'active_support/notifications'
require 'active_support/log_subscriber'

module Cassanity
  module Instrumentation
    class LogSubscriber < ::ActiveSupport::LogSubscriber
      def cql(event)
        return unless logger.debug?

        name = '%s (%.1fms)' % ["CQL Query", event.duration]

        # execute arguments are always an array where the first element is the
        # cql string and the rest are the bound variables.
        cql, *args = event.payload[:execute_arguments]
        arguments = args.map { |arg| arg.inspect }.join(', ')

        query = "#{cql}"
        query += " (#{arguments})" unless arguments.empty?

        debug "  #{color(name, CYAN, true)}  [ #{query} ]"
      end
    end
  end
end

Cassanity::Instrumentation::LogSubscriber.attach_to :cassanity

All the users of cassanity need to do to get logging of the CQL queries they are performing and their timing is require a file (and have activesupport in their gemfile):

require 'cassanity/instrumentation/log_subscriber'

And they get logging goodness like this in their terminal:

The Accuracy

But! BUT, you say. What about the tests? Well, my friend, I have that all wrapped up for you as well. Since it is so easy to pass through an instrumenter to our library, we should probably also have an in memory instrumenter that keeps track of the events instrumented, so you can test thoroughly, and ensure you don’t hose your users with incorrect instrumentation.

The previous sentence was quite a mouthful, so my next one will be short and sweet. For testing, I created an in-memory instrumenter that simply stores each instrumented event with name, payload, and the computed block result for later comparison. Check it:

module Foo
  module Instrumenters
    class Memory
      Event = Struct.new(:name, :payload, :result)

      attr_reader :events

      def initialize
        @events = []
      end

      def instrument(name, payload = {})
        result = if block_given?
          yield payload
        else
          nil
        end

        @events << Event.new(name, payload, result)

        result
      end
    end
  end
end

Now in your tests, you can do something like this when you want to check that your library is correctly instrumenting:

instrumenter = Foo::Instrumenters::Memory.new

client = Foo::Client.new({
  instrumenter: instrumenter,
})

client.execute(...)

payload = {... something .. }
event = instrumenter.events.last

assert_not_nil event
assert_equal 'client_execute.foo', event.name
assert_equal payload, event.payload

The End Result

With two instrumenters (noop, memory) and a belief in interfaces, we have created immense value.

Fin

Go forth and instrument all the things!

A Simple Tour of the Ruby MRI Source Code with Pat Shaughnessy

Peter Cooper — Mon, 03 Dec 2012 15:20:31 +0000

I'm not in Ruby core or, well, even a confident C coder anymore, but I've long enjoyed digging in the Ruby MRI source code to understand weird behavior and to pick up stuff for my Ruby course.

Pat Shaughnessy is also a fan of digging around in Ruby's internals and has written some great posts like How Ruby Executes Your Code, Objects, Classes and Modules, and Exploring Ruby's Regular Expression Algorithm.

When Pat released his Ruby Under a Microscope book, I knew it would be right up my street! He digs into how objects are represented internally, why MRI, Rubinius and JRuby act in certain ways and, of course, "lots more."

I invited Pat to take a very high level cruise through the MRI codebase with me so we could share that knowledge with Ruby programmers who haven't dared take a look 'under the hood' and to show it's not as scary or pointless as it may seem.

It's 100% free so enjoy it above or on YouTube in 720p HD.

P.S. Pat is happy to do another video digging deeper into how Ruby actually takes your code and executes it and he's able to walk through the actual virtual machine for us. If the reaction to this video is good, we'll sit down again and see if we can do it! :-)