Broadcasting Adam

Introducing FronendServer

me@broadcastingadam.com (Adam Hawkins) — Mon, 20 Feb 2012 00:00:00 +0200

Radium has turned out to be a real beast. After much delibration, we decided to move to a pure JS frontend written in Ember. He’s written in JS and basic CSS. No coffeescript or SASS. We needed an easy way to compile all the JS and CSS files into two different files and host them on the web. We also have a few other requirements.

Frontend code is completly separate of backend API.
Frontend code can be deployed and maintained independently.
Enable SASS/Coffeescript. I freaking love SASS.
Recompile assets in development
API proxy to avoid CORS issues in development
Easily deployable by non server guys.

So me being the server side guy, I had to sort this out. Josh turned me on to: @wycats’s rake-pipeline-web-filters. It’s an extension to rake-pipeline. Rake-pipeline is essentially the Rails asset pipeline done in a less bitchy way. It’s a rake extension that does what the rails asset pipeline does, except it’s much more powerful. You can easily define custom filters for things like minification.

Rake::Pipeline::WebFilters provides the heavy lifting. I wrote a simple Rack app to serve the stuff based on sensible defaults (convention over configuration) and voilla! We have asset compilation and a development server. Combine some more glue code to connect a proxy and other bits and we have a more complete solution.

Introducting FrontendServer. Here’s what it does:

Compile JS/CS into Minispade modules
Compile LESS/SASS/CSS into css files
Reload assets in development
Proxy the backend API
Compile into one public directory
Deployable to heroku by any git user

I’ve used the classic Backbone todos app for an example. The example is running here. Source here.

Rails 3.2, Heroku, & rails_log_stdout

me@broadcastingadam.com (Adam Hawkins) — Mon, 16 Jan 2012 00:00:00 +0200

I try to keep my applications on the bleeding edge of Rails. After being stuck Rails 2 for almost 1.5 years after Rails 3 was released, it’s been very nice. Of course the bleeding edge has it’s drawbacks. Enter Rails 3.2.0rc2 and Heroku. vendor/plugins is deprecated in Rails 3.2. Heroku dumps it’s configuration plugins into vendor/. This gives you a nice deprecration warning everytime a rails environment process starts.

I had to do some debugging with developers using the API. This means I need to be able to see what their applications are sending in realtime. I’ve been using Heroku for a long time, so I just fire up heroku logs --tail and watch away. Except, now with rails 3.2.0rc2 I don’t see anything coming from rails. Only the start up messages and the request accepted message. Well, there is no parameter information.

2012-01-16T13:26:50+00:00 app[web.1]: Started POST "/todos" for 82.181.188.119 at 2012-01-16 13:26:50 +0000
2012-01-16T13:26:51+00:00 heroku[router]: POST api.radiumcrm.com/todos dyno=web.1 queue=0 wait=0ms service=212ms status=422 bytes=92

Well shit. That doesn’t really tell me anything at all. I want to see this:

2012-01-16T13:26:50+00:00 app[web.1]: Started POST "/todos" for 82.181.188.119 at 2012-01-16 13:26:50 +0000
2012-01-16T13:26:50+00:00 app[web.1]: Processing by TodosController#create as JSON
2012-01-16T13:26:50+00:00 app[web.1]:   Parameters: {"todo"=>{"kind"=>"call", "finish_by"=>"2012-05-01T18:52Z"}}
2012-01-16T13:26:51+00:00 app[web.1]: Completed 422 Unprocessable Entity in 208ms (Views: 0.3ms | ActiveRecord: 110.7ms)
2012-01-16T13:26:51+00:00 app[web.1]: cache: [POST /todos] invalidate, pass
2012-01-16T13:26:51+00:00 heroku[router]: POST api.radiumcrm.com/todos dyno=web.1 queue=0 wait=0ms service=212ms status=422 bytes=92

Turns out there is problem with the [rails_log_stdout](https://github.com/ddollar/rails_log_stdout) plugin heroku uses do it’s business. The rails initialization process has changed slightly. ActionController::Base.logger was being assisigned to the standard log/production.log file somewhere before the std out logger was being set. Then through the magic of ||= it is already defined and not reassigned to the stdout logger. The plugin is one init.rb file which is now deprecated. Time to fix this!

I wrote a simple rails engine (rails 3+ only!) to fix this and stop the deprecation warnings. Hopefully heroku will use this code to make their platform rails 3.2 compatible. You can grab the code here.

Then stick this bad boy in your Gemfile:

group :production do
  gem 'rails_log_stdout', :git => 'git://github.com/Adman65/rails_log_stdout.git'
end

Now you have also have to tell Rails to explicitly load a whitelist of plugins. You can do this in production.rb

config.plugins = [ :rails_serve_static_assets ] # ONLY load static assets plugin

You’ll still see depreaction warnings because heroku is injecting code into your source, but logging will work as you expect. If you are using any other plugins make sure you add them to the list.

Happy log tailing!

Generating Routes for API Documentation

me@broadcastingadam.com (Adam Hawkins) — Sat, 14 Jan 2012 00:00:00 +0200

I recentely came across this situation. I was updating the documenation for our main API. I have a section which includes all the routes grouped by resource with HTTP verb. I ran rake routes and edited the output to get it how I wanted. Fast forward a few API revisions with new routes and routes removed. Doing this process over and over again would be very tedious. Here’s what I wanted to generate:

GET    /accounts/:id
PUT    /accounts/:id

POST   /users/:user_id/follow
GET    /users/:user_id/feed
GET    /users
GET    /users/:id
PUT    /users/:id
DELETE /users/:id

POST   /contacts/:contact_id/deals
POST   /contacts/:contact_id/notes
POST   /contacts/:contact_id/todos
POST   /contacts/:contact_id/follow
GET    /contacts/:contact_id/feed
GET    /contacts
POST   /contacts
GET    /contacts/:id
PUT    /contacts/:id
DELETE /contacts/:id

POST   /todos/:todo_id/comments
POST   /todos/:todo_id/reminders
GET    /todos
POST   /todos
GET    /todos/:id
PUT    /todos/:id
DELETE /todos/:id

POST   /deals/:deal_id/comments
POST   /deals/:deal_id/todos
POST   /deals/:deal_id/follow
GET    /deals/:deal_id/feed
GET    /deals
GET    /deals/:id
PUT    /deals/:id
DELETE /deals/:id

POST   /groups/:group_id/todos
POST   /groups/:group_id/follow
GET    /groups/:group_id/feed
GET    /groups
POST   /groups
GET    /groups/:id
PUT    /groups/:id
DELETE /groups/:id

POST   /announcements/:announcement_id/comments
GET    /announcements
POST   /announcements
GET    /announcements/:id
PUT    /announcements/:id
DELETE /announcements/:id

POST   /emails/:email_id/comments
POST   /emails/:email_id/todos
GET    /emails
POST   /emails
GET    /emails/:id
PUT    /emails/:id
DELETE /emails/:id

POST   /sms/:sms_id/comments
POST   /sms/:sms_id/todos
POST   /sms/import
GET    /sms
POST   /sms
GET    /sms/:id
PUT    /sms/:id
DELETE /sms/:id

POST   /ims/:im_id/comments
GET    /ims
POST   /ims
GET    /ims/:id
PUT    /ims/:id
DELETE /ims/:id

POST   /meetings/:meeting_id/comments
POST   /meetings/:meeting_id/todos
POST   /meetings/:meeting_id/reminders
PUT    /meetings/:id/reschedule
PUT    /meetings/:id/cancel
GET    /meetings
POST   /meetings
GET    /meetings/:id
PUT    /meetings/:id
DELETE /meetings/:id

PUT    /invitations/:id/confirm
PUT    /invitations/:id/reject

POST   /phone_calls/:phone_call_id/todos
POST   /phone_calls/:phone_call_id/comments
POST   /phone_calls/import
GET    /phone_calls
GET    /phone_calls/:id
PUT    /phone_calls/:id
DELETE /phone_calls/:id

POST   /notes/:note_id/todos
GET    /notes
GET    /notes/:id
PUT    /notes/:id
DELETE /notes/:id

GET    /reminders
POST   /reminders
GET    /reminders/:id
PUT    /reminders/:id
DELETE /reminders/:id

POST   /campaigns/:campaign_id/follow
POST   /campaigns/:campaign_id/comments
POST   /campaigns/:campaign_id/notes
POST   /campaigns/:campaign_id/call_lists
GET    /campaigns/:campaign_id/feed
GET    /campaigns
POST   /campaigns
GET    /campaigns/:id
PUT    /campaigns/:id
DELETE /campaigns/:id

PUT    /followings/:id/approve
GET    /followings
POST   /followings
GET    /followings/:id
PUT    /followings/:id
DELETE /followings/:id

GET    /products
POST   /products
GET    /products/:id
PUT    /products/:id
DELETE /products/:id

GET    /call_lists
GET    /call_lists/:id
PUT    /call_lists/:id
DELETE /call_lists/:id

GET    /comments
POST   /comments
GET    /comments/:id
DELETE /comments/:id

POST   /bug

POST   /echo

I also wanted to group them with spaces by the associated resource. This was actually pretty easy to do. I opened up github and hit t and searched routes.rake to find the source code. Then subclassed the generator and called it myself. Took a few tries to get all the bugs worked out. The code is pretty easy. Here’s the finished product. Drop it in api_routes.rake:

desc 'Print out all defined routes used for the api without rails specific information'
task :api_routes => :environment do
  Rails.application.reload_routes!
  all_routes = Rails.application.routes.routes

  require 'rails/application/route_inspector'

  class ApiRouteInspector < Rails::Application::RouteInspector
   def formatted_routes(routes)
      verb_width = routes.map{ |r| r[:verb].length }.max
      path_width = routes.map{ |r| r[:path].length }.max

      routes = routes.reject do |r|
        r[:reqs] =~ /#(new|edit)/
      end

      groups = routes.inject({}) do |set, r|
        section = r[:path].match(/\/([^\/|\(]+)/)[1]

        set[section] ||= []
        set[section] << r
        set
      end

      groups.values.map do |rs|
        rs.map do |r|
          line = "#{r[:verb].ljust(verb_width)} #{r[:path].gsub("(.:format)", '').ljust(path_width)}"
          line = "#{line}\n" if rs.last == r
          line
        end.join("\n")
      end
    end
  end

  inspector = ApiRouteInspector.new
  puts inspector.format(all_routes, ENV['CONTROLLER']).join "\n"
end

Then just run bundle exec rake api_routes and you’ll have some quick documentation to give to your devs.

Moving on from Rails: Part 2

me@broadcastingadam.com (Adam Hawkins) — Sun, 04 Dec 2011 00:00:00 +0200

I was quite surprised by how much feedback I received on the original post. You should read the first one before reading this. I’m writing this post to respond to some common questions, points, and concerns.

Setting The Record Straight

It seems the point of the previous post has been lost in the title. I guess this my fault. I am not quitting or bashing Rails. I am “moving on from Rails” because it is becoming increasingly less integral to how I construct my applications. My applications are JSON based communicating over HTTP. I still use Rails for all my web based projects with different components added/removed based on the application’s requirements.

Rails is becoming less important because I no longer rely on it to be my application. I try to use ActionPack as an interface to my code which is not bound to Rails in anyway. In this setup, I do not the traditional full stack setup. I don’t use ActiveRecord and I don’t use ActionView. I continue to use Rails mainly because of ActionPack since it’s (from my experience) the easiest way to handle and respond to HTTP requests.

Oh, I use ActiveModel a ton because it’s awesome as. I also use tons of pits and pieces from ActiveSupport because #present?, #blank? and #underscore etc are just too handy when you’re doing programming with strings.

Alternatives

I was surprised (again) by how many blasted me about not suggesting alternatives. I didn’t know I supposed to. Still, I don’t suggest any alternatives because I don’t think there are any for the way I structure my applications. I could use Sinatra (and I do) in less complicated scenarios. I like controller classes. I like routes.rb. With 30+ controllers and easily over 100 routes I think that would become an unweildly application. Sinatra is probably closer to what I want, but I’d end up building stuff from ActionPack into the application just to make it work. Someone also suggested: Renee but I have yet to play with it.

Tunnel Vision

This seems to be a recurring pattern with developers who have “discovered” design and architecture through frameworks and can not seem to separate that from the tools used.

I completely agree with that statement. I think this especially true of any full stack framework. If you are introduced to web development through that stack and work with only that stack for a very long time, it will be very hard to separate yourself from that train of thinking. This has happened to me and only has significantly changed after building much larger and more complicated web applications.

The other day I was speaking to someone who could not separate models from database tables. There is no actual MVC going on there. Many people think that models are just a wrapper around persistence. There is no concept what does this model actually do. What does it represent in your application? How does it interact with other models? Does it even need to be persisted? Tunnel vision is not unique to Rails, but I think it effects many new programmers who fall in love with the framework and use it for years.

PHP

I don’t like PHP. I used to love PHP because it let me do all sorts of cool stuff. I could create variable variables (I have no idea why that exists.) plus all the other trickery you can do in the language. After having programmed in a few more languages over the years, now I think that PHP is fundamentally flawed as a language. Therefore, I have no interest in using it or following it.

There are plenty of people who get paid good money write PHP. I’m sure there are plenty of them who love PHP. I’m sure there are plenty of them who love CakePHP or whatever framework they use (if they use it). Great for them. As long as you are happy with whatever you do and whatever tools you use then good for you.

Replying to Dor

Dor wrote his own post in response to mine. Since he was nice enough to write a post which is much longer than a tweet, I’ll reply in long-form.

So I think Adam is exaggerating and I think I can understand him but it seems to me he is looking for an answer at the wrong place. I am also not sure what is the alternative that he’s suggesting…

I’ve found the answer to what I need. ActionPack is the answer.

During the last 6 years there were so many posts of that kind… but the music is still on.

The music is still on, but the tune has drastically changed. You don’t scaffold anymore. There was no MongoDB. There was no such things as “JavaScript clients.” New user expectations and technologies have changed the a web framework’s role. You can see this with 3.1 with inclusion of the (much maligned) asset pipeline and seperation of ActiveModel into an abstract concept reusable in other applications.

I am nt sure why someone want to write such a negative post, to me it seems like a waste of time. Why don’t you write how much you enjoy the new JackRobinson(tm) technology and its remarkable advantages over Rails…

I don’t consider it a negative post at all. I think it is an interesting post which talks about the changing structure of web applications.

For example, the natural way in which CoffeeScript was made a default on Rails 3.1 stack. In many languages you can pull the first element of an array with the method #first. Did you know that in Rails you can also do .second, .third and so on until 40-something? DHH, before he’s a technology guy, he’s a great product and he thinks about the developers, that’s what I like most about Rails. Man, he’s the g– d— Steve Jobs of development.

Coffescript is ok. It’s cool that decided to make assets first class citizens in Rails 3.1. However, I think the asset pipeline is not going to be used when developer API based applications. Why do I need the asset pipeline to serve assets if all care about is JSON? Hell, why would I even worry about keeping assets in source control? It doesn’t make sense. These are two completely independent concepts. I use Rails to serve JSON. Something else does whatever else. You can have a single nginx box serving up static JS all day. I don’t mess with that at all. That’s what makes Rails great. You can just that off. I turn it off and pay no attention to it. I think the asset pipeline is generally a useless feature for developing modern applications.

I agree today the web is much more about client-side but that’s what backbone.js and spine.js are doing, extending Rails.

These things have nothing to do with Rails at all. There is a complete separation of concerns. They do not extend Rails at all. They extend ANY HTTP based application. I could have an assembly program serving up JSON it makes no difference. Backbone/Spine/Knockout/what have you simply make it easier to create modern interactive UI’s in the browser.

As for Ruby, the 1.9x version is fast and slick

Totally. 1.9.3 is a huge improvement over 1.8.7. I wish they would’ve just skipped what 1.9.0,1,2 and went to right to 1.9.3 :D

One final note about writing Rails applications: most of the fun and handy things you can do in code are done by ActiveSupport.

All in all, I believe there is place for many flavours. Just like in real life, mongrels are better than pure-breeds. The environments learn from each other and so improve each other and everyone are happier :)

Agree. Rails developers will learn to adapt framework to fit there needs or they will simply move on.

Summing It Up

tl;dr - web development is changing, the frontend side of things is gaining significance, while the backend is moving to, well, the background. No reason to be surprised there, right?

bphogan does a nice job cutting to the point as well:

This is less about moving on from Rails and more about moving on from building static pages from a database. Lots of web folks have been predicting this. I’ve been saying for about two years now that the days of serving entire HTML pages from the serverside are numbered. With things like Backbone, I can bring up a Rails app without views and do something pretty awesome. And then it becomes a question as to what Rails offers. (I bolded that for emphasis)

I love Rails. It got me back into web app development in 2005 after nearly burning out. But Rails isn’t exactly keeping up and people who need to move on are going to do that.

Div does a better job of summing up my thoughts than I can do. Props man:

It seems like the title is a bit poorly chosen.

The author talks about the importance of having a platform to cater to the multitude of devices and other apps out there, and that this means rails isn’t the center of the universe anymore.

To me, this does not necessarily mean moving on from rails.

It does mean moving on from writing all code in rails.

There could be a fullfledged backbone.js app powering a responsive ui, and a distributed clojure jobqueue making sure messages are fanned out to their destined networks in the backend. However, there is still room in this picture for Rails as a router of sorts.

Rails still makes it easy to quickly build a solid REST api, and easy to delegate long-running jobs to a separate system, in this type of architecture, Rails would have roughly a third of the responsibility / code that it has in a Rails only architecture, but it’s still a vital component.

I hope that his post clarified the original. Now we can all move on with our lives :D

Moving on from Rails

me@broadcastingadam.com (Adam Hawkins) — Sat, 26 Nov 2011 00:00:00 +0200

I’ve been doing Rails since 2006. My skill level and the framework as changed greatly since. I came from PHP web sites and Java through academics. Ruby was breath of fresh air. Rails was a thing a of beauty. It was exactly what I needed at the time. Now times are different.

A Changing Web Yields Changing Requirements

EDIT: I’ve written a follow up to this post that can be found here.

Web developers know there is an incredibly high turnover rate in technologies. Think about to 6 months ago. Did you know about Coffeescript? What about backbone.js? What about websockets/push? Not all of those things are technologies. Coffeescript is not a technology, it’s just a nice layer on top of Javascript. It is not a new technology like HTML5 Websockets. Socket based communications are not new by any means, but they are new for us since we can use them in the browser.

Did you also notice how all the things I rattled off were purely related to JavaScript? This isn’t a purely server side driven game anymore. The paradigm is shifting in how we architect modern internet applications. Before we got to where we are now we have to know where we came from.

I was coming from PHP. PHP was shit then and is still shit now. I just I think this is a common story for many Rails guys. I didn’t realize how shitty PHP as language was until I started learning Ruby through Rails. At the time, Rails was exactly what many devs wanted. It made common tasks like generating basic CRUD interfaces. You could simply scaffold and start building (but I never recommend that), but you could do it. It made managing a DB easy. There was schema.rb. There were ActiveRecord associations. ActionController and AJAX handling. It did everything and it made a huge splash in the industry. Rails made such a big impact it was able to drive the growth of 37signals into a major powerhouse. There are hundreds of committers and the framework is still very popular and useful. It made building Web 2.0 quick and easy. Hell, the generated application greets you with: “you’re riding on Rails.”

I look back on those days fondly. It was a much more simple time. The barrier to enter the market was low. You could churn out SaaS services so quickly. Generate a rails app, throw up some scaffolds, and boom. There ya go. Zero to sixty faster than an R8. Now things are different.

Think back to when Rails first came out. There was no iPhone. There was no Android. People still owned Nokias and actually bought stock in the damn company. Windows XP was still massively popular and we were fighting to get a decent version of IE (but that will never end). There was no such thing as a mobile web. No one was thinking about tablets. How old is the iPad? Not very old. The web has drastically changed due to the smartphone explosion. This is only the first step in a true multiplatform internet. A few years ago you could get by with having a mobile version of your web app. Now that’s not enough. That concept is almost laughable today. The bar is set pretty high. People expect applications for different devices. This is unfortunately our fault since we’ve trained them. I was just on lemon.com while showing it to my boss. There is a huge picture of numerous devices. People are off their computers now. Now it’s simply not enough to build a web application. You need to build a service. You need to build your own platform to build your own applications off of. You will not survive if you don’t. The changing web yields new requirements in how we design and structure modern applications.

SoA Platforms then Applications

Service oriented architecture has always been very fascinating concept. Jeff Bezos was way ahead of his time when he basically pulled a Hitler and dictated that everything would switch over to SoA. We are at point in the web where platforms represent core business value. You need to have a platform that’s accessible on any device either natively or with a browser. So now you’re faced with a question. How can you develop your platform and expose to different devices: HTTP. I like HTTP and I make my platforms speak HTTP. We can use HTTP to create JS applications for browsers. We can also use HTTP to communicate with an external system through OS level code. You can build a native application on Android backed by an HTTP service. You can build your JS application backed by HTTP. Don’t think that the browser is the only answer. Not every single application can exist in the browser. What if you need to write files? What if you need to access hardware? The browser is not the be-all & end all of internet applications. People will always be creating native applications. So don’t look at everything purely through the: HTML5 and JS lens. Sometimes applications need more.

I think many developers are realizing this. You can see it with things like Sproutcore (much love to SC) and Backbone. Less so than backbone and more so with SC < 2. There is this revolution in MVC on the client. Ya, you could call it that, but really it’s the beginning of a wall being built between clients and servers. Backbone is becoming increasingly popular for good reasons. It allows developers to make javascript applications with nice MVC trimmings. But it’s more than that. It makes people structure and organize their javascript as an application, and not simply a hodepodge of event bindings and AJAX related trickery. Applications are now being built in Javascript. This trend will only continue. Why? Because the interfaces are fucking fast. Things happen on the user’s computer and can be persisted back to the server later. Hell, Sproutcore even abstracted HTTP persistence completely out of the framework. There is a clear separation of concerns. The platform processes data and serves up JSON. Then end programs actually do all the “work” by presenting UI’s and all that good stuff. And now you’ve built the first consumer for your platform. With Sproutcore you may even build the entire application without even conntecting it to a server. At that point, you’re able to treat persistence as an implementation detail which drastically changes the way you think.

HTTP Backed Services

HTTP backed services are inherently easy to write and manage. They are much easier to write than traditional web applications because you do not have to worry about V–just MC. From my experience, generating a good UI (V) takes orders of magnitude more than implementing M and C. HTTP is perfect because you only have to worry about garbage in and garbage out. In Rails, this means you really only care about params and respond_with. Data in and data out. Now, when I look at these API’S I look at the responsibility to the end developer: return JSON. That is it. A single focus: return JSON. Of course there is processing involved but the main point is return JSON that end application can use for it’s own purposes. That makes developing this very easy because it removes many other requirement and makes you think about a fundamental thing: HTTP is only how the outside talks to my code. My application is not HTTP and it’s certainly not just HTML and JSON. This is only how I talk to the outside world. The code can be something totally different. You could have nginx talking to a haskell application. No one cares as long as the parameters and data coming out are according to spec. This has been extremely freeing for me because I no longer think rails is my application.

Changing How We Code

Avdi Grimm verbalized some of my current feelings in this regard. His object on Rails paper goes through the entire process of creating a blog using OOP techniques. The code is developed completely outside of Rails. Rails isn’t even introduced. Models are derived completely on their own. Collaborators and responsibilities are defined as well. Slowly the system takes shape. First you need ActiveModel for some validations. Everything is going well. Eventually you’re going to need persistence. He treats persistance as an implementation details Mdoels are more than ActiveRecord! Your code is more than Rails. Rails is not your code. You can and should develop the business logic completely on your own ISOLATED from whatever code you’ll eventually use to allow the outside world to consume your service. Corey Haines started something called fast tests. Fast tests are great. Everyone loves them. They make tdd faster blah blah. His essential point was to move code out of Rails into Ruby modules that can be tested in isolation. This is a step in the right direction but I think it’s a very small one. You should go all out and simply move your code out of whatever framework you’re eventually going to use. I highly recommend Avdi’s paper on Objects on Rails to see a very refreshing approach. The continuing theme here is dividing parts of your eventual platform. Most important is splitting access from application.

Moving Away From Rails

Rails has always been fantastic at one thing: accepting, handling, and returning body for a HTTP response. What you choose to put that body has always been up to you. Just recently, it’s been assumed you’re going to generate HTML. Now I just want to generate JSON. Designing my HTTP based services only require two things:

Accept http requests
Give me a nice API for determine response code and response body.

Rails has always been fantastic at those two things. But now, I do the rest. I’m not really worried about ActiveRecord. Let me handle the models. I don’t need ActionView. There are a lot of things I don’t need from Rails anymore. I think that’s good. What makes Rails 3 great is you can modularize it by removing (or including) components. Now I can get the most out of Rails by making it fit my requirements. But that’s it. I no longer think Rails is my application. It’s just a nice thing for handling how it’s accessed.

Moving on in Life

I’ve become increasingly disinterested with Rails over the past while. This is nothing to do with the framework itself. It’s just no longer related to my core interests. A few years ago, it was my only interest. It was simply how you build web applications. That time has come and gone. Now we are faced with applications with more diverse requirements. You may have a Rails based application for handling the majority of web requests working in conjunction with a Node process for evented stuff. Hell, you may at some point even have processes in different languages which have been optimized for different things. Look at Twitter with Scala. There are many permutations of different technologies for different requirements. There will be more permutation as the way we architect the modern web. For me, and I think many others, Rails will become and increasingly smaller piece of the puzzle. Why? Simply because their platform is more than Rails. Now I focus on other questions. Rails answers some of them, however I’m finding it’s the answer less and less.

Reviewing Rails 3 in Action

me@broadcastingadam.com (Adam Hawkins) — Mon, 21 Nov 2011 00:00:00 +0200

Ryan Bigg is writing a book called Rails 3 In Action. I’ve been lucky enough to help read the book and generally review its contents while it’s been written. I consider myself an advanced Rails developer, so when I read books like this I look for them to cover specific things that new developers need to know and how well the example integrate best practices. In this post, I review Rails 3 in Action.

About the Book

Rails 3 in Action is written to give people interested in learning Rails an indepth look into creating a basic Rails application using TDD. The book is divided into three sections and one appendix. Here is the table of contents:

Part 1: Getting Rolling

Ruby on Rails, the Framework
Testing Saves Your Bacon
Developing a Real Application
Oh CRUD!
Nested Resources
Authentication and Basic Authorization
More Authorization

Part 2: Putting on the Bling

File Uploads
Tracking State
Tagging
Sending Email
Designing an API
Deployment

Part 3: Further Enhancements

Alternative Authentication
Performance Enhancing Basics
Engines
Rack Applications

I was really looking forward to this book because the authors planned on covering more advanced topics. That way, the new readers can go from zero knowledge to going through an overview on advanced topics. The beginning chapters are well written and cover all the basics. The advanced chapters are very imformative. The engines chapter is most likely the best documentation to how to write an engine except existing code. One reason I really like this book is because current and popular gems are covered throughout the book. Devise, Paperclip, Cucumber, and RSpec are used throughout the book. I think this gives the book a very current feel and reflects what developers are doing at this point.

Part 1: Getting Rolling

I can honestly say that the first four chapters of this book are the best at showing how to wire up a Rails application. New developers are often confused by all the various moving wheels. They have questions like: how can I call code from a URL? What is a route anyways? What goes in a controller? How do I make a new model? How do I make a form? These are all very basic questions that must be answered before anyone can start making real applications. Bigg answers all these questions and more. He starts by giving an overview of the Rails framework and covers these Rails principles:

Convention over Configuration
REST
MVC.

It’s very important to cover these ideas upfront so readers are well aware of recurring themes in the book. The author demonstrates how to create a very basic application. This is where the stage is set for creating the example application. All important steps and pieces are introduce before moving on to the next step. Ryan covers all these basic parts:

Installing
Using a generator to create an application
Starting
Scaffolding
Migration
Views
Validations
Routing

These small portions are very effective because it demonstrates how to do common tasks. All new developers should know how to do these things before continuing onto the next step. This brings me to why this is my favorite Rails book I’ve ever read. The book uses pure TDD to develop the demo applications.

The next chapter is “Testing Saves Your Bacon.” The author goes through a small example of using RSpec for unit testing. He describes BDD. BDD stands for “Behavior Driven Development.” It is essentially the same thing as TDD, except tests respresent documentation and are readable by stake holders. You can express this in code:

class TestDriveDevelopment

  def strategy
    write_a_test(requirements)
    while(test_failing)
      write_implementation(test)
      run_test_suite
    end
  end
end

class BehaviorDrivenDevelopment < TestDrivenDevelopment
end

The testing saves your bacon chapter is very short and suite. It’s only real purpose it to introduce Cucumber and RSpec. (More information on BDD here) Now that the reader knows the basics of using these test frameworks, they can move through an excerise in creating the scaffold for a real application.

Before, the reader only used a generated application to move through the basic motions. Now it’s time to set the stage for creating the application used throughout the book. I like this chapter because it covers things that are usually overlooked. The author mentions the book will use Git. I find it funny that this fact is given very little attention and is more annecdotal than anything else. Granted this is not a book on version control, but I think that spending more time on why developers should use version control would be beneficial–especially because some readers may have no version control experience.

The sample application is a ticket tracking app. It is a very slimmed down version of Lighthouse. Ryan describes the application in high level terms that can be translated to automated acceptance tests. The rest of this section contists of writing cucumber features and then their implentation. This process is informative for new users.

There is one downside to these early sections. A large number of new tools are introduced. This may be confusing for brand new Rails developers. I’ve run into this problem before when I teach Rails. I use the same technique: Teach Rails using TDD with Cucumber and RSpec. I’ve noticed that new developers think all these tools are Rails. This is not the case. The case is also complicated by the fact that Cucumber uses Capybara to drive a browser. The line between all the different tools blurs for new developers. This is not a problem with this book, but a problem with any type of learning that overloads people with new things at the beginning. I don’t think there is anyway to get around this. It is very important to introduce all these concepts and tools in the beginning. The book can spend more time clearly defining each role the tools plays. I’ve found that the link between Cucumber and Capybara difficult. The readers will have a good understanding all the tools by the time they finish the book, but I think it’s wise to put more time up front explaining each tool and why Rails developers use them.

Now, that I have off my chest, I can cover the best examples for learning how to do basic CRUD I’ve ever seen. Chapters 4 & 5 are magnificent peices of work. I wish I could give some my students these chapters because they cover creating/viewing/editing/deleting in TDD fashion so well. I think these chapters work so well for a few reason. Readers may have a basic idea through previous experience, but they generally are familar with creating a new record in a computer system. New Rails programmers do not know how to make this process work. They know what they want to do, but now how to do it. Since the author uses TDD, he writes the test case first. The test suite is run. This shows that system does not currently support that use case. At this point, the reader has a very clear idea of what they are adding to the system. Then the implementation is written to make the test case pass. This demonstrates precisly what code is required to do each use case. This is process is very helpful because the reader will know exactly what the did to complete the use case. It also demonstrates the power of having a test suite that covers the application so futher development does not break existing functionality. After finishing the reader finishes this chapter they will have good understanding of these concepts:

Implementing RESTful routes
Using routing helpers to link to different pages
How to render new views
How to use partials
How to connect routes to controllers
And Most Importantly: How to do it all test first.

These are the fundamental skills all Rails developers should have. I guartenee that after reading this chapter you will know how to do CRUD.

The next chapter “Nested Resources” covers associations and building tickets for associated projects. This is essentially the same as the previous chapter, except for creating models that must be associated with other models. The Ticketee application has a Project model. A Project has many Tickets. This chapter covers nested routes and the basic of handling models with associations. It covers all the CRUD options as well.

After the “Nested Resources” chapter all foundation is there for creating a more complicated functionality. Every application allows users to log in. Once you can determine that someone is logged in, then it makes sense to allows specific users to do specific things. This describes authentications vs authorization. Part 1’s final two chapters cover this topic in depth. And, god damn. These chapters are long.

The book uses Devise to handle user registration/sign in/sign out/confirmation. It is currently the go-to solution for authenticating users inside Rails application. It is a Rails 3 engine. This fact is important because it it referenced later in the “Engines” chapter. The chapter follow the same flow as the others: Write a test for a new feature. Implement the code to make it pass. Chapter 6 covers creating users and allowing them to login. Basic access control is added. Namespaced controllers are introducted in this chapter. This is a helpful concept because it helps keep controllers organized. Once the groundwork is set, Admin users are created and a basic permissions system it implemented. This chapter is very informative. I felt it was much longer than it needed to be. Chapter 6 seems to drag on forever, where all the previous chapters are snappy and too the point. Once I got to part about creating Admin users, I had to skip ahead to see how much is left. I think the second half of this chapter could be split into it’s own section. The author fits a ton of information into this chapter. It may be too much for some readers.

Chapter 7 is the final chapter in part 1. It covers creating a more complicated persmission system using CanCan. This chapter is well written and thankfully shorter than the previous one. The information it presents is precise and useful.

Part 1 is really quite good. I can say that first few chapters are some the best I’ve read on on these topics. At this point the reader has transitioned to Rails newb to somewhere just before the intermediate level. Granted, they cannot be consider true developers until they have experience under their belt, but Part 1 covers all the skills people should have before moving onto the advanced stuff. This is where the book takes a different approach. In the beginning of the book, you did everything yourself. You wrote all the code and wrote all the tests. The last two chapters of part 1 and sections of part 2 focus on leveraging other people’s code through gems.

Part 2: Putting on the Bling

The first chapter in Part 2 covers uploading files with Paperclip. I like this chapter because it covers adding a feature that is important to serious applications. I also like that the author choose to use popular gems because it reflects current practices. The chapter does a nice job of showing how to implement file uploading and retreiving correctly. There is one thing I do not like about this chapter. It uses paperclip. Carrierwave would’ve been a much better choice, but at the time the book was written, paperclip was the best choice. We all know things change fast in the tech world. It’s very hard when writing a book. Writing a book takes forever by today’s standards. Rails 3.1 was in development while this book was being written. Hard to stay current sometimes.

The next two chapters add two features: state tracking and tagging. There is nothing too fancy about these chapters. The book covers writing a state tracking feature from scratch. I was suprised to see this, but I think it’s good for the new developers to learn how to do this sort of thing. Tagging is very common too. It’s not hard to implement, but it’s generally a solved problem so there are plenty of gems. The book covers how to write your own tagging system. It does use the author’s searcher gem to implement simple searching. My guess is the gem was written specifically for the book. I think it was a good choice to write the tagging system by hand and use a gem to create simple search functionality. This doesn’t distract from the main point: creating functionality to manage tags.

Chapter 11 is all about email. It provides a well thoughout overview to implementating email functionality inside Ticketee. The reader will learn how to setup an observer (in app/observers where they should be). I’ll admit I got a bit happy when I saw the author add chosen not to put them in app/models. I mean, why would you? They’re not models? They’re not presisted, they are completely separate! I digress. I only have one negative comment to say about this chapter: The book specifically assumes you have a Gmail account. The author also uses the Gmail gem to fetch email from a gmail account. I think it would be better to use the built in Mail retreiver methods instead. This keeps the toolbox small and my be less confusing to new developers. The test cases also involve the network in tests. This should come with a big warning saying: HERE BE DRAGONS! It’s not good to introduce the network into the test suite because it can create random failtures unrelated to the code. What happens if the connection goes down? Test fails. Is it the code or the network? This should be mentioned.

When I first started to talk to Ryan about his book he mentioned that he was writing chapters on creating API’s and deployment. I got really excited when I heard this. Why? Simply because you don’t see these topics covered in many Rails books. You could write an entire book on only deploying Rails applications–and they have. You could also write an entire text on creating well crafted HTTP API’s. Luckily for readers, the author has compressed these very large topics down into manageable portions that you can learn from!

I think the API chapter is damn good. It’s simple and straight forward. I think it demonstrates how easy it is to write RESTful API’s. This is where Rails really shines through. It’s so easy to return objects in XML or JSON. The author users JSON of course, then easily shows you can serve back XML if you are crazy enough. He also covers how to rate limit it. The RSpec tests read very nicely. The chapter also shows how to create versioned API’s by using namespace. This chapter is very well put together overall and shows Rail’s real strenghts. The author left one thing out: The params parser. It would’ve been nice to create an example application showing how to use the API. The params parser allows developers to POST/GET/PUT/DELETE with JSON/XML as the body of the request. The params parser will parse the text and return the proper data structure. This is an execellent way to pass complicated parameters. This may be outside the scope of the book, but it covers so many cool things I felt this one was left out.

The deployment chapter is awesome! I think these chapters really make this book stand out because it gives a complete overview being a Rails developer. Here’s the general process the author goes through:

Setup a VM
Install RVM
Install Ruby
Create a deploy user
Setup authentication w/SSH
Install a database
Using github for deployment (setting deploy keys)
Using capistrano
Configuring a web server
Configuring passenger to serve the Rails application.

There’s not really too much to say about this chapter even though it’s well done. It provides a blueprint for setting up a very basic application server. It is by no means a guide on how to setup a true production grade box. It does show you take a barebones system and throw a Rails app on there. Every rails developer should know how to do this. I consider this a mandatory skill when I think about hiring people–which is why I’m so glad the book covers it. After reading this chapter you’ll know how to do this: Setup Ubuntu with RVM, PostgreSQL, Nginx, and Passenger. This chapter is a good reference for anyone who’s never done it before.

That’s the end of Part 2. Part 3 holds much promise. Personally, I would buy this book for part 3 since that has stuff I’m interested in. The first two sections do not really apply to me. This is where we get into the advanced stuff–and I like it.

Part 3: Leveling Up

If you already know Rails and want to learn more then this section is for you. It covers some cool stuff that’s covered in Rails books. This book is book about developing Rails applications and not an exhaustive reference of the framework. Now we get to see how we can start to make our application do much more cool things.

The first chapter in part 3 describes how to setup external authentication using Devise + OmniAuth (which is seriously awesome). This is a very short chapter because it doesn’t need to be any longer! That’s how easy it is to implement third party authentication. After this chapter you’ll know how to integrate GitHub and Twitter authentication. Not bad for a short chapter.

The next chapter is on “Basic Performance Enhancements”. I think this is the weakest chapter in the entire book. I have a lot of experience working with caching and there are technique that I hoped the author would present, but did not. The title is “Basic Performance Enhancements”. The techniques in this chapter are basic and do not represent performance minded approach. The chapter focuses around the three out of the four Rails caching strategies: Page, Action, and Fragment Caching. The fourth involves Rails.cache which is essentially manual cache operation, but very powerful since you can store anything you like. Ticketee is a very basic application. It needs to be for the purposes of the book. Since it’s simple, you can use very simple caching techniques. Expiring with sweepers is very easy to do in a simple application. However, it does not account for associated events. For example, say you have a comments controller and tickets controller. If a comment is for a ticket, you’d have to sweep the ticket from the comments controller. This is not the case in the book, but is the case with using action caching and cache sweepers. I would’ve liked to see more discussion on these matters.

Fragment caching is discussed breifely, but not in major detail. It does not into detail on creating different keys for different fragment, but relies on Rails to do the dirty work. This hides some power from the reader. Page caching is mentioned, but rarely application to any legitmate web application. You can read my take on Advanced Caching in Rails if you want to see what I think is important. This information is outside the scope of the book. I’ve included the link as example of how complicated it can become and to show why things were left out of the book.

The author descides not to use memcached. I think this is a very bad decision. He uses default which provides functionality not available to memcached. The vast majority of people use memcached in production. If they follow the code provided in this chapter it will not work in production systems with memcached. The author uses cache expiration based on a regular expression. This makes it easy to address problems that arise from using cache sweepers. However, it is only applicable to cache stores that can iterate over all keys. Memcached cannot do this. I think this is the chapters biggest flaw because it uses techniques that are not suitable to production applications.

There is good news. It seems the author is working on addressing these problems. I think he is adding a section on memcached and how to use Rails.cache. I was suprised to see that he did not cover using the cache manually. All the examples are focused around caching HTML or preventing web request from hitting the server. He does not cover using Rails.cache in the model layer for complicated queries for example. I think this bad omission and hopefully will be added before the second printing.

He discusses two other enhancments. He address the N+1 problem and uses database indexes. The N+1 problem is a very basic performance enhancment. I think it should’ve been covered earlier in the text because it is not a true performance technique in the sense of implementing caching throughout an application. It does help with performance, but I don’t think it needs it’s own section.

The book also considers pagination a performance enhancment. I don’t think this is a performance enhancment. Granted, it will increase performance by cutting down the number of objects returned, but it seems more of a usability thing than anything else. Pagination is very simple and should be covered earlier on in the text.

I can easily forget about the “Basic Performance Enhancements” chapter because the “Engines” chapter is straigh GODLIKE. I think this chapter is the best one in the book by good measure for a few resons:

It provides excellent documenation and guidelines for creating engines. This is hard to find at the time of this writing.
The chapter shows how to test and develop Rails engines correctly.
It demonstrates Capybara’s raw power as a browser driver.
It makes you think about componentizing your application.
It shows you to release it as a gem. How many other books have you seen do this? It’s this kind of thing that makes this book awesome.

The only downside is: it’s god damn long. There is a ton of information in this chapter. It doesn’t feel long like the Authentication & Authorization chapter. It just owns. If you want to learn about engines. Read this chapter. It’s that good.

The “Rack-Based Applications” chapter is nicely written. It opens the hood to some of the undercover magic going on between Rails and the webserver. It’s a relativly short chapter, but does a nice job introduction conepts and showing you how you can write the most basic rack application. It shows you can you mount Rack applications in Rails and most importantly introduces middleware. It would be nice to see a real application of middleware, but it’s hard to find one for a simple application. The book shows how you can inject a peice of middleware to jumble the links. Nothing serious really–just a proof concept. I’m not sure what else the book could add to this chapter.

Part 4: Wrapping Up

There are some interesting tidbits in the end, but nothing really worth mentioning. The meat the book is the first 10 chapters. Things after that become more complicated and therefore take more time to cover and increase the scope of text. For instance: Deployment and designing an API can be separate books. It’s very hard to do them justice in a small time frame. That being said, the first chapters are absolutely fantastic! I think the first chapters do a wonderful job of teaching people how to build basic rails application. This is by no means a guide to building advanced applications but it will teach noobies how to get going.

Who Should Buy This Book?

You should buy this book if you are a newbie with little to no knowledge on Rails. This book will teach you everything you need to know to help you start your journey. You will also become familar with some intermediate concepts along the way. You will also learn it all test first which will help you in the long run.

Finally Shoutout: To Ryan Bigg for letting my get in on the writing process and helping him make his book better.

You can buy the book here.

Dear God?! What is this Bug?

me@broadcastingadam.com (Adam Hawkins) — Fri, 28 Oct 2011 00:00:00 +0300

A few days ago I had to do a very trivial task. One of my coworkers has translated our Rails app into finnish. He sent me the fi.yml file for me to add the application. I thought this process would take maybe ~30 mins. Turns out it took me 5 hours. I’ll tell you why.

It All Started on Windows

I sent my coworker the existing yml file. I told him to replace the text with the finnish version. He had a hard time just working with the yml file so he made an excel spreadsheet so he could see the existing english text in context. Then he put the finnish version next to it. When he was done he wrote a script to take the finnish columns and create the yml file. Sounds reasonable. I knew there would be formatting mistakes related to tabs, spaces, :’s etc. So I figured I’d just have to clean up the file before adding it to the locales. After a few days he sent me the file. Then after a few weeks I had time to put it in–so I downloaded fi.yml from gmail and off I went.

Drop the file in config/locales and start the server. BOOM. Syntax error. Line #4. Ok, np, open up the file, figure there would be a missing : or something. Nothing jumps out. Stare at it for about 30 minutes. Hmmm…there’s got to be something going on here. Open the file up in textmate so I can see “invisibles.” Nothing seems out of place. I convert all tabs to spaces (I know YML parsers are bitchy when it comes to this). Run the server again. Syntax error line #4. Le fu. At this point I have no clue what it could be. I save the file in UTF8 (just to be sure) and convert all line breaks to unix format. Sill no luck. At this point I’m out of ideas. So I just deleted the first block of text and retyped it. Syntax error line 79! HUZZAH! Progress. Head over to line 79. There are odvious issues which I take care off. All in all it took little over an hour to clean up the 1,000+ line yml file.

The Server Starts Finally

Now I’m ready to see the wondeful finnish version of the application. I open up the settings page and switch the locale to Suomen. Refresh the page and voilla! It’s in English. Hmmm, this is prolly just a dumb thing I did like forgetting the before_filter to set the locale or forgetting to save the form. Ya know, something simple. Do the quick status check. Everything is in proper order. My locale is set to fi in the DB. The before_filter :set_locale is being hit. Everything on my end seems to be as it should be. Now I have to do the fun stuff which happens way to often on this project: debug framework code. It’s time to take a dive into I18n.translate which of course is modified by Rails for trickery.

Into the Rabbit Hole

At this point, I just want to find out if the right locale and key is being passed into I18n. After another bit of reading code (and learning about I18n fallbacks) and I see that :fi is being passed into the various translate functions. So at this point, I know these things.

My code to manage the locale is correct
My locale is set to :fi
The :fi locale is correctly being passed into I18n.

Now that I know this, I’m able to try to figure out why every single key is falling back to english. After some more code reading I look squarely at this method: (source taken from I18n code)

def lookup(locale, key, scope = [], options = {})
  init_translations unless initialized?
  keys = I18n.normalize_keys(locale, key, scope, options[:separator])

  keys.inject(translations) do |result, _key|
    _key = _key.to_sym
    return nil unless result.is_a?(Hash) && result.has_key?(_key)
    result = result[_key]
    result = resolve(locale, _key, result, options.merge(:scope => nil)) if result.is_a?(Symbol)
    result
  end
end

The underlying code is pretty simple. It loops over the translation keys like: [en, dashboard, subkey, key, key] to find the actual value in the translations hash. Ok, seems easy enough (recurring theme over the course of this task), throw a debugger in and see what’s happening.

So I put a debugger here:

def lookup(locale, key, scope = [], options = {})
  init_translations unless initialized?
  keys = I18n.normalize_keys(locale, key, scope, options[:separator])

  keys.inject(translations) do |result, _key|
    _key = _key.to_sym
    debugger # <---------- Debugger added
    return nil unless result.is_a?(Hash) && result.has_key?(_key)
    result = result[_key]
    result = resolve(locale, _key, result, options.merge(:scope => nil)) if result.is_a?(Symbol)
    result
  end
end

So I restart the server and go to page. My perfectly placed debugger hits and I get the nice rdb prompt. This is where my brain starts to question everything it knows about Ruby.

1 + 1 = 1

Now that I’m in my debugger I can see that locale == :fi => true. I want to know why the key fi.navigation.dashboard is returning english. So I step. and the method exists. Hmm. Apparently the translations hash does not have the :fi key. What follows is something straight out of the X-Files.

I quit the process and start over again. This time I don’t step but inspect what’s going on in memory. Here’s me in the debugger

(rdb:2) translations.keys
[:"en-us", :"de-ch", :en, :fi :"en-gb"]
(rdb:2) translations[:fi]
nil
(rdb:2) translations[:en]
{:invitation_mailer=>{:rejection_notification=>{:description=>"%{name} has reje...
(rdb:2)

Well this is looking very suspect. I’m thinking symbols are globally unique! A :fi anywhere in any ruby source file in the same process is equal to any other :fi in the same process. How can this possibly be! Well, perhaps translations isn’t a simple Hash but something like HashWithIndifferentAccess or other trickery. A check to translations.class returns Hash. At this point I’m absolutely fucking confused because translations[:fi] is nil but translations[:en] is correct. AND translations.keys has :fi in the damn thing. So I start running around the room bouncing off walls and other thing that don’t make any sense because for some reason all I know about Ruby symbols is wrong and that’s causing my brain to meltdown.

I start playing in the debugger more.

(rdb:2) translations[translations.keys.first]
# a ton of finnish
(rdb:2) translations.keys[:fi]
nil # wait wut.

Does. Not. Compute. Brain shutting down. More debugging:

(rdb:2) translations.keys
[:fi, :"en-gb", :en, :"en-us", :"de-ch"]
(rdb:2) translations.keys.first == :fi # HMMMM. Highly suspect <------------ WTF!
false
(rdb:2) translations.keys.first
:fi
(rdb:2) translations.keys[2] == :en
true
(rdb:2) translations[:en]
# a ton of english
(rdb:2) translations[:fi]
# nil
(rdb:2) translations[translations.keys.first]

GAH. I cannot handle this. There has got to be some completly sinister going on here. Something I’ve never heard about. Something that only exists in comp.lang.c. Something that is out side of releam. Something going in the C implementation. Just something fucking crazy.

This sort of bug induced comma has been going on for a few hours now. Nearing the end of my rope I try some more things in the debugger:

/Users/adam/.rvm/gems/ree-1.8.7-2011.03/gems/i18n-0.6.0/lib/i18n/backend/simple.rb:33
locale = locale.to_sym
(rdb:1) locale
"fi"
(rdb:1) locale == "fi"
false
(rdb:1) locale <=> "fi"
1
(rdb:1) locale.length
5
(rdb:1) "fi".length
2

HOLY CHRISTMAS. There is the sinister bit! The keys are actually different! This is completely masked by any call to puts or to_sym. Now I have to figure out why in god’s name is the key for finnish in the translations 5 characters. There is only one other place that can cause this problem: Where the YML files are parsed and put into the translations file. I track that down and enter the debugger:

(rdb:1) locale.bytes
#<Enumerable::Enumerator:0x10ba360b8>
(rdb:1) locale.bytes.map(&:to_s)
["239", "187", "191", "102", "105"]
(rdb:1) "fi".bytes.map(&:to_s)
["102", "105"]

Encodings, You’ve Done it to me Again!

Astute readers will notice that is there is a BOM in the key that’s used in the translations hash! So when I pass the string “fi” into I18n.translate of course it doesn’t have the BOM in it which essentially equates to I18n thining that there is no such thing as that locale. There are more sinister things at play here. I18n will call to_sym for all keys that are entered into the translations hash. However, ruby will not remove the BOM from the string when to_sym is called. When you inspect that symbol in the debugger it will show as :fi. When you call locale.to_s it will show "fi" so everything seems right on the surface. Underneath the covers it is horribly wrong. Now I have to figure out if the problem is with my files or somee other piece of code. Let’s get a hex dump and figure out for sure.

Here is the hex dump of the en.yml file.

cs181226081:crm adam$ od config/locales/en.yml 
0000000    067145  005072  020040  067554  060543  062554  035163  020012

Now we have the eternally lovely fi.yml

cs181226081:crm adam$ od -ax config/locales/fi.yml 
0000000    ?   ?   ?   f   i   :  nl  sp  sp   l   o   c   a   l   e   s
         bbef    66bf    3a69    200a    6c20    636f    6c61    7365

Dear god. There is a BOM at the start of the file. That was it?! Yes folks, that was the problem. There was a BOM at the start of my locale file. YAML (however it’s coded) consumes the bytes and turns them to a key for hash. Question: how come BOM are used to create keys to this hash? Answer: Because I’m using Ruby 1.8.7 and everything’s wrong!

People bitch about the YML parser on 1.9. I welcome it’s strictness. I don’t think this would’ve happend on 1.9. There was some interesting twist of fate in how Ruby 1.8.7 handles BOM’s, encodings, and YAML. I don’t remember exactly what it was but I know this: It was the perfect storm of everything going the exactly wrong direction to create the most annoying bug I’ve ever seen. I like to describe these sitations with this phrase: “a long and constant stream of unfortunate mistakes.”

Squashing the Bug

I don’t hold anything aganist anyone. This is most likely some odd edge case. I attribute this to the file coming from Windows, generated then edited in god knows what way. I attribute it to encoding conversions. There are a lot of possible ways this situation could happen. One thing is for sure: any YML library on any version of Ruby should not allow BOM markers in keys! This is crazy! I cannot think of any use case for this behavior.

After I finally got my head around what exactly what had happened I could move forward. I copied the text to the clipboard and deleted the existing file. I made a new file in the ever trustworthy VIM and pasted it in. :w, then ./script/server, and a refresh later: I see my application in Finnish. Jesus christ. That took my a little over 5 hours. By this time I was completely mentally spent. A few fixes and commits later I deployed a finnish version of the application–then I didn’t work on Rails for the rest of the day.

Moral of the story

Use Ruby 1.9
Don’t trust files from Windows
Turn invisibles on in your editor when editing YML files
Be sure to remove the BOM
Upgrade from Ruby 1.8.7

P.S. Here are the gists if you want to relive the horror.

Debugging Session Hex Dumps

Handling Legacy APIs

me@broadcastingadam.com (Adam Hawkins) — Wed, 19 Oct 2011 00:00:00 +0300

Well, it’s been a while since I’ve written a post to my blog. I’m on a plane back to Helsinki for some intense work. I’ve got some trance on so I figure I’ll scratch and itch and try to get something written.

There are few things poeple never mention when they are teaching rails to rookies, or hell, really even talk about in general. Rails gets a lot of play for how easy it is to write RESTful APIs. It deserves it. However, no one ever mentions what it actually means to write a web API. I haven’t see anyone talk about freezing input formats or output formats. The common approach is to just throw in respond_to :json then go on your merry way. It works, but it’s got some problems.

Dealing with Changes

Let’s first examine the most basic and widely adverised Rails RESTful API controlller.

class FoosController < ApplicationController
  respond_to :json

  def new
    respond_with Foo.create(params[:foo])
  end

  def show
    respond_with Foo.find params[:id]
  end

  def index
    respond_with Foo.all
  end

  def update
    foo = Foo.find params[:id]

    respond_with foo.update_attributes(params[:foo])
  end

  def destroy
    foo = Foo.find params[:id]

    respond_with foo.destroy
  end
end

Zomg! You have an API! Well, no. You’ve just defined a simple webservice that communicates with JSON. You have not met the two fundamental requirements for writing an API:

Input parameters stay the same forever for a given API version
Output formats stay the same forever for a given API version.

Let’s say you change an attribute on the Foo class. You’ll likely be smacked with a lof of UnknownAttribute errors. Then on the other end, there previous parameter will be renamed in the outputted JSON. Congratulations, you’ve just made an API that has Facebook consistency. As the developer, you need to take steps to ensure that you’re API stays to the standards you’ve written.

You have two ways to handle this problem. You can build logic into the underlying model to handle deprecated methods. You could code some logic into the controller to convert legacy input parameters. You could write a middleware that will handle some changes. There are a few ways to tackel this problem. I’ll share how I did it for Radium.

Handling Legacy API Input Parameters

A long time ago. I wrote a webservice for a Symbian (kill it with fire) application. The API was supposed to exist for maybe a few months for prototyping. Turns out it’s taken over 1.5 years to develop (and it’s not done yet) the application. In that time, my simple API interface to some common models has had to stay the same. You can imagine over the course of 1.5 years the underyling schema and model layer will undergo changes. For examples, one record use to a series to timestamps and a state machine to track it’s progress. Now there is a simple boolean flag. Attributes have been renamed. Classes have changed. The underlying data model has matured but the API hadn’t. Now I’m facing a problem with supporting legacy input parameters and legacy output formats. I was using the wonderful to_json method.

I needed to take different param hashes, change values, rename keys, and some various other massaging to fit current model. So I decided to write a class for each different type of API call. There are todos, contacts, and meetings. Here is a snippet of one of the classes:

module SymbianAPI
  class LegacyTodoConverter
    def self.convert(params)
      params[:finished] = true if params.delete(:finished_at)
      params[:description] = params.delete :task
      params[:finish_by] = params.delete :due_at

      # more stuff

      params
    end
  end
end

I ended up writing 3 of these type’s of classes. They massage the legacy params for the API request and update them to fit the current model.

I took a more complicated route implementing them. I wanted my controllers to look as vanilla as possible. I didn’t want to have to reference these classes in every request or write a before_filter. I just wanted the controller to know that params[:foo] is good to go. I dedecied to write a middleware that would automatically convert the legacy parameters based on the route. So if the route mataches /api/customers, then I would use my LegacyCustomerConvertor to merge in the new params. There is some other trickery going on here, but I figure I’d share the code for anyone who is interested.

class SymbianApiAdapter
  def initialize(app)
    @app = app
  end

  def call(env)
    if request_for_symbian_api?(env)
      @app.call convert_legacy_input(env)
    else
      @app.call env
    end
  end

  private 
  def request_for_symbian_api?(env)
    parts = env['PATH_INFO'].split('/')
    parts[1] == 'api' && parts[2] != 'v2'
  end

  def convert_legacy_input(env)
    # No fucking clue why we have to do this trickery for PUTS
    if env['REQUEST_METHOD'] == 'PUT'
      params = Rack::Request.new(env).POST
    else
      params = env['rack.request.form_hash']
    end

    if params.present? && params['xml'].present?
      adapter = case env['PATH_INFO']
                when '/api/todos/sync'
                  Api::Adapters::TodosAdapter
                when /todo/
                  Api::Adapters::TodoAdapter
                when '/api/meetings/sync'
                  Api::Adapters::MeetingsAdapter
                when /meeting/
                  Api::Adapters::MeetingAdapter
                when '/api/customers/sync'
                  Api::Adapters::CustomersAdapter
                when /customer/
                  Api::Adapters::CustomerAdapter
                end

      env['rack.request.form_hash'] || {}

      if adapter
        xml = params.delete 'xml'
        hash = Hash.from_xml(xml)

        if hash.values.first.is_a?(Hash)
          converted_params = {}
          converted_params[hash.keys.first] = adapter.convert(hash.values.first.with_indifferent_access)
          env['rack.request.form_hash'].merge! converted_params
        elsif hash.values.first.is_a?(Array)
          env['rack.request.form_hash'].merge! adapter.convert(hash)
        end
      end
    end

    env
  end
end

This keeps my controllers small since they don’t have to worry about handling the paramters. They are just correct when the request finally hits the controller. Now there is also a wall between what comes in from the request and what actually hits the models. This makes it much easier to ensure future support. All I need to do is update those convertor classes and things will continue. Now at this point I can write this controller action and never worry about the params.

class FoosController < ApplicationController
  respond_to :json

  def create
    reapond_with Foo.create(params[:foo])
  end
end

Handling Legacy API Output Formats

There has finally been some talk about this sort of thing. There needs to be a way to easily generate different blocks of JSON depending on what API version is in use–essentially different views. There are a few ways to do this. You could use a fancy new JSON builder like Rabl. I have not used Rabl. I investigated it, but I find that using builders in views is cumbersom when you need to write some code. (And code doesn’t belong the view anyway). I opted for an easier approach. (And given that this is currently a Rails 2 app, there are no other options). I wrote another three classes that take the record to be returned and generate an output hash. That hash can then be used for to_json or to_xml. Here’s how they work.

module Api
  class LegacyContactAdapter
    def self.convert(contact)
     {
      :town => contact.city, # API was specified to return a 'town' attribute
      :postcode => contact.zip_code # API specificed a 'postcode' attribute

      # so on and so forth. Build up the hash with the specified
      # attributes
    }
  end
end

Now, the controller can use that class to return the required JSON.

def create
  foo = Foo.create params[:foo]

  if foo.save
    respond_to do |wants|
      wants.json { render :json => Api::LegacyContactAdapter.convert(contact), :status => :created }
    end
  else
    respond_to do |wants|
      wants.json { render :json => contact.errors, :status => :unprocessable_entity }
    end
  end

Using that classes ensure that it’s easy(ier) to support the legacy API into the future because there is a wall in the code. Also, if the contact model ever changes, or the api needs a new output paramter, you can just throw it into the various Adapter classes.

Wrapping it Up

Writing API’s is serious business. They represent a contact between your system and external developers. You need to do everything in your power to ensure that you hold up your end of the baragin. You need to ensure that the input parameters are always accepted and that you stick to the given output format no matter what changes. I’ve showed you some different ways you can hold up your end of the contract. A middleware based solution may not work in every situation. I could’ve easily used a before_filter but I didn’t like that. The important thing is to build walls between the API and the other parts of your code that way it’s easier to ensure support in the future.

Present Yourself - Presenters in Rails

me@broadcastingadam.com (Adam Hawkins) — Fri, 10 Jun 2011 00:00:00 +0300

Presenters are one of those interesting things that you don’t hear about very much. They are mentioned, and then flutter in the wind. If you google “presenters rails” you’ll get some links from a few years ago and that’s about it. There is perhaps some useful information there. Maybe you find something on cells. That’s most likely not what you want. I’m taking about presenters (insert Alan Iverson practice voice). Presenters are object designed to encapsulate information required to create a view. They slim down controllers and encourage view reusability. This post describes how I started using presenters and why it worked.

The Backstory - That’s a God Damn Ton of Instance Variables

I work on a complex project. It’s not one of those cookie-cutter Rails apps that deals with practically generated code. There is some real business going on here. Views are complicated things and it takes a fair amount of information to render views for certain objects. Here is an example of what I’m talking about for one page.

The record itself.
All the associated notes
All the associated todos
All the associated extra fields (think EAV)
All the associated deals
All the associate activities (with fancy filtering. This alone is massive where each activity as it’s own forms and required stuff)
Statistics (3 different tables, 30 different statistics, custom ranges)
A new email
A new sms
A new meeting
A new deal
The list goes on

Everyone is used to seeing this:

def show
  @customer = Customer.find params[:id]
end

That’s all well in good for simple applications. What if you have this?

def show
  @customer = Customer.find params[:id]

  # insert 30 more lines of instantiation
  # and other trickery to get the view to render
end

That’s gonna get old real quick–especially if you have to do that for many different pages. The controller is becoming ove run run with logic only required for the view. All of that extra cruft is not related to the actual controller action of taking params and finding and object. It’s just noise. The solution is to move all that stuff into an object that knows how to present that specific view. Why do you want to to this? I think there are a few reasons.

Keep controllers small and stupid. They should be kept small.
Extract logic into a class where it’s easily testable
Encourage view/template reusability since a view requires an object not a random assortment of instance variables.
Keep views stupid since they depend on one object for everything.

Now many of my previous complex controller actions look like this:

def show
  customer = Customer.find params[:id]
  @presenter = CustomerPresenter.new customer, current_user
end

At this point, the presenter does all the required instantiation and other trickery that the view needs.

Looking at a Presenter

I created a common base class for all the presenters in my application. I call it ApplicationPresenter. Here’s the code: (Rails 2.3)

class ApplicationPresenter
  extend ActiveSupport::Memoizable
  include ActionController::UrlWriter
  include ActionController::RecordIdentifier

  def self.default_url_options
    ActionMailer::Base.default_url_options
  end

  private
  def t(*args)
    I18n.translate(*args)
  end
end

This code enables me to do a few things:

Memoize all methods so instantiation/querying only happens once
Use named route helpers & url_for/polymorphic_url etc in the presenter
User dom_id and things like that. I use dom_id a ton in this project.
Provide t in the presenters. This is mostly to prevent the views from figuring out how to find text themselves.

The application is very compontentized. Pages are composed of different components. Each component has their own presenter. A page presenter will provide an interface for getting a presenter for each component. That presenter is passed into the partial as a local variable. It works out pretty well. Here is an example view. Most of the views look like this:

-# This is the dashboard type view

- title @presenter.title

= render :partial => 'announcements/announcement', :locals => {:presenter => @presenter.announcement_presenter}

= render :partial => "activities/activities" , :locals => {:presenter => @presenter.activities_presenter}

= render_statistics @presenter.statistics_presenter

- content_for :sidebar do

  = render :partial => "todos/widget", :locals => { :presenter => @presenter.todo_widget_presenter }

  = render :partial => 'users/widget', :locals => { :presenter => @presenter.user_widget_presenter }

  = render :partial => 'customers/search_widget'

  = render :partial => 'companies/widget', :locals => {:presenter => @presenter.company_widget_presenter}

Now you know what a basic view looks like, here’s the code for that page’s presenter.

class DashboardPresenter < ApplicationPresenter
  def initialize(user)
    @user = user
  end

  def user_widget_presenter
    UserWidgetPresenter.new @user 
  end
  memoize :user_widget_presenter

  def company_widget_presenter
    CompanyWidgetPresenter.new @user
  end
  memoize :company_widget_presenter

  def todo_widget_presenter
    TodoWidgetPresenter.new @user, @user
  end
  memoize :todo_widget_presenter

  def announcement_presenter
    AnnouncementPresenter.new @user
  end
  memoize :announcement_presenter

  def activities_presenter
    DashboardActivitiesPresenter.new @user, @user
  end
  memoize :activities_presenter

  def statistics_presenter
    DashboardStatisticsPresenter.new @user
  end
  memoize :statistics_presenter

  def title
    t 'dashboard.page_title'
  end
  memoize :title
end

The main page presenters really don’t have much to them. The just create presenters for all the different components I want on that page. However, some of the individual presenters can get pretty hairy. I’ll share a simple one first:

class NotesPresenter < ApplicationPresenter
  def initialize(record)
    @record = record
  end

  def notes
    @record.notes.all(:include => :user)
  end
  memoize :notes

  def show_explanation?
    @record.notes.count == 0
  end
  memoize :show_explanation?

  def explanation
    t('explanations.notes')
  end
  memoize :explanation
end

Now for the hairy one:

class DealsPresenter < ApplicationPresenter

  PER_PAGE = 35

  def initialize(user, params)
    @user = user
    @params = params
  end

  def deals
    if @user.is_a?(Manager)
      bucket = account.deals
    else
      bucket = @user.deals
    end

    case filter
    when 'user'
      bucket = bucket.with_user(account.users.find(@params[:user_id]))
    when 'status_pending'
      bucket = bucket.pending
    when 'status_closed'
      bucket = bucket.closed
    when 'status_rejected'
      bucket = bucket.rejected
    when 'status_paid'
      bucket = bucket.paid
    when 'due_this_week'
      bucket = bucket.due_between(Time.zone.now.beginning_of_week..Time.zone.now.end_of_week)
    when 'due_this_month'
      bucket = bucket.due_between(Time.zone.now.beginning_of_month..Time.zone.now.end_of_month)
    when 'due_overdue'
      bucket = bucket.overdue
    else
      bucket
    end

    bucket.paginate :order => "#{ordered_column} #{sort_direction}",
                    :include => [{:customer => :company}, :user],
                    :page => @params[:page],
                    :per_page => PER_PAGE
  end
  memoize :deals

  def filters_presenter
    presenter = FiltersPresenter.new

    presenter.filter I18n.translate('deals.filters.all_deals'), deals_path, :class => (filter == 'all' ? 'selected' : 'unselected')

    text = case filter
           when 'user'
             I18n.translate('deals.filters.filtered_by_user', :user => account.users.find(@params[:user_id]))
           else
             I18n.translate('deals.filters.by_user')
           end

    if @user.is_a?(Manager)
      presenter.dropdown text, :class => (filter == 'user' ? 'selected' : 'unselected') do |drop_down|
        account.users.alphabetical.except(@user).each do |user|
          drop_down.filter user, deals_path(:filter => :user, :user_id => user.id)
        end
      end
    end

    text = case filter
           when 'status_pending'
             I18n.translate('deals.filters.status_pending')
           when 'status_closed'
             I18n.translate('deals.filters.status_closed')
           when 'status_paid'
             I18n.translate('deals.filters.status_paid')
           when 'status_rejected'
             I18n.translate('deals.filters.status_rejected')
           else
             I18n.translate('deals.filters.status')
           end

    presenter.dropdown text, :class => (filter =~ /status/ ? 'selected' : 'unselected') do |drop_down|
      %w(pending closed paid rejected).each do |status|
        drop_down.filter I18n.translate("deals.states.#{status}"), deals_path(:filter => "status_#{status}") if filter != status
      end
    end

    presenter
  end
  memoize :filters_presenter

  def deal
    Deal.new
  end
  memoize :deal

  def sortable_options
    @params.slice(:filter, :user_id)
  end
  memoize :sortable_options

  def sort_column
    %w(user customer company amount due_on status).include?(@params[:sort]) ? @params[:sort] : 'user'
  end
  memoize :sort_column

  def sort_direction
    @params[:direction] == 'desc' ? 'desc' : 'asc'
  end
  memoize :sort_direction

  def ordered_column
    case sort_column
    when 'user'
      'users.name'
    when 'customer'
      'customers.name'
    when 'company'
      'companies.name'
    when 'amount'
      'deals.value'
    when 'status'
      'deals.state'
    when 'due_on'
      'deals.due_by'
    end
  end
  memoize :ordered_column

  def title
    I18n.translate 'plurals.deals'
  end
  memoize :title

  def statistics_presenter

  end
  memoize :statistics_presenter

  private
  def filter
    %w(user status_pending status_closed status_rejected status_paid
      due_this_week due_this_month due_overdue all).include?(@params[:filter]) ? @params[:filter] : 'all'
  end
  memoize :filter

  def account
    @user.account
  end
  memoize :account
end

Note: this particular presenter is waiting to be refactored. But it does give you an idea of some of the logic that I removed from the controlller. It also testifies to the logic required to construct a view and why it’s nice to remove it from the controller.

Testing Presenters

I like moving logic out of the controllers because testing controllers is such a pain in the ass. (I stopped doing it completely actually). Once your controller starts to do some real work, whatever mocks/stubs you had in place become too cumbersome to maintain. Sometimes I simply want to test that a new instance variable is created. Using my presenter, I could write a test like this:

class NotesPresenter < ApplicationPresenter
  def note
    Note.new
  end
  memoize :note
end

describe NotesPresenter do
  it "should provide a new note for a form" do
    subject.note.should be_new_record
    subject.note.should be_a(Note)
  end
end

Good luck doing that in a controller action with more complex logic. It’s very easy to test in an isolate class.

Most of my test cases don’t do heavy assertions, but verify that a specific interface is implemented. Most of the logic inside the method is trivial enough to ignore writing a test case. Instead, I use rspec’s it_should_behave_like to specify the presenter provides a certain interface. Here is the test for the previously mentioned DashboardPresenter:

require 'spec_helper'

describe DashboardPresenter do
  def mock_user(stubs = {})
    @mock_user ||= mock_model(User, stubs)
  end

  subject { DashboardPresenter.new(mock_user) }

  it_should_behave_like "a presenter with activities"

  it_should_behave_like "a presenter with todos"

  it_should_behave_like "a presenter with stats"

  it_should_behave_like "a presenter with companies"

  it_should_behave_like "a presenter with a page title"

  it { should respond_to(:user_widget_presenter) }

  it { should respond_to(:announcement_presenter) }
end

Now for a component presenter:

require 'spec_helper'

describe NotesPresenter do
  fixtures :customers

  subject { NotesPresenter.new customers(:teemu) }

  it_should_behave_like "a presenter with an explanation"

  it { should respond_to(:notes) }
end

Closing Thoughts

I’m very happy I did this. It makes my view layer much easier to maintain. It also makes my controllers easy to maintain because of how simple they are. It also gives me a common object I can pass off to a view if I need to render it. This happens to me a lot actually. There are ajax forms that hit different controllers on one page. For example, if you are on ‘companies/1’ there is a form to add a todo. Naturally this goes to TodosController. Now the UI for the company page has to be updated from the TodosController in a js.erb template. I can simply instantiate the todos component presenter has use that to rerender the partial. I don’t have to know anything else–the presenter does all the work for me.

You can learn more about presenters in this course. You can also learn more about presenters by reading Martin Flower’s papers.

Advanced Caching in Rails

me@broadcastingadam.com (Adam Hawkins) — Fri, 06 May 2011 00:00:00 +0300

Caching in Rails is covered occasionally. It is covered in very basic detail in the caching guide. Advanced caching is left to reader. Here’s where I come in. I recently read part of Ryan Bigg’s Rails 3 in Action upcoming Rails book (review in the works) where he covers caching. He does a wonderful job of giving the reader the basic sense of how you can use page, action, and fragment caching. The examples only work well in a simple application like he’s developing in the book. I’m going to show you how you can level up your caching with some new approaches.

Different Caching Layers

First, let’s start with a brief overview of the different types of caching:

1. Page Caching: PRAISE THE GODS if you actually can use page caching in your application. Page caching is the holy grail. Save the entire thing. Don’t hit the stack & give some prerendered stuff back. Great for worthless applications without authentication and other highly dynamic aspects.

2. Action Caching: Essentially the same as page caching, except all the before filters are run allowing you to check authentication and other stuff that may have prevented the request for rendering.

3. Fragment Caching: Store parts of views in the cache. Usually for caching partials or large bits of HTML that are independent from other parts. IE, a list of top stories or something like that.

4. Rails.cache: All cached content except cached pages are stored in the Rails.cache. Cached pages are stored as HTML on disk. We’ll use the fact that all the cached action and fragment content are simply stored in Rails.cache. You can cache arbitrary content in the Rails cache. You may cache a large complicated query that you don’t want to wait to reinstantiate a ton of AR::Base objects.

Under the Hood

All the caching layers are built on top of the next one. Page caching is the only exception because it does not use Rails.cache it writes content to disk. The cache is essentially a key-value store. Different things can be persisted. Strings are most common (for HTML fragments). More complicated objects can be persisted as well. Let’s go through some examples of manually using the cache to store things. I am using memcached with dalli for all these examples. Any driver that implements the cache store pattern should work.

# Rails.cache.write takes two value: key and a value
> Rails.cache.write 'foo', 'bar'
=> true

# We can read an object back with read
> Rails.cache.read 'foo'
=> "bar"

# We can store a complicated object as well
> hash = {:this => {:is => 'a hash'}}
> Rails.cache.write 'complicated-object', object
> Rails.cache.read 'complicated-object'
=> {:this=>{:is=>"a hash"}}

# If we want something that doesn't exist, we get nil
> Rails.cache.read 'we-havent-cached-this-yet'
=> nil

# "Fetch" is the most common pattern. You give it a key and a block
# to execute to store if the cache misses. The block is not executed
# if there is a cache hit.
> Rails.cache.fetch 'huge-array' do
    huge_array = Array.new
    1000000.times { |i| huge_array << i }
    huge_array # retrun value is stored in cache
  end
=> [huge array] # took some time to generate
> Rails.cache.read 'huge-array'
=> [huge array] # but returned instantly

# You can also delete everything from the cache
> Rails.cache.clear 
=> [true]

Those are the basics of interacting withe the Rails cache. The rails cache is a wrapper around whatever functionality is provided by the underlying storage system. Now we are ready to move up a layer.

Understanding Fragment Caching

Fragment caching is taking rendered HTML fragments and storing them in the cache. Rails provides a cache view helper for this. It’s most basic form takes no arguments besides a block. Whatever is rendered during the block will be written back to the cache. The basic principle behind fragment caching is that it takes much less time fetch pre-rendered HTML from the cache, then it takes to generate a fresh copy. This is very true. If you haven’t noticed, view generation can be very costly. Let’s say you have generated a basic scaffold for a post:

$ rails g scaffold post title:string content:text author:string
# that will generate some views to play with

Let’s start with the most common use case: caching information specific to one thing. IE: One post. Here is a show view:

<!-- nothing fancy going on here -->
<p>
  <b>Title:</b>
  <%= @post.title %>
</p>

<p>
  <b>Content:</b>
  <%= @post.content %>
</p>

<p>
  <b>Author:</b>
  <%= @post.author %>
</p>

Let’s say we wanted to cache fragment. Simple wrap it in cache and Rails will do it.

<%= cache "post-#{@post.id}" do %>
  <p>
    <b>Title:</b>
    <%= @post.title %>
  </p>

  <p>
    <b>Content:</b>
    <%= @post.content %>
  </p>

  <p>
    <b>Author:</b>
    <%= @post.author %>
  </p>
<% end %>

The first argument is the key for this fragment. The rendered HTML is stored with this key: views/posts-1. Wait what? Where did that ‘views’ come from? The cache view helper automatically prepends ‘view’ to all keys. This is important later. When you first load the page you’ll see this in the log:

Exist fragment? views/post-2 (1.6ms)
Write fragment views/post-2 (0.9ms)

You can see the key and the operations. Rails is checking to see if the specific key exists. It will fetch it or write it. In this case, it has not been stored so it is written. When you reload the page, you’ll see a cache hit:

Exist fragment? views/post-2 (0.6ms)
Read fragment views/post-2 (0.0ms)

There we go. We got HTML from the cache instead of rendering it. Look at the response times for the two requests:

Completed 200 OK in 17ms (Views: 11.6ms | ActiveRecord: 0.1ms)
Completed 200 OK in 16ms (Views: 9.7ms | ActiveRecord: 0.1ms)

Very small differences in this case. 2ms different in view generation. This is a very simple example, but it can make a world of difference in more complicated situations.

You are probably asking the question: “What happens when the post changes?” This is an excellent question! What well if the post changes, the cached content will not be correct. It is up to us to remove stuff from the cache or figure out a way to get new content from the cache. Let’s assume that our blog posts now have comments. What happens when a comment is created? How can handle this?

This is a very simple problem. What if we could figured out a solution to this problem: How can we create a cache miss when the associated object changes? We’ve already demonstrated how we can explicitly set a cache key. What if we made a key that’s dependent on the time the object was last updated? We can create a key composed of the record’s ID and it’s updated_at timestamp! This way the cache key will change as the content changes and we will not have to expire things manually. (We’ll come back to sweepers later). Let’s change our cache key to this:

<% cache "post-#{@post.id}", @post.updated_at.to_i do %>

Now we can see we have a new cache key that’s dependent on the objects timestamps. Check out the rails log:

Exist fragment? views/post-2/1304291241 (0.5ms)
Write fragment views/post-2/1304291241 (0.4ms)

Cool! Now let’s make it so creating a comment updates the post’s timestamp:

class Comment < ActiveRecord::Base
  belongs_to :post, :touch => true
end

Now all comments will touch the post and change the updated_at time stamp. You can see this in action by touch‘ing a post.

Post.find(1).touch

Exist fragment? views/post-2/1304292445 (0.4ms)
Write fragment views/post-2/1304292445 (0.4ms)

This concept is known as: auto expiring cache keys. You create a composite key with the normal key and a time stamp. This will create some memory build up as objects are updated and no longer create cache hits. For example. You have that fragment. It is cached. Then someone updates the post. You now have two versions of the fragment cached. If there are 10 updates, then there are 10 different versions. Luckily for you, this is not a problem for memcached! Memcached uses a LRU replacement policy. LRU stands for Least Recently Used. That means the key that hasn’t been request in the longest time will be replaced with new content needs to be stored. For example, assume your cache can only hold 10 posts. The next update will create a new key and hence new content. Version 0 will be deleted and version 11 will be stored in the cache. The total amount of memory is cycled between things that are requested. There are two things to consider in this approach. 1: You will not be able to ensure that content is kept in the cache as long as possible. 2. You will never have to worry about expiring things manually as long as timestamps are updated in the model layer. I’ve found it is orders of magnitude easier to add a few :touch => true’s to my relationships than it is to maintain sweepers. More on sweepers later. We must continue exploring cache keys.

Rails uses auto-expiring cache keys by default. The problem is they are not mentioned at all the documentation or in the guides. There is one very handy method: ActiveRecord::Base.cache_key. This will generate a key like this: posts/2-20110501232725. This is the exact same thing we did ourselves. This method is very important because depending on what type of arguments you pass into the cache method it will be called on them. For the time being, this code is functionally equal to our previous examples.

<%= cache @post do %>

The cache helper takes different forms for arguments. Here are some examples:

cache 'explicit-key'      # views/explicit-key
cache @post               # views/posts/2-1283479827349
cache [@post, 'sidebar']  # views/posts/2-2348719328478/sidebar
cache [@post, @comment]   # views/posts/2-2384193284878/comments/1-2384971487
cache :hash => :of_things # views/localhost:3000/posts/2?hash_of_things

If an Array is the first arguments, Rails will use cache key expansion to generate a string key. This means calling doing logic on each object then joining each result together with a ’/’. Essentially, if the object responds to cache_key, it will use that. Else it will do various things. Here’s the source for expand_cache_key:

def self.expand_cache_key(key, namespace = nil)
  expanded_cache_key = namespace ? "#{namespace}/" : ""

  prefix = ENV["RAILS_CACHE_ID"] || ENV["RAILS_APP_VERSION"]
  if prefix
    expanded_cache_key << "#{prefix}/"
  end

  expanded_cache_key <<
    if key.respond_to?(:cache_key)
      key.cache_key
    elsif key.is_a?(Array)
      if key.size > 1
        key.collect { |element| expand_cache_key(element) }.to_param
      else
        key.first.to_param
      end
    elsif key
      key.to_param
    end.to_s

  expanded_cache_key
end

This is where all the magic happens. Our simple fragment caching example could easily be converted into an idea like this: The post hasn’t changed, so cache the entire result of /posts/1. You can do with this action caching or page caching.

Moving on to Action Caching

Action caching is an around filter for specific controller actions. It is different from page caching since before filters are run and may prevent access to certain pages. For example, you only want to cache if the user is logged in. If the user is not logged in they should be redirect to the log in page. This is different than page caching. Page caching bypasses the rails stack completely. Most web applications for legitimate complexity cannot use page caching. Action caching is the next logical step for most web applications. Let’s break the idea down: If the post hasn’t changed, return the entire cached page as the HTTP response, else render the show view, cache it, and return that as the HTTP response. Or in code:

Rails.cache.fetch 'views/localhost:3000/posts/1' do
  @post = Post.find params[:id]
  render :show
end

Declaring action caching is easy. Here’s how you can cache the show action:

class PostsController < ApplicationController

  caches_action :show

  def show
    # do stuff
  end
end

Now refresh the page and look at what’s been cached.

Started GET "/posts/2" for 127.0.0.1 at 2011-05-01 16:54:43 -0700
  Processing by PostsController#show as HTML
  Parameters: {"id"=>"2"}
Read fragment views/localhost:3000/posts/2 (0.5ms)
Rendered posts/show.html.erb within layouts/application (6.1ms)
Write fragment views/localhost:3000/posts/2 (0.5ms)
Completed 200 OK in 16ms (Views: 8.6ms | ActiveRecord: 0.1ms)

Now that the show action for post #2 is cached, refresh the page and see what happens.

Started GET "/posts/2" for 127.0.0.1 at 2011-05-01 16:55:27 -0700
  Processing by PostsController#show as HTML
  Parameters: {"id"=>"2"}
Read fragment views/localhost:3000/posts/2 (0.6ms)
Completed 200 OK in 1ms

Damn. 16ms vs 1ms. You can see the difference! You can also see Rails reading that cache key. The cache key is generated off the url with action caching. Action caching is a combination of a before and around filter. The around filter is used to capture the output and the before filter is used to check to see if it’s been cached. It works like this:

Execute before filter to check to see if cache key exists?
Key exists? - Read from cache and return HTTP Response. This triggers a render and prevents any further code from being executed.
No key? - Call all controller and view code. Cache output using Rails.cache and return HTTP response.

Now you are probably asking the same question as before: “What do we do when the post changes?” We do the same thing as before: we create a composite key with a string and a time stamp. The question now is, how do we generate a special key using action caching?

Action caching generates a key from the current url. You can pass extra options using the :cache_path option. Whatever is in this value is passed into url_for using the current parameters. Remember in the view cache key examples what happened when we passed in a hash? We got a much different key than before:

views/localhost:3000/posts/2?hash_of_things

Rails generated a URL based key instead of the standard views key. This is because you may different servers and things like that. This ensures that each server has it’s own cache key. IE, server one does not collide with server 2. We could generate our own url for this resource by doing something like this:

url_for(@post, :tag => @post.updated_at.to_i)

This will generate this url:

http://localhost:3000/posts/1?tag=234897123978

Notice the ‘?tag=23481329847’. Look familiar from anywhere? Rails uses this method to tag GET urls for static assets. That way the browser does not send a new HTTP request when it sees ‘application.css?1234’ since it is caching it. We can use this strategy to with action caching as well.

caches_action :show, :cache_path => proc { |c|
  # c is the instance of the controller. Since action caching
  # is declared at the class level, we don't have access to instance
  # variables. If cache_path is a proc, it will be evaluated in the
  # the context of the current controller. This is the same idea
  # as validations with the :if and :unless options
  #
  # Remember, what is returned from this block will be passed in as
  # extra parameters to the url_for method.
  post = Post.find c.params[:id]
  {:tag => post.updated_at.to_i}
end

This calls url_for with the parameters already assigned by it through the router and whatever is returned by the block. Now if you refresh the page, you’ll have this:

Started GET "/posts/2" for 127.0.0.1 at 2011-05-01 17:11:22 -0700
  Processing by PostsController#show as HTML
  Parameters: {"id"=>"2"}
Read fragment views/localhost:3000/posts/2?tag=1304292445 (0.5ms)
Rendered posts/show.html.erb within layouts/application (1.7ms)
Write fragment views/localhost:3000/posts/2?tag=1304292445 (0.5ms)
Completed 200 OK in 16ms (Views: 4.4ms | ActiveRecord: 0.1ms)

And volia! Now we have an expiring cache key for our post! Let’s dig a little deeper. We know the key. Let’s look into the cache and see what it actually is! You can see the key from the log. Look it up in the cache.

> Rails.cache.read 'views/localhost:3000/posts/2?tag=1304292445'
=> "<!DOCTYPE html>\n<html>\n<head>....."

It’s just a straight HTML string. Easy to use and return as the body. This method works well for singular resources. How can we handle the index action? I’ve created 10,000 posts. It takes a good amount of time to render that page on my computer. It takes over 10 seconds. The question is, how can we cache this? We could use the most recently updated post for the time stamp. That way, when one post is updated, it will move to the top and create a new cache key. Here is the code without any action caching:

Started GET "/posts" for 127.0.0.1 at 2011-05-01 17:18:11 -0700
  Processing by PostsController#index as HTML
  Post Load (54.1ms)  SELECT "posts".* FROM "posts" ORDER BY updated_at DESC LIMIT 1
Dalli::Server#connect localhost:11212
Read fragment views/localhost:3000/posts?tag=1304292445 (1.5ms)
Rendered posts/index.html.erb within layouts/application (9532.3ms)
Write fragment views/localhost:3000/posts?tag=1304292445 (36.7ms)
Completed 200 OK in 10088ms (Views: 9535.6ms | ActiveRecord: 276.2ms)

Now with action caching:

Started GET "/posts" for 127.0.0.1 at 2011-05-01 17:20:47 -0700
  Processing by PostsController#index as HTML
Read fragment views/localhost:3000/posts?tag=1304295632 (1.0ms)
Completed 200 OK in 11ms

Here’s the code for action caching:

caches_action :index, :cache_path => proc {|c|
  post = Post.order('updated_at DESC').limit(1).first
  {:tag => post.updated_at.to_i}
}

These are simple examples designed to show you who can create auto expiring keys for different situations. At this point we have not add to expire any thing ourselves! The keys have done it all for us. However, there are some times when you want more precise control over how things exist in the cache. Enter Sweepers.

Sweepers

Sweepers are HTTP request dependent observers. They are loaded into controllers and observer models the same way standard observers do. However there is one very important different. They are only used through HTTP requests. This means if you have things being created outside the context of HTTP requests sweepers will do you know good. For example, say you have a background process running that syncs with an external system. Creating a new model will not make it to any sweeper. So, if you have anything cached. It is up to you to expire it. Everything I’ve demonstrated so far can be done with sweepers.

Each cache_* method has an opposite expire_* method. Here’s the mapping:

caches_page , expire_page
caches_action , expire_action
cache , expire_fragment

Their arguments work the same with using cache_key_expansion to find a key to read or delete. Depending on the complexity of your application, it may be very to use sweepers or it may be impossible. Our simple examples can use sweepers easily. We only need to tie into the save event. For example, when a update or delete happens we need to expire the cache for that specific post. When a create, update, or delete happens we need to expire the index action. Here’s what a the sweeper would look like:

class PostSweeper < ActionController::Caching::Sweeper
  observe Post

  def after_create(post)
    expire_action :index
    expire_action :show, :id => post
    # this is the same as the previous line
    expire_action :controller => :posts, :action => :show, :id => @post.id
  end
end

# then in the controller, load the sweeper
class PostsController < ApplicationController
  cache_sweeper :post_sweeper
end

I will not go into much depth on sweepers because they are the only thing covered in the rails caching guide. The work, but I feel they are clumsy for complex applications. Let’s say you have comments for posts. What do you do when a comment is created for a post? Well, you have to either create a comment sweeper or load the post sweeper into the comments controller. You can do either. However, depending on the complexity of your model layer, it may quickly infeasible to do cache expiration with sweepers. For example, let say you have a Customer. A customer has 15 different types of associated things. Do you want to put the sweeper into 15 different controllers? You can, but you may forget to at some point.

The real problem with sweepers is that they cannot be used once your application works outside of HTTP requests. They can also be clumsy. I personally feel it’s much easier to create auto expiring cache keys and only uses sweepers when I want to tie into very specific events.

Now you should have a good grasp on how the Rails caching methods work. We’ve covered how fragment caching uses the current view to generate a cache key. We introduced the concept of auto expiring cache keys using ActiveRecord#cache_key to automatically expire cached content. We introduced action caching and how it uses url_for to generate a cache key. Then we covered how you can pass things into url_for to generate a time stamped key to expire actions automatically. We’ve skipped page caching because it’s not applicable to many Rails applications. Now that we understand how caching works we can address shortcomings in the system.

Moving Away from the HTTP Request

Now we’re going to write some code to address problems in the Rails caching system. We know that action caching is dependent on URLS. Fragment caching is dependent on the view being rendered. However, we know that both of these methods use Rails.cache under the covers to store content. We can use Rails.cache any where in our code. Unlike caches_path, caches_action and cache that will no hit the cache if perform_caching is set to false, the Rails.cache methods will always execute against the cache. Ideally, it would be nice to create a simple observer for our models. What it would be cool if we had a class like this:

class Cache 
  def self.expire_page(*args)
    # do stuff
  end

  def self.expire_action(*args)
    # do stuff
  end

  def self.expire_fragment(*args)
    # do stuff
  end
end

Then we can use that utility class anywhere in our code to expire different things we have cached. First, we need to be able to generate URL’s from something other than a controller. You may be familiar with this problem. Mailers are not controllers, but you can still generate URL’s. You need a host name to generate paths. The controller have this information because they accept HTTP requests which have that information. Mailer do not. That’s why the host name must be configured in the different environments. We can create a frankenstein class that takes parts of ActionMailer to generate URLS. Once we can generate URL’s we can expire pages and actions. URL generation is included this module Rails.application.routes.url_helpers. That’s a shortcut method for the generated module which contains url_for, path_for and all the named route helpers. We also need a class level variable for the host name. Here’s what we can do so far:

class Cache
  include Rails.application.routes.url_helpers # for url generation

  def self.default_url_options
    ActionMailer::Base.default_url_options
  end

  def expire_action(*args)
    # do stuff
  end

  def expire_fragment(*args)
    # do stuff
  end
end

Now we can pull in some knowledge on how the cache system works to fill in the gaps. Some of this comes from reading the various source files and observation in generating the cache keys. Here is the complete class:

# will not work in Rails 2 -- Rails 3 only!
class Cache
  include Rails.application.routes.url_helpers # for url generation

  def self.default_url_options
    ActionMailer::Base.default_url_options
  end

  def expire_action(key, options = {})
    expire(key, options)
  end

  def expire_fragment(key, options={})
    expire(key, options)
  end

  private
  def caching_enabled?
    return ActionController::Base.perform_caching
  end

  def expire(key, options = {})
    return unless caching_enabled?
    Rails.cache.delete expand_cache_key(key), options
  end

  def expand_cache_key(key)
    # if the key is a hash, then use url for
    # else use expand_cache_key like fragment caching
    to_expand = key.is_a?(Hash) ? url_for(key).split('://').last : key
    ActiveSupport::Cache.expand_cache_key to_expand, :views
  end
end

Since action and fragment caching all use Rails.cache under the hood, we can simply generate the keys ourselves and remove them manually–all without the fuss of HTTP Requests. Now you can create an initializer to define a method on your application namespace so it’s globally accessible. I like this way because it’s easy to reference in any piece of code.

# config/initializers/cache.rb
require 'cache'

module App # whatever you application module is
  class << self
    def cache
      @cache ||= Cache.new
    end

    def expire_fragment(*args)
      cache.expire_fragment(*args)
    end

    def expire_action(*args)
      cache.expire_fragment(*args)
    end
  end
end

Now we can merrily go about our business expiring cached content from anywhere. Here are some examples:

App.cache # reference to a Cache instance

App.expire_fragment @post
App.expire_fragment [@post, 'sidebar']
App.expire_fragment 'explicit-key'

# in a controller
App.expire_fragment post_url(@post)
# Have to pass in the hash since it's most likely
# that you won't have access to the url helpers
# in whatever scope your're in.
App.expire_action :action => :show, :controller => :posts, :id => @post, :tag => @post.updated_at.to_i

The expire_fragment and expire_action methods work just like the ones described in the Rails guides. Only difference is, you can use them anywhere. Now we can easily call this code in an observer. The observer events will fire every time they happen anywhere in the codebase. Here’s an example. I am assuming a todo is created outside an HTTP request through a background process. The observer will capture the event.

class TodoObserver < ActivRecord::Observer
  def after_create
    App.expire_fragment :controller => :todos, :action => :index
  end
end

The beauty here is that we can use this code anywhere. If you have more complicated cache expirations you may have to use a background job. This may not be acceptable because of processing time, but in some situations you can afford a sweeping delay if the sweeping process takes a long time. You could easily use this code with DelayedJob or Resque if needed. After all, the generated rails code does reference a cache observer–now you know how to write one.

Tagged Based Caching

This is an approach I came up with to work in this situation:

Maintain control over how long things are cached
Large number of different associations. Actions or fragments no longer related to a specific resource.
Content could be invalidated through HTTP requests or any number of background process.
Hard to maintain specific keys. I thought of it as “resources”.

There is a ton of cached content in the system. Many different actions and fragments. There was also a cache hierarchy. Expiring a specific fragment would have to expire an action (so a cache miss would occur when a page was requested thus, causing the new fragment to be displayed) while other things on pages are still cached. One question to ask, is how can I expire groups of things based on certain events? Well, first you need a way to associate different keys. Once you can associate different keys, then you can expire them together. Since you’re tracking the keys being sent to Rails.cache, you can simply use Rails.cache to delete them. All of this is possible through one itty-bitty detail of the Rails caching system.

You may have noticed something in the Cache class in the previous section. There is a second argument for options. Anything in the option argument is passed to the cache store. This is where can tie in the grouping logic. Also, since action and fragment caching use the same mechanism to write to the cache, we simply have to override the write_fragment method to add our tagging logic.

Through all of this trickery, you’ll be able to express this type of statement:

App.cache.expire_tag 'stats' 
App.cache.expire_tag @account

The content could from anywhere, but all you know is that’s stale.

This is exactly where Cashier comes in. It is (my gem) that allows you associate actions and fragments with one or more tags, then expire based of tags. Of course you can expire the cache from anywhere in your code. Here are some examples:

caches_action :stats, :tag => proc {|c|
  "account-#{Account.find(c.params[:id]).id}"
}

caches_action :show, :tag => 'account'
caches_cation :show, :tag => %w(account customer)

<%= cache @post, :tag => 'customer' do %>

Then you can expire like this:

Cashier.expire 'account' # wipe all keys tagged 'account'

All this is possible through this module:

module Cashier
  module ControllerHelper
    def self.included(klass)
      klass.class_eval do
        def write_fragment_with_tagged_key(key, content, options = nil)
          if options && options[:tag] && Cashier.perform_caching? 
            tags = case options[:tag].class.to_s
                   when 'Proc', 'Lambda'
                     options[:tag].call(self)
                   else 
                     options[:tag]
                   end
            Cashier.store_fragment fragment_cache_key(key), *tags
          end
          write_fragment_without_tagged_key(key, content, options)
        end
        alias_method_chain :write_fragment, :tagged_key
      end
    end
  end
end

I higly recommend you checkout Cashier. It may be useful in your application especially if you have complicated relationships with high performance requirements.

Caching Complicated Actions (or Methods)

Let’s say you have an index action. However, it’s more complicated than a normal scaffold index. The user can search, filer, sort and apply different query options. Think for example a form build with MetaWhere or Sunspot. There are infinite number of combinations, but the data is always the same. That is, a search for “EC2” will always have the same results as another search for “EC2” as long as the underlying data hasn’t changed. We could easily cache the index action if we could figured how to represent each unique combination of input parameters as a key value. Memcached also has a key length limit. I don’t know what it is off the top of my head, but you should try to keep the key short. How can we do this? We use a cryptographic hash. A cryptographic hash is guaranteed to be unique given a unique set of input parameters. This means there no collisions.

hash(key1) != hash(key2) # will always be true

The Ruby Standard Library comes with SHA1. SHA1 is good hashing function so we’ll have no problems using it for these examples. It takes a string input and generates a hash. We’ll create a composite key with a timestamp and string representation of the input parameters.

require ‘digest/sha1’

class ComplicatedSearchController < ApplicationController

caches_action :search, :cache_path => proc {|c|
  timestamp = Model.most_recently_updated.updated_at
  string = timestamp + c.params.inspect
  {:tag => Digest::SHA.hexdigest(string)}
}

end

That will cache every combination of input parameters you can throw at it. This is perfect for actions with pagination as well. It’s perfect for anything that uses the same underlying data based on input parameters. This can save your bacon if a search takes a few seconds. If one user just did the same search, the second user won’t have to wait at all. Hell, they might even be impressed.

Bringing Caching into the Model Layer

Caching isn’t just for views. Some DB operations or methods make be computationally intensive. We can use Rails.cache inside the models to make them more efficient. Let’s say you wanted to cached the listing of all the top 100 posts on reddit.

class Post
  def self.top_100
    timestamp = Post.most_recently_updated.updated_at
    Rails.cache.fetch ['top-100', timestamp.to_i'].join('/') do
      order('vote_count DESC').limit(100).all
    end
  end
end

I’ve used the most_recently_updated method a few times. It is not a defined method, but a method named so that you understand what it is doing. We can use these concepts to do more fun stuff. My main project has companies and customers. An account has many customers and companies. It’s typical that I need to retrieve all the customers per an account. This can be 10000 records. That takes time. ActiveRecord instantiation on that order is not free. However, I only care about customers or companies in the scope of a specific account. That means, I only use the account and customers/companies association. Rails gives you the ability to specific a different attribute for :touch on belongs_to. I use this to my advantage to create an ‘association_name_updated_at’ column. Then specify :touch => ‘association_name_updated_at’. Here’s how it looks in code:

class Account < ActiveRecord::Base
  has_many :customers
end

class Customers < ActiveRecord::Base
  belongs_to :account, :touch => :customers_updated_at
end

That gives me a timestamp I can use to generate all keys. Now I can use Rails.cache to fetch different queries and keep them all cached. You can wrap this functionality in a module and include in other associations.

require 'digest/sha1'

module CachedFinderExtension
  def cached(options = {})
    key = Digest::SHA1.hexdigest(options.to_s)
    association_name = proxy_reflection.name
    owner_key = [proxy_owner.class.to_s.underscore, proxy_owner.id].join('/')
    tag = proxy_owner.send("#{association_name}_updated_at").to_i

    Rails.cache.fetch [owner_key, association_name, tag, key].join('/') do
      all options
    end
  end
end

all is a method that takes many options. We don’t really care what’s passed in, we just need to be able to generate a cache key based on the input parameters. Since we know when the association was last updated, the method will return fresh content depending if records have been modified. Include the extension in your association and you’re on your way!

class Account < ActiveRecord::Base
  has_many :customers, :extend => CachedFinderExentsion
end

# all find's now automatically cached and expired
@account.customers.cached(:conditions => {:name => 'Adam'})
@account.customers.cache(:order => 'name ASC', :limit => 10})

These are just examples of what you can do with caching in the model layer. You could even write the type of cached finder extension for ActiveRecord::Base. This is different from SQL caching since it only persists through request–this is cached throughout the entire application.

CSRF and form_authenticty_token

Rails uses a CSRF (Cross Site Request Forgery) token and a form authentic token to protect your application against attacks. These are generated per request and each pages get unique values each time. protect_from_forgery is added by default to ApplicationController. You may have run into the problem before. You may have tried to submit a POST and received an Unauthorized response. This is the form_authenticity_token in action. You can fiddle with it and see what happens to your application.

These tokens cause problems (depending on what Rails version) you’re using with cached HTML. Caching a page or an action with a form may generate unauthorized errors because the tokens were for a different session or request. There are parts of the cached pages that need to be replaced with new values before the application can be used. This is a simple process, but it will take another HTTP request.

You’ll need to create a controller to server up some configuration related information that’s never cached. That way, a cached action will load, then a separate request will be made for correct tokens.

NOTE: You may run into more problems with on Rails 2. This is because Rails 3 uses a form authenticity token and CSRF in a meta tag in the HEAD of the document. This is for AJAX requests. You may notice the rails.js file appends them to all AJAX requests. Forms submitted with AJAX with something like $(form).serialize() will send the form_authenticty_token since it’s automatically included in all forms generated with form_for or form_tag.

You need to create a new controller that responds_to JavaScript and return some JS for the browser to evaluate. Here’s how you can replace the information in the meta tag for Rails 3. You can also use this logic to update all form_authenticty_token inputs on the page.

$("meta[name='csrf-token']").attr('content', '<% Rack::Utils.escape_html(request_forgery_protection_token) %>');
$("meta[name='csrf-param']").attr('content', '<% Rack::Utils.escape_html(form_authenticity_token) %>');

// you may also want to supply current application status as well.
// for example, you may want to know the current users's ID
// for use in your application JS
MyApp.userId = '<%= current_user.id %>';

Dealing with Relative Dates (or other content)

Many Rails applications use distance_of_times_in_words throughout their application. This can cause major problems for any cached content with a data. For example, you have a fragment cached. That fragment was cached 1 month ago. 2 months ago, it’s still in the cache. Since you stored a relative date in the cache, the fragment contains ‘1 month ago’. This is no good. You can solve this problem easily with JavaScript.

JavaScript is better for handling dates/times than Rails is. This is because Rails needs to know what the user’s time zone is, then marshal all times into that time zone. JavaScript is better because it use the local time zone by default. How often do you want to display a time in a different zone than user’s current locale? You can dump the UTC representation of the date into the DOM, then use JS to parse them into relative or something like strftime. I’ve encapsulated this process in a helper in my Rails applications. Once all the data is in the DOM, you can do all the parsing in JavaScript.

def timestamp(time, options = {})
  classes = %w(timestamp)
  classes << 'past' if time.past?
  classes << 'future' if time.future?

  options[:class] ||= ""
  options[:class] += classes.join(' ')

  content_tag(:span, time.utc.iso8601, options)
end

Then, when the page loads you can use a library like date.js to create more user friendly dates.

Time to Cash Out

I’ve covered a ton of material in this article. I’ve given a through explanation of how all the Rails cache layers fit together and how to use the lowest level to it’s full potential. I’ve provided a solution for managin the cache outside the HTTP request cycle as well as shown you how to bring caching into the model layer. This is not the be-all-and-all of caching in Rails. It is a indepth look at caching in a Rails application. I’ll leave you with a quick summary of everything covered and some few goodies.

Page Caching

The honest to goodness best caching ever. Bypass Rails completely.
Usually not applicable to any web application. Have a form? No good, the form_authenticity_token will be no good and Rails will reject it.

Action Caching

Most bang for the buck. Can usually be applied in many different circumstances.
Uses fragment caching under the covers.
Generates a cache key based off the current url and whatever other options are passed in
Get more mileage by caching actions with an composite timestamped key.

Fragment Caching

Good for caching reusable bits of HTML. Think shared partials or forms.
Use a good cache key for each cache block.
Don’t go overboard. Requests to memcached are not free. Maximize benefits by caching a small number of large fragments instead of a large number of small fragments.
Use auto expiring cache keys to invalidate the cache automatically.

General Points

Don’t worry about sweepers unless you have too.
Understand the limitations of Rail’s HTTP request cycle
Use cryptographic hashes to generate cache keys when permutations of input parameters are invloved.
Don’t be afraid to use Rails.cache in your models.
Only use sweepers when you have to.
Tagged based caching is useful in certain situations.
Conslidate your cache expritation logic in one place so it’s easily testable.
Test with caching turned on in complex applications.
Look into Varnish for more epic wins.
belongs to with :touch => true is your friend.
Use association timestamps
Spend time upfront considering your cache strategy.
Be weary of examples with expire by regex. This only works on cache stores that have the ability to iterate over all keys. Memcached is not one of those.

Learning Rails: A Glossary

me@broadcastingadam.com (Adam Hawkins) — Sun, 24 Apr 2011 00:00:00 +0300

I’ve been teaching Rails to some people. One thing that’s hard for them to get straight is the large number of tools involved in Rails development. This post is glossary of terms you may come across. Hopefully it will clarify things for you.

Acceptance Testing

Acceptancing testing is the act of testing use cases. Test cases are written in a way that describes a use case. Then a test case is passing it can be accepted. Cucumber is a good tool for acceptance testings. Work with your stake holder to develop tests that represent use cases. When the test is complete the feature should be accepted. Acceptance testing is focused around people outside the code development accepting features.

Application Servers - Thin, Mongrel, Passenger, Unicorn

These are all application servers. They interact with your Ruby code and respond to requests. They are integrated with web servers like Nginx or Apache to server you application on the internet.

Authentication

Authentication is the process of matching credentials to a person and verifying them. Authentication is purely about identifiying who the user is–and not what they can do. Devise is an example of an authentication library.

Authorization

Authorization is the process for determine what a specific user can do. Authorization usually involves permission or role based systems. CanCan is an example of an authorization library.

Behavior Driven Development (BDD)

Is essentially the same as TDD except using a different set of tools to express code in terms of user facing behavior. Rspec and Cucumber are part of the BDD toolbox.

Bundler

Bundler reads a Gemfile and calculates a set of version requirements to make all the specified gems live happily together. It will prevent version conflicts and infamous ‘gem already activated error’. It allows you to install git gems or standard gems from rubygems.org. It does not require libraries, it simply makes them available. It is up to you require them in your programs. Bundler can be used outside of rails. You should use bundler when you do any ruby work.

Capistrano

Capistrano is a tool for executing command one groups of remote (or local) serves over SSH. It is primary used to deploy Ruby (on Rails) application. It has support for multistage environments. Example, staging and production. You can easily write your own tasks similar to writing rake task. It is the preferred way deploy Rails applications.

Capybara

Capybara is a gem designed to provide an abstraction layer between different browser drivers. It is primarily used in integration testing to interact with the web server. It provides an API to navigate between pages, click buttons, fill in forms, and other user interactions. It has adapters for many different browser drivers. Notable drivers include Selenium, rack-test and webrat.

Compass

Compass is a library built around SASS abstractions. It provides mixins for many common things like styling buttons and forms. It is also easy to extend and comes with many built in functions. The blueprint CSS framework is bundled by default.

Cucumber

Cucumber is a test framework for creating plain english acceptance tests. The tests can be executed automatically. Cucumber is used for integration testing web applications. The test suite is often used in CI (Continuous Integration). Cucumber uses a language called Gherkin to parse files into lines and match them against regular expressions. Regular expressions are matched with code blocks. Your test code lives in these blocks.

Cucumber tests are divided up into “Feature” files. Each feature has many “scenarios.” Features are like use cases. Scenarios are different permutations of that use case. Here is an example Feature file:

Feature: Make Widthdrawls from Accounts
  As an account holder
  I want to use my money
  In order to use it buy thing

  Background:
    Given I have account under "RubyX"
    And my account is activated

  Scenario: There is enough money in my account
    Given my account has "$1,000"
    And I'm at the bank
    When I widthdraw "$500"
    Then my account should have "$500"

  Scenario: There is not enough money in my account
    Given my account has "$1,000"
    And I'm at the bank
    When I widthdraw "$500"
    Then the teller should reject my transaction

Here is an example step definition:

Given /I'm at the bank/ do
  # set up pre conditions
end

Then /the teller should reject my transaction/ do
  # assert on things
end

DSL

DSL stands for Domain Specific Language. They are crafted to solve one or more problems very eloquently and nothing more. For example, a DSL created to declare work order would be horrible suited for writing Photoshop. DSLs are usually wrappers around more complicated methods that make it easier to express the intent of the underlying code from a programmer’s perspective. You may have used a DSL before and not realized it. Here is an example from Sunspot’s search functionality. It’s designed for describing a search and nothing more:

Post.search do
  fulltext 'best pizza'
  with :blog_id, 1
  with(:published_at).less_than Time.now
  order_by :published_at, :desc
  paginate :page => 2, :per_page => 15
  facet :category_ids, :author_id
end

ERB

ERB is Embedded Ruby. ERB is built into the Ruby core. It allows to to place Ruby inside other files. For example, placing Ruby inside HTML. Here is an example:

<div class="<%= @ticket.state %>"
  <p><%= @ticket.message %></p>
</div>

Factories - FactoryGirl & Machinist

These are two popular libraries for creating object factories. They are usually used in test suites and population scripts. They provide a default set of attributes and allow programmers to specify the attributes they care about at creation time.

Git

Git is a distributed version control system. Each user has a complete copy of the repository. Changes can be pushed back to the remote repositories for others to pull or push from. Linus Torvalds created Git because he was unsatisfied with other version control systems like CVS or SVN. Do not get GitHub confused with Git. GitHub is simply a service for hosting the main Git repository. You can use git independent of github, however most Ruby developers use github exclusively.

HAML

HAML is an HTML abstraction language. It’s great for structuring documents and horrible to content. It will autoclose tags and lets you specify attributes as a hash. You can also include ruby code inside the templates. Here is an example:

.post#post_5 
  .content= simple_format(@post.content)

Heroku

Ruby PaaS (Platform as a Service). They provide free cloud hosting for Rack applications with paid plans for increased resources. It is a very easy way to deploy your first application. Beware, they are easily owned by Amazon’s AWS failure.

Metaprogramming

Metaprogramming is a term for dynamically generating code at runtime. Metaprogramming is why Rails feel the way it does. ActiveRecord associations to dynamically add methods to your classed based on how to declare them. Metaprogramming is possible in Ruby because it’s a dynamic language interpreted at run time.

Open Classes & Monkey Patching

Ruby has open classes. This means you can simply declare methods insides a class that’s already been defined. ActiveSupport uses open classes to add all those nice methods to core Ruby objects. This is how you can add a method to the String class:

class String
  def wtf?
    puts "wtf? " * self.length
  end
end

Rake

Rake is like the Ruby version of make. You can create custom tasks that can be executed from the command line. rake db:migrate is a classic example. You can create as many tasks as you want. They can have prerequisites. They can also be in namespaces. A ’:’ designates tasks in different namespace. db:migrate means ‘db’ namespace, ‘migrate’ task. Multiple tasks can be executed in one go like so: rake db:create schema:load. They will be executed in the order they are listed. Rake was originally designed to be like make, but is often used to execute arbitrary code outside an application context. A cron job is a perfect example.

RJS (Ruby JavaScript)

RJS is an abomination. Don’t use it. RJS uses ruby helpers to generate JavaScript to dump into HTML attributes violating UJS.

RSpec

Rspec is a unit testing framework. It is based around the idea that test should describe behavior of classes in an english like way. Test files are called “specs”. Spec files are divided into “examples.” Examples contain matchers. Spec files can share examples. Here is an example spec_file:

require 'spec_helper'

describe Post do
  it { should have_many(:comments) }

  describe "Post#out_dated?" do
    subject { Post.new :created_at => 2.months.ago }

    it { should be_outdated }
  end
end

RVM

Rvm stands for Ruby Version Manager. It is a set of bash script designed to allow you switch out Ruby interpreters on the fly. It manages installed ruby interpreters and makes is very easy to install different implementations. It also manages Gemsets. Gemsets are groups of gemsets that are distinct from other groups (except the global gemset which shares gems between different ruby interpreters).

SASS & SCSS

SASS and SCSS are CSS abstraction languages. They are compiled down to CSS. They allow you use variables, modules and include other files. In short, they make it much easier to write and main large amounts of CSS.

Selenium

Selenium is a library that simulates user interaction with a browser. It runs the full browser. Selenium works best in FireFox, but can work in Chrome and other browsers. Commands are sent across as JavaScript which the browser evaluates to complete each action. Selenium is the most complete solution for simulating a user for your web application.

Test Driven Development (TDD)

The practice of writing a failing test first then completing the implementation. This makes the developer spend more time thinking about the code upfront while providing a solid test suite for the entire application. You can use Test::Unit for TDD in Ruby.

Test::Unit

Test::Unit is a unit test framework built into Ruby 1.8. It is known as MiniTest in 1.9. It provides functionality for writing test cases with standard setup and tear down. Rails generates test files built in Test::Unit by default. It provides basic assertions. It’s similar to jUnit or any member of the xUnit family. Here is an example:

require 'test_helper'

class PostTest < Test::Unit::TestCase 

  def test_out_dated? do
    post = Post.new :created_at => 2.months.ago
    assertTrue(post.out_dated?)
  end
end

UJS (Unobstrusive JavaScript)

Unobtrusive JavaScript means separating JavaScript from the HTML. Specifying an onClick attribute in HTML is consider obtrusive because it obfuscates the markup. It is also hard to maintain because your javascript is harder to maintain. You can do the same thing unobtrusively by using jQuery to find the element by a class name and applying a click handler. Essentially UJS means keep JavaScript in .js files and HTML in .html files. Separation of church and state if you will.

Webrat

Webrat is the original headless browser. It’s similar to selenium, but much more implemented. It does not execute JavaScript and does not execute in a GUI. It is the most basic driver and is perfect for interacting with simple websites.

Sproutcore Login Tutorial

me@broadcastingadam.com (Adam Hawkins) — Mon, 04 Apr 2011 00:00:00 +0300

Sproutcore is probably the coolest thing I’ve seen since I saw rails way back early in 2006. I think Sproutcore is the future of complex web application because it is an excellent way to create complicated (and elegant) UI’s using Javascript. It uses KVO (Key-Value-Observing) to keep the UI in sync with the model. Yehdua Katz put it this way: “the view always represents truth.” I will not write about Sproutcore (SC from now on) because there is plenty of information to read on what it is and how it works. They have a guides site to learn about it. However, there is still a lot of confusion on how to implement some simple stuff in SC! I’ve been dabbling with SC for a few months now–and now I’m finally ready to pass on some knowledge. I write SaaS applications, so the first thing I ever consider is “the user has to login.” That was my first hurdle with SC. I had to solve this problem: “How can I create a login form with sproutcore and authenticate against a database of users?” Well in order to solve this problem we need to do a few things:

Create an SC application with a form for username/password
Figure out how to submit that form
Create an HTTP API to determine if the form is valid
Return some information to SC

We’ll start out by creating a basic interface. We’ll have two pages. When the user goes to our application they’ll see the login form. If they login correcly, we’ll show them a different view.

We need some way to get the data from the form to pass it off for processing. SC.ObjectController will do this for us. SC.ObjectController is a proxy to some underling object. However, we never bind directly to the object, instead we bind to the controller. That way when some information changes in the controller (and thusly the underlying object) the various things bound to it will change. This is where KVO comes into play.

We also need something to process the data in the form. The processing code works along these lines:

Send a HTTP request to the server
Tell the UI to update according to the HTTP response

We’ll use a state chart framework to transition the application from a logged out state to the logged in state. State charts are good way to model applications. I suggest you google them to learn more about them. Our simple application will only have 2 states: logged in/out.

Once we have the SC application working correclty with some mock login info, we’ll hook up a simple web service written with Sinatra.

Creating the Sproutcore Application

Sproutcore comes with some generators (similar to Rails). It has a generator to create a basic application. I am using Sproutcore 1.5RC1 for this. Ensure the proper version is installed. You can install the gem and generate a new application like this:

$ # enter a fresh directory
$ rvm use 1.9.2@sproutcore-login-tutorial # if you want to use rvm
$ gem install sproutcore --pre
$ sc-init login_tutorial

Now you can start the server and see a hello world page

$ sc-server
# head over to localhost:4020 and choose 'login_tutorial'

Creating the Login Page

Take a peek at apps/login_tutorial/main_page.js. This file defines the intial view in our application. It creates a page with a label on it. Every SC.Page must have a SC.MainPane as the mainPane property. The mainPane contains the objects that are part of the view and displayed on the page. We’ll use this code as an example to create a new page that shows our login form. We’ll reserve the main page for the logged in state since it’s feasible to say that 99% of time the user is logged in.

Create a file named: apps/login_tutorial/resource/login_page.js.

Here is the skeleton:

LoginTutorial.loginPage = SC.Page.design({
    mainPane: SC.MainPane.design({
    })
});

Inside the mainPane there will be view with 2 text feilds and a button to submit the form. Here is the scaffold code you can use to create the view:

LoginTutorial.loginPage = SC.Page.design({
  mainPane: SC.MainPane.design({
    childViews: 'form'.w(),

    form: SC.View.design({
      layout: { width: 200, height: 160, centerX: 0, centerY: 0 },
      childViews: 'header userName password loginButton'.w(),

      header: SC.LabelView.design({
        layout: { width: 200, height: 24, top: 0, centerX: 0 },
        controlSize: SC.LARGE_CONTROL_SIZE,
        value: 'Login Required',
        textAlign: SC.ALIGN_CENTER
      }),

      userName: SC.TextFieldView.design({
        layout: { width: 150, height: 30, top: 30, centerX: 0},
        hint: 'Username'
      }),

      password: SC.TextFieldView.design({
        layout: {  width: 150, height: 30, top: 80, centerX: 0 },
        hint: 'Password',
        isPassword: YES
      }),

      loginButton: SC.ButtonView.design({
        layout: { width: 100, height: 30, top: 120, centerX: 0 },
        conrolSize: SC.HUGE_CONTROL_SIZE,
        title: 'Login'
      })
    })
  })
});

We still have to connect the controller to the data and setup the state chart. Let’s do that next.

Configuring the State Chart

Firs, update the Buildfile in the root directory to require the state chart framework. Change the only line in the file to:

config :all, :required => [:sproutcore, 'sproutcore/statechart']

Now create a file named: apps/login_tutorial/core_states.js. This file defines the state chart. It will have 2 states: loggedIn and loggedOut. When we enter the logged out state, the login form will be displayed. The form will be removed when we leave the state. The main page will be displayed when we enter the logged in state. Here is the code:

LoginTutorial.statechart = SC.Statechart.create({
  rootState: SC.State.design({
    initialSubstate: 'loggedOut',

    loggedOut: SC.State.design({
      enterState: function() {
        LoginTutorial.getPath('loginPage.mainPane').append();
      },

      exitState: function() {
        LoginTutorial.getPath('loginPage.mainPane').remove();
      }
    }),

    loggedIn: SC.State.design({
      enterState: function() {
        LoginTutorial.getPath('mainPage.mainPane').append();
      }
    })
  })
});

The state chart is in charge of handling the flow of the application. It needs to be started when the user goes to the page. Open up apps/login_tutorial/main.js and replace the content of the main function with:

LoginTutorial.main = function main() {

  LoginTutorial.statechart.initStatechart();

} ;

Now, reboot the server and head back to the application. You should see a shiny login form.

Binding with a Controller

Now we need to create a controller. A controller manages data for us. We’ll create a controller with two properties and bind them to the values in the login form. First generate a controller:

$ sc-gen controller LoginTutorial.loginController

Now create two attributes for the controller like so:

LoginTutorial.loginController = SC.ObjectController.create(
/** @scope LoginTutorial.loginController.prototype */ {

  userName: null,
  password: null

}) ;

Now we need to bind the controller to the inputs. We’ll set the valueBinding property on the text fields to the correct value on the controller. Then whenever the user types something in the form, the controller’s attributes will update. We’ll use the controller to get the data to actually login soon. Here is the code to bind the text fields to the controller:

userName: SC.TextFieldView.design({
  layout: { width: 150, height: 30, top: 30, centerX: 0},
  hint: 'Username',
  valueBinding: 'LoginTutorial.loginController.userName'
}),

password: SC.TextFieldView.design({
  layout: {  width: 150, height: 30, top: 80, centerX: 0 },
  hint: 'Password',
  isPassword: YES,
  valueBinding: 'LoginTutorial.loginController.password'
}),

Reload the page and now you can type stuff into the fields. Then you can check controller properties in the console. So for example, if you type ‘Adman65’ into the user name field you could evaluate this in the console:

> LoginTutorial.loginController.userName
"Adman65"

Conversely, you could also set the value of userName in the controller and it would update the UI:

LoginTutorial.loginController.set('userName', 'rpm')

Making the Form Do Something

The next step is to make the login button do something. Pressing the button fires an event. We can configure the button to call a method on a responder. A responder is an object that knows how to handle the action. We can set the responder property on a view so all actions are processed by the same object. Our statechart is the responder for this example. Here is the strategy:

Tell the view that state chart will handle events fired from it
Tell the button to call a method on the responder
Add a method to handle the action
Use that action to authenticate using the credentials

Set the responder like this:

LoginTutorial.loginPage = SC.Page.design({
  mainPane: SC.MainPane.design({
    defaultResponder: 'LoginTutorial.statechart',

    //...
  })
})

Update the button view like this:

loginButton: SC.ButtonView.design({
    layout: { width: 100, height: 30, top: 120, centerX: 0 },
    conrolSize: SC.HUGE_CONTROL_SIZE,
    title: 'Login',
    action: 'authenticate'
  })

Now add a method in the state chart to handle the action

LoginTutorial.statechart = SC.Statechart.create({
  rootState: SC.State.design({
    initialSubstate: 'loggedOut',

    loggedOut: SC.State.design({
      enterState: function() {
        LoginTutorial.getPath('loginPage.mainPane').append();
      },

      exitState: function() {
        LoginTutorial.getPath('loginPage.mainPane').remove();
      },

      authenticate: function() {
        // we'll fill this in later
        // you can call alert('weeeeee') to test it's working if you 
        // don't trust me :D
      }
    }),

    loggedIn: SC.State.design({
      enterState: function() {
        LoginTutorial.getPath('mainPage.mainPane').append();
      }
    })
  })
});

Mock Authentication

At this point we can check the credentials in the authenticate method we just added. For now we’ll just check to see if the user has filled in both things, then move to logged in state. We’ll show an error if either value is missing. Once the UI working, we’ll use a simple web service to authenticate.

authenticate: function() {
    var userName = LoginTutorial.getPath('loginController.userName');
    var password = LoginTutorial.getPath('loginController.password');

    if(!SC.empty(userName) && !SC.empty(password)) {
      this.gotoState('loggedIn');
    } else {
      SC.AlertPane.error("Login information incorrect!");
    }
  }

Now, when you fill in both fields and press login, then you should see the original welcome message. Otherwise, you get a popup error message.

Connecting to the Web

Now we’ll create a simple sinatra site that accepts a post and does the same basic checking. Here is webservice.rb:

require 'rubygems'
require 'sinatra' # make sure you install this gem
require 'json' # make sure you install this gem

post '/login' do
  data = JSON.parse request.body.read

  if data['user_name'] && data['password']
    200
  else
    412
  end
end

Sproutcore sends parameters as JSON encoded strings. We need to decode the JSON to get the parameters. Now you can run the file like this:

$ ruby webservice.rb

Now we have to update the authenticate method to post data to the server:

loggedOut: SC.State.design({
  enterState: function() {
    LoginTutorial.getPath('loginPage.mainPane').append();
  },

  exitState: function() {
    LoginTutorial.getPath('loginPage.mainPane').remove();
  },

  authenticate: function() {
    var userName = LoginTutorial.getPath('loginController.userName');
    var password = LoginTutorial.getPath('loginController.password');

    SC.Request.postUrl('/login', {user_name: userName, password: password}).
      notify(this, 'didCompleteAuthentication').json().send();
  },

  didCompleteAuthentication: function(response){
    if(SC.ok(response)) {
       this.gotoState('loggedIn');
     } else {
       SC.AlertPane.error("Login information incorrect!");
     } 
  }
}),

We use SC.Request to create a HTTP POST ajax call. The second argument is the body. Calling .json() will encode the body argument when sending the request. notify() adds a callback to handle the response. Finally, send() actually sends the request. The first argument to the callback is always the response. We check to see if it’s ok then go to logged in state, else show an error message.

Finally, we have to update the build file to proxy ‘/login’ to sinatra. Add this line to your Buildfile:

proxy '/login', :to => 'localhost:4567'

Wrapping Up

I moved through this example pretty fast since it’s very basic. It is intended to give you a rough overview of how you can string together a controller, view, state chart, an web service to authenticate users and update the UI accordingly. Here is some further reading:

The Ruby Gem Challenge

me@broadcastingadam.com (Adam Hawkins) — Sun, 20 Feb 2011 00:00:00 +0200

I am working on a gem ranking website. One of the metrics is test results. Getting test result for a random project can be a very difficult task. I’ve cloned 3,081 different gems onto my computer. I’ve written a simple bash script to execute this loop:

for repo in ~/repos/* ; do
  cd ${repo}
  rvm use 1.8.7@${repo}
  bundle
  rake
done

Nothing fancy going on there. However, the results are appalling. Very few gems work out of the box on a clean machine. I think this a huge failure for gem developers. If you don’t have a rake task that can execute tests in a clean environment then you have a problem! I thought to myself, surely this cannot be the case. I tested well known gems like devise. No dice. I tested cancan. That didn’t work as well. I tried to test some of gem’s I’ve used. Not much luck there. I have the script running in a console right now. AASM just worked with 100% passing. I’ve tweeted some gems that work as well. Mail and HTTParty worked out of the box. HTTParty even had cucumber features passing! I think the gem authors (myself included) should rise to the occasion and make it easier for other people to test our gems! I think this would speak very highly of the ruby community. Everyone should try the Ruby Gem challenge out on their favorite gem:

Clone repo into fresh directory
Create empty gemset and bundle
Bundle (If the gem does not use bunder, fail right there)
Execute rake
Report results

After running this test through many many gems, I have a new found respect for the authors of the gems that passed my tests. Try it out and let me and the authors know your results! Tweet me at @Adman65 and tweet the authors know too! They should get on that :D

PS. It will also be nearly impossible to get above an 80% ranking on whatgem when I implement this scheme. That will really sort out the good from the bad.

Why I Love Volume 1: Sprockets

me@broadcastingadam.com (Adam Hawkins) — Fri, 11 Feb 2011 00:00:00 +0200

Sprockets is one of the most handy gems I’ve ever used. It allows you separate javascript into multiple files and use require keywords. This was a god send for me. The application I’m working has a TON of js. I was able to tame it with sprockets. I also came up with a nice directory structure a long the way.

The Problem

It all starts with application.js. You have one file. The documentation says dump your javascript into this file–hell, put all your js into this file. So you start writing a few ajax calls and various trickery. 20 lines. Then 50 lines. Then 100 lines. Then 500 lines. Then maybe a couple thousand. Wait….a couple thousand? How did we get here? A couple thousand for what? What page is this JS for? How do i find what javascript I’m looking for? Hmmm. What about my jquery plugins? /public is starting to get pretty packed. Now lets say you’ve got 25 different pages. Each page needs their own JS to accomplish certain tasks. At this point, is it smart to keep dumping things into the same file? I say no. It’s time to get things whipped into shape. There’s one thing I really like about Rails: convention over configuration. The views folder is setup pretty nicely. There is a folder that corresponds to the controller that renders the view, and a file for the view name. It would be nice to have this same structure for our javascript. When your JS starts to become rather large, you’ll have some common code that is shared. This stuff belongs in an application.js file. So how can we keep all this code organized? Sprockets.

Hail the Conquering Hero

Sprockets is a Ruby library that preprocesses and concatenates JavaScript source files. It takes any number of source files and preprocesses them line-by-line in order to build a single concatenation. Specially formatted lines act as directives to the Sprockets preprocessor, telling it to require the contents of another file or library first or to provide a set of asset files (such as images or stylesheets) to the document root. Sprockets attempts to fulfill required dependencies by searching a set of directories called the load path.

Perfect. It can even combine all our js into one single file. We can even minmize that later if we choose too. This means we can setup this type of directory structure:

/app
  /javascripts
    /pages
      dashboard.js
      settings.js
    /shared
      utility.js
    application.js

! That is pretty handy if you ask me. It becomes very odvious how the JS is organized. It also makes it very easy to add isolated bits of javascript for specific pages/widgets/etc. You can also add other directories to the load path. This means you can create a /vendor directory for your javascript. I love this because I can keep my downloaded jquery plugins in /vendor with git submodules for easy updating. So you could create this sort of structure for your application:

/app
  /javascripts
    /so_on_and_so_forth
/vendor
  /javascripts
    jquery.js
    jquery-ui.js
    jquery.plugin1.js
    jquery.plugin2.js

Nice. You can also require other files as you would in ruby. For example, say you’re in application.js and you want to ensure that some other javascript (like jquery) is loaded before this code is ran.

//=require <jquery>

// jquery dependent stuff here

Another example, say you’re writing some JS for the dashboard and you need to the utilities methods.

//= require '../shared/utilities'

MyApp.utilities.flashNotice('oh hai');

A require statement tells Sprockets to insert the content of the required file before processing the rest of the document. I like this because it makes it very explicit what JS is needed. It also prevents those wonderful undefined method xxx for null errors. A require with <file> tells sprockets to search the load path. A require without means it is a relative path name.

How I Integrated Sprockets

I took a similar approach to what I outlined earlier. I wanted a javascript file for each separate page of the application and a some shared folder where shared code lived. Then I wanted a way to easily initialize the pages. Each page would live in it’s own specific object, so there would be no collisions. Here is the directory structure I came up with:

/app
  /javascripts
    /pages
      dashboard.js
      settings.js
    /components
      widget1.js
      widget2.js
    /shared
      utilities.js
    application.js
    jquery.js
/vendor
  /sprockets
    /jquery
      /src
        jquery-1.4.4.js
        jquery-ui.js
        jquery.plugin1.js
        jquery.plugin2.js

My application has many shared widgets. I called them components because they can be reused in any context in many different places. Sometimes the JS for these things can be quite long, so I wanted a specific file for each one so I knew where to look when something went wrong. The default configuration for sprocket-rails has /vendor/sprockets/*src on the load path. I didn’t feel like changing it, and this way it lets me group similar files. Sprockets will always process application.js first. I use this to set the stage by requiring all different JS my application needs.

// application.js

//=require 'jquery'

jquery.js is a file simple loads all the stuff in /vendor:

// jquery.js

//=require <jquery.1.4.4.>
//=require <jquery-ui>
//=require <jquery.plugin1>
//=require <jquery.plugin2>

Here’s what one of the page file looks like:

//= require '../components/widget1.js'
//= require '../components/widget2.js'

var DashboardPage = {
  init: function() {
      // do stuff, this is called when the page is loaded
  },
  // protip, use an ajax callback for the page automagically
  ajaxComplete: function() { }
};

I added a helper to initialize the page and attach the current page. It generates javascript like along these lines and embeds it into the page:

$(function(){
  #{page_name.titleize}Page.init();
  $('body').ajaxComplete(#{page_name.titleize}Page.ajaxComplete);
});

Then in the view (/app/views/dashboards/show):

<% initialize_page 'dashboard' %>

How it Worked Out

This was the best changed I’ve ever made to this application. Before the JS was spread out into random files and it was a pain in the ass to track down how it got included and where it was. Now this way I know there is /app/javascripts/pages/page_name.js file and by the time that code is executed, all the required code is added. It’s also been very easy to add new plugins. Drop the jquery plugin into /vendor and update the jquery.js in /app/javascripts. Boom. Available everywhere. It is also concatenated into one large file so instead of 20 or so (yes I know this is bad) requests we now only have 1. This made a big difference in the load time. If you haven’t used sprockets, I highly suggest you check it out–especially if you have a js centric application.

tl;dr: Sprockets is a cool gem to organize and manage js your own way. It also concatenates all js files into one single file. This makes your page load faster. Use sprockets for inceased sanity.

Ubuntu, Jetty, Solr & MultiCore

me@broadcastingadam.com (Adam Hawkins) — Wed, 02 Feb 2011 00:00:00 +0200

Solr is a wonderful fulltext search program. It can be configured to do a great many things. It can also be a royal pain to setup. Solr is written in Java. In order to use solr in your application you must configure a java application somewhere to serve up Solr. Solr runs as an web service. You index docouments by posting XML to the server for indexing. You can install a java application server like Tomcat or Jetty to host Solr. By default, Solr can only index one set of data. This means, if you need to host multiple applications on the same solr server, then you have a few options. You can create new Solr instances for each application, or you can use Solr’s MultiCore functionality to index different datasets. MultiCore is like creating another database for Postgres. I’ll show you how to get this up and running under Ubuntu.

Installing Jetty, Java and Solr.

This is one of the reasons I love ubuntu server. It just has packages. I don’t have to worry about downloading code from random place, it just has everything a boy could need in a server. Install these packages using apt:

sudo apt-get install solr-jetty openjdk-6-jdk

This pulls in ~60MB and a ton of packages so watch out for that :D

The next thing we want to do is setup Jetty to listen on all connections. By default the installation is only available on htt://localhost:8080. That’s great if you’re making a local server, but we need to open our box up to the world. Ubuntu uses a file /etc/defaults/jetty to manage daemon settings. Open this file in vim and replace this line:

#JETTY_HOST=$(uname -n)

With:

JETTY_HOST=

This will tell jetty to listen on all connections. You can also put in your own ip or domain name if you please. Feel free to change the port as well at this point.

Now navigate to the top of the file and /NO_START to go to the next setting we need to change. Replace the 1 with a 0 and wer’re in business. This will tell jetty to start when the server is loaded.

Now fire the server up with:

/etc/init.d/jetty start

Once all is good you should be able to navigate to http://yourhost.com:8080 and get welcome page. This is basically a “it works page”. Now you can move on to solr. Ubuntu already did a lot of extra work for you by installing a runnable version of solr into Jetty. You can access that at http://yourhost.com:8080. It is a very basic admin–but it’s something. Now we’re ready for MulitCore.

Multi Core

MultiCore is was the most complicated part for me to get setup, partially because everything I read conflicted with something–but don’t worry. It should be easy for you if you follow along. You can read the offical wiki page http://wiki.apache.org/solr/CoreAdmin.

Here’s what they don’t tell you, or assume you should know: In order for MultiCore to work, each core must have it’s own solrconfig.xml and schema.xml. That’s fantastic, but excuse me, where the hell do I put these files? That was my life for 3 hours. Mucking around with random configurations until POOF. \o/ It all worked.

You also have to create a solr.xml file separate from all the other config files that tells Solr to load multicore. This was outlined reasonably well in the documentation, but it still gave me headaches.

Step 1. Solr.xml

We need to create a file that tells Solr to load our cores. You can decided ahead of time what they are, or just use this as a template for now. Below is a template you can follow. You can create as many cores as you want. When Editing the file, be sure to replace all copies of “production” with whatever the name of your core is. In my setup, I needed 3 different cores. One for production code, one for staging code, and one for a beta code. Once you’ve created this file, save it as: /usr/share/solr/solr.xml.

<solr persistent="true">
 <cores adminPath="/admin/cores">
   <core name="production" instanceDir="production" dataDir="/var/lib/solr/production/data" />
   <core name="staging" instanceDir="staging" dataDir="/var/lib/solr/staging/data" />
   <core name="beta" instanceDir="beta" dataDir="/var/lib/solr/beta/data" />
 </cores>
</solr>

Don’t forget to set the dataDirectory attribute as well!

Step 2. Making Data Directories

Now we must create directories for our index data to live. They were specified earlier in the step 1. These directories must be writable by the jetty user. Apt automatically added this user for you when you installed jetty-solr. You can create them with this command.

sudo mkdir -p /var/lib/solr/production/data

Repeat this command for however many cores you need. Next make jetty the owner.

sudo chown -R jetty /var/lib/solr/

Step 3. Copying the Config Files

This was the hidden step. First thing we need to do is create directories for our cores to live. Each core has it’s own schema and config files. These need to be created. It’s just like we did in step 2, but with a different directory.

sudo mkdir /usr/share/solr/production

Now we need to create the configuration files. I’ve posted them in gists. These files are templates and just enough to get the server started. It is up to you to do the customization!

Download those files or keep them open. Now create a conf directory inside the directory you’ve already created

sudo mkdir /usr/share/solr/production/conf

Now enter the directory and paste those files.

cd /usr/share/solr/production/conf
# paste solrconfig.xml into your editor and save it
# paste schema.xml into your editor and save it

Now at this point, you can simply copy this directory for the other stages. This is especially helpful if you have 5 stages. You can duplicate the config like so

sudo cp -R /usr/share/solr/production /usr/share/solr/new_name1
sudo cp -R /usr/share/solr/production /usr/share/solr/new_name2
# and so on

Now you are ready to restart the server

sudo /etc/init.d/jetty stop
sudo /etc/init.d/jetty start

Now head back over to http://yourhost.com:8080/solr/admin And you should see links to all your cores!

Step 4. Customization

Depending your needs you may have different schemas and configurations for each core. You should configure them now. If you need to use the same configuration to each core, you should symlink the main solrconfig.xml to the various stages. For example, you should have the production and staging cores running the same config.

Feel free to hit me up on twitter at @Adman65 with questions or problems! Hope this helped.

Cucumber's env.rb & Dry Run Problems

me@broadcastingadam.com (Adam Hawkins) — Thu, 02 Dec 2010 00:00:00 +0200

Env.rb setsup Cucumber’s execution environment. The generated file from cucumber-rails essentially loads your rails env and setups up Capybara etc. That’s all well and good but what do you do if you need to add your own stuff. Once you’ve built up a sizeable cucumber test suite, it’s probable that you’ve got some modifications to env.rb. However, they should not be there since when you upgrade to a new version of cucumber (mainly cucumber-rails) it wants to regenerate that file. So what you need to do is split up your modifications into sepearate files. Here are some modifications you may have:

Modifying your Capybara driver (yay chrome)
Loading blueprints
Customizing specjour
Settings other constants
Insert random code

That’s all well and good but it’s not the correct way to do it. There are few options. You can split each modification into its own file and drop it in /features/support. Cucumber will autoload all files in side this directory. Technically it matches all files using this glob pattern: features/**/*.rb. However env.rb is loaded before all other files in features/support. This means you can drop create a file like this for specjour into features/support/specjour.rb:

# tell Capybara to start a server on any open port
# since specjour will start multiple workers on the same computer
# and hence Capybara will try to connect to the same port 
# locking up the test suite
if ENV['TEST_ENV_NUMBER']
  Capybara::Server.class_eval do
    def find_available_port
      server = TCPServer.new('127.0.0.1', 0)
      @port = server.addr[1]
    ensure
      server.close if server
    end
  end
end

That fill will be loaded after env.rb when you execute your tests. This does create an interesting wrinkle. I use dry run mode a lot in my suite. I refactor my features and steps quite often as I get a better understanding of the domain. I use dry run mode to check to see if all the steps are defined before executing the test suite. The test suite can take over an hour. :( Cucumber does not load env.rb in dry run mode, it does load all other files in /features/support. This creates a problem if you have files in features/support that require env.rb to be loaded. For instance the specjour example I posted requires the capybara gem to be loaded. You could add:

require 'capybara'

But it won’t be able to find the gem since the gem environment is not loaded when the file is required. In dry run mode cucumber does not load files that match this regular expression: support\/env\..*. Interestingly Cucumber does not simply select features/support/env.rb since that is the standard file. That means you can name files “env.specjour.rb” or “env.capybara.rb” to have them execluded in dry run mode. Although, this issue is only present when you run features using the cucumber binary. If you run features through rake then you will not have problems since the complete rails environment is loaded before cucumber is loaded.

tl;dr PROTIP: put things you would’ve added to env.rb in a file in /features/support/customer_modification_.rb. If you run those features with rake you’ll be ok. If you run those features with the cucumber command you’ll be ok. If those modifications required env.rb to be loaded and you run features with cucumber in dry run mode name them: features/support/env.modification.rb.

Lessons From a Startup

me@broadcastingadam.com (Adam Hawkins) — Sun, 31 Oct 2010 00:00:00 +0300

I’ve been working as the lead developer at Finnish start up for about 9 months now. We’re making a cool product that we’ve got high hopes for. We’re getting our first customers now, so I’m taking some time to reflect on our progress and what I’ve learned.

1. Don’t Release Unfinished Code

This seems like a no brainer, but sometimes the excitement about a new feature gets the best of you and you push it out before you’ve had adequate time to to test it. I’m not talking about just integration testing, but letting some test users play with and make sure it’s working to their requirements and not yours. Last month we had a major feature released, it had already been delayed due to bug fixes. We held off and kept working on it. When we did release the feature it turned out that it still wasn’t done and we had to go shock and awe on the bug list. It’s better to be late and correct, then early and look like a noob. We learned that lesson.

2. Don’t Trust Third Parties

Our product integrates pretty heavily with various third party products. At one point we were working with four third parties to provide crucial parts of our product. Their services seemed useful, but we ran into problems when it came into crunch time. Our problem was that our release dates where not inline with their time. For example, we were waiting a month for one company to provide some information/API stuff for us. We wanted to roll out the feature in two weeks, but as time went on, we had to cut the feature because the third party couldn’t deliver. We are starting another third party integration. In the planning phase, we just assumed it would take twice or even three times as long as they say–this is simply because they’ve got their own business too.

3. Have (small) Goals

We are big dreamers. We want our product to be the shit. We want it to be awesome. We want to be a swiss army knife made out of unicorns tears flying around with a gold cape. We also need to sell this thing and develop real features. It’s been easy for us to get distracted and not focus on getting useful things done. What we need to do is focus on small features and deliverables. There are quick wins and small steps. Remember to set goals you can make–and be realistic about them! You only hurt yourself by setting unrealastic goals for yourself. Know your limits and work within in them. We learned this lesson as well.

4. Have a Vision

The vision is the over arching goal and purpose of what you’re doing. Someone needs to have this, or you’re just a leaf in the wind. However, don’t get your vision confused with your goals! You set goals in order to meet the vision. Your goals build up to a product that fulfills the vision. Also, be realistic about your vision. If you’re not making any cash, then your vision must be to make some money and set goals to make that happen.

5. Have a Standard

We’ve had a few developers come and go through the course of the project. I’ve been the only constant. In the beginning, it was wild west. Anyone could commit and features were happening all over the place. We recently had someone new coming a few months ago to do front end work. The front end work is slowly transitioning into backend-ish work. That’s all good if you have people that can wear multiple hats, except there has to be rules. After a while I got fed up and wrote up a standards document. It laid out what would have to be in place for ommits/features to be accepted and what kind of workflow to use. It’s published in the repo’s readme. All current and future developers will be held to the standards laid out. Having standards should increase code quality over the project’s lifetime.

6. Invest in Testing

Test. Test. Test. Test. I love testing and you should to. Testing can save your life. Include time for testing in the release schedule. Include time for testing before production deploys. Invest developer time in creating a good test suite. Use tools like specjour to distribute your test suite. Our cucumber test suite took ~1hr. Spent some time setting up specjour, got that sucker down to ~10min. You’re integration suite should run quickly and you should run it often. If the suite takes too long to run, it will not run that often and you lose the benefits of automated testing. Invest time and keeping this process lean. It will pay you back.

7. Make Software that Customers Want (and will pay for)

Duh. But, sometimes it’s easy to get distracted with stuff you think is cool. Example: you think feature A is off-chain and it should totally be in the project. Customers are lined up waiting for feature B. Feature B isn’t as cool but is going to bring in some cash. Work on feature B. Make a product that will sell and bring in money. Our product is a game changer (I know you hear this every time you read about start ups). When customers see what we’ve got cooking they are completely befuddled, next awe struck, then really interested. The project leader and myself are way past that phase. We are no longer awe struck by what our product can do–but the customers still are! You should focus on developing features that customers want and not (all the time) developing feature for yourself.

8. Know You’re Gonna Make Mistakes

It goes without saying, but understand that you’re gonna mess up. When you’re working in a startup, the business plan and overall product is not set in stone. Remember to keep this mind and adapt to change.

app/observers -- Where They Should Be

me@broadcastingadam.com (Adam Hawkins) — Sat, 09 Oct 2010 00:00:00 +0300

Afer you’ve been doing Rails for a while you become old and cranky about how you want things. I love my observers in /app/obsevers. I do not understand why they are not their by default. Models, mailers, and controllers all have their own folders, why can’t observers by default. They don’t even make any sense in /app/models. #1 They don’t model anything and #2 They aren’t subclasses of ActiveRecord (or some other ORM). If all the classes in /app/models are subclasses of AR, then what is an observer breaks the pattern. In Rails 2 if you want to specify another directory to load code from you have to specify add it to the config.load_path variable. This is not the case in Rails 3. If you simply want to shove your observers into /app/observers, jsut make the directory and move the files in there. You will have to move them if you don’t patch the rails g observer command to generate them in a new directory. That takes care of Rails, but now that we have our observers separated, it’s safe to assume we want to be able to run rake spec:obsevers. This is not a problem either. All you have to do is create a rake spec task to only run files in spec/observers. So drop this bad boy in /lib/tasks

require 'rspec/core'
require 'rspec/core/rake_task'
Rake.application.instance_variable_get('@tasks')['default'].prerequisites.delete('test')

spec_prereq = Rails.configuration.generators.options[:rails][:orm] == :active_record ?  "db:test:prepare" : :noop

namespace :spec do
  [:observers].each do |sub|
    desc "Run the code examples in spec/#{sub}"
    RSpec::Core::RakeTask.new(sub => spec_prereq) do |t|
      t.pattern = "./spec/#{sub}/**/*_spec.rb"
    end
  end
end

Now we have our own rake task for testing our observers. But let’s take it one step further and make our observer specs a first class citizen in Rspec2. You know that when you write controller specs you can call the controller method or in helper specs there is a helper object that represents the object under test. This magic happens because rspec contains special code that runs when initial describe block matches something like xxxController or xxxHelper. If it matches, it loads some special code to make writing specs for these classes much easier. All you have to do is take a peek into the rspec-rails source to see where the magic happens. I advice you to look at those files and figure out what’s up. Creating a new observer example group is easy. Drop this bad boy in side /spec/support/observer_example_group.rb

module RSpec::Rails
  module ObserverExampleGroup    
    extend ActiveSupport::Concern
    extend RSpec::Rails::ModuleInclusion

    include RSpec::Rails::RailsExampleGroup

    def observer
      example.example_group.describes.instance
    end

    included do
      metadata[:type] = :observer
    end

    RSpec.configure &include_self_when_dir_matches('spec','observers')
  end
end

This module adds some sugar to all specs in an observer example group. If your spec is in spec/observers you can now do something like this:

describe AccountObserver do
  it "should send a welcome email" do
     AccountMailer.should_receive(:welcome_email).and_return(mock_mail)
     mock_mail.should_receive(:deliver)
     observer.after_create(mock_account) # notice observer is defined in the observer example group
  end
end

Nice! We no longer have to call AccountObserver.instance in all our tests or set an @observer in a before filter. This also allows us to do some more cool stuff for our observer examples. You can include support modules for certian example by doing something like this in your spec_helper.rb file

config.include ControllerHelpers, :type => :controller

Now we can do that for observers as well! You may by thinking where the hell did those mock_account and mock_email methods come from? You define then in an obesrver helper module inside the support directory then tell rspec to include them for all observers like so. First create this file: spec/support/observer_helpers.rb

module ObserverHelpers
  def mock_account(stubs = {}) 
    @mock_account ||= mock_model(Account, stubs)
  end

  def mock_mail(stubs = {}) 
    @mock_mail ||= mock(Mail, stubs)
  end
end

Now in your spec helper:

config.include ObserverHelpers, :type => :observer

Poof! All done. Now you can go on your way running rake spec:observers and treating your observer specs as first class citizens w/Rspec2. Happy testing.