vijay's technical spot

Why is HTTP Streaming a joke on Rails 3.1?

2011-11-18T05:13:00.001-06:00

I have been trying to convert a Rails 3.1 app to use HTTP Streaming. The interesting fact about this app, and others that I have been working on for the past few years is that most of the models are ActiveResource models - not ActiveRecord. In this context, having the controller logic evaluated lazily takes on more importance - but the phrase "Rails 3.1 supports HTTP Streaming" leaves a lot to be desired.

I first tried to follow Ryan Bates' excellent rails cast. The tips are good, and I was able to get streaming working in a basic rails app ie one that uses ARecord quite easily. The key to streaming is the lazy evaluation of any logic that takes time - namely hitting the database (ARecord) or webservices (AResource). For ARecord, the Rails developers have piggy-backed on ActiveRelation's proxy (scoped) method for lazy evaluation. This is fine for ARecord. But does not work for AResource. This is the place where Rails breaks down.

What would have been better is if Rails had implemented a way for ALL such instance variables to be defined inside a lambda or Proc (in the controller) - so that the lambda is defined but not evaluated right then and there. And the second piece of the puzzle would have been for the getter of these instance variables (usually used in the views) to automatically evaluate the block the first time around that they were invoked (should only do the eval once).

One other thing to note is that any logic (instance variables that get assigned in the controller using the '@' symbol) that depend on the above lazy loaded values/attributes have to also be lazy loaded. This can be done using the method outlined here (this is one of the cleaner methods - and I am not sure whether the author was the first one to do it this way - I found other sites with the same way of tackling the problem).

Active Record Validator

2011-10-29T23:22:00.003-05:00

As your rails project sees multiple releases, you probably have new migrations that were run on your database. This coupled with Rails' default (IMHO, incorrect) attitude of not enforcing foreign key constraints at the db level might cause your old data to become invalid based on new ActiveRecord validations (unless your incremental migrations have taken this into consideration and massaged your data correctly). In our project, we had created a home-grown solution called ActiveRecordValidator. This would find all records in each table in your database (it was intelligent enough to handle STI, etc and also had a mechanism where you could specify some tables that you did not want to process - for eg audit tables, etc), and invoke valid? on them. If there were any errors, these were printed out to stderr. Progress was printed to stderr. This proved very useful for us, even in development/test so as to validate that all our reference table data was correct. When run against a snapshot of our production database (we setup a run every week against this snapshot data), it would inform us of data that would become invalid when the next production deploy happened. Our development iteration had an ongoing story card to fix such invalid data after talking to our SMEs. Since this code was done for a client, I could not open source it, but found something similar: active_sanity.

Application constants and accessing these from within Rails.configure block

2011-10-29T23:02:00.002-05:00

Around 4 years ago, when I was on my first ruby project, we came up with a solution to extract all application constants into a yaml file grouped by the rails environment. This concept was not new - but I did not find any plugin that had this functionality. After some time, I came to know about app_constants from my colleague, Leornado Borges. I have since used this gem in a couple of projects, but am missing one of the functionalities that our home-grown solution had: Our plugin had a method that could be invoked for any environment (we were calling this as part of our rspec suite for every CI run), that would take the aggregate of all the keys in every environment, and list out those keys that were not present in all the environments. This provided us with a way to quickly determine if a developer had added a key, but had forgotten to add it into the test/cucumber/uat/production environments. This was also invoked during deployment to production - so that if the yml file is being overwritten by your deployment tool, then we can be sure that any missing keys are caught during your deployment window - rather than at runtime by your user! (I think I will fork app_constants and implement this functionality - unless leonardo beats me to it!)

Note: If you are using this gem to extract your constants, but are using these constants within your environment initialization file, you might get an error. The way around this is to extract those specific lines into config.after_initialize blocks.

For eg:

    config.after_initialize do
      # Enable serving of images, stylesheets, and JavaScripts from an asset server
      config.action_controller.asset_host = "http://#{AppConstants.ASSET_HOST}"
      config.action_mailer.asset_host = "http://#{AppConstants.ASSET_HOST}"
    end

ActiveResource filter params

2011-10-29T21:22:00.002-05:00

Rails provides a way to filter action controller params from being logged. But if your app is talking to a webservice and passing these sensitive parameters, and you have turned on logged for ActiveResource::Base, then these sensitive params are anyways logged! To overcome this, I wrote this snippet:
https://gist.github.com/1319094.js
This can obviously be improved by separating out the params to be filtered between your ActionController::Base params and ActiveResource::Base params - but I leave that up to you...(this works for me)

Yaml - Yes? No? Maybe?

2011-09-24T03:04:00.002-05:00

Another interesting thing I found out this past week: the default yaml ruby library auto-converts keys with yes/true/ etc to the boolean true. I kind of expected that yaml might play around with the values - but I didnt expect that I would find such behavior with keys! Anyways, the way to fix it is to explicitly make the key a string!

yaml and japanese - translate.rb

2011-09-24T02:55:00.000-05:00

I had an interesting problem which grew incrementally a couple of weeks ago - and I wanted to share the experience. The problem statement was simple enough: we needed to internationalize the application that I am working on. Since this seemed simple enough with the rails-i18n gem, I wanted to generate the corresponding yml files from a master one automatically by invoking the google translator service. This seemed simple enough - but there was something I did not take into account. And that was: that the default yaml library does not have the ability to store content in languages like japanese. The way it bypasses this is to convert all such characters into unicode and store it (while writing to the yml file) - but this does not play nice when the same file is read in by rails. Enter the ya2yaml gem. This gem was written (not by me) to solve exactly this problem! Finally, I could invoke the google translator service and write the yml file directly - and my site now shows up with katakana characters correctly! To help others with the same problem, here is the code.

junitRunner

2011-03-22T19:25:00.000-05:00

I have almost given up on working on Java for the past 3+ years, but I am currently assigned to a project that is heavily into Java. When I landed here, I was pleasantly surprised to see most of the devs using Eclipse. The daily workflow involved running a lot of junit tests, and when I was pairing with a dev, I noticed that he ran the tests using the mouse (all tests in a suite), and then reran only a specific failed test - using his mouse. This was exactly the usecase for the junitrunner eclipse plugin!

Stubbing constants for tests

2011-03-18T22:49:00.000-05:00

Yesterday, I read a blog post about stubbing class constants for testing. I think using the before(:all) and after(:all) hooks are not the best way to achieve this. The reasons being that:
1) This happens for all test methods in that spec file - and is not limited to the [typically] few test methods that do need overridden values.
2) Ruby has an elegant mechanism with closures and 'yield', which is not being used.

Instead of the method suggested above, IMHO, this is better:

The above code block ensures that as soon as the block returns, the constants are reset to the old values thus making each test clean up after itself.

Keep your local git checkout clean

2011-03-17T12:23:00.001-05:00

In my previous project, active development always happened in trunk except for certain scenarios: spikes, fixing bugs on production release branches, completing some unfinished work on the next production release branch, etc. This caused a lot of our dev pairing machines to have multiple branches. And since we used to rotate pairs very often (every morning), whenever I went to pair on a different machine, I would first run this script:

The reason was very simple. When I am working on master, and the machine had multiple other branches (some of them having local changes, while others not), each time I did a push to the remote origin, it would complain (since git tries to push all local branches that track remotes, by default). Of course, we could have written our commit [rake] task in such a way that it deciphered the current branch and only pushed that. But, IMHO, its better to have a clean workspace rather than finding ways to bypass it. I know others who have a different opinion - the most common one being git is a DVCS and that's how it is supposed to work. That may be the case - everyone is entitled to their opinions. Mine is just that my workspace should be clean.

The above script will go through all the local branches, and pull from their respective remotes. If there is no merge conflict (by parsing the status message), then it would revert back to the master branch and delete the other local branch. This step is repeated for each locally existing branch.

This script and some more that I used to loop through and maintain multiple git repositories are shared through my github account.

Destroy does not actually mean destroy!

2011-03-16T23:33:00.000-05:00

When I started learning Ruby and Rails, I was fascinated with the rails scaffold generated code. There I was typing away to glory to come up with a webpage on a Java stack barely managing to keep it following a pattern, and here was a simple command line script which when invoked with the correct parameters, would generate the same for me in Ruby - wow!

And then I learned the first step - dont trust the generated code implicitly. Case in point: we had a [business/domain] scenario where the destroy action on an ActiveRecord would need to verify that some condition was satisfied, and if these conditions were not met, the destroy action would raise an error. The scenario in our case was that the entity was in use in some other relationships, and until those were deleted first, this entity could not be destroyed.

For the case of the entity creation, Rails provides a convenient callback mechanism called validations. The same is the case for updates. But for destroys, Rails assumes that all is well. This is carried on into the code generated for the controllers specifically for the destroy action. It just returns a 200 status code. But, as evidenced in our case, that is not always the case, and we had to manually edit all such scaffold-generated code to "fix" the issue.

Backing up Firefox and Thunderbird profiles

2011-03-16T23:13:00.003-05:00

I have been using Firefox and Thunderbird for the past 10 years or so - since the time when hard drive space was a precious commodity. As I started using more than one machine (or when I completely trashed my OS and had to reinstall it from scratch), I came across a need to copy and maintain these profiles. Since I wanted them to be in some "known" location, and since both these excellent mozilla products provide a way to edit an ini file and thus "move" the profile folders, I have been doing that as well for the past several years. (For more information on where to find your profile directory for your OS, and what to change, please look here.)

I ran across an initial version of this script here
As I started experimenting with bash scripts, I enhanced the above script (that would check if the program/application was running currently, and if not, it would do some common house-cleaning within these profile folders), with some new functionality, like delete some cache folders, etc. This cleanup should never be run when the application is in use - so be warned. Once the cleaning was done, I also added a cmd-line switch that would go ahead and create a tgz file for backups.

In case anyone is interested, here is the script:

This script has been written in bash, and so can be run on windows if you use cygwin.

You can of course take the script and modify it as you wish. Just be warned (again) that if you do edit it, dont remove the check for skipping the house-cleaning if the application is running. Also, if you end up making any changes, I would appreciate it if you could send me the changes to be incorporated into my copy.

Git > TrueCrypt > Dropbox: A truly geeky way to backup your home folder across machines and OSes

2011-03-16T22:53:00.000-05:00

I was searching for ways to backup and sync up my profiles across different machines and across different *nix based OSes that I use (Mac OSX and Ubuntu). The stuff I wanted to sync was stuff like my .zshrc, .vimrc, .irbrc, .ssh, etc. At the same time, there were my firefox and thunderbird profiles that needed to be synced up as well (disregarding the fact that I use some add-ons for these which are OS-specific). Since some of these folders also contained sensitive information, any storage/sync mechanism that I arrived at would need to hold the data in an encrypted fashion. Lastly, I thought it would be good to have a historical backup of these same files/folders.

When I was in India on an assignment for my employer (Thoughtworks), I had some time during the evenings to play around with these requirements and came up with a geeky solution. I would create a git repository (most of the stuff I was storing were ASCII files, not binaries. This was important since git does not scale well for binary files. Next this git repository would go into a TrueCrypt container which was configured with sufficient space. Finally, the whole TrueCrypt container [file] would go into my DropBox account. Done! This would solve all my problems (at least I thought so at that time, but...)

There are some obvious limitations of this approach:

The TrueCrypt container once created, cannot be dynamically made to expand
Even if the TC file were recreated with a higher storage capacity, it would still need to adhere to the size of my dropbox account (they currently give 2GB free for the basic/free account)
Syncing a 1.5GB file across three machines - all of them wireless - was seriously overloading my home DSL setup!

Enter SparkleShare - the syncing software where you can store the data not on the cloud, but on your own drives! This was featured on one of the ubuntu blogs a few days ago - and it uses the same pattern for storing the data: your data/files/folders are actually encrypted (in lieu of my TrueCrypt step) and stored inside a git repo on whatever network storage device you configure! Yay!
The only hiccup right now is that their software is still in beta for both Mac and Linux - but maybe I will try it out sometime soon and post my review/experiences.

externalizing configurations across different rails environments

2011-03-07T06:29:00.000-06:00

In all the rails projects that I have worked on during the past 3 years, I have come across a pattern where some configurations need to be externalized. These configurations could include sensitive information (authentication passwords for a 3rd party system), application runtime information (like number of items to show in a paginated list), environment-specific information (like development vs uat vs production). To accomplish this, it is very trivial to write a Registry module that would load up a few yaml files into a hash in memory. Each file is loaded and if the same key exists from a previous file, the new value would super-impose the one in memory - so that ultimately, there is one "master" copy that the application runtime would use. (This concept has also been prevalent for more than a decade in the Java world - I used it in 2000). As I said before, this is pretty trivial to write. While browsing in github, I came across this project: https://github.com/lukeredpath/simpleconfig - which has almost exactly the same features, in a cleaner way! Now onto forking that project and converting it into a gem...

Consolidating rails migrations

2011-02-28T22:55:00.000-06:00

With the meteoric rise in adoption of rails as a framework to develop business applications, we as developers need to also remember that these same codebases need some amount of love/house-cleaning from time to time. One of the areas that I want to highlight is database migrations.

As new releases of the app is deployed to production, and development of new features (needing new migrations) continues, it would be a good practice to consolidate previously run migrations into a single file. The idea is simple:

Use the [default] VERSION environment variable to specify the migration till which you want to consolidate
Drop, create and migrate the database till this version
Copy the resulting schema.rb file into the migration (replacing the old content of the 'self.up' method) in migration with that same version number
Typically you will never go below this migration, so raise an 'ActiveRecord::IrreversibleMigration' exception in the corresponding 'self.down' method
Delete older migrations (prior to the specified one)
Commit into your scm

At my previous client, we implemented a rake task to accomplish these steps in an automated fashion. Once a release went to production and after a couple of days of "settling down", this rake task was run with the version of the [production] db.

The advantage of doing this is:

Typically, as design evolves, you might rename/drop columns, drop tables, add seed data, etc which are all collapsed into a single step
The number of files that need to be evaluated and run (each time the db is recreated, typically, when running tests) is smaller, and thus taking [fractionally] less amount of time
Forces you, the developer, to maintain seed data outside migrations

Rails tricks: Require correctly

2011-02-27T02:03:00.003-06:00

On one of my previous projects, we noticed some strange (to us) behavior: any extra spec behavior (extra methods, config.before(:all), etc) was being loaded/defined more than once while running the whole suite of tests, but when we tried to run a single spec class (ie all tests defined within one file), this behavior was not evident. Digging further, we found that this was caused by requiring the spec_helper. As each spec file is being "processed", the require statement at the top of the file got triggered, and when a new file (at a different relative directory) was encountered, the spec_helper line would usually trigger this scenario. For eg:



spec/a_spec.rb: require File.dirname(__FILE__) + '/spec_helper'

spec/models/b_spec.rb: require File.dirname(__FILE__) + '/../spec_helper'

spec/lib/level1/c.rb: require File.dirname(__FILE__) + '/../../spec_helper'

To solve this, we resorted to using the fully expanded path to the spec_helper - since this would resolve to the same file each time. In this case, the above files were changed so:



spec/a_spec.rb: require File.expand_path('spec_helper', File.dirname(__FILE__))

spec/models/b_spec.rb: require File.expand_path('../spec_helper', File.dirname(__FILE__))

spec/lib/level1/c.rb: require File.expand_path('../../spec_helper', File.dirname(__FILE__))

UPDATE: My good friend and colleague Brian Guthrie corrected me saying that this situation can be avoided if the load path is set correctly (to include the spec folder) through the rake task before actually invoking rspec. That sounds reasonable, but I have yet to try it out - and unfortunately, I have moved on from the project where this was sorely needed.