vijay's technical spot

One effective way to use a CMS

2015-06-14T05:53:00.001-05:00

In most projects involving content managed (for internalization requirements) using CMSes, developers, project managers and content maintainers always seem to be pulling in different directions. This is due to the fact that project managers are concerned with taking "it" to production, developers always go into the "solutioning" mode, whereas content curators (marketing?) are concerned with getting the "right" content in front of the target audience.

Typically, once the developers are done developing the functionality, they also should have extracted out the snippets/placeholders in the web app to some CMS. When this goes out to UAT, it typically also goes out to the [usually outsourced] content curators. Here comes the first dilemma: content (mostly single words or phrases) have meaning in the context of how/where they appear in the UI. Without this knowledge, the content authors/translators are handicapped. So, the way we solved this issue in a couple of the past projects was to hook upto the CMS (SiteCore in these cases) using a web service on the CMS side that exposes content over a HTTP url. Since content is usually requested at a page-level for most websites, grouping content at a page level is useful to avoid performance issues. To further solve performance issues (since we didnt want to tax our CMS, and not have to repeatedly fetch the same [unchanged] content data over the network, we also built in some level of caching on the application tier. This cache would be blasted using a hook on the CMS side whenever a content author published new content.

Also, for the content authors to view the context of the content, we exposed our UAT site with some special query string parameters on each page, so that the externalized divs would be "flagged" to the user visually when (s)he visits the page using a star image on the top right corner of that div/span. Hovering on the div using the mouse would show the key and its ancestral tree within the CMS where the user could navigate to so as to effect a change. Changing this on the CMS, and publishing, and refreshing the app page would show the new content. Using this method, we were able to decouple the need for developers to be involved in pushing the content changes to the site and the content authors were able to publish and effect changes on their own schedule. Once this process was adopted by all roles, developers could push to production with the site showing the default English content as a fallback for non-english locales, and as the content authors pushed their changes, the website would transform. Of course, the only caveat was that the content authors needed to translate whole pages and then export from their UAT instance into the production instance in one shot. Rinse and repeat FTW!

We also used some techniques to store the snippets at a page-level in an application-level caching mechanism (a poor mans memcached), which was also purged whenever the content author published via the CMS. All this was accomplished using a hook from the CMS calling into a service in our application.

Coding/Design Developer practices - a brain dump

2015-06-09T05:31:00.001-05:00

This is not a complete list. Its a growing/living document of the practices that I espouse and try to elucidate to all my team mates on different projects/clients....

1) Running tests and adding to suite as new functionality is added
2) Keep build/unit test time as small as possible
3) Commit often (multiple times a day) and commit on a task-basis.
4) Breaking the build is ok - as long as it is fixed or reverted quickly. Do not leave for home on a broken build. The whole team should feel empowered to rollback after giving sufficient time for the committer to rectify the build. If this becomes too long, then anyone from the team can revert the commit.
5) Use the best tools for the best job (eg git)
6) Get enough build agents - developers should not be waiting for build queue.
7) dev machines should be as beefy as possible.
8) Use modular code at solution level (for eg using nuget).
9) communicate early and clearly - both about blockers, current work stream + new knowledge (mostly about tech) that could benefit the team as a whole and instigate thinking in different ways to solve your current domain problems.
10) Use tools (that measure code quality metrics) for the tech stack to track the trend as well as break the build in case of going below agreed-upon threshold
11) Use common settings for VisualStudio/ReSharper/StyleCop/etc - and run all these locally as much as possible. (do not depend on the 'Personal build' feature to be the safety net). Any change to a setting by a single developer should also be visible to others (Resharper/StyleCop settings are shareable this way)
12) Do not pick up a card for development until the analysis and UI is fully understood. The same goes for the QA's understanding and documenting the basic Acceptance Criteria (even before development is started).
13) Do not accept a change in requirements while a card is being played. Work already done is anyways done.
14) Each build output should be for a modular deployable unit.
15) Stateless on server-side is good.
16) Common sql injection scenarios, etc should be safe-guarded against
17) Use language features - dont rewrite code from scratch that is already available as part of the core language
18) Keep up-to-date on language enhancements + new tools to do a better job of old problems
19) Each sprint should achieve a value-add to the deployable product. (In some rare cases, this need not achieve a direct business benefit - it could also be targetting fixing of NFRs or reducing tech debt, etc)
20) Keep tech debt documented as well as in check ie do not let it build up. Constant care is required to maintain a healthy codebase (this goes for team practices as well)
21) (DB Design) Two-phase commit in case of domain usage/necessity
22) Data integrity should never be compromised - ever!
23) Support for multiple versions of API services needs to be kept to a minimum possible.
24) DB and HD administration should be kept up constantly. Dont allow the db to build up tremendously.
25) Review what goes "over the wire" when designing/coming up with a solution for a specific requirement
26) Schedule Pre-IPMs and retros to achieve both team inclusivity as well as to blow off steam.
27) Tech huddles are vital for KT - this can also be across roles (not just devs)
28) Security is a big aspect of programming - and should be on the radar right from the beginning.

Why is HTTP Streaming a joke on Rails 3.1?

2011-11-18T05:13:00.001-06:00

I have been trying to convert a Rails 3.1 app to use HTTP Streaming. The interesting fact about this app, and others that I have been working on for the past few years is that most of the models are ActiveResource models - not ActiveRecord. In this context, having the controller logic evaluated lazily takes on more importance - but the phrase "Rails 3.1 supports HTTP Streaming" leaves a lot to be desired.

I first tried to follow Ryan Bates' excellent rails cast. The tips are good, and I was able to get streaming working in a basic rails app ie one that uses ARecord quite easily. The key to streaming is the lazy evaluation of any logic that takes time - namely hitting the database (ARecord) or webservices (AResource). For ARecord, the Rails developers have piggy-backed on ActiveRelation's proxy (scoped) method for lazy evaluation. This is fine for ARecord. But does not work for AResource. This is the place where Rails breaks down.

What would have been better is if Rails had implemented a way for ALL such instance variables to be defined inside a lambda or Proc (in the controller) - so that the lambda is defined but not evaluated right then and there. And the second piece of the puzzle would have been for the getter of these instance variables (usually used in the views) to automatically evaluate the block the first time around that they were invoked (should only do the eval once).

One other thing to note is that any logic (instance variables that get assigned in the controller using the '@' symbol) that depend on the above lazy loaded values/attributes have to also be lazy loaded. This can be done using the method outlined here (this is one of the cleaner methods - and I am not sure whether the author was the first one to do it this way - I found other sites with the same way of tackling the problem).

Active Record Validator

2011-10-29T23:22:00.003-05:00

As your rails project sees multiple releases, you probably have new migrations that were run on your database. This coupled with Rails' default (IMHO, incorrect) attitude of not enforcing foreign key constraints at the db level might cause your old data to become invalid based on new ActiveRecord validations (unless your incremental migrations have taken this into consideration and massaged your data correctly). In our project, we had created a home-grown solution called ActiveRecordValidator. This would find all records in each table in your database (it was intelligent enough to handle STI, etc and also had a mechanism where you could specify some tables that you did not want to process - for eg audit tables, etc), and invoke valid? on them. If there were any errors, these were printed out to stderr. Progress was printed to stderr. This proved very useful for us, even in development/test so as to validate that all our reference table data was correct. When run against a snapshot of our production database (we setup a run every week against this snapshot data), it would inform us of data that would become invalid when the next production deploy happened. Our development iteration had an ongoing story card to fix such invalid data after talking to our SMEs. Since this code was done for a client, I could not open source it, but found something similar: active_sanity.

Application constants and accessing these from within Rails.configure block

2011-10-29T23:02:00.002-05:00

Around 4 years ago, when I was on my first ruby project, we came up with a solution to extract all application constants into a yaml file grouped by the rails environment. This concept was not new - but I did not find any plugin that had this functionality. After some time, I came to know about app_constants from my colleague, Leornado Borges. I have since used this gem in a couple of projects, but am missing one of the functionalities that our home-grown solution had: Our plugin had a method that could be invoked for any environment (we were calling this as part of our rspec suite for every CI run), that would take the aggregate of all the keys in every environment, and list out those keys that were not present in all the environments. This provided us with a way to quickly determine if a developer had added a key, but had forgotten to add it into the test/cucumber/uat/production environments. This was also invoked during deployment to production - so that if the yml file is being overwritten by your deployment tool, then we can be sure that any missing keys are caught during your deployment window - rather than at runtime by your user! (I think I will fork app_constants and implement this functionality - unless leonardo beats me to it!)

Note: If you are using this gem to extract your constants, but are using these constants within your environment initialization file, you might get an error. The way around this is to extract those specific lines into config.after_initialize blocks.

For eg:

    config.after_initialize do
      # Enable serving of images, stylesheets, and JavaScripts from an asset server
      config.action_controller.asset_host = "http://#{AppConstants.ASSET_HOST}"
      config.action_mailer.asset_host = "http://#{AppConstants.ASSET_HOST}"
    end

ActiveResource filter params

2011-10-29T21:22:00.002-05:00

Rails provides a way to filter action controller params from being logged. But if your app is talking to a webservice and passing these sensitive parameters, and you have turned on logged for ActiveResource::Base, then these sensitive params are anyways logged! To overcome this, I wrote this snippet:
https://gist.github.com/1319094.js
This can obviously be improved by separating out the params to be filtered between your ActionController::Base params and ActiveResource::Base params - but I leave that up to you...(this works for me)

Yaml - Yes? No? Maybe?

2011-09-24T03:04:00.002-05:00

Another interesting thing I found out this past week: the default yaml ruby library auto-converts keys with yes/true/ etc to the boolean true. I kind of expected that yaml might play around with the values - but I didnt expect that I would find such behavior with keys! Anyways, the way to fix it is to explicitly make the key a string!

yaml and japanese - translate.rb

2011-09-24T02:55:00.000-05:00

I had an interesting problem which grew incrementally a couple of weeks ago - and I wanted to share the experience. The problem statement was simple enough: we needed to internationalize the application that I am working on. Since this seemed simple enough with the rails-i18n gem, I wanted to generate the corresponding yml files from a master one automatically by invoking the google translator service. This seemed simple enough - but there was something I did not take into account. And that was: that the default yaml library does not have the ability to store content in languages like japanese. The way it bypasses this is to convert all such characters into unicode and store it (while writing to the yml file) - but this does not play nice when the same file is read in by rails. Enter the ya2yaml gem. This gem was written (not by me) to solve exactly this problem! Finally, I could invoke the google translator service and write the yml file directly - and my site now shows up with katakana characters correctly! To help others with the same problem, here is the code.

junitRunner

2011-03-22T19:25:00.000-05:00

I have almost given up on working on Java for the past 3+ years, but I am currently assigned to a project that is heavily into Java. When I landed here, I was pleasantly surprised to see most of the devs using Eclipse. The daily workflow involved running a lot of junit tests, and when I was pairing with a dev, I noticed that he ran the tests using the mouse (all tests in a suite), and then reran only a specific failed test - using his mouse. This was exactly the usecase for the junitrunner eclipse plugin!

Stubbing constants for tests

2011-03-18T22:49:00.000-05:00

Yesterday, I read a blog post about stubbing class constants for testing. I think using the before(:all) and after(:all) hooks are not the best way to achieve this. The reasons being that:
1) This happens for all test methods in that spec file - and is not limited to the [typically] few test methods that do need overridden values.
2) Ruby has an elegant mechanism with closures and 'yield', which is not being used.

Instead of the method suggested above, IMHO, this is better:

The above code block ensures that as soon as the block returns, the constants are reset to the old values thus making each test clean up after itself.

Keep your local git checkout clean

2011-03-17T12:23:00.001-05:00

In my previous project, active development always happened in trunk except for certain scenarios: spikes, fixing bugs on production release branches, completing some unfinished work on the next production release branch, etc. This caused a lot of our dev pairing machines to have multiple branches. And since we used to rotate pairs very often (every morning), whenever I went to pair on a different machine, I would first run this script:

The reason was very simple. When I am working on master, and the machine had multiple other branches (some of them having local changes, while others not), each time I did a push to the remote origin, it would complain (since git tries to push all local branches that track remotes, by default). Of course, we could have written our commit [rake] task in such a way that it deciphered the current branch and only pushed that. But, IMHO, its better to have a clean workspace rather than finding ways to bypass it. I know others who have a different opinion - the most common one being git is a DVCS and that's how it is supposed to work. That may be the case - everyone is entitled to their opinions. Mine is just that my workspace should be clean.

The above script will go through all the local branches, and pull from their respective remotes. If there is no merge conflict (by parsing the status message), then it would revert back to the master branch and delete the other local branch. This step is repeated for each locally existing branch.

This script and some more that I used to loop through and maintain multiple git repositories are shared through my github account.

Destroy does not actually mean destroy!

2011-03-16T23:33:00.000-05:00

When I started learning Ruby and Rails, I was fascinated with the rails scaffold generated code. There I was typing away to glory to come up with a webpage on a Java stack barely managing to keep it following a pattern, and here was a simple command line script which when invoked with the correct parameters, would generate the same for me in Ruby - wow!

And then I learned the first step - dont trust the generated code implicitly. Case in point: we had a [business/domain] scenario where the destroy action on an ActiveRecord would need to verify that some condition was satisfied, and if these conditions were not met, the destroy action would raise an error. The scenario in our case was that the entity was in use in some other relationships, and until those were deleted first, this entity could not be destroyed.

For the case of the entity creation, Rails provides a convenient callback mechanism called validations. The same is the case for updates. But for destroys, Rails assumes that all is well. This is carried on into the code generated for the controllers specifically for the destroy action. It just returns a 200 status code. But, as evidenced in our case, that is not always the case, and we had to manually edit all such scaffold-generated code to "fix" the issue.

Backing up Firefox and Thunderbird profiles

2011-03-16T23:13:00.003-05:00

I have been using Firefox and Thunderbird for the past 10 years or so - since the time when hard drive space was a precious commodity. As I started using more than one machine (or when I completely trashed my OS and had to reinstall it from scratch), I came across a need to copy and maintain these profiles. Since I wanted them to be in some "known" location, and since both these excellent mozilla products provide a way to edit an ini file and thus "move" the profile folders, I have been doing that as well for the past several years. (For more information on where to find your profile directory for your OS, and what to change, please look here.)

I ran across an initial version of this script here
As I started experimenting with bash scripts, I enhanced the above script (that would check if the program/application was running currently, and if not, it would do some common house-cleaning within these profile folders), with some new functionality, like delete some cache folders, etc. This cleanup should never be run when the application is in use - so be warned. Once the cleaning was done, I also added a cmd-line switch that would go ahead and create a tgz file for backups.

In case anyone is interested, here is the script

This script has been written in bash, and so can be run on windows if you use cygwin.

You can of course take the script and modify it as you wish. Just be warned (again) that if you do edit it, dont remove the check for skipping the house-cleaning if the application is running. Also, if you end up making any changes, I would appreciate it if you could send me the changes to be incorporated into my copy.

Git > TrueCrypt > Dropbox: A truly geeky way to backup your home folder across machines and OSes

2011-03-16T22:53:00.000-05:00

I was searching for ways to backup and sync up my profiles across different machines and across different *nix based OSes that I use (Mac OSX and Ubuntu). The stuff I wanted to sync was stuff like my .zshrc, .vimrc, .irbrc, .ssh, etc. At the same time, there were my firefox and thunderbird profiles that needed to be synced up as well (disregarding the fact that I use some add-ons for these which are OS-specific). Since some of these folders also contained sensitive information, any storage/sync mechanism that I arrived at would need to hold the data in an encrypted fashion. Lastly, I thought it would be good to have a historical backup of these same files/folders.

When I was in India on an assignment for my employer (Thoughtworks), I had some time during the evenings to play around with these requirements and came up with a geeky solution. I would create a git repository (most of the stuff I was storing were ASCII files, not binaries). This was important since git does not scale well for binary files. Next this git repository would go into a TrueCrypt container which was configured with sufficient space. Finally, the whole TrueCrypt container [file] would go into my DropBox account. Done! This would solve all my problems (at least I thought so at that time, but...)

There are some obvious limitations of this approach:

The TrueCrypt container once created, cannot be dynamically made to expand
Even if the TC file were recreated with a higher storage capacity, it would still need to adhere to the size of my dropbox account (they currently give 2GB free for the basic/free account)
Syncing a 1.5GB file across three machines - all of them wireless - was seriously overloading my home DSL setup!

Enter SparkleShare - the syncing software where you can store the data not on the cloud, but on your own drives! This was featured on one of the ubuntu blogs a few days ago - and it uses the same pattern for storing the data: your data/files/folders are actually encrypted (in lieu of my TrueCrypt step) and stored inside a git repo on whatever network storage device you configure! Yay!
The only hiccup right now is that their software is still in beta for both Mac and Linux - but maybe I will try it out sometime soon and post my review/experiences.

externalizing configurations across different rails environments

2011-03-07T06:29:00.000-06:00

In all the rails projects that I have worked on during the past 3 years, I have come across a pattern where some configurations need to be externalized. These configurations could include sensitive information (authentication passwords for a 3rd party system), application runtime information (like number of items to show in a paginated list), environment-specific information (like development vs uat vs production). To accomplish this, it is very trivial to write a Registry module that would load up a few yaml files into a hash in memory. Each file is loaded and if the same key exists from a previous file, the new value would super-impose the one in memory - so that ultimately, there is one "master" copy that the application runtime would use. (This concept has also been prevalent for more than a decade in the Java world - I used it in 2000). As I said before, this is pretty trivial to write. While browsing in github, I came across this project: https://github.com/lukeredpath/simpleconfig - which has almost exactly the same features, in a cleaner way! Now onto forking that project and converting it into a gem...

Consolidating rails migrations

2011-02-28T22:55:00.000-06:00

With the meteoric rise in adoption of rails as a framework to develop business applications, we as developers need to also remember that these same codebases need some amount of love/house-cleaning from time to time. One of the areas that I want to highlight is database migrations.

As new releases of the app is deployed to production, and development of new features (needing new migrations) continues, it would be a good practice to consolidate previously run migrations into a single file. The idea is simple:

Use the [default] VERSION environment variable to specify the migration till which you want to consolidate
Drop, create and migrate the database till this version
Copy the resulting schema.rb file into the migration (replacing the old content of the 'self.up' method) in migration with that same version number
Typically you will never go below this migration, so raise an 'ActiveRecord::IrreversibleMigration' exception in the corresponding 'self.down' method
Delete older migrations (prior to the specified one)
Commit into your scm

At my previous client, we implemented a rake task to accomplish these steps in an automated fashion. Once a release went to production and after a couple of days of "settling down", this rake task was run with the version of the [production] db.

The advantage of doing this is:

Typically, as design evolves, you might rename/drop columns, drop tables, add seed data, etc which are all collapsed into a single step
The number of files that need to be evaluated and run (each time the db is recreated, typically, when running tests) is smaller, and thus taking [fractionally] less amount of time
Forces you, the developer, to maintain seed data outside migrations

Rails tricks: Require correctly

2011-02-27T02:03:00.003-06:00

On one of my previous projects, we noticed some strange (to us) behavior: any extra spec behavior (extra methods, config.before(:all), etc) was being loaded/defined more than once while running the whole suite of tests, but when we tried to run a single spec class (ie all tests defined within one file), this behavior was not evident. Digging further, we found that this was caused by requiring the spec_helper. As each spec file is being "processed", the require statement at the top of the file got triggered, and when a new file (at a different relative directory) was encountered, the spec_helper line would usually trigger this scenario. For eg:



spec/a_spec.rb: require File.dirname(__FILE__) + '/spec_helper'

spec/models/b_spec.rb: require File.dirname(__FILE__) + '/../spec_helper'

spec/lib/level1/c.rb: require File.dirname(__FILE__) + '/../../spec_helper'

To solve this, we resorted to using the fully expanded path to the spec_helper - since this would resolve to the same file each time. In this case, the above files were changed so:



spec/a_spec.rb: require File.expand_path('spec_helper', File.dirname(__FILE__))

spec/models/b_spec.rb: require File.expand_path('../spec_helper', File.dirname(__FILE__))

spec/lib/level1/c.rb: require File.expand_path('../../spec_helper', File.dirname(__FILE__))

UPDATE: My good friend and colleague Brian Guthrie corrected me saying that this situation can be avoided if the load path is set correctly (to include the spec folder) through the rake task before actually invoking rspec. That sounds reasonable, but I have yet to try it out - and unfortunately, I have moved on from the project where this was sorely needed.