29 Steps

Introducing Snipplet

Sat, 11 Feb 2012 15:56:00 +0000

Snipplets is a code hightlighter built using Rails and uses pygmentizer to highlight code blocks.

It allows for code embedding within HTML pages as well as oEmbed links which is an open format used to display rich content on HTML pages.

Snipplets runs on Heroku and processes the code snippets by sending them to Resque which adds it to a conversion queue. Extra configuration is used to only create the workers on demand as and when new code snippets are added to the queue.

To see it in action please visit: Snipplets site

A sample is shown below:

FFMPEG 2012 update - Installing frei0r plugin

Tue, 17 Jan 2012 21:21:00 +0000

frei0r is a plugin which allows for video effects and can be built into the ffmpeg program by enabling the '--enable-frei0r'option during compile. That requires building frei0r first.

I tried building frei0r from the git source and ran into several issues on Ubuntu Oneiric 11.

The first problem I came across after running './config.sh' to generate the makefile followed by 'make' is the following:

libtool: Version mismatch error. This is libtool 2.4.2, but the
libtool: definition of this LT_INIT comes from libtool 2.4.
libtool: You should recreate aclocal.m4 with macros from libtool 2.4.2
libtool: and run autoconf again.

To proceed with the build I did the following within the source directory:

aclocal

autoreconf -i -f

./configure

The second issue occurs when you try to compile frei0r with ffmpeg. The Makefile generated by frei0r does not include linking with the system libc library as indicated by the lack of LDFLAGS '-ldl' during the make output hence causing the compile crash later in ffmpeg with the following errors:

undefined reference to 'dlclose'
undefined reference to 'dlopen'
undefined reference to 'dlerror'

To solve this just link frei0r with the libc library during configure and compile like so:

./configure LIBS=-ldl

make -j3

Next during the actual ffmpeg configure after enabling the --enable-frei0r option, you would still need to enable the 'extra-libs' flag as so:

( from within the ffmpeg source directory )

./configure --extra-libs="-ldl" ... -enable-frei0r

There is also a prebuilt version of frei0r in the synaptic library if the above does not work. This has been tested with the latest ffmpeg git version and version 1.3 of the frei0r plugin from its git source.

If anyone out there has a better solution to compile frie0r from source please feel free to contribute. Additionally, I am also looking into how to compile frie0r on OSX Snow Leopard.

More or Less Heroku

Wed, 11 Jan 2012 21:34:00 +0000

LESS is a dynamic stylesheet language which I just discovered recently. It is powerful enough to allow for variables; mixins and functions.

In a recent Rails 2.3x project which I had to deploy to Heroku, it uses a rails plugin called More to generate the stylesheet from an existing LESS template. However the issue is Heroku uses protected directories and the More plugin tries to delete the current generated stylesheet whenever you do a deploy.

This is due to the following block of code within more/rails/init.rb

config.after_initialize {
  Less::More.clean
  Less::More.parse if Less::More.page_cache?
}

The line Less::More.clean will attempt to remove the stylesheet from the public/stylesheet directory which will cause a permission error since Heroku uses a readonly filesystem and does not allow for file deletion.

The second line Less::More.parse re-generates the css everytime if Less::More.page_cache? is true. In the case of Heroku, the plugin is looking for the HEROKU_ENV variable which is no longer valid, causing the heroku? method to return false and (not heroku?) to return true. Since page_cache_enabled_in_environment_configuration? calls on Rails.configuration.action_controller.perform_caching which is set to true in production, causing Less::More.parse to re generate all the css files.

def page_cache_enabled_in_environment_configuration?
  # returns true in production env
   Rails.configuration.action_controller.perform_caching
end

def page_cache?
  (not heroku?) && page_cache_enabled_in_environment_configuration?
end

# Returns true if the app is running on Heroku. When +heroku?+ is true,
# +page_cache?+ will always be false.
def heroku?
  !!ENV["HEROKU_ENV"]
end

Hence the css gets regenerated causing an error.

There are two approaches to solving this depending on the version of the More plugin you are using.

If you are using the old More plugin, which is version less than 0.1.0, then you would need to add the following config var to your heroku variables.

  heroku config:add HEROKU_ENV=production

This would set Less::More.parse to false hence not generating the css.

If you are using the latest version from github, you just need to generate the css by running rake more:generate and then committing the css into the deploy. Make sure that the directory holding the generated css is not in your gitignore.

Next use a skip_before_filter to stop Less::More.generate_all from running during deploy:

skip_before_filter :generate_css_from_less if Rails.env.production?

I finally upgraded to the newer version of More as I don't think the older version still works given the fixes suggested above.

Being AGILE in 2012

Tue, 10 Jan 2012 12:58:00 +0000

At 29 Steps we occasionally receive recommendations for new work through previous clients and there is no greater privilege
than to be recognised for the quality of the work you have done previously for your details to be passed on to a third party.

With the weeks leading up to XMAS, I had such an opportunity but after a series of email exchanges with the potential client
I soon realised it would be difficult working with him and hence I turned it down. The debate or argument centred around the preferred set
of working practices and once again AGILE was in the firing line.

The client, whose name I shall abbreviate to A.T., disagreed with our working arrangements. He was happy with most of the work
to be carried out remotely on the provision that I maintain a direct, open line of communication using Skype so he could walk me through
a set of specs which is 'fluid' and changeable at the last moment and also to provide some training in Ruby on Rails

My reply was to separate the two events and my concern was with the idea of 'changeable specs'. AGILE proposes that you need at least an initial
set of clear specs before you can even begin any development which gets refined throughout the process

(From past experiences that usually says to me 'I do not have a clue on what to do next but lets get the ball rolling and see how it goes'. )

When I mentioned my concerns to him this was his reply verbatim:

"My priority is not to learn Rails.
 
I can specify broadly what I need, so yes the requirements might change.
 
I appreciate not having a totally clear spec makes estimating duration complicated which is why Im happy to pay an hourly rate, and accept that certain things may need re-doing, and refactoring as the system evolves.
 
I have been a developer for 20 years, and have managed teams of programmers from 5 to 25 and have always worked in an industry which required a very agile approach like this."

Note the use of the phrases 'agile approach' and the more startling 'developer for 20 years'.

Its by reading through the email again that something
profound hit me: how could something such as AGILE, a set of best practices, be so misrepresented and misinterpreted in 2012?

Thats when I realised there is a fundamental difference between a developer's understanding and perspective of Agile compared to that of other stakeholders.

Why is it so bad?

One of the main causes is the way AGILE has been potrayed over the years, not as a set of practices or principles but as a 'one size fits all' approach to any issue that does not fit in with the norm. In A.T.'s case since he did not have a clear list of specs, he believed he could formulate them along the way by being 'AGILE' or working in an 'AGILE' manner ( the misunderstanding here is that AGILE allows for iterations and turnarounds provided you have some specs to work with rather than chop and change whenever you feel like it ).

Some individuals or companies like A.T. even go as far as hailing AGILE as their saving grace or bandied the term just to make themselves look savvy and in touch.

Before we go any further let us take a step back and examine this in perspective and ask ourselves this question:

What exactly is AGILE?

Agile to me means a set of best practices and guiding principles to facilitate a fluid, manageable and less error prone software development which allows for both the stakeholders and developers to come together to make a great product

The keyword here is 'principles'. Its NOT a set of RULES to live by or followed completely.

You pick out or adapt the relevant parts to fit your own development process and keep refining it until you are comfortable enough to do it without having to consciously think about it.

Just because it works for person or company A does not mean it will work for you in the same way or you have to do as they do. My plea is to read these AGILE success stories with a pinch of salt rather than follow it blindly.

AGILE is meant to be fluid and adaptable which means it allows you to make development changes rapidly since you are releasing at regular intervals.

Some clients I have come across such as A.T. above will seize the opportunity to cramp as much work as they can into each release cycle with n cycles of iterations until they are satisfied enough with the end result to justify paying you.

If you already have 20+ years management experience under your belt you would know that AGILE proposes or allows for the workflow to be broken down into simple manageable user stories to be developed iteratively and not for addressing all your problems at one big go. Please stop abusing the AGILE term to justify your unreasonable behavior.

Last but not least, AGILE is associated with TDD/BDD practices which in my opinion should be separate events.

It doesn't mean that as soon as I burst open Textmate to start on a client project, out comes rspec, cucumber and all the CI suites I can think of in order to be 'AGILE'. It depends on the nature and context of the project. I think I can at least get some agreement that it will be overkill to have a full integration test on a static one page website.

Likewise, some of the clients I have come across are not interested in pretty formatted PDF/HTML tests reports to read as they are comfortable with your ability to justify hiring you in the first place to solve a problem and let you get on with it.

At 29 Steps we appreciate and like to achieve a healthy balance as we believe in regular testing and having a strong test suite to further backup your work but please do it in context not because 'you have to' or as a rite of passage to become a rails / ruby developer

Another issue is with self-styled or industry recognised AGILE gurus who 'preaches' to you the must have tool to use if you want to stay AGILE in your development life span. A recent blog post I read on a renown website about testing introduced a new acceptance testing framework in order to improve your integration test suite but out of all the available features they recommend only 1 would have benefitted me so am I suppose to spend hours re writing my entire test suite to use this one tool just to fit in with what the gurus recommend?.

For 2012 let us developers reclaim the term AGILE and for once embrace it for what it truly is - a useful set of principles to have and to learn from
but not RULES which hinder what we love to do best - to play, create and make great software.

Unless of course you relish a challenge with clients such as A.T. and D.F ( friend of A.T who has the same outlook on AGILE ) and possess a lot of time on your hands.

As an aside, Zed Shaw, the author of Mongrel 2 server, has created a website which highlights some the thoughts I have mentioned so far but to a slightly greater extreme: http://programming-motherfucker.com/

DNode + Rails 3 = Realtime web

Thu, 29 Dec 2011 18:01:00 +0000

I recently had the opportunity to work on a project involving two totally separate skillsets: Rails 3 application running on top of a node.js application server which handles and distributes web sockets requests. In addition throw in e commerce and multi user functionality

The end product is SquareMarket.com which encompasses all the above which is why I have been keeping my head down for over a month since my last post. Below are some screenshots of the app in action.

The project has been a revelation for me in terms of the work that has to be done to get websockets running in production mode and to make it secure by running ssl.

The realtime aspect is handled by a node.js script using dnode which has built in support for websockets using socket.io

DNode 'Object.keys not found' error

Thu, 10 Nov 2011 12:17:00 +0000

In a recent project I had to integrate DNode in a node.js application which broacasts realtime events to connnected clients in the browser. For those who wonder what DNode is, its a node.js module which uses socket-io underneath to carry out RPC method calls between your browser and your backend application. More details about DNode can be found here: DNode github

In older browsers such as Firefox 3, it complains that a method called 'Object.keys' is not found. This is caused by the inclusion of 'dnode.js' in the browser script which makes a call to Object.keys.

To fix this I added in an additional piece of javascript from the older branch or version of dnode 0.5 as follows:


if (!Object.keys) Object.keys = function (obj) {
    var keys = [];
    for (var key in obj) {
        if (obj.hasOwnProperty(key))
            keys.push(key);
    }
    return keys;
};

if (typeof Object.create === 'undefined') {
    Object.create = function (o) {
        function F() {}
        F.prototype = o;
        return new F();
    };
}

Drop the above into an external js file, include it and restart dnode again and provided you are using the latest version of jquery the above should make dnode cross browser compliant.I have tested it on firefox 3 and 7; chrome and safari on the mac.

Due to significant rewrite of the dnode library, this only affects the latest master branch and the npm version. If you are using an older version such as 0.5 the changes above has already been merged in to the core. For more details refer to the git commits and this issue here

Non obtrusive contact form in Rails

Thu, 10 Nov 2011 11:29:00 +0000

In my development experience with Rails I always come to the point when I have to make some kind of contact form or contact us page and its always the part I leave to the very end for the reason that it is tedious ( write mailers; setup mail; views; controllers; tests ), sometimes buggy and difficult to test ( if you are collaborating on a project and the client only wants you to use specific testing gems ); and repetitive ( you will have to repeat the entire process again in another project.)

Because of this I have developed or adpated a simple system so I have a base of standard template code I can use between projects which I know does work.

For my contact forms setup, the first thing I do is to create an ActiveModel object or class called Contact or ContactForm depending on the what names are available in that project and drop it into the app/models folder. It will be prepended with Contact so I know what it is and does.

Why use ActiveModel and not ActiveRecord? In some circumstances I come across, there is alreay a model called Contact in some projects which represent a different entity of the system altogether. Also I don't want to use ActiveRecord unless I know I am saving data to the DB. In the case of a contact form I'm only interested in getting details from the form and mailing it - if there is a need to store client queries, another plugin or system will need to be used to handle that aspect as there is a world of difference between sending data in a contact form and filtering and storing client queries.

Once the model is setup I normally add in basic validations for the commonly used fields such as name, email, subject matter. It is imperative to keep these fields to a minimum or to just the required fields. For me personally, I just want the client to provide the basic information which I need to get in contact with them again. The code below illustrates this:


class Contact
  include ActiveModel::Validations
  include ActiveModel::Conversion
  extend ActiveModel::Naming

  attr_accessor :name, :email, :phone, :alternative_phone, :subject, :question

  validates_presence_of :name, :email, :subject, :question
  validates_format_of :email, :with => /^[-a-z0-9_+\.]+\@([-a-z0-9]+\.)+[a-z0-9]{2,4}$/i
  validates_length_of :question, :maximum => 2500

  def initialize(attributes = {})
    attributes.each do |name, value|
      send("#{name}=", value)
    end
  end

  def persisted?
    false
  end
  
end

To include the above into a form, just initialize it and if you are using the client-side-validations gem, the errors will still show up if some of the required fields are missing or malformed:


<%
  @contact = Contact.new
%>


<%= form_for @contact, :remote => true,:validate => true, :url => contact_path do |f| %>


        Your Name *

        <%= f.text_field :name, :class=>'text' %>
  


<% end %>

Assuming you have a contact_path to submit the form to, you can pass the contact object into a mailer once the validations have passed:


@contact = Contact.new(params[:contact])

unless @contact.errors.any?
  ContactMailer.query(@contact).deliver
end

Notice that in the form code sample above, I passed in remote as true to utilize the UJS capabilities in Rails 3. In the next article I will show you how to utilize this to build a less obtrusive contact form on the front end to integrate it with what we have so far.

Rails 3.1.1 + Asset Pipeline

Wed, 26 Oct 2011 19:32:00 +0000

I had the opportunity to try out the new Rails 3.1.1 asset pipeline and although I love the way it works, it does take some getting used to and understand the way it works.

The only issue I have with it is the sluggish speed in development mode - Sprockets compiles each asset individually and if you are using engines with nested stylesheets and js files, it grinds to a halt.

After much googling around, I found this gem

gem 'rails-dev-tweaks', '~> 0.5.1'

I then added the following default config into the application.rb file:


config.dev_tweaks.autoload_rules do
      keep :all

      skip '/favicon.ico'
      skip :assets
      skip :xhr
      keep :forced
 end

Restart the server and you should find that your assets are loaded normally.

Rails 3.1.1 + Refinery CMS + OAuth

Wed, 26 Oct 2011 19:17:00 +0000

I just completed a project where I had to use the edge branch of Refinery CMS in a Rails 3.1.1 application.

Because OAuth is used as part of the user login, I incorporated both twitter and facebook using the Omniauth middleware. However, the twitter oauth call keeps failing with 'uninitialized constant OAuth'.

I added to the gemfile:

gem 'oauth'

bundle install and it worked after restarting the server.

RefineryCMS pages have a greedy or catch all path which will throw a page not found error if you make your auth calls using '/auth/twitter' for example.

To overcome this, set up a route at the top of your routes file like so:

match "/auth/:provider" => "errors#error"

What the above route does is to call the method "error" within the "errors" controller. The code within the errors controller look like so:


class ErrorsController < ApplicationController
  def error
    render :file => "#{Rails.root}/public/404.html", :status => 404, :layout => false
  end
end

ErrorsController error method throws a 404 error which will then cause the Omniauth middleware to forward the request onto the OAuth provider by passing the Refinery catch all route.

Hope the above helps someone working with or in my case struggled with Refinery and Rails 3.1.1

Using node.js and Twitter Node - controlling the flow of stream

Fri, 14 Oct 2011 09:35:00 +0000

This is the second part in the series on using node-twitter plugin with sockets.io in a node.js app to stream real time data streams from twitter.

Node-twitter is an npm plugin which you can put into your node app to stream real time feeds from the Twitter Streaming API. The really cool part about it is that it supports multiple streaming api methods e.g. you can either 'track' a term or search for it in real time.

The article below assumes you have installed node-twitter using npm:

npm install twitter

Once a stream object is instantiated with the node-twitter plugin and a listener method attached to listen for any new incoming data, it stays within an event emitter loop which means if you need to stop the stream you will need to get a reference to the stream object and call destroy on it. A code example will illustrate this point further.

  
   var twitter = require('twitter');
   
   var my stream_obj;

   var twit = new twitter({
	consumer_key: 'STATE YOUR NAME',
	consumer_secret: 'STATE YOUR NAME',
	access_token_key: 'STATE YOUR NAME',
	access_token_secret: 'STATE YOUR NAME'
   });

   twit.stream('user', {track:'nodejs'}, function(stream) {
     // store the stream object elsewhere so we reference it later.
     my_stream_obj = stream;

     stream.on('data', function (data) {
       sys.puts(sys.inspect(data));
      });
   });


   // to cancel the stream later
   my_stream_obj.destroy();

Note that if you cancel the stream all data stops and if you are broadcasting twitter feeds to multiple clients using just one twitter object as above, it will stop all connections. The approach I would use in this case is to store the reference of each stream object into a hash and track it using a using term or session id.

This is important as I have found out to my expense when I deployed to a live system recently, I did not cancel the stream after a user has left the site and over time this caused an out of memory error, crashing the server.

If you are building a multi client or user streaming application, please make sure you close the stream after the user has left your site.