29 Steps

Using Pygments.rb within Rails 3.2

Sat, 19 May 2012 10:52:00 +0000

Snipplets, the syntax highlighting app I built a month ago, makes strong use of Pygmentizer to highlight user submitted code segments. It does so by making a POST request to an external pygmentizer app located on google app engines which then makes a successful request back to the app on completion of the highlighting.

While this serves the purpose of the app, it has some drawbacks. Snipplets relies on a Resque queue to wait for the request to be complete which means the user ends up with an inconsistent view of the codesnipplet; also what happens if the API service fails?

After watching the railscasts episode 207 which introduces pygments.rb I ported all the syntax highlighting code to using pygments.rb.

Whats kool about pygments.rb is that it still uses pygemntizer but the program itself is embedded within the gem and your app using ruby-python and libffi ( through the ffi gem ) which means any environment that supports or runs python can run pygments.rb and that includes Heroku.

However I ran into this peculiar bug whereby pygments.rb crashes the entire application server ( both Mongrel and Webrick as shown in the Railscasts) with a killed signal message on the console when running Pygments.highlight method.

If you are running into the same issue, try switching over to Thin which works for me without any issues. As yet I still cannot ascertain the error whether it is system dependent or not and ruby-debug did not thrown up any unusual bugs. Could have something to do with asynchronous processing but will look into it in a separate post.

If you are new to using pygments.rb and have just switched over you might notice that the line numbers have disappeared completely if you used to have line numbers enabled. This is off by default in the gem and can be enabled by passing the line numbers options to true inside the Pygments.highlight method.

Below is a short snipplet describing some of the workarounds for pygments.rb as discussed above:

The second issue you might run into when deploying onto Heroku is the version of RubyPython you are using. If you get errors such as 'lexer cannot be found' then it is related to RubyPython as pygments.rb cannot pick up on the Python interpreter. To do so simply create an initializer file ( I called mine 'rubypython.rb' and put it inside config/initializers) with the following code:

Essentially what the above does is to start a python interpreter session so that pygmentizer can run. I can't seem to find any other way to avoid this else the entire app will fail. The other gotcha is that it only seems to work with RubyPython 0.5.1 so I have the gemfiles locked down to that version. I tried it with both RubyPython 0.5.3 and the latest 0.6.1 to no avail - the cedar stack on heroku just refuses to communicate with rubypython.

Hope this helps someone with similar dilemma as pygmentizer is a great syntax highlighting tool.

29 Steps Shop live and March 2012 updates

Thu, 15 Mar 2012 21:32:00 +0000

February has been a fairly quiet month so we are using this free time to develop our own products.

We have been receiving requests for e commerce development work which is why we decided to learn and use Spree which is an open source e commerce platform. What we love about Spree is the ability to write custom extensions and there is a vibrant community of contributors out there. Also they have a really detailed set of guides to get you up and running at no time at all.

We have created our own shop to showcase this. The shop can be found at this link.The whole purpose of the shop is to say yes we can also do e commerce and if you need a bespoke / custom solution we can do it for you using the best open source tools out there.

The entire shop is built using edge version of Spree; running on Rails 3.2.2 and hosted on Heroku. It uses a set of extensions such as Spree Paypal gateway; Spree carrierwave to deal with S3 uploads on Heroku; and Essential Spree CMS for the custom pages.

We also learnt to use Deface to generate custom modifications to the templates ( which is the recommended way in Spree ) and also developed a small Featured Products extension on the homepage.

Why would you want your own e commerce shop / solution when there are plenty of ready made even free services out there to use? The simple answer to that is it depends on your requirements. Some sellers are quite happy with ready made solutions as they have only a small number of products which don't change frequently. My personal take on that is - if you choose to migrate your data away to another service you will have a job on your hands. That is the main issue I have with ready made services. Some services such as Shopify allows you to integrate your data through their API but it may so happen you pick and use a provider which doesn't have an API then what do you do? Re enter all your product info and images from scratch? Not for me.

I still think that having your own custom shop allows you maximum flexibility without the issue and headaches of data migration over platforms and it also allows you to customize it as you go along.

If you are interested in finding out more, please contact us for further details.

So ... what else is happening?

February alone is not just devoted to e commerce alone. We have also finishing off on our video encoding platform called Vide.IO. Screenshots and site coming really soon.

In addition we also found the time to rewrite Snipplets to use the Bootstrap CSS and JS and also enhanced features such as github integration through the github API so you can version control your own snipplets. We also added a social network touch to it and that should be completed over the weekend.

Last but not least, we are also developing a new rubygem for the Amazon DynamoDB. We are completing the specs for it at the moment so it should be launched fairly soon. The Amaze SNS rubygem has attracted over 24000 downloads as of today. We feel that an update is due soon as there has been some API changes.

Introducing Snipplets

Sat, 11 Feb 2012 15:56:00 +0000

Snipplets is a code hightlighter built using Rails and uses pygmentizer to highlight code blocks.

It allows for code embedding within HTML pages as well as oEmbed links which is an open format used to display rich content on HTML pages.

Snipplets runs on Heroku and processes the code snipplets by encoding it through RubyPython and pygments.rb

To see it in action please visit: Snipplets site

A sample is shown below:

FFMPEG 2012 update - Installing frei0r plugin

Tue, 17 Jan 2012 21:21:00 +0000

frei0r is a plugin which allows for video effects and can be built into the ffmpeg program by enabling the '--enable-frei0r'option during compile. That requires building frei0r first.

I tried building frei0r from the git source and ran into several issues on Ubuntu Oneiric 11.

The first problem I came across after running './config.sh' to generate the makefile followed by 'make' is the following:

libtool: Version mismatch error. This is libtool 2.4.2, but the
libtool: definition of this LT_INIT comes from libtool 2.4.
libtool: You should recreate aclocal.m4 with macros from libtool 2.4.2
libtool: and run autoconf again.

To proceed with the build I did the following within the source directory:

aclocal

autoreconf -i -f

./configure

The second issue occurs when you try to compile frei0r with ffmpeg. The Makefile generated by frei0r does not include linking with the system libc library as indicated by the lack of LDFLAGS '-ldl' during the make output hence causing the compile crash later in ffmpeg with the following errors:

undefined reference to 'dlclose'
undefined reference to 'dlopen'
undefined reference to 'dlerror'

To solve this just link frei0r with the libc library during configure and compile like so:

./configure LIBS=-ldl

make -j3

Next during the actual ffmpeg configure after enabling the --enable-frei0r option, you would still need to enable the 'extra-libs' flag as so:

( from within the ffmpeg source directory )

./configure --extra-libs="-ldl" ... -enable-frei0r

There is also a prebuilt version of frei0r in the synaptic library if the above does not work. This has been tested with the latest ffmpeg git version and version 1.3 of the frei0r plugin from its git source.

If anyone out there has a better solution to compile frie0r from source please feel free to contribute. Additionally, I am also looking into how to compile frie0r on OSX Snow Leopard.

More or Less Heroku

Wed, 11 Jan 2012 21:34:00 +0000

LESS is a dynamic stylesheet language which I just discovered recently. It is powerful enough to allow for variables; mixins and functions.

In a recent Rails 2.3x project which I had to deploy to Heroku, it uses a rails plugin called More to generate the stylesheet from an existing LESS template. However the issue is Heroku uses protected directories and the More plugin tries to delete the current generated stylesheet whenever you do a deploy.

This is due to the following block of code within more/rails/init.rb

config.after_initialize {
  Less::More.clean
  Less::More.parse if Less::More.page_cache?
}

The line Less::More.clean will attempt to remove the stylesheet from the public/stylesheet directory which will cause a permission error since Heroku uses a readonly filesystem and does not allow for file deletion.

The second line Less::More.parse re-generates the css everytime if Less::More.page_cache? is true. In the case of Heroku, the plugin is looking for the HEROKU_ENV variable which is no longer valid, causing the heroku? method to return false and (not heroku?) to return true. Since page_cache_enabled_in_environment_configuration? calls on Rails.configuration.action_controller.perform_caching which is set to true in production, causing Less::More.parse to re generate all the css files.

def page_cache_enabled_in_environment_configuration?
  # returns true in production env
   Rails.configuration.action_controller.perform_caching
end

def page_cache?
  (not heroku?) && page_cache_enabled_in_environment_configuration?
end

# Returns true if the app is running on Heroku. When +heroku?+ is true,
# +page_cache?+ will always be false.
def heroku?
  !!ENV["HEROKU_ENV"]
end

Hence the css gets regenerated causing an error.

There are two approaches to solving this depending on the version of the More plugin you are using.

If you are using the old More plugin, which is version less than 0.1.0, then you would need to add the following config var to your heroku variables.

  heroku config:add HEROKU_ENV=production

This would set Less::More.parse to false hence not generating the css.

If you are using the latest version from github, you just need to generate the css by running rake more:generate and then committing the css into the deploy. Make sure that the directory holding the generated css is not in your gitignore.

Next use a skip_before_filter to stop Less::More.generate_all from running during deploy:

skip_before_filter :generate_css_from_less if Rails.env.production?

I finally upgraded to the newer version of More as I don't think the older version still works given the fixes suggested above.

Being AGILE in 2012

Tue, 10 Jan 2012 12:58:00 +0000

At 29 Steps we occasionally receive recommendations for new work through previous clients and there is no greater privilege
than to be recognised for the quality of the work you have done previously for your details to be passed on to a third party.

With the weeks leading up to XMAS, I had such an opportunity but after a series of email exchanges with the potential client
I soon realised it would be difficult working with him and hence I turned it down. The debate or argument centred around the preferred set
of working practices and once again AGILE was in the firing line.

The client, whose name I shall abbreviate to A.T., disagreed with our working arrangements. He was happy with most of the work
to be carried out remotely on the provision that I maintain a direct, open line of communication using Skype so he could walk me through
a set of specs which is 'fluid' and changeable at the last moment and also to provide some training in Ruby on Rails

My reply was to separate the two events and my concern was with the idea of 'changeable specs'. AGILE proposes that you need at least an initial
set of clear specs before you can even begin any development which gets refined throughout the process

(From past experiences that usually says to me 'I do not have a clue on what to do next but lets get the ball rolling and see how it goes'. )

When I mentioned my concerns to him this was his reply verbatim:

"My priority is not to learn Rails.
 
I can specify broadly what I need, so yes the requirements might change.
 
I appreciate not having a totally clear spec makes estimating duration complicated which is why Im happy to pay an hourly rate, and accept that certain things may need re-doing, and refactoring as the system evolves.
 
I have been a developer for 20 years, and have managed teams of programmers from 5 to 25 and have always worked in an industry which required a very agile approach like this."

Note the use of the phrases 'agile approach' and the more startling 'developer for 20 years'.

Its by reading through the email again that something
profound hit me: how could something such as AGILE, a set of best practices, be so misrepresented and misinterpreted in 2012?

Thats when I realised there is a fundamental difference between a developer's understanding and perspective of Agile compared to that of other stakeholders.

Why is it so bad?

One of the main causes is the way AGILE has been potrayed over the years, not as a set of practices or principles but as a 'one size fits all' approach to any issue that does not fit in with the norm. In A.T.'s case since he did not have a clear list of specs, he believed he could formulate them along the way by being 'AGILE' or working in an 'AGILE' manner ( the misunderstanding here is that AGILE allows for iterations and turnarounds provided you have some specs to work with rather than chop and change whenever you feel like it ).

Some individuals or companies like A.T. even go as far as hailing AGILE as their saving grace or bandied the term just to make themselves look savvy and in touch.

Before we go any further let us take a step back and examine this in perspective and ask ourselves this question:

What exactly is AGILE?

Agile to me means a set of best practices and guiding principles to facilitate a fluid, manageable and less error prone software development which allows for both the stakeholders and developers to come together to make a great product

The keyword here is 'principles'. Its NOT a set of RULES to live by or followed completely.

You pick out or adapt the relevant parts to fit your own development process and keep refining it until you are comfortable enough to do it without having to consciously think about it.

Just because it works for person or company A does not mean it will work for you in the same way or you have to do as they do. My plea is to read these AGILE success stories with a pinch of salt rather than follow it blindly.

AGILE is meant to be fluid and adaptable which means it allows you to make development changes rapidly since you are releasing at regular intervals.

Some clients I have come across such as A.T. above will seize the opportunity to cramp as much work as they can into each release cycle with n cycles of iterations until they are satisfied enough with the end result to justify paying you.

If you already have 20+ years management experience under your belt you would know that AGILE proposes or allows for the workflow to be broken down into simple manageable user stories to be developed iteratively and not for addressing all your problems at one big go. Please stop abusing the AGILE term to justify your unreasonable behavior.

Last but not least, AGILE is associated with TDD/BDD practices which in my opinion should be separate events.

It doesn't mean that as soon as I burst open Textmate to start on a client project, out comes rspec, cucumber and all the CI suites I can think of in order to be 'AGILE'. It depends on the nature and context of the project. I think I can at least get some agreement that it will be overkill to have a full integration test on a static one page website.

Likewise, some of the clients I have come across are not interested in pretty formatted PDF/HTML tests reports to read as they are comfortable with your ability to justify hiring you in the first place to solve a problem and let you get on with it.

At 29 Steps we appreciate and like to achieve a healthy balance as we believe in regular testing and having a strong test suite to further backup your work but please do it in context not because 'you have to' or as a rite of passage to become a rails / ruby developer

Another issue is with self-styled or industry recognised AGILE gurus who 'preaches' to you the must have tool to use if you want to stay AGILE in your development life span. A recent blog post I read on a renown website about testing introduced a new acceptance testing framework in order to improve your integration test suite but out of all the available features they recommend only 1 would have benefitted me so am I suppose to spend hours re writing my entire test suite to use this one tool just to fit in with what the gurus recommend?.

For 2012 let us developers reclaim the term AGILE and for once embrace it for what it truly is - a useful set of principles to have and to learn from
but not RULES which hinder what we love to do best - to play, create and make great software.

Unless of course you relish a challenge with clients such as A.T. and D.F ( friend of A.T who has the same outlook on AGILE ) and possess a lot of time on your hands.

As an aside, Zed Shaw, the author of Mongrel 2 server, has created a website which highlights some the thoughts I have mentioned so far but to a slightly greater extreme: http://programming-motherfucker.com/

DNode + Rails 3 = Realtime web

Thu, 29 Dec 2011 18:01:00 +0000

I recently had the opportunity to work on a project involving two totally separate skillsets: Rails 3 application running on top of a node.js application server which handles and distributes web sockets requests. In addition throw in e commerce and multi user functionality

The end product is SquareMarket.com which encompasses all the above which is why I have been keeping my head down for over a month since my last post. Below are some screenshots of the app in action.

The project has been a revelation for me in terms of the work that has to be done to get websockets running in production mode and to make it secure by running ssl.

The realtime aspect is handled by a node.js script using dnode which has built in support for websockets using socket.io

DNode 'Object.keys not found' error

Thu, 10 Nov 2011 12:17:00 +0000

In a recent project I had to integrate DNode in a node.js application which broacasts realtime events to connnected clients in the browser. For those who wonder what DNode is, its a node.js module which uses socket-io underneath to carry out RPC method calls between your browser and your backend application. More details about DNode can be found here: DNode github

In older browsers such as Firefox 3, it complains that a method called 'Object.keys' is not found. This is caused by the inclusion of 'dnode.js' in the browser script which makes a call to Object.keys.

To fix this I added in an additional piece of javascript from the older branch or version of dnode 0.5 as follows:


if (!Object.keys) Object.keys = function (obj) {
    var keys = [];
    for (var key in obj) {
        if (obj.hasOwnProperty(key))
            keys.push(key);
    }
    return keys;
};

if (typeof Object.create === 'undefined') {
    Object.create = function (o) {
        function F() {}
        F.prototype = o;
        return new F();
    };
}

Drop the above into an external js file, include it and restart dnode again and provided you are using the latest version of jquery the above should make dnode cross browser compliant.I have tested it on firefox 3 and 7; chrome and safari on the mac.

Due to significant rewrite of the dnode library, this only affects the latest master branch and the npm version. If you are using an older version such as 0.5 the changes above has already been merged in to the core. For more details refer to the git commits and this issue here

Non obtrusive contact form in Rails

Thu, 10 Nov 2011 11:29:00 +0000

In my development experience with Rails I always come to the point when I have to make some kind of contact form or contact us page and its always the part I leave to the very end for the reason that it is tedious ( write mailers; setup mail; views; controllers; tests ), sometimes buggy and difficult to test ( if you are collaborating on a project and the client only wants you to use specific testing gems ); and repetitive ( you will have to repeat the entire process again in another project.)

Because of this I have developed or adpated a simple system so I have a base of standard template code I can use between projects which I know does work.

For my contact forms setup, the first thing I do is to create an ActiveModel object or class called Contact or ContactForm depending on the what names are available in that project and drop it into the app/models folder. It will be prepended with Contact so I know what it is and does.

Why use ActiveModel and not ActiveRecord? In some circumstances I come across, there is alreay a model called Contact in some projects which represent a different entity of the system altogether. Also I don't want to use ActiveRecord unless I know I am saving data to the DB. In the case of a contact form I'm only interested in getting details from the form and mailing it - if there is a need to store client queries, another plugin or system will need to be used to handle that aspect as there is a world of difference between sending data in a contact form and filtering and storing client queries.

Once the model is setup I normally add in basic validations for the commonly used fields such as name, email, subject matter. It is imperative to keep these fields to a minimum or to just the required fields. For me personally, I just want the client to provide the basic information which I need to get in contact with them again. The code below illustrates this:


class Contact
  include ActiveModel::Validations
  include ActiveModel::Conversion
  extend ActiveModel::Naming

  attr_accessor :name, :email, :phone, :alternative_phone, :subject, :question

  validates_presence_of :name, :email, :subject, :question
  validates_format_of :email, :with => /^[-a-z0-9_+\.]+\@([-a-z0-9]+\.)+[a-z0-9]{2,4}$/i
  validates_length_of :question, :maximum => 2500

  def initialize(attributes = {})
    attributes.each do |name, value|
      send("#{name}=", value)
    end
  end

  def persisted?
    false
  end
  
end

To include the above into a form, just initialize it and if you are using the client-side-validations gem, the errors will still show up if some of the required fields are missing or malformed:


<%
  @contact = Contact.new
%>


<%= form_for @contact, :remote => true,:validate => true, :url => contact_path do |f| %>


        Your Name *

        <%= f.text_field :name, :class=>'text' %>
  


<% end %>

Assuming you have a contact_path to submit the form to, you can pass the contact object into a mailer once the validations have passed:


@contact = Contact.new(params[:contact])

unless @contact.errors.any?
  ContactMailer.query(@contact).deliver
end

Notice that in the form code sample above, I passed in remote as true to utilize the UJS capabilities in Rails 3. In the next article I will show you how to utilize this to build a less obtrusive contact form on the front end to integrate it with what we have so far.

Rails 3.1.1 + Asset Pipeline

Wed, 26 Oct 2011 19:32:00 +0000

I had the opportunity to try out the new Rails 3.1.1 asset pipeline and although I love the way it works, it does take some getting used to and understand the way it works.

The only issue I have with it is the sluggish speed in development mode - Sprockets compiles each asset individually and if you are using engines with nested stylesheets and js files, it grinds to a halt.

After much googling around, I found this gem

gem 'rails-dev-tweaks', '~> 0.5.1'

I then added the following default config into the application.rb file:


config.dev_tweaks.autoload_rules do
      keep :all

      skip '/favicon.ico'
      skip :assets
      skip :xhr
      keep :forced
 end

Restart the server and you should find that your assets are loaded normally.