Emmanuel Oga's Blog

Fun with Google's Apps Script! Batch Processing Gmail emails By Host

2014-03-28T00:00:00+00:00

Like everybody that has had the same email acount for a while, I receive a lot of emails. Most of them are automatic responses or update from sites like twitter.com, linkedin.com, facebook.com, plus.google.com, etc.. Others are mailing lists I signed up for and lost interest in, and so on.

Sometimes I get tired of the amount of emails to manage, and I just want to tell Gmail: "delete all emails coming from host.com", for any number of hosts.

My email managemenet workflow goes something like this:

Locate by visual inspection a host I want to delete or archive
Search "from:host", mark all emails for deletion
Delete all marked emails
Repeat for as many hosts as needed.

Since it is unlikely Gmail will ever include a feature to automate such specific workflow, I decided to give Google's Apps Script a try (not to confuse with AppleScript :p).

I got tired of doing this by hand and I wrote a script to automate this task.

Here's how the resulting Apps Script looks like (you need to be logged into Gmail for it to work).

The script is also able to archive emails coming from a certain host, and/or mark them as important.

The standard disclaimer applies: this script will erase some of your emails! Normally it should only move emails from the hosts you select to your Trash folder, but use it at your own risk!

Developing this script was fun but I found a few caveats because I was lazy and decided not to use the ui services. Instead, I used the html services.

Apps Script uses caja under the hood, so a lot of things are not allowed (e.g.: it did not allow me to add pictures, like a loading gif, or svg, to the page).

All in all it was a fun experience and I think AppScript is definitely a very useful tool to have in your toolbox if you use Google products like Gmail or Spreadsheets.

Graphs are Everywhere! An overview of GraphConnect San Francisco 2013

2013-10-07T00:00:00+00:00

Neo4j is a very cool Graph Database which sports an embedded mode for Java applications but also an stand-alone server with a REST API. Drivers are available for a ton of different programming languages.

The developers of Neo4j organized GraphConnect @ San Francisco and I was lucky enough to be invited! Hereby I present some of the notes I took.

New Opportunities for Connected Data, by Emil Eifrem

To me, the take away on this keynote was the notion that, in any given data set, there are relations to be discovered: "There's a Graph for That!". Exploiting this relationships can give a company an strategic advantage over competitors.

Emil mentioned a paper published by Gartner: The Competitive Dynamics of the Consumer Web: Five Graphs Deliver a Sustainable Advantage. Hopefully I'll get a hold of that paper at some point, sounds like an interesting read! Many times relationships are not self evident. A question that we can ask ourselves when observing a given system is: What five graphs exist here?

Emil announced a free course on Neo4j will be made available on versal.com, probably by end of October.

He finished by demoing a bit the sweet new data browser for Neo4j, built with AngularJS.

What's new in Neo4j 2.0, by Andreas Kollegger

Labeling is a new feature that replaces the need to assign { type: "Type" } properties on Nodes or create (Node)-[:IS_A]->(Type) relationships. Labels are light weight and shouldn't be thought of as a rigid type system. Labels are automatically indexed, so it is efficient to query for all Nodes with a certain label.

Another new feature is constraints: for instance, we can now constraint that a given property, like title, should be unique across nodes.

CREATE UNIQUE is going away and will be replaced with MERGE.

Many other things mentioned in this talk are well documented in a couple of blog posts in a much better way than I could do:

DIY Graph Search, by Max De Marzi

Max demoed a search engine accepting natural language queries, similar to Facebook Search.

He implemented the parser using treetop, so not really a full natural language processor, but it was a nice hack for getting something working quickly. The app worked by converting human friendly search patterns to Cyper queries.

Graph Adoption at Gamesys, by Toby O'Rourke

I was looking forward this talk because I really like both subjects: Graphs and Video Games! :)

The talk was about how Neo4j is used on Gamesys to back many of their games.

An example was how they tracked referral of friends. By visually plotting the referrals, suspicious activity patterns can be quickly spotted.

Graphs can also be used to model game economy. By putting all monsters, characters, locations and items of an RPG on a Graph, an analysis can be done showing things like monster that are too easy to beat or items that are too cheap or too expensive. The talk was a bit vague though about how to exactly find this things - trade secrets? :).

Hacking Health Care with Neo4j, by Fred Trotter

DocGraph is a Data Set comprised of fifty million connections that shows how healthcare providers team to provide care.

This talk was about were that data came from and what can be done with it. If working with a big health data set sounds interesting to you, you should check this page.

Designing and Building a Graph Database Application, by Ian Robinson

This talk was about Domain Driven Design, how to build an application able to evolve over time, and how to use Test Driven Development during the process. Not surprisingly, these very same tools can be used to build any other application!

The talk showed how to identify the Domain Entities starting from user Stories to the specification of the Data Model. One big advantage when using Graph Databases is that the Entities identified during design phase are often directly translatable to the Nodes and Relationships, in a lot more straightforward and flexible way than when designing the schema for a relational database.

A Little Graph Theory for the Busy Developer, by Jim Webber

The closing talk from Jim Webber was very entertaining and informative! Some key points:

Complexity in data management is a function not only of the size, but also of the connectedness and uniformity of the data (people tend to only focus only on size).
Graph databases leverage the rich theory behind graphs to find the information hidden in the relations of the data.
Traditional databases, including relational and document oriented databases, imprison your data.
In Graph Databases we don't need to denormalize or lay the data in a special way (e.g. for map-reduce friendliness).

Some algorithms and concepts mentioned:

A recommended book to dig more into these subjects is Networks, Crowds, and Markets: Reasoning About a Highly Connected World.

I'm pretty sure this talk also mentioned the NoSQL distilled book as a good source of information regarding non relational databases.

A Digression on Graphs, Graph Databases and Graph Processing Systems

There's a distinction to be made between Graph Compute Engines (which are generally used for running offline algorithms over massive amounts of Nodes), and Graph Databases. These concepts are explained well in the freely available Graph Databases Book.

Arguably every big player in the web nowadays is using Graphs in one way or another (a point stressed in Emil's keynote):

Facebook has the Open Graph.
Yahoo's Marissa Mayer talked about the Interests Graph, although I still have to see what Yahoo is exactly doing to build this Graph and make use of it.
LinkedIn has the Connections Graph: there is a tool that allows you to visualize your own personal connections.
Google's Page Rank, the company's corner stone algorithm which powers Search, is based on Graph Theory (and so is Knowledge Graph).

Google has also published the Pregel paper describing a distributed graph processing system. Some open source Graph Processing systems, like Apache Giraph, have picked Pregel as the basis for their implementations, but Google itself has not released an Open Source Graph Database or Graph Processing Software package yet.

Using AngularJS with a Rails backend

2013-07-23T00:00:00+00:00

I created a sample AngularJS app which uses rails as a backend.

The running app on all its glory.
Github repository.

While it is possible to include AngularJS as part of the rails assets, I think it is better to setup the angular code base on a standalone folder, leaving the rails app as a (more or less) isolated backend.

There are several advantages to this setup:

Ability to manage the frontend app using Yeoman (including generators!).
No more questions about file layout: use rails layout for rails stuff, angular-generator layout for angular stuff.
Easily install external assets via bower (e.g. angularjs, jquery, twitter bootstrap, underscore, etc.).
Write client side unit tests and run them with karma.
Promotes unit-testing the js codebase and removes the temptation of integration-testing everything: testing with karma is a lot faster than using capybara or a similar solution.
LiveReload support for free with yeoman's angular generator! (uses websockets, no need to install a browser plugin).

Yeoman, Bower, Livereload and Karma Runner.

I want to stress out the convinience of working on your frontend using the stack yeoman sets up for you. By keeping the frontend isolated from the backend you get an amazingly fast development environment and draw a clear line between backend and frontend (great for making sure you are writing the right unit/integration tests). Yeoman sets up the test environment for you using jasmine as the testing library and karma as the runner. karma is possibly the fastest and most complete js test runner out there. And it is very well integrated with angular.

The experience of coding with Livereload is simply amazing. Immediate feedback for every little addition you save in your code editor while you are working, without having to reload the page in the browser!

True: you can use livereload with rails alone, and you can use bower with rails too. But Yeoman's angular generator sets everything right for you with a single command.

Rails was born in the request-response era of the web, and it shows. Yeoman sets up an environment with defaults that are better suited for developing web applications.

Setting the environment up

You'll need:

ruby 1.9.3 (rvm recommended for installation)
node 0.10.13 (nvm recommended for installation)
Two shell sessions!

Shell session one: the rails backend:

rvm use 1.9.3
git clone https://github.com/EmmanuelOga/simple-angular-rails-app.git
cd simple-angular-rails-app
bundle install
bundle exec rake db:migrate
bundle exec rails s -p 3000

NOTE: the angular application was generated using these commands.

npm install -g yo generator-angular
mkdir ngapp; cd ngapp
yo angular notes

Shell session two: a grunt server

nvm use 0.10.13
cd simple-angular-rails-app/ngapp
npm install -g grunt-cli
npm install
bower install
grunt server # opens a browser window... you are done!

What's going on?

During development, you need to run both rails and the grunt server. The grunt server takes care of serving the assets (yes, it can serve coffeescript if you like that) and doing the live reloads.

We simulate that the whole environment is a single web application by running a proxy inside the grunt server. At some point during the production deploy process we'll be consolidating the whole angular app as a set of static assets in rails' public/ folder.

Here's a diagram of the stack during development:

Would it make sense to go the extra mile and just host the angular app and the rails backend as separated processes? It would. But this setup allows you deploy the whole thing as a single app in the end, and have everything in a single domain easily.

The grunt server task proxies any url with path /api to the rails backend on localhost:3000.

Would sinatra be a better choice for such a small app? It would... but the whole point of this blog post is to show a possible way of using angular and rails together. This setup should be usable if you already have a running rails app and you want to add some additional single-pagy parts to it.

TESTING

To run both the backend tests and front end tests, you can run:

rake test PHANTOMJS_BIN=`which phantomjs`

This is done by reopening rails's test task and adding a step to run the karma runner. This design is a bit simplistic but it works. You may want to have something a bit more elaborate to make it so angular's tests are run even if rails tests fail to complete.

The PHANTOMJS_BIN env var is needed because the project configures karma to use phantom js, but it could be changed to run any other browser.

karma can be configured to watch the tests as opposed to do a single run. You should deffinitively look into that during development.

Deploying

If you run grunt build, grunt will package the whole angular app in a tidy package on the rails public/ folder. This packaging step could happen in the server to avoid having to commit the generated assets in your repository, analogous to how it is done for generating assets with rails' assets pipeline.

XSRF support

The rails app sets the XSRF token in the cookies. The cookies are accessible even when using the proxy because the port is not taken into account when restricting access to the cookies.

Check ApplicationController for some notes on the XSRF protection.

Tidying up an rspec suite with helpers

2012-04-21T00:00:00+00:00

Conditionally include Rspec Helpers depending on the kind of described thing.

With rpsec you can define helper methods in modules and include them only in places where it make sense. As an example I use this for specs related to ActionMailer functionality in rails:

module MailSpecHelpers
  class MailDeliveries < Struct.new(:subject)
    def deliveries
      deliveries = ActionMailer::Base.deliveries
      deliveries = deliveries.select { |m| m.subject =~ /#{subject}/ } if subject
      deliveries
    end

    def count
      deliveries.count
    end
  end

  # Returns action mailer deliveries filtered by subject
  def mail_deliveries(subject = nil)
    MailDeliveries.new(subject)
  end
end

module MailerSpecHelpers
  # returns an email from the currently described mailer class.
  def email_for(name, *args)
    described = self.class.describes || self.class.description
    described.send(name, *args)
  end

  # checks that the body of an emails is not blank
  def can_send_an_email_for(name, *args)
    email_for(name, *args).tap do |email|
      email.body.should_not be_blank
    end
  end
end

# Rspec Config File.

RSpec.configure do |config|
  config.include MailSpecHelpers
  config.include MailerSpecHelpers, :example_group => {
    :describes => lambda {|described| described < ActionMailer::Base }
  }
end

With these helpers in place I can do things like:

describe UserMailer do
  let(:user) { User.make! }
  it { can_send_an_email_for(:user_greeting, user) }
end

describe User do
  let(:user) { User.make! }
  let(:greetings) { mail_deliveries("Welcome #{user.first_name}!") }

  it "sends a greeting to the user after activation" do
    expect { user.activate }.to change(greetings, :count).by(1)
  end
end

Notice MailerSpecHelpers only gets included in specs that describe ActionMailer::Base derived classes, but MailSpecHelpers is included everywhere, because I only want to check if a mailer works in its own describe block, but I may check for mail deliveries anywhere in the spec suite.

Formatting expectations to make it easy to read on the spec

I have an xml generator class that outputs xml without pretty printing it. Making the code generate pretty output would be too expensive for my purposes, but I still would like to present the expected output in a nicely formatted way on the specs. Here's what I did:

module Rspec
  module CustomStringHelpers

    class LooksLike < String
      def failure_message
        "#{@me}\n\nis different than:\n\n#{@another}"
      end

      def matches?(another)
        @me, @another = normalize(self), normalize(another)
        @me == @another
      end

      def normalize(str)
        str.split("\n").map(&:strip).join("")
      end
    end

    def look_like(string)
      LooksLike.new(string)
    end

  end
end

# Rspec config file.

RSpec.configure do |config|
  config.include Rspec::CustomStringHelpers
end

# Specs

describe XMLGeneratorThingie do
  let(:schemer) { XMLGeneratorThingie.new }

  it "can add tag attributes to elements" do
    result = schemer.tag(:div, :thing) do |mb|
      mb.tag(:h1, :name, :class => "perico")
    end

    result.should look_like(<<-MARKUP)
      <div itemscope itemtype="http://schema.org/Thing">
        <h1 itemprop="name" class="perico">Spatula</h1>
      </div>
    MARKUP
  end
end

Blank space and new lines does not matter on xml and I can strip them from both strings I'm comparing using the look_like helper. Because I control the generation of the xml, I know that it will work for all different outputs my code can generate.

Redefining constants quietly to improve speed of the suite

I use machinist to instantiate actual db records in numerous places, which can be expensive. This small method allows me to reduce the number of record instantiations:

class Class
  # Allows redefinition of constants without warnings.
  def redefine_const(const_name, new_value)
    remove_const const_name
    const_set const_name, new_value
  end
end

Usually redefining a constant produces a noisy warning, but with the redefine_const method I can do that like a ninja:

class UserSearch
  LIMIT = 25 # max users to return per search

  def search(*args)
    # implementation...
  end
end

# Specs

describe UserSearch do
  # Reopening the class and overwriting this would normally be noisy.
  # Here we are being explicit and no noise is produced.
  UserSearch.redefine_const :LIMIT, 1

  let (:user1) { User.make! }
  let (:user2) { User.make! }

  def search(*args)
    UserSearch.new(*args)
  end

  context "query-less search" do
    before { user1; user2 }

    it "returns all users with a limit" do
      search.results.length.should == UserSearch::LIMIT
    end
  end
end

Instead of using redefine_const, I could replace the LIMIT constant with a class-attribute-accessor. But in the example code, LIMIT should be constant, something that should not ever change unless I rewrite its value and redeploy my code. I think is just natural to use a ruby constant when something is going to be, well..., constant.

Logging in users automagically

Before putting these helpers in place I had to define the current_user or admin in each controller spec, and sign in in a before-filter as needed. Now, all I have to do is include a parameter in the describe block to acknowledge whether there should be an admin or a user logged in that context.

module ControllerMacros

  def sign_in!(kind, *options)
    if kind == :admin
      let(:admin) { Admin.make!(*options) }
      before { sign_in :admin, admin }
    else
      let(:current_user) { User.make!(*options)
      before { sign_in :user, current_user }
    end
  end

end

# RSpec configuration file

RSpec.configure do |config|
  # So useful: if something goes wrong on the view, the spec run will blow up.
  config.render_views

  # Extend only controller specs
  config.extend ControllerMacros, :example_group => {
    :describes => lambda {|described| described < ActionController::Base }
  }

  # Funny flag... but allows me to write symbols instead of hashes.
  config.treat_symbols_as_metadata_keys_with_true_values = true

  # enables: describe(SomeController, :logged_in) { ... }
  config.before(:all, :type => :controller, :logged_in => true) { self.class.sign_in!(:user) }

  # enables: describe(SomeController, :logged_as => :admin / :user / etc..) { ... }
  [:user, :admin].each do |val|
    config.before(:all, :type => :controller, :logged_as => val) { self.class.sign_in!(val) }
  end
end

# Controller specs

describe Admin::Users do
  context "authorized", :logged_as => :admin do
    it "shows a list of users" do
      get :index
      response.should render("index")
    end
  end

  context "not authorized", :logged_as => :user do
    it "redirects the user to the front page" do
      get :index
      response.should redirect_to("/")
    end
  end
end

Filtering out specs

If you run a spec manually, you can use rspec spec/name_spec.rb:25 to only run the spec defined on the line 25. But if you are using a spec runner like autotest, or your line numbers change all the time because you keep adding or removing stuff, that can be annoying. Here is how you define spec filters:

# RSpec configuration file

def ec2_instance?
  # if you retrieve this url from an EC2 instance (and only then) it
  # returns an instance id, so if this does not fail we know we are
  # running inside EC2
  open("http://169.254.169.254/latest/meta-data/instance-id").read rescue nil
end

RSpec.configure do |config|
  # Only run focused specs:
  config.filter_run :focus => true

  # Yep, if there is nothing filtered, run the whole thing.
  config.run_all_when_everything_filtered = true

  # Do not run these specs on ec2
  config.filter_run_excluding :bypass_on_ec2 => true if ec2_instance?
end

# Specs

describe User do
  it "does something I'm checking RIGHT NOW.", :focus do
    # Stuff I'm working on.
  end

  context "when the external user profile is accessible", :bypass_on_ec2 do
    it "retrieves the file from the server and process it" do
      expect { subject.update_from_server }.to_not raise_error
    end
  end
end

There are two examples on how to use filtering. First one is simple: I just add the :focus simple to the block of code I'm currently working for.

The second one is a little different. Providing you have a good reason not to run some code when you are on an ec2 instance, then you can set up a block to skip the examples there.

Rails CMS landscape from 10,000ft high

2012-03-23T00:00:00+00:00

I'm needing a CMS to setup a semi dynamic web site. The features I'm looking for:

Ruby 1.9.2 compatible (for use with heroku's cedar stack).
If it uses Rails, it should be ~> 3.x.
Backed by a SQL database.
Easy to use by non developers.
Easily extensible with custom sections (e.g. adding a product catalogue).
I should be able to install, style it (bootstrap) and test it locally in less than 5 minutes (pretentious, I know).

I went through each of the options I found really quick, just to have a feel of each one from 10,000 ft high, so I'm sure I might have judged some of them unfairly.

Radiant

According to the Wikipedia article, the first release of http://radiantcms.org/ was 5 years ago, and it is still today alive and kicking. Unfortunately, it runs on Rails 2.x. I rather not have to deal with Rails 2.x for a new project if I can avoid it.

Refinery

The Refinery installation went on pretty well, but I couldn't find any sample themes for it.

Even though Ryan Bates in his screencast makes styling Refinery look easy as he just pastes some magic CSS code on application.css.scss, trying to adapt Refinery to use bootstrap is definitely a lot more involved.

To be clear, I think if I was creating the styles from scratch then Refinery would be great for me. The generated html looks pretty sane. But since I'm constrained to make it work with existing markup (the way bootstrap wants me to work), I find it a bit difficult to customize.

BrowserCMS

After installing the browsercms gem, the first thing I got with the very first command I run is:

undefined method `load_missing_constant' for module `ActiveSupport::Dependencies' (NameError)

An error so early in the process of installing the app lowered my confidence on the project... I wasn't sure I was going to succeed debugging the problem in just 5 minutes. I may inspect this option further in the future.

Locomotive

Since Locomotive looks pretty cool, I decided to obviate my decision of using a good'ol SQL database as the storage and created a free account on mongohq.com.

I spent a couple of minutes setting up the whole thing just to be confronted with this after following the site's instructions:

uninitialized constant CASClient::Tickets::Storage::ActiveRecord (NameError)

The one requirement I did not want to obviate was, again, my 5 minutes deadline. So I skipped to the next option.

Nesta

Nesta just worked. This is a good thing on my book. Also, it installed only a few gems.

There are a couple of ready made themes. Even a bootstrap one (which did not work too well out of the box, at least with the demo content... oh well).

Anyway, this is a very nice CMS for hackers, I can see myself using it on the future. It is too hardcore to casual users though: you need to create markdown files manually into the project directory to edit the site.

Zena

Zena looked promising but again it is based on Rails 2.x and I don't feel like using Rails 2.x for a new project.

adva2

I quote the adva-cms2 README:

"adva-cms2 is currently in an early developer sneak-peak preview stage."

Despite the warning I tried it and... I could not make it work. Some problem with gem dependencies I think.

Alchemy

Alchemy installation was almost straightforward. I got a couple of errors but I could correct manually without a lot of hoop-jumping, but once I ran the dashboard I was overwhelmed by the interface.

I just wanted to create a single page and test the result. I have the feeling this must be a pretty powerful CMS, but since it is difficult to use even for a seasoned web applications developer -as I am ;)- I'm guessing not so experienced users will have a hard time with it too.

Conclusion

From the whole list of CMS available the most promising ones for my purposes where Nesta and Refinery. Nesta is a bare-bones, good for static content, made for hackers CMS. I can see myself using it if I ever need something slightly more flexible than jekyll for my blog.

Refinery CMS looks pretty good, works out of the box and is easy to manage, but a little hard to style. Right now I'm not sure if it will work for me, but spending something more than 5 minutes trying to customize it seems worth the effort in order to avoid creating my website form scratch.

Net::HTTP not setting cookies... or so it looked like

2012-02-29T00:00:00+00:00

I was retrieving a web page from a site, but I found out it needed a cookie to be set in order to work. Otherwise, it was redirecting me to set up my language of preference. I thought no problem... let's set up that cookie.

require 'net/http'

url = "http://www.realtor.com/realestateandhomes-detail/3429-W-Lone-Mountain-Rd_North-Las-Vegas_NV_89031_M12163-28068"
uri = URI(url)
http = Net::HTTP.new(uri.host, 80)
request = Net::HTTP::Get.new(uri.request_uri)
request['Cookie'] = "preference=USA"
r = http.request(request)

Awesome! it worked! Sometimes...

After a bit of banging my head against the wall I decided to try with a different version of ruby, 1.9.3.

Locally I was using ree-1.8.7-2011.12, and on the c.i. server ruby-1.8.7-p352.

So what was the difference? Check it out.

Turns out 1.9.3 sets a default user agent. Even though it should be optional, looks like some servers do not look at the cookies header if the user agent is not set.

The takeaway: if you are setting cookies on your request, make sure the user agent is being set too:

request['User-Agent'] = 'MyCoolUserAgentString'

VIM config snippets

2011-11-28T00:00:00+00:00

There are lots of things you can do to better organize your vim configuration, starting with using pathogen.

Apart from using pathogen, what I like to do is having small snippets of configuration on separate files to tweak specific things. Here's my .vimrc:

" Setup for pathogen.
call pathogen#runtime_append_all_bundles()
call pathogen#helptags()

" Load additional plugin configurations and stuff
for rc in split(globpath(&rtp, "rc/*"), "\n")
  execute "source" rc
endfor

First part is the setup for pathogen. The second part is what I'm talking about, I go through every file in my .vim/rc directory and load every snippet of that folder.

A fun thing to try is to make every snippet "self contained". That way, if someone browsing your config thinks the thing you are setting up there is cool, all he has to do is copy that single file to get the specific piece of functionality.

BTW, since I use gvim most of the time, on the .gvimrc I just source the .vimrc, plus a few GUI tweaks (removing toolbars, menus, etc).

source ~/.vim/vimrc
" ... some gui settings.

Beer Drinking Contest: Ruby versus Lua. Fight!

2011-07-29T00:00:00+00:00

Grab a gist. You probably already know what's the output of these programs.

Ruby

class Crowd < Struct.new(:bottles)
  def drink()    puts("Take one down and pass it around."); bottles.drink(1); self;  end
  def buy_more() puts("Go to the store and buy some more."); bottles.buy(99); self; end
  def sing()     puts("#{bottles} on the wall, #{bottles}."); self; end
end

class Bottles < Struct.new(:qty)
  def drink(amount) self.qty -= amount; end
  def buy(amount)   self.qty += amount; end
  def to_s() qty == 0 ? "No more bottles of beer" : "#{qty} bottle#{'s' if qty > 1} of beer" end
end

Crowd.new(Bottles.new(99)).tap do |crowd|
  99.times { crowd.drink and crowd.sing }
  crowd.buy_more.sing
end

Lua

local Crowd   = {}
Crowd.drink   = function(self) print("Take one down and pass it around.") self.bottles:drink(1) return self end
Crowd.buymore = function(self) print("Go to the store and buy some more.") self.bottles:buy(99) return self end
Crowd.sing    = function(self) print(tostring(self.bottles).." on the wall, "..tostring(self.bottles)..".") return self end

local Bottles      = {}
Bottles.__index    = Bottles
Bottles.drink      = function(self, amount) self.qty = self.qty - amount end
Bottles.buy        = function(self, amount) self.qty = self.qty + amount end
Bottles.__tostring = function(self)
  if self.qty == 0 then return "No more bottles of beer"
  else return self.qty.." bottle"..(self.qty == 0 and '' or 's').." of beer"
  end
end

local crowd = setmetatable({bottles = setmetatable({qty=99}, Bottles)}, {__index=Crowd})

crowd:sing() for i = 99, 1, -1 do crowd:drink():sing() end crowd:buymore():sing()

Let the games begin!

#/usr/bin/env sh

echo 'Baseline echo'
time for i in {0..20..1}; do echo "A long line with a number: $i" > /dev/null ; done
# real  0m0.001s user   0m0.000s sys    0m0.000s

lua -v
time for i in {0..20..1}; do lua beer.lua > /dev/null ; done
# Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
# real  0m0.054s user   0m0.000s sys    0m0.007s

luajit-2.0.0-beta6 -v
time for i in {0..20..1}; do luajit-2.0.0-beta6 beer.lua > /dev/null ; done
# LuaJIT 2.0.0-beta6 -- Copyright (C) 2005-2011 Mike Pall. http://luajit.org/
# real  0m0.041s user   0m0.000s sys    0m0.007s

rvm 1.9.2 exec ruby -v
time for i in {0..20..1}; do rvm 1.9.2 exec ruby beer.rb > /dev/null ; done
# ruby 1.9.2p180 (2011-02-18 revision 30909) [x86_64-linux]
# real  0m17.076s user  0m12.116s sys   0m3.323s

rvm jruby exec jruby -v
time for i in {0..20..1}; do rvm jruby exec jruby beer.rb > /dev/null ; done
# jruby 1.6.2 (ruby-1.8.7-p330) (2011-05-23 e2ea975) (OpenJDK 64-Bit Server VM 1.6.0_22) [linux-amd64-java]
# real  1m35.087s user  2m21.537s sys   0m7.226s

rvm ree ruby -v
time for i in {0..20..1}; do rvm ree exec ruby beer.rb > /dev/null ; done
# ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2011.03
# real  0m18.075s user  0m12.619s sys   0m3.526s

Conclusions

I don't give any chance to the jit capable interpreters to work their magic.
I'm "measuring" the system's I/O too even when I'm directing the output to /dev/null, get over it.
Lua, including its start-up time, is FAST!. Also, object oriented programming in lua is flexible, simple and nice.
Rails does not scale.
I want a beer.

Taming a Capybara

2011-07-26T00:00:00+00:00

photo from raxdakkar.com

Running acceptance tests in our rails 3 application turned out to be non trivial, even though there are excellent tools out there, and they keep getting better.

We are currently using capybara with the capybara-webkit driver, which is great because it runs in headless mode, without annoying browser windows popping up. I heard the QT download can be pretty big for mac users though, so have some spare bandwidth around if you have a mac and are planning to give it a try.

As we kept adding tests, the size of our spec/support/capybara.rb file grew with a lot of hacks. Here is an anotated version of that file and some others related, should it be helpful to anybody out there. Perhaps somebody will want to comment on better solutions for some of the nasty hacks.

Gemfile

group :test do
  gem 'capybara', '~> 1.0.0'

  # The git version worked better for us at the time we installed it.
  gem 'capybara-webkit', :git => "git://github.com/thoughtbot/capybara-webkit"

  # This is needed by capybara's save_and_open_page method.
  gem 'launchy'

  # This *might* be needed in some setups
  gem 'database_cleaner'
end

config/initializers/session_store.rb

# We deploy our application to several different subdomains, and we need
# to configure the domain for cookies in order to avoid the sessions
# getting mixed.
# But the domain option was messing with capybara browsers' ability to
# remember cookies, so we just exclude the domain config on the test environment.
options = { :key => "_our_app_session_#{Rails.env}" }
options[:domain] = Settings.cookies_host unless Rails.env.test?

Rails.application.config.session_store :cookie_store, options

spec/spec_helper.rb

RSpec.configure do |config|
  # .. rspec stuff
end

# This is how we load our support files for rspec, this keeps the size of
# the spec_helper.rb file manageable.
Dir["#{File.dirname(__FILE__)}/support/**/*.rb"].each {|f| require f}

spec/support/capybara.rb

require "capybara/rails"
require 'capybara/rspec'

# Setup capybara webkit as the driver for javascript-enabled tests.
Capybara.javascript_driver = :webkit

# In our setup, for some reason the browsers capybara was driving were
# not openning the right host:port. Below, we force the correct
# host:port.
Capybara.server_port = 7787

# We have more than one controller inheriting from
# ActionController::Base, and, in our app, ApplicationController redefines
# the default_url_options method, so we need to redefine the method for
# the two classes.
[ApplicationController, ActionController::Base, Listing].each do |klass|
  klass.class_eval do
    def default_url_options(options = {})
      { :host => "127.0.0.1", :port => Capybara.server_port }.merge(options)
    end
  end
end

# Here we do some selective configuration for tests which run with the
# rack backend and tests which run with the webkit backend.
RSpec.configure do |config|
  # In order for the database to have the same data both for the
  # capybara process and the web application process, we need either to
  # disable transactional fixtures (which produces very slow runs),
  # or use the hack you can find below in this file.
  config.use_transactional_fixtures = true

  # Uncomment in case your db gets dirty somehow.
  # DatabaseCleaner.clean_with(:truncation)

  config.before :each do
    if Capybara.current_driver != :rack_test
      # With selenium/webkit the host is set automatically if it was nil.
      Capybara.app_host = nil
    # config.use_transactional_fixtures = false
    # $use_truncation = true
    else
      # We found Capybara.app_host needs to be nil when using rack backend,
      # but point to 127.0.0.1 when using selenium or webkit.
      Capybara.app_host = "http://127.0.0.1"
    # config.use_transactional_fixtures = true
    end
  end

# config.after :each do
#   DatabaseCleaner.clean_with(:truncation) if $use_truncation
# end
end

# Big Fat Hack (TM) so the ActiveRecord connections are shared across threads.
# This is a variation of a hack you can find all over the web to make
# capybara usable without having to switch to non transactional
# fixtures.
# http://groups.google.com/group/ruby-capybara/browse_thread/thread/248e89ae2acbf603/e5da9e9bfac733e0
Thread.main[:activerecord_connection] = ActiveRecord::Base.retrieve_connection

def (ActiveRecord::Base).connection
  Thread.main[:activerecord_connection]
end

# Last but not least, when using capybara-webkit sometimes the response
# is not available when capybara tries to retrieve the contents of the
# page. Of all the possible solutions, this was the simplest for us:
# introducing a fixed delay each time something is clicked in the browser.
# https://github.com/thoughtbot/capybara-webkit/issues/111
class Capybara::Driver::Webkit::Browser
  alias original_command command

  def command(name, *args)
    result = original_command(name, *args)
    sleep(1) if args.first == "click"
    result
  end
end

Final Thoughts

Even though the config above allows our acceptance tests to merrily run, I'm not completely satisfied with the hoops we needed to jump in order to get things to work.

I have the feeling having the acceptance tests completely isolated from the main application (even in a separate repository) could be a good thing. In this direction it would be worthy to try and write the acceptance tests using a tool like phantomjs. I'm not talking about getting a phantomjs driver for capybara, but directly writing the whole acceptance suite with phantomjs in javascript.

Downloading and unzipping a file with ruby

2011-06-21T00:00:00+00:00

Quick! I need to download a zip file from an HTTP server and unzip it! That's surely trivial with ruby... is it? Yes, it is. But! Turns out there are a bunch of zip handling gems out there, and I had to try a a couple before arriving to my final solution.

First, let's download the file. This particular resource is served via an HTTP POST method. Yeah, unusual for a zip file.

require 'net/http'

  url = "http://someweirdserver.com/a-zip-served-via-POST.zip"
  data = { "some-bizarre-params" => "which-are-needed" }

  zipbytes = Net::HTTP.post_form(URI.parse(url), data).body

Done. Moving forward, let's unzip the file. One detail to bear in mind, I have the zip file as a buffer (a String instance), and I don't want to go trough a tempfile to unzip it. The good news is that the zipruby gem can work off buffers.

require 'zipruby'

  Zip::Archive.open_buffer(zipbytes) do |zf|
    # this is a single file archive, so read the first file
    zf.fopen(zf.get_name(0)) do |f|
      unzipped = f.read
      # done! do something with the unzipped file.
    end
  end

Lua's multiple arguments and the "select" function

2010-12-09T00:00:00+00:00

When I first read about lua's select function I found it a little confusing.

According to lua's manual for version 5.1, select behaves as follows:

select (index, ···)

If index is a number, returns all arguments after argument number
index.  Otherwise, index must be the string "#", and select returns
the total number of extra arguments it received.

That wasn't clear to me at first, but then I remembered lua sports real multiple arguments. I remark the word real because other languages like ruby don't really support returning multiple arguments. In ruby, if a method returns multiple arguments they are automatically packed into an array. Lua, in the other hand, has built in multiple arguments, which you can convert into a table if you really need to:

function multiple_args(...)
  local arguments = {...}  -- pack the arguments in a table
  -- do something --
  return unpack(arguments) -- return multiple arguments from a table (unpack)
end

Remembering the nature of lua's multiple arguments, and the fact that multiple arguments are not returned on a lua table dressed as an array, the description on the manual started to make sense.

The select function takes as "index" the index of the first argument that you want to retrieve, and any number of arguments afterward (the count of arguments starts at 1).

print(select(1, 1, 2, 3)) --> 1, 2, 3
print(select(2, 1, 2, 3)) --> 2, 3
print(select(3, 1, 2, 3)) --> 3
print(select(4, 1, 2, 3)) --> NOTHING

print(select(1, {1, 2, 3})) --> table
print(select(2, {1, 2, 3})) --> nothing

In the last two examples, the number of variable arguments is just one, consisting of a single table.

By the way, note another subtlety of lua: if you select more that the number of arguments available you get absolutely nothing!. Not even nil: nothing. If you try to pack NOTHING into a table, you get an empty table:

print(#{(function()end)()}) --> 0

Finally, if you pass "#" as index, the function returns a count of the multiple arguments provided:

print(select("#")) --> 0
print(select("#", {1, 2, 3})) --> 1 (single table as argument)
print(select("#", 1, 2, 3)) --> 3
print(select("#", {1,2,3}, 4, 5, {6,7,8}) --> 4 (a table, 2 numbers, another table)

Ruby conferences in Brazil and Uruguay

2010-10-31T00:00:00+00:00

This year I could attend both rubyconf Brazil and Uruguay. Both conferences had great talks and summarizing everything would be overkill, so I'll instead post some brief notes.

My Notes

Brazil Opening

Fabio Akita's talk was great in motivating the audience to be more participative in open source projects as a way of continuously improving as developers.

Client side development

Yehuda Katz talked about focusing on the client side of web applications. Using Ajax techniques is a start but ajax still generates a lot of latency waiting for server responses. In order to have a better client side experience techniques like buffered DOM manipulation, object bindings and event loops on the client side, can help to do as much work on the client as possible.

Rails 3.1

José Valim talked about some great features that will be packaged on rails 3.1, as better asset packaging (in line with the functionality provided by the sprockets gems), making rails serve the stylesheets and javascript and use sass, and general improvements on a lot of areas (new ActiveRecord's identity map, better generators, etc.).

Rubinius

Both Evan Phoenix in Brazil and Brian Ford in Uruguay described great features packed on of rubinius. Some of the impressive features are the integrated console and the profiler (both triggable by command line, no additional gems required). They also talked about the modularity and extensibility of the tools bundled in the rubinius environment, made evident by the possibility to implement 3rd party languages (like the fancy programming language).

JRuby

Charles Nutter showcased some cool demos on jruby, including live coding a swt GUI from irb, using the VisualVM profiler to profile some code, running a multithreaded java environment from the command line using the trinidad gem, and a 3D game. He also showed a demo of the mirah programming language, which has a syntax similar to ruby's but contains type annotations and can produce either java code or jvm bytecode.

Webfinger

Blaine Cook talked about webfinger, a solution to provide users with an identity on the web in such a way they becomes the owners of it (as opposed as tying the identity on the web to any given service, like twitter, facebook or linkedin profile names). Slides here.

Ruby 1.9.x fibers

Aaron Patterson gave a talk about scheduling, fibers and coroutines.

OOP and SOLID

Jim Weirich gave a talk about OOP and the SOLID principles.

Git

Scott Chacon gave a tutorial for git, starting from the fundamentals and the differences with other VCS like rcs or svn.

Clojure

Federico Brubacher talked about clojure, a very nice functional programming language for the jvm, and a great alternative to scala as it is a much more simple language.

ActiveRecord and Identity Maps

Emilio Tagua talked the addition of an Identity Map to ActiveRecord to shield great memory and performance savings.

Programming patterns

Ignacio Facello talked about patterns for Ruby.

Metaprogramming

Santiago Pastorino talked about metaprogramming ruby techniques.

Android

Tim Bray gave a 30 minutes long advertisement on android, quickly dismissing ruby as a viable development option while it was on it. Despite that, android is a great platform to develop mobile applications.

Uruguay Keynote

Evan Henshaw-Plath (@rabble) closed the Uruguay conference with a great keynote where he talked about the characteristics of the ruby community.

See you at rubyconf.br!

2010-10-24T00:00:00+00:00

Tomorrow I'm travelling to Sao Pablo, Brazil, to attend my first http://rubyconf.com.br/.

Exiting times! See you there?