Rails on the Run - Home

Avoid using metaprogramming (seriously!)

2008-05-04T08:29:00Z

Ruby is sexy, Ruby is cool and its metaprogramming potential offers some really cook features. However you might not realize that your cleverness is slowing down your code.

Today I was working on cleaning up merb_helper a Merb plugin that brings a lot of the stuff Rails developers are used to. In Merb we aim for speed and try to avoid magic.

merb_plugin didn't receive a lot of love from the main contributors but few features were added by different contributors and the code became hard to maintain.

Looking at the code I quickly found this bad boy:

(Old Merb Time DSL using metaprogramming)


   
  module MetaTimeDSL

    {:second => 1, 
     :minute => 60, 
     :hour => 3600, 
     :day => [24,:hours], 
     :week => [7,:days], 
     :month => [30,:days], 
     :year => [364.25, :days]}.each do |meth, amount|
      define_method "m_#{meth}" do
        amount = amount.is_a?(Array) ? amount[0].send(amount[1]) : amount
        self * amount
      end
      alias_method "m_#{meth}s".intern, "m_#{meth}"
    end

  end
  Numeric.send :include, MetaTimeDSL

The above code looks awful to me and I decided to rewrite it a way I thought would be more efficient:


  module TimeDSL

    def second
      self * 1
    end
    alias_method :seconds, :second

    def minute
      self * 60
    end
    alias_method :minutes, :minute

    def hour
      self * 3600
    end
    alias_method :hours, :hour

    def day
      self * 86400
    end
    alias_method :days, :day

    def week
      self * 604800
    end
    alias_method :weeks, :week

    def month
      self * 2592000
    end
    alias_method :months, :month

    def year
      self * 31471200
    end
    alias_method :years, :year

  end
  Numeric.send :include, TimeDSL

To make sure I was right, I run the following benchmarks:


require 'benchmark'
TIMES = (ARGV[0] || 100_000).to_i

Benchmark.bmbm do |x|
  
  x.report("metaprogramming 360.seconds") do
    TIMES.times do    
      360.m_seconds
    end
  end
  
  x.report("no metaprogramming 360.hours") do
    TIMES.times do
      360.seconds
    end
  end
  
  x.report("metaprogramming 360.minutes") do
    TIMES.times do    
      360.m_minutes
    end
  end
  
  x.report("no metaprogramming 360.minutes") do
    TIMES.times do
      360.minutes
    end
  end
  
  x.report("metaprogramming 360.hours") do
    TIMES.times do    
      360.m_hours
    end
  end
  
  x.report("no metaprogramming 360.hours") do
    TIMES.times do
      360.hours
    end
  end
  
  x.report("metaprogramming 360.days") do
    TIMES.times do    
      360.m_days
    end
  end
  
  x.report("no metaprogramming 360.days") do
    TIMES.times do
      360.days
    end
  end
  
  x.report("metaprogramming 360.weeks") do
    TIMES.times do    
      360.m_weeks
    end
  end
  
  x.report("no metaprogramming 360.weeks") do
    TIMES.times do
      360.weeks
    end
  end

  x.report("metaprogramming 18.months") do
    TIMES.times do    
      18.m_months
    end
  end
  
  x.report("no metaprogramming 18.months") do
    TIMES.times do
      18.months
    end
  end
  
  x.report("metaprogramming 7.years") do
    TIMES.times do    
      7.m_years
    end
  end
  
  x.report("no metaprogramming 7.years") do
    TIMES.times do
      7.years
    end
  end
  
end


 Rehearsal ------------------------------------------------------------------
metaprogramming 360.seconds      0.130000   0.000000   0.130000 (  0.133164)
no metaprogramming 360.hours     0.050000   0.000000   0.050000 (  0.042655)
metaprogramming 360.minutes      0.130000   0.000000   0.130000 (  0.133327)
no metaprogramming 360.minutes   0.040000   0.000000   0.040000 (  0.042401)
metaprogramming 360.hours        0.140000   0.000000   0.140000 (  0.134312)
no metaprogramming 360.hours     0.040000   0.000000   0.040000 (  0.043125)
metaprogramming 360.days         0.130000   0.000000   0.130000 (  0.134949)
no metaprogramming 360.days      0.050000   0.000000   0.050000 (  0.043745)
metaprogramming 360.weeks        0.130000   0.000000   0.130000 (  0.135581)
no metaprogramming 360.weeks     0.050000   0.000000   0.050000 (  0.043544)
metaprogramming 18.months        0.130000   0.000000   0.130000 (  0.135234)
no metaprogramming 18.months     0.050000   0.000000   0.050000 (  0.044354)
metaprogramming 7.years          0.140000   0.000000   0.140000 (  0.144062)
no metaprogramming 7.years       0.050000   0.000000   0.050000 (  0.044392)
--------------------------------------------------------- total: 1.260000sec

                                     user     system      total        real
metaprogramming 360.seconds      0.130000   0.000000   0.130000 (  0.132567)
no metaprogramming 360.hours     0.040000   0.000000   0.040000 (  0.042777)
metaprogramming 360.minutes      0.140000   0.000000   0.140000 (  0.132554)
no metaprogramming 360.minutes   0.040000   0.000000   0.040000 (  0.043193)
metaprogramming 360.hours        0.130000   0.000000   0.130000 (  0.133027)
no metaprogramming 360.hours     0.050000   0.000000   0.050000 (  0.042613)
metaprogramming 360.days         0.130000   0.000000   0.130000 (  0.138637)
no metaprogramming 360.days      0.050000   0.000000   0.050000 (  0.043213)
metaprogramming 360.weeks        0.130000   0.000000   0.130000 (  0.134049)
no metaprogramming 360.weeks     0.040000   0.000000   0.040000 (  0.043713)
metaprogramming 18.months        0.140000   0.000000   0.140000 (  0.134941)
no metaprogramming 18.months     0.040000   0.000000   0.040000 (  0.043980)
metaprogramming 7.years          0.150000   0.000000   0.150000 (  0.143389)
no metaprogramming 7.years       0.040000   0.000000   0.040000 (  0.044585)
 0.136591)

The metaprogramming version of the same implementation is almost 3 times slower!

Moral of the story: only use metaprogramming if you really have to or if you don't care about speed of execution.

BarCamp San Diego rev.3

2008-04-27T17:24:00Z

Just a reminder, this coming week end, San Diego presents BarCamp for the third time.

This time, the chosen Venue is Microsoft in La Jolla

I was thinking about preparing 2 intro talks, one on Merb and one on Unobtrusive Javascript (jQuery, Prototype + LowPro etc...), then we'll see the crowd and what people are interested in. Feel free to give me your feedback, suggestions...

I also heard that on top of the awesome people from san Diego, some other important people are coming just for the event:

Kevin Clark from the Powerset hall of fame.
Derek Neighbors representing Phoenix's top Rails shop: Integrum

Warning

Based on previous BarCamps, you might end up seeing a guy wearing a kilt who talks about how he is his own imaginary friend, having to look at ActionScript code, meet a lot of interesting people and even maybe learn about lock picking.

I even heard rumors saying that Ryan Felton is organizing another wii tournament.

Freezing Rails with Git

2008-04-16T08:21:00Z

As you've probably heard, Rails now moved to its own GitHub repo.

If, like me you were a heavy piston user, you are wondering how you will be able to do the same thing if you switch to git.

First off, you need to know that Piston will soon support git. As a matter a fact it already does. At least you can download a beta version from François's blog.

You can also go with giston/braids which was meant to make the svn/switch easy on you. I heard rumors that evilchelu might not keep on developing this project. You might want to check with him.

Personally I didn't really like using any of these solutions. Rails also came with its' own approach. (rake rails:freeze:edge)

When I recently worked on Merb's freezer, I discovered the power of git submodules.

Submodules allow you to import "modules" from other git repos inside your own repo. Basically they do what piston does for SVN, apart that submodules are built-in git. Of course it has an expected limitation, you can only add git submodules.

The good news is that Rails moved to git and now you can "freeze" Rails as a submodule and update really easily!

First thing first, you need to move your project to git. If you are not confident it's a good move yet, you can use "git-svn". However, I would personally recommend you don't. I did that for few months and when I finally moved to git only, it was a pain to restructure the entire path of the app.

Anyways, let's say you created a new github project and if still wish to use git svn do:

1234	$ git-svn import svn://path-to-your-svn-repo project-name$ cd project-name$ git remote add origin git@github.com:matta/project-name.git$ git push origin master

Your project is now under git, but if you pistonized Rails, you can't update it anymore :(

Do not fear my dear friend and do as follows:

$ rm -rf vendor/rails
$ git commit
$ git submodule add git://github.com/rails/rails.git vendor/rails
$ git submodule init
$ git commit
$

That's it you are done :)

Next time you want to update just do:

123	$ cd vendor/rails$ git remote update$ git merge origin/master

or you can also do

12	$ cd vendor/rails$ git pull

(yes, each plugin acts a a normal git repo)

A quick note for gitHub users. If you browse your repo you won't see the vendor/rails folder and might freak out. Don't! Git is smart and wants to stay slim, instead of copying the files over, it just creates a reference to the original repo. If you try to pull your project in another folder you will see that the Rails folder gets created as expected.

Personally, when plugins are not available in a git repo I usually do a simple svn export to my project vendor's folder. If I need to modify one of these plugins, I just import it to github and work on it from there.

You might still want to stick to Piston or Braids and that's fine, but now you won't have an excuse not to switch to Git :)

UPDATE: I just found out that Graeme wrote a nice detailed post about tracking plugins using git, check it out

Merb tip - how to freeze a project?

2008-04-15T07:42:00Z

After few of my patches got accepted by the Merb lead team, I was given commit access to merb-more. Merb doesn't really have a core team per say. It's actually managed the same way Rubinius is managed meaning that few people such as Ezra, Wycats and Ivey lead the development while many other contributors have commit rights to the different repos.

Patches are handled via GitHub pull Request and LightHouse tickets. Read the following contribution documentation for more info.

Anyway, I've been working on a merb-more gem called merb-freezer.

We don't have a logo for the plugin yet so I picked a "cool" star from the late 80's to represent.

Vanilla Ice!

Right, so you might wonder what's the connection between a kitsch white rapper and a new Merb Gem? Not much, apart that they are both cool (or kinda cool).

Let's forget about "ice ice Baby" and focus on merb-freezer.

merb-freezer has a simple goal: let you "freeze" your application and run it without dependencies.

Why would you want to freeze your app?

(This is only valid for Merb 0.9.3 and 0.9.2 edge as of April 14)

You might have multiple applications on the same server/slice/cluster. Different applications might require different versions of Merb or some other Merb gems.
You might work with a team of developers and want everyone to be using the same version of the gems.
You are using Merb Edge and want to make sure that your coworkers are developing/testing against the same revision.

How to freeze your app?

First thing, in your init.rb file you need to require merb-freezer

require 'merb-freezer'

Now that you required the plugin when you get new rake tasks:


  rake freeze:core            # Freeze core from git://github.com/wycats/merb...
  rake freeze:more            # Freeze more from git://github.com/wycats/merb...
  rake freeze:plugins         # Freeze plugins from git://github.com/wycats/m...

The rake freeze tasks use by default git submodules to freeze the various components. That means that you need to have your project under git to use that feature. However, if you didn't switch to git yet, do no worry, we have a plan B.

When you run the freeze tasks a framework directory is created at the root of the folder and the gems are checked out there.

Not a git user? We thought of you and added an option for you to use rubygems.

12	rake freeze:core MODE=rubygems

When doing that, you are freezing the latest version available on the rubygems server or installed locally. Note that when using the default mode, you are pulling the latest version of Merb from the official git repo. If you want to do that using rubygems you will need to checkout the git repos locally and install the gems yourself before freezing them.

Also it's worth noting that you can also do all of that manually, as long as you follow the same conventions you should be fine.

How to use the freezing gems?

If you are a Rails user, you might expect that Merb uses the frozen gems by default, at least that what I expected. Turns out, it's not the case since Merb avoids too much magic and unless you ask for it, Merb will use the available system gems.

So how to start a frozen merb app? Easy enough:

frozen-merb

That's it, you can uninstall merb-core from your system and as long as you froze merb-core, you can start your app.

How to update a frozen app?

simply re-freeze but with the UPDATE=true param

12	rake freeze:core UPDATE=true

This option works for in both modes (rubygems and git submodules).

What's next?

I'd like to add a locking mechanism that would allow you to force your app to only run on specific versions of few gems. The main advantage of this approach is that you wouldn't need to freeze files in your repo as long as you have the required versions on your machine.

I would also like to extend the freezer to let you use the submodules to freeze other gems.

Other suggestions? Found a bug? Want to submit a patch? Leave me a comment or use the Merb LightHouse ticketing system

Rails or Merb, what's best for you?

2008-04-10T08:10:00Z

If you follow my blog, you already know what Merb is.

I love Rails and I truly believe it has changed web development. At least it has changed the way I do web development.

But Merb looks slick, apparently is way faster than Rails, and has less "fluff" and less magic.

Now that we are getting really close to a Merb 1.0 (scheduled for Rails Conf '08) it's time to evaluate if Merb is the good choice for some of my clients' projects.

However, according to Merb's author, Ezra, at MountainWest RubyConf 2008, Rails will get you there faster. In a client's case, they don't need to build a huge app but need a lot of speed and the ability to easily handle a heavy load right away without using caching. Also most of the traffic will go through an API so we won't have to manage too many views.

Let's see how fast Merb really is.

To test Merb's speed, I built the very same prototype using Merb 0.9.2 and Rails Edge (pre 2.1). Both apps use ActiveRecord and are connected to a UTF8 MySQL database, both apps have exactly the same views. (Note that Merb would run way faster using DataMapper, but I don't feel that DM 0.9x is production ready yet, also, using a rack handler would certainly be way faster but my goal was really to compare ActionPack vs Merb.)

Both apps use the same ActiveRecord class, their controllers are a bit different but basically do the same thing.

Here is what was tested:

The Merb/Rails app should receive a GET request with a JSON object in the query.
The Merb/Rails app should route the request to a controller and pass the JSON object to an AR class.
The AR class should parse the JSON object (which contains an array of objects), extract each object, and try to find them in the database using one of the attributes. If the object isn't found, it should be created, otherwise it should return the AR object. The amount of hits should be incremented by 1 and the object should be saved back to the database.
A simple HTML view should be rendered

Quick Merb benchmark

I setup Merb to run locally on my MacBook 2.16Ghz Core Duo 2, 2Gb Ram. To test the raw performance, Merb is started in production mode.

I then used httperf to make 10000 connections to the server at a rate of 500 (--rate=500 --send-buffer=4096 --recv-buffer=16384 --num-conns=10000 --num-calls=1)

Here are the results:

Maximum connect burst length: 29
Total: connections 4377 requests 4221 replies 2932 test-duration 41.629 s
Connection rate: 105.1 conn/s (9.5 ms/conn, <=1022 concurrent connections)
Connection time [ms]: min 41.0 avg 1920.4 max 35390.8 median 898.5 stddev 4887.3
Connection time [ms]: connect 2118.1
Connection length [replies/conn]: 1.000

Request rate: 101.4 req/s (9.9 ms/req)
Request size [B]: 321.0

*Reply rate [replies/s]: min 0.0 avg 73.3 max 143.0 stddev 65.8 (8 samples)*
Reply time [ms]: response 809.0 transfer 18.1
Reply size [B]: header 121.0 content 557.0 footer 0.0 (total 678.0)
*Reply status: 1xx=0 2xx=2932 3xx=0 4xx=0 5xx=0*

CPU time [s]: user 0.35 system 36.54 (user 0.8% system 87.8% total 88.6%)
Net I/O: 78.4 KB/s (0.6*10^6 bps)

Errors: total 7068 client-timo 0 socket-timo 0 connrefused 0 connreset 1445
Errors: fd-unavail 5623 addrunavail 0 ftab-full 0 other 0

What we care about is the reply rate/s. We have an average of 73.3 requests per second with a standard deviation of 65.8 using 8 samples.

We also make sure that all the replies were successful. (status == 2xx)

I also checked the database, made sure my AR object was created and that the hits were increased. AR object hits: 2932, which matches the amount of replies reported by httperf.

We don't care so much about the rest of the httperf. Let's move on to the Rails benchmark.

Quick Rails benchmark

Rails is set the same way, running locally in production mode, same httperf settings.

Here are the results:

Maximum connect burst length: 44

Total: connections 2923 requests 2825 replies 1672 test-duration 37.418 s

Connection rate: 78.1 conn/s (12.8 ms/conn, <=1022 concurrent connections)
Connection time [ms]: min 382.7 avg 5635.4 max 36384.5 median 1887.5 stddev 10103.1
Connection time [ms]: connect 3631.2
Connection length [replies/conn]: 1.000

Request rate: 75.5 req/s (13.2 ms/req)
Request size [B]: 319.0

*Reply rate [replies/s]: min 0.0 avg 43.4 max 75.2 stddev 30.8 (7 samples)*
Reply time [ms]: response 1568.1 transfer 36.7
Reply size [B]: header 471.0 content 581.0 footer 0.0 (total 1052.0)
*Reply status: 1xx=0 2xx=1672 3xx=0 4xx=0 5xx=0*

CPU time [s]: user 0.25 system 31.31 (user 0.7% system 83.7% total 84.4%)
Net I/O: 69.4 KB/s (0.6*10^6 bps)

DB hits: 1672

First thing, the database object was created properly and the hits incremented to 1672 which matches the amount of replies reported by httperf.

Then, we notice that on this test, we only got 7 samples, that's more than enough though. The standard deviation is 30.8 which is better than Merb's 65.8. That means that in our benchmarks, the reply speed difference in Merb's requests was bigger than Rails'. Not a big deal, this is not a scientific test but it's good to acknowledge it.

What we really care about is the average reply rate: 43.4

Let's also note that all the replies had a 2xx status, so everything went well.

Results

Based on this really basic benchmark, my Merb app had an average reply rate of 73.3 requests per second against Rails' 43.4 requests per second.

That means that in this very specific case,

Merb is 69% faster than Rails! Sexy!

In other words, my Merb prototype could handle 69% more requests than the Rails prototype in the same amount of time.

I heard people reporting than Merb was 3 to 5 times faster than Rails. Honestly, it really depends on what you do. By using ActiveRecord on both prototypes, I limited the speed difference since AR is not multithread and therefore Merb can't run as fast as it would using Sequel or DataMapper. By actually hitting the database on every single request, I also made sure to really compare ActionPack vs Merb.

Conclusion

The conclusion is simple, I recommended that my client go with Merb. Merb 1.0 is almost ready, the public API has been frozen. My client needs speed and simplicity. Using Merb I get exactly what I need and nothing more. Actually, we'll probably increase the performance by writing a rack handler and bypassing the entire framework for API calls (that should be wicked fast!). Also, as soon as DataMapper becomes production ready, we'll switch to DM and should get way better performance!

Am I suggesting to give up Rails and switch to Merb? Absolutely not! First off, Merb is a "lower level" framework. It requires a deeper understanding of Web Development in general and being more than just 'acquainted' with the Ruby language. So, unless you are an advanced developer or have time to learn, I would suggest to keep on using Rails (start using Merb on personal projects, it's a perfect way of learning). If you have a lot of views and/or use loads of AJAX, RJS, built-in helpers, you probably want to stick to Rails and start looking at how you can do all of that from scratch. By default Rails uses nasty helpers that create inline javascript, and is something you really want to avoid. RJS is fun, but it goes against Merb's philosophy, so you need to make sure you can live without it (note that you can reproduce the same behavior in Merb rendering JS, it just requries more work). If you rely a lot of Rails plugins, you might want to delay your switch, Merb is pretty new and doesn't have a mass-load of plugins yet.

Finally, Merb doesn't have a lot of documentation and changed a lot when 0.9 got released. To understand how Merb works, you will need to go through the source code, specs, Google, and ask on the Merb IRC channel.

It turns out that in our case we have experienced developers, a great need for speed, not too many views and are following Merb's development really closely . I honestly think it's the best choice for my client and I'm excited they accepted to use Merb.

Merb is addressing different issues than Rails and doing it well. I think there is a bright future for Merb. And don't even think that Rails is going away, that won't happen anytime soon!

Recently, Sony Playstation even posted a job post looking for a Rails/Merb developer. This is very promising for the Merb community!

Merb tips II

2008-04-08T02:11:00Z

I the previous post I covered few useful tips for Merb 0.9. The good news is that Merb should get its wiki setup over the week end!

Here is another batch of hopefully useful tips:

In init.rb, you can define a dependency and specify a version number: dependency "merb_fu", ">= 1.0"
If you want to run your application from a subdirectory, once again, in your init.rb file, add: c[:path_prefix] = "/your_prefix" (note, that you can also do that in a specific environment file.)
You feel like limiting a route to a specific request such as a DELETE? In your router.rb file add the following:

12	r.match("/:bucket_id", :method => :delete).to(:controller => "buckets", :action => "destroy")

Since we are talking about routes, what about an iPhone only route?

12	r.match(%r[^/(.+)], :user_agent => /iPhone/).to(:controller => "mobile", :title => "Welcome Apple FanBoy", :action => "show")

what about an admin section for my blogposts?

1234	r.match('/admin') do \|admin\| admin.resources :blogposts end

To finish with the routes, look at the following merb-core spec


  it "should allow you to restrict routes based on protocol" do
    Merb::Router.prepare do |r|
      r.match(:protocol => "http://").to(:controller => "foo", :action => "bar")
      r.default_routes
    end
    route_to("/foo/bar").should have_route(:controller => "foo", :action => "bar")
    route_to("/boo/hoo", :protocol => "https://").should have_route(:controller => "boo", :action => "hoo")
  end

You can set custom routes to only work when connected via SSL, that's just really nice!

Other quick tip. Last time we saw how to install locally all the required gems. Well, you can also freeze merb by doing:

12	merb-gen frozen-merb

Another common IRC question, how do I use Merb's logger. It's really easy:

12	Merb.logger.info('our stuff')

Where info is the debugging level you want to send your message to.

Finally, today in IRC a Rails user asked how reset a session in Merb. Rails has a reset_session method that resets the session by clearing out all the objects stored within and initializing a new session object. Merb simply uses a hash to store sessions, so session.clear will do it ;)

Feel free to add a comment with your Merb tips or leave a question regarding something you can't seem to be able to do with Merb.

Merb tips I

2008-04-05T05:59:00Z

I'm working on a post reporting a recent benchmark I did comparing Rails vs Merb performances for a client's app.

In the meantime, here are few tricks you might need when using Merb 0.9x

In the init.rb file, uncomment and rename c[:session_id_key] (in the Merb::Config.use block)
In the same block, add c[:log_level] = :debug to set a log level
By default, Merb logs to STDOUT, to log to a file, in the config block add c[:log_file] = Merb.log_path + '/development.log' (note that you need to create the file yourself, Merb won't do that)
to save your gems locally, do: sudo gem install gem_name -i gems
need basic HTTP auth? it's now available in core
don't forget to require any plugins, extra gems you need (such as merb_helpers or merb-assets)
don't forget to select your ORM before using the generator( so your generated goodies will be adapted to your ORM)
routes are easy to use. In the console (merb -i) type merb.show_routes to see all your named routes
if you want to use linkto, install merbassets
nested routes example:


  r.resources :channels do |channels|
    channels.resources :shows do |shows|
     shows.resources :episodes
    end
   end

usage:

1234	url(:channel_shows, :channel_id => channel) link_to h(channel.description), url(:channel, :id => channel)

That's it for today :)

In the meantime, check this Merb presentation by Ezra and this DataMapper presentation by Wycats

How Duplo (and I) offended people

2008-03-21T06:59:00Z

My recent and certainly provocative post generated a lot of noise. Just by reading some of the comments on my blog or on reddit I could only notice that some people took offense and some people tried to push the metaphor way too far. At the same time, it generated a lot of interesting discussions even outside of our small Ruby community.

A few clarifications:

Rails is an awesome framework, and to be honest I really like the fact that it's accessible to newbies while offering great tools
Rails Plugins that I like calling Duplo blocks are great and I use a bunch in most of my projects
I, myself wrote a bunch of plugins and gems never as great as the one mentioned in my post though
No, I don't think you are dumb if you don't have a clue how Rails or Rails plugins work

I also agree that the metaphor is very limited and doesn't stretch... but who cares.. it's funny.

What did I mean to say

Rails plugins are useful, use them and ABUSE them. Even though most of them are very well written I strongly believe you would become a better developer if you understand how they work.
A Duplo doesn't do exactly what you want it to do, don't bitch about it. Try submitting a patch or if you can write your own plugin.
If you are using a plugin on a regular basis, give it some love. Submit some documentation, write a blog post, send a thank you email.
A great framework and a bunch of nice plugins are awesome tools for developers but they won't replace your brain.

So, there you go, this time around I tried to be more straight forward and I even apologize if I offended you.

Finally, I don't believe the Rails community is a bunch of obnoxious-arrogant-wannabe-rockstars, and certainly not a ghetto. We all started as newbies and I hope a lot of newbies will join the fun and learn through Rails and later provide the community with a lot of awesome Duplos!

Ruby on Rails: the Duplo generation

2008-03-17T06:39:00Z

I'm sure, at least once in your life you played with Duplos.

Duplo bricks are eight times the size in volume, twice the length, height and width of traditional Lego bricks, and are easier to handle for younger children. Despite their size, they are still compatible with traditional Lego brick.

Duplos are great to introduce kids to the concept of Lego bricks and to get them to think about building their own creations. However you would freak out if your kid would grow up and not want to start playing with Legos and building more advanced/custom stuff.

Unfortunately, that's exactly what's going on in the Rails community right now. We created a generation of Duplo developers.

Rick Olson, AKA Technoweenie fathered a great majority of this Duplo generation. Rick is a Rails core member and a prolific Rails plugin developer. He has written very popular plugins and Rails apps such as:

If Rick is the father, the mother of this generation would obviously be David Heinemeier Hansson, creator of the Ruby on Rails framework. David has always wanted to make our lives easier, providing us with tools to avoid repeating ourselves and a mass-load of tools to create web apps in no time.

Rick, David and others worked hard to provide the community with tools that cut our development times by 20% to 30% and that's just awesome. They basically took their meccano applications and extracted Duplo blocks you can play with.

from to

The problem is that a generation of Rubyists has grown up being used to getting everything pre written for them. They haven't yet passed the "Duplo stage" and basically write applications putting a few blocks together, only writing 10 to 20% and barely understand 5%.

On top of that, what really annoys me is that, these very same developers complain about the existing plugins, always ask for more and don't give anything back to the community.

The problem is that it's always the same people giving and helping. Relatively quickly, the community grows and people supporting it get tired. I won't go as far as Zed and his funny rant but we need to wake up. We need to evolve, learn how Rails magic works, give up the cargo culting and start giving back.

The first thing would be to stop complaining about plugins you use on a daily basis and write your own or fork existing ones. GitHub is a good place to start forking existing projects, if you are a Rails/Ruby beginner, you can help with documentation or submit small patches. Josh Susser wrote a nice tutorial on how to commit changes (patches/documentation).

Why not blog about issues you have just faced and how you resolved them. Start writing small plugins/gems. Try helping people on the various mailing lists.

And finally, drop the Duplos and start playing with Legos - don't use plugins just because they are available to you, make sure you fully understand what the plugins you use do! Learn more about Rails guts and start using it in a way that makes sense to you.

Why not even switch to meccano and take a look at Merb

starting the migration to GitHub

2008-03-05T07:48:00Z

I started moving some of my projects to GitHub.

Here is my GitHub account.

Projects moved to GitHub:

I'm planning on moving GoogleCharts, RandomWordGenerator and some not released stuff to GitHub so people can have fun forking my projects.

Git and GitHub are the new cool things. GitHub is planning on setting up a gem server while they are already offering tarball download and a post-receive hook. (they also plan on becoming myspace for geeks, but that's another story)

Do you have to switch to git and github? Honestly, ...no you don't.. Git can act as SVN, but let's be honest, if you switch to a new SCM it needs to do more. I've been using Git for a couple of months and even though I still don't have a full understanding of this SCM, I really enjoy using it.

So, get over it, learn on your own or purchase this excellent peepcode

Email me to get a GitHub invite (Tom and Chris gave me some invites for readers) or/and try Gitorious.

The fact that some major players (Topfunky, technoweenie, Chris & PJ, jnunemaker and major projects such as capistrano, vlad the deployer and Merb use and support Git is a sign that it's the next big thing.

Also, I believe that a lot of developers will also be motivated to move their plugins/gems to GitHub because they simply can't always maintain their own libs and/or just hope people will fork their project and contribute back.

How to use github and submit a patch

2008-03-03T08:22:00Z

If you don't know about git and github yet, it's time you clean up your RSS feeds and find some good source of information.

Github is used by the Merb core team and I'll show you how to use github to fork Merb, make your modifications and "submit your patch".

This is the exact reason why github is simply awesome, it makes forking projects just super simple and submitting changes even easier.

First thing, you need to have a github account, if you don't have one yet, email me, I have a couple of invitations left, otherwise, just wait until github gets public.

Now, let's go to Merb's repository and fork Merb-core by clicking on the fork button.

Actually, for this example, I'll fork merb-plugins because I want to improve the ActiveRecord rake tasks.

Because I forked merb-plugins, I now have my own forked repo: !

I'll start by checking out/cloning my forked repo locally.

git clone git@github.com:matta/merb-plugins.git

Great I can now make my own changes.... but wait, what if the merb core team makes a change to the code? Well, I need to track their changes. Here is how:

git remote add coreteam git://github.com/wycats/merb-plugins.git

FYI, it adds the following to edit .git/config:

1234	[remote "coreteam"] url = git://github.com/wycats/merb-plugins.git fetch = +refs/heads/:refs/remotes/coreteam/

then

git fetch coreteam

and finally

git checkout -b coreteam coreteam/master

You can now track the latest change and merge them with your branch. Note that you can also track other forks and merge some other changes. (just that feature is worth using git)

Alright, now you can do your stuff, and push your local change to your remote repo at github.

Once you are done, you can simply click on the "pull request button"

fill up the form and select the recipient. (wycats in this example if you want him to merge your changes into the official version of Merb).

p.s: The github guys are working on a gem to make our loves easier, give http://github.com/defunkt/github-gem/tree/master a try. I'll post about the gem when it will be a bit more stable.

resolving git-svn conflicts

2008-02-29T22:40:00Z

I've been using git and git-svn for a little while and never had a problem... until today.

On one of my project, we have a SVN repo but since I prefer using Git, I'm using git-svn.

Git-svn has been great, it let me create my own local branches for each new set of features (that's when I don't forget to create a branch) and to commit all the changes back to svn.

The problem today happened after I did a simple git-svn rebase. I had some sort of error and my local repo looked like it got reverted to the head of the svn repo....

error: patch failed: trunk/app/models/view.rb:1
error: trunk/app/models/view.rb: patch does not apply
[blah blah]
sing index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...

I hadn't committed to SVN for 24 hours and had a lot of work that was just checked in locally... You can imagine the panic. Rob started digging in the .git repo to finally find the hash representing the latest delta before the rebase. With the help of the #caboose guys, I did a simple

git reset --hard hash-name

Which restore my repo to the pre SVN commit state. Awesome... however I still had issues to commit my stuff. After a little while I as able to commit again, worked a bit more and tried to commit again.... same error :(

But this time I noticed I could simply do

git rebase --abort

to restore the original branch.an

But I still couldn't commit properly... until I discovered that I just needed to fix the conflicts manually using

git-mergetool

git-mergetool uses whichever merge tool available: kdiff3 tkdiff xxdiff meld gvimdiff opendiff emerge vimdiff filemerge

I fixed my conflicts in no time, then did a

git rebase --continue

and finally

git-svn dcommit

Looking back, I wish I knew how to properly deal with conflicts when using git-svn, I wasted a bit of my precious time ;) hopefully this post will help you.

p.s: here is an interesting use of Sake to handle git-svn

GumGum launch, or how to avoid a disastrous launch.

2008-02-28T08:42:00Z

For the last few months I worked on a Rails app called GumGum. GumGum is a licensing and distribution platform for online content. GumGum believes paying a flat rate to license content for online use is illogical. Offline, the flat rate model works because distribution is finite. Online, thanks to Google and other search engines, a story lives forever. GumGum has developed two usage models to fairly monetize a license: pay-per-use and ad supported. Pay-per-use allows publishers to license content on a CPM basis. Ad supported subsidizes the licensing revenue through advertisements, providing the publisher a free license.

Simple concept and great team. When Ari Mir and Ophir Tanz approached me to work on their app I was impressed by their understanding of Web Business. It was not their first start up and as a consultant, you usually prefer to work with some experienced clients. Ari, GumGum's CPO was great at defining realistic expectations, and limit the scope of the application which was great since I'm an Agile/XP fanboy. I believe that really helped us meeting our dead line without feature creeping the app.

We were finally ready to enter a public beta phase, we were running our production server on a godaddy VPS (no comment) and were on the process of migrating to slicehost where we were running our staging environment. The plan was to go public on Tuesday 12 and have a feel for the load. We also setup an EC2 image to handle some of the background tasks and were thinking of moving to Engine Yard within few months.

Everything was ready for a launch, when the day before D Day, Ari calls me telling me: "Good news Matt, TechCrunch is going to publish a blog post about us!". While he sounded really excited, I couldn't believe me ears.

The app wasn't optimized, darn it all, I hadn't even properly indexed the database, no caching, nothing. (it was in the backlog, things I was planning on doing after the public beta launch)
We would never be able to handle the load. Running an app like ours definitely needs more than a 256Mb shared CPU shared host.
Ari asked me if they could embed one of our license object in TechCrunch home page.
I had 12 hours to get stuff ready.

Trying to be ready:

TechCrunch has 688k RSS subscribers! Even though I asked Ari not to embed one of our protected object inside TC's home page, I knew he couldn't say no to Mike Arrington. (little did I know, they would end up embedding 2 licenses)

I hurried up and started properly indexing the database, ask advice to some of my cabooser friends, did some quick re-factoring but obviously didn't have enough time to rewrite the way we handle tags and tooltips (the overlay popup window that displays when you rollover a picture in our browse page). Rob helped me getting ready and we transformed our staging environment into a secondary production server hoping we would split the load.

Since we didn't have a load balancer, we tried to cheat by setting up two A records for gumgum.com. The idea was that browsers would pick an ip randomly and since we were only using 1 database, everything should be fine. That was the theory, during our tests we realized that our Flash object was not using the same IP than the webpage and that was creating random issues with our licensing system.

We were running out of time, I turned on page caching on few actions (home page) so even though my mongrels would die, I knew we would at least be able to display something. Because of our complex authentication system, I couldn't cache much and I didn't have time to re-factor a lot of my code or to setup memcached. (on top of that I knew the product was stable and I didn't want to mess up with it).

The Storm:

The article got published and in seconds we saw the load on the main box going from 0.2 to 2.0! We were still dealing with the DNS issues and some pictures were not getting loaded. When Rails got above 60 req/s we started dropping some requests and returning 500s. That was bad, really bad. Our browse page (not optimized) was down and we handled around 80k requests in a matter of few hours. What you don't know is that each of our flash object generates a couple of requests, and that TechCrunch decided to embed 2 objects on their home page.

Remember, it was our launch, which was not meant to be a big launch, we were not prepared for that!

Dealing with the load:

The good thing is that my code didn't leak memory, Rails was stable and it was a good sign. We all know that "Rails CAN scale", but we also know that you need to have a proper hosting to do that. Godaddy was not handling the load and it was just the beginning. So what do you do when you have to scale and that your code isn't total junk? You call a real hosting company like EngineYard or Joyent. I was lucky Ezra was online. I quickly explained to him the situation, we discussed the reason why our site was not doing so great and defined the bottlenecks. It was simple, we needed more humph. The normal waiting time to get a slice on EY was around 2 weeks back then. Ezra knew that we couldn't wait 2 weeks :p He was simply awesome, managed to find us 2 slices, set them up for us and setup our app in less than 2 hours. He even helped us configuring our old server to get NginX to bounce the requests to EY while the DNS was propagating.

As soon as the switch was done, everything loaded perfectly, not a single request dropped, it was simply great.

Since we are talking about scaling, hosting and facing issues, I'd like to make a simple point: $349.00 a month for a slice is NOT expensive for what you get. First off, EY slices are setup superbly, you get awesome performances out of them, they come ready for your app and you don't have to do much. Secondly, I'm not a sys admin, I really hate dealing with servers, packages, configuration, compiling crap that won't compile because I'm lame... etc...
EY has a bunch of experts (both Ruby and Sys admin experts) available 24/7 that always go the extra mile to help you. Last time I checked I couldn't hire a good sys admin for, let's say the price of 3 or 4 slices. Now, we get a team of guys, available, knowing their stuff and helping me for way cheaper than hiring a sys admin dude. The other thing is, I'm a consultant, I don't want to deal with servers and charge my clients for that. I also want them to be reassured and know that "yes, we can scale". The price of an EY/Joyent slice is the price you pay to be confident you can take the load.

So what now?

We have a lot of optimization to do, we are working on moving more of our stuff to EC2 to avoid being charged for bandwidth twice (user => EY => S3), we are setting up a queuing system and we have a lot of nice features to deploy. But one thing for sure, I can focus on my code since I know EY has my back and I don't have to worry about our servers.

p.s: GumGum is doing great, a lot of people are really interested in the product, new features are coming up (we already auto import content from some major Paparazzi agencies) and we are getting a good press coverage

Misc tips and tricks

2008-01-30T08:00:00Z

I haven't posted for quite a long time. The thing is I moved to a new place and I'm really busy on working clients + setting up my new office + dealing with way too much paperwork.

Anyway, enough excuses, here are few tips that I believe will be useful to some of you:

ZenTest Autotest

I love autotest, but you might have noticed that sometimes (especially on big projects), ZenTest might start using more CPU than expected. On my machine, that results in the fan going off and annoying the crap out of me.

The solution is quite simple, exclude all folders you don't need to monitor. To do that, update ZenTest to version 3.8.X

sudo gem update ZenTest

(older version had a different syntax)

Now, edit your .autotest that should be located in ~/.autotest (if it doesn't exist, create it).

Finally add the following code:

1234	Autotest.add_hook :initialize do \|at\| %w{.svn .hg .git vendor}.each {\|exception\| at.add_exception(exception)} end

I personally freeze rails in vendor and I autotest is way happier when it doesn't have to monitor some extra files. (note that we also exclude folders such as .git or .svn) (you can also include files etc... read more there)

RSpec

RSpec is certainly my favorite Ruby tool and I'm glad to say that most of my SD.rb friends finally got convinced!

Now, few people complained to me about spec failures outputting the full stack such as:


 The Sessions controller should fail since it's a test' FAILED
 expected true, got false
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/expectations.rb:52:in `fail_with'
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/expectations/handler.rb:21:in `handle_matcher'
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/expectations/extensions/object.rb:34:in `should'
 ./spec/controllers/sessions_controller_spec.rb:25:
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/example/example_methods.rb:78:in `instance_eval'
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/example/example_methods.rb:78:in `run_with_description_capturing'
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/example/example_methods.rb:19:in `execute'
 /opt/local/lib/ruby/1.8/timeout.rb:48:in `timeout'
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/example/example_methods.rb:16:in `execute'
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/example/example_group_methods.rb:288:in `execute_examples'
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/example/example_group_methods.rb:287:in `each'
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/example/example_group_methods.rb:287:in `execute_examples'
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/example/example_group_methods.rb:121:in `run'
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/runner/example_group_runner.rb:22:in `run'
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/runner/example_group_runner.rb:21:in `each'
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/runner/example_group_runner.rb:21:in `run'
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/runner/options.rb:89:in `run_examples'
 test_app-git/trunk/vendor/plugins/rspec/lib/spec/runner/command_line.rb:19:in `run'
 script/spec:4:

  Finished in 6.035147 seconds

  400 examples, 1 failure

We can really easily change that, open you spec.opts file located in your spec folder.

it probably looks like that:


  --colour
  --format
  progress
  --loadby
  mtime
  --reverse
  --backtrace

Get rid of "--backtrace" and your new failure should look like:


  1)
  'The Sessions controller The Sessions controller should fail since it's a test' FAILED
  expected false, got true
  ./spec/controllers/sessions_controller_spec.rb:25:
  script/spec:4:

  Finished in 0.269956 seconds

  15 examples, 1 failure

Other stuff you may find interesting (in no particular order):

RSpec on Rails Matchers plugin

2008-01-04T03:46:00Z

RSpec is an awesome testing framework. On top of being the first Ruby BDD framework the core team is doing a great job in enhancing our testing experience and therefore the quality of our code.

This time, I don't want to introduce to the latest changes but instead showing you what Josh Knowles, Bryan Helmkamp and myself came up with.

RSpec on Rails matchers plugin + TextMate Bundle

Matchers are some sort of helpers that will help you cleaning up your tests. We simply came up with a collection of matchers that we think will make your like easier.

We divided the matchers in 3 categories:

Associations

Verify that the association has been defined. (doesn't verify that the association works!)

Usage examples:


    @post.should have_many(:comments)
  
    @comment.should belong_to(:post)
  
    @user.should have_one(:social_security_number)
  
    @project.should have_and_belong_to_many(:categories)

Validations

Verify that a validation has been defined. (doesn't test the validation itself)


    object.should validate_presence_of(:attribute)

    object.should validate_confirmation_of(:attribute)

    object.should validate_uniqueness_of(:attribute)

    object.should validate_length_of(:attribute, :between => 5..10)
    
    object.should validate_length_of(:attribute, :is => 5)

Views

My personal favorite matchers, you can now do stuff like:


    it "should render new form" do
        render "/users/new.html.erb"

        response.should have_form_posting_to(users_path) do
          with_text_field_for(:user_name)
          with_text_area_for(:user_address)
          with_text_field_for(:user_login)
          with_text_field_for(:user_email)
          with_submit_button
        end
    end

Check the readme for more information and details on the added matchers. I personally recommend you try the TextMate Bundle on top of being a perfect tool for lazy devs, it also lists all the available matchers and is an excellent way of learning.

We just released our first release yesterday, this is not a final version and we will keep on improving the code. If you have suggestions and patches feel free to open a ticket there.