Piotr Sarnacki

Mountable apps tutorial

2010-12-21T00:00:00-08:00

I finished my work on RubySOC a few months ago and I wanted to write this post quickly after that, but life, as usual, verified my plans. Unfortunately, attending a few great conferences left me with no money, so I needed some time of hard work to pay all the debts (nevertheless, all those trips were really worth it, especially rubyconf – I really felt in love with New Orleans!). I want to apologize to everyone who really waited for that, especially those who heard that this blog post will be written “very soon” ;) Now… let’s cut the emo talk and get to serious business!

I’ll start with a short introduction for those, who aren’t familiar with the concept of mountable apps in Rails 3 (edge) yet. The goal for implementing mountable apps was to allow running 2 rails applications in one process. Unfortunately, that goal was not achieved and we settled on a simpler approach. If you want to create a mountable application, you can use rails engines, which will be much more powerful in Rails 3.1. For the list of major changes, please check my last post on that topic.

Although almost all of the changes made to engines were usable shortly after summer, the process of setting up a new engine, suitable for mounting, was really rough. Of course, there is a great gem, enginex, by José, but at that time it only generated bare structure for any kind of rails extension and there were no things specific to mountable apps (as mountable apps haven’t existed during the time of writing enginex). That’s why I ported enginex to rails, with some changes that were needed to merge it to the core, to make that task simpler.

Let’s start with the tutorial. First things first. As the new APIs work only on rails edge, you need to get rails from github:

git clone https://github.com/rails/rails.git
cd rails
bundle install

To generate any rails extension in rails 3.1 you can use rails plugin new command. It generates a directory with lib directory, gemspec, tests and dummy application for testing. There are 2 really important things here:

gemspec is automatically generated, unless you use --skip-gemspec option – I hope that this will help to move more people to gem plugins
dummy application, which by default lives in test/dummy – this is standard rails application that you can test your plugin

To generate a mountable engine, we can use:

bundle exec ./bin/rails plugin new ../blog --edge --mountable
cd ../blog
bundle install

We need to add --dev or --edge option to ensure that Gemfile will point to rails edge version, either from local clone or github. I used edge here, cause it’s better for most of you – you can get newest rails version with simple bundle update. Mountable option will add files that will come in handy for developing a mountable engine, such as: integration tests, Blog::ApplicationController, config/routes.rb and so on.

Let’s see how the key files look like.

Probably the most important file is lib/blog/engine.rb, which keeps definition of the engine:

module Blog
  class Engine < Rails::Engine
    isolate_namespace Blog
  end
end

The thing that differs from what we could see in Rails 3.0 engine, is isolate_namespace which makes engine isolated from host application. If you don’t remember how it works exactly, please check the documentation or my last post.

Next important thing is config/routes.rb file:

Blog::Engine.routes.draw do
end

These are empty routes, but as you can see, they belong to the engine, not to the host application.

The last thing that you may not be familiar with is the dummy application located in test/dummy. This is a standard rails application that will be used to test engine, both with automated tests and manually during development. The nice thing about the new plugin generator is that with --mountable option, it automatically mounts engine in test/dummy/config/routes.rb

Rails.application.routes.draw do
  mount Blog::Engine => "/blog"
end

Ok, let’s write some code! Or… actually don’t write code, I’m too lazy, I’m gonna use scaffold generator:

rails g scaffold post title:string body:text

While the scaffold should behave exactly the same as the one generated in regular app, there are some differences. Notice that migration for that is called create_blog_posts instead of create_posts. Also almost all of the files are places in blog/ subdirectory. When you open app/models/blog/post.rb, you will see:

module Blog
  class Post < ActiveRecord::Base
  end
end

Everything is namespaced for a good reason: we want to avoid conflicts between engine and host application.

Ok, now we can migrate our database (sqlite3 by default):

rake db:migrate

Engine’s Rakefile includes all the tasks that are needed to manage database. Migrations are run from both engine’s and dummy’s application directories to make development and testing easier. Finally, it’s time to check it!

rails s

Now point your browser to http://localhost:3000/blog/posts. You should see the standard rails scaffold working – the only difference is that it’s namespaced with /blog path. It was easy, wasn’t it?

As you can see, creating a mountable application using engines is pretty straightforward. I haven’t shown any way to better integrate a mountable application with the host application (e.g. using User model from the host application or adding some configuration options to the engine). It’s on purpose ;) I haven’t got time to develop anything good yet. You can start playing with it yourself! Be sure to let me know if you come up with some good patterns.

Lightweight controllers with rails 3

2010-12-12T00:00:00-08:00

Some time ago I showed you, how easily you can reuse rails 3 modules. Today I want to present lightweight controllers, which is also really really easy in rails 3.

Why would you want controller to be more lightweight? Speed of course. We all know that rails can’t scale and ruby is slow, let’s make it a little bit faster ;)

Let’s start with getting familiar with new controller architecture. The base for all the controllers in rails is AbstractController::Base. If you want to build controllers without support for handling requests, you can start with that as a base. What if we want to create something looking more like standard rails controllers? We can check how does implementation of ActionController::Base look like: ActionController::Base

As you can see, it inherits from ActionController::Metal and includes bunch of modules. Looks like Metal will be a good start for our purpose. Let’s say that we want to render a simple JSON response generated from a model:

# app/controllers/api_controller.rb
class ApiController < ActionController::Metal
  include ActionController::Rendering

  def index
    render :text => "Good morning!"
  end
end

As you can see I created controller that inherits from ActionController::Metal and included ActionController::Rendering. To render simple JSON response, we will not need anything more.

How fast is it comparing to ActionController::Base? For comparison I’ve created controller with exactly the same action:

# app/controllers/home_controller.rb
class HomeController < ActionController::Base
  def index
    render :text => "Good morning!"
  end
end

I’ve done a few benchmarks, but don’t take it too seriously, these are not scientific, I just wanted to show you how approximately can it differ. Here are ab results: https://gist.github.com/738168. As you can see, ActionController::Metal is almost 40% faster. This can of course vary, depending on version of ruby, concurrency (I used 1) and other factors, but clearly it can give you nice boost.

Of course trained eye can easily spot an issue here. Do you think that someone will actually return a response like that? Hardcoded string?Srsly? I guess not. But there certainly are situations, when you can profit from using that kind of controllers. If all you need to do, is to return text response, that most of the time is fetched from cache and is hit frequently, this should work pretty well. Some of you may also ask, ~~why can’t we use Rails Metal, which is available for quite some time now~~ rack middleware (rails metal was removed in rails 3). We could, but then we loose possibility to use router and include other modules to ActionController::Metal (like ActionController::Helpers or other useful things).

I hope that this post will help you to experiment with new Rails 3 cool features! If you want more of that stuff, in better form and with much better explanation and depth, I highly recommend reading José Valim’s book Crafting Rails Applications!

Mountable engines - RSoC Wrap Up

2010-09-14T00:00:00-07:00

As some of you probably know, during the summer I was working on “Rails mountable applications” project thanks to Ruby Summer of Code. It was great experience, I learned a lot new things and I hope that I delivered something useful for the community. I tweeted that the other day, but I must emphasize: it would not be possible without great support from my mentors: Carl Lerche, Yehuda Katz and José Valim! They helped me a lot with both ideas and implementation. The biggest internet hug should go to José, who spent enormous amount of his time on discussions, reviewing my commits and helping me to set the goals. I also want to thank all the sponsors and people that helped to organize Ruby Summer of Code.

Getting back to my work. My main task was to extend capabilities of rails engines. Although at first I wanted to get straight to mountable full rails applications, after discussions with Carl and José, I knew that starting with the smaller target will be more sane way to go. The biggest problem with implementing mountable applications (that is running more than one rails application in same process) is configuration and application initialization in general. Right now config values are shared between all the railties classes, because they’re kept in class variables. The other problem with mountable apps is that they’re pretty new concept in rails community and we will need some time to settle standards. With those problems in mind, it is much better to test the concept with engines and implement truly mountable applications later, based on results and feedback. Right now, engines are almost as powerful as applications. The main difference is that engine can’t be run without an application. The best thing about such strategy is that API for using more than one application is already here, so if it is needed, converting engine to application will be as easy as changing a bunch of configuration files.

In that post, I would like to briefly describe my changes.

Extending engines

First thing needed to allow build entire applications on engines, was to add some of the application’s features to engines. That’s why engines got:

its own middleware stack
routes
plugins support
config/environment.rb

With all the capabilities that engines had before, engine is now almost as powerful as application.

Mounting engine

Since engine is now a rack app, you can simply mount it in your application’s routes:

Rails.application.routes.draw do
  mount Blog::Engine => "/blog"
end

This will mount Blog::Engine at /blog path. There are 2 huge benefits of such method:

you can change the engine’s path, which was much harder before
you can use all of the routes magic

Want to mount your engine with dynamic scope? No problem:

scope "/:username", :username => "default" do
  mount Blog::Engine => "/blog"
end

You can also use devise to force authentication for the engine:

authenticate :admin do
  mount Tolk::Engine => "/tolk"
end

It’s easy, isn’t it?

Cross applications routes API

Since you can mount engine with its own router inside application, there is a possibility of having more than one router in your app. To handle all the routers easily, there are new helpers providing access for each router. Application’s router is always available as main_app. That said, you can call any application route from engine just like that:

main_app.logout_path
main_app.root_path

Helpers for other routers are available after mounting an engine:

mount Blog::Engine => "/blog"

# default helper for such engine is "blog":
blog.posts_path
blog.root_path

If you need to change the helper name, just pass the :as attribute:

mount Blog::Engine => "/blog", :as => "my_blog"

# now helper is called my_blog:
my_blog.posts_path
my_blog.root_path

You can also use those helpers in polymorphic url (which is used, among the other places, in form_for):

polymorphic_url([blog, @post])
form_for([blog, @post])

Note that you have to explicitly add those helpers to polymorphic_url only when you need to call an engine route from your application or from another engine.

Namespacing

Having more than one source of controllers, models and helpers can cause conflicts. Imagine you have Post model in your application and you would like to install blog engine having model with the same name. To avoid that, you can put your engine inside the namespace. Although it will help you with conflicts, it’s not enough level of separation for some engines. There are basically two possible scenarios here. One of them is shared engine, when you want to share helpers between engine and application. A good example of such engine is Devise. The other use case is when you want to make isolated engine, which will not likely share anything. It could be an engine that provides tools for application like Tolk or the other app like blog or CMS.

Engines are shared by default, therefore you must explicitly mark it as isolated with isolate_namespace method:

module Blog
  class Engine < ::Rails::Engine
    isolate_namespace Blog
  end
end

With such engine definition, only the engine’s helpers and routes will be included inside engine’s controllers and views.

Migrations

If you are one of NoSQL (or rather schema less) database users, you can probably skip that point. In other cases you will probably need migrations. We decided that the easiest way to handle engine’s migrations is to copy them to application’s db/migrate directory and change their timestamps for not breaking the migrations timeline. It can be done by simply calling rake task:

rake railties:copy_migrations
# or to copy only selected engines
rake railties:copy_migrations RAILTIES=foo,bar

The nice side effect of copying migrations is that you can easily review them before applying.

Assets

The chances are that you will have some assets in your engine’s public directory. The default way of serving assets in development mode in Rails is ActionDispatch::Static middleware. If you have mounted any engine, it will automagically serve their assets. In production you have 2 options:

stay with ActionDispatch::Static by turning it on with: config.serve_static_assets = true in your environment.rb
create symlinks to engine’s public directories

You can automatically create symlinks with a rake task:

rake ralties:create_symlinks

If you want to change the default asset path, you can set it in Engine definition:

module Blog
  class Engine < ::Rails::Engine
    config.asset_path = "/my_blog_assets%s" # note %s at the end
  end
end

This will change both ActionDispatch::Static’s and create_symlinks rake behavior.

Why does it matter?

Although engines are available since Rails 2.3 and even ealier as a plugin, building isolated applications using them can be hard. With new API, building entire apps (like forum, blog or CMS) that can be reused, will be much easier. But hey, big components are evil, right? I would say that it depends on your needs. There is huge space for engines as tools to help develop your application. If you use i18n, there is Tolk. Most of applications could probably benefit from Rails Admin (it’s also one of the RSoC projects). Also the new Carlhuda’s ;-) work on Rails 3.1 assets is based on engines.

Another topic are applications. You can argue that making flexible application that would suit needs of many developers will be hard and inefficient… and you are probably right. However there are many situations when you need to quickly add some CMS or small forum to your application and you pretty much do not care about features. It’s not your core product, you just need something simple. Of course you can always do it by yourself, but what are the benefits? If you’re not trying to revolutionize CMS systems and just want an easy way to add a few articles to your site, you can use something generic. If it’s really needed, you can always write your own better suited engine later on and replace the previous one.

What’s next?

My work is already merged to rails master and most of the things are finished, but there are still some places that need more work. If you have any suggestions on engines, feel free to comment, add a ticket on lighthouse or send a pull request with changes.

There is also need for engines creators that will battle test the new API and new concepts. If you have any questions feel free to contact me on twitter, github or by email. I will also prepare a guide shortly, so stay tuned!

RSoC status: Namespacing engines

2010-09-06T00:00:00-07:00

It’s been a while since my last post. After finalizing my work on routes and few other things I went to Croatia for 2 weeks, to rest from the computer and programming. But fear not, I will not count that 2 weeks into RSoC time and I will work for 2 weeks longer to not shorten actual development time.

The big news is: my work has been merged to rails official repository recently! That means that you can start using the things that I described right away (you just need to use edge version of rails).

The next thing that needed to be done to allow mounting engines easily is namespacing. Namespacing is crucial to avoid conflicts between engines and application itself. Imagine that you have application with Article and Comment models and you want to mount blog engine into it. The chances are, blog engine will also have Comment model and corresponding controller and helper. Without namespacing such configuration would end up with conflicts. To avoid that, we can put all of the things in an engine into namespace:

# ENGINE/app/models/blog/comment.rb
module Blog
  class Comment < ActiveRecord::Base
  end
end

# ENGINE/app/controllers/blog/comments_controller.rb
module Blog
  # note that ApplicationController here is in fact Blog::ApplicationController
  class CommentsController < ApplicationController
  end
end

At first, it may seem that this will not require any changes to Rails, it’s standard Ruby technique, right? To some degree it’s true, but there are some places that made it hard to use namespaced engines before.

Let’s look at controllers as an example. In rails 3.0, when you create any class that inherits from ActionController::Base, it will get all of the application helpers. It’s default behavior and it’s perfectly ok when you have only one application. Problems appear when you want to mount engine with its own controllers and helpers. With Rails 3.0 it will be a bit hard. Even if you namespace your controllers and helpers, you will get all application’s helpers instead of engine’s ones. The simple solution would be to load only helpers within current namespace, but that would break some of the existing engines.

There are basically 2 types of engines. Some of the engines are considered to be a part of application, they provide helpers and controllers that will be used within application and there is no need to isolate them (take Devise as example of such engine). I will call it “shared engine”. The second use case is engine that should not share anything, like blog engine. I will call such constructs “isolated engines”.

The problem with having shared and isolated engines is the fact that in both situations you can namespace stuff inside, but the behavior should be slightly different. While you would probably like to share helpers from shared engine, even if they’re namespaced, you would not want that in case of isolated engine. To distinguish this situations you need to explicitly say that engine is isolated. Otherwise it will be shared. To make engine isolated, just use isolate_namespace() method in Engine definition:

module Blog
  class Engine < Rails::Engine
    isolate_namespace(Blog)
  end
end

With such code, all of the stuff namespaced with Blog will be isolated. There are quite a few effects of that. As I mentioned before, one of the problems are controllers. With engine explicitly marked as namesapaced, controllers will include only helpers from within the same namespace and they will include only an engine’s routes.

The next thing is router itself. When you have every controller inside the module, normally you would have to reflect that in routes:

Blog::Engine.routes.draw do
  scope(:module => :blog) do
    resources :posts
  end
end

This code will ensure that posts resource points to Blog::PostsController. With isolated engine this is not needed at all. You can ommit that scope, cause it will be applied automagically behind the scene. With engine marked as isolated you can just do:

Blog::Engine.routes.draw do
  # everything here is namespaced with :blog namespace
  resources :posts
end

The next thing to change was ActiveModel::Naming. It is used in many places in rails and by default it leaves namespace, e.g. it would convert Blog::Post to blog_post. While in some places it’s ok, it can be a major PITA in other ones. The first pain point is polymorphic_url, which is used among the other things in form_for. Let’s consider example of Blog::Post model:

# @post = Blog::Post.new

<%= form_for(@post) do |f|%>
  <%= f.text_field :title %>
  <%= f.submit %>
<% end %>

Without any changes in ActiveModel::Naming, such code would try to use blog_posts_path helper. Using namespace here is not needed at all, as you can’t have a conflict with helpers from other routers. That’s why in isolated engine form_for(@post) will use posts_path and thanks to that, we didn’t need to add that prefix to all the routes.

The next thing connected with forms and model name are param names. Normally, the last piece of code would generate field <input type="text" name="blog_post[title]" id="blog_post_title" />. That kind of prefixes are also not needed as it’s highly improbable that you would need params[:post] for some other thing (and even if it is the case for you, you can use :as => :some_other_name to change the default). The blog_ is also omitted here and in isolated engine the field would look just like that: <input type="text" name="post[title]" id="post_title"/>.

All of the other places where ActiveModel::Naming is used, like partials, generating dom elements or i18n, take the namespace into account.

As you can see, there was quite a few changes in Rails code to handle namespaces, but in the end they’re pretty transparent to for the developer.

Rails3 modularity

2010-07-31T00:00:00-07:00

Since I spend a lot more time with Rails 3 codebase, I am more and more amazed by its modularity. Rails 3 is much more extensible than I would have ever thought. As a part of my rubysoc project I wanted to port merb-parts to Rails 3. Some of you may never heard of merb parts and ask what the hell are parts? Do I need it? Parts are a bit like controllers, but much more lightweight and without overhead of handling requests. I like to think about parts as powerful partials. If you have a partial and you would like to add a bit more logic to it or fetch something from database, you could probably benefit from using parts. Let’s go with a simple example:

# app/parts/articles_part.rb
class ArticlesPart < Parts::Base
  def index
    @articles = Article.limit(params[:limit] || 10).order("created_at DESC")
  end
end

# app/parts/views/articles_part/index.html.erb
<ul>
  <% @articles.each do |article| %>
    <li><%= article.title %></li>
  <% end %>
</ul>

This simple part will fetch last params[:limit] articles or 10 if limit is not provided. Let’s use it in our view:

  <%= part(ArticlesPart => :index, :limit => 5) %>

Such call will render list of last 5 articles.

The question is: how much lines of code does it take to implement something like that with ability to render views, layouts, :inline, use helpers, filters and much more? Over 100 lines of code including part() helper and a railtie (railtie is used to plug it into rails) – if you don’t believe me grab the repo and check it yourself.

How is it possible? Let’s look at the implementation of Parts::Base:

require 'parts/default_layout'

module Parts
  class Base < AbstractController::Base
    attr_reader :params

    include AbstractController::Layouts
    include AbstractController::Translation
    include ActionController::Helpers
    include AbstractController::Rendering
    include ActionController::ImplicitRender
    include DefaultLayout
    include AbstractController::Callbacks

    def initialize(controller, params)
      @params = controller.params.dup
      @params.merge!(params) unless params.empty?
      self.formats = controller.formats
    end

    def self.inherited(klass)
      super
      klass.helper :all
    end
  end
end

That’s all… ? Yes!

As you cans see Parts::Base inherits from AbstractController::Base, which gives it really basic functionality. Additionaly a few helpers are included to add a bit more behavior. The only mixin that I needed to create myself is Parts::DefaultLayouts which ensures that layout with the name of the part is rendered by default, unless :layout => false or there is no such layout in layouts directory.

Ok, that all is nice and dandy, but how does it work internally?

The implementation of such pattern can be demonstrated with such code:

class Foo
  def foo
    puts 'foo'
  end
end

module Bar
  def foo
    puts 'bar'
    super
  end
end

module Baz
  def foo
    puts 'baz'
    super
  end
end

class Omg < Foo
  include Bar
  include Baz

  def foo
    puts 'omg'
    super
  end
end

Omg.new.foo #=> omg
            #   baz
            #   bar
            #   foo

There are 2 modules (Bar and Baz) and 2 classes (Foo and Omg) implementing foo method here. Omg class inherits from Foo and additionaly modules Bar and Baz are included in Omg. This code takes advantage of ruby object model. How does it work?

When you instantinate Omg object with Omg.new and call method foo on it, it looks for foo method in current class and its superclasses. So first method that will be actually called is Omg#foo. This method is calling super, so ruby will look for method foo also in Omg’s superclass. At first you could think that it’s Foo, but internally Ruby treats modules as superclasses. That said, the next superclass will be the last included module, which is Baz. After that Bar’s and Foo’s methods will be invoked.

This one of the best patterns I have seen in Ruby so far. Not only it allows to extend objects easily, but also to reuse small chunks of code making the whole thing more modular. If Rails would stop at standard inheritance that we know from Java and other similar OO languages, creating something like parts would be much harder. In Parts::Base I used a few modules from AbstractController, but also a module from ActionController. With implementation without mixins, I would need to take all or nothing approach.

If you’re a Ruby developer, please think about that pattern while writing your next library. It can really help to make your code easier to extend and reuse. I was amazed how easy it was to implement all that stuff and I love the way that Rails is going!

RSoC status: routes (a.k.a OMG it's hard)

2010-07-20T00:00:00-07:00

In my last post I briefly described some of the RSoC changes and plans. One of the things that I left is router. The topic is much harder and I think it deserves separate blog post.

Basically, the problem with engines and routes is that now you can have more than one router. That leads to many issues that need to be solved. As with my last post, things described here are not in Rails core yet, so you will have to wait a bit to actually use it.

For the good start, let’s identify things to consider:

recognition
generation
named routes from more than one router (for example posts_path)

Recognition

At the beginning I would like to explain how does recognition work. Here is a simple example of engine mounted in application:

# APP/config/routes.rb
Foo::Application.routes.draw do
  resources :users
  mount Blog::Engine => "/blog"
end

# ENGINE/config/routes.rb
Blog::Engine.routes.draw do
  resources :posts
end

As you can see engine provides :posts resource and it’s mounted at /blog path. Let’s study simple request to our blog:

GET /blog/posts

Application’s router will match the first part of the path, which is /blog, with /blog mount point and pass the request to rack app mounted there (which in that case is Blog::Engine). To allow engine recognizing path properly it needs to pass /posts as a path. The /blog (which I will call prefix later on) is not a part of mounted engine, but we don’t want to loose that information by simply removing it from PATH. In that case /blog prefix is attached to env["SCRIPT_NAME"] (SCRIPT_NAME is part of rack’s spec). That way, engine will get only /posts part as PATH, which will allow to properly recognize it.

The only one problem with recognition is connected with rack middlewares. Consider such example:

# APP/config/routes.rb
Foo::Application.routes.draw do
  mount Blog::Engine => "/blog"
  match "/blog/omg" => "omg#index"
end

# APP/app/controllers/omg_controller.rb
class OmgController < ApplicationController
  use SomeMiddleware
end

# ENGINE/lib/blog/engine.rb
class Blog::Engine < Rails::Engine
  config.use SomeMiddleware
end

What will happen when you will request /blog/omg path? It will call the Blog::Engine first, as it has higher priority than /blog/omg. Request will pass through the Engine’s middleware stack firing SomeMiddleware and hit the Engine’s router. If /omg is not a valid route for Blog::Engine it will return 404 and Application’s router will try next route, that is /blog/omg pointing to OmgController. Middleware stack for that controller also includes SomeMiddleware so it will be fired again, which is probably not something that we want.

The solution is to mount apps with lower priority than other routes.

Generation

The problem with generation is related to mount point, the place where engine is mounted. Consider such example:

# APP/config/routes.rb
MyRailsApp::Application.routes.draw do
  match "/" => "users#index"

  scope "/:user", :user => "drogus" do
    mount Blog::Engine => "/blog"
  end
end

# ENGINE/lib/blog/engine.rb
class Blog::Engine < Rails::Engine
end

# ENGINE/config/routes.rb
Blog::Engine.routes.draw do
  resources :posts
end

Blog::Engine is mounted at "/:user/blog" (let’s just suppose that blog implements multi user setup) and it provides posts resources. As you can remember from recognition part, /:user/blog part is engine’s prefix. Additionally, default user is set to “drogus”, so when :user is not specified it will be set to drogus. Imagine that you need to generate path to posts. Something that you would normally achieve by calling posts_path method. With more than one router the situation is not so simple. We can’t simply use named url helpers (like posts_path) inapplication’s controllers, so we need some other way to handle that (if you’re wondering why can’t we just use posts_path, it’s explained in named routes section of this post).

First problem connected with that is API. Jeremy Kemper came up with helper that will allow using mounted engine’s url helpers by simply calling some_engine.posts_path. The name of that helper is taken from Engine’s name or from :as option used in mount method. Because :as option is not provided in my example, helper’s name will be blog_engine (based on Blog::Engine). Using that helper we can generate paths for Engine:

blog_engine.posts_path
blog_engine.url_for @post
blog_engine.polymorphic_path @post
blog_engine.url_for :controller => "posts", :action => "index"

We can also generate application’s urls with app helper:

app.root_url

The next problem to solve is behavior of such helpers. At first, it seems that it’s fairly easy, but in reality there is quite a few cases to handle. The thing that is most important while considering url generation is context. We can generate urls in application’s controllers, engine’s controller’s, ActionMailer or regular classes not related to Rails itself. Let’s go through each case.

Generating engine’s url inside application’s controller

# APP/app/controllers/foo_controller.rb
class FooController < ApplicationController
  def index
    blog_engine.posts_path #=> "/drogus/blog/posts"
  end
end

Generating posts path inside one of application controllers (and consequently views and helpers) should generate prefix according to mount point. In that particular situation, default user will be inserted in place of :user, so the url will be "/drogus/blog/posts". You could also do blog_engine.posts_path(:user => "john"), which would generate "/john/blog/posts".

Generating engine’s url inside engine’s controller

# ENGINE/app/controllers/posts_controller.rb
class PostsController < ActionController::Base
  include Blog::Engine.routes.url_helpers

  def index
    posts_path #=> ??
  end
end

This situation is a bit different. The first thing you will probably notice is that I used posts_path instead of blog_engine.posts_path. This is possible because I included url_helpers from engine’s routes. From various reasons it can’t be included automagically for engine’s controllers, but the plan is to make it work without any explicit includes. Why can’t we just use blog_engine.posts_path? In that example it would work, but note that engine can be mounted with different :as option. With mount looking like:

mount Blog::Engine => "/blog", :as => "blog"

you would have to use blog.posts_path instead of blog_engine.posts_path. Basically engine should not need to know how it’s mounted. That said, we don’t have any information about options used to mount it, all that we know is what’s the request that was used to reach the engine.

But what about the generated path? Someone could say that it should also generate prefix, but that would not work as expected. Imagine that someone requested one of your users blog with path "/dhh/blog/posts/1". When you click on link with url generated by posts_path, you should stay in the same scope, so url should depend on you current path. This is achieved by using env["SCRIPT_NAME"] value. In request to "/dhh/blog/posts/1", the script name would be set to "/dhh/blog", as this is the part of path that does not belong to engine. It should be clear now, that example above will generate "/dhh/blog/posts" path for such request.

The next thing that is worth mentioning is _routes method. When you call posts_path directly, it must use routes object. This object is available through _routes method. This method is defined when you include url_helpers, so it points to application’s routes by default. When Blog::Engine.routes.url_helpers are included in PostsController, _routes is changed to use engine’s routes, and because of that we can use posts_path safely.

Generating application’s url inside engine’s controller

# ENGINE/app/controllers/posts_controller.rb
class PostsController < ActionController::Base
  def index
    app.root_path
  end
end

We would like to generate root_path from our application’s router. Obviously it should generate "/" path, without Engine’s prefix. The only exception is situation when app is hosted in a sub path (eg /myapp). This can be done with Phusion Passenger, using RailsBaseURI option. /myapp part would be passed as SCRIPT_NAME in such case. Nothing complicated, we already know how to use SCRIPT_NAME, right? Not really (wouldn’t it be to simple? ;-). The problem is, SCRIPT_NAME is kept as a string. Let’s see how the request to "/myapp/user/blog/posts" looks like.

At first it hits the application. SCRIPT_NAME is set to /myapp by Passenger and the path is /user/blog/posts. Now, application’s router recognizes that this request should get to engine, so Blog::Engine is called. As engine needs only /posts as path, the prefix (/user/blog) will be attached to env["SCRIPT_NAME"] resulting in /myapp/user/blog. As this is one string and we need to get just application’s script name, solution is not obvious. How do we get the original script name? Right now our approach is to use whatever is set in Rails.application.routes.default_url_options[:script_name], so it should be set to “/myapp” in that case.

Generating engine’s url in any other class (including ActionMailer)

In that case, url should be generated with prefix (which would be /user/blog in my example). In ActionMailer we are not inside request, so script_name is not available and with that in mind we need to generate the full path with "/user/blog/" at the beginning.

Solution

Although it may look complicated to handle all those use cases, solution is relatively easy. The first thing to do is to assume that prefix should be always generated. This gets all cases with prefix out of our way. What’s not so obvious, it also makes generating application’s url in engine work. As application’s routes are not mounted, prefix is nil, so we can safely add it to “generate prefix” case. What’s left? Generating engine’s url inside engine. We need to use env["SCRIPT_NAME"] here. How to check if we should attach script name? We need to check if routes used to generate url are the same as routes connected with current request.

Right now, to make it possible, router object is passed via env["action_dispatch.routes"]. When application is called, it sets it to Rails.application.routes and then when engine is called, it sets its own router there. That way, we always now in which controller are we. If _routes method points to the same routes as the env["action_dispatch.routes"] it means that we try to generate engine’s url inside engine and we should use the SCRIPT_NAME.

Named routes

The initial idea was to allow using named routes from 2 routers in one scope, just like that:

# APP/app/controllers/foo_controller.rb
class FooController < ActionController::Base
  # Rails.application.routes_url_helpers are included by default
  include Blog::Engine.routes.url_helpers

  def index
    posts_path #=> "/blog/posts" - path from engine's router
    root_path #=> "/" - path from application's router
  end
end

Although it would be handy if you will have to use paths from mounted engine a lot, it is the cause of many issues:

named route method must know the router that it was generated by, as now we could have multiple routers
there is a collision problem if 2 routers have the same helpers (situations when both routers will have for example posts_path is not so common, but problem will exist for root_path for sure)
currently, after including url_helpers twice _routes method will point to second routes (Blog::Engine.routes in last example), which will cause problems with using url helpers directly

There are some solutions for all of those problems, but as they’re all a bit hacky and complicated, we decided to leave that topic and not allow to use helpers from 2 routers in one scope. It means that you will have to explicitly say that you want to use router other then the default one (for example by using blog_engine.posts_path helper). It also means that including url_helpers from engine will overwrite current routes, so in case above, you can’t directly use application routes any more. In such case you should use app helper.

I hope that this post is good introduction to current status of router usage in mountable apps. As usual: if you have any ideas, feature requests, critique or any other thoughts that could help bringing mountable apps to life, you’re more than welcome ;-)

RSoC status: bringing engines closer to application

2010-07-06T00:00:00-07:00

I’ve been working for Ruby Summer of Code for last 2 weeks and so far it’s great! In this post I will try to sum up the work on engines and outline a couple of problems that are still not solved.

The first idea for RSoC was to bring Rails::Engine closer to Rails::Application. One of the long term targets is to allow to run more than one Application instance in one process. As I described in my last post, application is a bit more specialized engine, so while moving most of the functionalities from Application to Engine, I could identify and solve most of the problems with running several apps in one process.

First things first. What can Engine do right now and where is it used in Rails? When you drop anything in vendor/plugins directory, it will implicitly be declared as Engine. The features of engine are:

everything in app/* works as in application
it can load config/routes.rb
config/locales/* are automatically picked by I18n
config/initializers works as in application
engines also have initializers blocks inside the Rails::Engine class to customize rails booting
you can customize paths like in application (for example change where controllers are)
it can define custom generators
rake tasks are loaded from lib/tasks

All these features are great, but we can take it even further. Here is the plan for bringing Rails::Engine closer to Rails::Application. Rails::Engine should:

be a Rack app
have a middleware stack
have its own routes
allow to store assets in public/
be able to run its own migrations
be able to load plugins
allow to do more configuration
allow to namespace models and controller without problems

A few things from that list are already finished (not in rails master yet, on my fork for now). I will describe my changes, but beware, this is code that’s not currently a part of rails and it can be changed before merging it to rails. Its here for getting feedback mainly.

Engine can be now a rack application by providing rack endpoint:

class Blog::Engine < Rails::Engine
  endpoint AnyRackApp
end

That code would create engine with AnyRackApp as endpoint. Now you can mount it with:

MyRailsApp::Application.routes.draw do
  mount Blog::Engine => "/blog"
end

Mount method will tell application router that Blog::Engine is located at “/blog” path. Let’s investigate a request to “/blog/posts”. At first, it will hit the application and it will be passed through entire application’s middleware stack. The last middleware in application is the router. Router will recognize that “/blog” should point to Blog::Engine app, so it will pass the request to Blog::Engine. Then it will be passed through Engine’s middleware stack and finally it will hit Engine’s rack endpoint.

What’s the point of providing rack app inside engine? Probably this will not be a common scenario, but it allows you to add all functionalities provided by Engines, like initializers, generators, to your rack app. Besides, Engine has its own middleware stack now, so you can easily add some specific middlewares that will be fired only on requests for engine:

class Blog::Engine < Rails::Engine
  middleware.use Rack::Subdomain
end

By default endpoint is set to routes, probably a more common scenario.

Currently if you want to add your own routes, you have to hardcode it in application’s routes. It’s a bit limited, as you cannot easily change the place where the engine is mounted. With Engine being able to use its own router, you can mount it anywhere you want, even using dynamic scopes.

# APP/config/routes.rb
MyRailsApp::Application.routes.draw do
  mount Blog::Engine => "/blog"
end

# ENGINE/lib/blog/engine.rb
class Blog::Engine < Rails::Engine
end

# ENGINE/config/routes.rb
Blog::Engine.routes.draw do
  resources :posts
end

With such setup, Blog::Engine is mounted at “/blog”. A request to “/blog/posts” will fire posts controller inside the engine. It’s simple as that! However, as we are going to see in another blog post, if you want to mount an engine using a dynamic route, as “/:company/blog”, we need to consider different scenarios on URL Recognition and Generation. Currently if you use posts_path, it will generate /posts. The problem is, if you’re in application you should prepend prefix for mounted app, so it would be “/company/blog/posts”.

Next important thing for engines working as full rails apps is public folder. Currently you can for example symlink engine’s public folder in your application. It’s ok, but there is much cleaner way to do it: use ActionDispatch::Static, which will serve static files from given directory. At first when I heard that idea, I thought that it will much slower than serving those files with web server, but according to Yehuda Katz, it should be really fast in practice and many server setups could actually cache it (for example with Varnish). If it’s still not fast enough you can just not enable serving static files, make a symlink and serve it with webserver.

In addition to all that features, engines can load its own set of plugins now. Right now, it’s really simple implementation without any check for duplicating plugins, but I think we will consider such thing. Personally I hope that more and more people will just start using gem plugins with its own dependencies, but there are still many apps with vendor/plugins full of plugins.

One of the things that has not been implemented yet is migration support. There is quite long discussion on it on lighthouse and a blog post with a few solutions described, but there is sill no consensus on that one. If you have any thoughts on that topic, you can add a comment on lighthouseapp.

The last problem that I will talk about (but certainly not last problem that will appear) is namespacing. For better encapsulation all of the controllers, helpers and models should be namespaced. The problem is, currently if you would namespace a controller or model it will not work as intended. Given:

class Blog::PostsController < Blog::ApplicationController
end

it will affect application in many places. Using that controller in routes will also require namespacing it with: “blog/posts#index”, which is probably not something that we want for all the routes. In case of models, most of ORMs will name table for class Blog::Post as “blog_posts”. It could be actually ok, as it will help to avoid name conflicts, but it’s not always desired behavior.

I will appreciate any ideas and thoughts on that topic. Anything, like API ideas, feature requests or solutions to some problems will be welcome and will help me with delivering better mountable apps. Stay tuned for the next post on RSoC status: routes. I promise you will not have to wait for it for the next 2 weeks ;-)

Rails internals: Railties

2010-06-18T00:00:00-07:00

I’m going to start my RSoC work next week, but I as a preparation for real work I started reading through Rails codebase. I will try to document my findings on this blog, I hope it will be helpful for people that would like to know Rails internals.

One of the first findings of reading Rails 3 code is that it’s really simple compared to previous Rails versions. Boot process doesn’t need rubygems magic anymore. Ruby features are used to make Rails flexible without need of some crazy hacks. alias_method_chain is not used anymore, which makes it much easier to understand code. These are really great news for all Rails devs, especially plugins maintainers. But hey, enough already known facts, let’s get to the point!

The core class of Rails is Railtie. If you are a plugin developer, you will probably want to take a closer look at that. Although Railtie is not required in every plugin, it’s handy if you want to hook into Rails boot process, create intializers or add a rake tasks. For more information you can read about Railtie in docs So what’s Railtie? It’s a simple class that is capable of loading generators, tasks, middlewares and adding subscribers. One of the things that I’ve noticed in Railtie (and almost any other class in Rails) is usage of autoload method. I’ve never heard of autoload before and it seems that’s very handy method. From rubydocs:

Registers filename to be loaded (using Kernel::require) the first time that module (which may be a String or a symbol) is accessed.

  autoload(module, filename) => nil
  autoload(:MyModule, "/usr/local/lib/modules/my_module.rb")

The great thing in autoload is its laziness. It will not load given file, if module is not accessed. How does Rails use autoload in Railtie? It autoloads Configurable and Configuration modules. As Railtie doesn’t need configuration on its own, it includes it in classes that inherit from Railtie (code). That way, those 2 files will not be loaded unless Railtie subclass is created. Someone could argue that Railtie will not be probably used alone, without any subclasses, as Railtie is much more like abstract class, but it’s really good pattern and it’s consistently used across other classes in Rails.

I need to rectify one thing. Although autoload is very nice, it’s also known to be thread unsafe and according to Carl Lerche it will be removed from Rails. Use with consideration ;-)

Getting back to Rails itself. The next class in hierarchy is Engine. Engine is much closer to full stack Rails app. In fact application is a bit more specialized Engine. That makes sharing an application as easy as converting it to engine. So, what’s Engine? Basically it’s Railtie with ability to set load paths for views, controllers, helpers, locales etc. That’s it? Nothing more? That simple? Yep, that’s right.

Now it’s time for Application. Application is a subclass of Engine and it’s capable of booting the Rails app. As you can read in the docs, Application is singletone and that’s why 2 Rails apps can’t be run in a single process. So what does exactly Rails application do?

it loads default middleware stack (it’s amazing how much things can be handled by rack middlewares, allowing to keep Rails code simpler)
it loads plugins
it sets bunch of other things: database, logging, sessions, environment config, cache
it loads activesupport (interesting thing, you can set config.active_support.bare to not load ‘active_support/all’)

It’s also interesting how does Rails use OO model to make things simpler. After generating Rails 3 apps you can notice that frameworks are loaded by just requiring their railties:

require "action_mailer/railtie"
require "active_resource/railtie"

The same goes for engines. Ok… but how will Rails know all the railties and engines loaded? As railties and engines are subclasses of Rails::Railtie and Rails::Engine, it’s as easy as getting their subclasses: ::Rails::Railtie.subclasses or ::Rails::Engine.subclasses. Quick note on subclasses: it’s not ruby method, subclasses are gathered using inherited method (code).

Now let’s look how is that all used to boot rails application. As you can see in Rails 3 application, config.ru file looks like:

require ::File.expand_path('../config/environment',  __FILE__)
run MyApplication::Application

As you probably know, object passed to “run” method must respond to :call. Application class does not have “call” method defined, though. How does it work? When you send “call” message to MyApplication::Application, method_missing will be fired and message will be send to application instance. In fact all of the missing methods on MyApplication::Application are delegated to instance:

def method_missing(*args, &block)
  instance.send(*args, &block)
end

What is instance? As Application is singleton, call to instance method returns instantinated application or instantinates it and then returns the instance. Application instance is built on initialization, such behavior is triggered in Finisher

The next question is: how does request’s path look like? call() method runs call() on @app object, which is basically the middleware stack. Then, request goes through all the middlewares up to ActionDispatch::Routing::RouteSet, which is responsible for routing request to one of the controllers or other rack apps.

After reading this blog post you should have a pretty good overview of core classes of Rails. I strongly encourage you to dive into Rails code yourself, it’s a good way to learn new things about design, object oriented programming in ruby and to pick some cool patterns.

If anyone is interested, I plan to continue Rails internals posts with action pack and routing system. Feedback highly appreciated! Please let me know if you find this kind of information useful or if you think that I missed something or should give more details on any of the parts.

New address

2010-05-15T00:00:00-07:00

I haven’t written anything here for a while now. It was partially caused by a little mess in my blogs. I had 2 blogs with addresses blog.drogomir.com (for Polish blog) and drogomir.com/blog (for English blog)… I didn’t quite like addresses (hey? which one is in which language?) and keeping them on separate engines (typo and mephisto). Each time I wanted to start new post, I felt that I should change that right away. The problem is, I never had that much will to actually do something and after few minutes of checking solutions I usually stopped without any effects.

Finally I’ve merged my blogs and moved it to current address: piotrsarnacki.com.

The blog is hosted on github with help of jekyll. Jekyll is great so far. Its simplicity is impressive and I don’t have to worry about any server related issues. Layout is borrowed from Tom Preston-Werner’s site (layout is MIT licensed). It helped me start within a few minutes, but I will try to change it in a few days.

Although old RSS feed still works, I recommend to use new one: feeds.feedburner.com/piotrsarnacki. All the feeds available for this blog are:

feeds.feedburner.com/piotrsarnacki – English content
feeds.feedburner.com/piotrsarnacki-pl – Polish content (available on piotrsarnacki.com/pl)
feeds.feedburner.com/piotrsarnacki-all – All posts

I deleted some of the posts and marked some as deprecated. I try to not abandon that blog this time and post some new stuff.

On the side note. I have managed to be one of 20 students working on Ruby Summer of Code projects. I will work in the team with Bogdan Gaza on Rails 3 mountable apps and Rails admin panel (I will focus more on mountable apps part and Bogdan will work on rails admin panel). Carl Lerche and Erik Michaels-Ober will be our primary mentors.

I will try to post progress updates on this blog. Stay tuned!

Cucumber and Celerity - testing unobtrusive javascript

2009-06-16T00:00:00-07:00

In the world of javascript and ajax web apps I often find that writing applications supporting both javascript enabled and disabled is hard. For the sake of simplicity and due to deadlines I often write only javascript version of some of the features. But there are many situations where both scenarios should be supported.

The most important parts of app, especially the ones that must be crawled by google, should be written with unobtrusive style. And here comes the problem… I rarely see javascript testing and till today I haven’t done it myself. Celerity and cucumber (with help of culerity) can solve this problem. Here is short guide introducing this technique.

I will use Cucumber, celerity (which is HtmlUnit wrapper with API compatible with watir) and culerity. Culerity is a proxy between celerity and your app. Celerity requires JRuby and probably your app need MRI or REE – culerity resolves this problem.

Let’s get started.

You need to install JRuby in order to run celerity. After installing and adding jruby to your PATH install celerity gem (probably as a root):

jruby -S gem install celerity

Now you can create rails app and configure the environment:

rails culerity-example
sudo gem install cucumber rspec rspec-rails haml
# add config.gem "haml" to environment.rb
gem install langalex-culerity --source http://gems.github.com
cd culerity-example
./script/generate cucumber
./script/generate rspec
# now edit database.yml and set database options
rake db:migrate
rake features
rm features/step_definitions/webrat_steps.rb # cause we will be using celerity

At this point you should have cucumber configured and you should be able to run “rake features” with output similar to:

0 scenarios
0 steps
0m0.000s

Let’s add some tests! You will need step definitions and hooks. Culerity provides some basic step definitions and hooks which you can generate with “./script/generate culerity” but I’ve changed them a bit for my needs, so you can find them on this example repository.

Copy those files to your app.

The first file is just rewrite of webrat steps and the second file adds hooks for firing celerity server and browser. Let me explain the hooks.rb file:

require 'culerity'
`mongrel_rails start -e cucumber -p 3001 -d`

Before do
  $server ||= Culerity::run_server
  $browser = Culerity::RemoteBrowserProxy.new $server, {:browser => :firefox}
  $browser.webclient.setJavaScriptEnabled(false) 
  @host = 'http://localhost:3001'
end

Before("@js") do |scenario|
  $browser.webclient.setJavaScriptEnabled(true)
end

at_exit do
  $browser.exit
  $server.close
  `mongrel_rails stop`
end

‘Before’ hooks are run before each scenerio. In first ‘before’ hook server and browser are set up and host is set to “http://localhost:3001” (change it if you want to run app on other address or port). Notice the line: $browser.webclient.setJavaScriptEnabled(false) – it disables javascript by default.

Second Before hook is fired only for scenarios tagged with @js tag. It will be useful for explicitly saying which scenarios should be tested with javascript.

I’ve also added lines to start mongrel before the tests and stop it at exit. It’s handy if you don’t want to run and restart mongrel manually

Now it is time to write some scenarios. File features/javascript.feature

Feature: Javascript
  In order to test javascript
  As a developer
  I need a way to run test scenarios with javascript enabled or disabled

  @js
  Scenario: With javascript
    Given I am on the homepage
    And I follow "Click me!"
    Then I should see "Javascript rocks!"

  Scenario: Without javascript
    Given I am on the homepage
    And I follow "Click me!"
    Then I should see "I am also working without javascript!"

Both scenerios rely on “Click me!” link but have different expectations. ~~To run those tests start mongrel (or any other web server):~~

mongrel_rails start -e cucumber -p 3001 -d

~~This will fire mongrel in background on port 3001.~~
It is not necessary as mongrel control commands are in hooks.rb file

It’s best to operate on 2 console tabs while using celerity. One of the big drawbacks of using it, is lack of displaying rails exceptions. Because of that I run cucumber on one tab and “tail -f log/cucumber.log” on the other – I can see errors in the log without opening the browser.

Let’s run tests:

rake features

Of course both tests should fail with

Unable to locate Link, using :text and /Click me!/

and rails error in log file:

ActionController::RoutingError (No route matches “/” with {:method=>:get}):

Now we can fix it. We need some controller to show the link:

script/generate rspec_controller home

Add

map.root :action => “show”, :controller => “home”

to routes file.
Next copy app/views/layouts/main.html.haml (it just yields action and includes jquery.js and application.js) and jquery.js

You need to also set layout for home controller:

layout ‘main’

And here comes html and javascript.
app/views/home/show.html.haml

= link_to "Click me!", "?clicked=1", :id => "click_me"

#text
  - if params[:clicked]
    I am also working without javascript!

It displays a link and a text if params[:clicked] is present. So after clicking on that link page will be reloaded with parameter clicked=1 and the text will be displayed.

Let’s check if it’s passing:

mongrel_rails restart; rake features

We need to restart mongrel before “rake features” in order to load changes because of “cache_classes = true”. It’s one of the drawbacks of using this method, but I’m sure that someone will find out better way to do that.

If you did everything properly the second scenario should pass now. We’re green! :D

Add javascript code to your application in order to make the second scenario pass:

jQuery(function() {
  $("#click_me").click(function() {
    $("#text").html("Javascript rocks!");
    return false;
  });
})

Now both scenarios should pass.

To sum up, now you should be able to:

configure rails app with cucumber and celerity
specify how your tests should be run by placing @js tag on top of javascript scenarios
test unobtrusive javascript with ease

TODOs:

~~figure out better way to reload rails app while testing~~
provide better error explanations in cucumber with celurity to test without tailing logs

Cheers!

Additional resources

Celerity ajax examples by Alvin Schur

Fixing safe_erb with memcached improved

2009-05-12T00:00:00-07:00

Ruby has many structures that are weird and dangerous to some programmers that works with Java or other more “traditional” OO languages. For me it’s great and metaprogramming capabilities makes ruby killer in case of DSLs, nice looking APIs and avoiding repeating yourself.

But many problems that are solved with alias_method_chain or method_missing can be solved easier without the code smell. I sometimes find myself designing some weird complex solutions and then, after looking at it saying: “Hey! Why not just use OO features!”. I encourage everybody to look critically at every piece of your code that seems to be overkill and can be replaced with standard OO design.

One of examples is fixing memcached and safe_erb duet with alias_method_chain I don’t know if this solution worked in older versions of rails, but in 2.3 I got stack level too deep error. After some thinking I’ve just created subclass of MemCacheStore:

And then you can simply set it as cache store: config.cache_store = :my_mem_cache_store

And that’s it, no alias method chain :)

My git workflow

2009-04-26T00:00:00-07:00

I know that this post may be obvious for experienced git users, but it may be useful for some of you.

Github has added default branch picking recently. It’s great, as you don’t have to use master branch as your main branch. I usually have three “main” branches in my repo – production, staging and development.

The main point of it is to have separate branch for each environment (sometimes I skip staging, but it’s often wise to have some point between development and production). My main problem with git (or maybe rather with my lack of knowledge about git) in the very beginning of using git was to learn how to use branches efficiently. After some time with git I’ve noticed that it’s best to keep each fix or idea in separate branch.

One crucial thing is to inherit from production when it’s possible. Let’s say you have those main branches I mentioned before. There is a bug #345 in bug tracker and you want to fix it and merge into production after showing results to the client. The best way is to begin with production:


git checkout -b bug345 production

It will create new branch bug345 with production as a parent branch. Now:


# fix the bug
git commit -m "Fixed the nasty bug #345"

Sometimes client doesn’t need reviewing the changes but when he does you should have ability to show it and merge it to the production branch later.


git checkout staging
git merge bug345
# deploy staging

After the client approval of your solution you can merge the bugfix to the production branch:


git checkout production
git merge bug345
# deploy production
# you can remove bug345 branch now
git branch -d bug345

The most important thing in this process is to always inherit from production branch, unless you need changes from other branches. Otherwise you could merge some needless junk from other branches or have to cherry-pick individual commits. With above solution everything is clean and you are sure that that you will merge only what you need.

Hope it will help :)

JSON-P support for Apache upload progress

2009-02-10T00:00:00-08:00

Just a quick note about new feature added to apache upload progress and jquery upload progress libs.

Ron Evans aka deadprogrammer has added support for JSON-P in this commit. What does it mean? Cross domain requests are now possible, so if you need such a functionality pull the newest changes.

Here is Ron’s description from README:

- JSON-P Support

You can also request progress updates by using JSON-P, if you are uploading the file from a different domain or subdomain than the web server that is handling your original request. Adding a “callback=yourCallbackFunction” parameter to your request to the progress server will activate this functionality.

For example, a request like:
http://uploads.yourdomain.com/progress?callback=jsonp123&X-Progress-ID=1234

Would return the JSON-P function:
jsonp123(new Object({ ‘state’ : ‘uploading’, ‘received’ : 35587, ‘size’ : 716595, ‘speed’ : 35587 }));

The normal JSON request:
http://www.yourdomain.com/progress?X-Progress-ID=1234

Would return the JSON data:
new Object({ ‘state’ : ‘uploading’, ‘received’ : 35587, ‘size’ : 716595, ‘speed’ : 35587 })

Remember to update jquery upload progress also, to use jsonp.

Enjoy :)

Tweaking Rails app with jQuery, part I

2008-07-03T00:00:00-07:00

Quick links:

completed demo (demo is reseted every few hours)
sources of application on github (or download tarball)

I’m in the train from Zgorzelec to Warsaw returning from my girlfriend’s place. Polish trains are like turtles, so I will have pretty much time for writing ;-)

I’ve wrote (or maybe it’s better to say copy&paste) little rails app like in Mike Clark’s tutorial for attachment_fu. A few months ago there was Mugshots exhibition in Yours Gallery in Warsaw based on work of Peter Doyle. I saw it with Kathleene, she took some pictures. Great! I have material to fill my new app, what else could I possibly dream of?! (yeah… macbook, but it’s obvious ;-).

Now you can admire my hard work: mugshots.drogomir.com/js/no-javascript/mugshots/

But wait… It’s not so cool… where are all those shiny javascript effects? Don’t worry. I will show you how to spice this dish.

We will need:

I’ve pushed application to github, so you can see entire code. Clone it or grab the tarball

There is one thing that is not straight forward. @main_js variable in app/views/layouts/main.rhtml:


<%= javascript_include_tag @main_js %>

It’s there for changing javascript file loaded. When url is app.com/js/some_javascript_file/mugshots, @main_js should be “some_javascript_file.js”. I’ve done this to have possibility to show you app with different javascript files without changing the code. See routes and mugshots_controller.rb to find out how it was done (or run “rake routes” in app dir to see routes).

Lets begin.

What to do first? It’s all about uploading files, so I would add upload progress bar to form in mugshots.drogomir.com/mugshots/new. To implement it you will need some kind of server module:

You have to install and enable one of the above modules to make progress bar work.

Then add some javascript to applications.js. This example is using “LightBoxFu”: – little script that I wrote to show progress bar as an overlay. It’s based on Riddle’s work – all positioning is in CSS (except expressions for IE) so it’s really light and fast. Ideal for such a task. If you don’t like lightBoxFu you can use any other form of displaying message (you can use some other lightbox with displaying code function or even blockUI plugin).


// handy trick, if we can't use $ beaceuse jQuery.noConflict
// was used, jQuery is passed as argument in document ready
// so we can name it $
jQuery(function($) {
  // add upload progress to our form
  $('form.progress').uploadProgress({
    start:function(){
      // after starting upload open lightBoxFu with our bar as html
      $.lightBoxFu.open({
        html: '<div id="uploading"><span id="received"></span><span id="size"></span><br/><div id="progress" class="bar"><div id="progressbar">&nbsp;</div></div><span id="percent"></span></div>',
        width: "250px",
        closeOnClick: false
      });
      jQuery('#received').html("Upload starting.");
      jQuery('#percent').html("0%");
    },
    uploading: function(upload) {
      // update upload info on each /progress response
      jQuery('#received').html("Uploading: "+parseInt(upload.received/1024)+"/");
      jQuery('#size').html(parseInt(upload.size/1024)+" kB");
      jQuery('#percent').html(upload.percents+"%");
    },
    interval: 2000,
    /* if we are using images it's good to preload them, safari has problems with
       downloading anything after hitting submit button. these are images for lightBoxFu
       and progress bar */
    preloadImages: ["/images/overlay.png", "/images/ajax-loader.gif"]
  });
});

And some styling for progress bar:


  #progress {
  margin: 8px;
  width: 220px;
  height: 19px;
}
#progressbar {

  background: url(‘/images/ajax-loader.gif’) no-repeat;

  width: 0px;

  height: 19px;

}

That’s it, just add “progress” class to your form and progress bar is working:


<% form_for(:mugshot, :url => mugshots_path, 
                      :html => { :multipart => true, :class => "progress" }) do |f| -%>

Uploading files looks much better right now, check it here: http://mugshots.drogomir.com/js/progress/mugshots/new

So what now? I find the “add photo, click New mugshot, add photo” scenerio annoying. We could add more than one file on each submit. For that we will use jquery.MultiFile.js. This one is a bit tricky, cause we will need to tweak code handling uploads also.

Javascript enabling mutlifile:


jQuery(function($) {
  $('.multi-file').each(function() {
    // change name of element before applying MultiFile
    // so array of files can be send to server with mugshot[uploaded_data][]
    $(this).attr('name', $(this).attr('name') + '[]');
  }).MultiFile();
});

We must also add “multi-file” class to file field:


<%= f.file_field :uploaded_data, :class => 'multi-file' %>

From javascript point of view that’s all. Let’s see how uploaded photos are handled by rails app:


@mugshot = Mugshot.new(params[:mugshot])

So mugshot.uploaded_data is filled with data from params[:mugshot][:uploaded_data]. Good for one file. But with array of files we should create Mugshot for each file. I would add a method in model:


  def self.handle_upload(mugshot_params)
    # array for not saved mugshots
    mugshots = []
    if mugshot_params[:uploaded_data].kind_of?(Array)
      mugshot_params[:uploaded_data].each do |p| 
        unless p.blank?
          mugshot = Mugshot.new(:uploaded_data => p)
          mugshots << mugshot unless mugshot.save
        end
      end
    else
      mugshot = Mugshot.new(mugshot_params)
      mugshots << mugshot unless mugshot.save
    end
    mugshots
  end

and slightly change controller code:


  def create
    @mugshots = Mugshot.handle_upload(params[:mugshot])
      
    # if @mugshots is empty there are no errors
    if @mugshots.blank?
      flash[:notice] = 'Mugshot was successfully created.'
      redirect_to mugshots_url
    else
      render :action => :new
    end
  end

Only one problem left. Validation.

Easiest way is to change error_messages_for:


<%= error_messages_for :object => @mugshots %>

It works. But suppose you are uploading 3 files and 2 of them are too big. You will end with:

Size is not included in the list
Size is not included in the list

Which one was added? Some lottery here…

I would tweak attachment_fu error messages a bit. By default it uses validates_as_attachment method which simply adds:


  validates_presence_of :size, :content_type, :filename
  validate  :attachment_attributes_valid?

Instead validates_as_attachment we can isert our new code:


  validates_presence_of :size, :content_type, :filename, :message => Proc.new { |mugshot| "can't be blank (#{mugshot.filename})" }
  validate  :attachment_attributes_valid?
  
  def attachment_attributes_valid?
    [:size, :content_type].each do |attr_name|
      enum = attachment_options[attr_name]
      errors.add attr_name, "#{ActiveRecord::Errors.default_error_messages[:inclusion]} (#{self.filename})" unless enum.nil? || enum.include?(send(attr_name))
    end
  end

Now it’s a lot more readable:

Size is not included in the list (filename.jpg)
Size is not included in the list (filename1.jpg)

Submit form looks better now, but viewing files is still ugly. Maybe we could add some lightbox? No problem:


$('#mugshots li a').lightBox();

I used that lightbox cause I had it configured for my previous rails apps, but pick your favourite one, as there are gazilions of them.

This is first step of tweaking our app. Javascript is in step1.js file: mugshots.drogomir.com/js/step1/mugshots/new

What now? User can upload many files at one submit and see progress bar. What else do we need? Ajax! My preciousssss…

As all children know, XMLHttpRequest can’t upload files. What a shame… our new tweaked mugshots app is all about uploading files. Although you can’t do it with XHR, there is a way to imitate it. It is obtained by creating an iframe and uploading files to it.

Luckily Mike Malsup has done hard work for us writing jQuery form plugin.

First, we need our form. I would place it instead “New mugshot” link. Link has id=“new_mugshot_link”, so this piece of code will replace it with form:


  /* create upload form with multifile instead of new mugshot link */
  var form = $('<form method="post" enctype="multipart/form-data" class="progress ajax" action="/mugshots">');
  var label = $('<p><label for="mugshot_uploaded_data">Upload mugshot: </label></p>');
  var input = $('<input type="file" class="multi-file" id="mugshot_uploaded_data" size="30" name="mugshot[uploaded_data]"/>');
  label.append(input).appendTo(form);
  form.append('<p><input type="submit" value="Create" name="commit"/></p>');
  if (typeof(AUTH_TOKEN) != "undefined") form.append('<input type="hidden" value="'+AUTH_TOKEN+'" name="authenticity_token"/>'); 
  $('#new_mugshot_link').replaceWith(form);

Our form has to be send to an iframe, so we have to apply ajaxForm to it. After replacing link with form we can’t figure out when form is actually appended to DOM. To be sure that form is there, we can use livequery. It will fire callback function when ‘form.ajax’ will be available:


  $('form.ajax').livequery(function() {
    $(this).ajaxForm({iframe: true, success: function (responseText, statusText, form) {
      var url = $(form).attr('action');
      /* get new files */
      $.ajax({
          url: url,
          dataType: "script",
          beforeSend: function(xhr) {xhr.setRequestHeader("Accept", "text/javascript");},
	  /* we need to update lightbox array to include new files */
          complete: function() { $('#mugshots li a').lightBox(); }
      });
    }});
  });

When new form tag with class “ajax” will be available callback function will be run. iframe option tells form plugin to add hidden iframe (it will handle file upload).

The above code has ajax call to “/mugshots” url which will run index.js.erb (RJS), so we will need one:

app/views/mugshots/index.js.erb


  jQuery('#mugshots').html(<%= js render(:partial => 'mugshot', :collection => @mugshots) %>);

to handle it we need to use respond_to:


  respond_to do |format|
    format.html
    # layout => false is here beaceuse without it rails are looking
    # for layouts/index.js.erb
    format.js { render :layout => false }
  end

Normally I try not to use RJS to keep all my javascript (and ajax) logic in javascript files, but in case of images it isn’t so esay. I will write about it and about javascript templating systems in one of the next posts.

Take a look at: mugshots.drogomir.com/js/step2/mugshots Doesn’t it look nice?

There is only one problem :) No ajax validation. After submitting files, javascript can’t get any info about errors or uploaded files beaceuse it is treated like normal html request and response is loaded in an iframe. How to fix it? I’ll write about it in the next post. :)

Upload progress script with safari support

2008-06-30T00:00:00-07:00

Quick links:

Source on github for jquery version: http://github.com/drogus/jquery-upload-progress
Source on github for prototype version: http://github.com/drogus/prototype-upload-progress
jQuery demo: http://drogomir.com/files/blog/jquery-upload-progress/example
Prototype demo: http://drogomir.com/files/blog/prototype-upload-progress/example
Installing apache upload progress

Recently I’ve wrote about apache upload progress module. I work mainly on linux and I haven’t check my scripts on safari. It was working even on IE, what possibly could be harder to obtain? ;-) Some people reported that demo is not working on safari and Michele resolved the problem (thanks Michele :).

Solved! The only thing to do was to open WinXP on VirtualBox and check it on Safari 3. Michele’s solution worked well, but to make it work there must be html page with given structure and javascript. I like “one file” easy to use scripts without any issues with static files :) So what? Create an iframe dynamically, load scripts dynamically, one file, the only thing that user will have to set is path to scripts.

With Safari? No, not really. I’ve wrote it in a few minuttes and checked in firefox. It worked, great, now safari. Nope….

Although Safari have great CSS support, it is really terrible with Javascript. WYSIWYG, javascript history, ajax issues, now the upload progress and iframes. In edge case libraries I often see hacks for IE and safari mainly.

Luckily Apple released Windows version of safari and I can check my scripts and pages on safari. Good… developer tools don’t work, though… I had to do some alert-like IE style debugging ;-) After many hours of trying new more and more insane ways to create iframe and load scripts into it, code with document.write() worked!

Check the commit on github. Lines 18-22 especially. And 28-line issue with safari not waiting to load previous script.

It’s really sad that after working on firefox with firebug or opera with developer tools I have to fight with Safari which is supposed to be modern browser. I know that on Mac developing scripts for safari is easier, thanks to working debug tools, but hey! I work on Linux. Why do I have to run Safari on Wine or VirtualBox, lacking debug tools? It’s a pain. I feel like I’m working on explorer…

~~I will add prototype version~~ and possibly some usage page shortly (for know look at the demo code).

Upload progress bar with mod_passenger and apache

2008-06-18T00:00:00-07:00

UPDATE: I found 2 bugs in upload progress module. If you have already installed. update to at least 0.1 version: http://github.com/drogus/apache-upload-progress-module/commits/0.1

I’ve installed mod passenger on my server recently. It’s really great software. Now I don’t have to worry about monitoring, nginx proxy, load balancing, big file uploads… and it’s fast! With Ruby Enterprise Edition it’s even faster.

Personally I don’t care about people saying that phusion wants to promote themselves on REE as long as it gives faster ruby with lower memory use (but yes, I know, REE is not best choice for a name :).

After installing I’ve realised that my shiny upload progress bar (thanks to Upload Progress Module for nginx) was not working. Oh gods! What a tragedy!

But fear not. I’ve written apache upload progress module to have my lovely progress bar back again. As a lazy developer I’ve implemented reports in the same way as in nginx upload progress, so my applications are working without changing any signle line of code. If you were using nginx upload progress just drop the module, change your config and you’re good to go :)

I’m testing it in one of my production servers, but be carefull – it’s not well tested in other enviroments (i have gentoo with apache 2.2.8). Any feedback will be helpfull. Give me a note in comments if you encounter any problems.

So you want to be cool and have your own sexy progress bar in your app? Keep reading ;)

To install module you must download it using git:

git clone git://github.com/drogus/apache-upload-progress-module.git

or get the package: http://github.com/drogus/apache-upload-progress-module/tarball/master

To compile/install/activate you have to use apxs2:

apxs2 -c -i -a mod_upload_progress.c

-c is for compiling
-i is for installing (copy mod_upload_progress.so to apache library dir)
-a is for activating (add LoadModule option into your apache conf file)

If you want to install and activate run this command as a root. Otherwise you can just compile and add LoadModule to apache conf:

LoadModule upload_progress_module path/to/apache-upload-progress-module/.libs/mod_upload_progress.so

Currently there is only one global option:

UploadProgressSharedMemorySize 1024000

This sets shared memory size to 1M. By default it’s 100kB.

To add tracking and reporting upload for a virtual host in apache you will need to add:

<Location />
    # enable tracking uploads in /
    TrackUploads On
</Location>

<Location /progress>
    # enable upload progress reports in /progress
    ReportUploads On
</Location>

Now all uploads will be tracked and reports are under /progress

Format of the report is JSON. From nginx wiki:

The returned document is a JSON text with the possible 4 results:

the upload request hasn’t been registered yet or is unknown:

new Object({ ‘state’ : ‘starting’ })

the upload request has ended:

new Object({ ‘state’ : ‘done’ })

the upload request generated an HTTP error:

new Object({ ‘state’ : ‘error’, ‘status’ : })

One error code that is interesting to track for clients is HTTP error 413 (Request entity too large)

the upload request is in progress:

new Object({ ‘state’ : ‘uploading’, ‘received’ : , ‘size’ : })

The HTTP request to this location must have either an X-Progress-ID parameter or X-Progress-ID HTTP header containing the unique identifier as specified in your upload/POST request to the relevant tracked zone. If you are using the X-Progress-ID as a query-string parameter, ensure it is the LAST argument in the URL.

Now the last thing to do is to implement progress bar. I don’t like repeating others and there is great tutorial on setting up upload progress bar with nginx and merb

UPDATE: I released jquery upload progress library with Safari 3 support. More info here.
UPDATE2: I’ve upgraded prototype version to work in Safari.

It’s for merb and nginx but if you drop the scripts in your rails app and with apache-upload-progress-module it will work. :) Basically if you have your own code handling uploads (for example using attachment_fu) you can just add javascript and css – it’s unobtrusive.

If you’re using prototype I’ve rewritten script and made a demo. You can also grab files

I hope you enjoy this article. Progress bar is in my opinion one of the most useful technics – there is nothing more annoying than large file uploading without any info on state of an upload.