Owen Ou's Blog

PoEAA on Rails

2011-09-24T00:00:00-07:00

The book Patterns of Enterprise Application Architecture (PoEAA) laid the blueprints for Rails’ architecture. When choosing which enterprise design patterns to encode into the framework, Rails picked, to name a few, Active Record, Template View, Application Controller, etc. By covering these patterns with a sweet coating of convention-over-configuration, Rails simplifies pattern analysis a lot.

These design assumptions were absolutely pragmatic for the type of applications that Rails was targeting at. However, as applications growing more and more complex, developers are starting to realize these default architectural patterns may not scale very well. Typically, four main areas are overloaded in an enterprise application:

high coupling between domain model and data source,
bloated domain model with a mix of domain logic and application logic,
presentation behaviour leaked into views, and
high coupling between view data and template

To understand these problems better as well as to figure out possible solutions, I would like to walk you through some enterprise patterns from the same book that Rails’ architecture heavily bases upon.

Note that the patterns I am mentioning here will benefit more for complex applications, for example, an e-commerce web application selling digital goods. There’s clearly some overhead to use these patterns for simple projects where the default Rails patterns shine. Most importantly, you should only apply these patterns where they make sense. I am a strong believer in contextual solution and these patterns are definitely not the only way to go.

Data Mapper

Data Mapper is a layer that transfers data between objects and a database. It typically creates Domain Model objects by populating their attributes from the database.

Assuming we are building an e-commerce platform, we get a store object by going through the store mapper which connects to the database:

# app/controllers/stores_controller.rb

class StoreController < ApplicationController
  def show
    @store = StoreMapper.find(params[:id])
  end
end

The difference between Active Record and Data Mapper is actually that in the Active Record implementation, Domain Model not only encapsulates business logic, but also takes responsibility of database access, while in the Data Mapper implementation, Domain Model is ignorant of database and become Plain Old Ruby Object with business logic. Data Mapper allows database logic and the object model to evolve independently.

Note that the Active Record pattern and the Data Mapper pattern mentioned are not the Ruby libraries these patterns inspire. The active_record gem is an implementation of the Active Record pattern. The data_mapper gem is also a “Active Record”-ish implementation, although it has elements of the Data Mapper pattern. For those who are interested in seeing the implementation difference between these two gems, I created a gist for you.

Here is Martin Fowler on the choice of Active Record or Data Mapper for Domain Model:

An OO domain model will often look similar to a database model, yet it will still have a lot of differences. A Domain Model mingles data and process, has multivalued attributes and a complex web of associations, and uses inheritance.

As a result I see two styles of Domain Model in the field. A simple Domain Model looks very much like the database design with mostly one domain object for each database table. A rich Domain Model can look different from the database design, with inheritance, strategies, and other [Gang of Four] patterns, and complex webs of interconnected objects. A rich Domain Model is better for more complex logic, but is harder to map to the database. A simple Domain Model can use Active Record, whereas a rich Domain Model requires Data Mapper.

In an enterprise Rails application, it’s not rare to see complex domain models stuffed with associations, validations, scopes and business logics. It has become a growing pain to deal with such “fat models”. We need to separate the database concerns out into a new dedicated layer: the data mappers. However, as far as I know, there’s no ORM in Ruby giving us such separation yet, although it’s been said the upcoming 2.0 release of the data_mapper gem will fully implement the Data Mapper pattern. Before we are able to consume the new data_mapper gem, is there a way to mitigate existing overloaded Rails models? Besides, switching an ORM for existing code is not effortless.

[Updated 01/10 2012], Piotr Solnica from the data_mapper gem made an official announcement saying they are actively working on Data Mapper 2.0. Announcement goes here.

As a compromised solution, I would extract database related logic for each ActiveRecord::Base model (e.g., validations and scopes) out into a module and then mix it in:

# app/mappers/store_mapper.rb

module StoreMapper
  extend ActiveSupport::Concern

  included do
    # validations
    validates_presence_of :name
    ...

    # scopes
    scope :disabled, where(:disabled => true)
    ...
  end
end

# app/models/store.rb

class Store < ActiveRecord::Base
  include StoreMapper

  # associations
  has_many :products
  belongs_to :company
  ...

  # business logics
  def calculate_sells(from_date, to_date)
    # calculate sells from date to date
  end
  ...
end

This half-baked solution, although not migrating to the Data Mapper pattern, cleanly isolates the definitions of database logic with the ones of business logic. Of course, it’s recommended to use the Data Mapper pattern where possible.

Service Layer

Layering is one of the most common techniques to break apart a complex software system. The higher layer uses service defined by the lower layer, but the lower layer is unaware of the higher layer. Each layer usually hides its lower layers from the layers above. A Rails project is typically divided into three layers:

presentation layer, including views and controllers,
domain layer, including models, and
data source layer, which is hidden behind models extending from ActiveRecord::Base.

It’s not difficult to understand the Data Mapper pattern is an effort to break down #3 into another layer to lower the complexity of models. However, in the context of enterprise application, models are still overwhelmed by complicated business logic. As Fowler pointed out, business logic can be further divided into “domain logic” and “application logic”, and Service Layer is a pattern to encapsulate model’s “application logic” by establishing a boundary where the presentation layers interact with the application:

… Service Layer is a pattern for organization business logic. Many Designers, including me, like to divide “business logic” into two kinds: “domain logic”, having to do purely with the problem domain (such as strategies for calculating revenue recognition on a contract), and “application logic”, having to do with application responsibilities [Cockburn UC] (such as notifying contract administrators, and integrated applications, of revenue recognition calculations). Application logic is sometimes referred to as “workflow logic”, although different people have different interpretations of “workflow”.

He also pointed out, Service Layer is a good fit for coordinating operations among multiple models, as well as for talking to multiple presentation layers:

The benefit of Service Layer is that it defines a common set of application operations available to many kinds of clients and it coordinates an application’s response in each operation. …

The easier question to answer is probably when not to use it. You probably don’t need a Service Layer if your application’s business logic will only have one kind of client - say, a user interface - and its use case responses don’t involve multiple transactional resources.

To translate this into a code example, let’s assume in an e-commerce platform, we need to email monthly sells report to the store owner. In this example, StoreService acts as a coordinator of multiple models for the “mailing sells report” workflow:

# app/services/store_service.rb

class StoreService
  def mail_sells_report(store_id, from_date, to_date)
    store = StoreMapper.find(store_id)
    store_sells = store.calculate_sells(from_date, to_date)
    store_report = SellsReport.generate(store_sells)
    SellsReportMailer.report_mailer(store.owner, store_report).deliver
  end
end

Another place where Service Layer shines is reusing workflows for different controllers, for example, the same workflow in the above example is used in the Storefront::StoresController for store front and in the Api::StoresController for REST API calls. Service Layer becomes a Facade in this case.

Presentation Model

It’s not uncommon to present multiple models in a view. Most importantly, not all attributes of a model are needed for a certain presentation. In a complex system where there are lots of screens, views become a very busy place for extracting state and behavior from models. The Presentation Model comes to ease the pain:

Presentation Model pulls the state and behavior of the view out into a model class that is part of the presentation. The Presentation Model coordinates with the domain layer and provides an interface to the view that minimizes decision making in the view. The view either stores all its state in the Presentation Model or synchronizes its state with Presentation Model frequently.

As an example, let’s build a view for the scenario “creating a store for a user”:

# app/presenters/create_store_presenter.rb

class CreateStorePresenter
  attr_reader :store, :user

  delegate :name, :to => :store, :prefix => true
  delegate :email, :to => :user, :prefix => true

  def initialize(params)
    @store = Store.new(params[:store])
    @user = @store.build_user(params[:user])
  end

  def valid?
    @user.valid? && @store.valid?
  end

  def save
    @store.save
  end
end

<!-- app/views/stores/create_store.html.erb -->

<h1>Create a store</h1>

<%= form_for(@create_store_presenter) do |f| %>
  <p>
    <%= f.label :store_name %><br/>
    <%= f.text_field :store_name %>
  </p>

  <p>
    <%= f.label :user_email %><br/>
    <%= f.text_field :user_email %>
  </p>

  <p>
    <%= f.submit "Create" %>
  </p>
<% end %>

In the example, we wrap two models (Store and User) into a presenter and extract only the attributes needed (Store#name and User#email) for the “create store” screen. To summarize this pattern, I would like to once again consult Fowler:

Presentation Model is a pattern that pulls presentation behavior from a view. … It’s useful for allowing you to test without the UI, support for some form of multiple view and a separation of concerns which may make it easier to develop the user interface.

… Presentation Model allows you to write logic that is completely independent of the views used for display. You also do not need to rely on the view to store state. …

Two Step Views

Rails comes with a neat templating system that allows you to quickly create dynamic pages. However, this Template View pattern has drawbacks as Fowler pointed out, especially in a situation where the view is very complex:

… the common implementations make it too easy to put complicated logic in the page, thus making it hard to maintain, particularly by nonprogrammers. You need good discipline to keep the page simple and display oriented, putting logic in the helper. …

What’s worse, if the display of a view is based on conditions, for example in a multi-appearance application, you will find logic that determines which template to render leaked into many places in views or controllers. Again, this is fine for a simple Rails application. But it becomes unmanageable as the application growing more complex.

Let’s think about an example: an e-commerce platform supports multiple stores and each store has its own storefront to display a product. It’s common to see the following solution:

<!-- app/views/products/_product.html.erb -->

<% if store.amazon_store? %>
  <%= render :partial => '/amazon/products/_product.html.erb', :object => product %>
<% elsif store.apple_store? %>
  <%= render :partial => '/apple/products/_product.html.erb', :object => product %>
<% else %>
  <%= render :partial => '/default/products/_product.html.erb', :object => product %>
<% end %>

The determination logic for displaying a product based on store is leaked into the product partial. Apparently, to build the whole multi-appearance application, this approach does not scale. Thankfully, Fowler has something good for us - the Two Step View pattern:

… You may also want to make global changes to the appearance of the site easily, but common approaches using Template View or Transform View make this difficult because presentation decisions are often duplicated across multiple pages or transform modules. A global change can force you to change several files.

Two Step View deals with this problem by splitting the transformation into two stages. The first transforms the model data into a logical presentation without any special formatting; the second converts that logical presentation with the actual formatting needed. …

From the above diagram, the multi-storefront example can be reimplemented with the Two Step View pattern. The process is in two steps. The first step is to transform the product data into a logical presentation. The second step is to convert this logical presentation into different HTML.

Note that in the implementation, a gem called cells is used to help define logical presentation. The cells gem is very helpful in this respect although it was originally designed for other purposes.

For the logical presentation, we define it to display name, description, price and reviews of a product for every store:

<!-- app/views/products/_product.html.erb -->

<%= render_cell :product, :name, product.name %>
<%= render_cell :product, :description, product.description %>
<%= render_cell :product, :price, product.price %>
<%= render_cell :product, :reviews, product.reviews %>

We then define three strategies (ProductCell, Amazon::ProductCell, and Apple::ProductCell) to convert the logical presentation to different HTML. We make use of cells’ strategy builder to return strategy class based on current store in session:

# app/cells/product_cell.rb

class ProductCell < Cell::Rails
  # return strategy class based on current store in session
  build { "#{current_store.name}::ProductCell".classify.constantize rescue nil }

  def name(name)
    content_tag(:h1, name)
  end

  ...

  def reviews(reviews)
    # display reviews
  end
end

# app/cells/amazon/product_cell.rb

module Amazon
  class ProductCell < ::ProductCell
    def name(name)
      content_tag(:p, name)
    end

    ...

    def reviews(reviews)
      # display reviews for Amazon store
    end
  end
end

# app/cells/apple/product_cell.rb

module Apple
  class ProductCell < ::ProductCell
    def name(name)
      content_tag(:div, name, :class => 'title')
    end

    ...

    def reviews(reviews)
      # display reviews for Apple store
    end
  end
end

As you may see, the Two Step View pattern makes multi-appearance implementation manageable in a way that different appearance implementations are organized in a set of strategy classes to parse a common logial presentation.

Summary

The default enterprise design patterns encoded into Rails are perfect matches for small/medium size projects. They are light weight and easy to understand. However, as the application growing more mature, these patterns do not scale well due to layers taking too much responsibility. That said, a “fat” layer need to be broken into smaller ones:

Data Mapper is an effort to extract out data source layer from Domain Model that implements Active Record,
Service Layer is an endeavor to extract out application logic from Domain Model,
Presentation Model is an attempt to extract out presentation logic from a single or multiple Domain Model, and
Two Step View is a try to break down presentation logic into two processing steps so that a view can be generated in different formats.

In return for breaking apart a system into smaller layers, each layer becomes easier to maintain, reuse, test and scale.

Last but not least, I would like to thank Martin Fowler for his awesome book and would love to hear any feedback for you.

Automatic Testing of REST Web Services Client with Rails

2011-07-20T00:00:00-07:00

Testing REST web services client has never been easy. It requires a running web server, multiple threads, network conection and complex transaction management.

Ideally, REST web service client test should have the following characteristics:

The experience of testing REST resource is similar to that of testing a ActiveRecord model
Start up and shut down web server for the purpose of running REST web services
Rollback test data after each test
Control fixture creation for REST web services
All tests are automatic

In this article, I demonstrate solutions to each of those mentioned.

As an example throughout the article, let’s assume we have web services for a model called Task and we are testing its corresponding client code. Here is a sample action in the TasksController of the web server:

# server/app/controllers/tasks_controller.rb

def index
  @tasks = Task.all
  render :status => :ok, :json => @tasks
end

We render @tasks as the JSON format where to_json is called on the object. When you run ”curl http://localhost:3000/tasks.json”, you will get the following result:

$ curl http://localhost:3000/tasks.json
[{"id":1,"name":"Write a blog post","created_at":"2011-07-20T04:05:41Z","updated_at":"2011-07-20T04:05:41Z","ends_at":"2011-08-20T03:15:00Z"}]

ActiveResource

In order to test our REST web services, we need a HTTP client. There are lots of them out there, but I found ActiveResource the most enjoyable to use in a less complex situation. ActiveResource provides ActiveRecord compatible APIs, so when writing web service client tests, we feel like we are writing unit tests for a ActiveRecord model.

To start with, we just need to extend it from ActiveResource::Base and give it the web server URL and representation format. That’s it!

# client/app/models/task.rb

class Task < ActiveResource::Base
  self.site = "http://localhost:3000"
  self.format = :json
end

And we are using it as if you are using an ActiveRecord object:

# client/spec/models/task_spec.rb

describe Task do
  it "should return all the tasks" do
    @tasks = Task.all
    @tasks.size.should == 1
  end
end

Web Server

To maintain a zero-setup test environment, we’ll have our test control the stratup and shutdown of a web server. By having the tests start and stop the web server, they can be easily run with no external dependencies.

To control the startup and shutdown of a web server before and after all suites run, it’s as simple as having something like this:

# client/spec_helper.rb

RSpec.configure do |config|
  config.before :suite do
    @server = Server.new(server_path)
    @server.start
  end

  config.after :suite do
    @server.stop
  end
end

The implementation of Server is also dead simple. Execute “script/rails server -d” to daemonize the server and issue a kill to stop it:

# client/lib/server.rb

Class Server
  def initialize(server_path)
    @server_path = server_path
  end

  def start
    `#{rails_script} server -d -e test`
  end

  def stop
    pid = File.read(pidfile)
    `kill -9 #{pid}`
  end

  private

  def rails_script
    File.join(@server_path, 'script', 'rails')
  end

  def pidfile
    File.join(@server_path, 'tmp', 'pids', 'server.pid')
  end
end

Transaction Rollback

For testing strategies of web services, most people recommend to either truncate test data on each run or to mock out the request and response. These approaches are less ideal since they’re either less effective or they’re not testing full stack of the targeted web services.

Would it be possible to wrap web services calls in a transaction and rollback data after each test, like what Rails’s transactional fixture does?

Of course! But let’s first try to understand why making transaction rollback for web service calls is difficult:

Tests and web server are running in two separate threads, web server’s transactional boundary can’t expand to tests
Web service calls may commit its transaction
Web server doesn’t know when to rollback the test data

To overcome these problems, we’ll need to fully control the lifecycle of web server’s database connection in the client tests. But how are we able to do this in a client-server architecture?

dRuby to rescue!

For those who are not familiar with it, dRuby is as the Remote Method Invocation to Java as to Ruby. It allows methods to be called in one Ruby process upon a Ruby object located in another Ruby process. Here is a good introduction to brush you up.

We’ll make use of dRuby to directly control the lifecycle of web service’s database connection (ActiveRecord::Base.connection) in our web services client tests. To do that, we add the following code to web server’s ”config/environments/test.rb”:

# server/config/environments/test.rb

config.after_initialize do
  ActiveRecord::ConnectionAdapters::ConnectionPool.class_eval do
    alias_method :old_checkout, :checkout

    def checkout
      @cached_connection ||= old_checkout
    end
  end

  require 'drb'
  DRb.start_service("druby://localhost:8000", ActiveRecord::Base)
end

The above code snippet does two things:

Patch ActiveRecord::ConnectionAdapters::ConnectionPool#checkout to make sure only one connection is shared across threads
Start a dRuby service for ActiveRecord::Base to be used in tests

In case you are wondering why it’s necessary to share one database connection across threads: ActiveRecord creates one database connection for each thread in its connection pool. Our web service client tests run in a separate thread from the server’s so it’s impossible to track which connection to rollback data for the web services calls. What we are doing here is to make sure there is only one connection created and we always rollback data for this connection.

After the aforementioned setup, we are able to expand the transaction boundary to tests:

# client/spec/models/task_spec.rb

describe Task do
  before :all do
    @semaphore = Mutex.new
    DRb.start_service
    @remote_base = DRbObject.new nil, "druby://localhost:8000"
  end

  before :each do
    begin_remote_transaction
  end

  after :each do
    rollback_remote_transaction
  end

  it "creates a task through web serices" do
    task = Task.create(:name => "Write a blog post", :ends_at => Date.tomorrow)
    Task.find(task.id).should == task
  end

  private

  def begin_remote_transaction
    @semaphore.lock
    @remote_base.connection.increment_open_transactions
    @remote_base.connection.transaction_joinable = false
    @remote_base.connection.begin_db_transaction
  end

  def rollback_remote_transaction
    @remote_base.connection.rollback_db_transaction
    @remote_base.connection.decrement_open_transactions
    @remote_base.clear_active_connections!
    @semaphore.unlock
  end
end

Voila! With dRuby, we use begin+rollback to isolate changes of web services calls to the database, instead of having to delete+insert for every test case. A huge performance boost!

Note that the Mutex lock in the code is to make sure that multiple web service client tests can run concurrently, for exmaple, using the parallel_tests gem. Without this lock, while the remote ActiveRecord connection is shared, the tests will behavior strangely in a multi-threads environment. You can ignore those lines if your web service client tests never run concurrently.

We can easily refactor out the begin_remote_transaction method and the rollback_remote_transaction method to spec_helper.rb, so that our web services client tests have little difference from usual ActiveRecord unit tests.

# client/spec_helper.rb

RSpec.configure do |config|
  config.before :all do
    @semaphore = Mutex.new
    DRb.start_service
    @remote_base = DRbObject.new nil, "druby://localhost:8000"
  end

  config.before :each do
    @semaphore.lock
    @remote_base.connection.increment_open_transactions
    @remote_base.connection.transaction_joinable = false
    @remote_base.connection.begin_db_transaction
  end

  config.after :each do
    @remote_base.connection.rollback_db_transaction
    @remote_base.connection.decrement_open_transactions
    @remote_base.clear_active_connections!
    @semaphore.unlock
  end
end

# client/spec/models/task_spec.rb

describe Task do
  it "creates a task through web serices" do
    task = Task.create(:name => "Write a blog post", :ends_at => Date.tomorrow)
    Task.find(task.id).should == task
  end
end

Fixture Creation

Most of the time, we create test fixtures to quickly define prototypes for each of the models and ask for instances with properties that are important to the test at hand. But in the context of REST web services, we can’t create fixtures unless there is a REST API defined. To break this constraint, we use dRuby to open up another channel to directly interact with fixture data on web server.

Assuming we are using the factory_girl gem for fixture creation, We create a dRuby service for port discovery and a dRuby service for each fixture instance:

# server/lib/drb_active_record_instance_factory.rb

require 'factory_girl'

class DRbActiveRecordInstanceFactory
  def get_port_for_fixture_instance(factory_instance)
    port = create_port
    inst = Factory.create(factory_instance)
    DRb.start_service("druby://localhost:#{port}", inst)
    port
  end

  def create_port
    # create a random port
  end
end

DRb.start_service('druby://localhost:9000', DRbActiveRecordInstanceFactory.new)

In tests, we ask for the port of the fixture instance and query its corresponding remote reference:

# client/spec/models/task_spec.rb

describe Task do
  before :all do
    @drb_factory = DRbObject.new(nil, 'druby://localhost:9000')
  end

  before do
    remote_task_port = @drb_factory.get_port_for_fixture_instance(:task)
    @remote_task = DRbObject.new(nil, "druby://localhost:#{remote_task_port}")
  end

  it "should ..."
    # test REST web services calls with @remote_task
  end
end

Summary

Testing REST web services client can be less complex if we have full control over objects on the web server. ActiveResource and dRuby stand out to help! They make writing web service client tests feel like writing local unit tests.

Git Up Perforce with git-p4

2011-03-23T00:00:00-07:00

Distributed version control systems open up lots of flexibility and provide lots of efficiency in work-flow than their centralized cousins (e.g. Perforce). There’s often value to use DVCS in the team.

However, the case that I run into frequently is where there is a corporate standard for a deficient VCS, but the team wish to work efficiently by using a more powerful VCS. In that case, there are many successful stories of using multiple VCS: DVCS for local version control and committing to a shared centralized VCS with a DVCS-to-VCS bridge.

Git is particularly good at this aspect which provides tons of bridges to other VCS (That’s another good reason for preferring Git over Mercurial :-)). Lately I have been using the git-p4 bridge to synchronize codes between a centralized Perforce repository and a local Git repository. This post is a tutorial on how to set this up.

Setting up the Perforce command line client

The git-p4 bridge requires the p4 command line client properly set up. There is a lengthy tutorial on Perforce’s documentation website. The following is a short sums-up:

Install the p4 command line client (use MacPort or Homebrew if you are a Mac guy :-))
Create a .p4settings file in your home directory with the global environment settings for your Perforce repository. And in your .bashrc, export P4CONFIG=/path/to/your/.p4settings. For example, here is what my settings look like:
```
P4PORT=repo_url:repo_port
P4USER=user_name
P4PASSWD=password
P4CLIENT=client_workspace_name
P4EDITOR=vim
```
Run “p4 client” to define the workspace mappings
Run “p4 info” to verify the settings

After you have done all these, don’t forget to issue “p4 sync repo_url” to test whether you are able to checkout stuff from your repository.

Installing git-p4 bridge

The git-p4 bridge is a Python script to enable bidirectional operation between a Perforce depot and Git. It doesn’t come with the Git distribution by default. To make it invokable in your system, you need to download the script and put it into your system path. For me, I clone the Git source code from GitHub and make a soft-link to the script:

Clone the Git repository to somewhere

git clone https://github.com/git/git.git git_source_path

Create a soft-link to the git-p4 script in one of your system paths
```
ln -s git_source_path/contrib/fast-import/git-p4 /usr/bin/git-p4 
```

Windows users need to include the git-p4.bat file in the system path.

Commands

Four things to remember when using git-p4:

Instead of using “git push” to push local commits to remote repository, use “git-p4 submit”
Instead of using “git fetch” to fetch changes from remote repository to local, use “git-p4 sync”
Instead of using “git pull” to fetch and merge changes from remote repository to local, use “git-p4 rebase”
Instead of using “git merge” to merge local branches, use “git rebase”

For the last one, the reason is that when you run “git merge”, Git creates an extra commit on top of the stack for the merging. This is not something we wanna show in the remote non-git repository. So we merge code with “git rebase”. Detailed explanation of the difference between git-merge and git-rebase goes here.

Workflow

There is a detailed explanation on the usage of git-p4 in Git’s source. Here is an example almost covering daily usage:

Login with “p4 login” and clone a Perforce project:
```
git-p4 clone //depot/path/project
```
Make some changes to the project and commit locally to Git:
```
git commit changed_file
```
In the meantime somebody in the team submitted changes to the remote Perforce repository. Merge it to your local repository:
```
git-p4 rebase
```
Submit your local changes back to Perforce:
```
git-p4 submit
```

Ignoring the .gitignore file

Normally we check in a .gitignore file to ignore files that we don’t want to check into the remote repository for each project. However, since the remote repository is not Git, it’s meaningless to check in this .gitignore file. To ignore this file, simply add an entry to .git/info/exclude.

Under the hook

There is no magic happening with git-p4. What it does is simply invoking the p4 command line tool to download sources to local, and then clone a Git repository out of it. You can simply verify this by typing “git branch -a”:

You may be amazed once again by how flexible the design of Git is which makes it possible to bridge to multiple VCS!

Summary

To quote Martin Fowler’s opinions on dual VCS, “a lot of teams can benefit from this dual-VCS working style, particularly if there’s a lot of corporate ceremony enforced by their corporate VCS. Using dual-VCS can often make both the local development team happier and the corporate controllers happier as their motivations for VCS are often different”.

Speed Up J2EE Environment Setup With Jetty Maven Plugin

2011-02-23T00:00:00-08:00

Setting up a J2EE development environment is typically complex and full of repetition. It wastes a lot of programmers’ time before they can happily get the app up and running. They need to configure the IDE, set up the database, configure the server, deploy app to the server, etc.. What’s worse, they have to repeat all the setup steps or a subset of the setup steps if they have multiple workspaces. I am not saying all of these steps are unnecessary. But can we just think for a moment why this is causing pain? How can we be DRY in environment setup?

Let’s take a look at where other web frameworks such as Rails shine. If you carefully think about how you get a Rails app running, you will be surprised to find out almost no configuration is needed. And it’s so automatic! Here is how it rolls:

> git clone <some_app>
> cd <some_app>
> bundle install
> rails server

Boom! Your app is up with just 2 commands (ignoring the step to download the source)! The last command is especially interesting. It is to start up the Rails server with the app loaded. Can we do the same in the Java world? Of course, Maven + Jetty Maven Plugin.

Introducing Jetty Maven Plugin

Jetty is a web server and javax.servlet container. One of its greatest features is its ability to be embedded into any Java application. The Jetty Maven plugin is just a simple integration of Jetty into the Maven build process. But don’t get it wrong, being embeddable doesn’t mean being less performing. Jetty is actually the powering engine behind Google App Engine and VMware Zimbra. After all, scalability is more related to your app’s implementation than the container.

Running a single web app

To run a single web app, it is as simple as including the Jetty Maven Plugin as a plugin dependency in your pom file and run the ”mvn jetty:run” command:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<name>foo</name>
<artifactId>foo</artifactId>
<packaging>war</packaging>
<build>
	<finalName>foo</finalName>
	<plugins>
		<plugin>
			<groupId>org.mortbay.jetty</groupId>
			<artifactId>jetty-maven-plugin</artifactId>
			<version>7.2.2.v20101205</version>
		</plugin>
	</plugins>
</build>
</project>

There are lots of useful settings that you can fine-tune Jetty, such as “scanIntervalSeconds” for the time interval of automatic hot redeploy, “systemProperties” for System properties of the execution of the plugin, and “jettyEnvXml” for the file that defines JNDI bindings.

Running a single web app with JNDI

In the case that you are defining data source using JNDI, you need to describe the data source in a “jetty-env.xml” file and refer it in the Jetty Maven Plugin. The jetty-env.xml file may look something like this:

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "http://jetty.mortbay.org/configure.dtd">
<Configure class="org.eclipse.jetty.webapp.WebAppContext">
	<New id="jndi" class="org.eclipse.jetty.plus.jndi.Resource">
		<Arg>jdbc/jndi</Arg>
		<Arg>
			<New class="org.apache.commons.dbcp.BasicDataSource">
				<Set name="driverClassName">${db_connection_driver_class}</Set>
				<Set name="url">${db_connection_url}</Set>
				<Set name="username">${db_connection_username}</Set>
				<Set name="password">${db_connection_password}</Set>
			</New>
		</Arg>
	</New>
</Configure>

The pom.xml now becomes:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<name>foo</name>
<artifactId>foo</artifactId>
<packaging>war</packaging>
<build>
	<finalName>foo</finalName>
	<plugins>
		<plugin>
			<groupId>org.mortbay.jetty</groupId>
			<artifactId>jetty-maven-plugin</artifactId>
			<version>7.2.2.v20101205</version>
			<webAppConfig>
				<jettyEnvXml>${basedir}/src/over/here/jetty-env.xml</jettyEnvXml>
			</webAppConfig>		
		</plugin>
	</plugins>
</build>
</project>

Running multiple web apps

It’s a bit tricky to run multiple web apps at once with the Jetty Maven Plugin. You need to have a running web app to delegate the requests. If you are using JNDI to define data source, you also need to configure your web apps to load up their environment settings.

As an example, assuming you have two web apps foo and bar. They are organized in different Maven projects like the followings:

- pom.xml
- foo/
  - pom.xml
- bar/
  - pom.xml

If we wanna deploy both servers with the Jetty Maven Plugin, we need to either get foo running and delegate requests to bar, or the other way around. So it’s a good practice to keep foo and bar untouched and introduce a third server to do the request delegation. The ultimate folder structure will look like this:

- pom.xml
- foo/
  - pom.xml
- bar/
  - pom.xml
- servers/
  - pom.xml

The servers project is basically an empty server with nothing but a web.xml. In its pom file, we define the redirection path and make sure it loads up settings such as JNDI for each web app:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">	
<modelVersion>4.0.0</modelVersion>
<parent>
	<groupId>mavenJettyPluginExample</groupId>
	<artifactId>parent</artifactId>
	<version>1.0-SNAPSHOT</version>
</parent>
<artifactId>servers</artifactId>
<packaging>war</packaging>
<name>servers</name>
<build>
	<finalName>servers</finalName>
	<plugins>
		<plugin>
			<groupId>org.mortbay.jetty</groupId>
			<artifactId>jetty-maven-plugin</artifactId>
			<configuration>
				<contextHandlers>            
					<contextHandler implementation="org.eclipse.jetty.webapp.WebAppContext">
						<contextPath>/foo</contextPath>
						<resourceBase>${basedir}/../foo/target/foo</resourceBase>
						<configurationClasses>
							<configurationClass>org.eclipse.jetty.webapp.WebInfConfiguration</configurationClass>
							<configurationClass>org.eclipse.jetty.webapp.WebXmlConfiguration</configurationClass>
							<configurationClass>org.eclipse.jetty.webapp.MetaInfConfiguration</configurationClass>
							<configurationClass>org.eclipse.jetty.webapp.FragmentConfiguration</configurationClass>
							<configurationClass>org.eclipse.jetty.plus.webapp.EnvConfiguration</configurationClass>
							<configurationClass>org.eclipse.jetty.plus.webapp.PlusConfiguration</configurationClass>
							<configurationClass>org.eclipse.jetty.webapp.JettyWebXmlConfiguration</configurationClass>
							<configurationClass>org.eclipse.jetty.webapp.TagLibConfiguration</configurationClass>
						</configurationClasses>
					</contextHandler>
					<contextHandler implementation="org.eclipse.jetty.webapp.WebAppContext">
						<contextPath>/bar</contextPath>
						<resourceBase>${basedir}/../bar/target/bar</resourceBase>
						<configurationClasses>
							<configurationClass>org.eclipse.jetty.webapp.WebInfConfiguration</configurationClass>
							<configurationClass>org.eclipse.jetty.webapp.WebXmlConfiguration</configurationClass>
							<configurationClass>org.eclipse.jetty.webapp.MetaInfConfiguration</configurationClass>
							<configurationClass>org.eclipse.jetty.webapp.FragmentConfiguration</configurationClass>
							<configurationClass>org.eclipse.jetty.plus.webapp.EnvConfiguration</configurationClass>
							<configurationClass>org.eclipse.jetty.plus.webapp.PlusConfiguration</configurationClass>
							<configurationClass>org.eclipse.jetty.webapp.JettyWebXmlConfiguration</configurationClass>
							<configurationClass>org.eclipse.jetty.webapp.TagLibConfiguration</configurationClass>
						</configurationClasses>
					</contextHandler>
				</contextHandlers>  
			</configuration>
		</plugin>
	</plugins>
</build>
</project>

The “resourceBase” of each web app is set to its corresponding target folder. That means we have to compile the web app before running them. The “contextPath” is to tell Jetty the path of the web app. The list of “configurationClasses” is to tell Jetty to load corresponding configurations including the jetty-env.xml file for JNDI definition.

Again, running ”mvn jetty:run” will start up these two servers at once.

Summary

As you may see, after setting up Jetty in the build process, we are able to start up the server with just one command. It speeds up the setup of our development environment and free ourselves from repeating the setup steps in the future. Most importantly, your teammates will thank you for saving a lot of their precious time.

The source code of the example is available here: https://github.com/jingweno/maven_jetty_plugin_example.

Agile is an adjective, not a noun!

2011-02-07T00:00:00-08:00

Dave Thomas was right, there’s a growing tendency to treat the word “Agile” as a noun. Being an agile software developer means doing things in an agile manner, not just naming things that you do “Agile”. Being an agile software developer means persisting your agile way of doing things, not compromising it in any case. Being an agile software developer means evangelizing your core values, not isolating your agile self.

If you miss any of these, you are just fooling yourself.

Running a single JUnit test in Eclipse

2011-02-05T00:00:00-08:00

A lot of the test-driven developers may find it annoying to run a single JUnit test in Eclipse: if there is a test suite with multiple tests, when running a single unit test, either from the context menu of the code editor, or from the JUnit view, Eclipse seems to always insist on running the entire suite, rather than the single test.

Like the example below, running testOne will trigger running the rest of the tests in the suite which are grouped under the ”Unrooted Tests” node. This is quite irritating sometimes since running unexpected tests may slow down the workflow if they take time to finish. It’s also quite confusing since programmers get unwanted test results. Some of my coworkers solve this problem by commenting out unwanted tests each time before running a test. However, it’s not very effective and it’s error-prone.

This happens mainly because we are running JUnit 3 tests with the JUnit 4 runner. Before version 4.6, the JUnit 4 runner seems to have the problem of falling back to run a single JUnit 3 test.

So…the solution is quite simple:

change the JUnit runner to JUnit 3 in Eclipse, or
upgrade your JUnit 4 library to 4.6 or higher.

Now the problem is gone and the test-driven developers are happy :-).

Loading Path Gotchas in Rails 3

2011-01-20T00:00:00-08:00

The algorithm of load path inferring in Rails 3 is a bit weird when class caching is turned off. Assuming you add all subdirectories under app/models to the load paths by using the new “config.autoload_paths” setting in config/application.rb:

config.autoload_paths += Dir["#{config.root}/app/models/**/"]

And you have a subdirectory under app/models with name class1 and a file under this subdirectory with the same name class1.rb. When you are referring any classes in class1.rb under this subdirectory, you have to make sure they are in the namespace of Class1. Otherwise Rails will complain about your referred class is not in the namespace of Class1. In more details, if you have a directory structure like this:

- app
  - models
    - class1
      - class1.rb
      - class1_reference.rb

When you refer to Class1Reference in Class1, you will get ”Expected app/models/class1/class1_reference.rb to define Class1::Class1Reference”. If the subdirectory is not named class1, strangely this example will work… Let’s take a look at a comparing directory structure:

- app
  - models
    - not_class2
      - class2.rb
      - class2_reference.rb

This second example have everything the same as the first one (Class2 referring to Class2Reference) except that the subdirectory name (not_class2) is different from the file name (class2.rb). There is no exception raised and everything works as expected.

The result indicates either a (strange) directory naming convention in Rails 3 or a potential bug in its path inferring algorithm. Here are the lines in Rails 3 that does the magic:

# Load the constant named +const_name+ which is missing from +from_mod+. If
# it is not possible to load the constant into from_mod, try its parent module
# using const_missing.
def load_missing_constant(from_mod, const_name)
  ...

  file_path = search_for_file(path_suffix)

  if file_path && ! loaded.include?(File.expand_path(file_path)) # We found a matching file to load
    require_or_load file_path
    raise LoadError, "Expected #{file_path} to define #{qualified_name}" unless local_const_defined?(from_mod, const_name)
    return from_mod.const_get(const_name)
  elsif 
	...
end

Please note that if class caching is turned on, both cases work. Now everything becomes quite clear, the line with “unless” after the “raise” is causing the problem: when class caching is turned off, “local_const_defined?” always returns false hence raising the “LoadError”. I have created a bug report on this issue and I will post replies here once I get anything :).

The code of the two examples above are available here. It has a simple view to display the output of the load path exception. To get started, just run:

> bundle install
> rails server

And check two pages: http://localhost:3000/class1 and http://localhost:3000/class2.

Planning for CodeFaces 1.1.1

2010-09-21T00:00:00-07:00

For the pass couple months, amazing things have happened to CodeFaces: it has enough features and scalability to go live, it was listed as a demo app on Eclipse RAP’s demo page, good feedback has been heard after people using it, and constant feature requests have been documented and pushed to our backlog. That’s pretty encouraging both to me and to the CodeFaces Team.

In case you haven’t know, CodeFaces is a web-based source control client that targets at easing the pain of navigating codes from a source control system on a browser. It allows connecting to multiple source control systems and structurally navigating codes on a unified web interface.

It’s nice to see a potential community growing and we decided to continue making cool stuff happen. We have already marked down what you asked for: file searching, connecting to private repositories, flat structure on showing the folder trees, etc.. Right now we are on our way to the next CodeFaces release (v1.1.1). This release will be a maintenance one and the main theme is bug fixing, performance tuning and minor feature upgrade. We will soon scope it down in our issue tracker and open to everybody. Do not hesitate to ask us for new features. We greatly appreciate it! :)

Happy code navigating!

How to set up and run PDE build

2010-09-19T00:00:00-07:00

The Plug-in Development Environment (PDE) provides tools to build Eclipse plug-ins, fragments, features, update sites and RCP products. This post is about how to set up and run the PDE headless build. Headless build means making the build outside the Eclipse IDE. You can also manually build it inside Eclipse with its GUI.

Before running the PDE build, you need to set up the followings:

build directory
target platform
build configuration file

Build directory

The first step in setting up a build is to create the directory in which the build will take place. Next, create two subdirectories called ”plugins” and ”features”, and copy the plug-ins and features that you want to build respectively into these two folders. The directory structure for the build directory should look like this:

- build_directory
    - plugins
        - plugin_to_build_a
        - plugin_to_build_b
        - ...
    - features
        - feature_to_build
        - ...

Target platform

Features and plugins are compiled and run against a set of pre-built features and plugins. These dependencies constitute the target platform. For example, most RCP applications contribute to the UI by depending on the org.eclipse.ui plug-in.

In this step, you also need to create a directory with two subdirectories called ”plugins” and ”features”, and copy all the dependencies over. For developing RCP applications for multiple platforms, the RCP delta pack is a good option since it contains all the platform specific fragments from the Eclipse SDK. Download the delta pack and unzip it, then you finish setting up the target platform :). The directory structure for the target platform should look like this:

- target_platform
    - plugins
        - org.eclipse.core.runtime
        - org.eclipse.ui
        - ...
    - features
        - org.eclipse.equinox.launcher
        - ...

Build configuration file

After setting up the directory structure for the plugins/features to build and the target platform, we need to tell PDE how we want the build. We can configure it by adding a build.properties file. The template of this file is available in any Eclipse distribution:

#{eclipseInstall}/plugins/org.eclipse.pde.build_#{version}/templates/headless-build/build.properties

Create a directory for the configuration file, copy the template file over, comment out unnecessary properties in the template file. The directory structure should look something like this:

- build_config
    - build.properties

PDE build provides a variety of properties that we can configure for the build, from checking out source from CVS to controlling the whole build lifecycle. In this post, we only discuss the most essential ones:

topLevelElementType: feature or plugin
topLevelElementId: the id of the top level element we are building
archivePrefix: the prefix that will be used in the generated archive
collectingFolder: the location under which all of the build output will be collected
configs: the list of {os, ws, arch} configurations to build

Running PDE build

If you have followed me this far, you should have the following directory structure ready:

- build_directory
    - plugins
        - plugin_to_build_a
        - plugin_to_build_b
        - ...
    - features
        - feature_to_build
        - ...
- target_platform
    - plugins
        - org.eclipse.core.runtime
        - org.eclipse.ui
        - ...
    - features
        - org.eclipse.equinox.launcher
        - ...
- build_config
    - build.properties

Open up a terminal, type in:

java -jar #{eclipseInstall}/plugins/org.eclipse.equinox.launcher_#{version}.jar
     -application org.eclipse.ant.core.antRunner
     -buildfile #{eclipseInstall}/plugins/org.eclipse.pde.build_#{version}/scripts/build.xml
     -DbuildDirectory=#{path_to_the_build_directory}
     -DbaseLocation=#{path_to_the_target_platform_directory}
     -Dbuilder=#{path_to_the_build_configuration_directory}

You should see PDE starts building :).

Summary

Setting up the PDE build looks scary at first, but once you understand Equinox a little bit more, you will find everything make a lot of sense. All you need to get it to run is creating the directory structure for the plug-ins/features to build and their dependencies, and configuring some properties. The directory structure, probably as you have already realized, is Equinox’s convention of managing components.

You can also set up Ant tasks or Rake tasks to automate the whole process. Like the exercise that I have done lately, I used Rake to automate the PDE build for the Eclipse FITPro plugin. The source is available here: https://fitpro.svn.sourceforge.net/svnroot/fitpro/Eclipse/trunk/com.luxoft.eclipse.fit.runner.releng/.

Patching with Class Shadowing and Maven

2010-07-20T00:00:00-07:00

Before we start, you may speculate about the usefulness of class shadowing. Class shadowing can be used as a trick to patch or replace behaviours of classes at runtime, taking advantage of the first-come first-served algorithm of Java’s class loader. If there are more than one class with the same fully qualified name in the Java class loader, the first one that shows up always take precedence over the rest.

This is extremely useful in some cases. For example, in order to make some out-of-the-box (OOTB) methods extensible without directly changing the source, or to provide backward-compatibility support, class shadowing is a good technique to separate the OOTB code and the customized code. By creating a patch jar against the OOTB jar and putting it before the OOTB jar on classpath, classes with the same qualified name are merged at runtime.

Class shadowing is only one way of patching a class. It’s based on class loader’s runtime class resolution. Another way is to use class overlay. For example, the Maven war overlays plugin expands the war file and copy them on top of the host classes.

If you are already familiar with OSGi based technology, patch fragment is actually taking advantage of the class shadowing mechanism. This post is not targeting at building OSGi application but at standard Maven project.

To instruct Java’s class loader to load jars in a specific order, we can make use of the “-cp” option. For example,

java -cp patch.jar ootb.jar -jar main.jar

It tells the class loader that patch.jar is loaded before ootb.jar, hence any classes in patch.jar is overriding the ones in ootb.jar. Besides, we can also make use of the Class-Path attribute of the jar manifest to specify the classpath. We will adopt this method in the follow-up example.

Here is an example we are going to build. HelloWorldProxy is a proxy artifact that exports its classpath in such an order that classes in HelloWorldPatch are replacing classes in HelloWorld. HelloWorldTest depends on HelloWorldProxy and doesn’t know which implementation (HelloWorld or HelloWorldPatch) HelloWorldProxy is exporting. The dependency graph is as followed:

In the pom.xml of HelloWorldProxy, it has two dependencies and we put HelloWorldPatch before HelloWorld, since we would like to see classes in HelloWorldPatch replacing the ones in HelloWorld. As of Maven 2.0.9, the ordering of dependencies on the classpath is preserved. The code snippet is as followed:

<dependencies>
    <dependency>
        <groupId>HelloWorld</groupId>
        <artifactId>HelloWorldPatch</artifactId>
        <version>0.0.1-SNAPSHOT</version>
        <type>jar</type>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>HelloWorld</groupId>
        <artifactId>HelloWorld</artifactId>
        <version>0.0.1-SNAPSHOT</version>
        <type>jar</type>
        <scope>compile</scope>
    </dependency>
</dependencies>

We also need to make sure HelloWorldProxy exports the two jars in the Class-Path attribute of the MANIFEST.MF file with the correct ordering. The key is to set the addClasspath flag to true in the maven-jar-plugin:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jar-plugin</artifactId>
    <version>2.3.1</version>
    <configuration>
        <archive>
            <manifest>
                <addClasspath>true</addClasspath>
            </manifest>
        </archive>
    </configuration>
</plugin>

Run “mvn package” in HelloWorldProxy and take a look at the generated MANIFEST.MF:

Voila! That’s what we expect! HelloWorldPatch takes precedence over HelloWorld on HelloWorldProxy’s classpath!

Now HelloWorldTest can safely depends on HelloWorldProxy and expects that HelloWorldProxy will export HelloWorldPatch’s implementations at runtime:

<dependency>
    <groupId>HelloWorld</groupId>
    <artifactId>HelloWorldProxy</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <type>jar</type>
    <scope>compile</scope>
</dependency>

The source of this example is available on GitHub http://github.com/jingweno/patching_with_class_shadowing_and_maven. You can also view it directly with CodeFaces http://codefaces.org/http://github.com/jingweno/patching_with_class_shadowing_and_maven.

Introducing Eclipse RAP

2010-07-08T00:00:00-07:00

RAP, what? The Eclipse community is expanding its range to Hip Hop :)? Not exactly, although there is a chance. Eclipse Rich Ajax Platform (RAP) is a framework that empowers developers to build rich web clients “the Eclipse way.” From the latest Helios download, RAP Tooling has become a default component of the RCP bundle:

So what exactly is the Eclipse paradigm? Can we benefit from it?

Architecture

RAP is to the web as RCP to the desktop. It inherits all the goodness from RCP such as workbench extension points model, event-driven SWT/JFace APIs, and componentized OSGi design. As indicated below, the only difference between the architecture of RAP and that of RCP is the implementation of SWT/RWT. RWT is actually a bundle providing web-specific implementation of SWT’s widgets based on the qooxdoo toolkit. In RAP, almost no SWT API is changed.

The following is the architecture of RAP:

This architecture determines that developers can bring their RCP applications to the web with a single code base, also called single sourcing¹. Like building a RCP application, building a RAP application is a process of building plug-ins and bundles: On the UI side, contributing widget plug-ins; On the server side, since it is powered by server-side Equinox, contributing servlet plug-ins. Unsurprisingly, it also inherits the benefits of any OSGi application such as dynamically adding/removing bundles.

This architecture also determines that RAP is a server-centric AJAX framework². As indicated below, in a server-centric framework, the application is hosted on the application server and all processing is done on the server. The client browser is only used for data presentation. Consequently, it leaves small footprints on browsers: it waits for instructions from the server to create corresponding widgets on demand. In contrast, client-centric framework such as GWT statically compiles all widgets beforehand.

Both methods have pros and cons. Server-centric frameworks handle all communication automatically and leaves as few JavaScript codes on the browsers as possible, hence potentially having less security holes on the browsers. Client-centric frameworks are more flexible but require more work to step deep into low-level implementations of client-to-server communication, e.g., figuring out how to update widgets effectively in an asynchronous callback.

UI Testing

For a Java-based rich AJAX application, while server side can be tested with JUnit, testing UI is always a headache. Like GWT, testing UI has to rely on external JavaScript libraries such as JsUnit. But for RAP, since it’s server-centric, UI code can be tested using JUnit with the special RAPTestCase. Moreover, since the workbench APIs haven’t been changed, all existing SWT testing tools can be reused, e.g., Selenium or SWTBot.

Performance

In general, RAP pushes more work off the client onto the server, thus communicating more information and more frequently with the server. Some performance testings were available³.

Deployment

RAP is extremely easy to deploy on an OSGi based server ⁴, e.g., Spring dm server which has moved to the Eclipse Virgo project, because its server side implementation bases on the http OSGi standard. Deploying to a traditional J2EE such as Tomcat requires packaging an extra ”Servlet Bridge” into the war. A drawback of RAP is that it can’t be deployed to a plain web server such as Apache. You can do this to a GWT-based application without RPC and the generated JavaScript codes will still work fine on a browser. h4. Summary

RAP is created to provide developers a possibility to bring the goodness of RCP applications to the web without the need to digging through low-level web-technologies. This greatly reduces cost by reuse of knowledge and code. It provides a thin client with a rich widget set and a stageful server-side on top of OSGi, and reuses the Eclipse workbench paradigm. As OSGi-based web containers such as Spring dm server are getting more prevalent, RAP will definitely enjoy more popularity. h4. Reference

Why code navigation sucks on most SCM's web interfaces?

2010-07-05T00:00:00-07:00

I am not surprised that you feel the same. It sucks to navigate codes on most source control system’s web interfaces! Here are whys.

The display of files in most SCM’s web interface is not in a tree, but in a list. Files are in a tree structure; displaying them as a list is definitely a mismatch! The consequence of converting a tree structure to a list one is that the file context is lost during navigation. I believe you must feel annoying when clicking back and forth in order to find more than one file on GitHub’s web interface:
There is a disconnection between opened file’s context and opened file’s content. Take Google Code’s web interface as an example, although its display of files is in a tree structure, but after a file is opened, the context where the file belongs to is gone: only the path of the file is still kept:
Being impossible to open multiple files at once. File is a module to separate concerns in almost all programming languages. At most of the time, reading code only from one file is not informative enough. In a programming language sense, files have connections to each other and being able to show related file modules at once is critical to understand the application. Moreover, we have been used to exploring pages with multiple tabs on a browser, and have been used to navigate codes with multiple viewers in an IDE. This feature is definitely a must!
There is no unified UI to navigate codes across all source control systems. We have ViewVC, Gitweb, Warehouse, many many… But we don’t have a web interface that adapts to all kinds of version control systems.

If you feel there is a need to improve existing SCM’s web interface, I would like to introduce to you a neat tool that we built lately: CodeFaces. CodeFaces is a source control client made for browsers. It provides an IDE-looking page to structurally navigate codes from a version control system. Check out http://codefaces.org for more updates and here is a screencast that walks you through all its features: http://www.youtube.com/watch?v=JNVcyi0cDQ0&hd=1.

Pushing Adaptive Implementations with Eclipse's Extension Points

2010-04-12T00:00:00-07:00

A week ago, I blogged about implementing adaptive components using Eclipse’s optional plugin dependencies. In that post, I gave an example on displaying HTML files with different editors based on client’s environment: open the HTML file with the HTML editor from WTP if the plugin is installed, otherwise fallback to open with the default text editor. The final solution was to query the existence of the WTP editor through Equinox.

However, the implementation is far from enjoyable: our main plugin knows everything about WTP! In other words, in order to specify the query, our main plugin has to know about WTP, hence containing environment-specific implementations. The following graph shows the dependency:

The reason causing this environment-specific coupling is that our main plugin uses the pull method to retrieve the HTML editor implementations from WTP (code snippet goes here). To solve this, we can inverse the control and have another plugin to push specific implementations to the main plugin.

Equinox has already offered a very powerful tool for this purpose: extension points. In the dependency graph above, we separate the concerns by creating an adaptor plugin between the main plugin and the WTP plugin. We declare its dependency on WTP as optional to make sure the plugin will be installed on clients even if the WTP dependency is missing. If somehow WTP is installed in the future, the adaptor will automatically hook it up. Besides, we define an extension point in the main plugin so that the adaptor plugin knows what to contribute and where to contribute.

Here are steps to implement:

Create an extension point in the main plugin to allow editor contributions. We confine that the contributing editors should be an instance of ITextEditor. The schema looks something like this:

<element name="extension">
  <complextype>
    <sequence>
	  <element ref="sourceEditor"></element>
    </sequence>
	...
  </complextype>
</element>

<element name="sourceEditor">
  <complextype>
    <attribute name="id" type="string" use="required">
	  ...
	</attribute>
    <attribute name="class" type="string" use="required">
	  <annotation>
	    <appinfo>
		  <meta.attribute kind="java" basedon=":org.eclipse.ui.texteditor.ITextEditor"/>
	    </appinfo>
      </annotation>
    </attribute>
  </complextype>
</element>

Create a factory class in the main plugin that knows how to create an ITextEditor after reading the extensions. We also need to reconcile that only one editor extension is initialized (in the case of multiple contributions):

public class SourceEditorFactory {
	private static IConfigurationElement cachedConfigurationElement;

	private static final String ATTRIBUTE_CLASS = "class";

	public static ITextEditor createSourceEditor() {
		if (cachedConfigurationElement == null) {
			cachedConfigurationElement = readConfigurationElement();
		}

		if (cachedConfigurationElement != null) {
			try {
				return (ITextEditor) cachedConfigurationElement
						.createExecutableExtension(ATTRIBUTE_CLASS);
			} catch (Exception exception) {
				// errors when initializing the source editor
			}
		}

		return new TextEditor();
	}

	// Reads the first configuration element whose implemented class is not
	// TextEditor
	private static IConfigurationElement readConfigurationElement() {
		IExtensionPoint extensionPoint = Platform.getExtensionRegistry()
				.getExtensionPoint(FitPlugin.PLUGIN_ID, "fitSourceEditor");

		for (IConfigurationElement configurationElement : extensionPoint
				.getConfigurationElements()) {
			if (!StringUtils.equals(TextEditor.class.getCanonicalName(),
					configurationElement.getAttribute(ATTRIBUTE_CLASS))) {
				return configurationElement;
			}
		}

		return null;
	}
}

You might have noticed that lines 15 to 20 silently catch all exceptions and return the default text editor if there are any errors when initializing the extension. It ensures that even if WTP is not installed on the client, there is a fallback implementation.

Declare an extension in the adaptor plugin which refers to the HTML editor from WTP.

<extension point="com.luxoft.eclipse.fit.runner.fitSourceEditor">
  <sourceeditor class="org.eclipse.wst.sse.ui.StructuredTextEditor" 
		id="com.luxoft.eclipse.fit.runner.optional.wtpHTMLEditor"/>
</extension>

For more details, please check out the source.

From above, we learn how to push adaptive implementations using extension points, which decouples environment-specific implementations by inverting the controls. Another powerful tool that Equinox provides for adaptive implementations is Fragment. If you are careful enough, you might have noticed that our adaptor plugin actually acts as a fragment, a library that is specific to a particular operating system or windowing system. In the next blog post, I will talk more about it in details.

Adaptive Implementations with Eclipse's Optional Plugin Dependencies

2010-04-07T00:00:00-07:00

Equinox is an OSGI implementation that allows bundles to find out what capabilities are available on the system and adapt the functionality they can provide. This adaptivity is achieved through the OSGi service registry. In this blog post, I am gonna talk about how to make use of Eclipse’s optional plugin dependencies to support diverse runtime environment.

Let’s get started with an example: When a HTML file is clicked, it should be open with the HTML editor from WTP if the plugin is installed. Otherwise open it with a text editor. (This is a real world example from FITpro BTW :) ).

First, we need to add the WTP UI component as an optional dependency.

The MANIFEST.MF should look something like this:
```
Require-Bundle:
 ...
 org.eclipse.wst.sse.ui;resolution:=optional
```
Create a HTML editor if WTP is installed on the client, otherwise fallback to text editor. Wait…here we might have more than one solution:

First of all, we might consider directly initializing the HTML editor class from WTP and if it doesn’t exist, initialize the text editor. However, BundleLoader won’t let this happen since it throws out ClassNotFoundException if a class doesn’t exist (in the case of WTP not being installed on the client), and this exception can’t be caught in client implementations.

Secondly, we might also consider initializing the HTML editor class through reflection. In this way, we can catch our own ClassNotFoundException and fallback to create a text editor.

public TextEditor createHTMLTextEditor() {
	try {
		Class cls = Class
			.forName("org.eclipse.wst.sse.ui.StructuredTextEditor");
		for (Constructor constructor : cls.getDeclaredConstructors()) {
			if (constructor.getGenericParameterTypes().length == 0) {
				return (TextEditor) constructor.newInstance(new Object[0]);
			}
		}
	} catch (Throwable throwable) {
		// couldn't find the WTP plugin
	}

	return new TextEditor();
}

However…this is not quite “OSGI”. Actually, Equinox provides APIs to retrieve extension configurations and initialize their implemented classes. It uses the same way to initialize classes like what we are doing here.

We can ask Equinox for all the registered editor extensions and find out whether WTP’s HTML editor is one of them. If yes, just initialize it. Otherwise, create the text editor that is available to all clients.

private static final String HTML_EDITOR_ID = "org.eclipse.wst.sse.ui.StructuredTextEditor";

private static final String ATTRIBUTE_CLASS = "class";

private static final String ATTRIBUTE_ID = "id";

private static final String EDITOR_EXTENSION_POINT_ID = "org.eclipse.ui.editors";

public TextEditor createHTMLTextEditor() {
	try {
		IExtensionPoint extensionPoint = Platform.getExtensionRegistry()
			.getExtensionPoint(EDITOR_EXTENSION_POINT_ID);
		if (extensionPoint != null) {
			IConfigurationElement[] configurationElements = extensionPoint
				.getConfigurationElements();
			for (IConfigurationElement element : configurationElements) {
				if (StringUtils.equals(element.getAttribute(ATTRIBUTE_ID),
				HTML_EDITOR_ID)) {
					return (TextEditor) element.createExecutableExtension(ATTRIBUTE_CLASS);
				}
			}
		}
	} catch (Exception exception) {
		// couldn't find the WTP plugin
	}

	return new TextEditor();
}

Now we see how flexible it is to provide an adaptive Implementation with Eclipse’s optional plugin dependencies. With this power, we can also support something like upgrading a system without ruining the old working legacy components. A tips can be found here.

Judicious Serialization

2009-11-26T00:00:00-08:00

I was fixing a bug related to Java Serialization the other day: Server 1 tried to send out an email by using one of the web services from Server 2 with serialized objects passed along. It almost took me half a day to dig out the problem because the stack trace wasn’t helpful enough other than saying there were Serializable exceptions. After lots of debugging, I finally found out that the problem was one of the Serializable objects having non-Serializable attributes.

There must be some lessons we can learn from here! I quickly checked the hierarchy of the object. Okay…great! Its interface indirectly inherited from Serializable. A code smell came into my mind immediately: if a class is designed for inheritance (including interfaces), rarely extend it from Serializable. It isn’t too hard to understand why: once a class implements Serializable,

all its attributes have to either accept the fact that they are Serializable or transient, and
all its subclasses have to tightly couple to the Serializable club, including their attributes.

With Serializable exposed to interface level, the flexibility of the interface is highly reduced and we have to spend proportional effort on maintaining them. Just imagine there are 3 classes with 5 attributes each. To successfully transmit them through Internet, you need to make sure 18 items are Serializable (15 attributes + 3 classes). If you forget any of them, well, you could imagine how painful it is to count which one is missing with all your fingers (that’s exactly what I have been through!!). Needless to say how to test them. For testing, you must ensure 18 times both that the serialization-deserialization process succeeds and that it results in a faithful replica of the original object.

Alright, this is enough frustration! Serialization should be used judiciously! But how? First of all, of course, never expose Serializable to classes that are meant to be inherited. Do it in its subclasses like the following:

interface Foo { ... }
class Bar implements Foo, Serializable { ... }

Secondly, try to minimize as much as possible the scope of serialization. Never serialize more than you need. It will just make it harder to debug if anything wrong happened. For example, you are passing the following Serializable object from Server 1 to Server 2.

class Baz implements Serializable {
    private A a;
    private B b;
    private C c;
    ...
}

But somehow you know that only Baz.a and Baz.c are used on Server 2. So just pass Baz.a and Baz.c instead of the whole object! This is also true from the perspective of design: YAGNI, only pass the objects that you care about.

Finally, use as much as possible classes from the Java library in serialization, or use other alternative technologies. ArrayList, BigDecimal and many others come with the Serializable ability by default. Why reinventing the wheels and creating you own mess? Popular alternatives are XML, JSON, or YAML.

There are lots of great articles on this topic, e.g., Chapter 11 of Effective Java. I don’t wanna DRY myself here :). Last but not least, bewared when you see Serializable! It’s tricky!

Running Eclipse Java Compiler with Ant

2009-11-18T00:00:00-08:00

Eclipse Java Compiler (EJC) is an incremental compiler that is generally faster than any JDKs. It also fixes some incompatible problems related to JDK6, e.g. http://issues.apache.org/jira/browse/OPENJPA-640. This post is about how to run EJC with an ant target to compile your Java projects.

Install Ant 1.7+.
Download the latest JDT Core Batch Compiler and put it directly into your $ANT_HOME/lib folder. Or, if you have Eclipse installed, you have the compiler by default: Locate the org.eclipse.jdt.core_xxx.jar in the plugins directory of your Eclipse installation, put that jar file into the $ANT_HOME/lib folder, and at the same time unzip this jar file and make jdtCompilerAdapter.jar available to ant. As an alternative, you can specify the path of the jar file by using the “-lib” flag of ant.
You are now ready to test the Eclipse compiler in command line. The key is to direct ant to use EJC by specifying the compiler parameter -Dbuild.compiler to org.eclipse.jdt.core.JDTCompilerAdapter. Go to a Java project and type in:
```
ant -Dbuild.compiler=org.eclipse.jdt.core.JDTCompilerAdapter
    -Dant.build.javac.target=1.6
    -Dant.build.javac.source=1.6 compile
```

You can also set up an ant target to play around with it:

<target name="-eclipse-compile">
  <javac source="1.6" target="1.6" compiler="org.eclipse.jdt.core.JDTCompilerAdapter">
     …
  </javac>
</target>