bosy.DailyDev.org

Speed up Maven build

2009-02-17T16:03:00.001+01:00

From a sprint retrospective we got a lot of 'minuses' related to the speed of Maven build executed by every developer localy on their computers many times per day. The developers were frustrated that complete build tooks from 10 to 15 minutes and estimated that they spend from 30 to 60 minutes every day just waiting for build. What in a team containing 16 developers (it was two scrum teams) gives you from 8 to 16 manhours per day. It is really a lot so we decided to try to do something about it.

At the begining I have to say that if there is such problem in the project, it usualy mean that something is wrong with modularity, or your IDE should be replaced or used in a better way to support faster code-see iterations. But it wasn't true in our case. Our problem was legacy web application famework without any support in existing IDEs.

Since I was responsible for build process I started to work on this issue. At the first I started to talk with developers about how do they build it. They simply used the most common way of application building:

mvn clean install

to rebuild this all, or

mvn install

to just repack updated sources and resources.

So I started to "profile" the build and early found surprising fact that approximately 50% of the time is taken by 'clean' goal - I found first bottleneck.

To remove it I used simple trick - move/rename operation on the folder is immediate on most of the systems in comparison with delete operation.

...
<profile>

<id>qclean</id>
<build>

<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>

<id>rename_target</id>
<phase>pre-clean</phase>
<goals>

</goals>
<configuration>

<tasks>

<format property="targetTstamp"

pattern="yyyyMMdd-HHmmss" locale="en,US" />

</tstamp>
<move todir="trash/target-${targetTstamp}"

failonerror="false">

</move>

</tasks>

</configuration>

</execution>

</executions>

</plugin>

</plugins>

</build>

</profile>
...

In the code snapshot you can see definition of new qclean (means Quick Clean) profile. The profile contains configuration of maven-antrun-plugin to be executed in a 'pre-clean' phase. The Ant script at the first generates a timestamp string into a property 'targetTimestamp' using 'tstamp' task. Then it moves and renames 'target' folder to 'trash/target-${targetTstamp}'.

Now when developers executed build using following command they simply spent just half of the time waiting for build.

mvn clean install -Pqclean

Because the 'target' folder is just moved and not deleted it is necessary to clean also 'trash' folder sometimes. But you can do it as a background operation or while you are not using the computer at all. For this purpose I created one more profile named clean.

...
<profile>

<id>clean</id>
<build>

<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>

<id>clean_trash</id>
<phase>clean</phase>
<goals>

</goals>
<configuration>

<tasks>

</tasks>

</configuration>

</execution>

</executions>

</plugin>

</plugins>

</build>

</profile>
...

It can be executed using following command.

mvn clean -Pclean

During second profiling round I found again that there is 50% of time consumed by tasks which are without any use during quick code-see development. The tasks were unit tests (please do not beat me, I can explain it ;-), integration tests, unit test coverage, checkstyle, findbugs, and other checks.

Those task of course should be performed before every commit to SCM, or the continuous integration build will fail, if there are some violations.

What I wanted, was one more profile - qbuild - which will disable those tasks if active. The reason why I didn't want to include it in already existing qclean profile was that sometimes developers want to run build with qclean profile, but also wants to have all those checks enabled.

Switching off the unit tests is simple.

...
<profile>

<id>qbuild</id>
<properties>

<maven.test.skip>true</maven.test.skip>

</properties>

<profile>
...

It will neither compile them.

To switch off other stuff I had to know ids of the executions used to configure plugins within the build process definition.

...
<plugin>

<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<executions>

<id>verify-coverage</id>
<phase>verify</phase>
<goals>

<goal>check</goal>

</goals>

</execution>

</executions>

</plugin>
<plugin>

<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<executions>

<id>verify-checkstyle</id>
<phase>verify</phase>
<goals>

<goal>checkstyle</goal>

</goals>

</execution>

</executions>

</plugin>
...

The snapshot shows us example of such existing configurations from our POM file.

Then I disabled them in qbuild profile simply hooking to non existing build phase - noPhase.

...
<build>

<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<executions>

<id>verify-coverage</id>
<phase>noPhase</phase>

</execution>

</executions>

</plugin>
<plugin>

<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<executions>

<id>verify-checkstyle</id>
<phase>noPhase</phase>

</execution>

</executions>

</plugin>

</plugins>

</build>
...

The developers then were able to run build four times faster and without any unused build artifacts.

mvn clean install -Pqclean,qbuild

They were all very satisfied with the result of the tuning and the "minus" never appeared again in the further retrospectives !!!

Refactor singleton out of your code

2007-08-01T20:40:00.002+02:00

There are a lot of posts about that the Singleton design pattern [GoF] is not pattern but anti-pattern (eg. SINGLETON - the anti-pattern!). Some posts also proposes an alternatives for Singleton design pattern or solutions for already existing Singleton design pattern problem (eg. Patterns I Hate #1: Singleton). But I didn't found any post which describes solution for big project with bunch of singleton class usage without big effort of rewriting many places by hand.

Because I was in situation that I should refactor a big project containing bunch of Singletons, I was thinking about some automatic way how to remove them. I found a solution using several refactorings. After bellow described sequence of steps you can replace Singleton design pattern by Registry design pattern [P of EAA] which solves all crucial problems of a Singleton, like testing possibility, singleton subclassing, ....

Lets have simple singleton class which we want to refactor to access it's instance not as a singleton but using some registry.

public class SingletonClass {

private static final SingletonClass INSTANCE = new SingletonClass();

public static SingletonClass getInstance() {

return INSTANCE;

}

private SingletonClass() {

super();

}

public void voidMethod(String param) {

System.out.println("Doing something in method returning void");

}

public Object objectMethod(String param) {

System.out.println("Doing something in method returning Object");
return new String("This is an String");

}

and also lets have very simple client class which access the singleton

public class Client {

public void clientMethod() {

SingletonClass.getInstance().voidMethod("param");
Object object = SingletonClass.getInstance().objectMethod("param");

}

At the first I create simple interface for Registry with two simple methods - getter and setter for instance of class which is currently implemented as singleton (SingletonClass in my example). There are also other variants of Registry implementation. For example you can have one generic methods getService and setService with some selector as a parameter (String ID, service class, ...). But I choosed the simplest one.

public interface ISingletonRegistry {
SingletonClass getSingletonClass();
void setSingletonClass(SingletonClass singleton);
}
Make SingletonClass's constructor public
Then implement the interface as a simplest Registry design pattern implementation - SingletonRegistry.

public class SingletonRegistry implements ISingletonRegistry {

private static final SingletonRegistry INSTANCE = new SingletonRegistry();

private SingletonClass singleton;

public static ISingletonRegistry getInstance() {
return INSTANCE;
}

private SingletonRegistry() {
//this is the reason why the constructor should be public
singleton = new SingletonClass();
}

public SingletonClass getSingletonClass() {
return singleton;
}

public void setSingletonClass(SingletonClass singleton) {
this.singleton = singleton;
}
}

You can see that I implemented it as singleton (of course using practise described in one of my previous posts - Singleton pattern implementation in 4 steps :-). I know that a lot of readers will have animadversions that I am creating new singleton when I want to remove other singletons. But realize that singleton is not evil every time. Also Service Locator design pattern uses singleton to access ServiceLocator instance.
You can also see that constructor of the class initializes reference to SingletonClass instance. Of course this is just possibility. You can use some mechanism of configuration of the singleton instances. But note that it should be performed before you use SingletonRegistry by some client for the first time by. Anyway this implicit setup can stay here because you are able to set different instance of SingletonClass using setSingletonClass method.
Change getInstance method of SingletonClass to get instance from SingletonRegistry.

public static SingletonClass getInstance() {
return SingletonRegistry.getInstance().getSingletonClass();
}
Over getInstance method of SingletonClass perform Inline... refactoring (Alt+Shift+I), mark 'Delete method declaration' chekbox and click OK. The method getInstance from SingletonClass disappears and all it's client classes uses SingletonRegistry to access SingletonClass.

public class Client {
public void clientMethod() {
SingletonRegistry.getInstance() .getSingletonClass().voidMethod("param");
Object object = SingletonRegistry.getInstance() .getSingletonClass().objectMethod("param");
}
}
Delete INSTANCE constant from SingletonClass class.
As a cherry on top of a cake you can Extract Interface (Alt+Shift+T, E) from SingletonClass. Write just new interface name, select all methods from SingletonClass which you can extract into new interface and press OK. All references to SingletonClass will be refactored to references to your newly created interface.

And that is all. Now you have SingletonClass with totally same functionality but you are able to mock it, extend or replace by different implementation setting up your instance of SingletonClass to SingletonRegistry.

References:

My first real Java application

2007-07-02T19:05:00.006+02:00

I just found very funny Java application in my archive. It was year 1999 and I was studying college in Ostrava. We had strange theme called Java Technologies. Following applet was my semestral project for the theme. It is very simple, multi threaded (!!!), simulation of cross way. A colored squares are cars and yellow squares in their corners are flashers. The cars follows traffic lights and right-hand rule. Look and enjoy :-)

Do you want sources? Download crossway.zip

What I am missing on del.icio.us

2007-07-02T10:52:00.000+02:00

As member of a team I was looking for some solution which will enable us to share bookmarks across the team. Because I have some good experiences with del.icio.us bookmark manager and it also fulfil basic conditions, my focus was clear from the beginning.

The basic conditions were:

bookmarks should be shared among all team members
because the members are traveling and then they are out of corporate network we need access to bookmarks from anywhere
posting and using bookmark should be as simple as possible

Then I identified additional conditions:

some members uses IE, some members uses Mozilla
at least some members of team (eg. me :-) has it's own personal account on del.icio.us and they want to use bookmarks from both of them at a time.

My idea was that every member will have it's own account and will be logged in using browser extension. There will be also team specific account (TA) and every member will add it into it's network. At the beginning I thought that it will enable me to see my and TA's bookmarks on one heap and I will also be able to add for:ta tag from a suggestions when posting new team bookmark. But:

network member's bookmarks are not merged between user's bookmarks (I mean merged just for view)
for:ta tag suggestion is present, but it adds bookmarks just into "links for you" section of TA (BTW at the moment it is impossible to remove proposed bookmark - see del.icio.us FAQ)

Those two problems cause that:

we can't use personal and TA account together with browser extension
somebody should be responsible for accepting proposed bookmarks of TA from "links for you"

But I found workarounds (just for Mozilla which implements Live Bookmarks). Both bookmark sets (TA's bookmarks and TA's "links for you" bookmarks) provides RSS feeds. So every member will have two Live Bookmarks through which he can access both groups of shared bookmarks. But the solution still have several imperfections:

you cant use browser extension to access team bookmarks
you can't search bookmarks by tags (but you can create more Live Bookmarks for each or some tags)
when somebody save bookmarks from "links for you" to main group the bookmark is present in both groups

Maybe we are trying to get pears from apple tree. I also know that del.icio.us want to stay as simple as possible and they do it very well, but still I have some proposals to del.icio.us team:

improve "links for you" functionality

user can define list of trusted users and their proposed bookmarks will be added directly (with some flag) into "my bookmarks"
provide possibility to delete bookmarks from "links from you" group (in progress - see del.icio.us FAQ)

improve "network" functionality

provide possiblity to configure you network member to have his bookmarks merged (just for view) between your bookmarks

provide RSS feeds containing users tags

Anyway, at the moment we use it with workarounds and we are happy at least with this solutions, but it can be better.

Intercepting Filter Design Pattern

2007-06-14T20:00:00.000+02:00

Intercepting Filter is classified as an enterprise or J2EE design pattern. Intercepting Filter design pattern is typically used in cases where some messages should be processed but the target processing should be wrapped into chain of arbitrary pre/post processings as are authorization checks, message resources transformations (eg. translation, escaping), ...

Intercepting Filter design pattern is built from five basic components:

FilterManager
FilterChain
IFilter interface
Context
ITarget interface

FilterManager - FilterManager class is a boundary class of Intercepting Filter design pattern. It means that a client uses it as an interface to process some target operation (represented by ITarget interface) in a specific context (represented by abstract interface Context). FilterManager is then typically responsible for FilterChain creation, it's setup (concrete filter set configuration, target propagation, ...) and finally it's execution.

FilterChain - FilterChain class is basicaly data object carrying list of filters and the target, with just very simple logic of filter queue.

IFilter - IFilter interface has just one method. The method is typically implemented as captured on following diagram. It makes some pre processing, and if everything is OK than it calls FilterChain to process to following filter.

If something is wrong (some data are wrong or missing in Context) then it can return or throw exception.

When the target is reached through chain of filters, it is processed, and the thread is returning back same way, so it can make some post processing in every filter, during which it can also make some changes to Context (eg. response translation, escaping, ...).

ITarget - ITarget interface should be implemented as message receiver or it's wrapper.

Context - Context class in this presentation of Intercepting Filter design pattern should be considered as very abstract. It means that it can be totally different type of class or it can be set of classes or it can be some Map, etc. For example it can be ServletRequest together with ServletResponse when we are talking about Intercepting Filter design pattern implementation defined by Sun - Core J2EE Patterns - Intercepting Filter.

Implementation of Intercepting Filter design pattern is pretty simple. The most complex component can be FilterManager which can implement some sophisticated way of configuration which defines mapping filters to context and their configuration. But whenever it is implemented it is very simple to extend message processing by new features or change its behavior.

Related patterns:

Decorator [GoF]

Does not have FilterChain - the order of filters is defined wrapping one filter into another.

Template Method [GoF]

Very simple version of Intercepting Filter design pattern implementing Filter Chain (template method) and IFilter implementations (methods called from template method) in one class.

Chain of Responsibility [GoF]

Does not have FilterChain - the order of "interceptors" is defined as chain of successors.

Interceptor [POSA2]

Similar also to Chain of Responsibility but the chain of successors is not "hard coded" but propagated to "interceptor" as parameter of the method call.

Pipes and Filters [POSA1]

The Pipes and Filters design pattern is oriented basically for stream related use cases. It also doesn't provide implicit possibility of post processing.

References:

Singleton pattern implementation in 4 steps

2007-05-30T15:41:00.001+02:00

The article describes very simple, fast and efficient way how to create class implementing Singleton design pattern [GoF]. Everything is done in 4 steps using standard features of Eclipse IDE (refactoring and source code manipulation). But it is of course applicable for every IDE providing similar refactoring functionality.

Steps:

Create new empty 'SingletonClass' class - use New > Class action from popup menu over package or Ctrl+N shortcut. In the dialog just ensure that the source folder and project are specified and define the name of new class. Press Enter or click OK.

From the body of the class invoke Source > Generate Constructors from Superclass... action, of course using Alt+Shift+S, C shortcut. In the following diaglog just press Enter or click OK. (This step can be done together with prvious step checking 'Constructors from superclass' option in the 'New Java Class' dialog).

From the constructor signature definition (constructor name) invoke Refactor > Introduce Factory... action (using Alt+Shift+T, O shortcut). In an 'Introduce Factory' dialog change 'Factory method name' value to the name of singleton access method (eg. getInstance). Press Enter or click OK.

Select 'new SingletonClass()' from 'getInstance()' method and invoke Refactor > Extract Constant... action (Alt+ Shift + T, A).

In a dialog you can change 'Constant name' and visibility but typically the default value is OK. So just press Enter or click OK and voila ....

The result is full-value Singleton design pattern implementation created in 4 very simple steps.

Code Content Assistant Efficiency

2007-05-16T18:15:00.000+02:00

Purpose of this article is to evaluate efficiency of content assistant functionality provided by Eclipse with focus on type name completion. The results will be of course applicable not just for type names but also for field and method names.

Content assistant functionality
Eclipse content assistant is simply suggesting code completions while writing code considering to concrete context - type names, field names, method names, ... It also provides excellent feature called "camel case matches" (NPE matches to NullPointerException) which is also object of my evaluation.

Content assistant efficiency definition
Content assistant efficiency is number, which says how many times more you will get considering to what you spend. When we are talking about type names then this number is defined as typeNameLength/price. Where typeNameLength is simply the number of letters in the type name. The price is lowest number of letters and key presses (arrow down in suggestion list + Enter) necessary to be pressed to generate type name using content assistant.

For example class NullPointerException has 20 letters but using content assistant we can generate the name by 5 key presses (typing "NPE" + one down arrow press, because NoPermissionException is also in the list + Enter). It means that in case of NullPointerException class the efficiency of content assistant is 20/5=4. Basically it means that the result is four time bigger than the effort to generate it.

The example is talking just about one class. But it can be also applicable for set of classes (eg. jar) simply counting their lengths together and dividing by counted prices.

How I evaluated
Of course the biggest challenge was to find lowest number of letters and key presses necessary to be pressed to generate the type name. For this purpose I wrote simple application which for 3905 patterns (maximum 5 words with maximum 5 letters) distributes classes by name into groups where classes in one group matches the pattern with same result. After alphabetical sort of the classes in the group it counts price for every class as length of group id + position in the sorted list (first = 0) + 1 (Enter). The lowest price for a class over all groups is the number what we are looking for.

Example:
Pattern: ^([A-Z])[^A-Z]([A-Z])[^A-Z]([A-Z])[^A-Z].*$
Group Id: NPE
Sorted classes: NoPermissionException, NullPointerException
Prices: 3+0+1=4, 3+1+1=5

Pattern: ^([A-Z][^A-Z]{1})[^A-Z]([A-Z])[^A-Z]([A-Z])[^A-Z].*$
Group Id: NoPE
Sorted classes: NoPermissionException
Prices: 4+0+1=5

Lower price for NoPermissionException class is in the first group so it is the lowest price.

For evaluation I used classes from seven jars (rt.jar, catalina.jar, xercesImpl.jar, derby.jar, xalan.jar, axis.jar, junit.jar) with total number of 12707 classes.

Results
As we can see in the table below the content assistant efficiency counted for all seven jars together is 3.20.

It means that using content assistant is about three times more effectively than typing everything. The second benefit of content assistant usage is higher quality of code, because it is harder to make mistakes.

All this can be true just in case that the user uses content assistant effectively. About how to really use it and what are the best practises see TBD - article about effective useage is in progress.

Attachment
On chart in the image bellow we can see a histograms of class name length and class price. It shows number of classes with concrete length (blue) and number of classes for price (red). We can see the efficiency of the content assistant as a peek of red bars in comparison with blue "hill".

Don't write code - use Eclipse

2007-04-20T09:13:00.000+02:00

Eclipse IDE provides multiple functionalities which can help you when you are coding. The functionalities what I am talking about are typically very small and easy to use actions such as Content Assistant, Quick Fix, Refactoring and other code manipulations. Usage of those features can improve speed and quality of your coding.

In this article and also in the other articles under label coding tips I would like to describe usage of those features in the way where you will write as little code as possible and the rest of code will be generated by Eclipse.

Let start with very simple example - assignment of new instance of StringBuffer class to the local variable or to the field.

So what we want to have at the and will be something like:

StringBuffer buffer = new StringBuffer();

Steps:

write keyword 'new'

write 'S' and push Ctrl+Space (shortcut for Content Assistant)
you can see list of classes, interfaces and templates starting with 'S'

you can specify more accurately the name of requested class writing its following letters 'tr' or using Camel feature - just write 'B' as a Buffer. Then navigate selection to StringBuffer class.

press Enter. The name of the class is generated.

write '(' as an open parenthesis (the closing parenthesis is generated automatically) for constructor parameters and and push Ctrl+Space again.

select constructor you want to use and press Enter

write down the parameter values for the constructor, press End button, add semicolon ';' and save (off course using Ctr+S shortcut).

press Ctrl+1 (cursor should be somewhere on the line code). It is shortcut for Quick Fix proposal.

choose one of the proposals and press Enter. The Eclipse then generates all necessary code to have it compile able.
For more advanced usage you can use direct shortcuts for proposed actions (you can see them in the proposal) - Ctrl+2, L for assignment to local variable and Ctrl+2, F for assignment to field.
It also provides you possibility to give a name of created variable or field.

When you are sure about the name of the variable press Enter. It will offer you to specify type of variable in more detail. Of course the proposed types are super types of the type you instantiated.

When you will choose the right type press siply Enter again and once more for finalization.

Now we have what we wanted (55 characters) using 28 key presses.

Interceptor Design Pattern

2007-04-11T16:01:00.000+02:00

Usage of Interceptor Design Pattern [POSA2] can solve many problems in daily development. It is very easy to:

monitor what is the application doing inside
provide possibility to change some of application's behavior
provide possibility to extend application with new features

implementer of the extension need not to know rest of the application
implementer of the extension need not change existing code
new extensions can or can not affect current system

Interceptor design pattern is built from three basic components:

IInterceptor interface
Dispatcher
Context

IInterceptor - IInterceptor interface can define arbitrary number of operations which are executed by Dispatcher in specific cases (at the beginning of the request processing or at the end of the request processing). There is typically just one parameter for every method - Context - carrying input and output data. The interface is then implemented by specific interceptors (logging, authorization, transaction, ...).

Dispatcher - Dispatcher class typically implements IInterceptor interface. There are many ways how the dispatching can be implemented. For example the interceptors called by Dispatcher can be prioritized or exception thrown from interceptor can be handled as interception termination. The Dispatcher is the class which is called from "intercepting point" to perform intercepting.

Context - Context is typically simple data object used to carry data for interceptors or to keep results of intercepting (eg. transaction id generated by TransactionInterceptor). There can be different Context implementations not only for different IInterceptor interfaces but also for different methods of IInterceptor interface.

As mentioned in description of basic elements the Interceptor pattern has many degree of freedom:

IInterceptor

one/many methods
one/many Context types for many methods
result propagation to Dispatcher, intercepting point or following interceptors

Context - Dispatcher can decide about next step by status of the Context
Exception - Dispatcher can handle different exceptions different way

Dispatcher

implements/does not implement IInterceptor interface
dispatching mechanism (prioritized, random, based on context, ...)

Implementation of Interceptor Pattern is not the easiest work. But whenever it is implemented it is very simple to extend application by new features or change its behavior.

Related patterns:

Template Method [GoF]

Very simple version of Interceptor Pattern implementing Dispatcher (template method) and IInterceptor implementations (methods called from template method) in one class.

Chain of Responsibility [GoF]

Does not have Dispatcher - the order of "interceptors" is defined as chain of successors.

Intercepting Filter

Similar also to Chain of Responsibility but the chain of successors is not "hard coded" but propagated to "interceptor" as parameter of the method call.

References: