Just build it

Fun with git remote branches

2012-10-01T15:28:00.001+02:00

I am currently using the build-per-branch-"Plugin" to integrate my git-flow-enabled projects with Jenkins continuous integration.

After creating and removing a few feature/hotfix/release branches I noted that some branches that should have been deleted kept returning in my list of Jenkins jobs.

So, nobody likes zombie jobs, right?

I started digging into the build-per-branch plugin. It uses "git ls-remote --heads " to retrieve the list of remote branches.
And behold: my branches that "git branch -r" would no longer show were still listed there!

I did delete the branches using "git branch -r -d " and it told me that the remote branch was removed.

There is the other way for deleting remote branches: Pushing an empty repository to a remote branch with "git push origin :" (I confess that I often have used "git branch -r -d" because I could remember it more easily).

But as it turns out the only surefire way to get rid of a remote repository seems to be to use "git push origin :", becuase after this "git ls-remote" no longer lists the reference for the branch's head.

Update: I asked about this behavior ont the git mailing list:

Hi,

I came across an issue with deleting remote branches: When deleting a

branch using "git branch r d " the branch is deleted and no

longer shown by "git branch r".

"git ls-remote --heads" on the other hand still lists the ref for the last

branch commit.

If I delete the branch using "git push origin :" the branch is

removed completely (i.e. git ls-remote no longer returns a ref).

Is this a bug or is this behavior intentional? (I am currently using git

1.7.12).

Here is the response I got from the git mailing list:

All of the above are expected and working as designed. Remote

tracking branches are local _copies_ of what you have over there at

the remote repository. The latter is the authoritative version, and

you asked "ls-remote" to go over the network to view them.

If you are trying to remove the branches you show to others and

yourself who are observing that remote repository from your local

workstation, you can first "git push origin :" to remove it

in the remote repository. You can run your next fetch with "--prune"

to get rid of remotes/origin/, I think.

I think it clarifies the behavior. That thing that "git branch -r -d" deletes is only the local reference of the remote brach (the rempote tracking branch), i.e. the git branch command works fully locally.

Only git push will really change *anything* on a remote repository.

The perils of gradle wrapper and CI

2012-08-21T10:35:00.002+02:00

If you are building yours software with Gradle, using gradle-wrapper is a great thing to ensure that your build is able to run out of the box after checking out just by running "gradlew" in your project directory.

Just today I got a nice message from Hans Dockter (gradleware Germany), telling me that there have been a lot of downloads of gradle-1.0 (about 2100) in the course of one or two months, which caused some notable trafic on gradle.org.

2100 downloads check in at about 64 GB of data.

So, what was the cause? As we don't have an infinite number of monkeys using gradle here, the primary suspect is jenkins.
A good number of jobs (about 20) use Gradle for building, and each job is configured to use Gradle wrapper, because it is convenient and ensures the build runs exactly with the right Gradle version.

So it seems like every job was happily downloading Gradle over and over again, which puts some load on gradle.org (bad) and slows down the build (even worse).

The solution is simple: Just installGradle centrally on your Jenkins and configure the Jenkins jobs to use the centrally provided Gradle instance. I am dong this for my jobs right now and I expect them to behave better afterwards.

I am sure that gradle.org and the speed of your build jobs will thank you for it.

Encoding your passwords in Maven

2010-10-19T11:48:00.003+02:00

If you dislike storing your passwords in clear just like me then Maven 2.1.x or 3.x can help you out.

At the Maven site the documentation is somwhat hidden, so I'll just describe the process in short and put up a link to the relevant documentation.

How it works

Maven employs a two-step approach to securing your passwords.

The first step to security is to define a master password that is stored separateley from your normal Maven settings.xml file.

This is done by invoking Maven with the following commandline:

mvn -emp [master password]

This yields an ecrypted password that looks like:

{8XfdxChqWiIHuQtHizPd+kawVIzRF1WWmsKmGUbMcLw=}

Put the encrypted password in a file named settings-security.xml in your home folder ($HOME/.m2/settings-security.xml)

<settingsSecurity>
  <master>{8XfdxChqWiIHuQtHizPd+kawVIzRF1WWmsKmGUbMcLw=}</master>
</settingsSecurity>

After defining your master password you can simply encode your server passwords with

mvn -ep [server password]

and put the resulting encoded password in the password-Tag of your server-definitions in settings.xml.

If you are particularly concerned about your passwords you can also store your password on removable media.

This approach is described in detail in the Mini-HowTo at the Apache Maven site (see below for link).

For more information have a look at the original mini HowTo: http://maven.apache.org/guides/mini/guide-encryption.html

Best practices for multi-module project organisation

2010-10-14T10:21:00.001+02:00

When setting up a maven project consisting of multiple modules there are multiple ways to define the project dependencies.

Use one top level directory containing only module subfolders. The parent/module-POM resides in it's own module subdirectory (e.g. project-parent)
Use one top level directory containing module subfolders and the parent/module-POM.
Use one top level directory containing module subfolders and a POM file defining the modules of the project. The parent-POM resides in a submodule folder.

These are the three most common approaches to organize your project structure, each of which has it's own advantages and disadvantages.

Approach 1

Plus Side:

Works well with IDEs like Eclipse

Minus Side:

Hard to integrate with continous build systems because of the colocation of the definition of modules and the parent pom configuration. This makes standalone watchdog builds of modules almost impossible because the whole source tree is necessary to build a module.

Approach 2

Plus Side:

Only one single top level POM to maintain.

Minus Side:

Hard to integrate with continous build systems because of the colocation of the definition of modules and the parent pom configuration. This makes standalone watchdog builds of modules almost impossible because the whole source tree is necessary to build a module.
Top level POM is hard to integrate into project structures (e.g. eclipse) as it lies in a parent directory of the modules' project definitions.

Approach 3

Plus Side:

Parent POM is easily defineable as project in your IDE
Module definitions are separated from project configuration enabling the project to have multiple module set definition, e.g. have a definition to build only a subset of all modules - for impatient developers(like myself ;-)).
Good integration with continuous integration systems because module definitions do not interfere with standalone module builds.

Minus Side:

POMs containing module definitions may be difficult to integrate in IDE.

Conclusion

In my experience the third approach works best for most larger modules. Especially the compatibility with both IDEs and continuous build systems like Hudson make your work a lot easier.

So what does the recommended structure look like? Quite simple. You simply have a main directory that contains a top-level POM file and subdirectories for the parent module and submodules.

A sample minimal project can be downloaded from github (I guess I could make a Maven artifact for it but I just do not have the time right now).

Precompiling JSPs with Maven - Part 2

2009-09-14T14:58:00.005+02:00

If you use war dependencies for building your web applications the approach described in part 1 will not work for your application because the JSPC-Plugin by default works on your local JSP files and is executed during the lifecycle phase "process-classes".

Here is how to deal with this added requirement.

Moreover if you use tag files in your JSPs (especially if the Tag pages come from the included web application) the JSP parser will fail when trying to compile the JSP files because it needs access to the sources of all JSP pages referenced.

The solution to this problems exploits the fact that the war plugin checks for modifications before every step it takes.

The steps for the solution described here are acutally quite simple:

Configure an extra Execution of the WAR-Plugin to run during the lifecycle phase "process-classes" to exectue the goal "exploded".
Run the jspc plugin during the same execution but after the war plugin has run but configure it to operate on the exploded war archive.
Run the war plugin again with goal "war" during lifecycle phase "package".

Please see the next code snippet for the solution described above:

<plugin> 
  <groupId>org.codehaus.mojo.jspc</groupId> 
  <artifactId>jspc-maven-plugin</artifactId> 
  <executions> 
    <execution> 
      <id>precompileJsps</id> 
      <goals> 
        <goal>compile</goal> 
      </goals> 
      <phase>package</phase> 
      <configuration> 
        <sources> 
          <directory>${basedir}/target/${project.build.finalName}</directory> 
          <includes> 
            <include>**/*.jspx</include> 
          </includes> 
        </sources> 
        <source>1.5</source> 
        <target>1.5</target> 
        <verbose>1</verbose> 
        <inputWebXml>${basedir}/target/${project.build.finalName}/WEB-INF/web.xml</inputWebXml> 
      </configuration> 
    </execution> 
  </executions> 
  <dependencies> 
    <!-- Please see Part 1 for jspc dependencies -->
  </dependencies> 
</plugin>

The configuration of the war plugin looks like this:

<plugin> 
  <artifactId>maven-war-plugin</artifactId> 
  <executions> 
    <execution> 
      <id>precompileJsps</id> 
      <goals> 
        <goal>exploded</goal> 
      </goals> 
      <phase>process-classes</phase> 
      <configuration> 
        <warSourceDirectory>${basedir}/WebContent</warSourceDirectory> 
        <overlays> 
          <overlay> 
            <groupId>com.acme</groupId> 
            <artifactId>base-webapp</artifactId> 
          </overlay> 
        </overlays> 
      </configuration> 
    </execution> 
    <execution> 
      <goals> 
        <goal>war</goal> 
      </goals> 
      <configuration> 
        <warSourceDirectory>${basedir}/target/${project.build.finalName}</warSourceDirectory> 
        <overlays> 
          <overlay> 
            <groupId>com.acme</groupId> 
            <artifactId>base-webapp</artifactId> 
          </overlay> 
        </overlays> 
      </configuration> 
    </execution> 
  </executions> 
</plugin>

Important: The war plugin has to be configured before the war plugin in the POM because Maven executes the plugins within an execution in the order they are declared in the POM.

An that's all there is to precomipling JSPs in a multi project implementation using war dependencies.

Precompiling JSP Pages with Maven

2009-09-13T22:18:00.008+02:00

Recently I encountered the requirement to precompile JSP pages in a Maven webapp project using the following environment:

Java 5
Tomcat 5.x
Jsp pages with XML Syntax (jspx)

Solving the problem is not too difficult if you keep some things in mind.

Precompiling JSP pages has a number of advantages (and some disadvantages too):

The precompiled JSP pages are delivered as servlets which means that the application runs at full speed immediately after deploying it. Compiling JSP pages on demand may not matter with an application that uses a few JSP pages but it definitely matters when having 300+ JSP pages.
It is possible to tweak the Java Compiler for JSP pages to use optimisations and to leave out debug informations which can speed up the application a little further.
The Precompiler finds every error in every JSP (even in seldomly used pages/error pages)

There are some disadvantages which should be remembered as well:

The used JSP compiler version has to match the container version as close as possible. So if you update your tomcat installation you need to recompile your webapp or it may fail in a worst case scenario.
Turning on javac optimisations and turning off debug information makes it much harder to track down errors. So you'd better have good loggin in your application to get an idea why errors happened.
Not all servlet containers are supported by the used plugin. Currently it should work without problems with Jetty and Tomcat.

So, what is needed to include JSP precompilation in your Maven web project?

A recent Maven version (2+). I currently use Maven 2.2.1.
The Maven JspC Plugin from codehaus (currently 2.0-alpha-3)

The first step is to declare the plugins and configure them as with every other maven plugin. The jspc plugin documentation could use some improvement and updating. It took me some time to gather the required information to get it up and running. The results of this exptedition are found in the next code snippet:

<plugin>
  <groupid>org.codehaus.mojo.jspc</groupid>
  <artifactid>jspc-maven-plugin</artifactid>
  <executions>
    <execution>
      <id>precompileJsp</id>
      <goals>
        <goal>compile</goal>
      <goals>
      <configuration>
        <sources>
          <directory>${basedir}/WebContent</directory>
          <includes>
            <include>**/*.jspx</include>
          </includes>
        </sources>
        <source></source>1.5
        <target>1.5</target>
      </configuration>
    <execution>
  </execution>
  <dependencies>
    <dependency> 
      <groupid>org.codehaus.mojo.jspc</groupid> 
      <artifactid>jspc-compiler-tomcat5</artifactid> 
      <version>2.0-alpha-3</version> 
    </dependency> 
    <dependency> 
      <groupid>org.slf4j</groupid> 
      <artifactid>slf4j-log4j12</artifactid> 
      <version>1.5.6</version> 
    </dependency> 
    <dependency> 
      <groupid>org.slf4j</groupid> 
      <artifactid>jcl-over-slf4j</artifactid> 
      <version>1.5.6</version> 
    </dependency> 
  </dependencies>
</plugin>

There are some obvious and some, well, less obvious settings to configure.
The biggest pain with the jspc plugin is the incomplete declaration of dependencies of the jsp compiler that is actuall used.
For some reason you have to include the slf4j jars in the dependencies or the jspc plugin refuses to work with some misleading error.

The specific version of Tomcats JSP compiler implementation defaults to 5.5.23 for the jspc-plugin 2.0-alpha-3.
If you need to use another version of thew Jasper compiler, you have to declare the following dependencies for the jspc-maven-plugin:

<dependency>
  <groupid>tomcat</groupid>
  <artifactid>jasper-compiler</artifactid>
  <version>5.5.27</version>
</dependency>
<dependency>
  <groupid>tomcat</groupid>
  <artifactid>jasper-compiler-jdt</artifactid>
  <version>5.5.27</version>
</dependency>
<dependency>
  <groupid>tomcat</groupid>
  <artifactid>jasper-runtime</artifactid>
  <version>5.5.27</version>
</dependency>

With this configuration your WAR should grow by a few bytes to a few MB. If you are using Servlet 2.4+ you do not need to change your web.xml to add the servlet mappings.

If you are stuck with Servlet API 2.3 you'll have to paste the generated web.xml fragement into your file.

In the next post I will extend JSP precompilation to projects using WAR dependencies, which poses some additional implications and problems.