Dimitar's Blog

Common scenarios for Java application packaging

2011-09-14T04:59:00.000+09:00

It all started from an overgrown answer to a question on StackOverflow. I realized that I haven't seen an overview of best practices and patterns for packaging of Java applications. There are the platform specific packaging formats, but they have more to deal with installation, rather than the layout of the packaged application. Below are a few common scenarios I've come across over the last few years.

Note: I've written another, somewhat related article on SO, about common strategies for managing configuration files in Java

A single uberjar with MAIN-CLASS specified in the manifest is convenient distribution format for simple tool applications that are intended to be run on the command line. You can copy it around and plop it somewhere on the path or in your home directory. If you set the PATHEXT and file associations on Windows, you can run them as if they were native executables. One thing to watch out for is when you bundle libraries with custom entries in the manifest (you might need to manually merge them), when multiple jar's contain files with the same paths but different content (i.e. Spring namespace handlers, ServiceLoader implementations, LDAP SPI providers, etc.)

If for some reason you are bundling a library and all dependencies as a single jar (bad idea), you better take care to relocate the packages (for example, see how the JDK has relocated Xerces and BCEL under com.sun.org...) Not doing so can cause subtle problems to your clients which are extremely annoying, to say the least (*caugh*Weblogic*caugh*). On the other hand, relocating classes causes problem with smart usages of reflection relying on string concatenation. Also, you will most likely need to filter all XML and properties files, replacing the packages with the relocated names. All files under META-INF/services will need to be renamed too. All in all, bundling dependencies in a library jar causes far more problems than it solves.

When your application has more than a few options, you may find you want a config file. One approach I like is to package a template config in my app and look in a predefined location for override (i.e. in ~/.myapp) - you may provide instructions how to extract the template config and place it under the users home or you can have the app do it automatically on first usage. Which one you choose depends on your estimate how often an user will need to tweak the configs (we don't want to pollute the user home unnecessarily). A single directory app is usually installed in your user home directory.

If your application has dependencies on external services or special environment, you will need a startup script (shell or batch file) - in this case, the simplest deployment layout is single directory with uberjar and startup script. As you now have a directory, it is a good idea to add a README file and you may as well put the configs there. There are a number of variations on this theme. Usually, you want to add this directory to the app classpath as well, so any file there may be looked up as resource - that provides convenient way to override things like Spring descriptors, patch single classes, etc. These are usually installed in your home, /usr/local, or sometimes under opt.

If your application bundles many dependencies, has complex configuration, startup scripts for multiple platforms etc. then you will go with full blown deployment structure along these lines (used for big application deployed on multiple platforms):

/bin - startup scripts, rc3.d scripts, environment setup files, registry entries, etc.
/config - configuration files for app, should be possible to relocate to /etc by specifying env variable.
/logs - logs produced by app, safe to delete, config should allow us to move it to /var
/cache - temporary data produced by app, safe to delete, config should allow us to move it to /var
/data - data produced by application, config should allow us to move it to /etc
/lib - main application jar with all dependencies
/lib/win86 - platform specific JNI libs for all supported platforms, selected by the start script, passed as -Djava.library.path=...
/lib/linux86
/lib/linux64
/lib/...

The main application JAR should have a Main-Class attribute starting the app and Class-Path attribute specifying the ordering of the dependencies (all located in the same dir). The Class-Path should start with the ../config path that will be used to load configurations and patch classes. This allows to keep the startup scripts simple.

The last layout allows you to run the application from your home, work all from /opt, or install in read-only /opt and put the configs and writable dirs under /etc and /var like a decent Unix application. The last option also allows you to run a few instances with separate configs and data from the same binaries.

Groovy Classpath Scanner

2010-10-22T22:46:00.005+09:00

I just wanted a simple classpath scanner in Groovy - no library, no extra jars, no callback interfaces. I couldn't find any, so I wrote one. I'm posting it here and hopefully it would be useful to somebody. It is licensed under the established and permissive MIT License (if it precludes you from using it, let me know).

import java.util.zip.ZipFile
 
/**
 * <pre><code>
 * def cps = new GroovyClasspathScanner(packagePrefix: 'com.company.application')
 * cps.scanClasspathRoots(classLoader1) // optional
 * cps.scanClasspathRoots(classLoader2) // optional
 * ...
 * List<Class> classes = cps.scanClasses { Class it ->
 *    Event.isAssignableFrom(it)   ||
 *    Command.isAssignableFrom(it) ||
 *    it.isAnnotationPresent(MessageDescriptor)
 * }
 * </code></pre>
 */
class GroovyClasspathScanner {
  String packagePrefix = ''
  List<File> classpathRoots
 
  @SuppressWarnings("GroovyAssignabilityCheck")
  List<File> scanClasspathRoots(ClassLoader classLoader) {
    if (!classLoader) classLoader = getClass().classLoader
 
    def prefixPath = packagePrefix.replace((char) '.', (char) '/') + '/'
 
    def List<URL> urls = []
    for (URLClassLoader cl = classLoader; cl; cl = cl.parent) {
      urls.addAll cl.URLs
    }
 
    return urls
      .each { assert it.protocol == 'file' }
      .collect { new File(it.path) }
      .each { File it -> if (it.isFile()) assert it.name =~ /.*\.(?:jar|zip)$/ }
      .findAll { File it ->
        (it.isDirectory() && new File(it, prefixPath).exists()) ||
        (it.isFile() && new ZipFile(it).entries().find { it.name == prefixPath})
      }
  }
 
  List<String> scanClassNames() {
    if (!classpathRoots) classpathRoots = scanClasspathRoots()
 
    def classNames = []
    def collect = { it, String pathProp ->
      def normalizedPath = it[pathProp].replaceAll('[\\\\/]', '.')
      def packageRegex = packagePrefix.replace('.', '\\.')
      def classRegex = "\\.($packageRegex\\..+)\\.class\$"
 
      def match = normalizedPath =~ classRegex
      if (match) classNames << match[0][1]
    }
 
    classpathRoots.each {
      if (it.isDirectory()) {
        it.eachFileRecurse             { collect it, 'canonicalPath' }
      } else {
        new ZipFile(it).entries().each { collect it, 'name' }
      }
    }
 
    return classNames
  }
 
  List<Class> scanClasses(Closure predicate = { true } ) {
    return scanClassNames()
            .collect { try { Class.forName it } catch(Throwable e) { println "$it -> $e" } }
            .findAll { it }
            .findAll { predicate(it) }
  }
}

Concurrent development

2010-09-18T21:57:00.008+09:00

Background Story

That certain project was in its 4th year and after a few successful deployments in the US/LATAM regions, now the company was trying to push it to EU and APAC. The original developers were all in New York and over the last few months have been frantically working to adapt the product to EU's requirements (Asia was supposed to follow.) The scope was defined as 'whatever it takes', the methodology was a mix between 'trial and error' and bullying the users to contort the requirements to fit the delivered functionality. At the time, the EU team had realized that they would better spend their time staying on top of the changes and making sure the end product meets a minimum standard, so they were not really doing much development.

Time went on, requirements grew, scope shrinked. The project slipped past two deadlines and finally the Asia managers decided they need to take things into their hands and hire a local development team to avoid the communication gap that plagued the EU rollout and regain some control over the schedule. It was first time to have more than one people touching the code the core team was structured in a way that each component has an owner and the owner could do whatever they want. If you have a problem or need a change - ask the owner. The problem was not only that we were in different geographical location, in inconvenient timezone, but we were working on the same code, implementing requirements specified by separate BA teams, chasing schedules devised by separate project-management teams, and it all eventually converged in a common program-steering committee. I could go on, but suffice to say it was quite a mess - the bottom line is that moving from centralized sequential to distributed concurrent development models impose huge burden and the best advice one can give you would be "don't do it!".

Probably the biggest issue was that many people in the core team, just refused to change their way of working in order to accommodate our existence. Every second morning the trunk would not compile, often changes were checked in that prevent servers from starting, our changes were overwritten routinely because somebody's local modifications conflicted and they were unwilling to merge - you name it, we have it. The management layer was protecting them as "due to the years of accumulated experience, the productivity of the core team was much higher that ours, and the productivity hit they would suffer by addressing our petty complaints can not be justified in business terms". Luckily, there were some sensible guys and gradually we got to improve this, still I consider it one of the biggest organizational faults that for a long time the management efforts were focused on suppressing our complaints, rather than backing our suggestions on fixing the environment.

As the first QA delivery was approaching and the trunk was not giving any signs of getting more stable, we tried to think what can we do to stabilize the codebase. Some people said we should branch, others were weary of the cost of merging. The EU team had branched few months ago and all EU implementation were done on the branch and eventually (read 'sometimes') merged to the trunk. When the product was released in EU, they ended up with the problem how do they merge to the trunk. From what I hear it had been a terrible experience, including a lot of functionality rewrites, introduced bugs and regressions.

Knowing the EU problems and knowing that on one hand the trunk was still changing rapidly, on the other hand our requirements were dependent on code that was supposed to be delivered by US, we decided to branch, but keep developing on the trunk. All merges would be in direction trunk-to-branch and this would save us from the dreaded criss-cross merge conflicts. Since most of our problems to that date were with work-in-progress checkins, which we eventually wanted, we decided that we can treat the branch as a stable release line and trunk as unstable bleeding-edge code.

Unstable trunk + Release branch

I was tacitly elected as 'merge-master' and quickly I found myself following the same routine:

Every morning I would pull a list of all the unmerged commits and review them in a text editor. Then would move each commit into one of these categories:

WANTED - changes that are required or prerequisites for implementing our business functionality. These should be always merged.
BLOCKED - changes that we DO NOT want. These should be always marked as merged (no actual merging, just mark, so they will not appear in the list next time).
IRRELEVANT - changes that won't hurt us, but we don't strictly need them. We were merging these in the initial stages as keeping the branch close to trunk makes merging easier, as we got closer to the release, we flipped the policy to improve the stability.

When I merge or mark as merged the WANTED/IRRELEVANT/BLOCKED groups, I would put the category as a first word int he commit message. This made it easier to pick out the changes that were done directly in the branch (which should be kept to minimum and if necesarry ported manually to trunk). I didn't bother separating the individual changes, since the branch was not meant as a merge-source - this was saving me some time. Overall it was taking between 1 and 3 hours a day.
There would be a number of changes that didn't make it to any of the categories. For these I was contacting the comitter and following up. Often it was work in progress, sometimes after clarification they would be categorized on the next day. Usually I would post this communication as a tagged blog-entry in our wiki. There was a page displaying all the entries tagged in this way.

I found out that sorting the changes first by user and then by date simplifies the review significantly. Turned out that TextPad macros can be a very powerful tool for things like this.

The release branch worked well for some time, until a major feature for the next release was implemented on the trunk. We blocked it and ever since then, every commit that touched this dreaded component had to be hand-merged. Often, merging an one-line change resulted in tens of conflicts, so we resorted to rolling back the file in question and manually porting the change. The worst thing is that we tested the trunk extensively, but the change in our release-branch received only cursory examination until it reached QA.

Furthermore, once we reached the second phase of the Asia roll-out, our team split and started to work in parallel on three staged releases, which were supposed to deliver unrelated functionality within 2 months of each other startin 6 months from the date. This meant that we need better mechanism for dealing with divergent codebase and big changes in progress.

Exchange-trunk + Development & Release branches per stream

After taking a step back, we came up with a new branching scheme that satisfied all our requirements. For each pending project phase we would create two parallel branches - development and release (we would call the combination of two branches a 'stream'). In addition, we devised the following policies and procedures:

Developers always commit their code changes in the dev-branches.
Any code committed to the dev-branch MUST compile. If the branch is broken, people should not commit further unrelated changes until the CI says it's fixed.
Each commit in the dev-branch should contain work for a single feature. If there is certain code pertaining to two features, we pick one of them as primary and mark the other one as dependent in the issue-tracking system. Then, all the shared code goes to the primary feature and we know that we can not release the dependent on its own. It is not necesarry that the whole feature is committed in one go or that the dev-branch committed code actually works.
When we need some code from a different stream, we would wait until they publish it to trunk and only then we would merge from trunk to the dev-branch. Cross-stream merges are prohibited. We were calling this 'picking-up' the feature. Pick-up changesets should be marked in the commit message.
Each time we pick-up a feature, after we do the minimum conflict resolution, so the code works, we would commit the changeset immediately (that's the pickup changeset). This way, any additional enhancements, fixes. etc. will be committed in separate changeset, so it will be easier to merge them back to trunk later.
Once a feature is complete and dev-tested on the dev branch, all related changesets for that feature are merged as one consolidated changeset in the release branch. We call this 'feature-promotion'. This practice makes creating release notes relatively easy and allows us to do cool things such as rolling back the whole feature with one command.
When we promote a feature that has been picked from trunk, we immediately mark-as-merged this rel-branch commit into trunk to prevent double-merge conflicts. We would look if we have made any fixes on our branch and consolidate them into a single enhancement/bugfix changeset that will be merged directly from dev-branch to trunk (as in the the rel-branch we consolidate the pickup and enhancement changesets).
If QA finds that the feature did not work, we would add further bugfix changesets to the rel-branch, but we would strive to keep them to minimum.
When a release has passed QA, we would merge each feature-level commit that originated from this stream from the release branch to trunk ('publishing'). There it will be ready for picking up by the other streams (which will merge it in their dev-branch, promote it to release, etc).
For each release we would tag the release branch, since it was already stabilized. Bugfix releases were just further tags on that same branch. For urgent production changes, we would create a bugfix branch from the tag (happened only a few times).

Overall it worked well for us. Few months after we addopted this scheme I moved to another company, but I really hope the process is still useful and being improved. An interesting thing is that every time I explain this, the first reactions are along the lines of "does it have to be that complicated?" And while I can agree that complicated it is, I am still to find a simpler streategy that could work on this scale. Any ideas?

Setting up PuTTY to use public keys for authentication

2010-01-15T23:21:00.004+09:00

I've looked on the internet for a quick step-by-step guide how to get PuTTY to use public key authentication with OpenSSH daemon and it took me some time to figure. I'm posting these instructions in case anybody else has the same needs.

Prerequisites

Make sure that your OpenSSH configuration (usually /etc/ssh/sshd_config) contains the following line:

PubkeyAuthentication yes

In my case (CentOS 5.4) it was disabled by default.

Also, you would need the full PuTTY suite which can be downloaded form here (get putty.zip).

Generating the key

This is a way to generate the key with Putty. Alternatively you can generate it with OpenSSH's ssh-keygen tool and convert it to PuTTY format.

Start PUTTYGEN.EXE
In the parameters box at the bottom of the window, choose type of key 'SSH-2 RSA', set the bit size to 2048.
Click the Generate button, move the mouse over the blank area until the progres bar fills up.
Enter your notes in the comment line (this is displayed to you when you use the key, you can change later).
Enter key-phrase, make it long and complex, write it down in a secure place or print it and hide it somewhere in your freezer.
Save the private key (*.ppk) in a reasonably secure filesystem location. Even if somebody gets access to your private key, they will still need your passphrase to use it.
Copy the text from the text box under the 'Public key for pasting into OpenSSH authorized_keys file:' and paste it on one line in a new file called authorized_keys (we'll use that later). The file should contain a single line terminated by Unix-style new-line and there shall not be an empty line after it.
Close PUTTYGEN.EXE

Associating the key with your Unix account

Login to your unix account
Create a .ssh directory under your home if it does not exist
Copy the authorized_keys file there
Do chmod 700 ~/.ssh ; chmod 600 ~/.ssh/authorized_keys

This needs to be done for each machine you are connecting to. In this case it helps if your homw is NFS mounted.

Using the key directly

Start PuTTY
Specify user@host in the 'Session > Host Name' field.
Specify the path to your private key file in the 'Connection > SSH > Auth > Private key file' box.
Click the 'Open' button at the bottom of the PuTTY settings dialog.
When prompted, enter your private-key pass-phrase and you will be logged in without entering your Unix password

Setting up Pageant to cache the decrypted private key

Let's look what we have done. The good thing is that our password does not travel over the wire and is not susceptible to man-in-the-middle attacks. The bad thing is that we used to enter the short and easy password of our Unix account, while now we have to enter the long and difficult pass-phrase of our key every time we establish a new Unix connection. In order to avoid this, we can use PuTTY Pageant which is SSH authentication agent (Unix equivalent is ssh-agent)

Start PAGEANT.EXE
Click on the computer-with-hat icon in your system tray.
Choose the Add Key option and pick your private key (*.ppk)
Enter your pass-phrase
Close the pageant dialog

From now on, when establishing SSH session Putty will try to use the decrypted key from Pageant first and then fall back to password auth if none of the keys match.

You can create a shortcut, starting Pageant and passing the paths to your keys as arguments. This will start Pageant and load the keys in one step, but you will still need to specify the pass-phrase every time you do this (typically after system restart).

Keep in mind that Pageant holds the private key in memory unencrypted. If anybody captures a heap dump of the process, they can get access to your private key without knowing the pass-phrase. That's why, you might want to stop the Pageant if you are not using it for a long time or if you shae the machine in multi-user environment.

If using Pageant, you might also check the Putty option 'Connection > SSH > Auth agent forwarding', which will allow you to use your key from the remote machine on which you are logged on.

How fast an SSD drive do you need

2010-01-15T16:00:00.012+09:00

If you need Intel X25-E 32G SSD for 70% of the cheapest listed price (shipping extra), please let me know.

For a long time I thaught that the bottleneck of all builds was the HDD, so when I got my new notebook, the first thing I did was to add a spiffy extra Intel X25-E SSD hard drive to it. As I expected, the builds went much faster. To my surprise, the HDD throughput stayed fairly low during the build, which suggested that the benefit of SSD drives kicks in early and buying higher grade drives doesn't make much difference as the bottleneck moves to the CPU quite quickly. All this makes sense, considering that a typical application has hundreds to thousands of files and the HDD spends a lot of its time seeking rather than reading.

When I cleared the IntelliJ IDEA caches and opened the IDEA Community project (total 900MB, 66k files), during the indexing, one of the cores stayed pegged, the HDD read throughput did not exceed 4mb/s for 4 mins then for 1 minute it rose to top 25mb/s, avg I guess around 15mb/s. The write never exceeded 10mb/s, and for most part it was bellow 4mb/s, the last minute was between 5 and 7mb/s.

During initial compilation, the 2 cores of the CPU (2.53GHz T9400) were quite busy, staying above 80% all the time, the disk utilization during compilation stayed less than 4mb/sec with the ocasional peaks at 6mb/sec. The write peaks were 5mb/sec, for the most time bellow 1mb/sec.

At the end of the compilation, the index update took 50 seconds, with average read throughput ~10mb/s, peaking at 30.5mb/s, the write peaked at 7mb/sec. During that time the CPU utilization dropped around 50%, which only shows that IDEA's indexing is not using both CPUs.

The bottomline is - it's not worth byuing expensive SSDs for consumer usage - cheaper ones are just as good for the average home and software development workflow. Most of the applications do not involve transferring huge volumes of data and the slowness of the spinning-platter HDDs is mostly because of seek times and fragmentation. Expensive SDDs are warranted if you are working extensively with media files or are processing huge amounts of data on the disk.

Adding custom handler for JMX service URLs

2009-11-14T12:16:00.006+09:00

Recently, I've been creating some wiki pages describing the JMX management capabilities of the system I work on and I found it quite annoying to use the MX4J HTTP Connector or copy and paste the huge-ass service:jmx:rmi:///jndi/rmi://host:port/connector URLs, so I dug in MSDN, waisted some time and without further ado, I present you:


Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\SERVICE]
@="URL:Service Location Protocol"
"URL Protocol"=""

[HKEY_CLASSES_ROOT\SERVICE\shell]
@="open"

[HKEY_CLASSES_ROOT\SERVICE\shell\open]

[HKEY_CLASSES_ROOT\SERVICE\shell\open\command]
@="\"C:\\Program Files\\Java\\jdk1.6.0_06\\bin\\jconsole.exe\" %1"

Copy the text in the box above, paste it in Notepad, edit the blue text to point to your latest JDK, save as service-url-handler.reg and doubleclick to merge in the registry. If you make a mistake, rinse and repeat.

Once you are done, your can use HTML like this:


<a title="Static Data Server" href="service:jmx:rmi:///jndi/rmi://prodhost0123:9002/jmxrmi">manage SDS</a>

On clicking the link, the browser will open JConsole for the specified service URL.

The other half of the puzzle

2009-01-12T20:09:00.006+09:00

This post on service service granularity from Ross Mason (the original developer of Mule ESB) got me in retrospective mode. I've been thinking about this before and with the time, I have come up with some criteria how to break functionality into services. Unfortunately that turned out to be only half of the puzzle. The other half is the implementation and maintenance strategy that would allow you to achieve your nicely layered boxes-and-arrows vision, while delivering ROI on quarterly intervals and not falling into the JBOWS or stovepipe-system pitfalls.

I personally favor starting top-down, focusing on specific applications and task-level services, and harvesting entity and utility services from working production code. The downside of this is that it is generally more work, often seen as 'tinkering' and in case we consolidate utility code from multiple systems, will likely introduce temporary instabilities.

This makes it a difficult sell to management - what they see is the IT guys trying to get budget to work on some kind of metaphysical inner-beauty thing instead of delivering new features. Saying that this would reduce bugs is not a great argument, because you will be reminded that's why we pay the QA team and by the way, as a professional you are not supposed to write bugs in the first place.

On the other side, the PHBs of the world seems to love the idea of the Centers of Excellence, located in China or Malaysia, full of trained and cheap labour, building the utility services and coding the entity diagrams designed by the Architecture Group in their ivory tower, while the solution engineers use point and click tools to churn 'codeless' composite applications. Unfortunately this breaks at the other end - unexpected interactions are discovered at deployment time, maintenance takes too long as the whole turns out to be more than its parts, exhibiting "emergent behavior" (also known as "underspecified features") left and right.

So, there is no quick and easy way. I guess we need to become IT salesmen and we need to compete with the professional vendor scamsters and the consultancy loan-sharks (that's a topic for another post).

Mavenizaton - A real life scenario

2008-10-30T17:46:00.007+09:00

The other day I posted a message on a certain vendor’s support forum asking about the possibility to include minimal POMs in their jars (so they will be easier to mass-import into local repository) and I got the following reply:

We want to move from ant to maven for a long time but never got a chance to do it. Our ant script is pretty big and complex so we are afraid it will take us a while to migrate to maven. Since we have other important things to work on, we keep delaying this kind of infrastructure things that have relatively less impact than bugs. If you know any way to migrate from ant to maven quickly, I would love to hear.

So I asked if I can get access to their Ant script and they kindly sent me that 3000-lines of a beast.

The Challenge

The product is comprised of several modules which have well-defined dependencies on one another and can be used in various configurations. Each configuration is described in a build.properties file by xxx-source-path, xxx-properties-path and xxx-output-path properties. The source-path contains both Java files and misc. resources (there are no tests). The xxx-properties-path contains property files only.

In the end, for each module the build produces the following artifacts:

JAR files – depending on properties, these can be:
- Obfuscated with Zelix KlassMaster
- Debug builds (still obfuscated, but with line-numbers)
- Postprocessed by Retroweaver for Java 1.4 compatibility
Zipped sources
Zipped source stubs – public and protected method definitions and javadoc (no method bodies)
Zipped Javadocs
User manual in PDF format

In addition to these primary artifacts, the project:

Generates a couple of demo applications packaged as single-click JARs and JNLP.
Filters all property and some source files, substituting placeholders with values defined in the build.properties.
Compiles on the fly a doclet and then uses it to generate the source stubs.
Creates a bunch of bundles, containing different combinations of the aforementioned artifacts.

The vendor has a policy of releasing new demo-build every day which expires after 28 days.

The First Step – Fake It

In fact, we, the customers don’t care whether you are using Maven internally. All we care is to receive a POM with correct dependencies and (if possible) to be able to download the libraries from the official Maven repository (a.k.a. Repo1) or if not – your private repository (i.e. http://repo.your-company.com/demo or http://repo.your-company.com/licensed ).

In order to achieve this:

For each artifact, create a pom.properties file containing 3 properties: version, groupId and artifactId.
For each artifact, create a minimal POM containg the artifact group, name, version and dependencies information. Alternatively you can put placeholders and use Ant filtering to resolve the group/name/version from the pom.properties
Integrate the new files into your build process, treating them as yet another resource file and placing them under the META-INF/maven/your/artifact/group in each artifact JAR.
If desired use artifact:deploy task to push the artifacts to a Maven repository.

This is all you need to do for the ‘regular’ JARs. For the various other files, you would want to use the same artifact name and a classifier I would suggest using:

sources – standard Maven convention, since the full sources are not released to regular customers, it would make sense to have them either under a different classifier (e.g. fullsources) or same classifier, but from a private repo
javadocs – standard Maven convention
jdk14 – for retrozapped binaries
debug – for obfuscated binaries with line numbers

Then you Migrate

If we step back, the current build proces is something like this:

Preparation Phase
1. Set variables
2. Compile custom doclet
Generate Sources
1. Filter particular files to resolve values from properties
Compile
Copy resources
Package binaries (create jar file)
Obfuscate
Retrozap
Javadocs
Source Stubs
Assemble distribution archives
Build the demos

Here is how it would look after we convert the build to Maven:

All the path variables go away, replaced by standard Maven directory layout (you should be able to reorganize the source trees for a few hours on a Friday afternoon.) As a last resort you can define custom directories in the build section.
The company-name, version, etc. props go away replaced by the respective POM elements.
The rest of the settings are specified in profiles defined on project or user level.
The resource filtering is a built in feature for maven, but if you need to pre-process Java files, you would need to attach your own plugin to the grnerate-sources lifecycle phase.
Compile and Package are standard Maven targets and you don’t need to do anything.
The ZKM obfuscation would be tricky because there is no Maven plug in (or at least I couldn’t find any). I suggest that you use the AntRun plugin as a temporary hack.
The Javadoc plugin would automatically generate your Javadocs, you need to explicitly specify a second execution for your custom doclet
The assembly plugin should help you organize your different distributions.

This leaves out a lot of details, but ultimately some of the decisions are yours to make. I would suggest using a script to reorganize your sources, so you can do a number of dry-runs and verify that the result is as expected before you change the script to perform the move in the version control system.

Project Layout Considerations

Each Maven POM has exactly one parent, but can be part of multiple reactors. Both parent and reactors are pom-packaged artifacts, but their role is different. Most tutorials use the same pom for both purposes, but there is benefit in separating the functions (more about this in a bit).

POM Inheritance

The role of the parent is to provide defaults for the child POMs that inherit from it. The POMs are resolved by the normal Maven dependency resolution mechanism, which means that if you want the child projects to reflect a change in a parent-pom, you need to install/deploy the parent first. Alternatively, you can use the project/parent/relativePath element to explicitly specify the parent-pom’s path in the file layout (in this case, the changes are picked from the file-system.)

At the top-level parent you should use dependencyManagement to specify versions and transitive-dependency-exclusions for all artifacts used by the project. No child-artifacts should explicitly declare versions and exclusions.

You should use pluginManagement to fix the plugin versions and (optionally) specify common options for plugins. It’s ok (although not recommended) to override the plugin options in the children-pom’s, but you should not specify versions there. If you need the same overrides in multiple children, consider extracting them into intermediary-parent pom.

Finally, you should try to specify most of your profiles on a parent-pom level. This, combined with the pluginManagement allows you to define the bulk your build in the parent and leave the children DRY.

The children have to explicitly specify the master-pom version which makes it a bit of a pain to increment the version (on the other side, the master-pom usually changes much slower.)

Building in Dependency Order

The normal way for Maven to resolve dependencies between projects is through the local repository. This means that if project A depends on project B and we make a change in B, in order to test A against the new change, we need to first install or deploy B and then build A, so it would pick up the new changes (this is assuming we’re using SNAPSHOT versions).

The reactor projects take care about such closely related artifacts and automatically rebuild them in the correct order. Also they act as a grouping for the sub-projects (useful when you build assemblies). The reactors themselves are artifacts using pom-packaging (same as parents), but they explicitly list the relative path to each constituting module, and this is the reason that I recommend separation between parents and reactors:

Creating Distributions

The standard way of packaging distributions in Maven is using the Assembly plugin . It allows you to describe your distribution package in terms of project resources, artifactsIds and classifiers. A typical assembly descriptor would be:

Create a tar and zip files with classifier ‘distro’.
- Add all files from /src/main/assembly
  - excluding **.xml, and **.sh.
- Add all files from /src/main/assembly
  - that match **.sh
  - and set the unix permissions to 551.
- Add the maven artifact
  - with group-id=com.company.product.modules and artifact-id=module1 and classifier=obfuscated
  - put it under lib.
- Add all the transitive dependencies of this project (assuming this descriptor is in a reactor project)
  - and put them under lib/ext

You can also unpack and consolidate JARs, specify manifest-entries and signing for single-JAR distributions and so on.

Each artifact can produce more than one assemblies. If you want your assemblies to go into the repo, make sure you use the ‘attached’ goal. Most of the times, I use the reactor poms for distributions, but sometimes I just define the assembly in the JAR module’s POM.

Parents and Children vs. Reactors and Modules

Category	Parent/Child	Reactor/Module	Merged Parent+Reactor
Direction of Coupling	Children know the names of their parent, parents don’t know their children	Reactors knows the locations of their modules, modules don’t even know whether they are being built as a part of a reactor	Creates circular dependency between children/modules and parent/reactor artifacts.
Multiplicity	Each child has exactly one parent, specified in the child POM.	A module can be part of more than one reactors.	You must keep it to one reactor. If you define alternative reactor, the parent will still be resolved to the main parent-POM (as specified in the children POM’s), which can lead to a great deal of confusion (and undefined behaviour for some plugins).
Location	The parent is usually resolved through the repo (optionally through relative path)	The modules are always resolved through relative path	You need to physically layout your project, so the parent POM is accessible by relative path (which is not needed otherwise).
Change Drivers	The parent changes when your build process changes.	The reactor changes when you add a new module	By the artificial coupling, your POM is subject to more change. This is particularly bad for the parent role, because you need to manually update the parent-versions of the children.

Given this table, I’d recommend a project layout along these lines:

-+-+- project-root/
 |
 +-+- modules/
 | |
 | +-+- module1/ 
 | | |
 | | +--- pom.xml       | inherit from modules parent; specify dependencies 
 | | |                  | and module details (name, version, etc.)
 | | +-+- src/
 | |   +-+- main/
 | |   | +--- java/
 | |   | +--- resources/
 | |   |
 | |   +-+- test/
 | |     +--- java/
 | |     +--- resources/
 | |
 | | 
 | +-+- module2/ 
 | | |
 | | ...
 | |
 | +-+- module3/
 | | |
 | | ...
 | |
 | +-+- reactor1/       | e.g. all modules - used for the CI build and pre-commit
 | | |
 | | +--- pom.xml       | no parent; specify the modules as ../module1, ../module2, etc.
 | |
 | +-+- reactor2/       | e.g. only module2+module3 - used by the module3 developers only
 | | |
 | | +--- pom.xml   
 | |
 | +--- pom.xml         | modules parent - inherits from the main parent pom and specifies 
 |                      | module-specific profiles and plugin settings
 |
 +-+- applications/
 | |
 | +-+- application1/ 
 | | |
 | | +--- pom.xml       | inherit from apps parent; specify dependencies, module details 
 | | |                  | and whatever custom steps necesarry for the application
 | | ...
 | | 
 | +-+- application2/ 
 | | |
 | | ...
 | |
 | +-+- reactor/        | e.g. all applications
 | | |
 | | +--- pom.xml       | no parent; specify the modules as ../application1, ../application2, etc.
 | |  
 | |
 | +--- pom.xml         | apps parent - inherits from the main parent pom and specifies 
 |                      | module-specific profiles and plugin settings
 |
 +--- main-reactor      | specify as modules ../applications/reactor and ../modules/reactor1
 |
 +--- pom.xml           | the main parent pom

A few Tips

When You getStuck with a Bad Plugin

For all operations that you can’t easily achieve with the existing Maven plugins, you can fall back to the Antrun plugin and reuse snippets of your current build script. Yes, it’s a hack, but it gets the job done and you can fix it when it breaks.

Try to keep these hacks in the parent pom.

Dependency Management

When you define dependencies, take care to specify proper scopes (default is ‘compile’, also you should consider ‘provided’ and ‘test’). You should also mark all optional dependencies like POI HSSF. For some artifacts you might need to specify dependency-excludes, so Maven doesn’t bundle half of the Apache Commons with you.

Evaluation Builds

You might want to release the daily evaluation builds as snapshots. Maven has specific semantic for snapshots that it always refreshes them based on timestamp. For example, if you release an artifact my.group:module:1.0.0, Maven will download it once to the local repository and never look for another version unless you delete it manually (that's why you should NEVER re-cut a release once it's been pushed out - just declare the version broken and increment).

If you use a snapshot version (i.e. my.group:module:1.0.0-eval-SNAPSHOT), Maven will check once a day (unless forced with '-u') and automatically any new version base on timestamp. To achieve this you might want to cut an eval-branch from the tag of each release and change the version using a shell script or something.

Why ORM?

2008-10-25T15:24:00.009+09:00

I like Spring JDBC Template - it is simple, clean and saves a lot from the JDBC drudgery. It provides nice abstraction, simplifying the usage and management of prepared statements and overall if all you want is to do an ocasional query, it doesn't get any simpler. Oh, and you have full control on the underlying JDBC connection, allowing you to exploit any vendor-specific functionality. The provided functionality only makes the data access easier and supports only very basic property to column mapping out of the box (or you can write your own mapper in Java). Because of its lightweight nature, JDBC Template does not allow for transparent caching, horizontal table partitioning, vertical sharding, etc. It also doesn't offer lazy fetching of relations, identity tracking and other instrumentation magic.

Another tool I like is iBATIS. In comparison with JDBC template, it offers more structure, organizing my SQL statements in a logical way and supports most common forms of mapping the result sets to beans (including graphs of objects and joins). Ibatis does support caching (only on a whole query level), still no sharding and partitioning, lazy fetching and identity tracking. IBATIS doesn't try to parse your queries.

The Abator tool can generate iBATIS CRUD mappings, DDL from object model, and object model from a database, but frankly - in its generated form all these suck. The DDL uses only a few datatypes, has no RI or semantic constraints, nor indexes (I use it sometimes, but only as a boilerplate which I extensively tweak afterwards). The generated beans are isomorphic with the underlying tables, which is great if you are doing CRUD, but I find that most of my applications do data analysis, so the queries tend to be quite hairy, half of them encapsulated in stored procedures (for transaction processing I'd rather use Spring or plain JDBC anyway). The bottom line is that Abator handles the trivial cases, but for most of my projects it doesn't help much, so I'm back to manual coding (which is not that bad anyway.)

The first time I looked at ORM was when I checked TopLink for a project around 2000. Back then I got the impression that it introduces a lot of new concepts in order to simplify something that's already simple enough - loading a couple of beans from a table. For that project I used plain JDBC, it took me perhaps 20 mins to write and I can't remember taking much time to maintain that code - the total cost should have been less than 3 man-hours.

The second time I tried Hibernate and Toplink last year (2006) on a prototype with the same dataset and relatively small domain model (for which I finally ended up using iBATIS). The first thing that stroke me was how slow everything was - the startup time was noticeably slower than with iBATIS (about 500-800%), spent in initializing the JPA entity manager and the queries were consistently 30-50% slower. It might have been that I didn't know how to tune it, but I just couldn't achieve satisfactory results. I researched the facilities for mapping to externally defined schema and it stroke me that both Hibernate and TL are way clunkier than iBatis. There were also a number of restrictions that suggested that I need to modify my schema just to enable certain features of the tool... in the end, the identity tracking is a leaky abstraction - the whole story with the attached and detached entities smells to me.

Now the thousand dolar question, given that most applications out there DO NOT use horizontal partitioning or sharding, caching in the application layer is usually more efficient anyway, identity tracking and the transparent (hidden) lazy fetching often causes more problems than it solves (and countless hours lost in app tuning with dubious results), why are so many people obsessed with the idea of cramming an ORM into their application?

The other day I wrote a small application that had four composite entities and a couple of fairly sophisticated queries, the the data access part took me about less than an hour (that's implementation, testing and tuning), so:

How long would it have taken me if I used JPA? (to get reasonably well performing, tested code)
If you are using ORM, how many entities do you have? How much time (percentage) do you spend working on your persistence logic? Are you using the database as operational storage or are you trating it as a long-term asset? Do you share the DB with other applications?
Are the productivity gains realized only when your persistent model is so big that it's unmanageable by other means? Is it to futureproof a product (what is a reasonable timeframe for ROI, any examples?)?
How important is for you the ability to scale your data access to multiple database servers?
Have you ever switched an application from one SQL server to another? How much effort was it? What does the application do with the DB?
What are the benefits of using ORM smart caching and partial queries in comparison with in-memory database like TimesTen or H2?

Please enlighten me...

On Scopes

2008-09-10T01:21:00.002+09:00

Let's define component as a programming abstraction that exposes a number of interfaces and is instantiated and managed by a container. A component has implementation that sits behind the interface and provides the functionality, which could be a custom Java class, a proxy, a BPEL engine, a mock object, or even a mechanical turk solving captchas.

In the case of Java class components, they are usually composed of smaller Java classes like the ones in java.lang.* and java.util.* packages. All but the most simple Java objects are actually object graphs which are instantiated, wired togerher and orchestrated by the Java language and the JVM. The Java language specifies the composition structure and behavior, while JVM provides the means to invoke operations and share data. In contrast, the components are instantiated by a container and the container is responsible to wire them together (usually based on a blueprint in the form of a configuration or class annotations). The container can also provide a number of declarative services to its components: transaction management, access control, performance statistics and transparent distribution. Some popular containers are OSGi, Spring, EJB, ActiveX, MTS/COM+.

Even if a container can expose a local component over remote interface without requiring any code changes, practice has shown that taking a single-server application and distributing it in such fashion usually results in a disaster. This doesn't mean that the non-invasive distribution support is useless though. Given responsible usage, it can enable you to build distributed applications without having to deal with the transport API, using clean and reusable code (POJO). There are a number of new frameworks that are built around the concept and try to provide the right level of transparency and control (Mule ESB, Spring Remoting, Spring Integration are some popular choices).

Once you get too many components in the same contaner, autowiring starts doing most unexpected things, debugging becomes difficult and you start loosing track of your system. The solution is to add scoping. Spring supports limited scoping facilities in the form of parent-child application contexts. Although it solves the problem with configuration visibility, it does not provide facilities for controlling the public interface of the scope (explicitly exported objects). OSGi tackles this problem from a different direction using controlled classloaders, Spring OSGi is trying to be the best of both worlds, giving you scoped configuration and runtime autodiscoverable scoped components.

<interlude/>

I just finished reading Distributed Event Based Systems and they have an excellent chapter on utilizing scopes as means to structure a distributed event-driven application (chapter 6). They suggest a scoped architecture, where each component can belong to multiple scopes, scopes are linked in an arbitrary graph and notifications are governed by the different scope and component properties. The scopes described can declaratively provide a number of services (think AOP for messages). Some of these services are:

Containment - logical grouping of components and other scopes, allowing easy addressing for event dissemination and policy enforcement.
Scope Interface - defining which messages can cross the scope boundary and enforcing this policy
Transmission policy - determine which scope components will receive a certain message and/or which components have a right to publish/receive messages that cross the scope boundary.
Mappings - when a message crosses the scope boundary, enrich or transform it based on the source and destination scopes.

Scopes could be used to delineate groups with different QoS requirements, or to partition the application in geographical or security domains (or even both at the same time) and apply the appropriate policies. Integrating an external systems is also easier, because they are treated as yet another scope (as opposed to "enemy in the bee's nest") and you can provide rich access to the messages that the third party is entitled to without having to bolt on yet another authentication and authorization mechanism.

The book focuses in great detail and implementation strategies, analyzing different approaches ranging from simplistic (flooding) to very complex (integrated routing). Right now, the described architecture can not be implemented directly in any eventing system that I know of, though we can see some of the ideas in Jini (peer to peer with cooperative filtering), JMS (in implementation supporting topic hierarchies), OSGi (intra-JVM) or SCA (a lot of talk about scopes and composites, but I don't see them doing anything useful with them). The authors also propose a configuration language for managing scoped systems with the specified features. I can see AMQP as a flexible low-level specification that would allow to build something like this by implementing custom exchanges. We can use the scope configuration language as an intermediate representation, from which we can generate the custom exchange code.

Ultimately, I believe that in 10 years from now this scoping and routing facilities will become standard part of the messaging middleware (and perhaps hardware?) I'm looking in the general direction of AMQP, possibly SCA (despite their WS-* fetish) and new high level languages directly expressing architectural concerns like ACME and Einstein.

Dynamic components in Mule

2008-09-02T17:18:00.004+09:00

Imagine that you are given to implement the following scenario:

A system is composed of a number of uniform entities, each entity publishes its status on a separate pub-sub topic and accepting instructions over separate control queue.
Your application receives a stream of instructions over a control channel, instructing it to monitor and control or stop controlling certain entity.

or you receive a stream of active orders and want to subscribe to the right marketdata to calculate a relative performance benchmark, or...

Usually I write about how to implement stuff with Mule, but this time I'm going to write why Mule might not be your best choice if your use-case is anything like above.

The most straight forward approach for implementing the system above is to create a simple POJO, encapsulating the management algorithm and have a service listen to the control channel and instantiate/dispose a new component for each managed entity.

The problem is that in Mule the components are somewhat heavyweight and it's not so nice when you start having hundreds (approaching thousands) of them. Another problem is that the code for manually creating and registering a componment along with the whole paraphernalia of routers, endpoints, filters and transformers is quite verbose.

The latter problem is easier to solve. There are a couple of JIRA issues that aim to simplify the programmatic configuration (MULE-2228, MULE-3495, MULE-3495). The Annotations project at Muleforge is also worth watching (right now there's no documentation, so you'd need to dig in Subversion).

The problem with the too many components is more fundamental. The main application domain for Mule is service integration. Though it also happens to be a fairly good application platform, it does fall short when it comes to managing a lot of components. One of the important things missing is the ability to create groups of components with shared thread pools. Another missing thing is better support for variable subscriptions (provide an API to easily add and remove subscriptions to the same component using some kind of template). Another notable improvement would be to elaborate the models into full blown scope-controllers, allowing to create composite applications with proper interfaces, routing policies, etc. All of these are addressable, but there is a long way to go.

There are also some improvements that can be done to the JMX agent - use JMX relation service to connect component to its endpoint and statistics, all components of certain type, user defined groups of components. Allow the component to exercise some control over the Mule-generated ObjectName so users can organize the different components in groups and ease the monitoring by EMS tools by allowing to query the parameters from a specific group.

Please share if you've had similar issues, how you dealt with them and whether you think {with the benefit of a hindsight) that the solution was worth the effort. Also I'd like to hear from users/developers of other integration frameworks or composite application containers (Newton?) how they handle these problems.

Transformers vs. Components - Take 2

2008-09-01T01:14:00.002+09:00

In my previous article, I claimed that one of the main criteria for which Mule artifact to choose is the multiplicity of the inbound and outbound messages. I've been thinking about it and I want to add two more points:

Transformers are one instance per endpoint reference. This means that even if you use pooled components, all messages go through single transformer instance. In such case, you need to make sure that the transformer is threadsafe (which is not true if it is holding a JDBC connection or Hibernate session). The components do not have to be threadsafe, though if you are injecting some shared state you still need to properly synchronize the access to it.
Mule2 services allow you to configure different ways of instantiating the component (singleton, pooled, looked up from Spring/JNDI/OSGi/etc.). In contrast, all transformers defined using the mule-core namespace are prototype-scoped.
I actualy agree that direct database access belongs in a component. I still think that accessing a cache is kindof a gray area though.

InfoQ, Firefox Hacks and Media Snobbery

2008-08-16T01:06:00.007+09:00

I happen to be one of those people that don't like TV - it's not that I don't watch it when I get a chance, but I've made a conscious decision not to buy one and to get my news and movies from other sources. My primary issue is that TV as a media is too engaging, passive and slow. Radio is also passive and slow, but less engaging, hence better.

For example: in 10 mins browsing news sites, I can learn more than 10 mins of browsing newspapers, which still fare better than 10 mins of radio, which is just the same as 10 mins of TV, except that when I watch TV I don't do anything else.

I'm a much faster reader than listener. When I read I can always stop and mull over a sentence for a couple of minutes, go back or skip forward with natural ease. Not so with video and audio. Even though most streaming media these days have some kind of non-linear controls (skip-bar, etc.) it's still impossible to do the equivalent of scanning the headlines or speed-reading in the case of podcast or video content.

One of the reasons I like the InfoQ interviews is that they have full transcripts. One of the resons I hate them is that the transcript is enclosed in small box, 250px high in the middle of the screen and one has to click the open and close buttons and scroll like crazy. What I often end up doing is to fire DOM Inspector, cut off all the DOM nodes except the ancestors of #interviewContent, switch to CSS properties, delete the height and max-height props and this way I end up with full-screen view of the transcript, which I can print or browse at my convenience.

Today as I was debugging some stuff, I remembered that Firefox supports user stylesheets. By putting the bellow statements in my ${FIREFOX_PROFILE}/chrome/userContent.css, I was able to expand the transcript to be readable in a browser without inner scrolling and tweak the printable view to display only the relevant content (no empty squares in place of the flash player or vendor links I can't click on paper.)

/*
 * This file can be used to apply a style to all web pages you view 
 * Rules without !important are overruled by author rules if the author sets any. 
 * Rules with !important overrule author rules. 
 * 
 * For more examples see http://www.mozilla.org/unix/customizing.html 
 */ 
 
div#interviewContent, 
div#interviewContent * { 
    height: auto ! important; 
    max-height: none ! important; 
    overflow: visible ! important; 
} 
 
div#interviewContent div div div { 
    display: block ! important; 
} 
 
@media print { 
    * { 
        border: 0px ! important; 
        float: none ! important; 
    } 
 
    #content, 
    #container, 
    #content-wrapper, 
    .box .bottom-corners, 
    .box .bottom-corners div, 
    .box .top-corners, 
    .box .top-corners div, 
    .box .box-content, 
    .box .box-content-2, 
    .box .box-bottom, 
    .box .box-content-3 { 
        margin: 0 ! important; 
        padding: 0 ! important ; 
        float : none ! important; 
        width: 100% ! important; 
        line-height: 130% ! important; 
    } 
 
    /* these don't work on paper */ 
    object, 
    .comments-sort, 
    .vendor-content-box { 
        display: none ! important; 
    } 
 
    /* visual garbage */ 
    .tags3, 
    .comment-reply, 
    .comment-header > p, 
    .comment-footer, 
    #interviewContent img[alt='show all'], 
    #interviewContent img[alt='hide all'], 
    #content-wrapper > .box > h2, 
    #footer { 
        display: none ! important; 
    } 
 
    /* Fix comment headings */ 
    .comment-header * { 
        display: inline ! important; 
    } 
 
    div.comment-header { 
        border-top: 1px solid gray ! important; 
    } 
 
}

Now the only thing left is to convince the InfoQ guys to start providing transcripts for the presentations and downloadable slides :-)

Transformers vs Components

2008-07-23T23:53:00.002+09:00

When I started using Mule, for some time I used to wonder what's the big difference between a transformer and a component. Why don't we just stick with the plain pipes and filters model and we complicate it with the 'component' concept?

In general, transformers have conceptually simpler interface - they are supposed to get a payload and return a payload, possibly of different type. You should prefer transformers for reusable transformation tasks. On the other hand, transformers can not:

Swallow messages - to achieve this use filter or router.
Return multiple messages - use router.
Pause temporarily

In addition, despite their more complex configuration components provide some neat features:

Message queuing - if your message goes through multiple stages of enrichment, each taking variable time, you might be better off implementing the stages as components and separating them with VM queues.
Statistics - automatically measures the processing time and volume through each in or outbound endpoint.
Lifecycle control - you can start, stop or pause component multiple times (while a component is not running the messages will queue up.) The transformers are only initialized and disposed once.
A componen allows you to tack on multiple filters, transformers, routers, etc. A transformer is one part of this chain.

To put it together, here is an overview of Mule's component roles in a simple message routing sequence (I'm trying to explain the roles, not the exact sequence.)

The message enters through an inbound endpoint, where the transport extracts the payload and the message headers and bundles them in a message. (The message and some other objects are bundled in EventContext, but it is still a mystery to me why do we need it.)
After the message is created it passes through a filter chain, where the filters are applied one after another and each of them votes with true or false a full consensus is required or otherwise the message is rejected. Depending on the config the rejected messages can be routed somewhere else. The filters are not supposed to modify the message (in practice they can, but it's a very bad idea.)
If a message passes all filters it goes through a transformer chain, where each transformer is applied to the output of the preceding one. A transformer is not able to gracefully reject a message, but it can throw an exception.
We might have an interceptor wrapped around the component, which can log stuff before or after invocation or measure latency, etc.
Based on the payload and on the component type, Mule resolves the method it should call and if necessarry transforms the payload to something that fits. The return value of the method is used as a payload to a single outgoing message.
The outgoing message is passed to an outbound router, that can split it in multiple messages or just drop it. The router then usually sends the message(s) to arbitrary number of outgoing endpoints. The outgoing router uses the filter chains of the outgoing endpoints to decide which ones to use. The outgoing endpoints can be Mule internal endpoints or periphery endpoints, conecting the application to external system.

In general, I would recommend that you start with transformer and refactor it to a component if you realize that you want to add any of the aspects mentioned above. You can even easily use transformers as POJO components.

Testing Mule applications

2008-07-21T13:50:00.013+09:00

Antoine Borg has posted an intersting article at Ricston Blog about the feasibility of using mock objects for testing Mule applications. The bottom line is that in integration scenario is usually difficult to capture the application specification in mocks and often it's easier to write stub applications simulating the external systems (outside of your application process).

I've found that often I don't really test the whole application, but instead I test single components with a few attached transformers, filters and routers. I start by writing a simple Mule configuration - a single component using VM endpoints with queuing. Then I use MuleClient to feed in canned input data and assert the output from the outbound endpoint(s). As the application takes shape, I add transformers and routers as needed to approximate the real usage.

I could imagine that the next step would be to reuse my production configuration and extract the perimeter endpoint definitions in a new file (separating them from the model, connectors and internal endpoints) and pass the two configuration files to the config builder. This would allow me to create an alternative perimeter endpoints file using VM endpoints, so I can instantiate it in test case and use MuleClient for testing.

The benefits of the approach are that you are testing your component and routing logic and are not exposed to the peculiarities of the external system. Ideally we still want to have a full integration tests, including ones covering crash-failure, failover, connectivity loss and overload scenarios. We can achieve parts of this by stubbing the external system, but so far I usually find it difficult to reproduce the behaviour faithfully enough (especially when we have limited understanding about the external system).

No bulletpoints this time.

Build, Buy or Steal

2008-06-16T00:41:00.009+09:00

This post is based on a text I wrote at my day job. It is edited to reflect my personal opinions and does not represent my employer’s view in any way.

The classic 'build vs. buy' dilemma is further complicated by the new wave of quality open source components.

Even if we assume that the open-source alternatives are generally inferior to their closed-source counterparts, their low price makes them an attractive choice for getting started, provided that there is an easy migration path to more powerful solutions. Additionally, many open-source tools are backed by commercial companies which can provide support, consulting and guidance on demand.

Open-source components are often (not always) higher quality than the internally developed libraries. This is because of the usually pretty-good community QA (many users; no time pressures) and the small-vendor mentality of the backing firms (they need to be much better than the industry standard in order to convince a client). Also, a solid OS project provides a decent documentation and community support, often has books written about it and, in general, makes it easier to recruit people already familiar with it.

We do want to build components in-house when we believe that we can provide and need more value than their OS and commercial counterparts can provide. We always need to keep in account that building good library is a continuous investment in new features, bug fixing and documentation. Providing a sub-optimal internal component is the worst of all worlds.

Another big question when to contribute our changes back to an open source project. Conventional wisdom says that you shouldn't give our work for free. Still, given the previous paragraph, I would claim that it's in our best commercial interest to contribute back any changes to the external module's core. In such cases, the initial investment is small, the maintenance is high (as we need to reapply changes with every release) and the business value of the code is low. Alternative is to fork the open-source component, which brings us to the problem in the previous paragraph.

We might also decide to contribute or open source internally developed extensions of Open Source library, gaining free QA and possibly bugfixes. We should not open source code that gives us [our product] significant advantage over alternative solutions on the market. We must not share non-generic code, capturing: business processes, algorithms, site-specific logic, etc.

And now is the time for the mandatory list at the end of the post. I'm going to enumerate a few build vs buy decision anti-patterns:

Not Invented Here (NIH) Syndrome - you know, when we write your own thing because we are too lazy to do our research or read the docs.
The Wrapping Party - every external library is wrapped in order to integrate into the proprietary architectural framework. Though it might look like a good decision, it often leads to difficulties in the debugging, inability to apply best practices and tools, and generally inefficient use of the library
Nobody Got Fired for Buying Expensive Stuff - when technical decisions are taken by managers without enough information (or understanding). Sometimes this is rationalized as that all the products look the same on paper, so at least this one comes from a reputable company we can sue. Problem is that often the very expensive do-it-all products require a staff of rocket scientists (or vendor consultants) in order to deliver anything after that.
The First One is Free - some vendors try to promote their products as open source, while capturing your data and interfaces in proprietary formats and protocols and then selling services around them. The SOA RAD tools and BPM tools are particularly bad offenders here. The problem is that this limits the ways for evolving your platform. The way to prevent this is to always be aware what part of the solution is platform specific: configuration (is it documented file format), POJO vs proprietary interface components, can you get the whole solution as a bunch of text files, what protocols are used for communication, what is the data storage, can we plug our custom infrastructure, where?

The Shape of Complexity

2008-06-03T22:05:00.003+09:00

Some things are complex and there's no way around it. Still, even if we can't remove the complexity, often we can shape it into different forms. Some applications have shallow and wide complexity - many simple things with relatively few dependencies, but the overall system has mind-boggling emergent behavior (my favorite 25kloc perl scripts example). Other applications have narrow and deep complexity - small code base, but using so much infrastructure amd metaprogramming, that to understand what's going on, you need to be expert in the platform (think about Hello World written with EJB2 or any application using Ruby on Rails).

The horizontal complexity is pretty easy to deal with - find two similar things and create abstraction for them, rinse and repeat. It is important to stop and look every now and then for similar abstractions and for ugly usages. Eliminate the similar abstractions by extracting common functionality in helpers or superclasses; deal with the ugly usages by splitting the abstraction.

By definition, when we introduce abstractions, our complexity becomes 'narrower' and 'taller'. If we put too much stuff into the abstract classes or we nest them too much, we might transform the bunch of simple classes that we couldn't understand when taken together into a somewhat smaller bunch of more complex classes that we can't understand even in isolation. If go overboard in the other direction, e.g. adding too many facades and convenience methods, the API becomes too big without providing enough benefit to learn it (canonical example - I know a project that has a Strings class, containing constants for empty string, single digits, punctoation marks, single letters and other. Apart from being pointless, more verbose and difficult to apply consistently, this also couples all classes in that project to the util package).

So, how do we end up with code base that is neither too tall, neither too wide, but just the right shape?

Acknowledge that the right shape depends on the individual - some people can cope with more abstraction, while others can remember more facts. A metric for abstraction efficiency can be defined as delta-loc/n^{levels-of-indirection} (where n is a constant bigger than 2).
Keep in mind the choice of tools - for example IntelliJ IDEA excels at navigating well-factored code, while Vi people often prefer decoupled classes that can be changed with low risk of impacting other areas of code.
Consider the infrastructure maturity - using XA transactions with JMS is simple, debugging buggy JMS (caugh.. activemq.. caugh) is entirely different issue.
The experience of the team is important - for some people JMS is the most obvious way to send a piece of data once and only once to another system; others treat JMS as black magic and resort to FTP and cunning rename+move schemes, involving multiple directories and recovery scenarios.
And finaly, even if different pieces of infrastructure can provide similar functionality, sometimes they vary by the way they do it - in my previous article about component granularity, I mentioned the fat component vs fat Mule configuration scenarios. In this case, we can move the complexity between the Java code and the Mule code. Mule provides out of the box abstractions for threading, routing and transforming, but it is not great as a general-purpose process definition language. Java provides general purpose language, but does not provide high level primitives for threading and routhing. Another example: consider running OLAP against OLTP schema and dedicated warehouse schema - it works, but the difference in speed and CPU utilization can be orders of magnitude.

Pushing Files

2008-05-27T21:38:00.001+09:00

In the next couple of posts I'll try to capture a couple of scenarios that I think Mule (and in general ESB) fits in. The target audience are people like my friend who once asked me once 'Well, I got it - it's all about transports, transformers, routers and components. In the end, what would you use it for?'

When I started reading about Mule, the first thing that caught my eye was the huge collection of transports and third party modules. It allows you with a relatively little configuration to automate scenarios like sucking a file, uploading it to FTP and sending an email if it failed or moving it to a audit dir on success. Add a few lines and you can have Mule try to upload, if it fails, try to post to JMS queue and if it still fails automatically create a JIRA issue.

Still, even though Mule is easy to use, it's not simple. Actually even for basic use of Mule, you need to know XML, have idea about Enterprise Integration Patterns and be familiar with Java server applications. Suffice to say that many of my colleagues do not meet these requirements. Compare this with writing a bash or Perl script - it's much easier and quite often it is good enough. I wouldn't be a stretch to say that the TCO of a Perl script for the previous scenario is 10 times less than the TCO of an application built using Mule or any other free or commercial integrated solution.

Things change a bit when you have a big number of integration jobs. If the tasks are similar, a smart scripter would abstract away the processing, capturing it into a parameterized sub-routine or shell script, with multiple simple wrappers with different arguments. Unfortunately, much too often the strategy is copy-and-modify (especially when the guy who wrote the original script has left the company, and the guy who did the first 30 clones also). Cloning is a pragmatic, low-risk strategy since it minimizes the chance that you break stuff that works. Such developers are usually praised for their get-it-done, goal oriented attitude and management likes their low-risk approach. Still, the dirty secret that nobody likes to talk about is that the system grows in complexity until the moment when nobody knows what is it doing anymore (I think Skynet started that way).

Even we assume that the TCO of a single Perl script is 10 times less than the one of a Mule implementation, a single Mule instance with relatively little custom development can easily handle the job of all the 170 scripts that I counted in one directory on a production server I use. If we assume that these scripts were derived from 10 archetypes (actually it's less) this still gives us a cost reduction of 170%. Even more important - the Mule and Spring configurations give you a roadmap, which while not trivial is much easier to comprehend and audit than the 27,227 lines of repetitive Perl code.

To be fair, in this example I am comparing bad scripting code, written by multiple people over the time, without proper governance to an hypothetical Mule solution written by decent developers. Let's assume that you have a team of good scripting developers, writing well structured code and taking care to provide maintenance documentation and guidelines for extending the system. Actually, this would work fine. Effectively these guys will be defining a platform quite similar to what you get from an ESB (in the end it's all Turing-complete languages). The only catch is that in my experience, in a typical enterprise it's much less probable to meet such Perl coders than the above mentioned Java guys. The argument that Perl programs don't have to be messy is quite similar to "guns don't kill people".

Even if we assumed that the mythical Perl programers did exist (and you had all three of them in your IT department), there are still a couple of points that make an Mule an attractive solution.

Mule gives you a proven architectural framework, so you don't have to invent your own. This includes guidelines, some documentation (getting better) and abstract base classes for many of the EIP patterns.
The business code written against Mule is usually framework agnostic (POJO).
Various transports, transformers and out-of-the box components.
Handles advanced stuff as XA transactions and threading models that are just not possible using Perl.
Since the bulk of your application will be built on a platform, it will be easier to find people whi have some experience with it, which will shorten the learning curve.
Extensive JMX monitoring and management.

And here are some good reasons to keep cloning scripts:

Naturally resilient to regressions.
Allow you to hire cheaper developers (even if you hire expensive high-qualified ones, they will be just as good as the cheap ones).
If you plan to scrap the whole thing in the near future (i.e. because the whole system is being replaced)
If the job is not critical, doesn't change often and forking a script is so easy that there's no point investing in anything better.
leg·a·cy [ˈle-gə-sē] - code that actually works.

Component Granularity Musings

2008-05-26T19:40:00.002+09:00

This article was written based on my experience with Mule ESB, but the general principles should apply for other similar products as well.

Short description of an ESB application

A typical ESB (Enterprise Service Bus) application starts with an endpoint, which is an abstraction for a way to receive some 'event' which would make us do some work. The event carries some information, which could be structured (i.e. JMS message or HTTP request) or simple (the timer fired) - we'll call this information 'payload'. The endpoint passes the event (containing the payload) to a 'component', which does some kind of work and optionally emits a single output event. The output event goes to an outbound endpoint which can send the result back to the client, write it into a database, relay it to an external system, or do something else.

This was a high-level description - the benefits seen from here are that we get to reuse the communication logic between applications and since the endpoints are abstract, we can easily swap the actual transports. Still, a typical ESB provides some additional functionality. In the example above, when we receive an event it was relayed to a predefined component, but that doesn't have to be the case. We can have multiple components configured in the ESB and use 'routers' to determine the component that will process this particular event.

The routers could determine the actual component depending on the payload properties (in which case we call them CBR - content based routers) or based on anything else (e.g. we can have a support call router which routes to different endpoints based on the time of day). When router receives an event, it optionally emits one or many events (unlike a component that optionally emits one). Also, components are supposed to contain exclusively business logic, while routers are mostly concerned with mediating the events in proper manner. Other useful types of routers include aggregating, splitting, resequencing and chaining routers.

For the sake of completeness there are also 'transformers' which can be plugged between endpoint and component to apply some transformation on the event payload, converting the data or adding additional information (enriching); and 'filters' which can be used to discard the messages based on some criteria.

Since the routers, filters and transformers allow us to conditionally specify the components and endpoint, we can choose whether we want to implement our logic inside the component or outside (in the ESB configuration).

Ultra-fine grained routing (anemic components)

When each component contains no control-flow statements (straight-line imperative code), all the application logic is implemented in the ESB configuration. Some would argue that a system implemented in this fashion is more flexible and allows for faster turnaround and better introspection. While these statements are true, we need to consider the other side as well.

First, you are mixing business logic and communication logic at the same level, which is bad since it forces the maintainers to understand both (ideally they should be separated as much as possible). Not only this, but your code becomes platform-dependent (the configuration is code and in this case the ESB becomes more of a platform and less of a glue). This in turn increases the chance that upgrades of the ESB software would break your stuff (especially if you use undocumented or experimental features). We also need to keep in mind that the domain complexity has not changed - we are just expressing it using a different language.

Second, chances are that you have more developers knowing Java than the ESB config language. Also, during development it's much easier to step through the code of a single component rather than trace a message as it goes through multiple queues and thread pools.

From operations point of view, there are more knobs to turn, which increases the chance to turn the wrong ones. The monitoring can take advantage that there are more inspection points, but then you need the consider the usefulness of this information. Do you really care to know how many executions have you got for sell orders and how many for buy orders?

Many vendors (IONA, TIBCO, Progress) offer proprietary process modeling and metadata management facilities, allowing to express your business rules, routing and transformations in graphical notation and enforcing integrity based on metadata. These are expensive products and well worth their money if your project is big enough and you want to accept the vendor lock in. In that case, make sure that you make the most of it and take the time to learn how to use them instead of putting a half-assed simplification layer on top of them (just an example - so far I've seen two wrappers around Spring (in different companies) aiming to make it 'easier' to use. No need to say that none of them had any documentation).

Ultra-coarse grained routing (fat components)

In this scenario we have one component only. Multiple inbound endpoints go in (possibly passing through transformers and filters); and a single stream of events goes out. We use a content-based router to dispatch each event from the output stream, to one of the multiple outbound endpoints.

This is a code-centric approach (you won't make much use from a box-and-arrows editor). It allows one to use Java and standard Java tools and debuggers to implement, unit-test and debug the bulk of the application, while still abstracting the communication code and mundane stuff like transformations. If you want to expose application details for monitoring you have to do it by manually registering your custom MBeans or using the Spring JMX exporter.

One sign that you should consider splitting your component is if you start doing threading. This includes maintaining worker pools, doing half-sync/half-async dispatching, messing with locking, synchronization and notifications.

Sometimes there might be better abstractions for some pieces of code. For example, if your component is implementing a simple generic transformation and you are using Mule, consider extracting a Transformer. On the other hand, if the transformation is simple, but you feel that it is not generic enough then you have a choice to a) move it to a transformer and have your component focus on the business logic; b) leave it there and reduce the number of classes you maintain.

The Fine Line between Fat and Voluptuous... Components

...this time it is not the J Lo's booty. Actually, as much as I've looked I haven't found a good set of recommendations about how to structure components in an ESB. There's some stuff from the SOA guys, but they have some ill gotten assumptions that all the invocation between services has to be remote and marshaled through XML, which is not necessarily the case for a single application using ESB as a platform.

I am in no way authority on EAI, but here is my attempt at defining some guidelines (take them with a big lump of salt):

Multiple components make your application complex. A good starting point for a new application is sticking everything in one component and extract components as necessary using the following guidelines. In general I've found it easier to split components than to merge components (Java's method-invocation semantics is less expressive than ESB routing).
If you need to checkpoint your processing, consider splitting the stages into different components and separating them with durable queues. The queues could be either JMS or Mule VM queues with persistence enabled. With proper use of transactions, this would allow you to survive application crash or do a failover (you still need to take care of any non-transactional endpoints). Alternative is to keep using one component and checkpoint using a distributed transactional cache configured with redundancy.
If you need to control the resources allocated to certain part of the processing, you can extract it in a separate component and change the number of workers using a SEDA approach (check also this article; the Scatter-Gather and Composed Message Processor patterns). Another valid approach is to use a compute grid like GridGain, giving you more functionality, but increasing the total complexity of the application. We should also mention GigaSpaces, providing a platform based on JINI and JavaSpaces with good integration capabilities.
When parts of the processing have different state lifecycle. It is best illustrated by example: we receive a stream of events on an inbound endpoint (let's call it 'control stream'). Each event has a payload consisting of condition and a 'data stream' endpoint address . On certain conditions, we want to start receiving and processing the events on the data stream in a way which requires independent state. The implementation would be to have one component that would monitor the control stream and keep track of the registered processors, creating new ones as necessary. The processors are concerned only with their data stream of events and each of them has separate state. A clumsy alternative (antipattern?) is to have a single component subscribed for all data streams and use a cache to lookup the state based on some key derived from the incoming event. A possibly better alternative for this specific case is to embed an ESP processor like Esper
If you want to extract a component to reuse it, consider extracting it as a domain object first. If it encapsulates processing and is ESB aware, then perhaps it's better off as transformer, router or agent (using Mule terminology here). So far I haven't had the need to reuse a business-logic component between projects.
If two components are tightly coupled and do not take advantage of any ESB-supplied functionality on the connection between them, consider hiding them behind a single Java facade. It's easier to unit test a POJO than to integration-test a component.

As always, any opinions are welcome.

Transformations using Domain Adapters

2008-05-25T11:41:00.002+09:00

Actually, this entry started as a reply on the Mule-dev mailing list.

...

What caught my attention was your transformation challenge. Specifically, how you decided to have a less anemic domain model and move transformations there instead of dedicated transformers (hope I didn't misinterpret it). Could you shed more light on this move? This could be an interesting pattern for some cases.

Andrew

Well, the idea is that your domain uses adapters to wrap the source data and the accessors perform the transformation in place. Since it's usually a straight mapping, we haven't found the need to cache the values.

The mutators also store directly to the underlying source bean, except in the cases where the value is derived from multiple input fields and updating them would break some consistency rules (actually in such a case, it would be better to avoid providing accessor if you can). A third approach is to have a map for changed properties and have all your accessors check there first and all mutators write there. This way you don't have to do a deep copy when you move the message using the VM transport.

The technical part is that there is a transformer, which has its source and output classes configured in the Mule configuration (I've had to add a custom setter for the source class). In the transformer initialization, it resolves a constructor of the output class, that takes a single instance of the source class as argument. The transformation itself is invoking the constructor with the payload. Note that the specified output class has to be a concrete instance in this case. Perhaps I could have done something similar using expressions but I like the type safety of this approach (if one of the classes is missing it blows at runtime).

Pros:

You can easily trace why the data is the way it is.
Adding new field requires changes to only one class (the adapter).

Cons:

At least the first layer of adapters is coupled to your source objects (if you use regular transformers, the transformer clearly decouples the src and output models). I would advice putting thest in a separate packages.
Needs better regression testing. Usually one catches a good number of breaking data changes in the transformation step. Since we transform on demand, this means that you either need bigger unit test or might have problems go unnoticed until integration testing
You lug a lot of data around, I can imagine that the serialization and cloning overhead could become prohibitive. In such cases you can have a method like Adapter.pruneStuffIDontNeed() that removes the parts of the input message that have not been used until now (you also need to track them).

An Integration Story or 5 Ways to Transform a Message

2008-05-24T03:54:00.006+09:00

It all started when we decided to replace Moxie with Devissa*. Moxie was a decent system and it had aged well, but its years had started to show. The rigid data schema, the inflexible order representation, the monoloitic C++ server... Don't get me wrong, it was and still is working great, but with the time we realized that we need something more. Something that would let us define the way we do business instead of having us change the business to fit in its model.

* All names have been changed to protect the innocent

The global roll out of Devissa looked like a good opportunity to bring in a more capable trading system. Devissa itself, was a huge beast, composed of hundreds of instances of several native processes running with variety of configurations, held together by TCL code, cron jobs and a templated meta-configuration.

The Moxie communication protocol was simple - fixed length records sent in one direction, 32 bit status code in the other, over a TCP socket (actually 2 sockets - uplink and downlink). Devissa was much more complex - the messages were framed using XML-like self-describing hierarchical format (logically it was the standard map of strings-to-arrays of maps... ending up with primitive values at the leaf nodes). The session level protocol was simple and luckily there was a java library for it (I'll bitch about it some other time). On top of the sessions, sit a bunch of application level protocols, each with different QoS and MEP. There is also a registry, authentication service and a fcache/replicator/database/event processor thingie that sits in the center, but I am digressing.

I'm actually started this article to share some interesting stuff I learned while we migrated the order flow from Moxie to Devissa. The phase-zero was to make a point to point integration Devissa to Moxie using the FIX gateways of the respective products, routing orders entered into Devissa to Moxie, so the traders could work them in the familiar Moxie interface. It allowed us to receive flow from other offices which were already on the Devissa bandwagon and it was great because we didn't have to code data transformations and behaviour orchestration logic - it all 'just worked'.

The next task was to make sure that we can trade on Devissa and still be able to produce our end-of-day reports from a single point. Right then all reporting was done from Moxie, so what seemed to make most sense was to capture the reportable events from Devissa and feed them back to Moxie. I'll spare you the BA minutae for now.

As we were looking for a suitable base for creating a platform on which to build various applications around Devissa, I shortlisted a couple of ESB solutions (although it's an interesting topic, I won't talk about "what's an ESB and do I need one"). I looked at Artix, Tibco, Aqualogic, ServiceMix and Mule. I found that Artix ESB was great, Artix DS looked like a good match for our data mapping needs, the only thing I was concerned was the cost. Before I get in contact with the vendor, I asked my managers about our budget - thay replied with almost surprise that we don't know - if it was good and worth the money we might try to pitch it to the global architecture group - in other words, commercial product was not really an option. This ruled out pretty much everything, leaving ServiceMix and Mule (if I was starting now I would also consider Spring Integration). I read a bit about JBI. I tried to like it, I really did... still I couldn't swallow the idea about normalizing your data on each endpoint and being forced to handle all these chunks of XML flying arround. At that time Mule looked like the obvious answer for OS ESB.

The first thing I had to do was to build custom transport for Moxie and Devissa. That took about 2-3 days. They didn't have any fancy features (actually they barely worked), but I was able to receive a message from one and stuff a message in the other. During the following year both transport evolved a lot, ending up with full rewrite last month, porting them to Mule2 and adding goodies like container-managed dispatcher threading, half-sync support, support for all Devisa application protocols and others.

The second phase was to build a neutral domain model as described in the Eric Evans's "Domain Driven Design" which I had read recently. Then I wrote two transformers - Devissa2Domain and Domain2Moxie, implemented a simple POJO with about 15 lines of real code and voila - all our Devissa orders and Executions appeared in Moxie. Forking the flow to a database was really easy, since I could use the Mule JDBC connector and it took only 10 lines of config. Storing the messages in XML was also easy with the Mule XStream transformer and the Mule File connector. The world was great.

Not really. It turned out that the DB storage and the file-based audit were not real requirements, so we cut them really quick (or perhaps they made the first release). Soon, during UAT, it turned out that even though the the BAs had created quite detailed requirements, they didn't match what the business wanted. Even worse - the business itself wasn't sure what they wanted. We were going through a few iterations a day, discovering more data that needs to be mapped, formats that need to be converted, vital pieces of information that were present in one model and not in the other and they had to be either looked up from static table or calculated from couple of different fields and sometimes ended up stuck in a field that had different purpose, which we were not using right now.

During all this time, the domain model was growing. Each new piece of information was captured clearly and unambiguously in a Java bean with strongly typed properties, validation and stuff. We went live on December 14-th. On the next day the system broke. We kept tweaking the business logic for quite some time and for each tweak, there were always three places to change - the domain model, the inbound transformer and the outbound transformer.

One day I decided to see what would it be if we drop the domain model altogether and replace the inbound transformer with isomorphic conversion from the Devissa data classes to standard Java collections and then use a rule engine to build the outgoing Moxie message. Enter Drools. The experiment was success - in a couple of days, I was able to ditch my domain model (which has grown to be so specific to the application that it wasn't really neutral any more). Drools was working fine, though I had the feeling that something was wrong... I never asserted, nor retracted any facts in my consequences - I was abusing the Rete engine. Actyally, all I was doing was a glorified switch statement.

While I was at it, I decided to ditch Drools as well and use MVEL - one of the consequence-dialects of Drools, which turned out to be a nice, compact and easy to embed language. MVEL is designed mainly as expression language, though it has control-flow statements and other stuff. With MVEL, all my transformation fitted on one screen and had the familiar imperative look and feel, but without the cruft. I was able to plug some Java functions using the context object, which allowed me to hide some ugly processing; and the custom resolvers allowed me to resolve MVEL variables directly from the Devissa message and assign them directly to the properties of the Moxie message beans.

Some time after that, for different project, building on the same foundation, I decided to see if I can infer an XML schema from the XML serialization of the Devissa messages. After some massaging I used that schema to generate the domain model using JAXB and tried to see how it feels. It was a disaster. A typical Devissa message has more than 50 properties (often more than 100). Usually you need 10-20 of them. Alsi, the generated property names were ugly. Even after conversion from CONSTANT_CASE to camelCase, they were still ugly. The automatically generated beans was practically unusable, the XML looked not-human-editable, the XSD was not adding any real value since it lacked any semantic restrictions, so the whole thing felt like jumping through hoops. In the end I dropped the whole JAXB idea and went with MVEL again.

3rd time lucky, beginning of this March, I started a new project. This time I again decided to try a new approach - in the inbound transformer, I was wrapping the raw Devissa message in an adapter, exposing the fields I need as bean properties, but carrying the full dataset of the original messages. It works well. One particular benefit is that you can always look at the source data and see if there is anything there that might be useful.

In conclusion I'll try to summarize:

Neutral model plus double translation can yield benefits when the domain is well known, especially if it is externally defined (i.e. standard). On the other hand it's a pain in the ass to maintain, especially if the domain objects change frequently.
Rule engines are good when you have... ahem, rules. Think about complex condition and simple consequence. Actually, in the original Rete paper, the consequences are only meant to assert and retract facts. Changing an object in the working memory or doing anything else with side-effect behind the engine's back is considered a bad practice at best or (usually) plain wrong. Even when using fact invalidation (truth maintenance), it has big performance impact.
Direct mapping using expression language works well, especially for big and complex messages. The scripts are compact and deterministic, which makes them maintainable. You might need to write your own variable resolvers and extend the language with custom functions. Also, debugging could be a nusance, but if you keep your control-flow to minimum and use plugged Java functions, it's quite OK.
Adapters are a middle ground between double translation and direct mapping. They tend to work well to provide internal representation for the application, you can also stuff some intelligence in them without worrying that somebody might regenerate them. With a bean mapping framework like Dozer you can even automate the transformation to the output datatype, though for many cases that would be overkill (sometimes 200 lines of straight Java code are more maintainable than 50 lines of XML or 10 lines of LISP).
Xml works well if your output format is XML; if you need to apply transformations with XSLT or render it using XSL:FO. As we know, you can run XPath on bean and collection graphs using JXpath; also any expression language can provide sililar capabilities.

Next time, I'll write about component decomposition, content-based routing vs coarse-grained components and how to decide whether to do the transformation in a component or in a transformer.

Obfuscating the GUI

2007-12-01T19:48:00.000+09:00

When I was working on mobile applications, obfuscation was mandatory part of the build. When every byte counts, you can not afford to have long variable names or carry extra stuff if it's not critical to the application functionality. In fact we didn't really care about the actual obfuscation (it's quite difficult to take an app out of the phone anyway, and even then, the success of a mobile game usually does not depend on some top-secret algorithms). Back then it was all about jar size.

The other day, I got the task to obfuscate an application that we wanted to ship to external client. The app was a SWT GUI, making use of reflection, runtime generics and runtime attributes. Also, the idea was to merge all libraries in the app jar and wrap everything in a native launcher. First I tried to merge the JARs. There was a small issue with the order of merging, since one of the libraries needed some file in META-INF, which existed in more than one jars, but overall no major problems (good that we didn't use OSGI).

Next step was the obfuscation. Obfuscating a moble app is pretty straightforward - you define all the library interfaces as seeds and let the obfuscator do the rest... errr, I guess that wasn't very clear, perhaps I should step back and take a look at ProGuard (my weapon of choice when it comes to free obfuscators), but the principles should apply to most of the products on the market.

The ProGuard obfuscation consists of a couple of stages:

Shrinking

Starting from a specified seed classes or methods, analyze the control flow and remove all the reachable code. The different obfuscators have different ways of specifying the seeds, the simplest ones being "keep everything which is not part of my source" (this is actually enough for a mobile application) of "keep everything". You also need to include here any class which is accessed by reflection only (think plugins and DI), native methods, classes accessed exclusively from native code, classes used as default values of annotation attributes unless you always specified a proper value, etc.

The shrinking also removes all the attributes from the classes fields and methods. If this doesn't mean much to you, you are not alone - one usually doesn't think about what's in a class until things start breaking (and break they did).

The first problem was that all stacktraces did not contain line numbers. That was actually easy to fix - just keep the LineNumberTable attribute and replace the SourceFile and SourceDir with fixed string - both are quite easy with ProGuard.

Next problem was that the DI container could not read the generic attributes from the collections and was sticking inside strings instead of URLs. Again - the Signature attribute contains the information used by the runtime generics reflection.

Then I found that none of my runtime annotations were kept. After some time spent staring dumb at the JVM spec (Chapter 4), I learned that the annotations are kept in anoher set of attributes - namely RuntimeVisibleAnnotations and RuntimeVisibleParameterAnnotations. The annotation default values are kept in an AnnotationDefault attribute of the corresponting method in the annotation class (or interface if you prefer) - you can strip these if you specify explicit values for all annotations.

There were also some attributes related to enums, but it looks like they are not used at run time.

Optimization

Not really sure what it does exactly. I have seen it reduce the number of methods, but it has really only two settings - "optimize" (yes/no) and "number-of-passes". I guess that each pass does one level inlining if a method meets certain criteria, but it might also do many other whole-program optimizations.

One thing which might be interesting is a profile-guided optimization like the Intel C++ compiler, where the optimizer would first instrument your classes, adding probes to your bytecode. Then you would run your app a couple of times to generate execution profiles and then optimize your app using them. Of course that's partly what the Hotspot already does, but not everybody uses Hotspot and in any case it wouldn't hurt if the code takes the right branch without jump in the majority of the cases.

Another possible profile-guided optimization would be to identify the order of loading classes and separate them by that - the early loaded in one jar, the latter loaded in another and the barely used ones in third - it can reduce the classloading time (if you put them on the classpath in the right order) and combined with the Java Modules proposal can help one create slimmer applications where you can start the app with the minimal jar and the rest is streamed as you work.

Obfuscation

The goal of the obfuscation process is making your code more difficult to decompile. Please note that I didn't say "impossible" - although decompiling obfuscated code exposes much less information and usually does not produce runnable Java code, it is perfectly possible for a motivated person to reverse-engineer obfuscated bytecode - it's just going to take longer. In the end it boils to the cost/benefit perception - if somebody thinks it will be cheaper to hack your product, they will - the obfuscation raises the bar to do it, but if you really care about the bottom-line, you might be better off with ~~open~~available source and certain legal agreement (NDA, NCA and in some cases even patents might make sense).

Class/Method Renaming

The goal of the renaming is to make the classes and methods illegible. Usually this is achieved by changing the names to short identifiers (usualy one or two letters). This also reduces the size on disk and the perm-size by reducing the constant pool. If you specify the option to use lower and upper case letters for different class names, you can make the jar impoissible to extract on case-insensitive file systems as half of the classes would overwrite the other half. Another trick is to specify a dictionary of recommended identifiers, which contains all Java keywords. Since the keywords have meaning only in Java, but not in the bytecodes, a naive decompiler might produce funny uncompilable code (imagine for (int for=if; for<while.lenght; for++) else.add(while[for]);) - of course JAD handles this by recognizing and renaming the members, so I really consider this a wasted effort.

note: again, you will want to preserve the public interfaces, which is quite similar to the specification for the shrinking phase).

Flow Mangling

Since the decompilers recognize certain byte code patternsas result from a java statement, the obfuscator can reorder these, yielding semantically equivalent bytecode, which is impossible to map 1:1 to Java (JAD handles these with labels and goto). Also, I've seen obfuscated code using loops, breaks and exceptions to simulate IFs, but I'm not sure which obfuscator does these. (Zelix?)

String Encryption

I think it was Zelix KlassMaster that could substitute each string with encrypted version and insert code to decrypt them at runtime. This is very efficient measure as the strings usually give away a lot about what the code is doing (especially logging statements.)

Stack Map Generation (Preverification)

J2ME JVMs feature simplified class-loading mechanism which requires each method to declare how much stack space is it going to use in the worst case. The J2SE JVMs are smart enough to do this at runtime, stil this slows down the classloading. ProGuard can generate the correct StackMap attributes for the obfuscated code, so for slightly larger disk footprint you would get faster loading.

So that's about it. I figure that here is the place to throw in a couple of URLs:

ProGuard - free and quite decent. Nothing fancy.
yGuard - some people like its XML syntax. I don't think it's much different than ProGuard. The company producing it requires that you use it if you use their core product. It makes sense for them to want to take care about the actual protection of their IP.
JoGa was another tool that I used for J2ME, focused on bytecode optimization and had a nice GUI with many tweaks and gadgets. Unfortunately, it looks like the site is down.
Zelix KlassMaster - comercial - implements string encryption and advanced flow obfuscation (this is what JetBrains use for IntelliJ IDEA).

So all in all it took me about 6 hours to get everything obfuscated and in one jar. I had to disable the optimization and shrinking phase because I couldn't hunt down all the SWT JNI dependencies, still the resulting size was 2/3 of the original and the app was starting up noticeably faster.

The final touch was to wrap the single jar in Launch4J binary launcher, so the user would need resource editor to even get to the jar. Launch4J provides some small but nice features like JRE detection (from registry), JRE version checking, custom icon and Windows metadata.

Build Tools

2007-11-03T20:38:00.000+09:00

I had started writhng a post explaining how the Maven repository, artifact resolution and build lifecycle work, but I figured that I'm repeating the Maven Book and the Bullet-point Guide. Instead, think it would be more interesting to go down the memory lane and talk about how my build tools have changed through the years.

Why Build?

I stated programming on Apple][ using Basic, and at that time I stored my programs as source on a 5¼" diskette. Every time I wanted to run a program, I typed 'load MyProg <ENTER> run<ENTER>'. No compilation, no packaging, no complicated dependencies. It all Just Worked™.

Integrated Development Environments

Then, new machines came about, with bigger keyboards, bigger hard drives, bigger screens and (oh, horror) no built in Basic. At that time somebody told me that the Real Programmers don't use Pascal - somehow I failed to get the tongue-in-cheekness - it all made a lot of sense to me. Finally, after I couldn't find a Fortran compiler for IBM XT (actually Pravetz16), determined to become a Real Programmer I settled for QuickBasic. QB was very powerful - it had functions (which I thought were something like GOSUB but with a name) and you didn't have to put line numbers (still I didn't trust it, so I usually typed them in just in case). It also had a number of new commands, and almost none of the Apple II ones. Some time around 1992, I can't remember what happened, but I abandoned QBasic and joined the quiche-eating side of the power - I switched to Turbo Pascal 4.0.

Turbo Pascal was a big jump for me - it hade interesting new abstractions like units and scoping (the latter one being useless feature that only stops you from seeing your own variables), but one notable feature was that one program could be spread over multiple files. At that time text UIs were all the rage, so I had my own library for drawing animated windows, menus, etc. The whole thing was one file and I wrote wrote a couple of toy-apps, each of them having its own copy of the Library (notice the capital letter here). Every day I wanted to show my mom and dad to "what the computer can do" and I tried very hard to convince then start using my expense-tracking app (needless to say, my attempts were futile... My sister had a much better success at trying to get them to eat from her first cake).

At that time I didn't realize how much the IDE was doing for me - all I knew is that I press Ctrl+F9 and a couple of seconds later I get an EXE in the output directory. There was no packaging and I couldn't figure for the life of me why would anybody want to compile outside of the IDE.

Make?

As time went by, the IDEs changed (Turbo Pascal 5-7, Turbo/Borland C++ 2-4, Visual C++ 5-6), but my attitude stayed the same. Come summer 1999, I was working part-time as a developer in a small company and all of us ~20 developers were happily building release binaries with Visual Studio. There was a lone guy that tried to propose to use an obscure utility called make. It looked like you have to write yet another program that would do what the IDE does, but you needed to use obscure syntax, and call the "compiler" and "linker" directly, specifying every command-line parameter, and listing filenames manually - it was a lot more work. The benefit that he tried to put forward weren't very convincing either: "you can build from the command line!" - countered by "and why would you want to do that?" or "people that don't use Visual Studio can build the project", retorted by "are you crazy? Everybody uses Visual Studio." well, that guy was actually using VI... I think he didn't last very long there.

When I joined my next company I had to use Java. They were not using Visual Studio and in fact they didn't have a standard Java IDE. At that company they were building using make. Recursive make. Every directory had its own makefile, most of them containing only boilerplate code, including a toplevel template (quite annoying when you have to debug a build issue and count the number of '../' in the include), and to make it more interesting, some of the makefiles were not using the template, having their own goals, invoking OS commands, etc... Overall it worked (except when it didn't.) Most of the problems we had were related to incorrectly set environment variables and missing external programs. It was difficult to reason about the build process as the build files was spread all over the directory tree. In the end, one of the developers rewrote the whole build using Ant.

Ant

Ant worked. Much better than make. Looking back, I can say that this was because:

Ant is much less dependent on environment variables. In that case there was an build.properties file that everybody had to customize once and that was it.
Ant does not use OS commands. Everything an Ant build needs is either provided by the distribution or shipped with the source files (you don't have the habbit of plopping random jars in your $ANT_HONE/lib directory, do you?)
Ant's syntax is much more restricted than make. A syntactically invalid Ant script wouldn't run; a syntactically invalid makefile can erase your harddrive.
Ant was designed for Java, handling many common tasks right out of the box.

Many detractors say that Ant is too verbose and they are right. I personally don't have big problems with this as my editor usually autocompletes the tasknames and the attributes for me and warns me when I make a mistake. The modern Ant (1.6+) also allows you to factor your build fairly well by using includes, presetdefs and macrodefs. Actually Ant's biggest problem is that it is Turing complete. The target's dependency resolution, combined with the if and unless attributes is often abused to simulate control-flow statements, which pollutes the target namespace and complicates the dependencies. Too often the targets don't have good names because their only purpose is to hold a piece of code reused in some other targets (actually this use-case is served better by macrodefs, but many people still use targets). The assign-once-ignore-following semantics of the Ant properties is good for implementing overriding, but when we use Ant as a language given the lack of scoping, the namespace gets polluted really quickly and you might end up having strange interactions between unrelated targets.

As experiment I've tried useing Ant tasks from Jython. It works great - you have real variables, real control structures, the code is much more concise and you can use any other Java library you want. One downside is that for straight-forward builds (compile bunch of files, package them in a jar and zip them with some scripts) Ant is arguably easier to read, as there are fewer things one needs to be aware about. But the real dealbreaker is that you don't get any tool support - no IDE autocompletion, no syntax checking on-the-fly, no integrated build runners, nothing!

One thing I didn't mention is that Ant is very easy to customize - extend a class, provide some getters and setters, write your logic in the execute method and you are done! To use your custom task, you need to ship your jar with the build script and add an one-line definition to your build. If you have more tasks you can package them together with a simple descriptor and import them all at once using a namespace (this is called antlib).

Make!

Few years later, I was porting Java games for mobile phones from Doja to EzAppli and VSCL. I had common scripts for each of the platforms and every time I started a new port, I just had to tweak a template-script containing an import statement and a couple of properties. If I needed to port the same game for another platform, all I needed to change was the include statement. That was nice.

One day I got a port for a new system - it was called BREW and the API was in C. Initially I considered writing some Ant tasks to handle the native toolchain, but after some consideration, I read a couple of articles (see Recursive Make Considered Harmful) and decided to give make another try.

One of the useful make features is the dependency inference rules. This way, you can say that a *.c file generates *.o file. Then you just specify which *.o files your binary is comprised of and make will automatically guess your source files. If the source file is not newer then the object file, make is smart enough not to recompile it. The Java compiler does this by default (when using wildcards).

Make does not care about what commands you put in its goal definitions. That's why, out of the box it doesn't deal with transitive dependencies. To deal with this, most C compilers can generate a dependency listing in make format, which you can include in your makefile and regenerate when the dependency graph changes. In Java, the same thing can be achieved by using Ant's dependset task and some IDEs (like IntelliJ IDEA) can track all your dependencies (including transitives) as you type and recompile all impacted files.

In the end, I had a pretty well factored build system using make, requiring minimum configuration (much like the Java one), allowing for cross-compilation targeting x86 and ARM architectures, using different toolchains and everything. If I compare the Ant/makefile approach with the IDE, I'd say that the build scripts take more time to pay off. If you work on one project and your build is not complex and you don't need repeatable builds (because you work alone and your customer doesn't care), then the IDE might be a better proposition.

Shells, Perls and Pasta

Once again, I started on a new job and it turned out that in my department nobody uses a build tool. Everybody was usually building in their IDEs and copying straight to production or using ad-hock shell scripts or perl to build from the sources directly on the production box (the latter was rationalized as "this way we can fix bugs faster").

In the end, all scripts were simple compile+jar, sometimes even skipping the 'jar' step. They did get the job done and the business was happy. There were a number of things missing, like reproducible builds, reliable roll-back, etc. but it is a matter of tradeoff whether one wants to spend the necessarry time studying and implementing a build system or spend the same time implementing new functionality or fixing application bugs. There's nothing wrong with either way.

Maven

After spending some time working on an application with Ant Build from Hell, I was dreaming of a brave new world, where each application will be layed out in modules and packages with controlled dependencies, each module's build script would be simple and clean and one can focus on the actual application functionality.

Enter Maven (actually Maven2). After being burned by Maven1, I still thought that the ideas were good, and it was the actual implementation that sucked so bad. Maven2 is a new start, and a new chance to reinvent the wheel. The project developers have taken the working concepts from Maven1, pruned the ones that turned out to be a bad idea, and reimplemented everything from scratch. It's still not clear why did they decide to use their own DI container and classloader management (instead of say Spring and OSGI), but it works.

Maven has the chance to hit the sweet spot between a build-scripting tool and an IDE-style pure declarative build. In the core of Maven is the build lifecycle, which is just an abstract sequence of steps. Then, in your POM you define (or inherit) a packaging. The packaging defines a set of default plugins and executions. You can think about the plugins as a bunch of Ant-tasks (or 'goals' in mavenspeak), which are versioned together. The executions define parameters for the actual goal and are bound to a lifecycle phase.

Most of the parameters in a goal are optional, using sensible defaults. The defaults are either sensible constants or references to different parts of the POM. E.g. the compiler:compile goal would get the source directory from the POM reference ${pom.build.sourceDirectory} and use the constant "false" for it's fork parameter. All the POMs (or their parent POMs) in Maven2 inherit from a common "Super-POM". The common POM specifies many defaults (i.e. directory layout), so you don't need to, if you keep to the Maven Conventions. An important part is complying as much as possible to the standard Maven Directory Layout - it makes your life much easier.

There are some areas in Maven that are still rough. The release plugin is still quite limited (although there is work under way to implement features like release staging and binary promotion). There are a couple of annoying bugs in the assembly plugin, which are fixed on the head, but not released for more than an year. Some issues (like the explicit support for aggregator plugins) are being postponed for Maven 2.1 (which will probably ship around Q2-3 of 2008). But overall, I think it is an improvement.

Conclusions

So, I'm planning to use Maven2 for the time being and perhaps write a plugin or two for some tasks which it does not handle well (right now I'm still cheating, using the antrun plugin). I'm still (ab)using Ant for common scripting tasks like restarting a remote server through SSH connection, deleting files on remote machine, setting up a database table or deploying configuration files in remote environment. All these things do not fit in the build lifecycle and wouldn't benefit as much from writing Maven plugins for them. The main benefit in this case is that I get a simple, completely cross-platform scripting language, providing many common commands lacking from the normal Unix environment (btw did I mention that expect sucks?)

And finally, here are some more tools that I'm planning to check out:

scons and rake - build tools using Python amd Ruby respectively, each of them using the underlying platform and some clever code for doing build stuff
buildr - another Ruby tool that builds on rake, allows you to use Ant tasks, designed as a drop-in replacement for Maven2 (hopefully allowing for mixed environment).

On Maven2

2007-10-27T23:57:00.000+09:00

Maven is a tool with an interesting history dating back to 2001. In its first years it got deservedly bad reputation for being unstable, poorly documented and more or less experimental piece of work. The release of version 2.0 in 2005 fixed many of the early quirks and set right many of the short-sighted design decisions. After having some bad experience with Maven 1, I was weary to get on the M2 bandwagon, but when I moved to a new job in 2006 I decided to give it a go. So far there have been ups and downs, but I'm fairly happy with it. I still haven't abandoned all my Ant and shell scripts, but I find that I'm using Maven as a primary building tool for most of my projects.

The core proposition of Maven is that one should be able to declare what they are building in some sort of manifest file and the build tool should be able to figure how to build it. The manifest should contain only the information that is specific for the project and all the build procedures should be implemented as plugins. Each build should be related to exactly one artifact of certain type. The artifacts are stored in repositories (more about this later.)

In Maven parlance, the manifest file is called POM (that stands for Project Object Model). If a project adheres to a predefined filesystem layout, the actual XML one has to write can be very small. Here is a minimal example:

<project schemalocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd ">
  <modelversion>4.0.0</modelversion>
  <groupid>com.acme.foo</groupid>
  <artifactid>foobar</artifactid>
  <version>3.14-alpha</version>
</project>

This configuration is enough to enable Maven to build, package, install, deploy and clean your project. It would even generate the repots site if you run mvn site. That's the way I usually start my projects. Later, you can tack more elements as needed.

The top-level element defines the default schema for the POM. The schema has a version which has to match the modelversion element below. While the schema is not strictly necessary, it makes the POM editing much easier if you are using XML aware editor. It is well constrained and there are few extensibility points, so the autocompletion works very well. It also features annotations for each element, which means that if you use IntelliJ IDEA you can just press Ctrl+Q on any element and get instant documentation. The schema design is a bit annoying, but it is very regular and easy to understand:

No attributes - everything is an element.
No mini-languages - everything is an element and any custom textual notations are avoided as much as possible (though in many places they use URLs). This allows for simple parsing and processing.
If an element can be repeated more than once it is enclosed in container element that can appear only once. This ensures that all elements of one type are textually next to each other. Each container can contain only elements of the same type.

Overall, the POM bears a resemblance to an IDE configuration file and they serve the same functionality. Both Maven and IDEs runs a predefined build process, parameterized by the information in the project file (or POM). One major difference is that Maven is designed to be ran from the command line and also encapsulate all environment-specific factors into the POM and the settings.xml files. You can use Maven to generate project files for IDEA, Eclipse and Netbeans based on the information in the POM.

The machine-specific and user-specific configuration is specified in the settings.xml files. There are two of them, the machine-specific settings are stored under the $M2_HOME/conf directory and apply to all the users on the machine. Usually the contents is standardized within the team (internal repositories, proxy settings, etc.) In our company, this file is posted on the wiki where everybody can download it. Alternatively we could have built our internal Maven distribution with the file pre-included. The second settings.xml file resides under ~/.m2 and contains user-specific settings overriding the machine-specific. One can use the user-specific settings to keep login credentials, private keys, etc. On Unix machine, this file should be readable only by the user.

Though the POM is very flexible and can be tweaked to accommodate a number of different scenarios, it is very recommended to refrain from overriding the defaults and use the Maven conventions as much as possible. This way, new developers on the project can get up to speed faster and (important) it's much less likely that you get bitten by untested plugin 'feature'.

One of the thing that new users tend to dislike most is the standard directory layout. In brief, you have pom.xml in the root, and your files go under a directory called src. By default, all files (artifacts) generated during the build go under a directory called target which helps for an easy cleanup. Note that there is no 'lib' as all the libraries reside in the local repository (more about this in another post.)

So far so good, but then under source we usually have main and possibly test, integration and site directoryes and then under them we have java, resources and only there we put the actual source files. This means that we have at least 3 directory levels used for classification above our sources and if you jump between them using Windows Explorer or bash it makes for a lot of clicking/typing. On the other hand, this is the price one pays for the Maven's magic - each directory level means something to the plugins that build your project. E.g. the unit test goal knows that it should runt he tests under test and not the ones under integration. All the files in the resources directory are copied in the final JAR, while the ones under classes are not and so on and so forth.

This post became rather long, so I'll finish it here. Next week I'm going to cover Maven's dependencies management and repository organization. Again I'll try to talk more about "why's" and less about the "what's" that are already covered pretty well in the following tutorials:

Ninjava Presentation

2007-10-26T00:44:00.000+09:00

First Post!

A month ago I volunteered to give a presentation at Ninjava (a Java User Group in Tokyo). At first it looked easy - the topic was "Java Project Lifecycle" and that's something I've been doing for the last two years, so I should have a lot to say, right? Well, this actually turned to be my first problem - after putting my thoughts on paper, I realized that there are too many things and there is no way that I can fit the whole talk in one hour.

Since everything looked relevant to the main topic and everything looked important, I figured that I needed to tighten the scope. I thought that I can focus on a minimal process and how we can use tools to automate the chores like building, releasing and configuration management. In the end, I put together this plan to build a small application, starting with a domain layer (actually that might be too big of a word for a class multiplying two BigDecimals), test, add command-line interface, release, add simple GUI, release, show how the whole thing can be split into 3 modules and packaged in different configs - GUI only, CLI only or both. Finally I was going to add a feature on the head and backport it to a stable branch. I planned to use the following tools:

Subversion - version control system similar to CVS.
Maven2 - project automation tool from Apache. The tool is build around the POM (Project Object Model). The POM is an abstract model, but in practice, it is usually an XML document which describes information about the project. The POM does not describe what actions can be performed on a project.
Artifactory - repository management server for Maven2. Usually it's configured as caching proxy in front of the internet repository, sindicating its contents with the internal repo. The deployment can be done either through WebDAV, HTTP PUT or a convenient web UI for manual deployment.
TeamCity (check out the EAP for the 3.0 release)- a continuous integration (CI) server from JetBrains. Apart from the usual watch-and-build functionality, it also features some interesting stuff like conditional commit (send the changes to the server which will commit them only if the build passes), duplicates analyzer and static analysis tool working on AST level, plugins for many different IDEs, etc.
Jira - an issue tracker. Actually I should clarify - the best issue tracker I've used (especially with the Green Hopper plugin and the JIRA Client GUI)
Confluence - a damn good wiki. In our department it also doubles as reporting tool.
Fisheye + Crucible - a server indexing your version control system, with web interface for browsing changesets, files, revisions, filtering by committer and full text search. Crucible is an add-on to Fisheye, that lets you select a bunch of files and create a code review ticket, adding inline notes. Multiple people can collaborate, posting replies to the notes and proposing changes until the ticket is resolved.

The main factor for choosing these tools was that I am currently using them either at work of in the development of the Mule JMX Transport (BTW, all of them have free licenses for open-source). The first rehearsal was one evening when I stayed after work and went through all the steps on my workstation - it took me about 40. Perhaps 10 mins I spent looking up documentation on the internet, so I thought it was not too bad. On the presentation day I took a day-off from work, so I can have the time to set up Daniela's laptop as my primary demonstration machine (as I had already installed all the server apps on my poor Kohjinsha) and everything was going fine, until I started the dress rehearsal. After I chased Daniela out of the apartment, I went through the already familiar steps this time talking as I would on the presentation. Oops... it turned out that 15 minutes were gone and I was still at the very beginning (and I didn't even manage in detail explain what I was doing). Attempt #2 - this time I consciously tried to only say what I am doing and not what I am achieving or why I was doing it. This time the speed was much better, but I found out that I was getting these ~30 sec pauses when switching active files or when searching for symbol, which were ruining the whole flow.

A quick look in Sysinternals' ProcessExplorer and ProcessMonitor showed that the working set of java.exe is 300mb and it's paging like crazy. I stopped some services, closed JiraClient and all the other applications and tried again. This time it was better - it was also clear that this is not going to work. I was at step 13 and I was well past 1 hour in the presentation... The wall-clock was showing 4pm... It was to late to change the scope, so the only thing that seemed to make sense was to just go out, start talking and try to provoke questions, which will help me focus on something.

I showed up at Cerego around 7:30 (after meeting Nikolay at Shibuya station) and while I was thinking which kind of excuse should I use, when Peter proposed that I could use one of the machines in the conference room. There was nothing to lose, so I settled to set up an environment in the remaining 30 mins - I downloaded Maven, IDEA, JiraClient, JDK, set up environment variables and just in case checked out the JMX Connector. I tried to set up Jira Client against the JIRA instance running on the Kohjinsha, but for some reason it didn't resolve the host name, so I decided not to bother. By the time I finished there were about 10-15 people in the room.

I started talking the usual stuff about how some tools work well in some situations and suck in others (except SourceSafe, which sucks in every situation); how the primary criterion in choosing a is that they should help us get the job done in less time/risk and blah, blah.. (more about this in a latter post) and then I moved on to my current environment and to Maven. Then Curt had a question about the POM, then another and then just one more... well, I guess I have to thank him for this. Overall, we talked a lot about Maven, went quickly through TeamCity, just mentioned Crucible and didn't even touch most of the other stuff.

Now I realize that even my cut-down scope was huge and it was impossible to cover everything in one hour with enough detail. The other fundamental problem was that I started the presentaion without having "something to sell". That's why the whole talk was lacking direction, so only when I started "selling" Maven it started taking shape. It would have been better if I had picked Maven from the very beginning, so I could prepare more examples and not waste time with general stuff. In the end, people said they liked it and may be some really did.

Yesterday it came too me that a blog would probably be a much better place for all the ramblings I wanted to share, so here we are. This is the first post I'll try to publish at least one post weekly (hopefully this time I won't lose interest after the first few weeks)