Thomas Ferris Nicolaisen's blog

Metrics

2009-09-05T18:41:00.005+02:00

The last couple of weeks I've been spending a little effort on getting some code metrics up and running. I've blogged about measuring in software development before, and I think code is definitely one of the easiest and most important things to measure.

The most important thing about metrics is that you monitor them. I am of the firm belief that if you run a static code-analysis report manually, it will give you very little. Your team will say "wow, we have 3000 FindBugs warnings", fix a couple of them, and then forget about it. Then the next time you run the report, you'll find new bugs that have crept in, with no idea of when or who did it.

There are two ways to track these kind of metrics:

IDE warnings
Continuous integration reports

IDE warnings

Most of you have probably enabled the built-in warning system inside Eclipse (unused method, potential null-pointer, etc). Some of you strive to minimize these numbers of warnings, some of you have perhaps even turned on more than the default warnings in Eclipse. The key practice which I recommend you all adopt is the zero-warning-policy.

(I couldn't find a good place to quote, so I'll come up with my own definition:)

The zero-warning-policy is a programming practice which states that at any time, the number of warnings in a code base should be zero. With this in place, it is easy to discover and take care of any violations immediately, before commit. Gradually, new warning rules are introduced to the developer's team, and the existing violations are promptly removed, making sure the level is back to zero before every commit.

After some time, you may find that your team has exhausted the IDE's warning policies. At this point you have to introduce plugins to the IDE to facilitate more warnings. Now it gets a bit tricky, cause every team member needs to have the same IDE configuration.

Continuous integration reports

This approach works fine in coalition with the IDE warnings. The clue is to implement the warning policies into your CI-server. Some teams practice zero-warning-policies also here, the strictest variety would be that any violation will actually break the build. This is a good way to enforce zero warnings, but can be a bit too strict for most teams.

There are two strengths of CI reports: Firstly, they are centralized, so there's no need for the team members to set up any configuration or plugins. Secondly, they can track metrics historically, so you can see from day to day whether the number of violations are decreasing or increasing.

Historical data is a good substitute for the zero-warning policy. You may have 1000 violations in your code-base, but the key is to discover when new ones are introduced, and catch the violating developer in the act of producing bad code.

It is also interesting to track metrics over time to see whether coding practices have any effect on the quality of the code. Did introducing pair-programming/commit-mails/TDD lead to fewer violations? Now you have proof.

The landscape of metrics

Now that we have some motivation for putting code metrics into play, which tools should we use? There is a number of alternatives, both commercial and open source. Googling around will probably set you in the right direction, but here are my experiences:

Which reporting tools?

I like to divide into a families of metrics:

Test coverage: Crap4J, Clover (commercial), Cobertura and Emma. These tools also usually include some notion of complexity (execution paths), as coverage is a product of complexity.
Code style: CheckStyle, PMD
Duplication: Simian (commercial), PMD
Static code analysis: FindBugs, JavaNCSS, JDepend, Complexian

We played around with these tools, and found that the best combination for us now was JavaNCSS, FindBugs and PMD. We still haven't got enough test coverage to even bother measuring how low it is, unfortunately.

Which IDE?

Both Eclipse and IDEA have a good number of built-in warning policies, and most of the tools above come as plugins for both.

Ant or Maven?

Most of the tools above come as either Maven-reports or Ant tasks. Even though I'm a big Maven fan, I found it to be quite easy to set up all the tools with Ant. The Panopticode project, which aims to provide a number of metrics into one easy setup helped me a lot with finding the way to structure this into our existing Ant project. If you've got a Maven project, introducing a report should be as easy as five lines of XML in your pom.xml.

Which CI-server?

I've been using Hudson for so long now that I didn't bother with trying out anything else, but I'm guessing that none of the other products come close to Hudson in the number of tools supported by plugins. If I did have the money for Atlassian's Bamboo, I would definitely give the Bamboo + Clover mix a run.

Run it nightly

Finally, I found it best to not run these metrics after every commit. They take too long, and nobody inspects the changes in number of warnings from build to build. Instead, we run metrics nightly, so every morning we can see that there are X more/fewer violations in the code base. I'm really happy with how well it ended up working, and I hope this lends some inspiration for you to try the same.

Happy measuring!

Reason 3: Don't build everything all the time

2009-04-26T19:24:00.005+02:00

This post is a part of a tiny series I'm doing on why we use Maven, and you should too.

Previous posts:

Huge projects are hard to work with
Many projects that have grown over time will become too large to easily work with. The workspace gets too many classes, the IDE slows down and the build takes too long. If the developers focus on separating the concerns of the classes into packages of functionality, these packages can be organized into modules. As the contract and concern of a module becomes more defined, its rate of change will fall to a lower frequency than the rest of the project.

Once a module's activity has slowed down sufficiently, you can retire it into an external project, build a JAR-file from it and use this ready-built artifact as a classpath element for building and running the software instead. If changes occur later on, you can re-build the JAR-file.

Examples on where this might be appropriate:

Low level libraries and utilities, much like Java's own java.util package only changes every other year or so.
Contracts or adapters around external services, because integration with 3rd parties require stable data-formats.

What Maven does
Maven eases the work of maintaining such dependencies. As projects "subscribe" to their dependencies, they pull in the latest appropriate JAR-files automatically (snapshots) or by choice of versioned artifact. In an environment with a continuous integration server and a maven repository, the library developers need only commit their changes, and the artifact will be automatically built, deployed, and pulled into the depending project. A good term to describe this setup is Enterprise Maven Infrastructure (see this link for more information on how to actually implement this),

Again, understand that it is better to subscribe to your dependencies than to push them in.

Library developers do not have to deploy their artifacts to all clients
Clients keep the control of which libraries they use, in which version
Easier to maintain transitive dependencies (see my previous post)
Getting newer versions is easy

Just to clarify the last point: When you want a new version of a library, you change the version number in the pom.xml. If you want the latest/greatest build (handy for an adjacent project, or a library under heavy development), you make it a snapshot version number, and suck in the latest JAR file from your maven repository on every build.

I could go on on how bad it is to work on a project with a single huge build, but I imagine most developers out there are either struggling with it right now, or have done so some time in the past. There are many reasons why you want to break out modules of your project for architectural and code-design/quality reasons, but in this post I chose just to focus on the basic reasons. I came across one video from SonaType that illustrates this situation very well, so if you didn't get it from reading this post, please take a look at it (although the fancy staging functionality of Nexus is not that important, nor the point of this post).

Reason 2: Clean up your JAR-files

2009-04-13T01:50:00.002+02:00

Update: Added a summary section at the end of this post.

This post is a part of a tiny series I'm doing on why we use Maven, and you should too.

Previous posts:

Some background

Back a couple of months, I got the rewarding task of cleaning up our project's lib folder. You know the one: Crammed with JAR-files of various versions of the various dependencies your project has..

fizz-project
\
fizz-core
fizz-web
fizz-libs
 \
   junit.jar
   spring.jar
   common.jar
   lots and lots of others...
   ...

Yup, good old fizz-libs. It needs to be regularly cleaned up to reduce software rot. Over time, the developers try out new open source libraries and remove the use of old ones, but they seldom take care to clean out the libs-folder, because they don't know if there could be any hidden effects from removing JAR files.

Now, at our place we use Eclipse's .classpath file for specifying each module's dependencies. Each module is explicitly configured with which JAR-files (within fizz-libs) it depends on.

Unfortunately, Eclipse's .classpath file does not support any automatic reporting for analyzing and discovery of unused JAR-files. So I grabbed Jar Analyzer and set it loose on our libs folder (by the way, the author of the tool, Kirk, has a blog full of good thoughts on JAR-design, lately taking a humanly readable take on OSGi, recommended).

Jar Analyzer scans for compile dependencies, meaning that it can create a tree of which JAR-files are needed to compile which JAR-files that are needed to compile these JAR-files, and so on. You get a very nice report/graph which shows you all the JAR-files and why they are in there.

You can also see the JAR-files that don't have any connection to your code, remove them and their children. What I found in our libs folder was that about 20% of the 150 JAR files in our libs folder were unused at compile time, and these were potential JARs to be removed.

The big aber is that you don't get any hint on which JAR-files are used only at runtime by means of discovery and reflection. And this is where the real work begins.

The only way to find out whether a JAR file is used at runtime is basically to take it out, start up your application and test every functionality. If you have an application of moderate size, performing a 100% regression test takes many hours. So in practice, I ended up doing alot of guessing, quick and dirty testing, and asking around to find out which of the runtime dependencies were actually in use.

In the end, after two days of researching and testing, I ended up removing half of the compile-time-unused dependencies, crossing my fingers I didn't break anything in the process (which I did anyway).

The experience left something to be desired in the way we define dependencies: Eclipse's .classpath file simply does not allow you to express how and why which dependencies are in your project. You need something else: A tool that can define scope, version and transitivity of dependencies.

Scope

The total classpath for our running application is expressed in a dependencies module, which is an Eclipse project that solely exists to be used by Ant to build up which JAR-files should be part of our deployable WAR-file.

This dependencies module is alot like our web module, but it excludes references to JAR-files which are already available in the runtime of our application server. You could say that these JAR-files belong to a certain scope: they are not needed at compile time, and they're not needed at testing-time. Their scope is limited to runtime.

Another scope we often talk about is testing. These JAR-files are only needed for running and compiling tests. JUnit and mocking toolkits are typical examples. A good reason for keeping JAR-files with this scope seperate is that you do not want these JAR-files deployed along with the rest of your application.

So, let's take another look at our libs folder:

   fizz-libs
 \
   junit.jar  (test scope)
   mock.jar   (test scope)
   spring.jar (compile scope)
   common.jar (compile scope)
   jboss.jar  (runtime dependency)

So of these, only spring.jar and common.jar need to be brought along when we are deploying.

Instead of hacking together a special deploy-classpath configuration for Ant, Maven does exactly this when building WAR-files out of the box.

Transitive dependencies

Let us take a look at a couple of our modules (mea:

fizz-project
\
fizz-core
fizz-web

If fizz-web depends on fizz-core, and fizz-core depends on spring.jar for compilation, you can be pretty sure that fizz-web also depends on spring.jar indirectly. We say that fizz-web has a transitive dependence to spring.jar, or spring.jar is a transitive dependency of fizz-web.

In these terms, all dependencies in Eclipse are non-transitive until they are configured as being exported. This is the nearest thing Eclipse gets to having scoping on its dependencies.

Naturally, keeping track of transitive dependencies is imperative. Lack of control on this leads to missing class definitions at runtime, as well as unneeded JAR-files in your lib-folder that need to be cleaned up periodically (like I did).

Versions on dependencies

Now, add into this mix that spring.jar also depends on other JAR-files again, perhaps apache-commons or something else. And then you have to remember that you need to know which versions of these 3rd party library depends on which versions of their transitive deps, and
so on. This problem is expressed pretty well by Jason Van Zyl in his blog post Why Maven uses JAR names with versions, so I'm not going to write more on it right here. Basically, having an explicit notion of which version is an important part of controlling your dependencies.

Summary: Keeping order in your dependencies is easier with Maven

Since Maven makes us express the dependencies, their versions and their scope in the POM, an XML file dedicated to this purpose, we have a much easier task of maintaining our JAR-files. The easiest way to show this is to demonstrate the maven-dependency-plugin on a tiny example application I had lying around:

[INFO] [dependency:analyze]
[WARNING] Used undeclared dependencies found:
[WARNING] com.opensymphony:xwork:jar:2.0.4:compile
[WARNING] Unused declared dependencies found:
[WARNING] org.springframework:spring-mock:jar:2.0.5:test
[WARNING] org.springframework:spring-core:jar:2.0.5:test
[WARNING] javax.servlet:servlet-api:jar:2.4:provided
[WARNING] javax.servlet:jsp-api:jar:2.0:provided

The result of running mvn dependency:analyze is a report saying:

(a) which dependencies this project has that are unused (and can be removed),
(b) which dependencies this project has that are used at runtime, and
(c) which undeclared transitive dependencies are sucked in, but not declared in the pom.xml as it should be.

This is all the information I need to do the cleanup I spent two days on (given that all dependencies are correctly configured in our project).

I hope next time I'll be able to express why it's a good thing to publish/subscribe dependencies instead of pushing them into projects.

Reason 1: Get your libs-folder out of SVN

2009-04-13T00:09:00.004+02:00

This post is a part of a tiny series I'm doing on why we use Maven, and you should too.

Previous posts:

Introduction: Why you should use Maven

The libs folder

We have a libs-module in Subversion. When you check out the source code for our main product, this is one of the modules you get. It contains a hundred-and-some JAR-files. These are the dependencies for the sum of our modules:


fizz-project
\
 fizz-core
 fizz-web
 fizz-libs
   \
     junit.jar
     spring.jar
     common.jar

Downloading all these MB's of JAR files is something you'll have to do no matter which build system you use, so bandwith cost isn't an argument. Subversion also uses binary diffs, so copying or changing a JAR file doesn't increase repository size significantly.

The problem arises when you want more than one product. As soon as you want to split your product into two applications (or services), each project needs its own libs-folder:


fizz-project
\
 fizz-core
 fizz-web
 fizz-libs
   \
     junit.jar
     spring.jar
     common.jar
woop-project
\
 woop-core
 woop-web
 woop-libs
   \
     junit.jar
     spring.jar
     common.jar

By doing this, we almost efficiently duplicate our dependencies. While each project can manage its own dependencies without fearing for breaking functionality in the other project, efforts to maintain the libs is duplicated. All upgrading, figuring out bugs, transitive dependencies will have to be done in two places.

The alternative is to introduce a global libs folder:

fizz-project
\
 fizz-core
 fizz-web
woop-project
\
 woop-core
 woop-web
global-libs
\
 junit.jar
 spring.jar
 common.jar

... but unfortunately this would over time become a giant heap of libs of different JAR files in different versions, and for even the tiniest project you would need to check out the entire global-libs folder. Because you have no hard linking to which project a lib belongs, you can not actively do cleanups and get rid of unused JARs. Over time, it would grow to be hundreds of MB, perhaps even GB, containing every JAR file of very version all the products of the company ever used.

So neither of these approaches are particularly tasty for an organization with a large codebase and number of projects. The first approach enforces all-source-code-in-one-build with one gigantic sluggish workspace in your Eclipse, while the second enforces one gigantic global-libs JAR folder that everyone hates to download and deal with.

Maven's solution to this: There is one global JAR-repository like in the second approach, but download is done lazily. Each project has meta-data for which JAR-files it needs, and only upon building a particular project are the needed JAR-files downloaded from the company JAR-repository. Also, JAR-files are cached and referenced in a local Maven repository on each machine, so if you are working on three projects that use junit3.jar, you only have one junit3.jar in your ~/.m2/repository directory.

Additionally, Maven support inheriting dependency management. You can define parent projects that define dependencies, and have different projects inherit these.

This post is already long enough for publishing, so I'll continue next week with Reason 2: Clean up your JAR-files.

Why you should use Maven

2009-04-05T22:39:00.005+02:00

Update: I'm doing some work to split this blog post into some more fine grained posts in a series. Please, bear with me if these changes are giving you lots of noise in your feed-reader.

I used to work as a consultant. My job was to improve our clients' ways of software development. Among the most important things I tried to teach them were:

o Good code style and object orientation
o Practical use of good open source tools (don't re-invent the wheel)
o Automating tests of their software, and continuous integration
o Splitting code into modules, dependency-relationships and versions

The last point is perhaps the trickiest. It's hard to understand why it's needed in an isolated project, it is hard to explain how it should be done, and most importantly, there is no de-facto way or standard for managing modules, their relationships, versions and scope.

This is why me and my consultant buddies embraced Maven many years ago. After years of pushing and evangelizing at our clients', presentations at javaBin and JavaZone, Maven has at least become the way for managing and building Norwegian Java projects. There are still old Ant projects around, but I haven't heard of any new ones being started without Maven, or any old projects where a Ant/Maven migration path hasn't been thought of.

Here at IP Labs we still have an Ant build for our main project, and it works. (We have also started using Maven for a bunch of ad-hoc projects). There is a small, co-located and tightly communicating Java team that all use Eclipse with identical setup. In order to build our product, you check it out of Subversion and run "ant build-all", and that runs pretty similar to your average "mvn install": compiling, running tests and building binaries.

So here's the million dollar question: When our build system works perfectly fine, why should we go to the trouble of converting to Maven?

And instead of answering with some fluffy modular design theory, let me get hands-on, and give some real practical reasons in these following blog-posts. The first one is Reason 1: Get your libs-folder out of SVN.

Initializing Scrum

2009-03-15T17:05:00.003+01:00

After I began working at my new workplace, I've been gently pushing Agile here and there. We've been brainstorming how we can get agile working for our people, and so far we've adopted a few patterns and practices, but putting those plans into action has taken some time. Originally I hoped that things would go very quickly, but agile is about people, and people are hard to change.

Getting better, always.

We have a product that works really well, and pretty much all of the developers are happy. (Generally speaking, this state can be dangerous for a business. There is a danger of falling asleep, one day waking up to see that your competitors have gotten a good hold of your customers, with a better product and better marketing.) Having an environment that strives for improvement is a critical part of persisting success. And striving for improvement, well, that's what agile is.

Luckily, our team of developers do a lot of activities in the hunt for technological improvement. There is a team of architects (and by that I mean hands-on hard-core, coding Java gurus) that work across the other developers, singling out targets for refactoring and other improvements. Another practice I really dig is the weekly Java-team meeting, where one of the developer present a subject either on good programming practice, new tool/feature, or explaining a certain part of our domain model.

As well as pushing for technical excellence, we also want to push for process excellence. We want to give accurate estimates, we want to deliver fast, and we want to deliver the stuff that is most important first. This is where I (and the people who hired me) figured that Scrum would be a good try, but before I explain how the first week of scrumming has gone, I need to explain a little more about our team.

Who does what?

We've got nearly twenty developers working on our product (there are other teams in the company, but we're an independent gang for the most part). In this product there are many families of functionality, and there is no clear designation of which developer does which functionality. There is a lot of tacit knowledge on who does what: "Bob did a lot of work on the fizzerburpilator", or "Bill wrote the shackalock-editor, and occasionally does some maintenance on it" (features are given fantasy names for non-disclosure reasons).

Understandably, it is a bit fustrating for our customer-contacts to ask for new features, because often (a) they don't know which developer can do it, (b) the chief architect is too busy to spend time managing the tasks of every single developer, and (c) the developer who knows how to make the new feature is busy doing other critical stuff.

The solution to this frustration seems pretty clear: Organize into sub-teams, preferably by feature. However, we can't just jump right in and do a full-blown re-organization, we have to be careful about it, and we have been waiting for a fitting opportunity to try this out.

Taking Scrum for a spin

So, recently we had a big customer come to us with a feature request, let's call it "fizzburp hickup". It is a bit too big for one developer to do by himself, and there are some architectural challenges which would probably require a weighed consensus of developer opinions rather than the brilliant idea of a single programmer. The scope and definition of "fizzburp hickup" are pretty well described, and we have an ample time-frame, so we figured that this would be the perfect opportunity to try out Scrum with a small team of developers, called Team Espresso. If the experiment goes well, Team Espresso can continue existing after their feature has been developed, keep on maintaining it and adopting some other features as well.

The first sprint got scheduled to start this Monday. I knew that boiling up the backlog for the team would eat up alot of time, so I started the week before by chipping the requirement spec into user stories, using the customer-contact for verifying that I had gotten a good understanding of the domain, and also tried out User Story Mapping for the first time. I ran through the story cards with our chief architect and got a first-impression of estimates. Getting him to put estimates on user-features instead of architectural components was really hard, but in the end we came through with a rough estimate of the backlog-effort. We used 'tricky points' as our unit of estimation.

We then had half-a-day "kick-off" with the customer-contact and architects. First we went through the requirement spec and had the customer-contact (who unfortunately went on vacation and was represented by the, erm, chief architect) communicate the motivation for the new features. Naturally, the focus went technical early on, but I've been assured that there has already been so much communication with the customer on the "fizzburp hickup" already, that we shouldn't need any more input from them for the time being. Hmmm!

Sprint planning

After the kick-off, we gathered the team around the story map to do estimation and commitment to the first sprint (one week!). Naturally, the team zoomed down into technical discussion on every other story card (which is a good thing!). We discovered a good few open technical and functional questions that we would have to take back to the architects and customer-contacts. I thought this process was very fruitful, because we were finding challenges early that we would have to face sooner or later.

However, in the rest of the team there was a feeling of "let's just go and get started it", while I was forcing them to put tricky points on the user stories, and again it proved difficult to get them estimating features instead of technical components. Their point (as I understood it) was that in the end, we would have to deliver the same functionality anyway, and thinking in user-stories instead of components would only disturb our estimation thinking and architecture plans. That actually was a good point. My reply was "do all the architectural consideration you want to, but leave the story cards", in other words: bear with me on this. Finally, we committed to our first sprint, choosing three stories summing up to 30 tricky points.

And off we go!

On the first day of the sprint, I tried to get the team breaking the three stories down into tasks with hourly estimates. This proved even more difficult. I was inflicting even more Scrum overhead on a team which wanted to get to work. The point of breaking backlog items into tasks is to get a picture of remaining hours for sprint burndown, as well as assigning tasks to point person. Does this need to be explicitly done in such a small team for such a small project? I think so, because for a practiced team this overhead is virtually nothing. For my team in this sprint, however, we let it go.

Halfway through the sprint, the team got a bit nervous about delivering/presenting the result to the customer. Showing off that little progress would only anger and confuse the customer, making them come up with a burdening set of detailed requirements to "speed us up". So we extended our sprint to last two weeks instead of one.

We also changed around the planned story items. The architects found the time to do some smart fixes that allowed us to grab and do some of the more important user stories off the backlog.

At the end of the sprint now on friday, we had completed only one of the three committed stories, plus one extra because of the architecture fix. We had made alot of progress on the other items, but not completed them.

Retrospective

So, in order to help us reflect on how we should continue in the next sprint, let us sum up what practices we have tried and given up:

* Daily standup - stopped this, because we are already doing daily standup with the rest of the developers (I might write a post about how it was to get people adopting daily standup later).
* Story planning, we will continue this, although we are aware that it doesn't give us that much value without an on-site customer.
* Task tracking - stop this, too much overhead for such a small scope.
* One week sprints - go over to two week sprints instead.

I hope that we will be able to resume some of these, the 1-week-sprint in particular. There is a big organizational challenge which is making it difficult for us, and that is a quite common one in organizations adopting Scrum: We are part of two teams: On the one hand, we are Team Espresso, an agile group dedicated to deliver the "fizzburp hickup" functionality. On the other hand, we are part of the larger developer team, supporting and maintaining all the bits and pieces of our software, getting ready for our coming release ("fizzburp" is scheduled to be released in the next one after this one). Juggling these two on an hourly basis is proving quite difficult for the team.

I'll try to post back with an update after the next sprint. It could be that my expectations for how smoothly this would go were a bit too high, but hopefully I will be able to post back with a bit more uplifting results by then.

First month of work done!

2009-01-31T17:07:00.006+01:00

So! As I blogged about previously, I've begun working at a new company right after New Year's. I figured I'd get some thoughts out on how the first month has been.

Usually, when someone begins in a new job they're excited, nervous and above all very humble towards the existing practices that exist in the new workplace. Of course, I had these feelings, but being an agile fanatic, I was pretty sure that I would be able to find some improvement points. Here are some of the specific steps I took during my first weeks.

Step 0: Start writing a diary

I don't think an honest worker was ever harmed by being transparent in his work. A practice I've been doing ever since I started working is The Diary. Basically, just a personal wiki-page where you note down a few lines each day on what you're working on. Apart from being a good way to keep focus throughout the day and being a nice place to paste notes and code, it sends out a message: that you're interested in letting others know what you are working on. I'm going to be asking people what they're doing a lot, so being transparent in my own activities is a good start.

Step 1: Get Continuous Integration set up

One of my first reactions when I went to work on the code was a certain sense of claustrophobia. Here there were about twenty developers working on the same code base, and I really didn't have any feeling of who was committing what when. I first tried to get commit-mails set up, but due to some issues with commit-hooks in their Subversion setup, we couldn't get that working right away. So I turned to set up continuous integration with Hudson. My hope was that the CI changeset pages could work as a temporary commit-mail replacement, and since there already was a ViewSVN installation running, this worked out quite nicely.

From Drop Box

Note that the screen shot is not from our place, but from Sonatype's CI farm (tip o' the hat to Alf!).

Not long after that we were able to also get a nightly deployment of the build to the testing server, so now testers and designers can always experience the latest running on some local servers.

Note that getting this far in a couple of weeks would never had been possible in a traditional company (in my experience). The credit of this smooth setup belongs to some of their developers and admins, who put together the necessary scripts and server setup in record time. They actually had a continuous integration already, running the build each night as a cron-job. All I did was drop in the idea of using Hudson, and running the build on every commit.

Step 2: Do some agile preaching

I wanted to give the other developers some agile 101 very early, and luckily a slot opened up in our second weekly team meeting. I did this somewhat critical presentation on Scrum, so now when I accidentally throw terms like velocity and backlog around, they will have an idea of what I'm talking about.

Step 3: Whiteboards, whiteboards, whiteboards!

One of the things I did the first week was to get some large whiteboards ordered. I am of the opinion that you can never get enough whiteboards, and management agreed to spend a small fortune on this (again, something that wouldn't be easy in a traditional company). They finally arrived last week, and on Friday we finally got our team-lounge set up:

From Blogger-bilder

It's quite neat, I think. My plan so far is to use the far-left one for backlog stuff, the two horizontals for Sprint-charts, and the one on the right for anything else people would like to draw, be it architecture or obscenities ;)

It will be quite interesting to see how this works out. Some of the developers were speculating that the setup will be working for the first few weeks, and then fade away and people won't bother any more. Well, if people don't need this amount of communication in order to work efficiently, we will of course not force people to participate. I think everyone is better off if they spend 10 minutes of their day with the rest of their team on a daily standup in front of these whiteboards, but if people still don't like it after giving it an honest try, we'll remove the practice. After all, that's what being agile is all about.

Not consulting any more. Or am I?

2009-01-04T19:54:00.004+01:00

As I've hinted in some earlier blog-posts, I have now moved to Bonn, Germany, and as of last Monday, I'm working as a developer for IP Labs.

This means I won't be a consultant anymore, as I have been for the last 2,5 years. I am now a "fast ansatt", or steady employee. It's a bit interesting to ask myself:

Does this mean my rocketeer knowledge increasing career has come to an end?
Will I go to work with a much more relaxed attitude, getting used to stable work environments and fewer changes?
Will I spend less time overall doing stuff related to my profession?

I hope the answer is to all three question a ringing 'no!'. I really enjoyed being a consultant because it allowed me to do stuff I'm really good at. I'm not a super-programmer, in fact I know lots of programmers who are better than me (well, even more that are worse than me), but I hope I still will be able to contribute lots to the company by doing what I always do: pushing for more agility and sound practices.

So far in my career, I've worked closely with about 5 different teams, from 1 to 10 in size. My role in these projects have either been enforcing soft values (typical agile stuff, like planning, estimation and practices), enforcing hard values (code style, tool-use), or both. This effort has been spent mostly on the individual level, some on the organizational level, and then some on the community level (although not as much as I would've liked).

At IP Labs, I've found that the individual/hard values are already at quite a high level. They (or we, as I should say) are already well-endowed with pride, professionalism and quality, as well as excellent Java coding skills. There are some practices that can easily be introduced to make things a little better, but things are working, and the developers are delivering lots of functionality with steady velocity.

Also on the organizational level, they are focused on developer-satisfaction as well as customer-satisfaction. Already from day one, you feel that your developer needs are being taken care of, getting free choice of laptop and equipment (I'm now the proud owner of a MacBook, which I so far utterly despise, but hope I will learn to love in time). The infrastructure is very developer oriented, and Windows is frowned upon.

There is no crappy Exchange mail server here that forces everyone to use Outlook: It's simply IMAP - use the client you prefer. They don't use Messenger or SameTime with proprietary protocols, they use Jabber instead. Again - use the client you prefer. All in all, one kick-ass place to work :)

Well, back to the ranting. ..

The luxury of not being a consultant is that my voice can be a little louder. No more tip-toeing around, packaging plans into sweet little convincing ideas and planting them in other people, being sure not to make it sound like you're not selling anything, even if you're not selling anything but common sense.The idea is that if you're a steady employee, you can criticize, argue and be taken seriously without too much beating around the bush, subterfuge and convincing.

Or maybe it really isn't that different. As a consultant, you have to be careful not to be to rash, try not to insult anyone, be fair, objective, honest and overall being very very patient. Shouldn't the same rules apply to all software workers? Just a thought...

I think after all my job won't be that different. Maybe I'll get some more elbow-room, but all in all, consulting is perhaps nothing else than being an understanding, careful and humble colleague.

Projects are bad for software.

2008-12-06T19:36:00.005+01:00

This post is mildly inspired by the discussions from this year's Smidig conference (Oslo's agile conference) inspired by Marcus Ahnve's talk about "totality outside the project", roughly translated. I didn't see the talk myself (looking at it right now though), but when I heard the announcement of the title of the open space, I immediately recognized the idea as something that I've been lingering around for years, without being able to identify properly.

In my last post, I wrote about how I learned about project management in uni, and how it is the way for building software. In my own opinion, that is not true at all.

Over the handful of years I've been working with software, both in project and maintenance mode, I can just about sum it up in one sentence:

Projects are bad for software.

There is one exception (discussed below), but in all practical cases, I find this to be the reality, and if you think about it a little bit, I'm sure you'll agree. Here are some points that come to mind:

Projects create irregularities in software
Two times I've worked on larger projects that involved improving the functionality on existing software. Basically, the process involves boosting the maintenance team (0-2 people) with an additional team of 5-10 (typically consultants like myself). The super-team introduces lots of fancy new technologies and methods and grows new functionality into the codebase over some months. Unfortunately, the rushed nature of the project doesn't give the fresh team time to understand the existing domain model, and instead of morphing the code into doing the new stuff, they add and pack stuff on top of existing code that does things differently. Now we might have two different technologies that do the same thing, or two domain objects that are similar, but have no notion of re-use. Broken windows, if you will.

Projects push people in, and afterwards yank them out
For some reason, after project delivery the team is cut down to the minimum required to keep the software operational. The super-team are pretty happy with the work they did, because they understand the new stuff perfectly. Unfortunately, the ones stuck maintaining the project probably weren't able to absorb the new domain bits and techonologies in time, and are stuck maintaining something they don't understand. This is bad knowledge management.

Projects are meant to be temporary, but software is long-lasting
The majority of work on software is done after the initial development. It is maintained, monitored, bug-fix-patched, upgraded, expanded and extended. Throughout this phase, we spend a lot of time fighting bad stuff that was forced into the code-base during intensive project development. All the developers that contributed during the project who would be great aid in cleaning up the mess are long-gone, and besides the maintaners are too busy with bug-fixes and small functionality changes from the business side, so there's no resouces left to get quality back to a respectable level.

Projects are based on the notion that software go into production and live happily ever after. This is not the case! Start measuring how many change requests come in from your IT-department or business side in just a month.

Yet, we have created an artificial boundary between the software development and software maintenance because of some twisted desire that maintaining software should be cheaper than developing it. Does anyone seriously believe that performing change on software that is running in production is cheaper than changing software in development? If so, they probably have never worked with a database, or software at all.

So when is project the right thing?

There is only one scenario that justifies the use of project as an organisation form: Termination of a system. You are migrating data away from a legacy database, shutting down a server, quitting the use of a service provider, etc. There is a clear deadline and date in sight for when the development will be finished, and after that, all the code that was written can be thrown out and forgotten.

(Oh, a second case: Cancelled projects.)

There are some scenarios like these taking place, but often they are part of something larger, like building the new system, or using a new service provider, and these larger invests are meant for a longer life, and do not (or should not) use project as a form of organization.

So how come we ended up doing projects?

I think it was a sort of panic reaction from leaders to try to control quickly escelating IT costs. Time-boxing, money-boxing (budgeting). There's nothing wrong with budgets or time-boxes. They are perfect planning tools that ask the team the right questions.

But business side wanted to be more efficient, so they put cheap labour doing maintenance (because they think it is easy), and smart expensive labour doing development (because it is hard). Truth is, development and maintenance is equally hard. In fact, they are the same thing, and by seperating them we are only doing these people a favour: consultants. It's an entire industry that feeds on the fact that their customers are unable to manage projects and maintenance on their own.

I could go on about how some large organizations I know of control and drive development projects by intensive consultant development, and afterwards dump them over to the maintenance team in the IT-department. You've probably seen this yourself, and how much fustration and problems it creates.

Now let us do this instead..
Instead of hiring ten project developers and one maintainer, hire three developer/maintainers instead. Believe me, they will deliver just as much functionality, maybe not in the first three months, but over time. The team of three developers through a whole year will give you a much higher velocity over one year than boosting the team to ten developers the first three months.

It's cheaper too. Here's the math:

10 devs for three months + 9 months of 1 maintainer = 39 work months. And alot of these are probably expensive consultants as well.

3 devs for twelve months = 36 work months.

Besides, a stable team of three developers will give you a stable code base with better quality that grows in rythm and improves with the skills of the developers.

It's time we start making consultants their money worth
Don't use them as developers. Use them as advisors, teachers, coaches. Have them assist those three developers, not code for them. And when the consultants seem to run out of new stuff they can learn your developers, chuff them out.

I'm afraid this post cranked a bit too many thoughts into one, but it's still a field where I only have my own limited experience to base the ideas on. I'm sure there are many other smarter people who have some thoughts on this, and I would appreciate if you could give me some pointers in their direction.

What I learned about Project Management in school

2008-11-11T21:19:00.007+01:00

Back when I started studying at uni, I took a course called "Organization and management of technical projects" (Norwegian course page link). The course was structured in a very interesting way, and dealt with a measure of topics which are still very relevant for me today (team psychology and leadership, project planning, estimating, etc). It was also held in parallell with a practical software development project (Norwegian link).

I won't write too much about the course itself, but I want to write about the final "exam". We were asked to write a ~15 page essay reflecting on the subject based on our experiences from the practical project combined with the curricular theory.

I handed in my copy in June 2004, just about as I started earning money as a software developer. Some times recently I've wondered if there are any nuggets in that essay (entitled Project management for the Nintendo generation) that I can still reason about today.

(Skip to the bottom of this post if you would rather read the whole original essay in Norwegian).

The essay is primarily about Projects, clearly apparent from the oozy introduction:

Software is no longer only a tool. It is a product in its own right, and project work is what produces it.

Very flash. There are also plenty of footnotes, like:

Svada (Norwegian word a bit like claptrap): A field of knowledge that consists of obivous'ites and only exists for the purpose of its own consultants.

Hello satire! There are also some interesting psychological aspects of project heroism, showing some self-insight:

There were times [in the project] where I saw lack of management, and again I hungered to take control of the project to get the work going. I wonder whether this need was rooted in my task-oriented, or self-oriented self. The task-oriented part wanted control to get the job done, wile the self-oriented part wanted to put me in the hero-role so others could see my skill.

There is also some skimming of the nature of legacy systems:

It is a well known fact that extensive requirement specifications and customer interaction is necessary for a successful product. We have [with our project] confirmed this yet again. Even though it can be useful to re-use existing resources and design, it is healthy to engage upgrades of IT-systems as entire new projects.

Hmm, notice at this point that I might not fully agree today with what I wrote 4 years ago. This notion continues in the next section:

It is within the complex nature of project work that a comprehensive and documented plan is necessary. The project plan should be the project group's bible. It should be a reference guide and framework for all things regarding control, activities and mangement in the project.

Quite. But hey, later on I reframe the view somewhat:

It is easy to speculate on what a project plan should contain, and one often ends up with a project plan containing a lot [of stuff]. There is a trend of plans growing so heavy that they are ignored as soon as they are complete. The project team believes that the plan contains a series of obvious facts that are only relevant for management and customer, and feel no ownership in the document.

Hear, hear. Now, later on I described the project implementation in some detail, and this is where I'm particularly proud of my younger self:

The implementation phase is often experienced to be the biggest part of the project. It is then one actually makes the product. While being a concrete and actual task, it is still very unclear in the beginning. By the completion of the product, it is odd how clear the approach was when one looks back to the start.

Implementation never goes as expected. It is like finding the easiest way out of a circle drawn up by the analysis [earlier project phase], and naturally one follows the radian directly outwards.

The larger reqirements and analysis, the larger the circle becomes. Throughout the implementation, the developers analyze deeper and extend the requirements on their own. The analysis [circle] will grow and move depending on how thorough the original analysis was. Often it will become appearant that another angle out of the circle (meaning different technology or architecture) will be shorter, and in the worst case, especially early on in the implementation, one has to turn around entirely to follow another direction [out of the circle].

Today that looks to me as a strike for agile development :)

I also complained about my fellow students unwillingness to embrace English as language used in code, noting "It would be interesting to quantify what effect this has on Norwegian system development". This very day I'm working with projects that are suffering from having large masses of Norwegian code, while external consultants are struggling to figure out what's going on. Oh, and it's still a "company rule" that all code must be in Norwegian. Get a grip, you nitwits! Your Norwegian code won't protect you when outsourcing threatens. Rather it will be an good reason for throwing your code out, and you along with it! Learn English, or learn a different profession! [sorry for ranting]

The essay contains some glorification of the project as a working structure, and the project manager her/himself. I drew a prallell from the project manager to ancient human traditions of tribal leaders, and how a project manager appeals to our natural order. A practical assignment demonstrated that the team was actually less efficient when all had insight into the orchestration of a task. Today I'm not so certain we should prominate the project manager as the leader role it is set out to be. Scrum trainers have on several occasions claimed that there is no place for project managers in agile projects.. but that is all thought for another blog post.

Finishing off the essay, I gave a brief summary on each of the seven sections:

Expectations should not be built too high, but the team should be made aware that the project will demand full attention, so they anchor their devotion to a high level.

Humans have different motivations. Their true roles should be discovered so they can receieve proper motivation and fitting tasks in the project.

The mission must be well studied/understood. Stable customer contacts and project management must be used in the customers best interest to achieve the best product.

The plan requires quality over quantity. A good plan is used continously by management from start till finish.

Implementation is the goal of the project and can quickly become the hardest part. The team will find themselves within valleys of shadow [dødsskyggedaler], and it is important for the project manager to involve her/himself, maintain motivation and synchronize development with the requirements of the customer.

The end of the project should be emphasized. Before the team is disbanded, they should analyze their experiences and document what knowledge they have acquired throughout the project.

Leadership demands a person with a multitude of talents. She/he must be fully focused on the project. Both the team and external actors must be duly stimulated for the project to succeed.

.. and then I finished very dramatically:

It is important to remember that we do not use this form of management [meaning project] for its own sake, but because it brings out the most productive in humans.

Projects are our generation's way of organization for system development. It is important that we prepare ourselves for the real world by doing projects in a realistic atmosphere so we may experience that leadership is important, and not just another chapter in a textbook.

It would be fitting to end this arcticle in the same manner as it began; by the wise words of David Hume:

"Teaching has suffered greatly from having being locked up in Universities and expelled from the World and good Company."

Very fitting end to an essay I handed over to a professor, don't you think? Mind that the Hume quote is doubly translated back from Norwegian because I couldn't find the original source right now.

For those who wish to read the essay in Norwegian, it's available on Scribd.

Looking for work in Bonn (update: fixed)

2008-10-25T15:26:00.004+02:00

Update: I was in Bonn doing an interview on Friday, and the outcome is more or less decided: I've found my new employer. Thanks for all the input and pointers! Note that I've removed links to some documents with some sensitive information.

For those of you who don't already know: In about two months I'm moving from Oslo, Norway to Bonn, Germany. I am currently looking for work down there, and that's what this post is about.

Here is the information any potential employer should be interested in:

Elevator pitch:

Norwegian MSc graduate from the University of Oslo, Department of Informatics. Four years of working experience with Java, consulting, agile methods, product development and open-source.

I'll be ready to begin working on the 5th of January, 2009. I will be staying indefinitely.

My résumé

I will be visiting Bonn a couple of times in the following months, and it will be possible to meet me for interviews in Bonn on these dates:

7-10th of November

27th of November - 2nd of December

I would appreciate any advice anyone has on working in Germany, or can recommend any particular employer. I wold also appreciate if you will forward this post or application to any potential employer you could think of (I have already sent an application to some positions I've found on monster.de or stepstone.de).

Please contact me if you have any questions. My Gmail account is tfnico.

What I'm doing these days

2008-10-23T22:51:00.004+02:00

I'm quite busy these days doing preperations for *a big change in my life* that will be the subject of my next blog-post, but I figured I'd give a quick status update that's a bit too big for my twitter stream.

I was recently appointed to lead Objectware's Java-department's focus group on Agile. We've been doing agile for a long while on the company level, but it's first now that our department has its own group on it. My goal is to nurture it to being a very active micro-community that will produce knowledge both internally and externally. Much like we already did, but now with more, well, budget :)

Our first accomplishment was to attend the Smidig 2008 conference with a respectable amount of Objectware speakers. There were alot of interesting discussions going on in the succeeding open spaces, my favourite topics incuded some hard-core web-testing (of course), when (not to) re-use and some others I can't remember right now.

As part of my new responsibilities, I'm also trying to seed some content into the methodology part of our community wiki. Oh, and we also just had our first internal meeting with the interesting title of "Code monsters. How are they created and what can we do about them". I will be adding snips of the conclusions to the wiki as we go along.

Debate about Open Source in Oslo next wednesday

2008-10-23T00:31:00.005+02:00

Just a quick note. The alumni organization from my old university is arranging a meetup/debate about open source, and I'd promise I'd help communicate the event. The tine-up looks very promising:

Heidi Arnesen Austlid, Friprogsenteret
Per Hove, Oracle Norge
Shahzad Rana, Questpoint
Johannes Brodwall, Steria

The event will be held in The Scotsman, where they sell beer! And there will be free pizza! Doesn't get any better than that :)

I took the opportunity of uploading the full invitation here.

So, set aside next wednesday evening and I'll see you there!

My JavaZone video is out

2008-10-15T20:47:00.004+02:00

I was very happy when my JavaZone talk about "How I learned to love and hate web-testing" managed to fill up a whole room. For those of you you who didn't get in, or didn't manage to get a hold of one of the very rare overflow-headsets, or were busy at one of the other talks, here's the recorded video for your viewing pleasure:

All hail web-testing! Click the image to get to the talk, hit the little Presentation link at the top and off you go.

For the best viewing please, and this goes for all the JavaZone videos, I recommend the following:

1) Click the Max Slide button to get the slides in a seperate window
2) Right click the video for the flash menu
3) Click Zoom in, so you get a bigger video (click and drag if the focus is wrong)

On a cool note, after the first count (yesterday), my video was the 7-most viewed video (of a total of 86 videos). Woohoo! But I'm likely to drop once people start realizing that All the videos from JavaZone 2008 are available online! Now, go watch!

Some books for agile programmers

2008-09-27T21:17:00.001+02:00

I'm giving docs.google.com another try. I tried it over two years ago (wow, time flies) when Google Docs was still Writely. Oh, my prediction did come through (they got bought by Google), wuhu! So if this post gets doubly published it's because it got screwed up.

Update: docs.google.com didn't manage to publish the title properly. I also have to apply some tags manually. All the pictures and headings came through allright, though.

New books

I just got these books:

They're all from The Pragmatic Bookshelf, although I got them from Amazon (who sent them from New-Zealand, which gave me sort of a bad eco-concience for a while..) cause they were alot cheaper there.

The first title, The Pragmatic Programmer. Well, I'm sort of ashamed that I haven't read it before, being as famous as it is. However, I'm looking through the TOC, and I sort of get the feeling I've touched most of the points, either by hard realization and experience, or by advice from friends and colleagues. The book still provides me with a sturdy, well sorted and well formulated set of practices, which will help me communicate the QWAN to my consulted clients.

Practices of an Agile Developer looks like a follow-up to TPP. It's published in 2007, 7 years after. It seems to have more of a focus on collaboration and project management than on effective tool use (although the subjects do overlap).

The last book is written by two guys working for SAS, neither are part of the original PP pair. I got it because it's been a long while since I read any modern works on agile development, and agile stuff is quickly turning into my pressure point for some reasons yet to be announced publically :) The book also has a very nice looking section on Common problems and how to fix them. My team at work just shrunk down to 1,5 people, and we recently suffered almost a two month long sprint stumble (long story, and I'm still having trouble figuring out how I could've gotten it right). Hopefully the book has some advice I can apply to my own project.

All three books make up for three areas I care, and I think every IT-professional should care deeply about: Tools, languages and practices.

I just asked on twitter which book I should read first. I think I'll do Ship it! first, but might follow the advice of Olve and do TPP first. To be honest, after I finished studying I've been horrible at reading books. I have several books now that I started but never finished.

Oh, here's a wordle of my blog:

Very flash. I can't for the life of me remember who's blog I just read who pointed me at it. Sorry. There are a bit too many normal English words in the cloud, but some of the words are nice: pretty, people, good, stuff :)

Ok, let's push this thing to Blogger and see how it goes..

JavaZone 2008 and GeekCruise

2008-09-24T22:23:00.002+02:00

So, JavaZone is done. I did my presentation, tada (please view full screen to get rid of the chippy graphics):

How I Learned to Love and Hate Web-Testing

View SlideShare presentation or Upload your own. (tags: jz08 javazone)

I got good feedback. In short: The slides are funny and original, my presentation moves were allright (good voice, good gestures, movement around the stage), the content was pretty good and interesting. The talk was at an introductory level, and I feel I hit the audience pretty well. Will be very interesting to see the video, as well as getting the feedback.

I was also in the JavaZone arranging committee. You might've seen me scurrying around, commanding the guys in the orange T-shirts (they were the volunteers). It was alot of work, but it was alot of fun as well. Tomorrow evening we're throwing a party in their honour at the University, so if you're in the neighborhood, give me a ping and drop by.

But anyways, I didn't get too see many talks. Only one (excluding my own), to be honest, although I'm sure I could've squeezed into a couple of more. But you crowd could've gone a bit easier on the headsets! Jeez, it's called overflow because they're for when the rooms flow over.

If the room is full, go to another talk
If you can't find any place, get an overflow headset
Don't sit or walk around with a headset all day, you ¤%&/(

We only had like 450 headsets, which should be plenty thinking the rooms fit over 1700 people. So people, no sitting around. It's a conference. If you want to look at video, stay at home and look on youtube, or Parleys.com, or TED or something.

Well, enough ranting. On friday we had a Objectware-VIP presentation from Sun, more specifically from Alexis Moussine-Pouchkine on GlassFish. It looks as sexy as ever, v2 and v3 for its respective uses. I figured my next little fiddling will be trying to replace our JettyLauncher with a GlassFishV3Launcher, but I doubt I'll be able to convince my customer into converting from WebSphere to GlassFish V2 just yet.

Geek Cruise
Saturday morning I was looking forward to a nice weekend of doing nothing else than relaxing, and I was lying in my bed when somebody called and dragged me along to the infamous GeekCruise! I had previously turned down the available slot because I hadn't had time to prepare myself (because of JavaZone), but Anders Norås backed out, so I had no excuse anymore. Thanks Anders :)

We were only 16 participants, among them some pretty respected people like, Rickard Öberg, Johannes Brodwall and Totto. Some new to me, but obviously very bright people: Emil Eifrem, Michael Hunger and Jon-Anders Teigen. A big part of the group were Objectware people, but hey, we paid for the stuff! And I'd better mention Kaare Nilsen since he saved my ass on saturday evening. Your site is not responding, btw, Kaare.

Alot of the discussions and content have been documented in our community wiki, but the real value I think was the insight some of us gained, and the input some others got to their respectful projects. Communicating all the knowledge that flowed on that cruise.. well, it's pretty hard to do in written form (and maybe that's exactly why such events are a *good thing*). And most importantly, I got some new friends.

JavaZone'08 coming up

2008-08-10T22:30:00.005+02:00

I'm still going through my post-summer awakening, so I still haven't gotten the wheels of this blog running again. But current events are sort of relevant to the people I'm guessing are part of this blog's readership.

The biggest happening coming along is JavaZone 2008. Now this is especially important for me for two reasons. First reason:

I've been a part of the crew preparing the conference. My role has been gathering a crew of volunteers to help out with all the practical work during the conference. Hooray, very rewarding work, getting 40 people to work two days for virtually nothing!

Luckily there's a great crowd of Java-interested students in Oslo, and they can't afford a conference-ticket! So our evil schemes of acquiring a cheap workforce are proceeding as planned. Actually, there is a win-win relationship going on here. Student-organizations like to profile themselves in front of companies, cause companies have a tendency to give away free stuff when they do presentations at universities in order to recruit more newly educated computer scientists. Never underestimate the value of free food for a student.

Companies, however, are wondering "So how do we get to these students?" - when it comes to the University of Oslo, Institute of Informatics (the biggest contributor of volunteers to JavaZone), the answer is through events like dagen@ifi, and student organizations like ProsIT and Navet. You (recruiters) can meet members of these organizations at JavaZone. But try not to bother them while they're busy with the crucial tasks of keeping the conference running smoothly :)

We still need more volunteers, though, so if you know anyone who'll chip in some hours of work in exchange for some freebies and free attendance to the conference, send them over to the JavaZone volunteer sign-up page:

www.javazone.no/funk

Second reason: My proposal for JavaZone'08 talk got accepted! The title of it is "How I Learned to Love and Hate Web Testing", and that's pretty much what's it about as well. If you want more details you can read my last blog post on the subject. I haven't gotten to prepare the talk half as much as I've wanted, but I've got most of the content pretty clear up in my head.

I've started writing a manuscript for the entire talk. Now before you think "Stop it!" I won't print it and read it out loud when I do the talk. It's more like a movie script. I know too well that Norwegian audiences beyond a certain size are very hard to read, and therefore hard to adjust to, so I'm going to perform the talk exactly has I have planned it in advance. What I loose in adaptability I'll gain in stability and content. Or that's what I hope anyway. Comments on how to prepare for talks in front of big JavaZone audiences are appreciated.

Sharing the knowledge

2008-06-06T22:44:00.003+02:00

This post is going to a bit about what I've been doing lately. Well, like any other developer I've been mostly busy with getting the projects released before the big July. But I've also been past-timing a bit on setting up a new wiki.

Yay, I don't have to be the wiki techie any more!

It's been well over a year since I started managing Objectware's Confluence installation. As expected, it was very little work. Confluence basically runs itself once it's set up. Mostly it was upgrading, administering content (deleting and cleaning), doing the odd restart or log-watching. Eventually, after a year or so, the hosting of the wiki was outsourced to a nearby hosting company, cause our instance's database was starting to run a bit slow, and my billing rates were a bit high (hey, I'm a consultant, after all) :)

So good riddance, finally no more booring IT-support work.

Or do I...

Well, not until two months later, anyway. I was asked by a colleague, who's a core driver of our internal Solaris-group (I'll let him remain nameless right now, but he desevers all credit for this stuff), to participate in our latest venture, which I'll get back to, but first..

About two years ago..

I was discussing the still immature wiki with the boss. Already we were seeing effects of decay and staleness in the quickly growing content. He asked me what we should do about it, and I said "I honestly think we should just open it up.". He gave a small laugh and said that was not very realistic. And it wasn't. Too much confidential customer stuff was mixed in with the pure knowledge about various open source projects and tools.

Well, since then our wiki has grown and matured. It is better structured and filled with a lot of good content (much to the credit of the nameless guy mentioned earlier).

Some stats..

It's a meager 3.418 pages. You could be worried that it's a bunch of useless pages, but these pages count 29.555 edits, or nearly 9 edits per page, which I'm guessing is a good number for an internal wiki. 159 different authors have contributed to the content, and there are over 2.000 attachments.

Now, finally due to a lot of internal efforts (not many of which were my own, to be frank), the political descision has been made to open up *some* of the content.

Enter wiki.community.objectware.no

So we took an evening and set it up on our new fancy Sun/Solaris box. Did I mention how easy/great it was to set up a Solaris with a database and Confluence? Well, I did get someone else to do the dirty work of setting up and configuring the zone, but after that it was real easy :)

But why, you ask? Why take our hard-earned intellectual property worth hundreds of hours of consultant hours and just share them with everyone?

Open source, open knowledge

Well, first thing is that the knowledge in those pages isn't worth more than how much they are used. If more people use them, the (especially Java) community can benefit as a whole, and make our work easier/more fun. We have to work with our competitors on many projects, so if we can help them for example understand how to set up a Maven enterprise infrastructure, then we will benefit as well in the long run.

Good PR

Well, obvious argument isn't it :)

We needed a good wiki for these open source projects

We wished to host a Confluence for the Backward Compatibility Tester and the EDR project. More will probably follow soon. Sorry CollabNet, but your wiki sucks.

We need help!

We want input from other people who have experiences with the tools we use. We're hoping that people will contribute back with either their knowledge, or link to places where it exists on the net.

Why it might be a bad idea..

My main concern is that we'll get some negative reactions claiming we are duplicating/harnessing documentation which should reside other places. Well, we're too impatient to wait for these other places to update their FAQs and wiki pages with our stuff. If they wish they can just copy the content back to their sites whenever they feel like it.

There are some other concerns, like what license should we use, spam-prevention, and cross-wiki search (well, atleast now we can search the wiki with Google).

And finally, it might just not take off because people might be too shy to write stuff on the public wiki. Well, we'll see how it goes. No matter what, it will be my (and some others) way of sharing stuff with partners and other people in the community when we want to get some content *out there* quickly. Feel free to use it for the same.

It's not more than 81 pages out there yet, but take a look at the Smidig 2.0 stuff. It has a lot of good stuff about Maven and testing strategies, among other things. We've tried to keep all of it in English but some of it is still Norwegian, unfortunately. Smidig 2.0 roughly translates into "Agile 2.0" and is our pet-name for practices that have evolved after the first wave of agile bubbles. If you know Norwegian you can take a look at Totto's presentation from the Smidig 2007 conference for some more thoughts on that.

Here's a nice way of taking notes

2008-04-27T00:34:00.004+02:00

Inspired by Johannes' post-its and Anders' moleskin, I'd like to share my tip'o'the day for taking notes.

Step 1: Find a stack of A3 sheets, big ones!

Step 2: Get a good thick nice pen (mine could've been thicker)

Step 3: Lay the sheets on your desk between yourself and your keyboard (warning, not sure this is ergonomically wise thing to do).

Step 4: Whenever you show off stuff, especially code and stuff you've got on yer computer, support your presentations with drawings and notes on the sheet.

Step 5: When you're done, hand the sheet off to the person you drew it for, or whenever the drawing ends up particularly pretty, smack it up on the wall behind you. Or throw it out if it's all squabbly wabbly.

This way, I don't have to get up to use a flip-over, I don't have to take useless digital pics of the white-board, and I get to show off my l33t UML-skills without having to use some crappy software. Sweet!

Diclaimer: I didn't come up with this myself! Credit is due to our architect, who will be revealed if he so wishes :)

Why I keep badgering on about Web Testing..

2008-04-27T00:03:00.006+02:00

Lately I've become somewhat obsessed with testing. A specific kind of testing. I'm not quite sure about the term right now, but it's close to system testing, or maybe user testing. Ye know, that kind of black-box testing that makes sure your stuff works from the user's perspective. In web applications (of which I probably do about 90% of the time) they happen to blend in with web tests.

I suppose it started off with last autumn when I put some thoughts on this into my lightning talk at the Smidig 2007 conference. Well, it actually started all the way back in 2006 when I did a lightning talk about Selenium in XP-meetup based on my experiences with Selenium vs. gargantuan use-case descriptions in a project (see slides).

A few months ago I was down visiting a friend in Bonn, Germany. He works for this CMS-vendor/host called WebFactory. I offered him to drop by their workplace and do a quick demo on Selenium, as I figured they're (a) probably doing web applications, and (b) they're probably not doing web testing (in the automated figure o' speech).

Strictly speaking, I didn't actually do the demo; I had my friend do the demo instead. I took some time in advance to teach him how, and naturally he made a much better point of how to use the tool in their shop than I could. Neat sales trick, eh?

Anyhow, one of the guys' impression was "We had no idea it was that easy!", which is the usual response triggered by showing off Selenium. The 10 minute demo eloped into hours of interesting discussion on all things web, testing, JavaScript, Comet, etc, and the whole thing sort of re-invigorated my feeling of that people still do not know enough about web testing, and the world would be a bit better off if more people knew about it.

If you don't know what it is, there it is (in its simplest sense). A Firefox plugin that records your browsing instructions and can replay them later on a lot faster than you can with mouse and keyboard.

A few weeks later I went on to do a 30-minute session on web testing for my current customer's Java developers. Some days later on I held a 2 hour guest lecture in the University of Oslo's open source course.

I figured, hey, I'm starting to get quite good at presenting this web testing stuff. So I piggy-backed my own effort when writing an abstract submitted to this year's JavaZone conference. I have to admit it's not the most academical subject I would've liked to talk about. But I still see people (a) doing web applications, and (b) not automating their tests. I think it's good talk material because it is:

Simple and easy to convey
Really useful
(and I'm not trying to push any proprietary stuff here, I'm not even a Selenium committer)

So that's why I keep badgering on about Selenium and web testing. I could write page after page on web testing, but for now, if you're interested, take a look at the links in this post and take Selenium for a spin if you haven't already (the new 1.0 beta version is pretty sweet). And give me a shout on what you would have me write about next time. For instance:

* How web tests fit into the bigger picture (testing strategies and all that)
* Why I exaggerated when I once said Selenium tests can replace use-cases
* Why manual testing is still important
* Putting Selenium tests into a Continous Integration server (i.e. expanding my last post on the subject)

Wrapping Scrum in Evolutionary method(s)

2008-03-04T20:39:00.003+01:00

The best grade I ever achieved in college (apart from English) was of all things in the subject of statistics. It wasn't particularly fun or fascinating for me. It was just easy, the rules were straight and the formulas were logical (well, I kinda dropped off when we reached quantum statistics, but luckily that wasn't within the curriculum).

The reason I mention it it that I just came back from this 3-hour presentation about Evo Project Management, signed Tom Gilb and Junior (Kai Gilb). In short, Evo is an agile method focused on measurements. They have a lot of other sound agile practices in there, but that's the main thing ringing in my head after the presentation: They use measurements to aid them in prioritizing the backlog.

Now Tom's been around the Oslo agile scene for quite a while, he did a presentation in XP-meetup a year ago. I also saw him around on the Smidig 2007 conference, doing an open-space on agile estimation or something.. Didn't attend it myself unfortunately (well, I even managed to miss my own open-space for that matter). Anyhow, this stuff isn't new to me, but it's good to get some freshup on it, cause each time I hear about it I think "this is good stuff that makes sense!".

And sense it makes. For instance, Kai started off by presenting the problems with requirement specs today: Requirements are expressed through describing the solution. By example: "Requirement 12a: The solution must be password protected.".

What's the problem with requirement 12a above? It describes the solution instead of stating the real requirement: Security. First state your requirements as qualities, then state the level of quality: The solution should be so secure that it takes a professional security-team more than 4 hours to break in.

Maybe your solution is better off using a fingerprint-scanner or smart-card solution for security.

State your requirements, make measurable goals, then let the developers decide on what solutions best serve the requirements.

This makes sense because it gives you a better way of saying when you're done with a task (big problem in agile methods).

In the long run, the method becomes a self-fulfilling prophecy. If you measure the implementations by their rate of measured of success, it means your discovering which developer efforts are making you money. Supposedly, the method has a track-record of zero failures recorded by the Gilbs. If one follows the method strictly, I'm not surprised. You will only do stuff that proves to reward yourself or your customers.

There is plenty of free documentation around on the subject, so I suggest you take a read, starting off with the Gilbs' blog.

Now onwards to the problems of the methodology. One of the last things that were said in the course was that it is a top-down method only. Errmmright. That will be a problem, being the floor-mopping, monkey-coder low level consultant with no political power that I am (I have to sneak stuff in) :)

------

So here we are: It's not a how-to, but more of a 10 ideas to get you started measuring success bottom-up for average agilists:

1. Learn statistics! Or Zed will kill you.

2. Managers love numbers, statistics and proof. Show them concrete figures and they will love you for it.

3. Try to avoid absolute requirements (meaning political requirements, or solution constraints). If you do get them, try to connect to a quality requirement - "Why do you want XHTML-compliance? Oh, usability, well there are plenty of other things we can do for usability that will be much cheaper and more effective than trying to be XHTML-compliant.."

4. Put some numbers on your backlog. If you've already got an agile method in use in your project, you've probably got a backlog shoved full of requirements (we've got more than we could ever hope to achieve in 2 years). Apply measurable goals to those tasks that you can do so for. "Fixing that bug will probably increase up-time to 5 nines."

5. Other things you could perhaps measure: Traffic in your webapp, number of error-lines in your log-files, number of sales. You're probably already measuring velocity on the function points you are delivering each sprint, but be sure to measure the tasks in something else than hours (measure them in difficulty/complexity and/or business value points).

6. Measure time-to-fix period. This could come in handy when convincing management to buy into larger rewrites/refactoring efforts. Right now, our mean time to get a larger functionality fix into production is 2-3 sprints. Given refactoring module X, the time will be reduced to 1 sprint.

7. Don't kill yourself measuring. Like in any other practice, be reasonable and use common sense. If it's gonna take you 2 weeks to get a web-traffic analyzing Apache instance up and running in front of your website, drop it (and put Google Analytics in or something). Use subsets, subjective input (you don't have to use empirical research data to measure), quick'n'dirty measurements.

Well, I'll try applying the list above when I get into work tomorrow. If I achieve anything remarkable with it I'll post back on the subject later. My biggest worry is that the backlog is choked with political requirements that are not based on measurable factors. Priorities are partly based on which stakeholder has got the biggest mouth or the biggest money-bag.

Legacy Code, Broken Windows and Code Quality

2008-02-04T22:19:00.000+01:00

Long time since last post. It's not that I have been particularly busy, just haven't had anything *that* interesting to write about. Well, this might be interesting:

I know this legacy system. Well, it's not that much of a legacy system, it's barely a couple of years old. But it sure didn't take it too long to become difficult to maintain. Maybe you've seen similar scenarios... A big team of developers and consultants with lots of funding creates a big-bang super solution. Struggling to reach a deadline, at some point quality was left behind (or post-poned). We're talking compiler warnings, copy/paste code, hacks, quick'n'dirty (quoting Uncle Bob: There is no quick'n'dirty. Dirty means slow.), bad object-oriented design, the lot.

Well, I have to say that 80% of the code was golden: agile method, top of the line modern open source lightweight technologies, test-driven development, code reviews, continous integration, etc. But that last 20% of patchwork really was the start of things getting worse. We did have this great plan prepared for how we would get things right as soon as we reached the deadline, oh yes, we were going to make everything right again.

So we made the deadline and the project was a huge success. And what happened? Business people were happy and choffed all the consultants out. One maintainer was left behind to do "critical bug fixes". The broken windows remained and the make-things-right-again-plan was buried.

So a year went by. Business decided the solution needed new features. The maintainers desperately tried to keep up with business demands, again consultants were hired to satisfy the need for new functionality. This time, with developers unfamiliar with the existing code-base, more patch-work was done. The coding standards, patterns and technologies were unknown, unclear or inconsistent.

The existing broken window code had a devestating effect on the work that was done by the new consultants, and of course they were under the same pressure from the business-people, if not more. At the same time the maintainers were trying to do bug fixes in parallell, so the code was branched for several months. Upon merging again, the codebase suffered from alot of conflicts, but still the team didn't dare roll back to the stable code-base. The merge had to be used. It took several months of heroic effort from the maintainers (again, the consultant were choffed out prematurely) to get the codebase stable again, adding up to a total of 6 months without releases in a business critical system. That's 12 sprints the way we counted them.

I suppose scenarios like these end up with the system slowly decaying into the awful legacy system everyone loves to hate until finally its decided to be replaced with an all new big-bang system. And so the circle continues..

But not this time! The business has for some mind-perplexing reason actually realised that there is money to be saved from restoring the quality of this system! There will be a series of iterations were the focus is to rid the system of its technical debt (man, I love that term). Of course some business value will be delivered in each iteration, but hopefully this time quality tasks will be top priority.

The moral of the story comes two-fold:

1) Business-people can be taught the value of software quality. They can understand broken windows, it just takes alot of time to get it into their heads. Keep trying. Use good arguments, references and your experience (if you have any), and convince them that quality is the most pragmatic and profit-bound way. And if the project manager won't listen, go to the one above him and tell so. If they won't listen, find another place to work.

2) As a professional programmer, it is your duty to follow a respectful amount of common sense of ethichs and software craftmanship. Listen to the tiny voice inside your head saying "this code stinks" and never leave it behind before the little voice says "this code is good enough".

I've had it with people who write lousy code, calling themselves "pragmatic". From now on, I want to be an idealist :)

I'll finish off with another Uncle Bob quote (or maybe it was Jeff Atwood, good blog btw):

Always leave the code better than when you found it.

PS: This post was actually going be about how I wriggled around with Scala yesterday, getting my project to build Scala with Maven and still work on it from Eclipse (got it working somewhat but not 100%). Appearantly I have forgotten everything I learned about functional languages in Uni (ML), but I have a feeling this approach *might* be the easiest path down the concurrency landscape to come for us Java devs.

My excuse for not blogging the last month

2007-12-01T17:15:00.000+01:00

Recent activities eating away blog-activity include mostly JAFS - our annual internal weekend getaway. JAFS essentially means hauling our ~40 Java-programming colleagues off to a desolate hotel for a weekend, doing not so much else than having fun programming (and shooting some paintball).

The concept of the weekend's tasks are based on the following:

Divide the gang into 5 groups
Present some technologies that are core, meaning everyone of us should know by heart (for instance Spring and Maven)
Present some new technologies that are hot, and might come in handy in the near future (for instance JRuby and OpenESB)
Do some cool programming competition that makes use of above technologies

These points demand a mass of preperations. We've been 5 people spending evenings and weekends planning the event for the last month.

First we spent a whole load of time settling on which tehcnologies to focus on. See we've got four different camps of technology groups in Objectware, each of which are tugging in different directions, mainly core, web, integration and enterprise design & architecture. I like to believe we achieved doing a fair divide between the different camps, since these four camps are paying for the whole ordeal ;)

We also spent alot of time agreeing on the programming competition. Well, we actually settled on not doing a programming competition, instead focusing on cooperation between the 5 teams. Instead of having different teams compete over who could do the best implementation using different technologies (like we did last year), each team were to implement parts of a larger system which would cooperate using services across an ESB.

The system itself was sort of a collaboration suite framework, based on Subversion commits, Continous Integration and a robot. The ultimate functionality of the system: If you break the build, lil' Timmy here will come driving into your cubicle and steal your beer.

To be honest the solution will not be used directly, but the codebase does contain alot of good reference implementation of the various technologies we involved. We also got a good heap of discussions regarding among other things:

Shared domain model among the teams
Contract-driven development
A bunch of problems using JRuby (hint: it's too early)
The Enterprise Domain Repository pattern
Struts2 Best Practices

Oh, another thing we had to do in advance: we spent a whole lot of time implementing Proof-of-Concepts on each tecnhology to help the teams get started once they got up there. Now there was a learning experience :)

Aaand, the number one rule of bringing 40 programmers to a hotel: DO NOT TRUST THE NETWORK! Excactly. Even if the hotel has promised you full wireless, it ain't gonna work with 30 people who are trying to connect to the Maven repository, Subversion, etc etc.

So, we set up a mighty fine Ubuntu box with the following:

DHCP-service for allocating all the teams an address on our subnet (jafs.objectware.no)
DNS/bind for all the interal addresses on the domain
Subversion repo on svn.jafs.objectware.no
Continous Integration server on ci.jafs.objectware.no
Maven repositories on repo.jafs.objectware.no (snapshot-repo, release-repo and site-repo)
IRC server (of course!) on irc.jafs.objectware.no

And of course we brought along 40 TP-cables and a handful of switches to provide each contender with wired network.

Now it might seem like a bunch of uneccesary mumbo to setup all these wires, aliases and services, but believe me we all appreciated being able to just hook into a 100/1000-MBit network and be up and running. Zero network problems, I tell ye :)

So that was JAFS... Right after we were done I had to get ready for delivering my 10 minute lightning talk on Smidig 2007 (the Norwegian Agile-community's first conference). The conference was as impressive in implementation as was its lack of up-front planning (taking about two months to prepare, was it?).

The conference left me inspired and hungry for more. If I remember, the next post will be about distributed continous integration.

So you thought Spring's autowiring was a no-no?

2007-10-22T22:29:00.000+02:00

Think again.

Yet another Spring 2.5 feature, the @Autowired annotation gives you back the control of autowiring, by-type mind. Back in Spring 2.0, autowiring was considered by many to be an anti-pattern due to the lack of control and overview of which beans would actually be injected at runtime. With annotation driven injection (meaning you've got

in some Spring XML file), objects are only injected when you explicitly state so.

This allows you to reduce the amount of dependency mapping in your Spring XML config files and replace them with annotations in your Java files. If you've got multiple implementations that are possible autowiring candiates, you segment them by using the @Qualifier annotation.

Rather than riddle this blog post with snippets of Java and XML, I want you to get the Spring source (you can import it into Eclipse easily) and check out this test:

/spring/tiger/test/org/springframework/beans/factory/xml/QualifierAnnotationTests.java

I've been playing around with these features lately and must say they are working nicely. Beware of how/whether you decide to use the

instead of explicitly stating your beans in XML files. Yup, this means you can pretty much avoid XML completely in your Spring configuration.

It's a matter of taste really, but be consistent in the way you declare your components (either by XML or annotation). Make sure your whole team are aware of what is going on, as it can still feel quite auto-magical to the non-initiated.

If you're not interested in fiddling with the Spring source (which you should be - it does after all provide quite a few examples on how the Spring devs are eating their own dogfood), you can click some of the links below:

Mark Fisher blogged about these features a while back, and Rod Johnson recently posted an extensive article on TheServerSide about Spring 2.5. The tutorial gives a good introduction to the entire set of 2.5 features, including the annotation driven injection I've mentioned in this post and the Spring TestContext Framework I blogged about earlier.

I'll take this opportunity to remind any Oslo developers that we're doing a Spring course together with Interface21 in November, an excellent opportunity to jump on the Spring wagon and get a kick-start on lightweight system development and architecture.

Spring TestContext Framework

2007-09-30T21:07:00.000+02:00

I'm somewhat surprised that there hasn't been more noise around in the blogosphere about Spring 2.1's TestContext Framework. I had the pleasure of working with Juergen Hoeller for a day after JavaZone. Among other things, I learned about the fancy @Autowired annotation that you can use to make your autowiring much safer. Normally autowiring is considered bad practice, but it was Juergen's advice that now it is okay to use. This means in practice that all your service beans will still be declared in an applicationContext.xml file, but the dependencies and object graphs will be annotated in the Java files (or atleast where you see fit).

After this I had a deeper look into the features of Spring 2.1. Note the point about "our next-generation Spring TestContext Framework with support for JUnit4". Now this is cool.

Why we need it (or something like it)
For a long while I (and many of my colleagues and peers) have been annoyed by the disability to manage tests as the amount grows. There hasn't been any good framework around for doing any kind of categorization of tests. Each project would have to invent their own set of hacks in order to control their test suites, be it a TestFactory picking up system variables, or using different Maven profiles and source folders for different categories of tests. In short, inventing some sort of convention for seperating tests: put them in different folders, use name conventions like *Test.java, and *IntegrationTest.java.

Have you ever been in a project where you and your team dropped automated testing because you couldn't manage the mass of tests? Well, tools are starting to appear now that can save your hiney.

TestNG solves this. Why don't you just use that?
The first solution that many have become aware of is TestNG. I really like the feauteres, but as of Alex presentation on JavaZone, Maven's surefire plugin still has issues running TestNG. Also, Eclipse (our standard IDE) has built in support for JUnit4. And finally, Spring comes with a bunch of lovely standard testbase classes that make your integration tests nicer. Oh, wait. You don't want your test to inherit from a Spring class? Read on to find out. Final reason: everyone knows JUnit and not too many know TestNG (sorry Alex..).

There's a new kid in town
Spring TestContext Framework introduces a wide range of new annotations for doing tests, my absolute favourite being the @IfProfileValue that you can use to specify testing environment (in other words implement the test categorization I spoke of above). There is only one implemented ProfileValueSource (being the logic that decides which profile it is) that checks for system variables, but it is easy to extend with your own ProfileValueSource class that contains the logic for deciding which tests to run.

As promised: avoid extending Spring's testbase classes: Use the @RunWith(SpringJUnit4ClassRunner.class annotation instead.

You got the regular nifty annotations like @Repeat (guess what it does) @Timed (fails if the tests exceeds a time limit, different from JUnit 4's timeout attribute), plus a bunch of Spring environment controlling transactions and Spring contexts. You don't need to use the @ExpectedException any more because JUnit4 supplied us with the @Test(expected=RuntimeException.class) functionality. Spring people recommend that you use JUnit's way of doing it.

"Caching between testing", and "injection of test fixtures". Sounds lovely, doesn't it? Sounds alot like the presentation of TestNG, but hey, it's JUnit :)

There is also a pretty cool @ContextConfiguration annotation for declaring which Spring bean XML files you want to use for a particular test.

Actually, instead of me repeating the whole documentation, why don't you take a look for yourself. Note that I haven't actually played around with this properly yet (well, we got as far as getting the M4 build into our maven repository) cause I left my laptop in work. I'm gonna try setting up the machine here tonight and see if I can get something done before bedtime.

By the way, some shameless company promotion: My employer is doing a course on Spring 2 with Interface21 in Oslo in the beginning of November. Per is a really friendly guy, so don't hesitate to give him a call and ask about it if you're curious :)