For the past few months we’ve been working with scrum teams who are based in the UK, the US, Eastern Europe and Latin America. In general it’s been a great success but we have certainly learnt things along the way and I thought it would be useful to share these experiences. We used Martin Fowler’s model as a starting point for our own system and this proved to be very effective. We adopted pretty much all of Martin’s recommendations and I would highly recommend his article to anyone looking to implement their own distributed agile model.

Firstly let’s look at our logical team structure:

As can be seen, we use functional teams with component guardians providing the technical oversight of the application tiers. From a geographic perspective we have product owners based in the UK and US, architects and scrum masters based in the UK, US, Argentina and Estonia and scrum teams in the UK, Argentina and Estonia. Support elements are largely based in the UK and US.

We spent a fair amount of time working out the best geographic team distribution but in the end we came to one conclusion:

Timezone is the most important factor, everything else can be overcome

As a refinement of that broad statement we can say that product owners and scrum masters must be in the same timezone as their respective scrum teams, however none of them need to be situated in the same physical office. This gave us a logical structure – UK based product owners would work with scrum teams in Estonia (only a 2 hour difference) and US product owners would work with teams in Argentina (again only a 2 hour difference). We then had to decide whether to situate scrum masters with the scrum teams or with the product owner. On balance we found it better to situate them with the product owner because technical people were able to collaborate online easier than non-technical people.

Make full use of online tools

Technology has evolved so much over the past years that it’s now possible to do pretty much everything online in a collaborate environment. Some specific tools that are worthy of mention include:

Skype – Probably the most useful tool we use, being able to talk to people face to face has really revolutionized the way we work. Skype has also allowed us to build a social community across borders … we can point the webcam out of the window to show other team members the horrible weather, or the not so horrible girl walking outside.

Atlassian tools – We’ve used JIRA for some time but we’ve recently adopted the other Atlassian tools (Greenhopper, Confluence, Fisheye, Crucible) – There was certainly a learning curve, especially for Greenhopper which does not directly map to scrum in the way VersionOne does for example. It was worth the effort though because we now keep everything online. I was never a great fan of post it notes (being a tidy person) but Greenhopper seems to give us the same flexibility with more control

GoToMeeting/Webex - Being able to share a screen proved useful for technical and non-technical people. The product owners were able to illustrate things whilst the technies could adopt a pseudo pair-programming approach – they would share their computers and ask someone else to look at a problem they were facing.

Amazon web services – One challenge we faced was where to situate our development and test servers, how could we offer a small team in Argentina the same infrastructure as our much larger UK team? The solution was to allow teams (even individual developers) to spin up servers as and when needed.  AWS allowed us to offer a full performance test infrastructure to every team involved in the project.  It also helped to overcome the head office/regional office mentality in which developers in the head office can get “special treatment” from the engineering guys who are based alongside them.

What else did we learn?

• Don’t rely on email – It’s very easy to slip into long email threads which become unmanageable, we also found that people were hesitant to ask questions which they thought may have been answered in a previous email. They would spend 15 minutes searching through their email history and if they did find the answer it was often out of date.  Much better to ping someone on IM “can I ask you someting?” then call them using Skype
• Watch IM – chat can be addictive but in our experience it’s not as effective as a call … most people can talk quicker than they type and IM is inherently asynchronous – you can’t interrupt someone when all you see is “Dave is typing …”
• Share your screen, don’t check in bad code – Developers were often tempted to check in unfinished code so they could share it with their colleagues. Although source control should be used for collaboration it’s not a dumping ground.  Screen sharing was the answer
• Keep an online calendar showing key dates including public holidays – Initially we were caught out when a guy in the US would sit waiting on information from someone in Argentina, not realizing it was a public holiday. Google calendar works well for this purpose

TestNG (the NG stands for Next Generation) offers the same features as JUnit but also offers some significant enhancements that are lacking in JUnit. A detailed comparison of the two frameworks can be found in this excellent blog post. In our experience the most usable features of TestNG are test groups, parametrized testing and thread support.

Test Groups

Take this scenario … we have a traditional multi tiered web application with anaemic domain classes and stateless services. Having changed a service we want to run our unit tests but we don’t necessarily want to run all our domain tests because the domain classes should not be affected by a change in the service tier. We could create two test suites, one for the service tests and one for the domain tests but TestNG offers us test groups, basically tags that we can assign to individual tests (not just test classes).

@Test(groups = "service")
public void testMyCode() { ... }

We can then tell maven to execute tests with a particular tag …

 <plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-surefire-plugin</artifactId>
 <configuration>
 <groups>${groups}</groups>
 </configuration>
 </plugin>

$mvn -Dgroups=”service”

The TestNG ecliupse plugin also offers a simple way of executing tests with a particular tag:

Paramaterized testing

Writing good unit tests is only half the battle, maintaining them is equally challenging. One way to reduce the amount of boilerplate code is to write paramaterized tests. JUnit offers paramaterized support but it’s not clean. We need to create a static method which returns a collection of paramaters, create instance variables to hold the current paramaters, then implement a constructor which ties the static method and the instance variables together:

@RunWith(value = Parameterized.class)
public class JunitTest6 {

 private int number;

 public JunitTest6(int number) {
 this.number = number;
 }

 @Parameters
 public static Collection<Object[]> data() {
 Object[][] data = new Object[][] { { 1 }, { 2 }, { 3 }, { 4 } };
 return Arrays.asList(data);
 }

 @Test
 public void pushTest() {
 System.out.println("Parameterized Number is : " + number);
 }
}

In contrast TestNG’s approach is much cleaner:

public class TestNGTest {

 @DataProvider("numbers")
 public Object[][] getNumbers() {
 Object[][] data = new Object[][] { { 1 }, { 2 }, { 3 }, { 4 } };
 return data;
 }

 @Test(dataProvider="numbers")
 public void pushTest(int number) {
 System.out.println("Parameterized Number is : " + number);
 }
}

The distinction between JUnit and TestNG becomes even more apparent when we need to support multiple groups of paramaters:

public class TestNGTest {

 @DataProvider("numbers")
 public Object[][] getNumbers() {
 Object[][] data = new Object[][] { { 1 }, { 2 }, { 3 }, { 4 } };
 return data;
 }

 @DataProvider("names")
 public Object[][] getNames(){
 Object[][] data = new Object[][] { { "fred" }, { "harry" }, { "bob" }, { "alice" } };
 return data;
 }

 @Test(dataProvider="numbers")
 public void pushNumers(int number) {
 System.out.println("Parameterized Number is : " + number);
 }

 @Test(dataProvider="names")
 public void pushStrings(String name) {
 System.out.println("Parameterized name is : " + name);
 }

}

JUnit’s big limitation is that it only support a single dataprovider method which must set up all parameters for all tests.

XML Driven paramaters

TestNG also allows us to specify our parameters in an XML file which is often more readable:

<!DOCTYPE suite SYSTEM "<a href="http://beust.com/testng/testng-1.0.dtd" target="_blank">http://beust.com/testng/testng-1.0.dtd</a>" >
 <suite name="a">
 <test name="testing">

 <parameter name="numbers" value="1"/>
 <parameter name="numbers" value="2"/>
 <parameter name="numbers" value="3"/>

 <classes>
 <class name="com.fsecure.demo.testng.TestNGTest6_0" />
 </classes>

 </test>
 </suite>

 @Test(dataProvider="numbers")
 public void pushTest(int number) {
 System.out.println("Parameterized Number is : " + number);
 }

Testing threads

Anyone who has written a high performance application will have dealt with threads and thread pools. Some libraries (e.g. Spring Security) use threads behind the scenes. Our code may well run under a single thread of execution but what happens when it’s being hit by 10 concurrent threads?

Let’s take some naive code:

package com.test;

public class NaiveClass {

	private int value = 0;

	public void increment(int amount) {
		value += amount;
	}

	public int getValue() {
		return value;
	}

	public void setValue(int value) {
		this.value = value;
	}

}

And our JUnit test:

package com.test;

import static org.junit.Assert.*;
import java.util.Random;

import org.junit.Before;
import org.junit.Test;

public class TestNaiveClass {

	private NaiveClass classUnderTest;
	private Random generator;

	@Before
	public void setup() {
		classUnderTest = new NaiveClass();
		generator = new Random();
	}

	@Test
	public void testNaiveClass() {
		int startValue = generator.nextInt(100);
		int incrementValue = generator.nextInt(100);
		classUnderTest.setValue(startValue);
		classUnderTest.increment(incrementValue);
		int expected = startValue + incrementValue;
		assertEquals(expected, classUnderTest.getValue());
	}
}

and the result …

We know our code is flaky but no matter how many times we run the test it will always pass … not good!

Of course our test runs in a single thread so we can’t test thread safety. We need to fire off multiple threads:

package com.test;

import static org.junit.Assert.assertEquals;

import java.util.Random;

import org.junit.Test;

public class TestNaiveClassThreaded {

	private final NaiveClass classUnderTest = new NaiveClass();
	private final Random generator = new Random();

	@Test
	public void testNaiveClass() {
		for (int i=0; i<100; i++) {
			ThreadedTest threadTest = new ThreadedTest();
			Thread t = new Thread(threadTest);
			t.start();
		}
	}

	class ThreadedTest implements Runnable {

		@Override
		public void run() {
			try {
				int startValue = generator.nextInt(100);
				int incrementValue = generator.nextInt(100);
				classUnderTest.setValue(startValue);
				Thread.sleep(100);
				classUnderTest.increment(incrementValue);
				int expected = startValue + incrementValue;
				assertEquals(expected, classUnderTest.getValue());
			} catch (InterruptedException e) {
				throw new RuntimeException(e);
			}
		}

	}
}

This should fail …

Our test still passes! Something strange is happening here … the threads should sleep for 10 seconds but our test passes immediately. The problem lies with JUnit’s TestRunner which doesn’t support threads … it fires off threads then returns immediately. We can easily prove this:

package com.test;

import static org.junit.Assert.fail;

import org.junit.Test;

public class TestFail {

	@Test
	public void testNaiveClass() {
		for (int i=0; i<100; i++) {
			ThreadedTest threadTest = new ThreadedTest();
			Thread t = new Thread(threadTest);
			t.start();
		}
	}

	class ThreadedTest implements Runnable {

		@Override
		public void run() {
			fail();
		}

	}
}

This test always passes!

TestNG supports threads

In contrast TestNG offers explicit support for threads. We can easily verify that our NaiveClass isn’t thread safe by writing a simple test:

package com.test;

import static org.testng.Assert.assertEquals;

import java.util.Random;

import org.testng.annotations.Test;

public class TestNaiveClassTestNG {

	private final NaiveClass classUnderTest = new NaiveClass();
	private final Random generator = new Random();

	@Test(threadPoolSize=10, invocationCount=100)
	public void testNaiveClass() throws Exception {
		int startValue = generator.nextInt(100);
		int incrementValue = generator.nextInt(100);
		classUnderTest.setValue(startValue);
		Thread.sleep(10);
		classUnderTest.increment(incrementValue);
		int expected = startValue + incrementValue;
		assertEquals(classUnderTest.getValue(), expected);
	}
}

TestNG will create 10 threads and execute the test 100 times. As we can see the test fails because multiple threads are modifying the instance variable …

Conclusion

We believe TestNG offers some significant enhancements over JUnit and from what we have seen there are no drawbacks … it really is the Next Generation testing framework!

Cloud technology has been around for some time but it’s generally been confined to large public clouds like Amazon and Google app engine. The world is changing … organizations are now starting to build their own clouds to support their agile teams.

In this talk Toby Hobson will give an overview of cloud technology before exploring the benefits and pitfalls of private cloud technology for agile development teams. The talk will conclude with a comparison of the commercial and open source technologies on the market today. Topics addressed will include:

• Why infrastructure is the Achilles heel of agile development
• How clouds can empower development teams
• Development teams as customers
• Current trends
• Dealing with legacy systems

Technologies covered will include:

XEN, KVM, VM Ware, Eucalyptus/Ubuntu Enterprise cloud, 3Tera and OpenQRM along with an overview of Amazon’s private cloud offering

More information can be found on the Skillsmatter site

For those of you who don’t know about JRebel it’s a JVM agent that allows developers to change code on the fly without needing to rebuild and redeploy their apps. Sounds like hotswap? well actually it’s much more advanced than traditional jvm hot swap because it allows you to modify the structure of classes on the fly so you can add and remove methods, fields etc. It also has some neat plugins like the spring plugin which lets us add beans on the fly (hot swap definitely won’t let you do that!)

We’ve found that fast turnaround really helps us to improve the quality of our code. Take this scenario … you want to test a small change (e.g. adding a field to a JSP) but every time you change something it take 2 mins for your app to start up. What happens? you don’t make and test changes in isolation because it’s too painful so you end up changing lots of stuff and hope it all works!

The tool not only allows us to be more productive, it also brings a little bit of happiness into our day … there’s nothing more frustrating that waiting for your app server to restart.

Give it a go … you may be surprised!

Whilst Grails and Roo both aim to simplify the developer’s life they approach it in different ways. Grails is built on top of groovy and adds a lot of “magic” at runtime whereas Roo is essentially a code generation utility.

Performance

The benefit of compile time code generation is of course performance. A Roo powered application will almost always be faster than a Grails powered app but the performance “hit” associated with Grails is not as great as people sometimes think.

When we have run into performance problems in Grails the bottleneck in invariably a couple of methods or closures which perform computationally expensive calculations. Groovy implicitly uses BigIntegers for division which imposes a big performance hit. Extracting such calculations to static java methods which use floats often speeds up execution time by as much as 90%. Another area where Grails performance is lacking is the rendering engine behind the GSPs and specifically the way tag libraries are used. Switching to another rendering engine can significantly improve performance.

Both Roo and Grails can be tweaked to improve performance without making changes to the code base. Judicious use of caching and transaction management will give significant performance increases for both platforms. The use of transactions is especially significant for grails apps because by default grails will try to create a transaction for every request which is far from ideal.

So in summary out of the box grails is considerably slower than a Roo powered app but with a bit of tweaking we have found that grails apps can come very close to “traditional” java ones and certainly acceptable for all but the most demanding of applications.

Skills

To make the best use of Grails a developer must have a good grasp of Groovy whereas Roo is pure java so there is no need to learn another language. This shouldn’t be underestimated, I believe that any experienced Spring developer will have no trouble learning Groovy … the question is do you have the time to learn Groovy. Top developers are in high demand and getting time off to study something new is often easier said than done!

Grails and groovy is not hard but to become proficient you do need to spend a fair amount of time with real world examples. If you are a java developer and you have the chance to work on a grails project I would say jump at it … you’ll love it!

In contract a competent java developer can be up to speed and developing productive Roo apps in days. The code that is generated will look familiar and you’ll feel confident to experiment to get the best results. Roo is also ideally suited to more junior developers who may not be comfortable with some of the more “esoteric” features of groovy and grails but just want a framework that will save them a bit of time.

Availability of libraries/plugins

Roo is built on top of Maven so dependency management is a breeze and there is an extensive library of code available. In contrast grails relies on plugins which have a limited availablity. Grails can be used with Maven but it’s a painful experience and not something we would recommend. This is an area where Roo wins hands down. Ide support for grails projects is also lacking, even Spring own ide (STS) doesn’t have a specific groovy editor and debugging a grails application is not for the faint hearted. Grails generates a lot of code at runtime so the code you see in your debugger bears little relation to the code actually running. This can be a major headache but careful use of breakpoints and logging statements can ease the pain somewhat.

Maintenance

We’re not big fans of code generation tools, my own view is that if you find yourself writing or generating a lot of boilerplate code and config there is something wrong with the structure of your application. The core spring framework has been moving towards convention over configuration in recent releases, annotation driven controllers are the best example of this. Roo builds on this but it’s nowhere near Grails in terms of ease of maintenance. We are still amazed at just how little code we need to write to deliver a fully functional Grails powered app.

We have not yet developed and maintained any production grade Roo apps but my guess is that it could actually act as an impediment to the long term success of the project. I say this because I suspect developers who may be under pressure with tight deadlines will tend to rely on the code generation rather than thinking how they can structure the application to minimize code duplication.

Conclusion

Grails and Roo are both great frameworks. The framework you choose will depend on your need for performance vs maintainability but ultimately I think the big factor will the level of skills in your team. An advanced developer will be able to develop amazing Grails apps that are far more maintainable than those developed using Roo. Junior developers will be able to develop a basic grails app but the moment you hit performance or threading issues you’ll need to climb quite a high wall. Roo can be used by developers of all levels to boost their productivity.

The most basic spring security configuration looks something like this:

<http auto-config='true'>
  <intercept-url pattern="/**" access="ROLE_USER" />
</http>

This configuration tells spring to restrict access to the application to only those users who have the role of “ROLE_USER”. Behind the scenes spring creates and wires up several beans that all play a role in the authorization process. I’m going to assume that we are protecting web urls (known as FilterInvocation’s in Spring) but I’ll cover method level security in a later post

Before we start let’s have a look at the key collaborators involved in spring authorization:

DelegatingFilterProxy

You will most likely have defined and mapped an instance of the DelegatingFilterProxy in your web.xml. This proxy allows Spring to create a number of filters needed for authentication and authorization. One such filter which is relevant to url authorization is the FilterSecurityInterceptor

FilterSecurityInterceptor

This is the primary entry point for url authorization, it intercepts requests and passes control to the AccessDecisionManager

AccessDecisionManager (Interface)

This is the primary interface responsible for Authorization. As the name implies Implementors are responsible for deciding whether to grant or deny access to a particular resource (known as a secure object in Spring). In the context of web based url security the secure object will be an instance of the FilterInvocation class. AccessDecisionManager’s key method is

void decide(Authentication authentication, Object secureObject,
 ConfigAttributeDefinition config)
 throws AccessDeniedException, InsufficientAuthenticationException

The authentication represents the principle (or “user”) accessing the system, the secureObject will be an instance of FilterInvocation. The config is basically a collection of name/value pairs that are passed to the class if they are needed.

Out of the box spring will create an instance of the AccessDecisionManager which supports role based authorization. We can write our own implementation if we need some custom behavior. Lets say we want to restrict access to admin user’s with particular IP addresses:

package com.test.security;

import java.util.Collection;

import org.springframework.security.access.AccessDecisionManager;
import org.springframework.security.access.AccessDeniedException;
import org.springframework.security.access.ConfigAttribute;
import org.springframework.security.authentication.InsufficientAuthenticationException;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.GrantedAuthority;
import org.springframework.security.web.FilterInvocation;

public class NaiveAccessDecisionManager implements AccessDecisionManager {

 public void decide(Authentication authentication, Object secureObject,
     Collection attributes) throws AccessDeniedException,
     InsufficientAuthenticationException {

   // The supports method ensures we are dealing with FilterInvocations
   //  so we can safely cast the secure object
   FilterInvocation invocation = (FilterInvocation) secureObject;
   WebAuthenticationDetails requestDetails = (WebAuthenticationDetails) authentication.getDetails();

   if ("/secret.do".equals(invocation.getRequestUrl())) {
     boolean authorized = false;

     // only allow access if the user has the admin role a
     // and is within the correct IP range
     for (GrantedAuthority authority : authentication.getAuthorities()) {
       if ("ROLE_ADMIN".equals(authority.getAuthority()) && requestDetails.getRemoteAddress().startsWith("192.168.0")) {
         authorized = true;
       }
     }

     if (!authorized) {
       throw new AccessDeniedException("go away");
     }
   }
 }

 public boolean supports(Class clazz) {
   // This manager should be used for FilterInvocations (authorizing web
   // requests)
   return FilterInvocation.class.isAssignableFrom(clazz);
 }

 public boolean supports(ConfigAttribute config) {
   return true;
 }

}

We now need to tell spring to use our manager:

<bean:bean id="naiveAccessDecisionManager" class="com.test.security.NaiveAccessDecisionManager" />
<pre><pre>
<http auto-config="true" access-decision-manager-ref="naiveAccessDecisionManager">
 <!-- no need for interceptor element here because our accessDecisionManager has this hard coded -->
</http>

That would work but it’s not very nice.  For one thing we’re reimplementing spring’s role based access control which doesn’t make sense. It can also become messy if we have conditional logic

To simplify things Spring offers us the concept of “voters” which are polled by an AccessDecisionManager.

AccessDecisionVoter (interface)

Let’s have a quick look at the key method in the AccessDecisionVoter interface:

int vote(Authentication authentication, Object secureObject, ConfigAttributeDefinition config);

Voter’s can be chained together by an AccessDecisionManager to build complex authorization strategies. The way an AccessDecisionManager handles “votes” depends on it’s type; the Unanimous AccessDecisionManager requires all voters to say yes before allowing access. In contrast the Affirmative AccessDecisionManager just needs any one of the voters to say yes. Spring offers several concrete implementations of the AccessDecisionVoter, the most common of which is the RoleVoter.

A real world example

We recently did some work for a media company whose licensing and rights policy meant that they could only display content to users in the UK. We chose to implement a custom voter which would try to geolocate the visitors IP address and allow access if it was reported as being UK based:

package com.x;

public class IPVoter implements AccessDecisionVoter {

  public int vote(Authentication authentication, Object secureObject, Collection<ConfigAttribute> attributes) {

    if (authentication.getDetails() == null) {
      return AccessDecisionVoter.ACCESS_DENIED;
    }

    WebAuthenticationDetails details = (WebAuthenticationDetails) authentication.getDetails();
    String remoteIPAddress = details.getRemoteAddress();
    if (StringUtils.isBlank(remoteIPAddress)) {
      return AccessDecisionVoter.ACCESS_DENIED;
    }

    // checks the IP address using Quova
    return validateIPAddress(remoteIPAddress);
  }

  public boolean supports(ConfigAttribute att) {
    return true;
  }

  public boolean supports(Class clazz) {
    return FilterInvocation.class.isAssignableFrom(clazz);
  }
}

We then needed to wire up our new voter along with Spring’s RoleVoter:

<bean id="accessDecisionManager" class="org.springframework.security.vote.UnanimousBased">
  <property name="decisionVoters">
    <list>
      <bean class="org.springframework.security.vote.RoleVoter" />
      <bean class="com.x.ipVoter">
        <property name="quova" ref="quova" />
      </bean>
    </list>
</bean>

<security:http access-decision-manager-ref="accessDecisionManager">
...
</security>