You’ve probably read a lot on this blog about why automated tests are so awesome. But there’s another reason why they are really awesome: they are quite possibly the strictest critic of the code you write. More strict than your QA person, your pairing partner and the browser itself (yes, even IE ).

Automated browser tests can catch some of the smallest “bugs”. And I put the word bugs in quotes for a reason; browser tests can catch coding mistakes that are technically bugs, but aren’t actually visible to the user.

One category of these types of bugs is memory leaks. There have been several instances where our browser tests have become flaky or flat-out failed because some of our UI components and/or pages weren’t destroying themselves properly when they are no longer visible to the user. If the page/component is still in the DOM, the test will be able to interact with it which can cause the test to be flaky or fail. This issue has more potential to show itself if you re-use the same components in different parts of your app.

Memory leaks are a good example of something that wouldn’t typically have been found by QA or a regular user but a browser test will have no problem showing you that you messed up. Like I said, harshest critic you can find but they make you a better developer for sure .

Another quick Javascript example:
We had a boolean method on a Javascript object that looked something like this:

isParent: function() {
  return this.getChildren().length > 0;
}

We needed to make a change because this.getChildren() could now return undefined so we changed our method to this:

isParent: function() {
  var children = this.getChildren();
  return children && children.length > 0;
}

Luckily, we had also written a test to verify that isParent() would return false when children is undefined. It was failing and we couldn’t figure out why! Our change had even fixed the JavaScript error we were getting in the browser when getChildren() was returning undefined and everything looked great in the browser.

After some more inspection (our actual case was a bit more complicated so give us a break ) we realized that if children was undefined, isParent() would actually return undefined instead of a boolean value like the API says it should. Undefined is false-y in Javascript so the caller of isParent() didn’t happen to care that undefined was being returned instead of false in this case so everything looked great in the browser but the test showed us isParent() wasn’t adhering to its API.

I was really excited how writing a test made us realize that we had made a mistake!! A pretty small mistake all-in-all but nevertheless a mistake.

We recently wrapped up another hackathon. Like I did last summer, I spent my time in javascript. This time, I learned a few things about the key codes browsers send to javascript event handlers when non-alphanumeric keys are pressed.

First, in the modern browsers I tested (FF, Chrome, and (ick) IE9) the forward-slash key is registered as a key code of 191. This surprised me because the ASCII code is 47.

Along the same lines, the question mark key is registered as 191 with a shiftKey value of true. Again, I’d have expected the ASCII code of 63.

Here’s the curve ball in the whole thing. The one really random non-conforming browser is Firefox, which registers the question mark key as a key code of 0 (zero) with a shiftKey value of true. Shame on me for thinking IE would be the oddball.

Unlike the last time I spent time hackathoning in javascript, this project was handled all along with the intent of putting it into production. Keep an eye out for something cool involving the keyboard soon in Rally.

This week, Rally is holding the second annual RallyON conference. Today, Steve and I are presenting a brief history of (mostly automated) testing at Rally. There shouldn’t be a ton that’s new for regular readers of this blog, but I’m happy to share the slides with anyone who’s interested.

Track what’s happening at RallyON using the hashtag #RallyOn2012.

Sometimes I run across a task where I need to attach event listeners to many different dom elements of the same type. I am usually tempted to do something similar to the code below.

Ext.each(Ext.query('a', parentNode), function(el) {
    el.on('click', doSomethingAwesome);
});

This is ExtJs’s way to loop through each dom element matching the passed css selector and attach an event listener to it. I’m lazy and it’s less code than the correct implementation, which would be to add a listener to the parent element and filter for the correct target element inside the handler function. The disadvantages to the lazy method are that multiple listeners are created, all of which have some amount of overhead attached, and that listeners need to be removed and/or reattached whenever the innerHTML of the parent element changes.

An ExtJs specific example of when the ‘lazy’ method doesn’t work is when event listeners need to be attached to elements created by a renderer inside of a GridPanel cell. The GridPanel does not expose any row or cell render events, so there is no reliable way to add event listeners to dom elements located inside cells.

Fortunately ExtJs’s Element.on method has a helpful ‘delegate’ option that does all of this for you automatically. Use ExtJs’s Element.on method to attach a listener to the parent dom element and specify a css selector for the ‘delegate’ option to filter out events whose target node does not match the passed css selector.

var parentEl = Ext.fly(parentNode);
parentEl.on('click', function(event, target, options) {
    console.log('the "target" argument is the anchor element that is a child of parentNode');
}, this, {
    delegate: 'a'
});

On the Rally Engineering blog we’ve written many articles on systems’ performance, monitoring, resilience, and recovery. An interesting event just happened at our Denver production facility that pushed our systems to their limits. It helped us answer the question:

How well does production survive in the face of all-out failures?

Maybe inspired by Netflix’s Chaos Monkey, some local wildlife found its was into our server room and litterally tore apart our hardware.

Enter the “Rally Chaos Raccoon”

Last weekend the Chaos Raccoon, we now call “Cyril”*, chewed his way into our production server room. From security video footage we saw that at first he was cautious and quiet. Just hiding behind the Dell 4210s — where it’s warm.

But after a full day of no food Cyril got angry.

Pissed angry.

He gnawed on the mounting brackets of our UPS and ripped open a SAN unit in use by app-server-01.

The instant app-01 failed our backup app-server spun into action and took over the production traffic from app-01. The process all had to happen automatically — when Aaron tried to enter the cage for a manual failover Cyril attacked and bit his cheek (yes, that cheek).

As Aaron left the server room, seeking medical attention and rabies shots, the Chaos Raccoon ripped into another Dell 4210 and chewed its innards into shreds. Our system smoothly recovered, load-balancing network traffic whilst simultaneously paging the operations team another warning. When Cyril finally chewed through the UPS he electrocuted himself to death.

It smelled bad.

In the future we’ll probably test our processes with a “Bad-ass Bear”, “Crazy Coy Carp”, or “Janky Jackalope”. Opening up your production systems to wildlife attack demonstrates confidence in monitoring, recovery, and backup processes. It stretches your failover strategies to their limits. You may think your systems are ready for anything but when a raccoon attacks there are no rules.

If you’re going to implement your own Chaos Raccoon we recommend you first deploy an array of recovery tools and test with non-endangered creatures. It’s organic and the ecologically sound option.

* real names changed to protect the innocent

IntelliJ is an awesome IDE from JetBrains. jconsole is a powerful tool for examining the state of an executing JVM. When you’re debugging JVM code in IntelliJ, you might find that jconsole can’t connect to your JVM while you’re sitting on a breakpoint. This is because the default breakpoint behavior in IntelliJ is to stop the whole VM, not just the thread that’s sitting on the breakpoint. If you try to connect jconsole to the JVM in this state, it will time out.

This is easy to fix:
1. Right click on your breakpoint and choose “Properties”
2. Change the suspend policy from “All” to “Thread”

“…and you’re done kid.” Imagine I said that in a Boston accent.

You can now connect to the JVM while sitting on that breakpoint. Problem solved.

Help! My Java application keeps crashing! OutOfMemoryError?! My hair is on fire! Please don’t let the invisible fire burn my friend!!!

1. Increase max heap size

Some applications are complex and large enough that they simply need more memory.
Try increasing the max heap size when you start the JVM by adding the flag -Xmx. So your start-up will look something like this: java -Xmx1024m MyApp (which is equivalent to java -Xmx1g MyApp)

The downside to this is that your application will suffer from longer garbage collection cycles, it may be better to fix the greedy code.

But what if your application still runs out of memory? Seems like you have a leak.

2. Detect memory leak

Learn to use a tools like jmap and Thread Dump Analyser (TDA). Run jmap -histo
(where pid is the process-id of your Java application) a few times and compare the output. Read my earlier blog posts on fun with heap dump analysis.

3. Carefully read your code

There are many ways to leak memory in Java code. Watch out for thread local variables — especially in a thread pool. Each thread stores its own state so anything you put on there will last the lifetime of the thread. If your threads are recycled through a thread pool that state may hang around for a long time.

Mutable static collections are bad. Static fields are retained by the class and therefore its classloader. That means your collection will hang around for the lifetime of that class. Come to think of it, it is better not to use mutable statics.

If you re-implement the hashcode and equals methods and get it wrong you’ll shoot yourself in the foot. Here is why: when you store an object in a HashMap the collection will call your (wrong) hashcode method to determine that object’s position. On look-up, the map uses the (wrong) hashcode method again and then calls (wrong) equals method to check it retrieved the correct object. If you break the implementations of these two methods you can add objects into the HashMap but can’t get them back. Worse, you can add the same thing repeatedly and it will not overwrite the old value because hashcode keeps returning different values for the same object.

4. Circuit breakers

If you do have to compute over potentially large collections of objects then use weak references as circuit breakers. I wrote a series of blog posts that will serve as a good primer for weak references. In particular: JVM Memory Primer, Java References and Reachability Explained.

Monitoring JVM memory usage and tweaking system options is something we do all the time. Once you’ve fixed your leaks you need to continue to monitor systems and run load tests to project usage into the future. Be proactive; don’t wait for an OutOfMemoryError to remind you.

This one’s going to be short and sweet, with no sample code. Recently, I’ve been blogging about writing effective integration tests.

If you’re going to make the time investment (both development time and execution time) in writing integration tests, you need to make them work as much like your production environment as possible. Exercise as much of the same stack as you can.

If you’re writing to a database, you should be using the same connection pooling and transaction management mechanism as production. If you’re using an inversion of control container, you should be using the same one as production and loading as many of the same components as possible.

Why? If you don’t, your integration tests are not really proving anything. If you’re committing to a database, but not the same way you will in production, that leaves room for data inconsistencies. If you’re only loading part of your IoC container, that leaves room for unexpected wiring of dependencies.

Make your integration tests like production. It’ll save you from bugs later.

In my last post I promised to share some tips that can help developers write effective integration tests.

Committing to a database is expensive. In many standard SQL databases, it’s pretty much the most expensive thing you can do. By comparison, querying is relatively inexpensive. If possible, you should structure your integration tests in such a way that you assert as much as possible for each commit that’s made.

Consider this test. Rally supports the notion of a hierarchy of projects, structured into a tree. Individual users can have access to any node(s) within the tree and should only see data from those projects. This test creates users and projects and verifies our permission model.

public void whenOnParentScopedDownOnlyIncludeChildProjectsWhereUserHasAccess(PersistenceContext context) {
  Project parent = createProject();
  Project editorAccessChild = createProject(parent);
  Project viewerAccessChild = createProject(editorAccessChild);
  Project noAccessChild = createProject(parent);

  User user = createUserAsProjectEditor(parent);
  makeProjectEditor(user, editorAccessChild);
  makeProjectViewer(user, viewerAccessChild);
  context.commitAndReset();

  ProjectOidsInScope scope = scopeTo(parent).as(user).includeChildren();

  assertThat(scope.get(), containsOidsOf(parent, editorAccessChild, viewerAccessChild));
}

public void whenOnChildScopedUpOnlyIncludeParentProjectsWhereUserHasAccess(PersistenceContext context) {
  Project noAccessGrandParent = createProject();
  Project viewerParent = createProject(noAccessGrandParent);
  Project editorProject = createProject(viewerParent);

  User user = createUserAsProjectEditor(editorProject);
  makeProjectViewer(user, viewerParent);
  context.commitAndReset();

  ProjectOidsInScope scope = scopeTo(editorProject).as(user).includeParents();

  assertThat(scope.get(), containsOidsOf(editorProject, viewerParent));
}

These are good tests. With a little understanding of the Rally domain, they’re highly readable and they test the right things.

So what’s the problem? We’re committing twice when once will do. While the impact on any individual test is fairly minimal, the impact of unnecessary commits is compounded when spread across thousands of tests. Even within this test class, these aren’t the only two tests. There are ten.

What if we wrote the test like this?

private Project editorProject;
private Project editorAccessChild;
private Project viewerAccessChild;
private Project noAccessChild;
private User user;
private Project noAccessGrandParent;
private Project viewerParent;

@BeforeMethod
protected void setupData(PersistenceContext context) {
  if(dataCreated) return;
  dataCreated = true;
  noAccessGrandParent = createProject();
  viewerParent = createProject(noAccessGrandParent);
  editorProject = createProject(viewerParent);
  editorAccessChild = createProject(editorProject);
  viewerAccessChild = createProject(editorAccessChild);
  noAccessChild = createProject(editorProject);

  user = createUserAsProjectEditor(editorProject);

  makeProjectViewer(user, viewerParent);
  makeProjectEditor(user, editorAccessChild);
  makeProjectViewer(user, viewerAccessChild);

  context.commitAndReset();
}

public void whenOnParentScopedDownOnlyIncludeChildProjectsWhereUserHasAccess(PersistenceContext context) {
  ProjectOidsInScope scope = scopeTo(editorProject).as(user).includeChildren();

  assertThat(scope.get(), containsOidsOf(editorProject, editorAccessChild, viewerAccessChild));
}

public void whenOnChildScopedUpOnlyIncludeParentProjectsWhereUserHasAccess(PersistenceContext context) {
  ProjectOidsInScope scope = scopeTo(editorProject).as(user).includeParents();

  assertThat(scope.get(), containsOidsOf(editorProject, viewerParent));
}

We’ve moved the data creation into a setup method that runs (and commits) only once. We’ve sacrificed a small amount of readability within each test because the data setup isn’t directly adjacent to the actions and assertions. We have, however, gained some consistency across tests. Once we know the data structure we’re working with, we don’t need to mentally parse each test separately to figure out what shape the data is in before reading the rest of the test.

Integration tests are expensive, so achieving maximum bang for buck is important. If you can verify multiple behaviors while only doing the expensive part of the test once, DO IT!

I love TDD and Unit Testing. That said, there are times and places where Integration Testing is more appropriate. In this series of posts, I’m going to explore patterns (and anti-patterns) of integration testing and try to identify when each is appropriate.

First off, what’s an integration test? The loosest definition I can come up with is that an integration test involves either (1) access to out of process resources like databases or file systems or (2) exercises several parts of a larger system working together. For the most part, I’m going to focus on scenario 1.

In no particular order, here’s what I plan on blogging about in this series:

• Enforce your conventions!
• Limit your commits
• Multiple asserts are OK
• Set up the data as infrequently as possible
• Don’t test shite that should be tested in a faster test
• This shite still needs to be fast
• Make your tests like production
• No, really, this shite needs to be fast!
• Readability is still damn important

There’s certainly some overlap in that topic list, so I’m sure I’ll cover multiple topics in every post. I’ll be sure to get to all of the topics, though.