Bits in Motion

Gallio v3.2 Beta

2010-05-10T00:10:00.001-07:00

A couple weeks ago I posted that the new beta with R# 5.0 and VSTS 2010 support would be available. Unfortunately I forgot to go back and post an update with a link.

Here’s the link: http://ccnet.gallio.org/Distributables/

Interim release notes: http://www.gallio.org/wiki/doku.php?id=release_notes

Happy testing!

Gallio v3.2 Beta Later this Week

2010-04-14T11:22:00.001-07:00

This is just a short note to say that we will be releasing Gallio v3.2 Beta later this week with support for Visual Studio 2010 RTM and ReSharper 5.0.

Sneaky Equality Bugs

2010-03-23T01:28:00.001-07:00

While trying to figure out why running a single test with Gallio causes ReSharper 5.0 to throw up the test failure in a message box (which it never did it before), I found this sneaky little bug.

public abstract class RemoteTask : IEquatable<RemoteTask>
{
    public string RunnerId { get; set; }

    public override bool Equals(RemoteTask other)
    {
        return Equals(other as RemoteTask);
    }

    public bool Equals(RemoteTask other)
    {
        return other != null && RunnerId == other.RunnerId;
    }

    public override int GetHashCode()
    {
        return RunnerId.GetHashCode();
    }
}

Can you spot the bug?

Magic Triad: Dictionary<TKey, TValue>, EqualityComparer<T> and IEquatable<T>

Whenever we construct a Dictionary<TKey, TValue> without providing an explicit IEqualityComparer<TKey> in the constructor (which is most of the time), the .Net framework automatically provides one for us. Namely, it uses EqualityComparer<TKey>.Default.

EqualityComparer<T> is one of those really useful generic classes that is really easy to overlook. Its job is to choose an appropriate implementation of IEqualityComparer<T> for any given type T. Here are the cases it considers.

If T is type byte, then use a ByteEqualityComparer that does the obvious byte comparison.
If T is assignable to type IEquatable<T>, then use a GenericEqualityComparer<T> that will call IEquatable<T>.Equals method to determine equality.
If T is nullable and its value type TV is assignable to type IEquatable<TV>, then use a NullableEqualityComparer<T> that performs a null-check then calls IEquatable<TV>.Equals(TV) method to determine equality of non-null values.
If T is an enum whose underlying type is int, then use an EnumEqualityComparer<T> that compares enum values as integers.
Otherwise, use an ObjectEqualityComparer<T> that performs a null-check then compares objects using Object.Equals.

All of this magic is designed to do one thing: minimize the number of boxing conversions and casts required to compare objects.

The Bug

Did you notice that RemoteTask is abstract? That tells you that it is intended to be subclassed.

To ensure correct equality and hash code semantics, a subclass must override at least the following two methods:

bool IEquatable<RemoteTask>.Equals(RemoteTask)
int Object.GetHashCode()

(Note: It is not necessary to override object.Equals(object) because all it does is delegate to IEquatable<RemoteTask>.Equals(RemoteTask). Due to covariance of the equality relation, this code will work fine for all subclasses as long as they override IEquatable<RemoteTask>.Equals(RemoteTask) with their extra logic.)

Unfortunately RemoteTask’s implementation of IEquatable<RemoteTask>.Equals(RemoteTask) is not virtual! Consequently subclasses cannot override it. No matter what they do, they are stuck with an implementation that only considers the RunnerId field when determining equality.

As a matter of fact, all subclasses of RemoteTask override object.Equals(object) but not IEquatable<RemoteTask>.Equals(RemoteTask). That’s really bad because the methods will probably return different results! Any code that uses IEquatable<RemoteTask> to compare RemoteTask instances will be broken.

This bug has been in ReSharper for a few releases now but it only just reared its ugly head due to an internal refactoring.

Previous Usage of RemoteTask.Equals (Hides the Bug)

Previously, ReSharper performed a lookup of tasks using linear search, something like this:

private List<KeyValuePair<RemoteTask, UnitTestElement> tasks;

public UnitTestElement GetTestElement(RemoteTask task)
{
    foreach (RemoteTask candidate in tasks)
    {
        if (object.Equals(task, candidate.Key))
            return candidate.Value;
    }
    return null;
}

Notice that this code calls the object.Equals(object, object) static helper function which performs a null check then determines equality using object.Equals(object). In this case, subclasses that just override object.Equals(object) will be fine.

Refactored Usage of RemoteTask.Equals (Triggers the Bug)

For efficiency, ReSharper now performs a lookup of tasks using a hash table, something like this:

private Dictionary<RemoteTask, UnitTestElement> tasks;

public UnitTestElement GetTestElement(RemoteTask task)
{
    UnitTestElement element;
    tasks.TryGetValue(task, out element);
    return element;
}

What’s new here is that under the hood, the Dictionary<RemoteTask, UnitTestElement> uses EqualityComparer<RemoteTask>.Default. That in turn compares values using IEquatable<RemoteTask>.Equals(RemoteTask)…

Oh yeah, that’s the method that subclasses can’t override…

The net result is that the hash table considers all values to be equal as long as they have the same RunnerId, regardless of their type or whatever other properties they may consider in their implementation of object.Equals(object).

For the most part, this is only a problem when there are hash code collisions. If the hash codes are always distinct then it won’t matter to the hash table that the equality comparer it is using returns true more often than it should…

In other words, the fact that this code appears to work at all today is a matter of pure luck given that hash code collisions are fairly rare (for well designed hash functions, at least).

Moral

If you implement IEquatable<T> in a class that is intended to be subclasses, make sure to make it virtual! Also make sure the subclasses override it as needed!

Accessing Non-Public Members Using Mirrors

2010-02-11T23:05:00.001-08:00

MbUnit v3.2 introduces a new feature called Mirrors for accessing non-public members.

Please tell me what you think. :-)

Setting Up a Distributed Build – Part 1

2010-02-11T18:55:00.001-08:00

Earlier today I wrote about the fact that I need to rebuild the Gallio build server. Then I went downstairs to get ready to leave for work and I promptly twisted my ankle.

So… now lying in bed with an ice pack strapped to my ankle, this seems like as good a time as any to start planning.

Requirements

Gallio currently ships with plugins for several 3rd party tools. It supports the .Net CLR 2.0 and 4.0 on 32bit and 64bit Windows platforms and aims to eventually support Mono on Windows and on Linux. All told there are quite a few configurations to be tested.

Here’s what we need:

Distributed compilation: components are compiled on platforms that provide the required dependencies.
Distributed testing: components are tested on platforms that reflect a significant subset of the supported configurations.
Distributed packaging: groups of components need to be assembled into packages on platforms that support the required packaging tools.
Fully automatic continuous integration.

Here are some nice things to have:

Centralized control of builds and maintenance procedures.
Centralized reporting of test results, code coverage, performance metrics and defects.
Centralized publication of all artifacts such as binaries and documentation.
Hermetic build environment.

Observation:

Most components are binary portable and should only be compiled once but may need to be tested more than once depending on how the configuration of the environment affects the component.

Hermetic Build Environment

Hermetic means “sealed.”

A hermetic build environment is one which encapsulates all of its dependencies so that the result of the build is entirely reproducible.

Think for a moment about the dependencies of your own build process.

Source code.
3rd party libraries and tools.
SDKs, compilers, frameworks, header files.
Test tools, coverage tools, build automation tools, documentation tools, linting tools.
Scripting languages, command-line tools, scripts.
Configuration files, registry entries.
Custom patches, tweaks.
Operating System revision and patch level.
Crazy glue.
Position of the mouse pointer on the screen. (Please don’t depend on this!)
Time of day. (Ditto.)

The more dependencies you control, the more reproducible will be the build.

The idea of establishing a hermetic build environment is to ensure that the dependencies are tightly controlled. If I change line 101 of Foo.cs to fix a bug, I need to know that the next build will be identical to the first except that it will contain my specific bug fix because that is the only change in the source tree.

Let’s suppose I upgraded the compiler in place. Now it’s possible that due to some changes in the compiler, the same source code that I compiled yesterday might produce different output today. That’s really bad. It means that I could introduce new bugs otherwise unchanged versions of my code just by recompiling it!

One solution is to indicate in the source code exactly which version of the compiler should be used. That way if I recompile yesterday’s code, then I will use exactly the same version of the compiler that was originally specified yesterday. The only way to introduce a change would be to commit a change in the source tree indicating that a new compiler version should be used instead.

The configuration of a hermetic build environment is well-known and consists of fully specified dependencies. The only way to change the output of the build is by changing something in the source.

Crucially, the hermetic build environment should not be permanently altered by the build process itself. For example, running tests shouldn’t leave extra temporary files floating around that remain for the next build to trip over. The environment should be pristine!

How do we implement something like this?

In Linux, we might run the whole build inside of a very carefully constructed chroot directory assuming that we don’t care about the kernel version or that we control it via other means.

In Windows, it’s a lot harder because important configuration information could in principle reside anywhere in the file system or system registry. For example, what really happens when we install a new version of .Net? Hmm… A more practical concern for Gallio is what happens when we upgrade to a new ReSharper beta, for example.

The case of the ReSharper dependency is interesting. To compile Gallio’s ReSharper extensions, it would be sufficient to have all of the ReSharper libraries checked into the source tree. However, to test those extensions we actually need a working installation of ReSharper, which itself requires Visual Studio, which itself requires many things. Checking in libraries isn’t enough. We’re going to need a virtual machine snapshot.

Gather-Scatter-Gather-…

A build process can be described as a sequence of gathering and scattering operations.

Here’s a fairly straightforward example:

Task 1

Gather: Check out all of the sources for the core components from the source tree for a specific reversion.
Scatter: Compile.
Gather: Copy the compiled binaries to a common directory.

Task 2

Gather: Grab the compiled binaries from the first task.
Scatter: Compile extensions with complex environmental dependencies.
Gather: Copy the compiled extensions to a common directory. Generate the documentation and the installers.

Task 3

Gather: Grab the installer from the second task.
Scatter: Install and test extensions with complex environmental dependencies.
Gather: Copy all test results into a test database.

Task 4

Gather: Grab the installer and documentation from the second task.
Scatter: Publish the build artifacts and documentation to the web.

In general, tasks gather stuff from the source tree or from the output of other tasks, scatter a bunch of derived artifacts around, then eventually gather them together again into a form that can be consumed by subsequent tasks.

We can represent the dependencies as a linear sequence of steps (as outlined above) except that we would be better off if we could take advantage of the inherent parallelism of certain stages. For example, there’s no reason we couldn’t be compiling and testing the ReSharper 4.5 extension at the same time as the ReSharper 5.0 extension as long as both processes ran independently (in different virtual machines). Instead we should model dependencies as a graph of parallel or sequential tasks.

To make this work, we will need a continuous integration tool that coordinates task dependencies elegantly.

Outlining a Solution

Here’s what the solution will look like for us.

Virtual Hermetic Build Environment

Because we need to build and test many components on Windows, the most robust and perhaps simplest way to establish a hermetic build environment will be to use a suite of virtual machines.

To that end, the virtual machine host will maintain a collection virtual machine snapshots. Each snapshot will have a unique name, like “Win7 x64, VS 2010, R# 5.0.1161.9 (Snapshot 1.0)”.

There are many virtual machines that we could use, but for our purposes we will use VirtualBox. VirtualBox is free and performs well although I have found it to be somewhat unstable at times. Even so, it’s perfect for what we need.

We’re going to have a lot of snapshots. Every time we tweak one of the virtual machines, we will create a new snapshot and then check in a change to the source tree to use the new snapshot for builds of subsequent revisions. Fortunately VirtualBox supports differencing disk images and snapshot trees. As a result, we only need to store the differences between snapshots. What’s more, we can easily reset the virtual machine to a previous snapshot state just by throwing away the latest differencing disk image which is very fast.

Over time, we may choose to throw away some snapshots when they are no longer relevant for practical purposes since there will come a time when we will never need to rebuild most old revisions. In the meantime, gigabytes are cheap and plentiful, time is not!

How many virtual machines will there be? I’m not sure… there may be as many as a dozen.

Windows Server 2003, continuous integration server, build agents and other infrastructure.
Windows 7 x86, .Net Framework 2.0 & 4.0, Mono, NCover.
Windows 7 x64, .Net Framework 2.0 & 4.0, Mono, NCover.
Windows 7 x86, ReSharper 4.5, Visual Studio 2008, NCover.
Windows 7 x86, ReSharper 5.0, Visual Studio 2010, NCover.
Windows 7 x86, TypeMock, NCover.
Windows 7 x86, AutoCAD 2010, NCover.
Windows 7 x86, TeamCity, NCover. (for testing TeamCity extensions)
Windows 7 x86, CCNet, NCover. (for testing CCNet extensions)
Ubuntu, Mono, MonoCov.
… more… ?

Continuous Integration

The build process will be broken down into a graph of tasks. Each task will specify dependencies on other tasks whose output it consumes. We’ll use a continuous integration build manager to run the tasks and report progress.

There are many different continuous integration tools that we could use, but for our purposes we will use TeamCity. TeamCity does a very good job of managing dependencies and the professional edition is available for free and does everything we need.

Interestingly, because we will be using virtual machine snapshots as part of our hermetic build environment, we will not use TeamCity’s built-in support for cross-platform builds.

JetBrain’s recommended approach for cross-platform builds is to run multiple TeamCity build agents on multiple machines with different configurations. However for that to work, the machine has to persist the state of the built agent. The problem is that we will always wipe our virtual machines back to their snapshot state after each build. So if the agents ran inside the virtual machine they would lose their state changes completely between builds which would probably make TeamCity somewhat unhappy.

Another problem with running separate build agents inside each virtual machine is that we would have to keep all of the virtual machines running all of the time. Otherwise, the TeamCity primary server would not be able to connect to the build agents. Running so many virtual machines (dozens?) would be a prohibitively expensive waste of resources. Moreover, we would need to keep a lot of build agents running and it costs money to buy licenses for more than 3.

Our solution will be to run TeamCity build agents only on the virtual machine host (or in a virtual machine with access to the host). Each build task will be responsible for starting up the appropriate virtual machine, invoking a remote command inside the virtual machine, gathering results from inside the virtual machine, and then shutting down the virtual machine (discarding changes to its state).

All we need is one TeamCity build agent! That one build agent can manage as many virtual machines as we like.

In practical terms, we’ll probably run 3 TeamCity build agents. That way we can run 3 tasks in parallel to improve performance while still fitting within the license constraints of the free TeamCity Professional Edition.

Odds and Ends

We’re also going to need a few other things.

Test reporting. (Gallio Archimedes)
Code coverage. (NCover + Gallio Archimedes)
Linting. (NDepend + FxCop + Gallio Archimedes)

Err, right… I guess we need to build Archimedes too…

Technorati Tags: gallio,mbunit,software engineering

Gallio Build Server Down for Maintenance

2010-02-11T12:13:00.001-08:00

The build server has been in an unhappy state for quite some time now (registry corruption due to frequent crashes of the VirtualBox VM). Now it finally seems to have gone over the edge.

As a result I’ll be rebuilding the build server sometime in the next week. I think I'll set the new one up a bit differently if I have time.

Long Term Ideas

1. Split up the build based on component dependencies.

Gallio ships with plugins for ReSharper 3.1, 4.0, 4.1, 4.5 and 5.0. The trick here is that in order to build and test those plugins, all 5 versions of ReSharper need to be installed. Ouch! (Ok, we will probably be dropping 3.1, 4.0 and 4.1 soon…)

Of course Gallio also ships with plugins for NCover, TypeMock, AutoCAD, TestDriven.Net, Visual Studio 2008, Visual Studio 2010, etc…

Imagine all of those components installed on the same VM? Yeah… it’s a little scary. Fortunately most of them do not conflict with the others so it mostly works out fine.

Ideally we should just build a bunch of VMs with different groups of dependencies and build them all independently. That way we can add new dependencies at will without worrying quite so much about how they interact (or worry about the 12 hours it will take to reinstall everything if the VM blows up).

2. Multiple Platforms

We should really be testing on at least the following platforms:

Windows 7 64bit
Windows 7 32bit
Mono on Windows
Mono on Linux

Basically that means we’ll need a whole bunch of VMs for various configurations. Tricky.

3. Add White Box Tests.

Right now there are a bunch of things we don’t test rigorously in an automated way. Here are a few of them:

The installer.
The Visual Studio templates.
The Visual Studio add-in, besides checking MSTest integration.
The ReSharper plugin, besides integration tests for the reflection API.
The UIs, besides unit tests for the Models and Presenters.

I spend a lot of time before each release manually testing this stuff. It just doesn’t scale.

All we really need are a few tests to get sufficient confidence that the system isn’t broken in really boneheaded ways. For example, Visual Studio shouldn’t pop up a dialog complaining about the Gallio add-in encountering an error during initialization. (Usually that means the installer is broken somehow.)

4. Faster Release Cycles

All it should take for a release is for us to spend a couple of minutes writing release notes, adding some more documentation to the wiki, and pushing a big red button to promote the current nightly build to stable.

Paradoxically, the first step in this process is that unstable builds need to be more visible.

What I mean is that we should maintain consistently up to date release notes and documentation for our pre-release builds too. We should get more people using those pre-release builds and perform more sanity-checking with automation so we’re not chasing last-minute bugs.

When it comes time to release, all we should have to do is post a new stable build label to the website and spam the newsgroups. :-)

Duh.

Technorati Tags: mbunit,gallio

Infrastructure and Documentation

2010-02-06T21:19:00.001-08:00

With my new job at Google, I’ve found it harder and harder to spend as much time as I would like on Gallio and MbUnit. It’s hard to find a good solid block of time to work on stuff without too many other distractions so I’ve been spending my time on little infrastructure projects.

The theory is that if I can build a big enough lever… then I can pretend there really are 32 hours in a day. :-)

New Stuff

1. Auto-Publish to Web

Most of the Gallio and MbUnit web site content including documentation resources are now published to the web automatically by our Continuous Integration build server.

In other words, any committer can change the website just by checking in some code, and waiting 3-5 minutes for the web site to be built and the changes to propagate.

This kind of setup is really cool. Seriously seriously cool. So cool I know Oren has blogged about it.

Given my time resources are dwindling (and will even more once I start working on Android stuff), automating the whole pipeline is golden. I can practically pretend that it’s real-time. Hit commit and move on to another task.

2. New Wiki!

Let me be blunt. The Gallio documentation is woefully incomplete.

The main problem is that I expected that the Gallio Book writing to progress more rapidly than it has so I didn’t spend much time developing alternative documentation resources.

Of course, there have been many offers to help write content for the book but little has materialized so far. That’s just how Open Source is sometimes. Lots of people want to help but there is always a barrier to contribution. In this case I’ve lowered the barrier a bit by automating the book pipeline from commit through to publication on the web but writing high quality documentation still takes time and preparation (especially in book form).

So…

Say hello to quick & dirty documentation by the masses for the masses.

The new Gallio Wiki is here!

It’s a little empty... Please help fill it up. :-)

Also, if you are interested, the old MbUnit v2 Wiki is still available ~~on the~~ ~~Internet Archive~~ on the Gallio website. We should probably try to port some of this content over, or something.

3. Old MbUnit v2 Documentation Is Back

A couple of years ago, Ben Hall invested quite a lot of time getting Sandcastle and DocProject to play nicely together and assembling the documentation.

Unfortunately when we moved the MbUnit.com website a couple of years ago, we lost some content. Specifically, we lost the documentation site.

Oh… we still had the code to generate the documentation site, of course, but the specific versions of Sandcastle and DocProject that we used were no longer available and the new versions were not compatible with the old code.

Great. Bit rot. For two years, I was sitting on a pile of documentation that I couldn’t compile.

Last night I decided to bite the bullet and try again. It was a total nightmare! The whole ordeal took me maybe 12 hours. I spent a good 8 hours trying to upgrade to a newer version of DocProject and getting nowhere fast.

Edit: I should point out that I don't blame my problems on DocProject or Sandcastle themselves. The real problem is that I didn't really understand how it all worked so I wasted a lot of time trying stupid things that failed.

Today I finally gave up and ported the essential parts of the documentation to the latest version of Sandcastle and the Sandcastle Help File Builder. It’s a little worse for wear but it’s still readable.

Anyways, now the old MbUnit v2 documentation is online again here: MbUnit v2 Documentation.

Maybe we can port some of that content to the wiki or book and update it for MbUnit v3…

Technorati Tags: mbunit,gallio

Edit: Posted new link to old MbUnit v2 wiki content.

Announcing Gallio v3.1 Update 2

2009-11-18T09:00:00.001-08:00

Today we are releasing Gallio v3.1 Update 2. This releases fixes several problems on x64 platforms and includes support for Visual Studio 2010 and .Net Framework 4.0 Beta 2.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

Earlier release notes: v3.1 update 1, v3.1, all versions

Changes

Improved startup performance by fixing a problem with pre-generated XmlSerializers.
Added support for Visual Studio 2010 and .Net Framework 4.0 Beta 2.
Fixed an AccessViolationException in Icarus. Special thanks to Kent Hansen for identifying the root cause and proposing a fix!
Fixed installer bugs on x64 which caused some components to not be installed.
Fixed bugs running MSTest tests on x64.
Fixed a problem that caused long-running tests to be aborted prematurely by Visual Studio.

Technorati Tags: gallio,mbunit

XmlSerializers, ModuleVersionId, ILMerge, and You

2009-11-12T01:24:00.001-08:00

I solved a minor mystery today.

Gallio internally uses the .Net XmlSerializer class to load plug-in metadata and save reports. You probably know all about XmlSerializer already, but maybe you don’t know just how badly using it can impact application start-up performance.

About XmlSerializer Code Generation

The heart of the issue is that XmlSerializer uses code generation to perform serialization and deserialization. Specifically, XmlSerializer generates some fresh C# code, compiles it out of process with csc.exe, then loads the resulting assembly. This work takes time. Seconds… feels like forever to a user. The generated assemblies are not cached across runs so there is a significant performance penalty on every launch.

There is a way to avoid this cost: pre-generate serializers assemblies and distribute them with the application.

To pre-generate a serializers assembly, you are supposed to use SGen.exe a bit like this:

SGen.exe /assembly:MyAssembly /type:MyRootXmlType

This command will emit MyAssembly.XmlSerializers.dll. All you need to do then is copy it next to MyAssembly.dll and you’re done. The cost of code generation is gone from every launch.

Two problems:

SGen only lets you specify one root type. If you use multiple Xml document types in your assembly then you will need to write a custom tool like this: Gallio.BuildTools.SGen.
Make absolutely sure that you keep MyAssembly.dll and MyAssembly.XmlSerializers.dll in sync. If you change MyAssembly.dll in any way then you must regenerate the serializers assembly. This is easy to set up once in your build scripts and forget about.

Troubleshooting

So let’s say you do all of this work and you run your program and you still see csc.exe starting up while your program runs.

To find out why, add the following to your application’s config file. (eg. MyApp.exe.config)

<configuration>

    <system.diagnostics>
        <switches>
            <add name=”XmlSerialization.PregenEventLog” value=”1” />
        </switches>
    </system.diagnostics>

</configuration>

Then run your program again and look at the Windows Event Log. There should be some information in there to help you out. (Another thing to try is fuslogvw.exe.)

Here’s the message that I saw:

Pre-generated serializer ‘Gallio.XmlSerializers’ has expired. You need to re-generate serializer for ‘Gallio.Runtime.Extensibility.Schema.Cache’.

This message is telling me that I violated rule #2 above: the serializers assembly must always be kept in sync with its parent assembly!

Why? I’ve got fancy build scripts…

ModuleVersionId

XmlSerializer verifies that the serializer assembly is in sync with its parent assembly by comparing the module version id of the parent assembly with an id that it previously cached when it generated the serializer assembly.

Let’s check whether the assemblies are in sync using Reflector.

Gallio.dll is the parent assembly, it has a module version id of “fbe34432-817a-46dd-832f-3e5bc679ecff”.

Gallio.XmlSerializers.dll is the serializers assembly, it has a special assembly-level attribute called XmlSerializerVersion that was emitted during code generation. It expects the parent assembly id to be “3c420916-3f3c-45cb-a79c-9a4bbc81014a".

Pedantic note: An assembly can have multiple modules each and each module has its own id. So the XmlSerializerVersion attribute actually contains a comma-delimited list of all module version ids of the parent asssembly sorted in increasing order.

Ok, so XmlSerializer thinks the pre-generated serializers assembly is out of sync because the module id is out of sync. That seems bananas because my build scripts always regenerate Gallio.XmlSerializers.dll whenever Gallio.dll is recompiled.

ILMerge

Well, my build scripts do one more thing: they run ILMerge on Gallio.dll to internalize certain dependencies that might otherwise get in the way of running unit tests. (Bad things would happen if your unit testing tool exposed a dependency on one version of Mono.Cecil.dll whereas your tests depended on a different version.)

When ILMerge internalizes dependencies, it has to regenerate the assembly. As a result, the new freshly merged Gallio.dll gets a brand new ModuleVersionId. Unfortunately we pre-generated Gallio.XmlSerializers.dll before running ILMerge so it is now out of sync.

All we need to do is regenerate Gallio.XmlSerializers.dll after the ILMerge. Problem solved.

Announcing Gallio and MbUnit v3.1 Update 1

2009-10-26T00:55:00.001-07:00

Today we are releasing Gallio and MbUnit v3.1 Update 1. This is mainly a bug fix release with a few new little goodies.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

Earlier release notes: v3.1, all versions

Reminder: The Visual Studio Test Tools integration requires Visual Studio 2008 SP1 or Visual Studio 2010 to work. If you still don't see tests in the Test View then ensure that your project has the necessary ProjectTypeGuids and that the Gallio add-in is enabled in the Visual Studio Add-In Manager.

Changes

Samples:

[New!] Added samples for web testing with WatiN. In particular demonstrates the creation of a custom [Browser] attribute to support running tests with multiple browsers.
[New!] Added samples for GUI testing with White.
Improved basic MbUnit samples a little to make them easier to understand.

General:

[New!] Added a Sources.txt file to the installation to help users find out where to check out the original source code for the release.
Fixed a bug with Hint Directories being ignored by the test runners.

MbUnit:

Upgraded MVC templates for ASP.Net MVC 2.
Fixed bug with transitive dependencies not being considered by [DependsOn] attribute when deciding whether to skip a test.
Improved the error message displayed when attempting to use a [StaticTestFactory] in ReSharper which is not supported due to technical limitations.
Fixed bug with use of [Factory] attribute on a fixture type or property when the type was not explicitly specified.
Improved error reporting and workarounds for cases where the assertions or test context are being used incorrectly to help a test author diagnose the problem.

Icarus:

[New!] Added support for using the Delete key to delete items from the Project Explorer.
[New!] Pending and Ignored tests are now unchecked by default.
Added filtering to the test report view to only include reports that match the current report name format.
Fixed some threading issues.
Fixed some test filter issues.
Fixed test counts.
Fixed window positioning issues.
Fixed display of test categories.

Reports:

[New!] Added support for viewing embedded text, html and video attachments in Icarus and Visual Studio.
Fixed rendering of embedded plain text attachments to preserve formatting.
Fixed UTF-8 character encoding issue with html reports.
Fixed attachment links and code navigation links in reports on 64-bit platforms and in Internet Explorer Protected Mode.

xUnit.Net Adapter:

Upgraded to v1.5 RTM.

AutoCAD Integration:

Improved detection of AutoCAD window.

ReSharper Integration:

Added support for certain reflection operations which were needed by NUnit to support derived test fixtures in ReSharper.
Fixed a bug that caused duplicate NUnit tests to appear in ReSharper.

NAnt Integration:

Changed how the severity of certain test results are reported so that NAnt does not incorrectly consider a test run to have encountered a non-fatal error when in fact a warning should have been logged.

TeamCity Integration:

Fixed a bug in the test event processing that indirectly caused TeamCity test result reporting to stall when an empty test assembly was encountered.

Technorati Tags: gallio,mbunit

Gallio Next Steps

2009-09-17T23:20:00.001-07:00

With Gallio v3.1 released, I have begun planning for what's ahead.

My first priority is to fix most of the issues that have been reported and to release a minor update to Gallio v3.1 in a week or two.

After that, the hard work begins.

Testing as a Service

As it stands, the Gallio platform mainly consists of one core library and a whole bunch of plug-ins. The majority of the code is written in C# and runs on .Net Framework 2.0.

I want to go further.

The main goal of Gallio v3.2 will be to adapt the Gallio platform to operate as a collection of services that communicate via a message bus. We will publish a specification of the messaging protocol so that other tools -- not necessarily written in .Net -- will be able to exchange messages with Gallio tools and participate in test execution, monitoring and reporting.

Why not pool our talents to develop great testing tools that work together across platform boundaries?

Gallio v3.2 and Beyond

Here's what you can expect from upcoming releases.

Open messaging protocol.
Test data warehouse.
Web-based test reporting tool and test case manager. (Archimedes)
Test rigs for remote test execution.
Data collection API for monitoring performance, gathering metrics and computing statistics.
Improved IDE integration.
Improved cross-platform support.
More openness about the development process in general.
Performance optimization, particularly of UI responsiveness and program start time.

How to Participate

I want to work with other members of the community to grow a practical lingua franca for test data interchange. While at first we will be discussing how to design a suitable messaging protocol for Gallio, I hope that it will eventually evolve into an independent open standard that anyone can implement.

Please join us on the gallio-dev mailing list and tell us what you think! :-)

The Gallio Plug-in Model

2009-09-14T23:21:00.001-07:00

One of the biggest changes in Gallio v3.1 was is a new plug-in model.

The new plug-in model features comprehensive support for late-binding of components, provides a rich declarative metadata model, and provides enough information to support the installation and verification of plug-in assemblies and resources.

Concepts

Late-Binding and Declarative Metadata

Gallio includes many plug-ins out of the box and more can be installed later but not all of the plug-ins need to be loaded at all times. To improve startup time, Gallio defers plug-in assembly loading until the last possible moment. Consequently Gallio's plug-in model relies heavily on late-binding and declarative metadata that is external to the plug-in assemblies.

We don't use .Net custom attributes to provide most of this metadata because then we would have to load assemblies in order to read it out. (Note: We could read metadata from custom attributes once and cache it like MEF does but it is not practical for some purposes such as plug-in installation.)

So the basic idea is to pack a bunch of useful information in XML.

Drop-In Plug-In Installation and Caching

Gallio strives to support XCopy-style deployment as much as possible. This feature makes it possible to check-out a copy of Gallio from your source tree and just run it with no strings attached (except for special features that require installation like Visual Studio integration).

When designing the Gallio plug-in model it was important to ensure that Gallio plug-ins could be installed just by dropping files in the right place. At run-time Gallio locates its plug-ins by searching plug-in directories recursively for .plugin files. There is one major problem with this process: recursively scanning directories and loading lots of little files can be very time-consuming. So the new plug-in model incorporates a cache.

Gallio maintains a per-user cache of installed plug-ins in any given list of plug-in directories. This way when it starts up, it only needs to load the cache file and move on.

However there is a small problem with this approach. By caching plug-in metadata, it is no longer sufficient to just drop in new plug-in files to install them. The cache must be cleared somehow.

Gallio implements three strategies for refreshing its cache:

When loading a cache file, it checks the timestamps on the original plug-in files to verify that they have not been changed. This way Gallio can automatically detect an edit to a .plugin file and refresh the cache. Unfortunately it cannot detect the presence of new .plugin files.
A user can explicitly clear the plug-in cache using the "Gallio.Utility.exe ClearUserPluginCache" command. Unfortunately if there are multiple users of the machine then all users will need to have their plug-in cache cleared to detect the presence of a new plug-in.
A admin user can explicitly reset the Gallio "installation id" of the machine by running "Gallio.Utility.exe ResetInstallationId". This command effectively invalidates the caches of all users running on the machine at once.

It's not ideal but it works well for now. Plug-in metadata caching has significantly improved Gallio start-up performance.

Plug-in Metadata

.plugin files

Each Gallio plug-ins has a ".plugin" file that describes the plug-in using XML. The description includes the id of the plug-in, some traits, dependencies, associated files, referenced assemblies, services and components.

Here is an excerpt of Gallio.plugin.

<?xml version="1.0" encoding="utf-8" ?>
<plugin pluginId="Gallio" recommendedInstallationPath="" xmlns="http://www.gallio.org/">
<traits>
    <name>Gallio</name>
    <version>0.0.0.0</version>
    <description>The heart of Gallio.</description>
    <icon>plugin://Gallio/Resources/Gallio.ico</icon>
</traits>

<dependencies>
    <dependency pluginId="BuiltIn" />
</dependencies>

<files>
    <file path="Gallio.plugin" />
    <file path="Gallio.dll" />

    

</files>

<assemblies>
    <assembly fullName="Gallio, Version=0.0.0.0, Culture=neutral, PublicKeyToken=eb9cfa67ee6ab36e" codeBase="Gallio.dll" qualifyPartialName="true" />
</assemblies>

<services>
    <service serviceId="Gallio.TestFramework"
             serviceType="Gallio.Model.ITestFramework, Gallio" />

    <service serviceId="Gallio.TestFrameworkManager"
             serviceType="Gallio.Model.ITestFrameworkManager, Gallio" />

    

</services>

<components>
    <component componentId="Gallio.FallbackTestFramework"
               serviceId="Gallio.TestFramework"
               componentType="Gallio.Model.FallbackTestFramework, Gallio">
      <traits>
        <name>Fallback</name>
        <icon>plugin://Gallio/Resources/Gallio.ico</icon>
      </traits>
    </component>

    <component componentId="Gallio.TestFrameworkManager"
               serviceId="Gallio.TestFrameworkManager"
               componentType="Gallio.Model.DefaultTestFrameworkManager, Gallio">
      <parameters>
        <fallbackTestFrameworkHandle>${Gallio.FallbackTestFramework}</fallbackTestFrameworkHandle>
      </parameters>
    </component>

    

</components>
</plugin>

Services and Components

Following the nomenclature used by several .Net Inversion of Control Containers, Gallio refers to its extension points as services and its extensions as components. Services nominally represent interfaces. Components provide implementations of those interfaces.

Internally Gallio refers to all service and component types by name using a TypeName object until it needs to instantiate it. This is different from most .Net IoCs which immediately resolve most type names to runtime Type objects during initialization which causes most assemblies to be loaded up front.

Dependency Injection and Configuration

Gallio's plug-in model supports constructor and property dependency injection and configuration for components.

In the .plugin XML, the component parameters appear in a <parameters> element which contains an element for each constructor parameter or property that is to be bound. If no value is specified for a given constructor parameter, then Gallio will attempt to resolve a service of the appropriate type to bind. Otherwise if a value is specified, then Gallio will attempt to convert the provided string to an object of the required type using a DefaultObjectDependencyResolver.

The default object dependency resolver knows how to resolve all sorts of object types including strings, arrays, components, ComponentHandles, IComponentDescriptors, Versions, Guids, Icons, Images, Conditions, FileInfos, DirectoryInfos, Assemblies, AssemblySignatures, AssemblyNames, Types, TypeNames, and explicit references to other components (using the ${component.id} syntax seen in the excerpt above.)

Gallio resolves Icons, FileInfos, and other disk-based plug-in resources using a special Uri scheme called "plugin". For example, a component might require a reference to a file that belongs to the plug-in so it declares a constructor parameter of type FileInfo. In the XML, it provides the path of the file relative to the plug-in base directory by specifing a parameter with a Uri like: "plugin://MyPlugin/Path/To/File.txt". When the component is instantiated, Gallio will resolve the Uri and provide a FileInfo with the appropriate full path to the file.

Plug-in and Component Traits

Traits are collections of typed properties that Gallio uses to describe plug-ins and components.

At runtime, plug-in traits are represented by instances of the PluginTraits class which looks a bit like this:

public class PluginTraits : Traits
{
    public PluginTraits(string name) { ... }
    public string Name { get; }
    public string Version { get; set; }
    public Icon Icon { get; set; }
    // SNIP...
}

Likewise, component traits are represented by instances of the service's associated Traits type. To establish that association, a service type may have an optional TraitsAttribute applied.

[Traits(typeof(MyServiceTraits))]
public interface MyService
{
    void DoSomething();
}

public class MyServiceTraits : Traits
{
    public string SomeInformation { get; set; }
    public string[] MoreInformation { get; set; }
    public Icon MyIcon { get; set; }
}

Traits parameters are bound just like component parameters except that they appear in the <traits> element of the ,plugin XML.

Plug-in Dependencies

In the .plugin XML, the <dependencies> element specifies the ids of other plug-ins that are required by this plug-in.

The dependency information allows Gallio to do a few useful things:

If a plug-in dependency is missing or disabled then all dependent plug-ins are automatically disabled.
Before activating a plug-in, all of its dependencies are activated. (Note: Not implemented in v3.1 but may appear in a later version.)
When verifying the plug-in installation, Gallio can check that all of the services required by a plug-in are provided by that plug-in or by one of its dependencies. This check helps developers find errors in component registrations.

Assembly References and Probing Paths

Each plug-in describes the list of assemblies that it required in the <assemblies> element of its plug-in XML. Before attempting to use any components defined by a plug-in, Gallio first ensures that the plug-in's assemblies can be resolved. Assembly references can also be qualified with binding redirects and other loader options.

A plug-in can also declare custom probing paths in a <probingPaths> element.

Here are a few possibilities:

<probingPaths>
    <probingPath>customBins</probingPaths>
</probingPaths>

<assemblies>
    <assembly fullName="MyAssembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=eb9cfa67ee6ab36e" codeBase="MyAssembly.dll" qualifyPartialName="true" applyPublisherPolicy="true">
        <bindingRedirects>
            <bindingRedirect oldVersions="0.0.0.0-1.1.0.0" newVersion="2.0.0.0" />
        </bindingRedirects>
    </assembly>

    <assembly fullName="MyAssemblyInGAC, Version=1.0.0.0, Culture=neutral, PublicKeyToken=eb9cfa67ee6ab36e" />
</assemblies>

Plug-in Files and Recommended Installation Path

In order to facilitate plug-in installation, each plug-in includes two pieces of information:

A list of files in a <files> element.
A recommended installation path (relative to the Gallio bin directory) in the recommendedInstallationPath attribute of the <plugin> element.

Gallio uses this information in two ways:

During plug-in installation it is able to determine which files belong to which plug-ins so that it can copy or delete just the necessary files. (Note: Automatic plug-in installation is not implemented in Gallio v3.1 and will appear in a later release.)
On request, it checks the information to verify that all plug-ins are installed as expected and that no files are missing. This is part of what happens when you run the "Gallio.Utility.exe VerifyInstallation" command.

Conditional Plugin Enablement

Plug-ins can include a conditional expression that specifies when the plug-in should be enabled based on characteristics of the environment. For example, some plug-ins related to Visual Studio integration are only enabled when Gallio is hosted by the appropriate version of Visual Studio.

The conditional expression appears in the enableCondition attribute of the <plugin> element of the .plugin XML files.

Conditions are represented in the code by Condition objects. Each Condition consists of a simple boolean expression of terms drawn from a ConditionContext and combined using logical "or". For plug-in enablement the runtime uses the RuntimeConditionContext object to query properties in the environment.

The following condition properties are currently supported.

"${env:ENVIRONMENTVARIABLE}": Satisfied when the environment contains a variable called "ENVIRONMENTVARIABLE".
"${minFramework:NET20}", "${minFramework:NET30}", "${minFramework:NET35}", "${minFramework:NET40}": Satisfied when the currently running .Net runtime version is at least the specified version.
"${process:PROC.EXE}", "${process:PROC.EXE_V1}", "${process:PROC.EXE_V1.2}", "${process:PROC.EXE_V1.2.3}", "${process:PROC.EXE_V1.2.3.4}": Satisfied when the currently running process main module is "PROC.EXE" and exactly matches the specified file version components (if any).

For example, here's part of the .plugin XML file for the Gallio35 plug-in which provides .Net 3.5 specific extensions.

<plugin pluginId="Gallio35"
        recommendedInstallationPath=""
        enableCondition="${minFramework:NET35}"
        xmlns="http://www.gallio.org/">

Runtime API

Gallio clients mainly interact with the plug-in model by way of the following types:

RuntimeAccessor: Accessor for the core runtime objects.
IRuntime: Configures and provides access to the plug-in registry, service locator, runtime logger, and other runtime parameters.
IRegistry: A registry of all installed plug-ins, services and components.
IServiceLocator: A service locator for resolving services and components provided by plug-ins.

The initialization process looks a bit like this:

Client code calls RuntimeBootstrap.Initialize() to initialize the runtime.
The runtime creates a CachingPluginLoader.
The caching plugin loader loads plug-in metadata into a PluginCatalog. The metadata is represented in the catalog as PluginRegistration, ServiceRegistration, and ComponentRegistration objects.
The runtime augments the catalog with a few built-in services of its own.
The runtime creates a new Registry and populates it from the catalog. Gallio performs many checks during this stage to ensure that plug-ins are well-formed and that all of their assemblies are accessible. It automatically disables plug-ins that have problems.

Once the registry has been initialized, client code can use it in different ways.

The registry provides methods for getting IPluginDescriptor, IServiceDescriptor, and IComponentDescriptor objects. Descriptors are used to get information about plug-ins, services and components as well as to obtain instances of components and traits objects.
The service locator provides methods for resolving services. These are the same kinds of familiar operations exposed by most .Net IoCs such as Resolve, and ResolveByComponentId along with a couple novel ones like ResolveHandle and ResolveHandleByComponentId which obtain late-bound ComponentHandles. Gallio does not use the service locator explicitly very often. Most service location is performed implicitly by the runtime as part of dependency injection.

The ComponentHandle type is interesting in that it allows clients to obtain a typed reference to a component descriptor without resolving the component instance itself until needed. Here is an example usage of a component handle to enumerate installed components and choose to instantiate one or more of them based on some criterion.

var frameworks = new List<ITestFramework>()
ComponentHandle<ITestFramework, TestFrameworkTraits>[] frameworkHandles = // ...

foreach (var frameworkHandle in frameworkHandles)
{
    TestFrameworkTraits traits = frameworkHandle.GetTraits();
    if (IsSupportedFileType(traits.FileTypes, testFile))
    {
        frameworks.Add(frameworkHandle.GetComponent());
    }
}

Brief Comparison with Other Plug-In Models

Gallio's plug-in model is similar to the Eclipse plug-in model, Mono.Addins and MEF.

Eclipse and Mono.Addins both use XML to register plug-ins and to declare extensions points (services) and extensions (components). Extensions can provide ample metadata in the form of XML extension configuration data. Both of these frameworks also have provisions for packaging plug-ins as archives that can be installed incrementally. (Of course, Mono.Addins was inspired by Eclipse so this is no surprise.)

MEF supports late-binding Exports (component handles) and makes available ExportMetadata (traits) for clients to use. MEF is mainly attribute-based but can be configured to use external metadata files by providing an appropriate Catalog implementation. It is quite powerful and will be included as a standard system component in the .Net Framework 4.0 release.

Aside: The original Gallio plug-in model used the Castle Windsor inversion of control container as its foundation. Unfortunately Windsor does not support late-binding (it resolves all type names to Types on initialization) and it has limited support for component metadata. It is a great tool (and I highly recommend it for many projects) but just not well suited to Gallio's needs.

Why Not Use Mono.Addins or MEF for Gallio?

This was a tough call.

I took a good long look at Mono.Addins and MEF beforehand. Both are good options but I had a couple of problems with each of them.

I rejected Mono.Addins because I was not confident that I would be able to morph it into the kind of shape I needed for Gallio. I found the code to be tightly coupled in several places that I knew I would have to change dramatically to get the kind of dependency injection and late-binding features that I wanted.

And MEF unfortunately required .Net 3.5 as a minimum whereas I had selected a target framework of .Net 2.0 for Gallio. I briefly considered porting MEF to .Net 2.0 but gave up once I saw how much of System.Core it used. Also it was clear that the attribute model would not work well for Gallio's late-binding needs so I would not benefit from those features. Oh well. I look forward to trying MEF out another day with a .Net 4.0 project perhaps.

So it just seemed more straightforward to start over which would give me a maximum of flexibility in the implementation. As it happens, I have taken advantage of that flexibility many times over to make things better and I can say I am very happy I did not try to shoehorn Gallio into Mono.Addins or MEF. For example, some of the plug-in metadata Gallio uses would be very awkward to implement and consume with either of these libraries.

YMMV.

Future Directions

Here are some things to expect from future versions of the Gallio plug-in model:

A plug-in manager tool for downloading, installing, and maintaining plug-ins.
A plug-in activation process to enable plug-ins to have more control over their early initialization.
Extensible object dependency resolvers.
More optimizations for start-up performance.
Component scopes so that test frameworks can install special extensions while they run. eg. custom object formatters for MbUnit.

Technorati Tags: gallio

Announcing Gallio and MbUnit v3.1!

2009-09-11T11:03:00.001-07:00

Gallio v3.1 is a major upgrade to the platform. This release includes many new features, better performance, and improved robustness.

Highlights:

Video capture and embedding in test reports.
RSpec!
Support for .Net Framework 4.0.
Support for Visual Studio 2010 Beta 1.
Control Panel application.
Brand new plug-in model with improved startup performance.
Major internal redesign.
More MbUnit goodness.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

Earlier release notes: v3.0.6, v3.0.6 update 1, v3.0.6 update 2, all versions

Installation Notes

We recommend completely uninstalling any previous installation of Gallio before installing the new one. Be sure to shut down Visual Studio while uninstalling because some files may still be in use.

We strongly recommend installing KB963676 before installing Gallio. This patch fixes some Visual Studio crashes that may occur while editing ASP.Net pages.

New and Noteworthy

Video Capture and Embedding

Gallio now includes support for capturing video screen recordings and embedding them in test reports.

Why?

Imagine you were writing integration tests for a Web or GUI application through the UI using a library like WatiN or White. When the test fails, it is very handy to have a record of just what happened.

Magic

In the next couple of sections I'm going to show how you can capture and embed screen recordings like the following into Gallio test reports.

Screen Capture and Embedding

This is all it takes to capture and embed a screenshot in the test report.

Or if we just want to capture a screenshot when a test fails or is inconclusive, here's how we can do it. We can also scale the image by 25% to keep the report small while still preserving enough detail to read.

As you can see the Capture class has some interesting methods.

Creating and Embedding Videos

A video is just an encoded sequence of frames. So we can build up a screen recording by taking screenshots at intervals and adding them to an instance of the Video class such as a FlashScreenVideo.

Notice that you can create videos of anything you like. You are not limited to screen recordings.

Screen Recording and Embedding

Gallio offers a few shortcuts to simplify screen recording in particular.

The ScreenRecorder class captures screenshots in a background thread and assembles them into a Video that can later be embedded.

The Capture class also provides a method for automatically embedding a screen recording based on the outcome of the test. It starts a screen recorder in the background and when the test finishes decides whether to embed the video or discard it.

This is a very powerful one-liner...

Here we set things up to capture a screen recording at 5 frames per second and 25% zoom factor and cause it to be automatically embedded if the test fails or is inconclusive.

Overlays

Raw screen recordings are very useful but we can enrich them with additional information. For example, while playing back the recording, it can be useful to see a transcript of what the system was doing when it captured the frame.

Gallio supports applying any number of Overlays to any screen recording. An overlay is a basically an object that paints itself onto an image.

We're going to use a CaptionOverlay to add a caption to the video as we record it. We can change the caption between frames to display a variety of information.

That's a lot of work so the Capture class provides a few convenience methods for setting a caption overlay for its screen captures and screen recordings.

Magic Recipe for WatiN

Here's the whole recipe for hooking up automatic screen recordings with WatiN 2.0.

public abstract class WatiNFixture
{
    protected Browser Browser { get; private set; }

    [SetUp]
    public void SetUp()
    {
        Logger.LogWriter = new GallioLogger();

        Capture.SetCaptionAlignment(HorizontalAlignment.Center, VerticalAlignment.Bottom);
        Capture.SetCaptionFontSize(32);
        Capture.AutoEmbedRecording(TriggerEvent.TestFailedOrInconclusive,
            "Screen Recording",
            new CaptureParameters() { Zoom = 0.25 }, 5 /*frames per second*/);

        IE ie = new IE();
        ie.ShowWindow(NativeMethods.WindowShowStyle.ShowMaximized);
        Browser = ie;
    }

    [TearDown]
    public void TearDown()
    {
        Capture.SetCaption("");

        if (Browser != null)
        {
            Browser.Close();
            Browser = null;
        }
    }

    private sealed class GallioLogger : ILogWriter
    {
        public void LogAction(string message)
        {
            Capture.SetCaption(message);
            TestLog.WriteLine(message);
        }

        public void LogInfo(string message)
        {
            TestLog.WriteLine(message);
        }

        public void LogDebug(string message)
        {
            // Ignore these messages.
        }
    }
}

public class SampleTest : WatiNFixture
{
    [Test]
    public void GoogleSearchAndVisitMaps()
    {
        Browser.GoTo("http://www.google.com");

        Browser.TextField(Find.ByName("q")).TypeText("Ottawa");
        Browser.Button(Find.ByName("btnG")).Click();

        Browser.Link(Find.ByText("Ottawa, ON Canada")).Click();

        Browser.Link(Find.ByText("Parliament Hill")).Click();

        Browser.Link(Find.ByText("Full article")).Click();

        Assert.Fail("Just for show.");
    }
}

RSpec and IronRuby

Now for something completely different: RSpec integration.

Big News

The big news here is that Gallio now supports running tests that are defined in files instead of just .Net test assemblies. This means you can pretty easily create custom test framework adapters for any kind of file that you know how to process and then run those tests using Icarus and other tools.

Yes, that means you can implement 5 methods of ITestDriver (or one of the helper subclasses Gallio provides) and have a custom test framework of your own using whatever kinds test files you like be they spreadsheets, XML files, native executables, or programming language text.

RSpec in Action

Let's suppose we have implemented a "Bowling" class that computes the score of a bowling game. (This example is borrowed from the RSpec site.)

class Bowling
    def hit(pins)
    end

    def score
        0
end
end

Obviously this implementation will not get us very far. So we write some specs to guide us along.

require 'bowling'

describe Bowling do
it "should score 0 for gutter game" do
    bowling = Bowling.new

    20.times { bowling.hit(0) }

    bowling.score.should == 0
end

it "should score 300 for perfect game" do
    bowling = Bowling.new

    12.times { bowling.hit(10) }

    bowling.score.should == 300
end

it "should score 20 for single pin hit each ball" do
    pending("Scoring to be implemented.")
end
end

RSpec + Icarus

Now we just add the bowling_spec.rb file to Icarus, and run...

RSpec + Icarus + AutoRun = Bliss

When programming in a BDD way you will probably find it useful to enable the Icarus auto-run feature. Then whenever you edit the spec, they will be reloaded and re-executed.

It's a little slow right now but expect it to get better in future versions.

MbUnit Features

Yann Trévin has been busy cooking up new features for MbUnit v3.1.

Retry.Until

The Retry class makes it easy to implement polling loops in tests. Polling loops are useful when the test must wait for an asynchronous operation to complete but does not know how long it might take.

More information about Retry.Until.

Assert.Sorted and Assert.Distinct

Two new assertions for checking whether the elements in a collection are sorted or are distinct.

More information about Assert.Sorted.

More information about Assert.Distinct.

Assert.ForAll and Assert.Exists

Two new assertions for checking whether a property holds true for all elements of a collection or for at least one of them. These assertions are incredibly useful.

More information about Assert.ForAll and Assert.Exists.

Assert.Throws for Inner Exceptions

Assert.Throws now accepts an optional second type parameter to specify an inner exception type.

More information about Assert.Throws in general.

StructuralEqualityComparer

Did you ever have a data structure that you wanted to check for equality but it did not override Object.Equals? Ever grumble about having to implement this check manually?

MbUnit v3.1 includes a new feature called a structural equality comparer. The idea is to make it super easy to compare all members of a structure according to some rule. Basically you just provide a list of lambda expressions to select structure members to compare and then you optionally provide a comparison predicate. MbUnit will then compare the structure member by member using the provided selectors and predicates.

More information about StructuralEqualityComparer.

EnumData, SequentialNumbers, RandomNumbers and RandomStrings

Yann wrote a bunch more nifty data source attributes. Here they are in one big combinatorial test.

[MultipleCulture]

The [MultipleCulture] attribute runs a test repeatedly with different cultures.

[Impersonate]

The [Impersonate] attribute runs a test using different user credentials.

TestContext.AutoExecute

TestContext.AutoExecute makes it easy to register actions to perform when a test passes or fails without needing to write a [TearDown] method.

This example captures and embeds a screenshot in the test report whenever the test fails.

Control Panel

Gallio now has a control panel application where you can set all sorts of preferences.

Feel free to suggest other things for us to add to the control panel.

Gallio Runtime Settings

The Gallio runtime settings let you tweak some global parameters. Currently it lets you set custom plug-in paths in case you have installed other Gallio plug-ins elsewhere in your system.

Expect this list to grow...

Icarus Settings

All of the Icarus settings have moved to the Control Panel so you can configure them in the same place as other settings.

TestDriven.Net Settings

Now we have a place to configure settings for TestDriven.Net.

The Frameworks page lets you configure whether Gallio will be the default or preferred test runner for different frameworks. For example if you have csUnit installed and you prefer to use Gallio to run csUnit tests with TestDriven.Net then you can set Gallio's priority to "Preferred". If you set Gallio's priority to "Default" then Gallio will only be used if there is no other installed test runner available for a particular framework.

AutoCAD Settings

Mike Sandberg contributed a new preference panel for configuring the AutoCAD integration.

The AutoCAD integration still supports the "AcadAttachToExisting" and "AcadExePath" runner properties but having a GUI is certainly much nicer...

Plugin Diagnostics

Just for completeness, Gallio also displays a list of all installed plug-ins.

(Later on we will be adding a proper plug-in manager with the ability to download and install plug-ins separately.)

Visual Studio Test Tools Enhancements

Visual Studio 2010 Beta 1

Gallio v3.1 now includes support for Visual Studio 2010 Beta 1 out of the box.

Data-Driven Test Results (Fixed)

Gallio now correctly displays the results of data-driven tests like MbUnit row tests.

In the following screenshot you can clearly see both test results of a data-driven test. Previous versions of Gallio would only show one result.

AutoCAD Integration

In addition to the new preference panel, we have made a couple of minor enhancements to the AutoCAD testing experience.

Debugging AutoCAD Tests

With Gallio v3.1, you can now debug your AutoCAD tests using Icarus just like any other tests.

Previously debugging did not work because the Visual Studio debugger is unable to attach to a process that uses fibers (like AutoCAD does). So Gallio v3.1 launches AutoCAD with the debugger attached from the start and it works.

We also support zero-installation execution of AutoCAD tests. This means you can run tests on a machine with AutoCAD installed without having to separately install Gallio as long as the files have been copied to the local disk somewhere.

TeamCity Integration

Gallio v3.1 now automatically detects when it is being run by TeamCity and enables the TeamCity test runner extension. This means you do not need the following in your build scripts any more: "TeamCityExtension,Gallio.TeamCityIntegration".

There have also been some fixes to more correctly output test results from parallelizable tests.

Performance

We have invested some time improving the performance of Gallio and MbUnit v3.1 in several key areas.

Startup Time

Gallio v3.1 is much faster to start up than previous versions of Gallio.

Much of this gain comes from three important changes. First, the new plug-in model in Gallio exposes more declarative metadata up front so that we can defer loading assemblies until they are actually needed. Second, once we have identified all of the plug-ins that are installed, we cache the composite metadata for subsequent runs. Third, we generate XmlSerializers ahead of time.

On my slow laptop Icarus startup time dropped from over 20 seconds to about 2 seconds. These startup time improvements also carry over to the Visual Studio and ReSharper integration.

Memory Usage

Gallio v3.1 uses significantly less memory when running multiple test assemblies than it did in v3.0.6.

Older versions of Gallio used to keep all of the test assemblies loaded in memory for the duration of the test run. When running dozens of test assemblies together at once (say, during a continuous integration build) Gallio would end up consuming quite a lot of memory in proportion to the number of test assemblies.

Now Gallio v3.1 loads and unloads each test assembly before moving on to the next one. This change improves responsiveness and keeps memory usage in check.

Debugging and Profiling

One of the more perplexing performance problems we addressed in MbUnit v3.1 was the performance of the debugger.

In older versions of MbUnit v3 it would sometimes take half a second or more to single-step while running a test in the debugger. The problem is that the stack was just too deep. During a typical test run, the stack could be 160 frames deep!

Now MbUnit v3.1 is more carefully tuned to minimize its stack depth. We have reduced the effective stack depth by a factor of 5 so single-stepping is zippy and responsive just like it should be.

One happy side-benefit of reduced stack depth is that it significantly easier to figure out what it going on when running tests with a code profiler.

.Net 4.0 Support and x86/x64 Processor Architecture Enhancements

Gallio v3.1 includes out of the box support for .Net 4.0. We also improved our support for running tests in different processor architectures.

One key change is that Gallio runs each test assembly in its own process by default. Gallio automatically detects the required .Net runtime version and processor architecture by examining the assembly metadata. You can run tests designed for different .Net runtimes or processors architectures side-by-side if you like. All you need to do is compile your test assemblies with the right options.

It just works.

Robustness

We have worked hard to improve the robustness of Gallio v3.1 while adding new features.

UAC Privilege Elevation on Vista and Windows 7

Did you notice that the Control Panel application displayed the UAC shield icon? Here it is again:

The shield indicates that Gallio has detected that the Control Panel is running in a lower privilege environment and that certain settings that have changed require elevation. Of course when you click on the button you will receive a UAC prompt after which Gallio will carry out the required operation in an elevated context.

ReSharper Integration Bug Fixes

We have spent some time improving the robustness of the ReSharper integration. It used to throw up annoying exceptions here and there and occasionally would crash. Hopefully this is all a thing of the past.

Here's some bad code that used to cause Visual Studio to crash but not anymore:

public class A : B
{
}

public class B : A
{
}

Gallio.Utility Power Tool

Gallio.Utility.exe is a new command-line application included with Gallio to perform various functions related to maintaining a Gallio installation.

These are the supported utility commands at this time:

ClearCurrentUserPluginCache: Clears the plugin cache of the current user.
ResetInstallationId: Resets the installation id. The plugin list will be refreshed the next time a Gallio application is started.
Setup: Installs or uninstalls components.
VerifyInstallation: Checks for runtime installation errors.

More Goodies

MbUnit:

MbUnit assertions offer EqualityComparison<T> and IEqualityComparer<T> overloads.
Parallelizable tests run with the correct number of threads and start up immediately without requiring any prior "warm-up" period as was seen in earlier versions due to some bad logic.
Support the use of Assert.Inconclusive and Assert.Terminate within Assert.Multiple blocks.
Contract verifiers can be "static" in which case they behave like static test factories instead of like dynamic test factories. In a "static" contract verifier, each individual verification will appear as a separate test in the test runner.

csUnit:

Upgraded to csUnit v2.6.

NUnit:

Upgraded to NUnit v2.5.2 but retained support for NUnit v2.4.8 also.

MSTest:

Rewrote most of the MSTest integration to be faster and more robust.

Command-line tools:

Gallio.Echo.exe supports wildcards.
The NAnt, MSBuild and PowerShell tasks support specifying the Verbosity.
Echo support a /no-progress switch to disable progress reporting.

Icarus:

Icarus now has a "Recent Projects" menu.
Icarus test tree is a bit faster.
Lots of Icarus bug fixes.
New error dialog.
Support keyboard navigation.
Disabled the watchdog timer during debugging runs so you can debug at leisure without worrying about the test runner killing the process prematurely.

Test Reports:

Support embedded HTML content and Flash videos.
Workaround issues when clicking links to attachments in IE due to the Local Machine Zone Lockdown.
Display test kind icons in the report.
Added a condensed text report type.
Test output is normalized before writing it to the log. This is to catch problems like attempting to write a null character or BOM into the report.
Test framework nodes are no longer emitted.

Debugging and Code Coverage:

Gallio launches the debugger with the "Managed and Native" debugging engine selected so that you can debug into native code from managed code.
NCover v1.5.8 works on x64 but only supports running tests in x86 mode.
Using NCoverExplorer or NCover.Reporting to merge NCover coverage reports when running test assemblies in separate processes.

ReSharper:

Lots of bugfixes.

TestDriven.Net:

Gallio now supports zero-installation TestDriven.Net test runs with recent beta builds of TestDriven.Net. I expect Jamie will say more about this feature soon...

Pex:

MbUnit.Pex no longer included. The Pex project has taken over the maintenance of this extension.

CruiseControl.Net:

Added support for CruiseControl.Net v1.4.4 and more recent versions.

Documentation:

Yann performed a massive documentation sweep to clean up formatting.
Switched to Sandcastle instead of NDoc 2 Alpha.

Core:

Brand new test framework extension API. Only 5 methods to implement.
Early Transition to a message-based protocol for test framework integration.
Massive refactoring of namespaces and interfaces.

Technorati Tags: gallio,mbunit

Toward New Horizons

2009-07-30T01:09:00.001-07:00

The short story is that I am now between jobs.

The long story is that I have left Ingenio / Yellowpages.com / AT&T Interactive and will be joining Google in late September. Yes that's the same Google I turned down last year; my priorities and perspective have changed.

I am looking forward to living in Mountain View biking distance from work and walking distance to yuppie downtown amenities. I also look forward to joining any number of Google's social and volunteer organizations. I have pretty much decided that, as soon as I figure out how, I will find a way to teach the fine art (and joy) of programming and computer science to children, adult students and peers.

Now I have two months break! Feels a bit like summer vacation. Nothing planned but possibilities and opportunities abound everywhere. I'll probably sing, play music, learn pastels and paint, work on Open Source, take a dip in a warm ocean someplace, and relax.

My last day at my old job was Friday the 24th so I am still getting over the early morning nagging sensation that I have to be someplace. I'm sure that my coming week at Lark Camp in Mendocino will fix that. :-)

Coming Soon: Gallio and MbUnit v3.1

2009-07-19T00:08:00.001-07:00

The next release of Gallio and MbUnit is just around the corner. I need to work on a proper write-up of all of the new stuff but here are a few highlights:

Significantly faster start-up time.
Embedded videos in test reports including built-in screen recording. (Try playing with the Capture class.)
Gallio can run RSpec tests via IronRuby.
New Control Panel application for tweaking global settings.
More MbUnit assertions and attributes for manipulating data and the environment.
More Icarus GUI goodness.
Better API documentation.
Much simpler story for test framework adapters and test runners.
Gallio can run tests in arbitrary files including tests written in non .Net languages given a suitable framework adapter (which you could write in a day).

The final release of v3.1 will probably happen within the next two weeks.

We're looking for beta testers.

To help us out, try out one of the recent v3.1 builds from: http://ccnet.gallio.org/Distributables/

Please report any suggestions or issues you find here: http://code.google.com/p/mb-unit/issues/list

You can also join our developer mailing-list: http://groups.google.com/group/gallio-dev

API documentation: http://www.gallio.org/api/index.aspx

Thanks!

Technorati Tags: gallio,mbunit

Upcoming Events

2009-06-05T01:46:00.001-07:00

I will be attending ALT.Net Canada in Vancouver, BC from June 12th through 14th. My plan is to follow that up with a week-long vacation and visiting friends and family in Ottawa, Toronto, Montreal or Kitchener-Waterloo.

I rather desperately need a vacation...

As for Gallio, I am currently investing most of my spare time into adding support for test files that are not .Net assemblies. When I am finished, Gallio will be able to run tests written in other languages such as C++, Ruby, various XML-based DSLs or anything else that is represented as a file of a type for which a test framework plugin has been installed. As a proof of concept Gallio v3.0.7 will include support for RSpec by way of the Dynamic Language Runtime.

Announcing Gallio and MbUnit v3.0.6 Update 2.

2009-05-29T16:21:00.000-07:00

Today we are releasing Gallio and MbUnit v3.0.6 Update 2. This is mainly a bug fix release.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

Earlier release notes: v3.0.6, v3.0.6 update 1, all versions

Changes

MbUnit:

[New!] Added [Parallelizable(TestScope.All)] and [Parallelizable(TestScope.Descendants)] to recursively mark tests as parallelizable such as all tests within a fixture or all fixtures in an assembly.
Fixed "test was orphaned" errors involving Parallelizable tests.

xUnit.net:

Upgraded adpater to xUnit.net v1.1 RTM.

NCover:

Added support for running NCover v1.5.8 as a 32-bit process on 64-bit machines.

Echo:

Fixed a regression that was causing test reports not to be generated when canceled.

ReSharper:

Fixed ClassCastExceptions in the ReSharper runner.
Fixed missing hotfix lightbulbs in the ReSharper within test files.
Fixed red strikethrough that would appear in the Unit Test Session window when running MbUnit tests declared in subclassed test fixtures.

AutoCAD:

Fixed a regression in the AutoCAD test driver.

TeamCity:

Improved the handling of parallel tests so that TeamCity test statistics make more sense.

Miscellaneous:

Improved the installer to support in-place upgrades.
Fixed a regression that occasionally would prevent dynamic tests from running.

Technorati Tags: gallio,mbunit

Registry Weirdness, UAC and the Vista Virtual Store

2009-05-28T17:45:00.001-07:00

It appears that two different can see a different view of the registry content. I seem to recall seeing mention about registry values potentially being seen differently by elevated and non-elevated applications when UAC is active.

In this case, ProcessInvocation.exe sees the following value for a particular key: (which is wrong!)

C:\Source\MbUnit\v3\src\Extensions\TDNet\Gallio.TDNetRunner\bin\Gallio.TDNetRunner.dll

But regedit.exe sees the following value: (which is correct!)

C:\Program Files\Gallio\bin\TDNet\Gallio.TDNetRunner.dll

The first value was used for local development of Gallio v3.0.6 but I deleted that value! In fact I deleted it a week ago and I don't think I've recreated it since (because Gallio v3.0.7 uses different keys).

The second value was set by the Gallio v3.0.6 installer (uses the old key). This is the value I expect to see globally. All interesting applications were closed at the the time of installation.

So why are some applications seeing the first value and other applications seeing the second value?

No, this is not a simple question of refreshing or shutting down applications. I have shut them both down and tried again and the weird view remains persistent!

Next up, reboot. Argh.

After the Reboot...

Same behavior.

Why?

Here's what I found when I searched the registry for the incorrect value: (SID masked)

HKEY_USERS\S-1-5-21-xxxxxxxxxx-xxxxxxxxx-xxxxxxxxx-xxxx\Software\Classes\VirtualStore\MACHINE\SOFTWARE\MutantDesign\TestDriven.NET\TestRunners\Gallio_MbUnit

Whoa. So at least I'm not dreaming.

The UAC Virtual Store

On Vista, normal users are forbidden from writing to certain folders (like %ProgramFiles% and %SystemRoot%) and into the HKEY_LOCAL_MACHINE part of the registry. Even local administrator users are forbidden from writing into these folders, unless elevated.

Well, not quite. As a compatibility feature, instead of generating an error, a write by a non-elevated application running as a local administrator may be redirected to a Virtual Store on the file system or in the registry as appropriate. The Virtual Store functions as a shallow per-user overlay of protected parts of the file system and registry. Each user has a separate Virtual Store from all other users so misbehaving old-style applications can keep pretending they have write access to places they don't but the impact remains local to the user.

The file system Virtual Store is in %LOCALAPPDATA%\VirtualStore. Here are a few things that mine contains:

_vimrc files
*.pyc files created by GNU Solfege
Reflector.cfg
Trillian crash dumps and user data (this explains a few oddities I've seen!)

The registry Virtual Store is in HKEY_CURRENT_USER\Software\Classes\VirtualStore. Here are a few things that mine contains:

Old obsolete registry keys I created for debugging that got me into this mess.
A EULA key for Acrobat Reader. (Nowadays I use FoxIt Reader.)
Configuration for a few audio codecs.
An MRU list for DirectDraw (weird).
A flag set by my 3G wireless software.
Configuration for OpenVPN.

What Actually Happened

Once upon a time, I ran a little script without elevation to create some registry keys for local development. Since the script was not elevated, the new registry keys were created within the Virtual Store.

Later on, I ran a script to delete these keys. But in the interim I had changed the script to run with elevation. Because it ran with elevation it did not see the keys in the Virtual Store so it left them there.

Then I installed Gallio v3.0.6 which, being an installer, created some new registry keys with elevation in the expected location.

When I tried to run a test, the ProcessInvocation.exe component of TestDriven.Net enumerated these keys in the registry, but without elevation. Consequently it saw the old keys in the Virtual Store and got extremely confused.

Crash!

Moral

Normally the Virtual Store on Vista is not a problem. Actually it's a pretty clever solution to a serious compatibility problem. However, if you ever run into issues like mine you might just need to learn about it so that you can delete or repair the magic virtualized content if needed.

More info: http://support.microsoft.com/kb/927387

Technorati Tags: gallio

Vista UAC and Dog Food

2009-05-28T01:57:00.001-07:00

A few days ago I re-enabled Vista UAC on my development laptop.

Why?

I was developing a few features for Gallio that required being able to perform privilege elevations on Vista. So unless I got my act together to play nice with UAC, those features were just not going to work!

With reluctance I re-enabled UAC. Sometimes dog food, while nutritious, contains unpleasant additives.

Teaser

Here's one of the new features from Gallio v3.0.7 in trunk:

When certain settings are changed, we show shield buttons:

And of course when clicking on one of those buttons we'll get the standard UAC elevation dialog. I can't show you that because by design UAC blocks out other applications including SnagIt.

How it works (very briefly)

To display shield buttons, set the Button's FlatStyle to System and then send the BCM_SETSHIELD message to the button using SendMessage. (Or borrow my ShieldButton class. Ignore the Mono bits.)

Before performing privilege elevation, check whether it is even needed for the current user. Here's one way. A better way might be to examine the Win32 ProcessToken directly.

public static bool CurrentUserHasElevatedPrivileges()
{
WindowsPrincipal principal = new WindowsPrincipal(WindowsIdentity.GetCurrent());
return principal.IsInRole(WindowsBuiltInRole.Administrator);
}

To perform privilege elevation, first split the code into two parts: non-privileged and privileged. The privileged part will have to run in a separate process one way or another. Then decide how to perform the elevation using one of these options.

Use the COM elevation moniker as in the CoCreateInstanceAsAdmin example to instantiate a COM object within an elevated process provided by the system. This method is convenient but it requires that you have registered your privileged code as a COMVisible managed class ahead of time. Consequently, we do not use this method in Gallio which is expected to be xcopy deployable with no prior installation of components required (besides copying files).
Create a separate executable process that contains the privileged code. Be sure to embed an appropriate application manifest like the following either using the Visual Studio manifest build
option or mt.exe.

Our incantation of mt.exe looks a bit like this: "mt.exe -nologo -manifest Elevated.manifest -managedassemblyname:Gallio.Host.Elevated.exe -nodependency -outputresource:Gallio.Host.Elevated.exe;#1". Keep in mind that if you use mt.exe on a signed assembly that the signature will be invalidated and the assembly will need to be re-signed with "sn.exe -R ...". You might consider delay-signing the assembly at first, then merging the manifest, then re-signing the assembly. This way you can tell whether you're working with an assembly with a proper manifest because the original delay-signed copy will fail validation (normally) but the patched and re-signed copy will be good.

Here is the Elevated.manifest file that we use for Gallio.Host.Elevated.exe:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
<description>Provides out of process hosting for Gallio components that require privilege elevation.</description>
<trustInfo xmlns="urn:schemas-microsoft-com:asm.v2">
    <security>
      <requestedPrivileges xmlns="urn:schemas-microsoft-com:asm.v3">
        <requestedExecutionLevel level="requireAdministrator" uiAccess="false"/>
      </requestedPrivileges>
    </security>
</trustInfo>
</assembly>

Launch the privileged process directly being sure to set a few extra properties:

var processStartInfo = new ProcessStartInfo("PrivilegedApp.exe");
processStartInfo.Verb = "runas"; // optional if you embedded a manifest
processStartInfo.UseShellExecute = true; // mandatory
processStartInfo.ErrorDialog = true; // mandatory
processStartInfo.ErrorDialogOwnerHandle = ownerForm.Handle; // recommended

Ok, I just glossed over a million little details...

If you are ever in a position to have to perform privilege elevation in a managed application, I highly recommend reading the relevant MSDN articles three times over.

Technorati Tags: gallio

Gallio on Visual Studio 2010, Redux

2009-05-19T22:23:00.001-07:00

After the Microsoft PDC of 2008, I released a variant of Gallio v3.0.5 with support for the Visual Studio 2010 pre-release bits.

So today I have been working on getting Gallio v3.0.7 running on Visual Studio 2010 Beta 1.

The migration has been pretty easy for the most part. With the PDC bits, I had a hard time producing a build because I had to install all sorts of tools inside the PDC VM and deal with rather unstable bits. With the Beta 1 release, the process is much smoother since I can just install the framework on the build server. Visual Studio 2010 Beta 1 is working ok although I have filed a couple of bugs already.

Unfortunately since not all contributors to Gallio will be running VS 2010 anytime soon I will have to maintain a parallel set of project files and solutions. I guess I will be maintaining *.vs2008.csproj and *.vs2010.csproj files side-by-side for a little while. I wish Microsoft would invest some effort in providing a smooth backward-compatible upgrade path for projects to enable developers on newer tools to work with projects created using older tools without having to actually migrate them over. (This is my 3rd Visual Studio upgrade and I'm getting really tired of this story.)

The first pre-release installer should be available later this week.

Looks promising so far...

Technorati Tags: gallio,mbunit

NDepend v2.12 Rocks.

2009-05-08T23:47:00.001-07:00

I have used NDepend on and off for the past couple of years. It has never been an everyday thing, but whenever I have an itch to find out just how convoluted my code dependency graph has become, NDepend has been there for me.

Full disclosure: Patrick Smacchia sent me a free professional license of NDepend to play with some time ago. To be honest, I didn't really need the license because I was already using the OSS edition of NDepend at the time. I just think this is a cool tool. :-)

Download and Installation

Installing NDepend is pretty easy. Just visit the NDepend download page, type in your email address for the OSS or trial version or your license code for the progressional addition, then download the NDepend ZIP file. Extract the ZIP someplace useful (like C:\Program Files) then pin a shortcut to VisualNDepend.exe to your Quick Launch bar.

Tools Integration

Now launch VisualNDepend. To enable Visual Studio and Reflector integration click on the button in the UI. A couple more clicks and you're done.

I have to say, unboxing NDepend is very slick! There are a bunch of links to little screencast demos to explain how stuff works. I have to admit I have tool envy here. I wish my tools could help the user along like this.

That said, I do wish NDepend (and Reflector also) had an MSI installer flavor. It's harder to explain to others how to extract ZIPs and create shortcuts than to just ask them to run through a standard installer process.

Dissecting Gallio with NDepend

After installing the NDepend add-in for Visual Studio, I opened up the Gallio solution. There's a lot of stuff in here, so for this exploration I'm going to start by looking at the dependencies within Gallio.dll.

It's been a few months since I looked at this so I'm cringing here... What will I discover?

Up pops an instance of Visual NDepend with this dialog listing all of the assemblies in my solution. Pretty slick. I don't have to create a new NDepend project from scratch. On the other hand, I'm only interested in Gallio.dll and I did right-click specifically on that project in Visual Studio so I will remove everything else.

NDepend Report

The next thing that pops up is an NDepend report. It shows up in my default web browser (FireFox). I think it might be cooler if it appeared inside of the Visual NDepend application itself using an embedded web browser (eg. System.Windows.Forms.WebBrowser) but this report is pretty useful anyways.

Bunch of metrics regarding both the source code and the compiled binaries. Looks like Gallio.dll consists 66% of comments. Heh. There's all of the XML documentation!

One thing to keep in mind when reading the report is that not all of the information is equally useful. NDepend computes lots of different metrics to help you find hot spots. Of course, if there's nothing really wrong then what it finds might not be very interesting. You need to read the report intelligently.

Top 10 Methods to Refactor?

For example, here are the top 10 methods in Gallio.dll that I should refactor, according to NDepend.

Whoa, so the top 2 are compiler generated and most of the others are pretty simple methods. Based on the criteria used (see CQL below).

The main thing I'm hitting up against is the IL nesting depth and number of variables. None of these methods have a particularly troubling cyclomatic complexity except for CreateFilteredClosure which has a CC of 20 but can't really be improved much more given what it does.

So in sum, I guess I'm doing ok! Cool. However I bet if I threw this at some production code of my employer, we'd find some interesting things to talk about.

Let's keep digging.

Finding Large Methods

Out of the box, the NDepend report shows a summary of large methods. This is worth paying close attention to.

Here are the 10 biggest methods in Gallio.dll.

I'm amused that FindMiddleSnake made the list. That is one monster method! Unfortunately I can't really do much to improve it because of the nature of the algorithm. Calculating diffs is hard stuff! The topological sort is in a similar category.

However, the PluginCatalog.ApplyTo and DefaultRuntime.VerifyInstallation methods are long for no good reason. So with a little help from ReSharper's Extract Method refactoring here's what ApplyTo now looks like:

Overlap with FxCop

Some of the metrics that NDepend computes out of the box produce similar results to those that could be obtained using FxCop. This is rather interesting since NDepend's CQL language actually has the potential to be much more expressive and configurable than the FxCop verifications, if you like. However there can be some quirks.

Here are some methods and constructors that appear in abstract classes that were marked "public" but that NDepend thinks could be "protected".

The NativeCodeElementWrapper, NativeMemberWrapper and NativeFunctionWrapper constructors I understand (fixed!), the others not so much. TestTypePatternAttribute is a public attribute class whose constructor really does need to be public because it is public API although NDepend doesn't know that. The NativeCodeElementWrapper.Target property is only used internally in Gallio.dll so it could be made protected or internal if desired.

What it comes down to is that you still have to use your head when you interpret the data. It's worth the effort though!

Finding Inadvertent Coupling

Of all of the many metrics and views provided by NDepend, this one is by far my favorite. It shows the afferent coupling (incoming dependencies) in blue, efferent coupling (outgoing dependencies) in green, and mutual coupling (both incoming and outgoing dependencies) in black. The following chart is displaying coupling among namespaces.

Oh no! There's some black in there. The black means that some code in one namespace is using code in another namespace which also happens to use code in the former.

Namespaces are frequently used to separate subsystems from one another. Coupling among namespaces can indicate one of the following problems:

One or more classes are in the wrong namespace and should be moved.
One or more classes have dependencies on the wrong things and should be refactored to use dependency injection or use inversion of control to access foreign services.
The namespaces are not very cohesive and the subsystems they represent should be restructured to be more tightly encapsulated or have fewer responsibilities.
There are common services partially defined in both namespaces that should be extracted and moved elsewhere.

To take a deeper look I just double-click on one of the black squares to drill down.

Ah ha! It looks like the Gallio.Model.Diagnostics.ExceptionData class is being used by several of the classes in the Gallio.Runtime.Logging namespace. This is bad.

The Runtime subsystem is the lowest tier of Gallio and provides the core foundational services needed to get Gallio up and running. The Model subsystem is one of the tiers above the Runtime and its job is to define the test object model. What we have here is a cyclic dependency that crosses tiers.

In a tiered architecture, dependencies should always flow from one tier down to the ones below it and never the other way around. If you let dependencies flow both ways then you may quickly find yourself enmeshed in a bowl of spaghetti!

It looks like in order to fix this problem, I'm going to have to rearrange a few of these dependencies. The upside is that the structure of the system will be cleaner when I am done.

CQL: A Recipe for Analysis

NDepend provides a code query language called CQL which enables all sorts of fancy analysis. In addition to answering various questions interactively, you can incorporate NDepend CQL constraints into your build process such that the build will fail if a constraint is violated. By using CQL as part of a continuous integration process you can validate system design conventions early and often. For example, you can add a CQL constraint to detect when cyclic dependencies are accidentally created across tiers.

To be honest, I have not delved into CQL very much myself. Using it as part of the build process looks like it could be a very useful tool for managing large teams of developers. (I just need to find more time to do architecture work...)

Summary

NDepend is a good tool that repays careful analysis. It won't fix your system design defects for you but it will help you find the trouble spots.

As a result of writing this post, I fixed a few big methods and performed a major refactoring of Gallio's primary namespaces guided mainly by NDepend's dependency matrix view. This is not the first time I have used NDepend to improve Gallio. Here is how that dependency matrix is looking now:

All in all, I feel much better now!

Underfoot

2009-04-20T14:33:00.001-07:00

Every now and then we talk about the "overhead" associated with implementing better practices.

For example, designing a stratified architecture requires some up-front planning to separate the tiers, identify roles, decide on the interfaces, establish conventions, and resolve cross-cutting concerns. During implementation, there will almost certainly be additional code required to marshal messages across tiers, additional configuration required to glue them together and additional infrastructure required to host them. Quite probably the team will also need to learn new skills and adopt other ancillary design/implementation practices at the same time to make it all work out. This effort all takes time, effort, and money.

Likewise adopting TDD/BDD/DbE, using continuous integration, leveraging IoC, adopting one-way asynchronous message passing (instead of RPC), applying command-query separation, and "going SOLID" all have one-time and recurring costs... (and benefits of course.)

I think "overhead" is a misleading label for these costs. It presumes that the costs are excess terms that could be eliminated through greater efficiency. However these costs in fact purchase greater efficiency. They are an investment in laying a foundation for future. They pad the space under our feet not the space over our heads. They let us reach greater heights!

Let's try this paragraph with underfoot instead of overhead.

Building this proposed multi-tier platform will have an initial 20% underfoot while we retool and lay the foundation. Once that is done we will be 50% more efficient though we will have to spend an additional recurring 5% underfoot to keep the axles greased.

Which do you prefer?

Announcing Gallio and MbUnit v3.0.6 Update 1.

2009-04-02T02:00:00.000-07:00

Today we are releasing Gallio and MbUnit v3.0.6 Update 1. This release fixes a regression in the ReSharper support and adds a couple of new features.

Highlights:

Added support for TestDriven.Net category filters.
Added support for more powerful inclusion/exclusion test filter expressions.
Fixed ReSharper v3.1, v4.0 and v4.1 hangs.
Increased NCover v1.5.8 timeout.
Adopted new assembly version numbering scheme.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

Earlier release notes: v3.0.6, all versions

Changes

Inclusion / Exclusion Filters

Gallio provides a mechanism for filtering the set of tests to run. This is typically used to select some subset, for example to run all tests except for integration tests.

However, the filter grammar used in previous versions could not correctly express an exclusion constraint. Several people noted that filters like "not Category: Integration" did not in fact work due to a logical error introduced by a change in how Gallio applies filters. Oops.

In this update we have added a more concrete concept of inclusion and exclusion filters by extending the filter syntax with include and exclude clauses.

Here are a few examples using Gallio.Echo. The same filter syntax is used universally throughout Gallio so you can use them with NAnt, MSBuild, and PowerShell as well.

Gallio.Echo.exe MyTestAssembly.dll "/f:exclude Category: Integration"
Gallio.Echo.exe MyTestAssembly.dll "/f:include Type: MyFixture"
(here the "include" is optional)
Gallio.Echo.exe MyTestAssembly.dll "/f:AuthorName: Jeff and Category: SmokeTests exclude Type: BrokenFixture, AnotherBrokenFixture exclude Importance: NoOneReallyCaresAbout"

TestDriven.Net Category Filters

As of v2.17, TestDriven.Net has a Visual Studio preferences page that allows the user to configure a couple of different settings.

In this update of Gallio we have added support for the TestDriven.Net category filter options. This feature was absent in previous releases because we did not then have a way to express exclusion filters properly.

New Assembly Version Numbering Scheme

Plug-in authors rejoice! I've changed how assembly version numbers are constructed in a subtle but important way.

Previously the assembly version number would change after each build. As a result, code linked to Gallio v3.0.6.749 wouldn't work with Gallio v3.0.6.762 despite there being no significant API changes between those versions.

Now the last component of the assembly version number is always 0. This way there can be no assembly version conflicts within a given release branch. However, we still need a way to distinguish different builds. So the file version number contains the full version number.

For example, in this release the version numbers are set as follows:

Assembly version: v3.0.6.0.
File version: v3.0.6.763.

As a result, you should no longer need to recompile your plug-ins except to make them work with a new release branch.

Aside: There is now a separate branch for Gallio and MbUnit v3.0.6 in case you want to check out the code and play with it. This will also help us keep maintenance updates on track.

Gallio is not Alpha Anymore (And Has Not Been Since Last Year)

2009-04-01T18:55:00.001-07:00

I sometimes receive questions from people about when Gallio and MbUnit v3 will be out of alpha.

The answer is that they have been out of alpha officially since the v3.0.4 release last year. The current releases should be considered stable and final. Sure, there are still bugs to be fixed and enhancements yet to be made but we do consider these releases quite acceptable for use in production systems.

So there you go. :-)

Technorati Tags: gallio,mbunit

How to enable or disable Visual Studio Team Test integration

2009-04-01T12:30:00.001-07:00

Gallio provides integration with Visual Studio Team Test so that you can run your tests using the built-in Visual Studio Test View and other tools.

However to make this work, the project file must be configured with an extra "ProjectTypeGuid" that that informs Visual Studio that the project should be considered a Test Project. By adding or removing this element you can enable or disable support for Visual Studio Team Test.

Caveats

We do not install the Visual Studio Team Test integration by default anymore beginning with v3.0.6 if you perform a "Typical" install. You will need to perform a "Complete" installation or ensure that the "Visual Studio Team System Test Runner" component under "Test Runners" is installed as part of a "Custom" installation.
The built-in MbUnit project templates already contain the appropriate ProjectTypeGuids, so if you create a new project that way then it should just work out of the box.
Some editions of Visual Studio lack support for Test Projects and will refuse to open up any project that contains the special guid! In this case you may have to remove the offending guid so that you can work with your projects.
The Visual Studio Team Test integration is very rough. It has poor performance due to several issues including the fact that Visual Studio wants to populate its list of tests in the foreground after compiling a test project. So the GUI may hang for some time after each build if the Test View is open and you have a big test project. I really hope they fix this in Visual Studio 2010.
At this time, the integration only works with MbUnit v2, MbUnit v3 and xUnit.net. Unfotunately csUnit and NUnit do not support the use of an abstract reflection policy so they are not supported by this feature. However, there does exist another Visual Studio Team Test add-in for NUnit.

In all honesty, I would recommend using TestDriven.Net or ReSharper to run your tests inside Visual Studio instead of using the Test View.

Editing Project Files

Visual Studio uses a magic "ProjectTypeGuid" to determine whether it should consider a given project to be a Test Project. Only Test Projects are considered when populating the Test View.

And the magic guid is... {3AC096D0-A1C2-E12C-1390-A8335801FDAB}

Here's how to edit the files:

1. Open up the *.csproj or *.vbproj file in a text editor. It's just XML (actually an MSBuild build script).

2. Within the first <PropertyGroup> element at the top of the file, look for an existing <ProjectTypeGuids> element.

2a. If you found the <ProjectTypeGuids> element, then add the aforementioned Guid to the list within if not already present. Be sure to separate the guids with semicolons.

2b. If you did not find a <ProjectTypeGuids> element, then copy and paste in the appropriate element below.

C# Projects: *.csproj

<ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};
{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>

Visual Basic Projects: *.vbproj

<ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};
{F184B08F-C81C-45F6-A57F-5ABD9991F28F}</ProjectTypeGuids>

3. Save the project file.

4. Reload the project in Visual Studio.

5. Open up the Test View. The tests in your project should now be populated.

It's Too Slow... How Do I Turn It Off?

If you read the caveats above, I warned you... :-)

There are three ways to turn this off. Any one of them will suffice.

1. Remove the ProjectTypeGuid just added above then reload the project.

2. Disable the Gallio add-in using the Visual Studio Add-In Manager. Contrary to what some people expect, this will not in any way affect the ReSharper integration.

3. Uninstall the "Visual Studio Team System Test Runner" component. You don't need to uninstall all of Gallio for this. Use the Add/Remove programs control panel to Change the set of installed features and look under the "Test Runners" part of the tree.

What The Future Holds...

I do have a hack in mind for improving performance by populating the Visual Studio test list asynchronously although it would run contrary to the stated API contracts. So if the performance is a problem for you but you really really want to use the Visual Studio Team Test controls, please let me know and I'll raise the priority of this task. Also I could use a volunteer to help make the integration more compelling overall.

Also, some time ago I started working on a different native Visual Studio add-in for Gallio called Sail. It's not ready yet but something I've been thinking about again lately... If you'd like to help, let me know.

Technorati Tags: gallio,mbunit