Bits in Motion

Upcoming Events

2009-06-05T01:46:00.001-07:00

I will be attending ALT.Net Canada in Vancouver, BC from June 12th through 14th. My plan is to follow that up with a week-long vacation and visiting friends and family in Ottawa, Toronto, Montreal or Kitchener-Waterloo.

I rather desperately need a vacation...

As for Gallio, I am currently investing most of my spare time into adding support for test files that are not .Net assemblies. When I am finished, Gallio will be able to run tests written in other languages such as C++, Ruby, various XML-based DSLs or anything else that is represented as a file of a type for which a test framework plugin has been installed. As a proof of concept Gallio v3.0.7 will include support for RSpec by way of the Dynamic Language Runtime.

Announcing Gallio and MbUnit v3.0.6 Update 2.

2009-05-29T16:21:00.000-07:00

Today we are releasing Gallio and MbUnit v3.0.6 Update 2. This is mainly a bug fix release.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

Earlier release notes: v3.0.6, v3.0.6 update 1, all versions

Changes

MbUnit:

[New!] Added [Parallelizable(TestScope.All)] and [Parallelizable(TestScope.Descendants)] to recursively mark tests as parallelizable such as all tests within a fixture or all fixtures in an assembly.
Fixed "test was orphaned" errors involving Parallelizable tests.

xUnit.net:

Upgraded adpater to xUnit.net v1.1 RTM.

NCover:

Added support for running NCover v1.5.8 as a 32-bit process on 64-bit machines.

Echo:

Fixed a regression that was causing test reports not to be generated when canceled.

ReSharper:

Fixed ClassCastExceptions in the ReSharper runner.
Fixed missing hotfix lightbulbs in the ReSharper within test files.
Fixed red strikethrough that would appear in the Unit Test Session window when running MbUnit tests declared in subclassed test fixtures.

AutoCAD:

Fixed a regression in the AutoCAD test driver.

TeamCity:

Improved the handling of parallel tests so that TeamCity test statistics make more sense.

Miscellaneous:

Improved the installer to support in-place upgrades.
Fixed a regression that occasionally would prevent dynamic tests from running.

Technorati Tags: gallio,mbunit

Registry Weirdness, UAC and the Vista Virtual Store

2009-05-28T17:45:00.001-07:00

It appears that two different can see a different view of the registry content. I seem to recall seeing mention about registry values potentially being seen differently by elevated and non-elevated applications when UAC is active.

In this case, ProcessInvocation.exe sees the following value for a particular key: (which is wrong!)

C:\Source\MbUnit\v3\src\Extensions\TDNet\Gallio.TDNetRunner\bin\Gallio.TDNetRunner.dll

But regedit.exe sees the following value: (which is correct!)

C:\Program Files\Gallio\bin\TDNet\Gallio.TDNetRunner.dll

The first value was used for local development of Gallio v3.0.6 but I deleted that value! In fact I deleted it a week ago and I don't think I've recreated it since (because Gallio v3.0.7 uses different keys).

The second value was set by the Gallio v3.0.6 installer (uses the old key). This is the value I expect to see globally. All interesting applications were closed at the the time of installation.

So why are some applications seeing the first value and other applications seeing the second value?

No, this is not a simple question of refreshing or shutting down applications. I have shut them both down and tried again and the weird view remains persistent!

Next up, reboot. Argh.

After the Reboot...

Same behavior.

Why?

Here's what I found when I searched the registry for the incorrect value: (SID masked)

HKEY_USERS\S-1-5-21-xxxxxxxxxx-xxxxxxxxx-xxxxxxxxx-xxxx\Software\Classes\VirtualStore\MACHINE\SOFTWARE\MutantDesign\TestDriven.NET\TestRunners\Gallio_MbUnit

Whoa. So at least I'm not dreaming.

The UAC Virtual Store

On Vista, normal users are forbidden from writing to certain folders (like %ProgramFiles% and %SystemRoot%) and into the HKEY_LOCAL_MACHINE part of the registry. Even local administrator users are forbidden from writing into these folders, unless elevated.

Well, not quite. As a compatibility feature, instead of generating an error, a write by a non-elevated application running as a local administrator may be redirected to a Virtual Store on the file system or in the registry as appropriate. The Virtual Store functions as a shallow per-user overlay of protected parts of the file system and registry. Each user has a separate Virtual Store from all other users so misbehaving old-style applications can keep pretending they have write access to places they don't but the impact remains local to the user.

The file system Virtual Store is in %LOCALAPPDATA%\VirtualStore. Here are a few things that mine contains:

_vimrc files
*.pyc files created by GNU Solfege
Reflector.cfg
Trillian crash dumps and user data (this explains a few oddities I've seen!)

The registry Virtual Store is in HKEY_CURRENT_USER\Software\Classes\VirtualStore. Here are a few things that mine contains:

Old obsolete registry keys I created for debugging that got me into this mess.
A EULA key for Acrobat Reader. (Nowadays I use FoxIt Reader.)
Configuration for a few audio codecs.
An MRU list for DirectDraw (weird).
A flag set by my 3G wireless software.
Configuration for OpenVPN.

What Actually Happened

Once upon a time, I ran a little script without elevation to create some registry keys for local development. Since the script was not elevated, the new registry keys were created within the Virtual Store.

Later on, I ran a script to delete these keys. But in the interim I had changed the script to run with elevation. Because it ran with elevation it did not see the keys in the Virtual Store so it left them there.

Then I installed Gallio v3.0.6 which, being an installer, created some new registry keys with elevation in the expected location.

When I tried to run a test, the ProcessInvocation.exe component of TestDriven.Net enumerated these keys in the registry, but without elevation. Consequently it saw the old keys in the Virtual Store and got extremely confused.

Crash!

Moral

Normally the Virtual Store on Vista is not a problem. Actually it's a pretty clever solution to a serious compatibility problem. However, if you ever run into issues like mine you might just need to learn about it so that you can delete or repair the magic virtualized content if needed.

More info: http://support.microsoft.com/kb/927387

Technorati Tags: gallio

Vista UAC and Dog Food

2009-05-28T01:57:00.001-07:00

A few days ago I re-enabled Vista UAC on my development laptop.

Why?

I was developing a few features for Gallio that required being able to perform privilege elevations on Vista. So unless I got my act together to play nice with UAC, those features were just not going to work!

With reluctance I re-enabled UAC. Sometimes dog food, while nutritious, contains unpleasant additives.

Teaser

Here's one of the new features from Gallio v3.0.7 in trunk:

When certain settings are changed, we show shield buttons:

And of course when clicking on one of those buttons we'll get the standard UAC elevation dialog. I can't show you that because by design UAC blocks out other applications including SnagIt.

How it works (very briefly)

To display shield buttons, set the Button's FlatStyle to System and then send the BCM_SETSHIELD message to the button using SendMessage. (Or borrow my ShieldButton class. Ignore the Mono bits.)

Before performing privilege elevation, check whether it is even needed for the current user. Here's one way. A better way might be to examine the Win32 ProcessToken directly.

public static bool CurrentUserHasElevatedPrivileges()
{
WindowsPrincipal principal = new WindowsPrincipal(WindowsIdentity.GetCurrent());
return principal.IsInRole(WindowsBuiltInRole.Administrator);
}

To perform privilege elevation, first split the code into two parts: non-privileged and privileged. The privileged part will have to run in a separate process one way or another. Then decide how to perform the elevation using one of these options.

Use the COM elevation moniker as in the CoCreateInstanceAsAdmin example to instantiate a COM object within an elevated process provided by the system. This method is convenient but it requires that you have registered your privileged code as a COMVisible managed class ahead of time. Consequently, we do not use this method in Gallio which is expected to be xcopy deployable with no prior installation of components required (besides copying files).
Create a separate executable process that contains the privileged code. Be sure to embed an appropriate application manifest like the following either using the Visual Studio manifest build
option or mt.exe.

Our incantation of mt.exe looks a bit like this: "mt.exe -nologo -manifest Elevated.manifest -managedassemblyname:Gallio.Host.Elevated.exe -nodependency -outputresource:Gallio.Host.Elevated.exe;#1". Keep in mind that if you use mt.exe on a signed assembly that the signature will be invalidated and the assembly will need to be re-signed with "sn.exe -R ...". You might consider delay-signing the assembly at first, then merging the manifest, then re-signing the assembly. This way you can tell whether you're working with an assembly with a proper manifest because the original delay-signed copy will fail validation (normally) but the patched and re-signed copy will be good.

Here is the Elevated.manifest file that we use for Gallio.Host.Elevated.exe:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
<description>Provides out of process hosting for Gallio components that require privilege elevation.</description>
<trustInfo xmlns="urn:schemas-microsoft-com:asm.v2">
    <security>
      <requestedPrivileges xmlns="urn:schemas-microsoft-com:asm.v3">
        <requestedExecutionLevel level="requireAdministrator" uiAccess="false"/>
      </requestedPrivileges>
    </security>
</trustInfo>
</assembly>

Launch the privileged process directly being sure to set a few extra properties:

var processStartInfo = new ProcessStartInfo("PrivilegedApp.exe");
processStartInfo.Verb = "runas"; // optional if you embedded a manifest
processStartInfo.UseShellExecute = true; // mandatory
processStartInfo.ErrorDialog = true; // mandatory
processStartInfo.ErrorDialogOwnerHandle = ownerForm.Handle; // recommended

Ok, I just glossed over a million little details...

If you are ever in a position to have to perform privilege elevation in a managed application, I highly recommend reading the relevant MSDN articles three times over.

Technorati Tags: gallio

Gallio on Visual Studio 2010, Redux

2009-05-19T22:23:00.001-07:00

After the Microsoft PDC of 2008, I released a variant of Gallio v3.0.5 with support for the Visual Studio 2010 pre-release bits.

So today I have been working on getting Gallio v3.0.7 running on Visual Studio 2010 Beta 1.

The migration has been pretty easy for the most part. With the PDC bits, I had a hard time producing a build because I had to install all sorts of tools inside the PDC VM and deal with rather unstable bits. With the Beta 1 release, the process is much smoother since I can just install the framework on the build server. Visual Studio 2010 Beta 1 is working ok although I have filed a couple of bugs already.

Unfortunately since not all contributors to Gallio will be running VS 2010 anytime soon I will have to maintain a parallel set of project files and solutions. I guess I will be maintaining *.vs2008.csproj and *.vs2010.csproj files side-by-side for a little while. I wish Microsoft would invest some effort in providing a smooth backward-compatible upgrade path for projects to enable developers on newer tools to work with projects created using older tools without having to actually migrate them over. (This is my 3rd Visual Studio upgrade and I'm getting really tired of this story.)

The first pre-release installer should be available later this week.

Looks promising so far...

Technorati Tags: gallio,mbunit

NDepend v2.12 Rocks.

2009-05-08T23:47:00.001-07:00

I have used NDepend on and off for the past couple of years. It has never been an everyday thing, but whenever I have an itch to find out just how convoluted my code dependency graph has become, NDepend has been there for me.

Full disclosure: Patrick Smacchia sent me a free professional license of NDepend to play with some time ago. To be honest, I didn't really need the license because I was already using the OSS edition of NDepend at the time. I just think this is a cool tool. :-)

Download and Installation

Installing NDepend is pretty easy. Just visit the NDepend download page, type in your email address for the OSS or trial version or your license code for the progressional addition, then download the NDepend ZIP file. Extract the ZIP someplace useful (like C:\Program Files) then pin a shortcut to VisualNDepend.exe to your Quick Launch bar.

Tools Integration

Now launch VisualNDepend. To enable Visual Studio and Reflector integration click on the button in the UI. A couple more clicks and you're done.

I have to say, unboxing NDepend is very slick! There are a bunch of links to little screencast demos to explain how stuff works. I have to admit I have tool envy here. I wish my tools could help the user along like this.

That said, I do wish NDepend (and Reflector also) had an MSI installer flavor. It's harder to explain to others how to extract ZIPs and create shortcuts than to just ask them to run through a standard installer process.

Dissecting Gallio with NDepend

After installing the NDepend add-in for Visual Studio, I opened up the Gallio solution. There's a lot of stuff in here, so for this exploration I'm going to start by looking at the dependencies within Gallio.dll.

It's been a few months since I looked at this so I'm cringing here... What will I discover?

Up pops an instance of Visual NDepend with this dialog listing all of the assemblies in my solution. Pretty slick. I don't have to create a new NDepend project from scratch. On the other hand, I'm only interested in Gallio.dll and I did right-click specifically on that project in Visual Studio so I will remove everything else.

NDepend Report

The next thing that pops up is an NDepend report. It shows up in my default web browser (FireFox). I think it might be cooler if it appeared inside of the Visual NDepend application itself using an embedded web browser (eg. System.Windows.Forms.WebBrowser) but this report is pretty useful anyways.

Bunch of metrics regarding both the source code and the compiled binaries. Looks like Gallio.dll consists 66% of comments. Heh. There's all of the XML documentation!

One thing to keep in mind when reading the report is that not all of the information is equally useful. NDepend computes lots of different metrics to help you find hot spots. Of course, if there's nothing really wrong then what it finds might not be very interesting. You need to read the report intelligently.

Top 10 Methods to Refactor?

For example, here are the top 10 methods in Gallio.dll that I should refactor, according to NDepend.

Whoa, so the top 2 are compiler generated and most of the others are pretty simple methods. Based on the criteria used (see CQL below).

The main thing I'm hitting up against is the IL nesting depth and number of variables. None of these methods have a particularly troubling cyclomatic complexity except for CreateFilteredClosure which has a CC of 20 but can't really be improved much more given what it does.

So in sum, I guess I'm doing ok! Cool. However I bet if I threw this at some production code of my employer, we'd find some interesting things to talk about.

Let's keep digging.

Finding Large Methods

Out of the box, the NDepend report shows a summary of large methods. This is worth paying close attention to.

Here are the 10 biggest methods in Gallio.dll.

I'm amused that FindMiddleSnake made the list. That is one monster method! Unfortunately I can't really do much to improve it because of the nature of the algorithm. Calculating diffs is hard stuff! The topological sort is in a similar category.

However, the PluginCatalog.ApplyTo and DefaultRuntime.VerifyInstallation methods are long for no good reason. So with a little help from ReSharper's Extract Method refactoring here's what ApplyTo now looks like:

Overlap with FxCop

Some of the metrics that NDepend computes out of the box produce similar results to those that could be obtained using FxCop. This is rather interesting since NDepend's CQL language actually has the potential to be much more expressive and configurable than the FxCop verifications, if you like. However there can be some quirks.

Here are some methods and constructors that appear in abstract classes that were marked "public" but that NDepend thinks could be "protected".

The NativeCodeElementWrapper, NativeMemberWrapper and NativeFunctionWrapper constructors I understand (fixed!), the others not so much. TestTypePatternAttribute is a public attribute class whose constructor really does need to be public because it is public API although NDepend doesn't know that. The NativeCodeElementWrapper.Target property is only used internally in Gallio.dll so it could be made protected or internal if desired.

What it comes down to is that you still have to use your head when you interpret the data. It's worth the effort though!

Finding Inadvertent Coupling

Of all of the many metrics and views provided by NDepend, this one is by far my favorite. It shows the afferent coupling (incoming dependencies) in blue, efferent coupling (outgoing dependencies) in green, and mutual coupling (both incoming and outgoing dependencies) in black. The following chart is displaying coupling among namespaces.

Oh no! There's some black in there. The black means that some code in one namespace is using code in another namespace which also happens to use code in the former.

Namespaces are frequently used to separate subsystems from one another. Coupling among namespaces can indicate one of the following problems:

One or more classes are in the wrong namespace and should be moved.
One or more classes have dependencies on the wrong things and should be refactored to use dependency injection or use inversion of control to access foreign services.
The namespaces are not very cohesive and the subsystems they represent should be restructured to be more tightly encapsulated or have fewer responsibilities.
There are common services partially defined in both namespaces that should be extracted and moved elsewhere.

To take a deeper look I just double-click on one of the black squares to drill down.

Ah ha! It looks like the Gallio.Model.Diagnostics.ExceptionData class is being used by several of the classes in the Gallio.Runtime.Logging namespace. This is bad.

The Runtime subsystem is the lowest tier of Gallio and provides the core foundational services needed to get Gallio up and running. The Model subsystem is one of the tiers above the Runtime and its job is to define the test object model. What we have here is a cyclic dependency that crosses tiers.

In a tiered architecture, dependencies should always flow from one tier down to the ones below it and never the other way around. If you let dependencies flow both ways then you may quickly find yourself enmeshed in a bowl of spaghetti!

It looks like in order to fix this problem, I'm going to have to rearrange a few of these dependencies. The upside is that the structure of the system will be cleaner when I am done.

CQL: A Recipe for Analysis

NDepend provides a code query language called CQL which enables all sorts of fancy analysis. In addition to answering various questions interactively, you can incorporate NDepend CQL constraints into your build process such that the build will fail if a constraint is violated. By using CQL as part of a continuous integration process you can validate system design conventions early and often. For example, you can add a CQL constraint to detect when cyclic dependencies are accidentally created across tiers.

To be honest, I have not delved into CQL very much myself. Using it as part of the build process looks like it could be a very useful tool for managing large teams of developers. (I just need to find more time to do architecture work...)

Summary

NDepend is a good tool that repays careful analysis. It won't fix your system design defects for you but it will help you find the trouble spots.

As a result of writing this post, I fixed a few big methods and performed a major refactoring of Gallio's primary namespaces guided mainly by NDepend's dependency matrix view. This is not the first time I have used NDepend to improve Gallio. Here is how that dependency matrix is looking now:

All in all, I feel much better now!

Underfoot

2009-04-20T14:33:00.001-07:00

Every now and then we talk about the "overhead" associated with implementing better practices.

For example, designing a stratified architecture requires some up-front planning to separate the tiers, identify roles, decide on the interfaces, establish conventions, and resolve cross-cutting concerns. During implementation, there will almost certainly be additional code required to marshal messages across tiers, additional configuration required to glue them together and additional infrastructure required to host them. Quite probably the team will also need to learn new skills and adopt other ancillary design/implementation practices at the same time to make it all work out. This effort all takes time, effort, and money.

Likewise adopting TDD/BDD/DbE, using continuous integration, leveraging IoC, adopting one-way asynchronous message passing (instead of RPC), applying command-query separation, and "going SOLID" all have one-time and recurring costs... (and benefits of course.)

I think "overhead" is a misleading label for these costs. It presumes that the costs are excess terms that could be eliminated through greater efficiency. However these costs in fact purchase greater efficiency. They are an investment in laying a foundation for future. They pad the space under our feet not the space over our heads. They let us reach greater heights!

Let's try this paragraph with underfoot instead of overhead.

Building this proposed multi-tier platform will have an initial 20% underfoot while we retool and lay the foundation. Once that is done we will be 50% more efficient though we will have to spend an additional recurring 5% underfoot to keep the axles greased.

Which do you prefer?

Announcing Gallio and MbUnit v3.0.6 Update 1.

2009-04-02T02:00:00.000-07:00

Today we are releasing Gallio and MbUnit v3.0.6 Update 1. This release fixes a regression in the ReSharper support and adds a couple of new features.

Highlights:

Added support for TestDriven.Net category filters.
Added support for more powerful inclusion/exclusion test filter expressions.
Fixed ReSharper v3.1, v4.0 and v4.1 hangs.
Increased NCover v1.5.8 timeout.
Adopted new assembly version numbering scheme.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

Earlier release notes: v3.0.6, all versions

Changes

Inclusion / Exclusion Filters

Gallio provides a mechanism for filtering the set of tests to run. This is typically used to select some subset, for example to run all tests except for integration tests.

However, the filter grammar used in previous versions could not correctly express an exclusion constraint. Several people noted that filters like "not Category: Integration" did not in fact work due to a logical error introduced by a change in how Gallio applies filters. Oops.

In this update we have added a more concrete concept of inclusion and exclusion filters by extending the filter syntax with include and exclude clauses.

Here are a few examples using Gallio.Echo. The same filter syntax is used universally throughout Gallio so you can use them with NAnt, MSBuild, and PowerShell as well.

Gallio.Echo.exe MyTestAssembly.dll "/f:exclude Category: Integration"
Gallio.Echo.exe MyTestAssembly.dll "/f:include Type: MyFixture"
(here the "include" is optional)
Gallio.Echo.exe MyTestAssembly.dll "/f:AuthorName: Jeff and Category: SmokeTests exclude Type: BrokenFixture, AnotherBrokenFixture exclude Importance: NoOneReallyCaresAbout"

TestDriven.Net Category Filters

As of v2.17, TestDriven.Net has a Visual Studio preferences page that allows the user to configure a couple of different settings.

In this update of Gallio we have added support for the TestDriven.Net category filter options. This feature was absent in previous releases because we did not then have a way to express exclusion filters properly.

New Assembly Version Numbering Scheme

Plug-in authors rejoice! I've changed how assembly version numbers are constructed in a subtle but important way.

Previously the assembly version number would change after each build. As a result, code linked to Gallio v3.0.6.749 wouldn't work with Gallio v3.0.6.762 despite there being no significant API changes between those versions.

Now the last component of the assembly version number is always 0. This way there can be no assembly version conflicts within a given release branch. However, we still need a way to distinguish different builds. So the file version number contains the full version number.

For example, in this release the version numbers are set as follows:

Assembly version: v3.0.6.0.
File version: v3.0.6.763.

As a result, you should no longer need to recompile your plug-ins except to make them work with a new release branch.

Aside: There is now a separate branch for Gallio and MbUnit v3.0.6 in case you want to check out the code and play with it. This will also help us keep maintenance updates on track.

Gallio is not Alpha Anymore (And Has Not Been Since Last Year)

2009-04-01T18:55:00.001-07:00

I sometimes receive questions from people about when Gallio and MbUnit v3 will be out of alpha.

The answer is that they have been out of alpha officially since the v3.0.4 release last year. The current releases should be considered stable and final. Sure, there are still bugs to be fixed and enhancements yet to be made but we do consider these releases quite acceptable for use in production systems.

So there you go. :-)

Technorati Tags: gallio,mbunit

How to enable or disable Visual Studio Team Test integration

2009-04-01T12:30:00.001-07:00

Gallio provides integration with Visual Studio Team Test so that you can run your tests using the built-in Visual Studio Test View and other tools.

However to make this work, the project file must be configured with an extra "ProjectTypeGuid" that that informs Visual Studio that the project should be considered a Test Project. By adding or removing this element you can enable or disable support for Visual Studio Team Test.

Caveats

We do not install the Visual Studio Team Test integration by default anymore beginning with v3.0.6 if you perform a "Typical" install. You will need to perform a "Complete" installation or ensure that the "Visual Studio Team System Test Runner" component under "Test Runners" is installed as part of a "Custom" installation.
The built-in MbUnit project templates already contain the appropriate ProjectTypeGuids, so if you create a new project that way then it should just work out of the box.
Some editions of Visual Studio lack support for Test Projects and will refuse to open up any project that contains the special guid! In this case you may have to remove the offending guid so that you can work with your projects.
The Visual Studio Team Test integration is very rough. It has poor performance due to several issues including the fact that Visual Studio wants to populate its list of tests in the foreground after compiling a test project. So the GUI may hang for some time after each build if the Test View is open and you have a big test project. I really hope they fix this in Visual Studio 2010.
At this time, the integration only works with MbUnit v2, MbUnit v3 and xUnit.net. Unfotunately csUnit and NUnit do not support the use of an abstract reflection policy so they are not supported by this feature. However, there does exist another Visual Studio Team Test add-in for NUnit.

In all honesty, I would recommend using TestDriven.Net or ReSharper to run your tests inside Visual Studio instead of using the Test View.

Editing Project Files

Visual Studio uses a magic "ProjectTypeGuid" to determine whether it should consider a given project to be a Test Project. Only Test Projects are considered when populating the Test View.

And the magic guid is... {3AC096D0-A1C2-E12C-1390-A8335801FDAB}

Here's how to edit the files:

1. Open up the *.csproj or *.vbproj file in a text editor. It's just XML (actually an MSBuild build script).

2. Within the first <PropertyGroup> element at the top of the file, look for an existing <ProjectTypeGuids> element.

2a. If you found the <ProjectTypeGuids> element, then add the aforementioned Guid to the list within if not already present. Be sure to separate the guids with semicolons.

2b. If you did not find a <ProjectTypeGuids> element, then copy and paste in the appropriate element below.

C# Projects: *.csproj

<ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};
{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>

Visual Basic Projects: *.vbproj

<ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};
{F184B08F-C81C-45F6-A57F-5ABD9991F28F}</ProjectTypeGuids>

3. Save the project file.

4. Reload the project in Visual Studio.

5. Open up the Test View. The tests in your project should now be populated.

It's Too Slow... How Do I Turn It Off?

If you read the caveats above, I warned you... :-)

There are three ways to turn this off. Any one of them will suffice.

1. Remove the ProjectTypeGuid just added above then reload the project.

2. Disable the Gallio add-in using the Visual Studio Add-In Manager. Contrary to what some people expect, this will not in any way affect the ReSharper integration.

3. Uninstall the "Visual Studio Team System Test Runner" component. You don't need to uninstall all of Gallio for this. Use the Add/Remove programs control panel to Change the set of installed features and look under the "Test Runners" part of the tree.

What The Future Holds...

I do have a hack in mind for improving performance by populating the Visual Studio test list asynchronously although it would run contrary to the stated API contracts. So if the performance is a problem for you but you really really want to use the Visual Studio Team Test controls, please let me know and I'll raise the priority of this task. Also I could use a volunteer to help make the integration more compelling overall.

Also, some time ago I started working on a different native Visual Studio add-in for Gallio called Sail. It's not ready yet but something I've been thinking about again lately... If you'd like to help, let me know.

Technorati Tags: gallio,mbunit

Debugging Tests with Gallio

2009-04-01T11:48:00.001-07:00

Here's the rundown of how to debug.

Icarus: Click the new "Debug" button in v3.0.6.

Visual Studio Team Test: Select the tests in the Test View and click Debug. (This assumes that your test project has the appropriate ProjectTypeGuid to be recognized by Visual Studio as a Test Project. See next post.)

TestDriven.Net: Right-click on a test or fixture and select "Test With... Debugger"

ReSharper: Select a test in the editor or test session view and use the Debug drop-down or toolbar button as appropriate.

Echo from command-line: Run tests with /debug argument. This will automatically attach to Visual Studio (or launch it if it is not running).

PowerShell: Use the "/DebugTests" switch for similar behavior to Echo.

NAnt: Add the debug="true" attribute for similar behavior to Echo.

MSBuild: Add the Debug="true" attribute for similar behavior to Echo.

It is no longer necessary to set the test runner type to IsolatedAppDomain if you use any of the above methods to run tests with the debugger.

On the other hand if you are attempting to attach to an existing test run of Gallio that was not started using any of the debugging mechanisms described above and the test runner type is IsolatedProcess (the default) be sure to attach to the Gallio.Host.exe or Gallio.Host.x86.exe process instead of Gallio.Icarus.exe or whatever.

Announcing Gallio and MbUnit v3.0.6

2009-03-22T17:13:00.001-07:00

Today we are releasing Gallio and MbUnit v3.0.6. Lots of new stuff, including new features and performance enhancements.

Highlights:

Parallelizable tests.
One-click test debugging from Icarus.
Auto-reload and auto-run tests from Icarus after each build.
ReSharper 4.5 beta support with improved performance.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

Release notes: All release notes from all versions.

Remark (added 3/27): We strongly recommend upgrading to ReSharper v4.5 beta and TestDriven.Net v2.18+ before installing this release since there have been significant performance improvements. Also it appears there may be a regression in the Gallio support for older ReSharper versions that can result in deadlock during test exploration. (If for some reason you cannot upgrade at this time and you are experiencing this problem, you may try a recent nightly build instead.)

Acknowledgements

In this release I would like to especially acknowledge Graham Hay for his work enhancing Icarus with new features such as running tests with the debugger and also Yann Trévin for his work designing and building the MbUnit v3 contract verifiers. Eddy Young also helped add support for running MSTest 2005 tests (previously we only supported MSTest 2008).

I would also like to thank all of the other members of the community who contributed valuable feedback, patches and support during this release cycle.

Thanks!

Edit: Fixed Eddy Young's name in the acknowledgements. Thanks Eddy!

MbUnit

Parallelizable

With v3.0.6, MbUnit helps you get the most out of your multi-core CPU. Mark any test [Parallelizable] and it will be permitted to run in parallel with other parallelizable tests in the same fixture.

Fixtures can also be marked parallelizable to enable them to be run in parallel with other parallelizable fixtures.

Please note that if you want all tests within a fixture to be considered parallelizable then you still need to add [Parallelizable] to each of them. (We might add a feature to set this at the fixture or assembly level later based on user feedback.)

Also note that just because a test or fixture is marked parallelizable does not mean it will run in parallel with other tests in particular. For the sake of efficiency, we limit the number of active tests threads based on the configured degree of parallelism. If you want a specific number of instances of a test to run in parallel with each other, consider using [ThreadedRepeat].

DegreeOfParallelism

The degree of parallelism setting controls the maximum number of tests that MbUnit will attempt to run in parallel with one another. By default, the degree of parallelism equals the number of CPUs you have, or 2 at a minimum.

If you don't like the default then you can override the degree of parallelism at the assembly-level like this:

DefaultTestCaseTimeout (Breaking Change!)

In MbUnit v2 and earlier versions of MbUnit v3, there was a non-configurable default timeout of 10 minutes per test fixture. Now we have a configurable default timeout of 10 minutes per test case.

Old behaviour:

10 minute timeout per fixture
non-configurable
override by setting [Timeout] attribute on the fixture

New behaviour:

10 minute timeout per test case
configurable with [DefaultTestCaseTimeout] on the assembly
override by setting [Timeout] attribute on the test

We have a default timeout in place because it's useful for the sake of robustness. Otherwise tests could just run forever if left unattended. This is different from NUnit which has no timeout by default (even if you use Gallio to run your NUnit tests).

If you absolutely hate timeouts... here's what you need to do:

Contact Verifiers Redux (Breaking Change!)

Contract verifiers simplify the process of testing that an implementation of a common contract is correct. For example, they can help you test your custom Exception class, verify that your datatype's Equals & GetHashCode are consistent, and check IList<T> invariants.

We originally introduced this feature in MbUnit v3.0.4, updated it in v3.0.5 and Yann Trévin has been busy making it better for v3.0.6!

Instead of using custom attributes to use a contract verifier, we now use a read-only field in the test fixture. This change makes the syntax much more expressive, but you will have to update any tests written in the old style.

More documentation here: Contract Verifiers Chapter in Gallio Book (url subject to change)

In this example, we are testing whether the Parity class implements the Equatable contract correctly. The contract verifier ensures that all members of the equivalence classes are considered equal to each other (and have the same hashcode); that all members of different equivalence classes are considered unequal; ensures that null values are handled correctly and that both Equals(object) and Equals(Parity) behave the same way. It could also check operator overloads, if we implemented them.

The subject under test...

The test...

Xml Data Source

The CSV data source has been very popular. Did you know you can associate metadata with CSV data rows, like expected exceptions? Just add a column with a header like "[ExpectedException]", "[Description]", or some other metadata.

In v3.0.6, we now have an XML data source too.

Some data, including a little custom metadata...

The test...

The output in the report, notice the metadata...

Next up, does anyone want a SQL data source?

Serialization Asserts

This release includes a couple of new Assertions to help with testing serialization.

.Net serialization protocol asserts:

Assert.IsSerializableType: Asserts that a type supports .Net serialization by way of the [Serializable] attribute. Also ensures that type has a deserialization constructor if it also implements ISerializable.
Assert.Serialize: Serializes a value using a .Net serialization formatter. Asserts that serialization succeeded and returns resulting serialized value as a stream.
Assert.Deserialize: Deserializes a value using a .Net serialization formatter. Asserts that deserialization succeeded and returns the resulting deserialized value.
Assert.SerializeThenDeserialize: Serializes then deserializes a value using a .Net serialization formatter. Asserts that both serialization and deserialization succeeded and returns the resulting deserialized value after the round-trip.
Assert.BinarySerialize: Shortcut for Assert.Serialize using a BinaryFormatter.
Assert.BinaryDeserialize: Shortcut for Assert.Deserialize using a BinaryFormatter.
Assert.BinarySerializeThenDeserialize: Shortcut for Assert.SerializeThenDeserialize using a BinaryFormatter.

Xml serialization protocol asserts:

Assert.IsXmlSerializableType: Asserts that an XmlSerializer can be constructed for a given type without error.
Assert.XmlSerialize: Serializes a value using an XmlSerializer. Asserts that serialization succeeded and returns the resulting Xml as a string.
Assert.XmlDeserialize: Deserializes a value using an XmlSerializer. Asserts that deserialization succeeded and returns the resulting deserialized value.
Assert.XmlSerializeThenDeserialize: Serializes then deserialized a value using an XmlSerializer. Asserts that both serialization and deserialization succeeded and returns the resulting deserialized value after the round-trip.

Typical usage...

Terminating Tests with Specific Outcomes

MbUnit v3 includes several methods for controlling the outcome of a test, including a couple of new ones introduced in v3.0.6.

Assert.Fail: Raises an assertion failure by throwing an AssertionFailedException, resulting in a failed outcome. If executed within an Assert.Multiple block or in a test with [MultipleAsserts] then the failure is logged but the exception is deferred until the end of the block or of the test.
Assert.Inconclusive: Terminates a test by throwing a TestInconclusiveException, resulting in an inconclusive outcome.
[new!] Assert.Terminate: Terminates a test by throwing a TestTerminatedException, resulting in a user-specified outcome.
[new!] Assert.TerminateSilently: Terminates a test by throwing a SilentTestException, resulting in a user-specified outcome. Unlike Assert.Terminate, does not log a stack trace at the point of termination.
[new!] Assert.IsFailurePending: Indicates whether an assertion failure has been raised but the AssertionFailedException has not yet been thrown. This is used in conjunction with Assert.Multiple to determine whether any failures have already been observed within the block.

Reminder: There are two mechanisms in MbUnit to enable a test with multiple assertions to keep running even if some of them fail.

Assert.Multiple: Runs a block of code that may contain multiple assertions. If any assertions fail then the test will be terminated with a failed outcome when the block finishes.
[MultipleAsserts]: You may add the MultipleAsserts attribute to a test to effectively run the whole test within an Assert.Multiple block.

Diagnostic Log

A few people have asked how to print progress messages from tests. They try TestLog, Debug.WriteLine and Console.WriteLine, and sure enough, nothing gets printed to the console, by design. We capture all of this information and write it to the test report.

Now you can use DiagnosticLog.WriteLine to write progress messages to the TestDriven.Net output window, the Icarus Runtime Log view or the console when running Echo, NAnt, MSBuild or PowerShell.

Icarus

One-Click Debugging

Icarus can now run tests under the debugger with just one click of the "Debug" button. It will attach (or launch) the Visual Studio debugger to the test process and start the tests.

There is no longer any need to change the test runner type or to manually attach to the Icarus process.

Auto-Reload and Auto-Run

Icarus now supports automatically reloading and running all loaded tests whenever they are recompiled.

In v3.0.6, these features are controlled by a Preference. In future releases we will probably make them available from the toolbar so that it is easier to switch modes on demand.

Additional Test Runner Enhancements

Passing Additional Arguments to NCover

Gallio includes support for launching tests with NCover already attached. By default, the coverage report is written out to "Coverage.xml" in the current directory of the test run.

In v3.0.6 we have added the ability to change the location of the coverage file and to pass additional arguments to NCover. Typically you will want to pass additional arguments to inform NCover to exclude certain assemblies from code coverage.

Here is an example using Gallio.Echo to run the tests from the command-line. If you are using NAnt, MSBuild or Powershell then the process is similar. Refer to the Documentation for details of how to pass the NCoverCoverageFile and NCoverArguments "runner properties" in each case.

Gallio.Echo.exe Widget.Tests.dll /runner:NCover /runner-property:NCoverCoverageFile='C:\Temp\WidgetCoverage.xml' /runner-property:NCoverArguments='//eas .*.Tests;Gallio;MbUnit;OtherIgnoredAssembly'

Echo Support for Test Projects

Echo now supports running test projects such as are created by Icarus. Just pass it in on the command-line.

Gallio.Echo.exe WidgetTests.gallio

Launch Tests With Debugger Attached

It is now possible to launch tests with the debugger attached from Echo, NAnt, MSBuild or PowerShell by adding the "debug" argument.

Here is an example for Echo. The other runners are similar. Refer to the Documentation for details of how to pass the Debug option in each case.

Gallio.Echo.exe Widget.Tests.dll /debug

Total Run-Time Limit

To limit the total amount of time that the tests are allowed to run, specify the total run-time limit. This is a good idea when incorporating tests into a Continuous Integration build to ensure that runaway tests don't get to run for too long.

Here is an example for Echo. The other runners are similar. Refer to the Documentation for details of how to pass the RunTimeLimit option in each case. The time span is specified in seconds.

Gallio.Echo.exe Widget.Tests.dll /run-time-limit:600

External Tools Compatibility

We've updated integration with a variety of external tools.

CCNet v1.0.4
CSUnit v2.0.5
NUnit v2.4.8
Pex v0.9 Academic
TypeMock v5.1
TestDriven.Net v2.19
NCover v1.5.8, v2, v3.
ReSharper v3.1, v4.0, v4.1 , v4.5 Beta
TeamCity v4.0
xUnit.Net v1.0.3
Visual Studio 2005, 2008 (for VS 2010 support, use Gallio v3.0.5 for now)

Change Summary

Gallio Core

Redesigned test lifecycle to significantly improve performance when running tests in multiple test assemblies as a batch.
Added a debugging API, use to provide new debugging features in test runners.
Fixed an issue with significant whitespace being unintentionally dropped from the report due to XML whitespace handling.
Added an XSD file for Gallio reports.
Dropped dependency on ISymWrapper. There should be no more Exception Assistant dialogs about COMExceptions popping up in Visual Studio while launching the tests.
Removed a stray UTF8 Byte Order Mark that appeared in HTML formatted reports.
Gallio registers its TestDriven.Net extensions even if TestDriven.Net is not yet installed.
Added support for passing custom properties to test hosts and report formatters.
Improved diagnostics when running tests out of process.
Various API changes and refactoring.
Added support for jumping to the source code of a test from the report.
Added support for capturing diagnostic log messages during test execution.
Improved the Navigator feature (jump to source from a report) to automatically launch Visual Studio if needed.

MbUnit v3

Added [Parallelizable], [DegreeOfParallelism], [DefaultTestCaseTimeout] and [XmlData].
Added assertions and helpers: Terminate, TerminateSilently, IsFailurePending, IsBinarySerializable, IsXmlSerializable.
Added DiagnosticLog class for writing to the log.
Fixed [Factory] data sources defined on generic types.
Fixed dynamic test outcome reporting.
Fixed project templates for VB.Net.
Fixed Assert.EndsWith failure message.
Added generic variations of Assert.Is[Not]InstanceOfType and Assert.Is[Not]AssignableFrom.
Fixed a faulty assertion in the Exception contract verifier regarding the setting of the Exception's Message property.
Fixed an issue enumerating tests in nested types that inherit from one of their own containing types.
Modified [Rollback] to enter a COM+ transaction context like MbUnit v2's original rollback attribute. This should fix compatibility issues others have noted.
Fixed inheritance of [Row] attribute (and others) on test methods in derived classes.
Added TestStep.RunStepAndVerifyOutcome.
Fixed an issue with [MultipleAssert] affecting nested test steps.
Renamed MetadataKeys.CategoryName to MetadataKeys.Category.
Modified [StaticTestFactory] to run tests in the exact order specified.
Now tolerates missing metadata columns in CSV files.
Applied neoeinstein's patch providing IComparer<T> based overloads for relational asserts such as Assert.IsLessThan.
Added DiagnosticLog class to enable users to write messages to the console during test execution.
Eliminated a deep recursion issue in BaseTest.IsLocalIdUniqueAmongSiblings that caused a StackOverflowException.
Changed timeout policy to apply per-test case not per fixture.

MbUnit v2 Adapter

Fixed MbUnit v2 adapter top-level assembly outcome reporting.
Ported old MbUnit v2 XmlAsserts over to MbUnit.Compatibility.dll.
Fixed a problem where the CCNet integration documentation was not being installed.
Fixed a problem displaying attachments in CCNet.

MSTest Adapter

Auto-detect the version of the MSTest framework referenced by tests. MSTest 2005 not currently supported. (volunteer needed)

Icarus

Added "Debug" command to run tests with the debugger!
Fixed Icarus filter inconclusive.
Improved Icarus view source from the test tree.
Check to ensure assemblies exist before reloading.
Icarus now remembers its position and size between executions.

Echo

Added support for running Icarus projects.
Added /debug switch to run tests with the debugger.
Added /runtimelimit switch to set a time limit on test execution.
Added /runner-property option to pass custom options to the test runner.
Added /report-formatter-property option to pass custom options to the report formatter.

NAnt

Added debug attribute to run tests with the debugger.
Added run-time-limit attribute to set a time limit on test execution.
Added runner-property element to pass custom options to the test runner.
Added report-formatter-property element to pass custom options to the report formatter.

MSBuild

Added Debug attribute to run tests with the debugger.
Added RunTimeLimit attribute to set a time limit on test execution.
Added RunnerProperties element to pass custom options to the test runner.
Added ReportFormatterProperties element to pass custom options to the report formatter.

PowerShell

Added -DebugTests switch to run tests with the debugger.
Added -RunTimeLimit switch to set a time limit on test execution.
Added -RunnerProperties option to pass custom options to the test runner.
Added -ReportFormatterProperties option to pass custom options to the report formatter.

ReSharper

Added support for ReSharper v4.5 beta.
Added memoization to the reflection API to improve performance.

Visual Studio Team Test

Now runs tests in a background thread to provide better feedback during execution.
Added memoization to the reflection API to improve performance.
The MSTest adapter now support MSTest 2005 in addition to 2008.

NCover

Added support for NCover v3.
Added "NCoverCoverageFile" and "NCoverArguments" properties to tweak the command-line.
Fixed several issues that caused long-running coverage runs to fail.

TypeMock

Updated to TypeMock v5.1 integration assembly.

Pex

Upgraded Pex support for v0.9 Academic.

AutoCAD

Read Gallio's process id more robustly.

Technorati Tags: gallio,mbunit

Gallio v3.1 may use a message bus.

2009-03-16T23:44:00.001-07:00

After hanging out at QCon San Francisco last fall with the likes of Dave Laribee, Greg Young, Dru Sellers and Chris Patterson, I got to thinking about some problems I've been having with Gallio's communication patterns.

Why Use a Message Bus?

1. One-way asynchronous message passing is useful for publishing incremental status updates, notifications and interruption requests during the execution of long-running processes. Doing this with remote procedure calls requires callback interfaces, bi-directional channels, and possibly a background notification queue, ugh.

2. Message passing is explicit about connectivity. This is a good thing because communication faults should be handled differently from other application-level faults.

3. Assuming the messaging protocol is non-proprietary then we can open up participation to non-.Net agents.

4. Multiple subscribers can listen for messages published to a common message bus. Multiple publishers can publish to the same message bus. More importantly: they don't need central coordination.

5. A subscriber's lifetime is decoupled from that of the publisher and vice-versa. New subscribers and publishers can join the bus at any time which opens up all kinds of possibilities for auditing, reporting, parallel execution, passive monitoring, and maintaining a test grid.

6. Cross-Process communication becomes completely straightforward.

7. The API can evolve dramatically in very useful ways. In particular, testing can become an open-ended activity rather than a single long-running process. If you like, you could have testing processes running on a grid all day publishing results from their latest exhaustive trials.

Which Message Bus?

I don't know.

Help me choose.

Here are a few criteria:

1. Should run cross-platform, at least under CLR and Mono.

2. Should not depend on proprietary / non-free infrastructure components.

3. Should not be tightly coupled to .Net protocols such as object serialization.

4. Should be possible (and easy) to implement clients in different languages and runtime environments.

5. Should use well-known and straightforward standard message formats like XML or JSON with as little extra noise in the envelope as possible.

6. Should not require any up-front installation or configuration. Each endpoint can be self-managed if desired. The whole bus might be considered transient, something to be set up and torn down as needed. In fact, we probably don't need a full-blown bus. We just need some kind of efficient inter-process asynchronous messaging fabric. (peer to peer?)

7. Should support auto-discovery of services among a group of related processes.

8. Should be able to inform a client of communication faults but does not required durable messaging.

Technorati Tags: gallio

Kudos to NCover for Excellent Support

2009-03-06T15:46:00.001-08:00

I want to mention here that Stephen Ward over at NCover went above and beyond the call of duty to diagnose a problem I was having with NCover spontaneously aborting on the Gallio build server during a coverage run. He checked out the code, figured out how it worked, tried running it locally and when that didn't reproduce the problem he logged into the build server and fixed it there.

It turns out that the problem was even my fault! A 2-minute timeout during host process termination that I had forgotten about. Unfortunately we're talking about a massive NCover coverage log here on a severely underpowered build server, so it needs a bit more time to write it out.

Lessons learned:

The NCover team is really cool.
If I'm going to abnormally terminate a process after a timeout, I should at least write a message to the log when I do it.

Thanks!

NCover v3 Magic...

2009-02-18T14:24:00.001-08:00

I just upgraded the Gallio NCover integration to support NCover v3. It all looked okay except that I could not get the integration test to pass!

No matter what I did, NCover would exclude coverage of any Gallio code at all including my test assembly! However it would include coverage information about Castle assemblies.

On a whim, I ran grep Gallio * over the NCover program files folder, and here's what I found in a file called NCover.Console.exe.config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <configSections>
        <section name="ncover.systemAssemblies" type="NCover.Framework.SystemAssemblySectionHandler, NCover.Framework"/>
    </configSections>
    <startup>
    </startup>
    <ncover.systemAssemblies>
        <assembly>msvc[mrp].*</assembly>
        <assembly>mscorcfg</assembly>
        <assembly>mscorlib</assembly>
        <assembly>sysglobl</assembly>
        <assembly>CppCodeProvider</assembly>
        <assembly>Microsoft.*</assembly>
        <assembly>PresentationBuildTasks</assembly>
        <assembly>PresentationCore</assembly>
        <assembly>PresentationFramework.*</assembly>
        <assembly>ReachFramework</assembly>
        <assembly>System.*</assembly>
        <assembly>VSLangProj.*</assembly>
        <assembly>VSWebSite\.Interop</assembly>
        <assembly>VSWebSite\.Interop90</assembly>
        <assembly>SMDiagnostics</assembly>
        <assembly>UIAutomationClient</assembly>
        <assembly>UIAutomationClientsideProviders</assembly>
        <assembly>UIAutomationProvider</assembly>
        <assembly>UIAutomationTypes</assembly>
        <assembly>WindowsBase</assembly>
        <assembly>WindowsFormsIntegration</assembly>

        <assembly>nunit\..*</assembly>
        <assembly>nunit-.*</assembly>
        <assembly>xunit.*</assembly>
        <assembly>MbUnit.*</assembly>
        <assembly>MSTest.*</assembly>
        <assembly>log4net</assembly>
        <assembly>QuickGraph.*</assembly>
        <assembly>WiseOwl.Obfuscation</assembly>
        <assembly>Rhino.Mocks</assembly>
       <assembly>Gallio.*</assembly>
        <assembly>mbunit.console</assembly>
        <assembly>Gallio.Echo</assembly>
        <assembly>Gallio.Icarus</assembly>
    </ncover.systemAssemblies>
</configuration>

Aside: This list seems to contain some redundant or incorrect entries. For example, the old MbUnit v2 console application is called MbUnit.Cons, not MbUnit.Console.

Whoo!

Ok, so out of the box NCover will exclude from coverage analysis anything related to several popular testing packages. In principle, it's not a bad idea since excluding these assemblies is a common support request from users... but in practice this worries me because it can become overbroad and lead to user confusion (as it did for me). Gallio in particular is in a good position to take care of specifying its own necessary exclusions automatically (while taking plug-ins into account also).

To work around this issue, it suffices to edit the file. The list of system assemblies seems to override anything specified in the list of assembly inclusion arguments.

I will send the author a proposal to add a new switch to disable some of this behavior...

A Duty to Learn

2009-02-15T14:32:00.001-08:00

I was called to write this post after reading Jeff Atwood's The Ferengi Programmer and Rob Connery's apt response Patterns, Purists and Sinkholes. Their central argument is about whether the software development process can be codified.

Could there ever be a sufficiently precise set of rules to successfully build software by the book? Possibly.

Would it stifle the art? Certainly.

Are "principles and practices" useful? Yes! They provide scaffolding for reasoned analysis and discussion.

Are they universally good and complete? No. They are changeable embodiments of ideals.

Is it still worth learning and sharing principles and practices written down by others? Absolutely!

A Duty to Teach

What is a "best practice"?

First, let's start with the name. "Best" suggests that in some objective sense, these practices alone were judged to be absolutely superior to all others. Not merely are they good, they are superlative!

Second, the term is often applied to particular mechanisms used to achieve a particular policy. For example, "Thou shalt write application log files using protocol XYZ." There's more to learn about best practice than particular interfaces.

Let's call these things what they are instead: "standard practice".

Standard practice is a compromise by an organization to achieve a mutually agreeable implementation of an ideal principle. It is a tool to promote uniformity and consensus but it is no substitute for education. Standard practice should never be used to enforce unquestioned conformity and to stifle further debate.

Merely stating a standard practice does not free the speaker from the duty to teach!

An Unquestioning Mind Remains Unhired

Every professional software practitioner has a duty to learn. Those who shirk this duty eventually land in the pool of the millions of unemployed programmers who bang at the door of the software industry every day. They boast about their services yet fail time and again to invest their hearts and minds as barter for the money and recognition they feel is their due.

Let me share a secret...

If you say any of the following statements during an interview, I will probably not hire you. I don't care what your level of ability or experience is. These statements tell me you are on the path of eventual stagnation:

"I don't know anything about X but it's too complicated and I believe everything should be kept as simple as possible."

Translation: I am unwilling to learn on my own and I am unwilling to admit it might help my career.
"X is obviously better than Y." (I can't explain it but it is.)

Translation: I am unwilling to learn on my own and I cannot tell the difference.
"In all of the X years of my experience I have never had to do Y." (So you must be wrong about thinking it might be important.)

Translation: I am unwilling to learn on my own and I will drag everyone who disagrees with my dogma into the pit of irrelevance.
"I did X at all of my other jobs. X is the solution to all of your problems." (It slices, it dices.)

Translation: I am unwilling to learn on my own and I will not let go of my sledgehammer.
"All of my work conforms to industry best practices X, Y and Z." (But I can't explain them.)

Translation: I am unwilling to learn on my own and I like to impress people with buzzwords.
"I really want to work here but in two years do you think I can graduate to department X?" (X is so much cooler than Y even though I don't know X or Y very well, like development vs. testing.)

Translation: I am unwilling to learn on my own and I think what you do is beneath my status.

"I learned everything on my own. It's so easy and obvious. Schools and books are for losers."

Translation: I am unwilling to accept that others might still have things to teach me and I am stunting my own growth.
"I can learn any X. They're all the same." (Programming languages)

Translation: I am unwilling to accept that others might still have things to teach me and that there there are many ways of doing things; some vastly different from what I know.

I would rather endorse and train a candidate fresh out of school full of passion and vitality than a highly experienced person in denial of a closed mind.

A Broken Heart

A friend of mine recently told me this: "I am not sure whether I want to stay in software development much longer. I always have to learn new things each year and I feel I will never catch up. I think I should go into management or business intelligence since those fields stay the same all of the time and never change."

It broke my heart.

First, that he was planning to embark on a path of stagnation. Second, that he unfairly judged those in other disciplines as being somehow less sophisticated. Third, that he seemed to be making unhealthy decisions by default.

We talked for a long while after that.

Ultimately, I don't mind if he leaves software development (although I will be sad). I just hope he goes where his passion leads him. I will support him either way. Still, for me it was a worthwhile reminder that not everyone values their careers the same way...

Do want you want, but do it with all your heart and mind engaged. Learn and grow and change...

Edge Cases

2009-02-06T16:04:00.001-08:00

It's nice to pretend edge cases don't exist or aren't important, but they do and they are... Try as you might, someone will stumble upon it and the world will collapse under the weight of invalidated assumptions.

Here is one that I've been dealing with lately:

public class Container<T>
{
    public class Outer
    {
        public class Inner : Outer
        {
        }
    }
}

Quiz

1. What does typeof(Inner).GetGenericArguments().Length return?

2a. What does typeof(Inner).GetGenericArguments()[0].ToString() return?

2b. How about typeof(Inner).GetGenericArguments()[0].FullName?

3. What does typeof(Inner).GetGenericArguments()[0].DeclaringType.ToString() return?

3b. How about typeof(Inner).GetGenericArguments()[0].DeclaringType.FullName?

4a. What does typeof(Inner).BaseType.GetGenericArguments()[0].DeclaringType.ToString() return?

4b. How about typeof(Inner).BaseType.GetGenericArguments()[0].DeclaringType.FullName?

5a. What does typeof(Inner).BaseType.ToString() return?

5b. How about typeof(Inner).BaseType.FullName?

Answers

(Select text to reveal.)

1. 1

2a. "T"

2b. null

3a. "Container`1+Outer+Inner[T]"

3b. "Container`1+Outer+Inner"

4a. "Container`1+Outer+Inner[T]"

4b. "Container`1+Outer+Inner"

5a. "Container`1+Outer[T]"

5b. null

Surprise!

Gallio Update

2009-01-30T19:29:00.001-08:00

Work on Gallio v3.0.6 is progressing along after a brief holiday hiatus. Yes, I actually put the computers away for a couple of weeks to spend some quality time with friends, family and books (not to mention on other projects at work).

The plan is to ship v3.0.6 within the next couple of weeks.

Current primary objectives are:

Performance: improve start-up time, ReSharper integration, Visual Studio integration.
Extensibility: new plug-in model.
New features: XmlData, parallel tests, new contract verifiers, mixins (may be deferred).
UI affordances: simplify debugging, improve documentation.

And here's roughly where we're at:

Added [Parallelizable] attribute to run tests in parallel.
Added [XmlData] attribute to specify Xml data sources.
Global test execution timeout setting in Echo, MSBuild, NAnt and PowerShell.
Significant improvements to Contract Verifier syntax and functionality.
New developer content in the book.
Upgraded to Pex v0.9 Academic License.
Preliminary support for ReSharper 4.5 EAP.
Links to test source location from report.
Icarus can start tests with the debugger.
Icarus contains link to online documentation.
Started work to support MSTest 2005.
Added a report XSD file to assist contributors who wish to build report processors.
Started optimizing the reflection wrappers. Should improve ReSharper plugin performance slightly.
Added generic versions of Assert.Is[Not]AssignableFrom and Assert.Is[Not]InstanceOfType.
Started working on new plugin model.

Bug fixes:

Fixed MbUnit v2 adapter test outcome reporting and tear down glitches.
Fixed VB.Net project templates.
Fixed missing sample files.
Fixed XML encoding issues with actual / expected outcomes that caused whitespace differences to be hidden.
Fixed use of [Factory] data source with open generic types.
Fixed missing CCNet add-in documentation in the installer.
Fixed regression downloading report attachments in CCNet.
Fixed issue with Icarus go to source command invoked on non-UI thread.
Fixed inconsistent failure message for Assert.EndsWith.
No more spurious COMExceptions during startup when reading symbols from PDB files.

Announcing Gallio and MbUnit v3.0.5

2008-11-18T02:37:00.001-08:00

Today we are releasing Gallio and MbUnit v3.0.5. This is primarily a bug fix release but we are introducing a few new features.

Assert.AreApproximatelyEqual, test factories, structural object formatting, AutoCAD integration, x86 tests on x64, Visual Studio CTP 2010.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

Release notes: All release notes from all versions.

MbUnit

Assert.AreApproximatelyEqual

We have added Assert.AreApproximatelyEqual to express an assertion about equality within some delta (aka. tolerance). Likewise there is Assert.AreNotApproximatelyEqual.

Examples:

Assert.AreApproximatelyEqual(/*expectedValue*/ 2.0, /*actualValue*/ 2.2, /*delta*/ 0.3);
-> Success: 2.2 is in the range 2.0 +/- 0.3.

Assert.AreApproximatelyEqual(/*expectedValue*/ 2.0, /*actualValue*/ 2.2, /*delta*/ 0.1);
-> Failure: 2.2 is not in the range 2.0 +/- 0.1.

Assert.AreNotApproximatelyEqual(/*expectedValue*/ 2.0, /*actualValue*/ 2.2, /*delta*/ 0.1);
-> Success: 2.2 is not in the range 2.0 +/- 0.1.

The implementation supports any comparable type that defines a subtraction operator, and for which the result of subtraction is also comparable.

Examples:

Assert.AreApproximatelyEqual(-5, 2, 10);
-> Success: 2 is in the range -5 +/- 10.

Assert.AreApproximatelyEqual(new DateTime(2008, 11, 17), new DateTime(2008, 11, 18), TimeSpan.FromDays(2));
-> Success: 2008/11/18 is in the range 2008/11/17 +/- 1 day.

Static and Dynamic Test Factories (aka. Test Suites)

Some people were wondering where the MbUnit v2 [TestSuite] mechanism went. Now it's back with a few interesting twists.

A test factory in MbUnit v3 is a method that generates a test suite programmatically. They are typically used when the structure of a test is derived from custom metadata. For example, a test factory might produce a test suite generated from some information stored in the database.

A test suite produced by a test factory differs from a standard data-driven test (such as a Row-test) in that the tester has more control over the generated structure of the test suite. Instead of plugging in values to a parameterized test, whole new tests can be created arbitrarily.

There are two kinds of test factories.

A static test factory adds new tests to the static test tree. This makes the generated test suite part of the static test model that UI-based test runners such as Icarus use to present tests to the user.

Pros: The structure of the test suite is determined before tests actually run.

Cons: The structure of the test suite is fixed at test exploration time. Moreover, any data that was needed to produce the tests will be loaded all at once up front regardless of whether the test suite will actually be executed. Also, for this mechanism to work at all, the code that defines the factory must be executable at test exploration time. This is not the case in ReSharper or Visual Studio Test Tools so static test suites will not appear there, alas. (At least not as things are written today...)
A dynamic test factory adds new tests to the dynamic test tree, aka. the test step tree. The tests are not included in the static test model which makes it impossible for the user to pick a subset of the dynamic tests to run (basically they are all-or-nothing).

Pros: The structure of the test suite is determined when the fixture that contains it runs so it can vary based on input data determined at runtime. The test factory can be parameterized so it can produce multiple test suites with different input data. Likewise, a dynamic test factory can emit an unbounded number of tests. It can keep on producing tests to run for as long as it likes.

Cons: The dynamic tests are not visible to the user until test execution time.

It might help to think of a static test factory as working something like a [TestFixture] or [Test] where the test suite is constructed ahead of time using an algorithm of your choice. On the other hand, a dynamic test factory is more like TestStep.Run which is used to create dynamic test steps within a running test or like parameterized tests which pull contents from a data source at runtime.

Clear as mud huh? How about some examples.

Static Test Factory Example

In this example, we create a few silly tests statically in the CreateTests static method. Notice that it returns an enumeration of Test objects. It is also static because no instance of the fixture exists at the time this method is called. It follows that we cannot define a static test factory as a member of a generic test fixture because we won't know what values are intended to be plugged into the fixture's type parameters.

There are currently 3 kinds of built-in Test subclasses (you can make your own too):

A TestCase is a kind of Test that contains executable code to run. It can have a name, description and other custom metadata.
A TestSuite is another kind of Test that composes a list of other Tests (including other suites) to run. It can also have a name, description, and other custom metadata. In addition to its children, it can also include SetUp, TearDown, SuiteSetUp, and SuiteTearDown functions.
A TestFixtureReference is yet another kind of Test. We do not show it here but basically it allows you to create test suites that reference .

When we run the test fixture, it will include a test called "Sample Test" as well as a suite called "Sample Suite" with two silly test cases within.

Dynamic Test Factory Example

A dynamic test factory is quite similar to a static test factory. It just runs at runtime instead.

The first thing you will notice is that this dynamic test factory is not declared static. It can be if you like, but that's optional. The second thing is that this test factory can actually be parameterized!

Here we create 3 groups of dynamically generated test cases using different input data each time.

When we run the test fixture, it will produce 1 test case when n = 1, 4 test cases when n = 2 and 9 test cases when n = 3.

Of course this example is silly. I suppose next time I could write up something like a miniature file-based DSL integrated using test factories.

Structural Object Formatting

How often has this happened to you?

So you write this test...

And then it fails...

Ok, now actually it's pretty cool that it tells you that both values look the same when printed but are actually distinct instances, but that's not really good enough.

The problem is that the Data object does not provide a custom ToString() override. We could add one to that class, but maybe we can't or we don't want to.

Gallio provides a pluggable object formatter that MbUnit and other frameworks can use. It provides built-in support for primitives, arrays, lists, and many other common types.

Now we also provide a built-in structural formatter for objects. In the absence of a better formatting rule or a custom ToString() override, it uses reflection over the public properties and fields of an object to try to display its component parts. So we don't need to define useless ToString() methods for test diagnostics anymore.

Here is the result... can you tell what the problem is?

AutoCAD Integration

Mike Sandberg has added support for testing AutoCAD plugins.

It turns out that AutoCAD has a managed extensibility model so you can create your own plugins using .Net and the ObjectARX toolkit. Unfortunately it is somewhat difficult to write unit tests for plugins becuase they must run within the main UI thread of AutoCAD.

The AutoCAD integration for Gallio works by loading a shim into the AutoCAD application from which it can launch tests. To enable this integration, specify the "AutoCAD" runner type to the Echo, Icarus, MSBuild, NAnt or PowerShell runners.

For example:

Gallio.Echo.exe MyTestAssembly.dll /r:AutoCAD [other options...]

AutoCAD integration is not yet available from within the IDE. We will be working to improve this use case in the future.

Running x86 tests as 32-bit on x64

We now support running x86 tests within a 32-bit process on x64. This is necessary when the subject under test is linked to native 32-bit components since the 32-bit and 64-bit Application Binary Interfaces are incompatible.

To ensure that tests run in an x86 process, simply compile them for the x86 platform.

The only caveat is that this support might not work in embedded test runners such as TestDriven.Net, ReSharper or Visual Studio Test Tools because Gallio does not control process creation there.

However, it will work from the Icarus GUI, the build tasks and other tools as long as the tests are configured to run out of process (which is the default).

Visual Studio 2010 CTP Support

As mentioned previously, we are also shipping a preview of Gallio and MbUnit with support for .Net Framework 4.0 and Visual Studio 2010 CTP.

This is currently distributed as a separate download. It has only been lightly tested so please provide feedback.

External Tools Compatibility

We've updated integration with a variety of external tools.

CCNet v1.0.4
CSUnit v2.0.5
NUnit v2.4.8
Pex v0.8
ReSharper v3.1, v4.0, v4.1
TeamCity v4.0 EAP
xUnit.Net v1.0.3
Visual Studio 2005, 2008, 2010 (separate installer req'd for 2010)

Other Improvements and Bug Fixes

Improved the robustness of ReSharper, TestDriven.Net and Visual Studio Test Tools integration by loading Gallio in its own isolated AppDomain to prevent test assemblies from interfering with Gallio's runtime.
Added a "resident" test runner for TestDriven.Net to improve start-up performance. Use TestDriven.Net v2.17 or newer for this feature.
Fixed some issues relates to the setting of the application base directory and working directory. We were inadvertently attempting to create files within the Gallio installation folder at runtime which should not have been happening.
Cleaned up the documentation of MbUnit assertions and attributes. Repaired a few minor API inconsistencies along the way.
Added support for the ReSharper shadow copy and application base directory options.
Fixed a minor re-entrance issue when the Gallio MSBuild task was launched from within Visual Studio while other Gallio plug-ins for Visual Studio were already active.
Icarus tweaks for performance and robustness.
Added code to the isolated process hosting to interpret stack overflow related exit codes and log them.
Fixed a couple of bugs related logging the result of an asynchronous ThreadTask or ProcessTask spawned by a test using the Tasks abstraction.
Enhanced [ThreadedRepeat] and [Repeat] attributes to work on fixtures and to ensure that the setup/teardown runs.
Improved the reporting of xUnit.Net theories.
Fixed a bug that would sometimes prevent MbUnit v3 generic tests from running.
Fixed TeamCity reporting to ensure that it includes data-driven tests.
Fixed a bug in Assert.Throws that caused a StackOverflowException.
Fixed a bug printing the inner exceptions of MbUnit v2 tests.

Technorati Tags: gallio,mbunit

Don't change semantics of well-known types!

2008-11-18T01:23:00.001-08:00

I was writing a little quick and dirty PowerShell script to compare items in Visual Studio 2008 and Visual Studio 2010 project files side-by-side to ensure they are kept in sync.

Now I will admit that I am a total PowerShell newbie but I was not prepared to deal with this oddity:

$xml = New-Object System.Xml.XmlDocument
$xml.Load($path)
$nsmgr = New-Object System.Xml.XmlNamespaceManager $xml.psbase.NameTable

(...crash... cannot resolve the constructor for XmlNamespaceManager...)

See that little psbase? Well, because without it, PowerShell will apply some kind of built-in adapter semantics. Instead of giving back the NameTable, it will actually try to look for an element or attribute called "NameTable" and return that. Of course, since it doesn't find one, it returns Null instead!

Now I'm sure PowerShell provides some cool Xml slinging features I don't know about right now. However it's also a .Net language bound to the .Net framework. It just diverged pretty dramatically from my mental model of some pretty fundamental .Net framework types. It takes time to figure that out and recover...

This isn't like adding a fairly innocuous Ruby-ish to_s() method to types as an alias for ToString(). This is instead rather intrusive behavior that changes what member selection means for all XmlDocument objects. I think it would be better if PowerShell introduced its own Xml type instead. Then it could play with the semantics all it wanted while still preserving the expected behavior of the member selection operator.

Are there any other landmines I should beware of?

Speaking at QCon on Thursday

2008-11-17T08:18:00.001-08:00

This week I'll be at QCon speaking about Gallio and MbUnit in the ALT.Net track.

I've been looking forward to this for months! ;-)

C# Signatures

2008-11-06T19:55:00.001-08:00

On a thread in the Visual Studio 2010 C# and VB Feedback forum, DMeglio posted a question about whether C# 4.0 could be extended to support extended generic constraints.

I'm just curious, is the feature set for C# 4.0 locked? There is one thing that I feel is a glaring omission, and really hurts C#. The fact that generics do not support operator constraints. This is a huge issue for many of us. Why can't we do:

public class List<K> where operator+

I know this would make my life easier, I guess I'm just wondering if it's definitely been tabled or if it's still a possibility for 4.0

So I thought about this for a bit and proposed something similar to the C++ signature abstraction or a Haskell Type Class.

Nominative vs. Structural Typing

Consider the following types.

public class Cat
{
void Speak() { Console.WriteLine("Meow!"); }
}

public class Dog
{
void Speak() { Console.WriteLine("Woof!"); }
}

Well, they both have a Speak() method but they do not implement any common interfaces or inherit from a common type (besides Object). If we want to call Speak() polymorphically on Cat and Dog then we could modify them to implement an ISpeakable interface or Animal abstract base class.

That's because the CLR uses Nominative Typing. Two types are related only in the ways they explicitly declare as part of their definition. We take this for granted every day, but it's not the only way to do things. There's also Structural Typing.

The Cat and Dog classes obviously share structural similarities even if they don't declare them. They both have Speak methods...

C# Signature Proposal

Consider the following pseudo-code.

public signature Numeric<T>
{
static T operator+(T left, T right);
static T Zero { get; }
}

Any type can be said to satisfy this signature if it has the specified members. It need not actually declare that is satisfies the signature.

Assuming this notation, we can define methods like the following:

public static class Math
{
    public static T Sum(IEnumerable<T> values)
        where T : Numeric<T> // signifying conformance to a signature, NOT inheritance
    {
        T total = T.Zero; // the type must have a Zero property, per the signature
        foreach (T value in values)
            total += value; // valid because the type has an operator+, per the signature
        return total;
    }
}

Intuitively, the Sum method can be used on any type that supports all of the Numeric<T> operations.

Ok, but Int32 does not actually have operator+, or Zero defined on it. So those methods must come from somewhere else! Some of them could be defined as a default implementation provided by the signature but that's not enough.

C# Extension Properties and Static Methods

In C# 3.0, we got some fancy compiler syntax for extension methods. They are defined like this:

public static class EnumerableExtensions
{
    public static int Count<T>(this IEnumerable<T> values)
    {
        int count = 0;
        foreach (T value in values)
            count += 1;
        return count;
    }
}

This is all well and good but there is no syntax for extension properties and extension events, or for static extension methods, et. al.

It turns out that the current syntax for extension methods is very awkward to extend. Where will you insert this in those extension property and event declarations?

C# Signature Extension Proposal

Ah! But what if we allow signatures to extend one another or to extend existing types?

Then we can define new behaviors on those signatures that will apply to any types that satisfy the signature. If the signature extends another signature then it incorporates all parts of that other signature. Likewise if it extends a type then it takes on all aspects of that type.

But the compiler could also suppose that if a type conforms to a signature that is in scope, then all members of that signature can be used implicitly as if they were members of that type. Just like extension methods.

public signature Int32Extensions : Int32, Numeric<Int32>
{
    public static Int32 Zero
    {
        get { return 0; }
    }

    public static Int32 operator+ (Int32 a, Int32 b)
    {
        return a + b;
    }
}

Voila! Static extension methods and properties.

How about non-static extension methods and properties?

public signature Int32Extensions : Int32
{
    // I'm an extension property of Int32!
    public bool IsZero
    {
        return this == 0;
    }
}

Simplifications

If all we want is support for things like extension properties then it's clear that we can simplify the proposal greatly. Instead of defining something new like a signature we could define something like an extension class. However I feel that the structural typing ideas are quite useful on their own.

Gallio Store on Zazzle!

2008-11-05T16:27:00.001-08:00

Thanks to Mark Haley, our fantastic graphics designer, we are now offering a selection of Gallio branded products on Zazzle. Expect MbUnit branded products to arrive soon.

Proceeds from the store will be used to offset our hosting costs.

Full Text of the Announcement

You can now show your support for Gallio with merchandise bearing the Gallio Logo from a store we have set up on Zazzle.

At present there is a selection of T-Shirts, Polos, Hooded Sweatshirts, Hats, Mugs and Bumper Stickers available. This first run has one particular style, but other styles may follow. We intend to add MbUnit branded items in the future as well, so stay tuned!

A new menu item has been added to the gallio.org website that will direct you to the Zazzle site, or the direct link is here:
http://www.zazzle.com/gallio

Please note: most items can be customised extensively on the product page. For example, shirt colors or even style may be adjusted from the default style shown, so don't be afraid to experiment to get it just how you like it.

Technorati Tags: gallio,mbunit

Gallio and MbUnit v3 for Visual Studio 2010 CTP

2008-11-05T04:07:00.001-08:00

I suppose by now you have all downloaded the Visual Studio 2010 CTP. (Well, maybe not.)

But if you have and you are looking for a great unit testing framework for your first experiments with .Net Framework 4.0, I have published an early preview of Gallio and MbUnit v3 designed to work with the CTP.

Out of the box, it supports the VSTS test runner so you can run your tests using the Visual Studio Test View. Just be sure to create a new test project using the MbUnit v3 Test Project template.

Try it out!

Download: Gallio and MbUnit v3 for Visual Studio 2010 CTP

Screenshot

Technorati Tags: mbunit,gallio

See you at Microsoft PDC

2008-10-25T00:58:00.001-07:00

I will be at the Microsoft PDC.

If you're there then find me walking around wearing a kilt, as usual. I will also be a member of a lunchtime panel on the Future of Unit Testing, probably on Wednesday.

See you there!