Team Foundation Server has a great story when it comes to integrating several development practices into a single data warehouse for running reports on, and out of the box (at least) the Developer and Test editions of Visual Studio integrates into this warehouse with MS’s own unit testing framework, MSTest.
We played around a bit with MSTest a while ago in a couple of projects I’m on at work, but came to the conclusion that it fell short of MbUnit which provides a much better story when it comes with integrating with other test runner tools (TestDriven.Net / ReSharper) which I think is far superior to the one in Visual Studio and also has a larger arsenal of extensions and out-of-the-box asserts.
During the time we have been using MbUnit we have always been using the HTML reports which MbUnit provides and adding a link to this report in the build report provided by Team Build. The ability to have the test results in TFS have been something we have wanted the whole time though, and lately I found a project called NUnit for Team Build on CodePlex. This triggered me to try NUnit with a subset of our tests to try to get this thing running and this solution provides exactly what we have wanted.
There wasn’t much hassle getting the unit tests converted from MbUnit to NUnit, as it mostly consisted of replacing the using statements to point to NUnit.Framework instead of MbUnit.Framework and then rename/edit some of the asserts.
What to do
NUnit for Team Build doesn’t perform any kind of voodoo magic here. The main idea of this project is to simply use XSLT to transform the XML test results from NUnit to the XML format which MSTest generates and then trigger the MSTest executable to publish these results to TFS. What NUnit for Team Build provides is the XSLT document to be used for the transformation, and as such it should be feasible to create your own XSLT document to enable other unit testing frameworks to publish their results to TFS.
The actions necessary to set this up in Team Build is already described on the project site for NUnit for Team Build so I don’t want to repeat that here, but you can find it here.
I did have to alter their XSLT document a bit to make it work though. Specifically:
- Had to modify the transformation of the paths to the test assemblies
- The error message NUnit adds when expected values does not match contains a line break so the expected value and real value is displayed on separate lines. I wanted them on a single line so that error messages for these cases where displayed on a single line which fits the report view in Visual Studio better.
- Had to add a error message if tests are not executed for some reason (we had some tests which for some reason where inside private text fixtures, which caused them to not be executed)
- Added support for the description attribute so that test descriptions are included in the TFS results
The first item was easy enough. I just compared where the XSLT document expected to find items inside the NUnit XML files and modified them.
For the error messages modifications I looked in the <Output> element in the XSLT document. The original transformation is like this:
<xsl:for-each select="./failure">
<ErrorInfo>
<Message>
<xsl:value-of select="./message"/>
</Message>
<StackTrace>
<xsl:value-of select="./stack-trace"/>
</StackTrace>
</ErrorInfo>
</xsl:for-each>
I added a simple conditional test on the message and if it contains the text ‘Expected’, then simply remove line breaks in the message:
<xsl:for-each select="./failure">
<ErrorInfo>
<Message>
<xsl:choose>
<xsl:when test="contains(./message, 'Expected')">
<xsl:value-of select="translate(./message, '
', '')"/>
</xsl:when>
<xsl:otherwise>
<xsl:value-of select="./message"/>
</xsl:otherwise>
</xsl:choose>
</Message>
<StackTrace>
<xsl:value-of select="./stack-trace"/>
</StackTrace>
</ErrorInfo>
</xsl:for-each>
Adding an error message for not executed tests where also pretty trivial. I added the following inside the same <Output> element, below the above failure message handling:
<xsl:for-each select="./reason">
<ErrorInfo>
<Message>
<xsl:value-of select="translate(./message, '
', '')"/>
</Message>
<StackTrace></StackTrace>
</ErrorInfo>
</xsl:for-each>
My last modification is that I added support for the description field. This was done by adding the following inside the <UnitTest> element, just before the <Css> element:
<Description><xsl:value-of select="@description"/></Description>
Some afterthoughts
There is a couple of things which I still want to update in the XSLT document though, but which I haven’t had time to look at yet:
- The Test Results window in Visual Studio does not contain any information in the Project column in the test lists. This won’t be a problem as long as all tests is inside the same assembly, but if you have tests in multiple assemblies, then it would be nice to see where tests are located in this view.
- All dates and time data is hardcoded in the XSLT document and the transformation of the duration times seems to be weak. I would like to add some better transforms for this, but this is also something which I haven’t got to yet.
If anyone have some ideas for handling these two issues, then I would love to see them.
Note: I have also seen some references to NUnitForVS, where NUnit tests are executed by the test runner in Visual Studio:
This add-in for Visual Studio Team System will allow you to run NUnit tests just like Visual Studio Unit tests integrated in Visual Studio. It makes Visual Studio to see NUnit test as mstest. That means everything you can do with Visual Studio Unit tests you can also do with NUnit tests. For example publishing of results in TFS or code coverage.
This should be a better solution than the one I have described in this post, simply because it adds code coverage too. I plan to take a look at this one in the near future, but if anyone already has done so and has some feedback, please add them in the comments 
]]>
