ISerializable - Roy Osherove's Blog : Art Of Unit Testing

Minimizing unit Test Fragility – 8 features in Typemock Isolator to help

RoyOsherove — Thu, 09 Jul 2009 22:52:50 GMT

in continuation to my challenge (which no one had bothered answering, lazy web!)

One of the things that inhibits unit tests in organizations is the idea of fragile tests. a Fragile test is a test that can easily break when the production code changes. That’s not to say that tests should never break as you change production code, but there are ways to minimize this effect so that when tests do break, they break for the right reason.

the right reason is when a feature in production code isn’t working.

the wrong reasons can be many and varied. Of the most common ones is the idea of over specification in tests. The more your test expects of your code’s internal implementation, the more “specified” it is, the more “expectations” it has on the internals of the code. since internal code is more prone to change, so will the test break more often. so the situation can occur that the production code changed, but still gets the job done, but since it’s doing it differently internally, the test breaks even though the end result is still OK.

One of the main ideas in the new Isolator API was to reduce the fragility of tests by default. there are several features in Typemock Isolator that remove lots of fragility from tests by default, compared to other tools:

All Fakes are non strict by default.
Recursive Fakes
Method stubs ignore parameters by default
“From now on” Method behavior as a default
Isolate.Swap.AllInstances
Isolated.NonPublic APIs will not break when refactoring from private to public
Ignore one or all static methods on a type
True Properties

All Fakes are non strict by default.

when a fake object is strict, then it will throw an exception when someone calls one of its methods that wasn’t specifically “expected” by the test (or a different number of times than expected). since internal code can change and start interacting with other code in different ways, strict fakes will almost always fail your test on the lightest nuance of change. Most frameworks used to have strict fakes by default.

Isolator has all fakes non strict by default. in fact, there is no “strict” mode at all.

var fake = Isolate.Fake.Instance<SomeType>();

//code under test can now call any method without the test needing to specify it. the test only needs to specify behavior of methods it cares about.

Recursive Fakes

This is a feature first introduced by Isolator more than a couple of years ago and then adopted by Moq and Rhino Mocks. the idea is simple : allow chained calls in your code without the test needing to specify fakes at each hierarchy level. for example, calling MyFake.SomeProperty.DoSomething() will, by default, do nothing since its root is a fake object. the “SomeProperty” will be initialized by default to a fake object as well, recursively down the line.

With other frameworks this will only work with interfaces, methods which are virtual and classes that are non sealed. Isolator can make this work with anything, including static methods and non sealed classes.

var fake = Isolate.Fake.Instance<SomeType>();

//code under test can

fake.SomeProperty.DoSomething()

//or

var propObj = fake.SomeProperty;

propObj.DoSomething();

As your internal code is refactored, or changed to call a method down the hierarchy, this should not break a test (as long as the method does not matter to the test)

Method stubs ignore parameters by default

all other frameworks except Isolator will use the values being sent to methods of fake objects during the “setup” phase, as expectations of these actual values. Isolator by default ignores values being sent in, and just returns the fake result we tell it to, no matter what was being sent in. Of course, you *could* tell it to use the exact argument values, but that is not the default behavior.

other frameworks will require that you put in special constraints on each parameter value to say that you don’t care about the actual value.

var o = Isolate.Fake.Instance<SomeType>();

const int IGNORED_VALUE=10;

object IGNORED_OBJECT= null;

Isolate.WhenCalled(()=> o.SomeMethodWithParams(“ignored string”,IGNORED_VALUE,IGNORED_OBJECT) ).WillReturn(3);

this method will return 3 no matter what parameters are being sent to it. so internal code can change parameters slightly and it will still work.

by specifying only on specify param values, we could be over specifying the test.

“From now on” Method behavior as a default

with Isolator, once you set behavior on a method (“Ignore”, WillReturn(..), WillThrowException(..) ), it will use that as its default behavior for all subsequent calls. That way, if the internal code ends up calling a method more than once, the test should not care about it (as long as the result does not hurt the test). This is great for ignore calls to “Log” objects and such, where you don’t know or care how many times it will be called. you just want to ignore them without having to worry if the code will use them or not.

for example, the method from the code above will *always* return 3, no matter how many times it will be called.

if we specified it twice with different return values, it will return the *last* behavior as the default, after it was called once with the previous behavior.

Isolate.Swap.AllInstances

This beauty will replace all existing and future instances of a specific type in the code under test with the fake type. that way you don’t care how many times some object might be created, or how many instances of it are used. for “util” type objects, this makes things a breeze to setup for a test that could be very complicated or impossible in other frameworks.

as the internal code changes with usage of these types, the test will still pass.

var fake = Isolate.Fake.Instance<SomeType>();

Isolate.WhenCalled( ()=> fake.Foo() ).IgnoreCall();

Isolate.Swap.AllInstances<SomeType>().With(fake);

//now production code can do this:

var x = new SomeType();//<—this will behave like the fake object we created

Isolated.NonPublic APIs will not break when refactoring from private to public

Isolator allows setting method behavior on non public methods as well:

var x = new SomeObject();

Isolate.NonPublic.WhenCalled(x,”SomeMethod”).WillReturn(3);

The beauty is that if you do refactor the method to make it public, the test won’t nudge you or throw an exception. It will just work with public methods as well. You can then refactor the test without being a slave to it.

Ignore one or all static methods on a type

Isolate.Fake.StaticMethods<SomeType>() will ignore all void methods on that type, and return recursive fakes on all functions. That way, if you add more methods to that type, the test won’t care about it. great for utility style classes with lots of static methods.

True Properties

“True properties” is a feature that Isolator picked up from Rhino Mocks – you can set properties on a fake object as if they were real properties.

The nice thing is that if internal code then sets these properties to values you don’t care about, the test will work just fine. it’s also very readable.

var fake = Isolate.Fake.Instance<SomeType>();

fake.SomeProperty=3; //will behave like a real property and return 3;

Art of Unit Testing on Hanselminutes

RoyOsherove — Tue, 07 Jul 2009 12:28:40 GMT

Scott Hanselman took me in for a 30 minute interview about Unit Testing Dos and Don’ts in his podcast, Hanselminutes. It was a pleasure, and I hope to be there once more about other topics. Maybe in TechEd Berlin..

Have a listen

NDC 2009 – Done!

RoyOsherove — Mon, 22 Jun 2009 11:46:14 GMT

NDC 2009 was a blast!

Thanks for all the great conversations :)

Here’s what it looked like when you look up at a full room in NDC

Questions every team and dev lead should ask themselves

RoyOsherove — Tue, 16 Jun 2009 05:43:34 GMT

here are the questions that teams and team leads should be asking themselves on a daily\weekly basis.

There are more, but these are the basics, to me. It’s part of the summary for the talk “Beautiful teams I am giving at SEConf and NDC. we do a lot of this stuff over at work, and it’s proving itself on a daily basis.

Whole team

What can we automate?
where are we "Reinventing the wheel"?
what are the tools that slow us down?
what tools can we use better?
are there bugs that I could have found earlier? how do I make sure I find them earlier?
when do we find out we built the right thing?
when do we find out our code\design sucks? how can we make that earlier?
How do we show progress at the team level? at the management level?
How many meetings does each dev have every week? how can we remove them?
Are we building by feature or by layer?
can we make all our team sit in the same place?

Team Lead

daily: what bottlenecks exist in the team? what have I solved?
will my devs be better in a month or two than they were before? if not, how do I make that happen?
what prevents my devs from working? what am I doing about this?

Art Of Unit Testing available at Amazon

RoyOsherove — Sun, 07 Jun 2009 00:43:57 GMT

My book, The Art Of Unit Testing, is now in stock at Amazon. If you’ve read the book, I’d love it if you put in a review on that page.

Using Explicit Arrange,Act,Assert scopes in tests – thoughts?

RoyOsherove — Wed, 03 Jun 2009 00:55:11 GMT

What are your thoughts on this style of writing?

Specifically the “Explicit” way of defining arrange, act and assert scopes.
also, given the name of the test, where should the call to “OnApplyTemplate” be (if you’ve never seen the code)?
which version makes more sense to you? A, or B?

version A:

version B:

Testing that an event was raised

RoyOsherove — Sat, 23 May 2009 18:25:20 GMT

This question keeps coming up: “How can I test that an event was actually raised from my class under test?”

actually, there is an easy way to check if an event was raised.
you subscribe to the event in your test, and in the event handler you set a boolean flag to true. then you just assert on the flag.
here is a quick example:

Code:

public void Test()
{
bool wasRaised=false;
var button = new Button;
button.Click += ()=> wasRaised=true;
button.DoSomethingThatShouldHaveTriggeredTheEvent();
Assert.IsTrue(wasRaised);
}

If you feel less comfortable using lambdas:

public void Test()
{
bool wasRaised=false;
var button = new Button;
button.Click += delegate

{

wasRaised=true

};

button.DoSomethingThatShouldHaveTriggeredTheEvent();
Assert.IsTrue(wasRaised);
}

Unfortunately, with VB.NET’s current version, doing this in a single method is next to impossible, so you are forced to register to the event with a method at the class level, and check that:

Code (VB.NET):

dim wasRaised as Boolean=false;

public sub test()

wasRaised=false
dim button = new Button()
AddHandler( button.Click , AddressOf(OnClick))

button.DoSomethingThatShouldHaveTriggeredTheEvent()
Assert.IsTrue(wasRaised);

end sub

public sub OnClick(source as object,e as EventArgs)

wasRaised=true

end sub

Art Of Unit Testing (The Samurai Book)– Get it now, it’s done.

RoyOsherove — Wed, 20 May 2009 21:22:14 GMT

The time has actually come. After 2.5 years, and two kids, my book is finally done and is available in full form as an EBook. at the end of the month it will be in print form. Now would be the time to get it, when it is still at a “pre-order” pricing. Get the preorder price either at Manning(along with the EBook) or at the amazon page.

I love the cover image. How about we call this “The Samurai book” from now on?

This is the book that I wished I had when I started out writing my first tests, and that combines my knowledge about unit testing from the past 5-6 years or so working with companies on real projects (and real failures).

It contains things I have not seen in other places – writing readable, maintainable and trustworthy tests, as well as guidelines on how to review someone else’s tests and what to watch out for.

I was lucky to have the foreword written by no other than Michael-Legacy-Code-Feathers himself, and with some great quotes including one from Kent Beck about the book.

Here is the Table of Contents.

Free Chapters

The book comes with two free chapters:

Chapter 1 – The basics of unit testing(PDF)

Chapter 3 – Using Stubs to Break Dependencies (PDF)

Or get the full book

Book description:

Unit testing, done right, can mean the difference between a failed project and a successful one, between a maintainable code base and a code base that no one dares touch, and between getting home at 2 AM or getting home in time for dinner, even before a release deadline.

The Art of Unit Testing builds on top of what's already been written about this important topic. It guides you step by step from simple tests to tests that are maintainable, readable, and trustworthy. It covers advanced subjects like mocks, stubs, and frameworks such as Typemock Isolator and Rhino Mocks. And you'll learn about advanced test patterns and organization, working with legacy code and even untestable code. The book discusses tools you need when testing databases and other technologies. It's written for .NET developers but others will also benefit from this book.

What’s inside:

Test Review Guidelines
How to create readable, maintainable, trustworthy tests
Stubs, mock objects, and automated frameworks
Working with .NET tools, including NUnit, Rhino Mocks and Typemock Isolator

Test Review #3 – Unity

RoyOsherove — Mon, 23 Mar 2009 05:58:55 GMT

Watch previous videos:

In this video I go over the tests for Microsoft Unity Application Block.

Overall the quality of the tests in Unity is pretty good! I could certainly recommend that people look at them as examples of a bunch of tests against a framework, which are mostly very readable and maintainable.

Things I walk through:

Using [Ignore]
exploration testing
magic numbers and strings
un-needed asserts
Stub hard coded behavior
handling config files in tests
Asserts hidden in a utility method
more naming conventions
Separating integration tests
“smart” code that is less readable
Meaningless “isNotNull” tests
cleaner ways of expecting exceptions

Test Review #2 – ASP.NET MVC Unit Tests

RoyOsherove — Sun, 22 Mar 2009 03:42:38 GMT

See other reviews:

Review #1: NerdDinner

Here’s the second video review of Unit Tests. This is another one written by Microsoft – ASP.NET MVC (source).

First, it’s important to state how surprised I was by the high quality of the tests in MVC. The tests are readable, maintainable and trustworthy, with very little issues that I could find. whatever Issues I found are rather easy to fix. In any case, if one is looking for examples of systems written in what seems almost entirely in TDD, or at the minimum with very good testing guidance, ASP.NET MVC should be a good stop to look at.

Issues discussed in this video:

Implementing RowTest with MSTest, and the importance of naming (14:00)
Verify() that is splitted from the mock expectations (17:00)
some naming conventions
Over-specification in tests (mainly more than one mock object per test)
verifying mocks when it’s not required
logic inside tests (concatenation)
test factory methods with too much logic
a very good example of when multiple asserts is really bad (11:50)

Again – I’m very pleased with the test quality. Now is the time to make sure the things above are fixed. they are still important!

Test Review #1 - NerdDinner

RoyOsherove — Fri, 20 Mar 2009 11:45:53 GMT

I’ve decided to start doing some test reviews of tests that I see in the wild. I figured it’s the best way to show people what I mean when I say they do not have readable, maintainable or trustworthy tests.

The first episode is the review of the tests from the NerdDinner MVC source code. It’s 30 minutes long. and it was shot at 2AM, so I’m quite cranky. But the tests sure don’t try to ease my mind, and only make me crankier.

If you have tests you’d like me to review send an email to Roy AT osherove.com with the subject “Test Review [project name]”

problems that are dealt with in this video:

Naming conventions
Magic strings in tests
Hard coded values in hand written stubs
Magic assertions (expecting a value that no one understand why it should be that value)
Lack of tests

API design values and decision matrix

RoyOsherove — Tue, 17 Mar 2009 14:56:56 GMT

We’ve come to a nice way of deciding upon features in the Isolator API. This is what happens when your end product is an API for programmers.

First, we realized that we must argue about it. For each type of new feature in the isolator API, we put up on the white board several version of the API that we can’t seem to decide about, and then ask everyone on the team to say what they think. If everyone agrees, then we ask people to defend the opposite side and explain why an API is not good.

We came to several agreed upon values for the C# API, that help us judge the usability of it:

Consistency – is the new API consistent with the other APIs that already exist, or does it go against the regular way things are done?
Discoverability – If a user knew they wanted to do thing X, would they know to use the API without help? simply from intellisense?
Explicitness – we decided early on to be as explicit as possible about the API, so that the least guessing needs to take place by the user
Single point of entry – everything should start from a single point (Isolate.Something())
Readability for the reader, not the writer (if you didn’t write the test would you find it easy to understand what it does?)
Single way to achieve things – is there more than one way to do a task with the new API?
Backwards compatibility – do we break an existing feature and cause some heartache for users?

We measure the various versions of an API by putting it in a matrix and setting “V” or “X” on each of them (if we can’t agree it’s a V with a line on it).

then we have a better idea of what makes more sense.

I admit we still haven’t come up with a single set of values for the VB API, but I’ll try to define it here:

Does not require use of Action<T>
Consistent with the way a VBer would use other APIs
Explicit
Readable
Single way to achieve things
Backwards compatible

note that “single point of entry” is not included. That was a design decision we made early on.

Art of unit Testing goes to print in April

RoyOsherove — Mon, 16 Mar 2009 23:33:45 GMT

The book of never ending production is now actually near the end.

the projected print date of “Art Of Unit Testing” is now April 30 and I’m happy to see the Amazon page for the book already has one review of the early access version (though he totally destroys my spelling and book formatting abilities, he likes the content, so I’m happy)

Unit Testing in silverlight land with Typemock Isolator

RoyOsherove — Sat, 27 Dec 2008 21:54:07 GMT

I’ve been asked quite a lot recently whether one can write unit tests and isolate logical code that runs inside a silverlight application. Up until today my initial answer was ‘no’ because silverlight runs under different versions of mscorlib.dll.

however.

Today I actually gave it a try and realized that writing unit tests (not integration tests, as the silverlight test framework allows) against silverlight based code is possible and quite easy. Just like any other code that relies on a third party platform (like sharepoint code) the silverlight related code might have various dependencies.

I’m going to show how to use Typemock Isolator to overcome a couple of simple silverlight dependencies (using HtmlPage) and how to setup a test project against a silverlight project (with NUnit or MsTest)

Assumptions:

You have an open solution
the solution contains a silverlight class library or silverlight application project

To setup a test project against silverlight using MSTest:

Add a new test project to the solution
Remove all the existing classes (Database test, ordered test etc..) so that you are only left with the unit test class (UnitTest1).
Remove all useless comments and crud code from the test class so that you are only left with a test method (no comments, not even the TestContext)
Add a reference to the silverlight versions of “System.dll”, “System.Windows.dll” to the test project. (usually located under “C:\Program Files\Microsoft SDKs\Silverlight\v2.0\Reference Assemblies\” (remove existing reference to system.dll if you need to first)
Add a reference to the project under test
You can now write tests against the object model (standard classes)

To setup a test project against silverlight using NUnit or MbUnit:

Add a new class library to the solution as your test project
Add a reference to the silverlight versions of “System.dll”, “System.Windows.dll” to the test project. (usually located under “C:\Program Files\Microsoft SDKs\Silverlight\v2.0\Reference Assemblies\” (remove existing reference to system.dll if you need to first)
Add a reference to the project under test
You can now write tests against the object model (standard classes)

How to break the dependencies

Now, here is a simple example of code that has a silverlight dependency we’d like to test:

Let’s say we have a class in the silverlight project called ChatSession (I’m basing this on ScottGu’s Chat demo). but it’s constructor looks like this:

What if we wanted to control during our test whether the page is enabled or disabled?

Here’s one way to do it using Isolator:

In your test project add two references under the .NET tab: “Typemock Isolator” and “Typemock ArrangeActAssert”
Write the following test:

Using Isolate.WhenCalled() we are able to circumvent any method (static or not) to return whatever we want, or throw an exception.

Here’s a more interesting case. Let’s say we have a method that modifies the current page and shows some html to the user in a span tag:

here is one way to write a test that makes sure that the right text is set into the message element in the page, without needing to have a real page present:

There are several things to note here:

HtmlElement is Sealed has an internal constructor, but we can still create a fake instance of it using Isolator. None of the other frameworks can do this.
We are returning a fake object from a method call chain (HtmlPage.Document.GetElementById ) without needing to create a separate stub for document. None of the other frameworks can do this (especially since it is based on a static method call to begin with)
We assert in the end that the method was indeed called with the correct arguments without needing to change a single piece of code. Granted, I wouldn’t write code like this (I like decoupling!) but sometimes you just don’t have the ability to change existing code.

If you are developing an open source silverlight project, it is important to note that there is a free full version of isolator for open source projects with full functionality.

These are just the beginning of my journey into silverlight unit testing. I’m looking for good code examples that you might want to test, with various dependencies that need breaking. your comments are appreciated.

Isolator Feature Focus: Duck Typing and Isolate.Swap

RoyOsherove — Sun, 02 Nov 2008 12:25:00 GMT

Typemock Isolator 5.1.1 has been released, and this release brings with it some awesome (seriously) features that are unique fro any other framework I've seen.

a good overview of features can be found here.

The Isolator Swap feature allows swapping calls between real and fake objects (kind like redirects) so that any relevant calls made against the real object will be redirected and invoked on the fake object.

Here's how you use it:

Unlike doing a standard "WhenCalled()" on some object method and telling what its custom behavior will be, "Swapping" objects redirects all relevant calls(I'll explain what "relevant" means in a second) to the fake object which you have created.

Duck Typing Awesomeness

Another cool thing about it is that "Duck" and "Dog" don't have to have a shared interface or base class. The "Swap" feature will redirect a method call if it exists on the "fake" object, but if it does not, it will invoke the original object. This is one interpretation of what's called "Duck Typing".

The "CallsOn" and "WithCallsTo" methods take an object type, so you can send in anything you want. if a method on the fake object matches the signature and name of a called method on the real object, the fake method will be invoked.