Scott MacLellan

The Worst Week of My Life

2016-06-07T13:55:00-04:00

In January I had the worst week of my life. My wife and I joked we wanted to start 2016 in February. Within a single week I lost my job and we had a miscarriage. In this post I want to tell you about this traumatic experience, how this massive change has turned out positive and what I learned from it.

Before

I started this year with all the optimism in the world. Throughout the holidays leading up to 2016, I tried to refocus myself. The year started with new habits and influences which improved my outlook on life. I was determined to be more positive and work hard at doing a great job.

This was a stark contrast to the previous several months which were filled with ups and downs.

My time with the Exterminators was a blast. I felt re-energized and excited about software development again. While with the team I found new growth areas and doubled down on existing skills. This left me wanting more.

Returning to my previous team was jarring. When I left I was excited about a new project which would have a huge impact. Incorrectly, I thought I would get to dive into it right away. After I had started prototyping portions of it, the whole project was given to another team.

Instead, I was relegated to ongoing maintenance on a different project. My learning slowed. Shortly afterwards our team split in two. Another project I was excited about went with the other half of the team. I felt like I had been punted to the B team. My heart was not in my work. Focus and quality suffered.

In the months leading up to January I was contemplating a larger change. The essential question was whether I could still have an impact in my job or company. Getting things done was frustrating and I did not deal with this well. Over time my morale eroded further as I considered greener pastures elsewhere.

However, during a company-wide developer conference and the holidays I started to see life differently. In my role I might have plateaued, but I realized D2L as a whole still has many great things to offer. I worked with phenomenal people like Craig and Daryl whom I could still learn from. D2L operates on a large scale and have different challenges which are exciting. Lastly, I fully believe in D2L’s mission of transforming learning.

A fantastic gift from the D2L developer conference celebrating my 5 years there.

In January I hit the ground running. I reset my perspective. I was excited again. Each task I worked on received my complete attention. I was crushing it again. Coming to work each day was fantastic. We had great projects lined up and we were making a difference.

Terminus

The company mentioned there might be some changes come later in the month. In the past our team had been completely immune. After all, our team was doing critical work which could dramatically improve how the company operates. I considered myself an important contributing member of team and in my hubris didn’t think anything of it.

On that fateful January 26th, my manager who seemed a little choked up came and got me at my desk. This was not like him. I felt a pang of fear, then it happened. My time at D2L was finished. I was given the paper work, told the terms and then escorted to a conveniently located exit nearby.

I was absolutely stunned. It was difficult to speak. On my way out I bumped into one of my mentors and could barely tell them what happened. It took everything to stop myself from crying.

We had planned our life out around my job. This was one thing I thought I could depend on. I had worked there for over 5.5 years. In the previous few years we bought a house and had our son, Jude. How were we going to afford everything? Even in our wildest contingencies we were not ready for me to lose my job. The news was overwhelming.

Then the second bombshell went off. My wife was expecting our second child. She was having signs things were not right. The day after I was let go we lost the baby. Even months later, I am immediately brought back to the devastation we felt. All we could do was hold each other and cry. As I write this I am crying again as the memories come flooding back.

A commemorative necklace my wife wears for the miscarriage.

This was the worst week of my life.

Doubt

You are your own worst enemy. I spent too much of the next few months rethinking everything. Was there something I did? Something I didn’t do? Why was I chosen? Not knowing why is perhaps the hardest part.

Even now I doubt myself. Have I been a phony for all these years? Am I a horrible programmer or worse? There is an interesting cognitive bias known as the Dunning-Kruger effect where the truly incompetent cannot possibly know how bad they are. They lack the meta-thinking to understand their own faults. In short, you can be so bad you don’t realize it. Is that me?

I talked a little bit about my self-doubts in my post ‘Are you your code review?’. They have always been with me. This event fanned the flames of those doubts even more.

Leading up to the exit interview I wrote lists of things which I thought might have led to this outcome. Talking to my former boss helped clear up my lists. He highlighted areas for improvement and made me feel like I was not a failure.

Getting past my self-doubt and focusing on moving forward was the first step. This easily becomes a dark line of thinking which only leads to self-destruction.

Acceptance

Our family took the first week to recover and get back to normal. There was no changing what had happened. Daily life was permanently altered. We needed to accept this new reality and move forward.

Oh Josh, you are so silly.

During those first few days the weirdest thing that I needed to accept was my coworkers were suddenly not there. Waking up every morning I was confronted with the fact that I would not be seeing Travis and Josh today. Many of my coworkers had grown into close friends on whom I rely a great deal. I was always happy to see them again or beat them at foosball. Our relationship would need to change if it was to survive.

Another odd thought experiment was the other choices D2L could have made. Was I chosen instead of other people? Would me staying mean someone else leaving? In the end I felt it was better that I was chosen instead of someone else. Better I go through this than one of my friends. After our life had normalized a bit, I was optimistic we would be okay and didn’t want them to have the same burden.

Although being asked to leave was not great, it did open the door for new opportunities. With our last several projects I had been dipping my toes into different technologies and was really enjoying it. Now I had the chance to make a bigger shift in my career.

Searching

Getting a job in the past was a fairly straight forward process:

Write a resume
Find jobs
Write cover letters
Apply
Interview
Repeat or accept the offer

This time around everything happened at the same time and looked like:

Network, find jobs and apply all at once
Interview
Review and accept offers

During this job hunt I learned how invaluable your connections are in finding new jobs. Over the years I had worked with many different people who had moved on from D2L to other companies. I had great role models at D2L who were willing to be references.

Thanks to my network, finding potential jobs changed dramatically. The first step in my search was to talk to my connections. Right away I was introduced to potential companies. I went for coffee to learn about open positions. Casual meetings turned into interviews.

Suddenly leaving a company does wonders for your LinkedIn profile.

Finding potential jobs in Kitchener-Waterloo is also extremely easy thanks to services through Communitech. They run an active job board which has hundreds of fantastic postings available. I found many interesting companies with a diverse range of sizes and needs.

A number of recruiters reached out with various opportunities. While I did not use their services, they showed me positions typically outside of my existing network. They helped restore my confidence that I would be able to find a new job and move past my doubts.

Resumes and cover letters are a must. Within the first week I had mine updated and was ready to share it with the world. It slowly got better over time with more and more feedback. Luckily, I had updated it sporadically with different achievements and milestones.

Interviews are intimidating. It had been over 5 years since I last interviewed at all. Co-op terms at university gave me a lot of practice, but without using those skills for years I found I was very rusty. My friends helped me through mock interviews to get me ready. Even still, I bombed a few of them which was tough.

Regardless of how an interview went I always sent a follow up email to:

Thank the interviewer
Emphasize what went well
Address any hiccups
Reiterate why I would be a great fit

Due to my background I am less good at algorithms and get caught up on some standard interview questions. I spent a lot of time practicing interview problems and reading up on algorithms. When that wasn’t enough I tried learning Ruby and refreshing my knowledge of JavaScript frameworks/idioms. After coming home from interviews, I implemented the programming questions I was given to double check my solutions.

Success

Things started slowly then picked up pace. At first I just had coffee with former colleagues/mentors. Interviews started to trickle in and then really accelerated. During the last week I had a flurry of final interviews leading up to a difficult choice.

In the final stages I was fortunate to have several offers from great companies. Each presented different opportunities and challenges. I don’t think I could have gone wrong choosing from any of the options. Other interviews were not successful and I will need to try again another time.

Over the weekend I made my decision. Due to the timing I had another week off to relax and decompress before starting work. The end was in sight after the harrowing month which preceded it.

On March 7^th I started working at Vidyard full time! Even before the first day I was blown away by the amazing culture. They were so welcoming and helped me get connected right away. On my second day I pushed code to production! I am thrilled to be part of the team.

Lessons

What may have been the worst week of my life at the time, will now be the start of a bigger change in my life. This entire experience has taught me so much and helped me understand what matters most.

At D2L, I poured a lot of time, effort and self-worth into my job, only to see it come to an end. I was slowly learning work should not be my #1 priority. This experience brought that realization painfully to the forefront.

These are the people I care about most.

What really matters is our relationships and loved ones. The quiet moments as a family during the first week will stay with me forever. I cannot thank enough the many friends and family members who supported us through this challenging period. We are so fortunate to have them in our lives.

We needed to accept what had happened to move on. Hopefully some day we will have other children. I found another job. We can learn from our past, but should not dwell on it.

Interviewing is a skill you need to keep practicing. From now on I plan on regularly interviewing and updating my resume so I will not become rusty again. The goal is not to job hop every year. Instead, I want to understand what people are looking for and be able to confidently present myself.

I was due for a change and believe I have found what I needed. While my time at D2L was a great learning experience, I was restless and wanted something else. During several low points while at D2L I had felt like leaving. I was never able to make the choice on my own and now the choice was made for me.

Epilogue

Why write this post now? I am finishing my probation and feeling more comfortable at Vidyard. After the event I was so shaken up I decided to put the blog on hold. Starting a new job I wanted to make sure my first 3 months were solid.

Well, mostly solid. Almost immediately after starting I was sick for a full 3 weeks. In the first month I worked from a bed at home more than I did at the office. Although I was sweating bullets while coughing up lung chunks, my manager was extremely supportive. My coworkers all told me not to worry and how later I could look back and laugh about it. I had their trust to do what I needed to do and the space to get better. The entire experience was yet another testament to why Vidyard is awesome.

Now, I have shipped a few features and am working on something bigger. It has been a blast. I feel like I have learned so much in very little time. Hopefully, in the coming weeks I will be able to share what I have learned with you.

Thanks to my former co-worker Josh for helping me with the grammars. Sorry for turning down your edit about who wins our foosball. Thanks for the many years we spent together at D2L. I miss you canoe buddy.

I would like to thank my lovely wife Angela for helping review this post. I would be lost without you. I love you.

Vanilla JS Tetris - Good Luck, Have Fun

2016-01-31T07:05:07-05:00

It has been quite the week. Sometimes you need to just relax. Try this simple Tetris clone made with only the best Vanilla JS. Good luck, have fun.

How to Self-Host Nancy Without Locking Your DLLs: Shadow Copying

2016-01-12T17:11:07-05:00

This is in response to a GitHub issue for Nancy. The user is trying to self-host Nancy without locking their DLLs. One easy way to do this is to create a wrapper program which runs the actual program with shadow copying assemblies. No DLLs are locked and the changes are minimal.

I will first show a simple application which self-hosts Nancy and can serve a request to "/". The program will start listening, wait for a key to be pressed and then exit. This is the code:

using System;
using Nancy;
using Nancy.Hosting.Self;

class Program {
    static void Main( string[] args ) {

        var url = new Uri( "http://localhost:12345" );
        using ( var host = new NancyHost( new DefaultNancyBootstrapper(), url ) ) {
            host.Start();

            Console.WriteLine( "Now listening, have fun!" );

            Console.ReadLine();
        }

    }
}

We will add a simple NancyModule so we can test the application at http://localhost:12345/:

using System;
using Nancy;

public class HelloWorldService : NancyModule {
    public HelloWorldService() {

        Get["/"] = x => {
            return Response.AsText( "Hello World" );
        };

    }
}

This program will work as is. The problem is that it locks any assemblies it references.

To work around this problem, add another executable to wrap the actual implementation. In order to keep the DLLs unlocked, we will run the implementation from a separate AppDomain with Shadow Copying enabled.

AppDomains are a very powerful feature of the Common Language Runtime for isolating code. They can use different security contexts, modify how assemblies are loaded and can be managed independently. There can be multiple AppDomains within a single process and can achieve some of the same isolation benefits as processes.

Using the separate AppDomain allows us to set the ShadowCopyFiles option to "true". This option will cause the assembly loading process to copy each assembly into a different directory and then load them from the new location. The local copies are left unlocked. For more information on Shadow Copying Assemblies refer to MSDN.

The whole solution would look like the diagram below:

This is the wrapper program to call the actual executable Implementation.exe:

using System;

class Program {
    static int Main( string[] args ) {

        AppDomainSetup setup = new AppDomainSetup {
            ShadowCopyFiles = "true" // This is key
        };

        var domain = AppDomain.CreateDomain( "Real AppDomain", null, setup );

        // Execute your real application in the new app domain
        int result = domain.ExecuteAssembly(
            "Implementation.exe",
            args
        );

        return result;

    }
}

That is all there is to it. Don’t want your DLLs to be locked? The easy solution is to use another AppDomain with Shadow Copying enabled.

All the code for this blog post can be found in this sample project.

Introduction to Dependency Injection

2016-01-08T05:33:07-05:00

I had the privilege of mentoring several co-workers in 2015. One of the topics they found confusing was Dependency Injection. We use it everywhere. To them it felt like magic. The code just fits together through mystical Containers. In this post we will break down the powerful concepts surrounding Dependency Injection.

Let’s start with some simple classes:

public class Foo {
    public void Hello( string message ) {
        Console.WriteLine( "Hello {0}", message );
    }
}

public class Bar {
    Foo m_foo;

    public Bar() {
        m_foo = new Foo();
    }

    public void Example() {
        m_foo.Hello( "World" );
    }
}

These are all concrete classes. No fancy dependency magic here. What this code does is very clear. Bar creates a Foo then uses it to print Hello World.

A complete program using this code is also straightforward:

public class Program {
    static void Main() {
        Bar bar = new Bar();

        bar.Example();
    }
}

Create a new Bar then call Example. Hello World!

Dependency Inversion Principle Applied

Let’s spice things up! Instead of creating the Foo in Bar’s constructor we can pass it in. Better yet, we can switch to an interface with all the same methods as Foo.

public interface IFoo {
    void Hello( string message );
}

public class Foo : IFoo {
    public void Hello( string message ) {
        Console.WriteLine( "Hello {0}", message );
    }
}

public class Bar {
    IFoo m_foo;

    public Bar( IFoo foo ) {
        m_foo = foo;
    }

    public void Example() {
        m_foo.Hello( "World" );
    }
}

So what have we gained? Well, the low level Bar class no longer knows anything about the IFoo implementation it is using. That is now up to callers using Bar. We have switched from a concrete implementation to a higher level abstraction. This decouples the code making it easier to maintain.

We have applied the Dependency Inversion Principle ¹. As defined by Robert Martin it is:

A. High level modules should not depend upon low level modules. Both should depend upon abstractions.

B. Abstractions should not depend upon details. Details should depend upon abstractions.

The abstraction could be anything. Typically it will be an interface, but can also be a base class, delegate or another abstraction. The key is shifting the code from the implementation to a higher level.

What About Dependency Injection?

Another concept closely related to Dependency Inversion is Dependency Injection. In fact, we used it without really knowing it. Where Dependency Inversion was all about the abstractions and layers, Dependency Injection is all about how dependencies are provided.

Don’t worry, it is a really simple idea. Here is the demystified definition by James Shore:

Dependency injection means giving an object its instance variables.

Literally, injecting dependencies into a class. In our previous example, we injected our dependencies using constructor parameters. This is the most common approach, but you can also inject dependencies using properties or specialized methods.

Look look at the program from the last example:

public class Program {
    static void Main() {
        IFoo foo = new Foo();

        // Boom, IFoo injected!
        Bar bar = new Bar( foo );

        bar.Example();
    }
}

We inject the IFoo into the constructor of the Bar. The program can now choose which IFoo to use. This is more flexible than when the choice was buried in the Bar class.

The Benefits

We have inverted our dependencies and injected them into our classes. This is fantastic! Our code is nicely decoupled. We can easily change what is injected for testing or introducing new features.

Implementations are hidden behind abstractions and can be easily replaced. Want an IFoo which writes out to files? No problem. You can change the code to your new FileFoo without ever modifying Bar.

The original Bar is impossible to test in isolation. The direct dependency on Foo forces the two classes to be tested together. Changes to Foo could break the tests for Bar.

By injecting the dependencies, we can use a fake IFoo in tests to do whatever we want. This is a great way to set up specific scenarios and/or avoid external systems (i.e. databases or services).

Dependencies For Everyone!

Creating classes is more challenging when you use Dependency Injection and Dependency Inversion frequently.

We have moved where dependencies are created. This poses a problem for consumers of the original classes. They must both choose what to inject and create all the dependencies. I mean ALL the dependencies.

Your dependencies start to have their own dependencies. While this is not too bad with a few dependencies, once you get into chains of dependencies it gets nasty.

Think about it. One class has a dependency, the dependency has more dependencies and those dependencies have their own dependencies. This is the tip of the iceberg:

SqlConnection connection = new SqlConnection( "connection string" );

FooRepository repository = new FooRepository( connection );

Logger logger = new ConsoleLogger();

BazService controller = new BazService( repository, logger );

If every class repeated setups like this it would be a big problem. Creating anything would be a nightmare. Thankfully there is a better solution, Dependency Injection Containers.

Dependency Injection Containers

With all this dependency madness we need to find a better way to create classes. Don’t worry! There are fantastic libraries to address this problem. They are commonly referred to as Dependency Injection Containers or Containers for short.

Containers contain and seamlessly connect all of your dependencies. Within your application they are used to instantiate dependencies they know about.

Before we get to the real thing I want to walk you through a mental models for Containers.

Externally they are like one massive Factory for any type
Internally they are like a Dictionary of Factories

Enter the Factories

A Factory is a common creational pattern. They allow you to abstract what is being created and how it is created. Dependency Injection Containers behave like Super Factories which can create any type they know about.

Want an IFoo? Use the FooFactory!

public class FooFactory {
    public IFoo Create() {
        return new ConsoleFoo();
    }
}

internal class ConsoleFoo : IFoo {
    public void Hello( string message ) {
        Console.WriteLine( "Hello {0}", message );
    }
}

The Factory can be used any time an IFoo is needed without any knowledge of which IFoo is created. We could easily update the FooFactory to create a FileFoo.

public class FooFactory {
    public IFoo Create() {
        return new FileFoo();
    }
}

internal class FileFoo : IFoo {
    public void Hello( string message ) {
        File.AppendAllText( "C:\\foo.txt", "Hola " + message );
    }
}

Factories can partially contain the sprawling dependencies. The more dependencies you have the more factories you will need. Factories will need to call other factories to create nested dependencies. This can get complicated when many dependencies are needed. The extra classes and glue code are tedious to maintain.

Prior to using IFoo we need to create one using the factory:

public class Program {
    static void Main() {
        FooFactory factory = new FooFactory();
        IFoo foo = factory.Create();

        Bar bar = new Bar( foo );

        bar.Example();
    }
}

Having to use the factory everywhere is not fun. We can do better.

Poor Man’s Dependency Injection Container

What if we could use a single class to get any dependency we wanted? We could use a Dictionary of Factories! The Dictionary would be keyed on types where the values would define how to create their respective types. We could then create any class the Dictionary knows about.

In this section, we are going to create a really simple class to do just that. Our very own simple Dependency Injection Container.

Question: Why is it called a “Container”? It will contain all our application’s dependencies. When our application starts we will give it all the dependencies we want to create and classes we want to inject the dependencies into.

The Container needs to:

Resolve types our application needs
Register types our application provides

Enough with the words! Onto the code!

public interface ISimpleContainer {
    public T Resolve<T>();

    public void Register<T>( Func<T> factory );
}

Not bad. One method to Register dependencies and another to Resolve them. The methods line up with our “Factory for any type” and “Dictionary of Factories” mental models. We accept Func<T>’s as simple factories for each type. Once all the types have been registered, Resolve behaves like a Factory for any type.

Let’s implement our simple version:

public class SimpleContainer : ISimpleContainer {
    private readonly Dictionary<Type, Delegate> m_registrations =
        new Dictionary<Type, Delegate>();

    public T Resolve<T>() {
        Func<T> factory = (Func<T>)m_registrations[typeof(T)];
        return factory();
    }

    public void Register<T>( Func<T> factory ) {
        m_registrations.Add( typeof(T), factory );
    }
}

Internally we have the “Dictionary of Factories”. Resolve uses the Func<T>’s we registered.

Using the Container is easy. In the next example, we register all the types (lines 5-11) then resolve them (lines 8 and 13). Once we have resolved Bar we can use it normally!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
public class Program {
    static void Main() {
        SimpleContainer container = new SimpleContainer();

        container.Register<IFoo>( () => new ConsoleFoo() );
        container.Register<Bar>(
            () => {
                IFoo foo = container.Resolve<IFoo>();
                return new Bar( foo );
            }
        );

        Bar bar = container.Resolve<Bar>();

        bar.Example();
    }
}

Cool. We have a Container. I would not use it in a real project. It is awkward to use. The classes need to be wired together manually. We had to tell it exactly how to create a Bar even though it already knew how to create an IFoo.

This is where real Dependency Injection Containers are fantastic. They solve the wiring problem and automatically inject dependencies they know about. We can use the Container like glue to bind everything together.

Abstractions are registered as concrete types. Consumers can resolve and use those abstractions directly. They have no knowledge of the concrete types being injected. Instead they rely on the Container. This preserves the Dependency Inversion principle and helps decouple our code.

Awesome Containers

Thankfully, there are many great open source Dependency Injection Containers. The following three are my favourites. Our team has used them on different projects.

Autofac

Autofac is new, super clean and powerful. The registration API is fun! They also have great support for controlling lifetimes/scoping and cleaning up for you. This would be my first choice when starting a new application.

StructureMap

StructureMap is battle hardened having been the original .NET Dependency Injection Container. The latest version of StructureMap was a massive step forward. The authors incoperated many improvements they learned from 10 years of supporting the project. A great choice you should definitely check out.

TinyIoC via Nancy

Lastly, we use Nancy a lot! For the simpler applications, we exclusively use the built-in TinyIoC. It is simpler than the other Containers and is missing some advanced options. We periodically consider switching to one of the other libraries for these features which we believe would simplify our configuration.

More Out of the Box

These libraries greatly enhance how you register and resolve components. Often these capabilities are connected; features used when registering define how objects are resolved.

All the Containers can wire together classes based on what they need injected. You could register Bar and when it is resolved the Container would automatically inject an IFoo based on what was registered for IFoo.

Here is an example of our application using Autofac:

using System;
using Autofac;

public class Program {
    static void Main() {
        ContainerBuilder builder = new ContainerBuilder();

        builder.RegisterType<ConsoleFoo>().As<IFoo>();
        builder.RegisterType<Bar>().AsSelf();

        IContainer container = builder.Build();

        Bar bar = container.Resolve<Bar>();

        bar.Example();
    }
}

It does the right thing and gives Bar the registered ConsoleFoo.

Many Containers have shortcuts for simple transformations, i.e. from T to Lazy<T>. Containers often support resolving/registering open generic types.

Containers can offer the ability to register sets of dependencies in Modules or Registries. This provides a simple way to group registrations together or split them apart. For example, you could register all database related classes in one module separate from your logging module.

Most Containers provide mechanisms for registering your types based on conventions so you do not need to configure everything by hand. You can register all classes implementing a similar interface name, i.e. Foo would be registered for IFoo. This is cool for people who like conventions over configuration, but can be too much magic other people. We use this approach and only configure classes which violate our simple conventions.

Perhaps the greatest benefit is how they integrate with various frameworks. Containers often have shortcuts to hook into popular web frameworks, like ASP.NET MVC or Nancy. The framework can use the Container to resolve types it needs. We use this to create Controllers and automatically inject their dependencies. This lets you use Dependency Injection while decoupling your code from the Container itself. Everything magically fits together.

The larger our applications become the more benefit we get from using Dependency Injection Containers. We no longer worry about how we are going to wire our classes together. Instead, we can focus on designing our interfaces and classes.

Connecting the Dots

Phew, you made it this far! I hope this helped shed some light on Dependency Injection and the surrounding concepts.

Instead of using concrete classes we switched to higher level dependencies, applying Dependency Inversion. We needed to get those dependencies from somewhere so we used Dependency Injection, via constructor parameters, to inject the dependencies we wanted.

We explored Dependency Injection Containers using these mental models:

Externally they are like one massive Factory for any type
Internally they are like a Dictionary of Factories

Then we dug into complete Dependency Injection Containers and their extra features.

Have fun decoupling your dependencies!

Footnotes

1. I could not get this link to work. I found it via this great article explaining how an abstraction is not synonymous with interfaces.

2. For some applications we intentionally call the Container from our tests. We treat the Container configuration as part of our integration tests. I will agree this is not ideal, but it simplifies creating various types and better mimics what our users will run.

Thanks

Thanks to my gracious 2015 mentees for letting me practice this on you.

Thanks again to my co-worker Josh who helped review this article. He had the great recommendation of renaming the “Poor Man’s DI Container section” to “Man with too much time on his hands’ DI Container”. Maybe I need to go write more code.

Rock-Solid PowerShell Projects

2015-12-30T18:45:07-05:00

We write a lot of PowerShell. We didn’t realize it when we started, but our projects have gotten much bigger. What seemed like a small bit of glue scripting is now the core project. As the projects grew we learnt some lessons in how to keep them maintainable.

Our first release was really simple. It was just enough to meet our project goals. We didn’t think through how the different pieces would fit together. Fast forward 2 years and the older projects felt very thrown together.

When we started a new project I wanted to make it feel like a professional software project. Instead of the good enough we had with the first project, I wanted everything to fit together just right. I want to apply all the best practices we use with any other development. Just being glue was not enough.

This list might seem straightforward and mundane. That is okay! I think these guidelines are essential for any project worth your time to maintain. With a looser language like PowerShell, these conventions helped give our new project even more structure. Ultimately, they led to better code we can all understand.

Consistent Layout

At first, we tossed our files in one big directory and a bunch of messy dot sourced files. This was brutal. Finding anything was impossible.

Now we have all entry scripts at the root of the project, all modules in a lib folder and tests on their own. More folders are added as needed. A consistent layout for files makes it easy to find your way around.

We took this even further. We applied the same basic layout to all the projects and libraries we maintain. Every repository now includes a build file, release notes and a README.md in addition to consistent directories. New developers (or you after 4 months away) can open any project and start contributing right away.

Test Thoroughly

We were naive when we started our first project. What the project did was really simple. As a result, we thought all we needed to do was run the code once through the happy path. As the code grew this was no longer enough.

We now diligently test each module independently using Pester then again with more comprehensive integration tests together. Pester is an amazing testing library for PowerShell. We love Pester. If you have not used it, go check it out now.

Pester has greatly improved our unit testing. Our previous testing only validated large scenarios and missed the edge cases. We now test individual functions in isolation to expose more permutations caused by the lower levels.

Modules for Everything

Initially, any reusable functions were placed in magical library.ps1 scripts, which would be dot sourced in every other script (i.e. . $PsDir\library.ps1 everywhere, boo). Everything was written as a script to either be dot sourced or directly called. This was great when we started but did not scale as the project became more complex.

With newer projects we place everything into independent modules. Each module has a single responsibility, i.e. setting up part of our application. This keeps every module small and focused.

Within each module, we intentionally keep some functions private. This allows us to shrink the module’s surface area without losing functionality. With our included scripts this would not have been possible.

Mandatory Continuous Integration

From the beginning, our projects have had the ability to check them out, run all tests and publish releases. This allowed us to rapidly add functionality while keeping up our basic hygiene. The code remains clean and we can make sure it always meets our basic requirements.

As we continued to improve, most of our projects have added Preflights. This complemented our existing continuous integration. Now we stop problems from ever reaching our master branch.

Our added emphasis on testing at various levels has improved our confidence. Every build/test run covers even more of the application and edge cases.

Summary

We have learnt a lot from maintaining PowerShell projects over the past few years. Our new projects are rock-solid. We try to have the following in every project:

Consistent Layout
Thorough Testing
Single Responsibility Modules
Continuous Integration

This has made our projects easier to understand and update.

Have your glue scripts turned into something bigger? Is it time to take it to the next level?

No Managed-Only Finalizers

2015-12-28T18:55:07-05:00

We cleaned up a fruity issues caused by using .NET Finalizers with managed classes. Finalizers are a magical method called by the Garbage Collector. Like most magic you need to be careful or you might saw your legs off. While Finalizers might sound like a cool way to clean up after yourself they should only be used to cleanup unmanaged resources. Otherwise, they are not recommended and can lead to instability.

There are two primary mechanisms for cleaning up resources in .NET: Finalizers and IDisposable.

Finalizers are meant to ensure unmanaged resources are cleaned up by the Garbage Collector. They feel like a hold over from unmanaged code and I get uncomfortable when I see them. In 99.9% of the managed code I have worked in they don’t belong.

Whereas most .NET classes implement the IDisposable interface to tidy up after themselves. Closing database connections and cleaning up files are standard examples. The interface works great for allowing your consumers clean up resources when they want to instead of waiting for the Garbage Collector.

We were using Finalizers and were doing it wrong.

What went Wrong

With Finalizers, like their C++ sibling destructors, it is important to pay attention to how objects are connected. We ran into a problem where a Finalizer was disposing the object which created it.

The classes TestData and TestUtility were used together in integration tests to setup/cleanup data. The tests intermittently failed because the DataWeCareAbout was deleted when the TestData Finalizer ran.

Our classes looked roughly like this (minus a bunch of boring code):

internal class TestData {
    private TestHelper m_helper;

    public TestData( TestHelper helper ) {
        m_helper = helper;
    }

    // The Bad Finalizer
    ~TestData() {
        m_helper.Dispose();
    }
}

internal class TestHelper : IDisposable {

    public TestData CreateData() {
        return new TestData( this );
    }

    public void Dispose() {
        m_conn.Execute(
            "DELETE DataWeCareAbout WHERE Id = @Id",
            new { Id }
        );
    }

}

Since Finalizers are called directly by the Garbage Collector the failures were not consistent. They would happen if the Garbage Collector ran early in the test run. We eventually found it by stepping through the code to isolate what was deleting the data during the test. This led us to the Finalizers.

Daryl, a great developer I work with, insisted Finalizers should not be used with only managed code. Their sole purpose is to ensure unmanaged resources can be cleaned up even if the programmer forgets.

We had two problems:

TestData disposed the TestUtility which deleted DataWeCareAbout too soon
We should not have used a Finalizer at all

Finding More Finalizers

We looked through our code base to see if more Finalizers were used elsewhere. We only found one where it was needed and many others which needed to be removed.

The extra Finalizers were not needed at all! They had no unmanaged resources! We deleted them without a second thought.

As an added bonus, they were using a fun pattern which mixes Finalizers and IDisposable. Consumers can proactively call Dispose and clean up the data. If that is forgotten then the Finalizer will be used. This was complete overkill for the Finalizers we didn’t even need.

You can read more about this pattern and implementing Finalizers on msdn. For completeness here is a sample of the pattern based on our code/msdn:

public class Example : IDisposable {

    private bool m_disposed = false;
    private ScaryUnmanagedResource m_resource;

    public Example() {
        m_resource = new ScaryUnmanagedResource();
    }

    ~Example() {
        Dispose( false );
    }

    public void Dispose() {
        Dispose( true );
        GC.SupressFinalize( this );
    }

    protected virtual void Dispose( bool disposing ) {

        if( m_disposed ) {
            return;
        }

        // Only if explicitly disposing should you clean up unmanaged resources
        // We don't have any so there is no work to be done
        // if( disposing ) {
        //}

        // Clean up all unmanaged resources
        m_resource.Delete();
    }

}

Learning More

It turns out having a Finalizer makes classes special. Here are a few highlights which further convinced me I should avoid them like the plague:

They are handled differently which degrades performance
When they are called is non-deterministic
Any managed references may already be garbage collected and be horribly broken
Are only for cleaning up unmanaged resources

Don’t Do It

Odds are you will never need a Finalizer. If you are not sure then you shouldn’t add one.

We were using it for automatic clean up with only managed code. This was both lazy and wrong. Instead, we should have used IDisposable to clean up at the end of the test.

We learnt a valuable lesson: We didn’t need Finalizers

I would like to thank my co-workers, Daryl, Chris, Michael Swart and Derek, for digging into this issue with me. It was fun to find the eventual root cause, fix this weird issue and learn more about Finalizers.

Are you your Code Reviews?

2015-12-17T20:17:07-05:00

I recently attended several fantastic developer talks where I work. My co-worker, Chris, had a very interesting talk sharing lessons he learnt during 7 months of code reviews with a remote team. He had some great insights I have been thinking through. The one which sticks with me is: how your code reviews are a reflection of you.

Chris had the benefit of helping a remote team get started. It was really hard. I appreciate the work his team put in. His presentation reviewed what he learnt after months of working with the same people. Code reviews were the focal point for most of his interactions.

Things change when code reviews are the primary way you interact with someone. You don’t know them as a person and you cannot see them on a daily basis. If they keep giving crappy code reviews your opinion of them forms quickly. Likewise, consistently great code can help them stand apart.

This was the reality for Chris while working with the remote team. Being a talk on code reviews, he found special examples highlighting lessons learnt. He felt after a while, the code reviews became the main representation of a person’s ability.

This got me thinking about being evaluated and represented by each of my code reviews. I thought about how I often become too attached to my code. It also caused me to reflect on several great blog posts about separating your worth from the code you write. These posts, in a nutshell, explain how ‘you are not your code’. In the end, I think I found a way I can bring these differing ideas together.

Reminder: My blog is my own opinion and not necessarily that of my employer. That said thanks for the great sessions!

Invested

Connecting with what Chris said, I take feedback to code reviews I make to heart. On the receiving end reviewing code from others, I care about maintaining the quality of what we built. When I send code reviews I try to be careful in how I approach making changes. I hope I don’t look stupid.

The dark side is when I treat comments I give or receive like an attack. Reviews should be an open discussion and not used to showcase how brilliant you are. Instead, humility and listening need to be applied to every code review.

It is really easy to become invested in your code. I put lots of effort and care into the work I do. I am emotionally invested. Too often I behave defensively about my code and take feedback too personally.

Feelings of ownership intensify the longer I have been on a project and what role I had in its formation. We have been on our current project since it began and seen it go through many changes. I care deeply about it and want to see it succeed.

Over the last few weeks, I wrapped up a project I worked on over the course of 7 months. At the same set of talks, I presented about how this project evolved. I feel personally responsible for the code. It feels like any feedback on the project or code reflects on me as a developer.

I am self-conscious about by ability as a developer. I often feel like an imposter. Combining those developer doubts with putting myself out there in code reviews, you get a potent mix of feelings and lack of confidence. My inner phony is on display.

You Are Not Your Code

There are many great blog posts about how you are not your code.

Each of these posts is a reminder to not be so serious. The code you write is not your worth as a person. You are so much more. Don’t direct the suggestions and criticism inward. Instead, use the constructive criticism for getting better.

This is a message I need to take to heart. I need to separate myself from suggestions for the code I have created.

In preparing for my presentation I had some great questions from others which forced me to re-evaluate the decisions I thought were rock solid. Incorporating the extra feedback from others made the finished product was even better.

How does ‘you are not your code’ fit if your colleagues think about you based on the code you share in code reviews? This seems like the opposite perspective. Wrestling through this contradiction caused me to rethink Chris’ talk.

Caring Professionally

I think both ideas can fit together. The sum total of your being is not the code you write. How you present yourself, learn, care and interact with others is a clear reflection of what you value. Code reviews are one avenue for expressing yourself.

In talking with Chris, he cares about the humans behind their code reviews. In teasing apart what he said, I think caring about what you do is the most important part of each code review. Not getting emotionally attached to your code to the point where it consumes you. Care about your work and put your best foot forward.

If someone else is going to take the time to review your code be kind to them. One recommendation from the talk was for you to thoroughly review your code before sharing it. Go through the changes on your own and clean up any loose ends. Add comments explaining the decisions you made. Show you care.

I think constantly improving and trying your best is part of being a professional programmer. Learn from your code reviews. Try to make the most out of each review and the feedback you receive.

Be supportive when interactive with others. Don’t stomp around and shoot down their ideas. Treat others how you want to be treated and you cannot go wrong.

Although I don’t count myself as a great developer, like Jeff Atwood I care too much.

Unfortunately, the world is full of people who don’t give a damn about their work. Those of us who love programming enough to become highly skilled at it tend to have the opposite problem – we care too much

The solution is to convert caring into actions which help others.

You are not your code, but your code reviews are a reflection of your workmanship. Care about your work and do your best in every code review.

P.S. Thanks Josh for helping me review this blog post. Without your help I would be stuck in the present and past tense at the same time.

Don't Use A Singleton

2015-12-06T19:12:07-05:00

Singleton objects are not normally what you want to use to solve problems. It is great for optimizing creating objects but otherwise has limited utility. Recently, I had a change where I briefly considered using a Singleton to deal with it. Using a Singleton to solve my problem would have been wrong and led to more problems.

I was refactoring some code which had some unusual data access. Each user used a key/value store to save data and then later retrieve it. The data being saved was pretty simple and the code treated the key/value as a really fancy cache.

We decided to use the same key/value store for all users. Based on the data it made sense for every user to see the same values. Sharing between users would make the cache more effective.

I thought I could take a shortcut. Since the values would be identical for all users then why not use a Singleton object to store the value in memory. Every time the class was used we could return the same object and store the necessary value in a field. We could remove the key/value store completely and simplify the code. Or so I thought.

In the amazing image below, you can see what the code would look like before and after using a Singleton to refactor the code.

This isn't the complete Singleton Pattern. My use case would have reused the same object for the entire application without needing to restructure how the class was accessed. I have included links discussing the full Singleton Pattern at the end of this post. They come to the same conclusion.

Intervention

But Scott you say, what about other servers! Other servers would need the Singleton too! Umm, Scott … they would have different objects. Oh right.

The Singleton object starts to have issues immediately coordinating the same value between multiple servers. There are a bunch of problems like:

Different servers would have different objects
Access to the data needs to be coordinated across threads
Restarting the application loses the saved values

Thankfully my co-worker, Daryl, reminded me of the issues a Singleton would cause. Instead, we decided to store the value in the application’s database. It also meant we could stop worrying about cache invalidation and simplify the code since the database would always have the right value. Great. Problem solved.

Using a Singleton to make this change is wrong. It would introduce other issues which could be trivially avoided using other solutions.

The Bigger Picture

After this brief lapse of sanity, we talked a little more about why Singletons don’t make sense in your architecture. By definition, Singletons are a single point of failure and cannot be scaled out. For any system with enough load you want neither of those.

Some Singletons in your architecture are HARD to avoid, like a centralized database you use everywhere. Do whatever you can to make it redundant and continue to scale as your application grows. Such Singletons may just be part of doing business and once baked into the architecture are not worth the cost to replace.

You may have operations which should only run in one place. Not a problem, use a semaphore (or leader election like the cool kids) to coordinate who runs the operation then only run it in one place. Whatever runs the operation can be scaled, redundant and perform many other useful operations.

A better question is why do you have this operation/service which can only be run in one place? If the code is important enough and needs more throughput you may need to deal with your Singleton. Do what you can to allow it to run in multiple places at the same time. If that fails, shrink the code which needs to be exclusive as much as possible to avoid contention. Again consider making who and where it runs more flexible.

An Optimization

So when does a Singleton object make sense? To avoid allocating many small objects or very expensive objects.

Both these cases are optimizations to existing code based on specific circumstances. Prior to using a Singleton, you should investigate the problem completely and determine all options you have available.

Using a Singleton like this should be a specific micro-optimization which is transparent to the rest of the application. In our applications, we have some classes which are created once then injected into any class which needs them. The consuming classes have no idea that they are all using the same object. This works great for functional classes without any state or which talk directly to our centralized database.

Depending on how your component is used you need to consider concurrency. The CPU/memory gains from reusing the same object may be completely lost due to synchronization overhead.

Not the Tool for the Job

In the end, a Singleton object was not the right tool for the job. In thinking about it further, it is probably not the right tool for many jobs. We were able to easily refactor the code using other techniques.

My example wasn’t even using the complete Singleton Pattern and it was still bad! For more reading on why Singletons are the worst:

To recap: Singletons are bad, stupid and evil.

If you think a Singleton is a perfect solution to your problem, double check. There is probably different solution which fits better.

My Time As An Exterminator

2015-11-30T18:37:07-05:00

This week I am presenting to a large number of coworkers about work I started as an Exterminator. For those which want to learn more about that time I wanted to summarize all of the posts I had done.

Earlier this year I went on a 4-month rotation to a different team. It a great opportunity I learnt a great deal from doing. While on rotation I tried to regularly blog what I was doing and what I was learning from it. Here is a summary of all those posts.

I volunteer as tribute. This was my first post. I was really excited to join the Exterminator team. Learning new things and helping people.

The 4 Stages of Legacy Code. I had dived straight in and was having fun digging into code without tests. I was accustomed to thorough test suites and working on brand new code. In the end I was able to get through it and find good ways to approach this new challenge.

Focus and Quality. In order to keep myself honest, I blogged about my goals for the year and how I intended to approach them. This post and my goals, focus and quality, were a great way to ensure I was concentrating on the right things while I was an Exterminator. The links included with this post are fantastic and with the year end approaching it is time for me to read them again.

On Comments. We talked about what makes for good comments in code. Which comments are not necessary or too verbose. I highlight a few cases where I think comments really add to the code. There is also some amazing C code with silly comments.

DAMP Unfiltered AAA. With my added focus on quality, I spent lots of time writing tests. I delve into how trying to not repeat yourself and stay DRY makes tests much harder to understand. Instead, you want descriptive and meaningful phrases (DAMP) so yours tests are clear.

Pull Requests. I tried dabbling with breaking up a larger chunk of work into small pull requests. This led to a pull request explosion! Everything became a small pull request layered upon other pull requests. This fun exercise set the tone for the rest of my time as an Exterminator.

Code Reviews. I share my approach to code reviews. I strive to learn/share ideas and understand the underlying logic being updated. Hopefully, this approach leads to better conversations about the code change.

Discuss into Action. We had a retrospective which did not sit well with me. It was a good conversation but did not lead to any action items. I was left wanting more.

The Wrong Time to Refactor. I like refactoring A LOT. You might even say too much. We were the day before a release and fixing a late bug. I wanted to refactor the code more. We decided as a team this was not the right thing to do. Later, we were able to safely complete the refactoring.

Caught By Conventions. Oops. I learnt a lesson by swimming upstream. I was excited to try a new tool, Specflow, and started by contributing code to another project. I missed the memo and broke all the conventions within their project. It was very frustrating and then I realized I had done this to myself. I was able to learn from my bad pull request and teach others the lessons I encountered.

My Nemesis, Email. I was very distracted this week. I go over some techniques for reducing your interruptions, particularly from email.

Finding the Bottleneck. This is a basic introduction to some of the performance testing tools in Visual Studio. We were able to find hot spots in our code and apply improvements. I then break down the process, highlight some of our biases and lessons for next time. Go deeper.

I hope enjoyed these posts from my time as an Exterminator.

Crafting Code with C#

2015-11-22T18:42:07-05:00

In this post. I am going to go show how to use the keywords in C# to restrict your code and help you intentionally craft your APIs. While you can apply the same ideas to external APIs, I will focus on backend C# code here. The following examples are inspired by our code to highlight different techniques we use to achieve the exact code behaviour and visibility we want.

A simple example: A Factory

Let’s try a simple example. If you have a factory class the class being created can be made internal instead of public. Here is what you do NOT want to do:

public class WidgetFactory {
    IWidget Create() {
        return new CoolWidget();
    }
}

public class CoolWidget : IWidget {
    ...
}

If CoolWidget is public then why would you need the factory? People could start calling the class directly which would make the factory unnecessary. With other classes directly using CoolWidget changing the constructor, dependencies or creating a different class in the factory are all breaking changes.

Instead, CoolWidget should be internal! Much better. The factory cannot be bypassed.

internal class CoolWidget : IWidget {
    ...
}

Hide Internals: A Worker Class

Doing work in classes can get complicated. I looked through a whole bunch of our code and found we often use private methods for doing little bits of work.

We have worker classes for processing queued work which then send callbacks indicating the work is done or has failed. Within the class, there was a handy method for sending callbacks. The helper method would format the callback URL and other parameters then use other classes to actually send the callback.

public class Worker {
    public void Process( WorkItem work ) {
        ...
        SendCallback( work.CallbackUrl )
    }

    public void SendCallback( Uri url ) { ... }
}

The sample above shares too much. Why would SendCallback be public? It doesn’t fit with Worker’s purpose: processing work items. The method is only used by Worker and is nicely isolated to the class. Since we would never want other classes using the method, it should be private!

public class Worker {
    private void SendCallback( Uri url ) { ... }
}

Much better. Had we left this method public it could accidentally be used. Keeping it private allows it to continue to evolve separately from the API of Worker.

Sounds good, right? There is a catch, testing these methods is harder. You can’t test them as easily because they are now private. Instead, you need to test them indirectly through the inputs/outputs of other methods or using their behaviours. If the logic is really complicated you might want to move it into separate classes and interfaces.

In the example above we have separated sending the callback into a different class. This lets us test the Worker and sending callbacks separately. The logic for sending callbacks is consolidated in the dedicated class. The private method in Worker is still useful and allows us to prepare the Payload to be sent. The resulting helper classes look like this:

internal interface ISender {
    void SendCallback( Uri url, Payload payload );
}

internal class Sender {
    void SendCallback( Uri url, Payload payload ) { ... }
}

This lets us keep the logic testable and the external API small. There are now more classes which might make the code more complicated.

We intentionally keep these helper classes internal. While they are useful on their own, they have been created for use only within this assembly. Right now we don’t think anyone else would want to send their callbacks the same way. Due to this we have left them out of the public API until we are proven otherwise.

Inheritance: Family Planning

Controlling inheritance is useful. Most developers I work with avoid using it like the plague due to issues they had in the past. Composition over inheritance is not only recommended, it is enforced by marking most classes as sealed:

public sealed class CantTouchThis { ... }

Marking classes as sealed prevent the class from being inherited. This stops inheritance abuse. To be honest, I think it is overkill since there are very few cases where you need to explicitly block inheritance. More often than not you don’t have to worry about it. People don’t willy nilly start inheriting from classes when none of the methods can be overridden and there are no protected fields.

Inheritance: Template Class

I think base classes can be useful when used correctly. I use them to setup template methods or share common/optional functionality. This isn’t a technique I use too often. I like to treat it as yet another code design tool.

To enforce the purpose you envisioned for your base classes, I recommend using abstract classes and methods. This keyword ensures your desired methods must be implemented by child classes. The base class itself cannot be instantiated which further clarifies its purpose as a base class.

public abstract class TaxesCalculatorBase {

    public decimal CalculateTaxes() {
        ...
        decimal multiplier = GetTaxesMultiplier( totalIncome );
        ...
    }

    protected decimal abstract GetTaxesMultiplier( decimal totalIncome );
}

public sealed class FlatTaxesCalculator : TaxesCalculatorBase {

    protected override GetTaxesMultiplier( decimal totalIncome ) {
        return 0.05;
    }
}

public sealed class ProgressiveTaxesCalculator : TaxesCalculatorBase{

    protected override GetTaxesMultiplier( decimal totalIncome ) {
        if( totalIncome > 75000) {
            return 0.15;
        } else if( totalIncome > 50000) {
            return 0.12;
        } else if( totalIncome > 25000) {
            return 0.08;
        } else
            return 0.05;
        }
    }
}

The TaxesCalculatorBase is an abstract class with the template method CalculateTaxes which uses the abstract GetTaxesMultiplier method. Classes inheriting from TaxesCalculatorBase must implement the required methods and can do so however they like. To prevent the hierarchy from growing out of hand, you can optionally mark the child classes as sealed.

Another usage of base classes is implementing common or optional functions. I used this recently to make optionally implementing part of an API easier. The class had a method to return an IEnumerable. The default implementation in the base class returns an empty list. When the team was ready we could update the child classes one at a time to provide the new functionality.

public abstract class ExampleProviderBase {
    public virtual IEnumerable<IExample> GetExamples() {
        return Enumerable.Empty<IExample>();
    }
}

All Action: Helper Classes

Every now and then we have helper classes with only methods and no state. Stateless classes can be made static to prevent them from being instantiated or having instance variables added. If you had a helper classes for Uri’s it might look like this:

internal static class UrlHelpers {
    public static Uri FormatUri( Uri baseUrl, string route ) { ... }

    public static Uri TruncateUri( Uri baseUrl ) { ... }

    public static Uri SomethingCoolWithAUri( Uri baseUrl ) { ... }
}

The static constraint helps the class stay stateless and cannot be instantiated. Having the class be static ensures all the methods must also be declared as static.

Manage state: Immutable Classes

I think it is worth the minor effort to control whether fields/properties can be modified. Limiting the number of ways data can be modified and passed around can help highlight the right way to use your classes. The extreme version of this are immutable classes which cannot have their values changed and must be fully initialized when they are created.

In this simple example, I have made a Person class which is immutable. There would be other classes for retrieving and updating the data. Anyone trying to create a Person must provide the necessary value at creation time. This is a great opportunity to validate any inputs to prevent invalid objects from being created.

internal sealed class Person {
    public string FirstName { get; private set; }
    public string LastName { get; private set; }

    public Person( string first, string last ) {
        FirstName = first;
        LastName = last;
    }
}

You can take this even further if you would like. I used properties to make writing FirstName and LastName easier. You can just as easily use readonly to force the fields to be initialized inside the constructor. This allows the compiler to enforce keeping the class immutable.

internal sealed class Person {
    private readonly string m_firstName;
    private readonly string m_lastName;

    public string FirstName { get { return m_firstName; } }
    public string LastName { get { return m_lastName; } }

    public Person( string first, string last ) {
        m_firstName = first;
        m_lastName = last;
    }
}

In C# 6 readonly fields become even easier thanks to getter only properties. This is the same as the first example except for the missing private set on the properties. Like the second example, the compiler will again ensure the properites are not modified outside the constructor.

internal sealed class Person {
    public string FirstName { get; }
    public string LastName { get; }

    public Person( string first, string last ) {
        FirstName = first;
        LastName = last;
    }
}

Another great immutability technique is to create a new object with every operation which would otherwise modify the current object. Great examples of this approach are the DateTime and string classes:

DateTime now = DateTime.Now;
DateTime future = now.AddDays( 2.5 );

if( now == future ) {
    Console.WriteLine( "The future {0} is different than now {1}", future, now );
}

string example = "Hello World";
string updated = example.Replace( "l", "" );

if( updated == "Heo Word" ) {
    Console.WriteLine( "The example phrase was updated!")
}

Immutable classes strongly affect how users interact with them. They can reinforce readonly parts your API and highlight how you want data to be updated.

Generics: A Different Animal

Generics are an extremely powerful way to stay flexible while remaining strongly typed. Adding constraints to generic parameters, ensure the types used with your class match your expectations. If nothing else I find the constraints are a fun way to strictly enforce strong typing.

// T is both an Exception and has a default constructor
public class ExceptionThrower<T> where T: Exception, new() {}

// Using generics to keep callers type safe
public static class Cloner {
    public static IEnumerable<T> For<T>( T original, int n )
        where T : ICloneable {

        for( int i = 0; i < n; i++ ) {
            yield return (T)original.Clone();
        }
    }
}

// Fun with generics to create type safe factories
public interface IFactory<T> where T : class {
    T Create();
}

public class DefaultFactory<T> : IFactory<T> where T: class, new() {
    public T Create() {
        return new T();
    }
}

Conclusion

I hope you enjoyed these examples showcasing the C# keywords and how you can use them to create the exact API you want. I believe it is important to intentionally limit the surface area of your API by restricting how your classes can be used and the properties/methods they expose. Carefully crafting your API should make future maintenance easier.

Fun with Generics

I decided to move this to the footer. I have been known to abuse generics in the past. Here is some fun code which has constraints against multiple types. This ensures stronger types throughout the API. Within the class, less restrictive types are used to avoid needing a common interface for the inputs.

I would discourage using this particular class in your code. Use a DI container like Autofac instead.

public class FactoryCollection {
    private readonly Dictionary<Type, object> m_factories =
        new Dictionary<Type, object>();

    public void Add<TDependencyType, TFactoryType>( TFactoryType factory )
        where TDependencyType : class
        where TFactoryType : IFactory<TDependencyType> {

        m_factories[typeof(TDependencyType)] = factory;
    }

    public void Add<TDependencyType, TFactoryType>( TFactoryType factory )
        where TDependencyType : class
        where TFactoryType : IFactory<TDependencyType>, new() {

        Add<TDependencyType, TFactoryType>( new TFactoryType() );
    }

    public void Add<TDependencyType>()
        where TDependencyType : class, new() {

        Add<TDependencyType, DefaultFactory<TDependencyType>>(
            new DefaultFactory<TDependencyType>()
        );
    }

    public bool TryCreate<TDependencyType>( out TDependencyType result )
        where TDependencyType : class {

        Type type = typeof(TDependencyType);
        if( !m_factories.ContainsKey( type ) ) {
            result = default( TDependencyType );
            return false;
        }

        IFactory<TDependencyType> factory = (IFactory<TDependencyType>)m_factories[type];
        result = factory.Create();
        return true;
    }
}

Do what must be done

2015-11-09T18:16:07-05:00

This is going to be an interesting week. Deadlines are tight. Life doesn’t stop and has more than its fair share of fun. Instead of bemoaning our lot I have decided to do something about it. Work needs to get done and so we can step up. We will do what must be done.

We have a whole bunch of things stacked against us this week. There have been some surprises. People we depend on are not available. New projects are appearing and demanding our attention. It will be a challenge.

To add insult to injury we are dealing with some adversity at home, which will affect us for the coming year. I will need to miss work because of it. This will not be an easy thing to get past and we are very thankful for our incredibly supportive friends and family.

I am excited to meet our problems head on. Not only do I think we will get all the projects done, I think we will excel if we put our minds to it. I think there are a few things I can do to help make this week a success.

Be positive. It would be easier to curl up into a ball and try to ignore what is happening. Although the circumstances are disheartening there is hope. We need to remain positive and strive for the best. If we focus on the negative it will make everything else worse.

Be patient. Not everything is going to be magical. We will need to deal with more fun as it rolls in. Instead of being overwhelmed by the changes we need to patiently deal with them. The situation might not be ideal, but we can get through it.

Be proactive. Don’t sit back and let the challenges come to you. Embrace and overcome them. I plan on doing what I can to help out and get things done. To step up where I am needed most. My normal work might suffer, but we will get through our deadlines.

Although it will be hard I am optimistic about this week. If we remain positive, stay patient and are proactive we will get through it. This week I have decided to step up and do what must be done.

Dependency Injection Fun in NancyFx/TinyIoC

2015-11-05T19:34:07-05:00

This week my coworkers, Chris and Viktor, had a fun issue where it was hard to find the root cause. They added some caching to speed up a heavily used API which led to some tests breaking unexpectedly. In this post I wanted to dig into the root cause behind why the tests broke.

The one major lesson I learnt last year was the importance of getting to the root cause for unknown behaviour. My coworkers were doing a focused performance improvement. On the surface the change was straight forward. They wanted to add caching to a specific request which queried the same values multiple times.

To simplify the logic they decided to reuse the same values ever time it was used during a request. This kept the caching rules really simple and meant we did not have to care about cache invalidation. A new request could get new values and we did not need to determine how long cached values are valid.

The solution was to create a class per request which would read the underlying value once then save it to a field. To get the underlying value we delegate to the real provider. Every future query for the value would return the saved value from the field. The code for this new caching looks like:

class CachingProvider : IProvider {
    string m_value;
    readonly IProvider m_inner;

    internal CachingProvider( IProvider inner ) {
        m_inner = inner;
    }

    public string GetValue() {
        if( m_value != null ) {
            return m_value;
        }

        m_value = m_inner.GetValue();
        return m_value;
    }
}

We adjusted our dependency injection configuration to create one CachingProvider per request. For this application we are using NancyFX with the built in TinyIoC dependency injection container. To perform the change we updated our NancyBootstrapper configuration so TinyIoc would reuse the same CachingProvider for each request. Then we were off to the races!

class NancyBootstrapper : DefaultNancyBootstrapper {

    protected override void ConfigureRequestContainer(
        TinyIoCContainer container,
        NancyContext context
    ) {
        // Register other types here

        container.Register<IProvider>( (x, options) => {
            IProvider implementation = x.Resolve<RealProvider>();
            return new CachingProvider( implementation );
        } );
    }
}

Trouble In Paradise

The code cached the new values fantastically. The bad pages were now much better. However, the change broke an integration tests which seemed unrelated.

The broken test was a large integration test which faked a bunch of tricky dependencies. These special dependencies needed to be updated so we could control the test behaviour.

With some more investigation we found out IProvider was one of the types we were faking out. We tracked down the test code which registered it’s own implementation of IProvider to overwrite the real type. This was again done via the dependency injection container like so:

class TestBootstrapper : NancyBootstrapper {

    protected override void ConfigureApplicationContainer(
        TinyIoCContainer container
    ) {
        // Register other types here

        container.Register<IProvider, TestProvider>();
    }
}

Although we had found out what broke the test, we still had not fixed it or learnt why it broke.

Getting It Working

After many hours of debugging and head to desk slamming Chris and Viktor managed to fix the issue by changing how the registration was overridden. They replaced how IProvider was registered with a method which could be overridden. This change was pretty invasive and also moved around how other types were registered (not shown here).

In the normal bootstrapper they added a virtual method to register the updated type:

class NancyBootstrapper : DefaultNancyBootstrapper {

    protected override void ConfigureRequestContainer(
        TinyIoCContainer container,
        NancyContext context
    ) {
        // Register other dependencies

        RegisterProvider( container );
    }

    // The new method they added
    protected virtual void RegisterProvider(
        TinyIoCContainer container
    ) {
        container.Register<IProvider>( (x, options) => {
            IProvider implementation = x.Resolve<RealProvider>();
            return new CachingProvider( implementation );
        } );
    }
}

Then in the test bootstrapper they override the new method to register the fake IProvider:

class TestBootstrapper : NancyBootstrapper {

    protected override void RegisterProvider(
        TinyIoCContainer container
    ) {
        container.Register<IProvider, TestProvider>();
    }
}

This worked but they still had not gotten the root cause as to why the test was broken. They had only addressed the symptom which was causing the test to fail. It is not shown here, but several other methods were moved around to accommodate the change. We did not like this and knew there must be a better way.

Going Deeper: The Root Cause

If you look closely at the code samples above you can see the type switched from being registered for the Application to registered for each Request. While this seems fine it meant the tests would resolve the actual type instead of the fake IProvider they should be using.

The per Request dependencies are registered with a separate TinyIoCContainer which is a child of the Application container. When the types are resolved the Request container is tried first and if nothing is found then the Application container is tried. If nothing is found the dependency fails to resolve which can cause an exception.

The test failed because it found the new CachingProvider instead of the TestProvider it needed.

With this new knowledge the fix was relatively straight forward. We registered the fake IProvider per Request. This overwrites the CachingProvider registration with the TestProvider and solves the problem.

// Fixed Test Boot
class TestBootstrapper : NancyBootstrapper {

    protected override void ConfigureApplicationContainer(
        TinyIoCContainer container
    ) {
        // Register other types here

        // Previously configured here
        // container.Register<IProvider, TestProvider>();
    }

    protected override void ConfigureRequestContainer(
        TinyIoCContainer container,
        NancyContext context
    ) {
        // Register other types here

        // Fixes the root cause
        container.Register<IProvider, TestProvider>();
    }
}

Public Service Announcement

Phew. You made it this far. What is the major lesson?

It is important to understand how your code works! Especially when it doesn’t.

We went deeper into Nancy and figured out how the dependency injection worked! It was great. Afterwards, we could make the fix in a much simpler way.

Next time you are troubleshooting an issue make sure you find the root cause. Going deeper is worth the extra time. You will understand the problem better and the next time something comes up you will know what to do.

Until then happy troubleshooting.

I would like to thank Chris and Viktor for letting me write up this post and doing all the leg work for this issue.

Basic PowerShell Remoting

2015-11-01T18:45:07-05:00

Using remote PowerShell commands is a great way to manage servers. I have been spending more time using Windows 2012 Server Core which makes using remote tools essential. Instead of connecting using Remote Desktop, I try to do everything using PowerShell remotely. In this post, I will show off some extremely basic remote PowerShell commands, Enter-PSSession and Invoke-Command.

Let’s start with a simple command:

Enter-PSSession -ComputerName 'Target'

This amazing command uses your current PowerShell prompt to run PowerShell commands interactively on the remote computer, “Target”. When you are done type exit. With this command you now have the complete might of a fully operational PowerShell!

By The Conmunity - Pop Culture Geek from Los Angeles, CA, USA CC BY 2.0, via Wikimedia Commons

You can run single or multiple commands using Invoke-Command. Unlike Enter-PSSession, this command is not interactive and will still stream results back to your current prompt. This is great for one line commands like restarting IIS on the remote server “Target”:

Invoke-Command -ComputerName 'Target' -ScriptBlock { iisreset }

Remote commands are allowed by default on Windows Server 2012 and beyond. On older operating systems you can run Enabled-PSRemoting -Force from an Administrator PowerShell prompt on the target machine to enable remoting. You can then test the connection by running the following from another computer:

Invoke-Command -ComputerName 'Target' -ScriptBlock { echo 'hello' }

There are many other commands which natively support remote operations. These commands will often have a ComputerName parameter. You can see a list of commands with the ComputerName parameter by using:

Get-Command -ParameterName 'ComputerName'

Among my favourites is Get-EventLog. It is a great way to look at messages from a remote server without ever leaving the terminal. This example retrieves, formats and displays the last 5 error messages from the remote server “Target”:

Get-EventLog -ComputerName 'Target' -LogName 'Application' -Newest 5 -EntryType 'Error' `
	| Format-List TimeWritten, Message `
	| more

I hope you liked this mini intro to PowerShell remoting. Now go run some commands!

Being Shippable At All Times

2015-10-29T21:00:07-04:00

In this post I explore two approaches to being shippable at all times; knowing your Last Known Good package or having a Golden Master. Our team thoroughly tracks our last good package we have built and tested. Other teams stay shippable by keeping whatever is currently checked in stable. They can ship their master branch and we can ship our Last Known Good package.

In order to ship software you need to do the following:

Choose what to ship.
Produce code worth shipping.

If your code is no good or always breaks the build then you will not have anything to ship. You shouldn’t choose to ship garbage.

By releasing something you implicitly choose what you want to ship. Often there will be multiple builds/commits to choose from which could be shipped.

With solid Continuous Integration you can improve the confidence in each commit. The confidence from great unit and acceptance testing can be enough to declare any build shippable.

Lately, teams we work with have been intentionally trying to stay shippable. They want to support releasing at any point and other Continuous Delivery norms. Our team has long been trying to track the last package we deemed shippable. Both approaches are effective and have their own trade-offs.

We focus on choosing what to ship based on previous good builds. It means we can always ship something when we want to release. We keep track of the last build to be fully tested, what went into it and where it has been deployed. From this we choose our Last Known Good version. When it comes time to release into production we ship that version.

The other teams focus on keeping their master branch shippable at all times. This goes above and beyond keeping their Continuous Integration process green. No changes can be merged which would prevent the code from releasing. Risky changes should be broken down. When it is time to release the latest commit to the master branch is used.

Last Known Good

The Last Known Good package picked from the possible builds from the master branch.

We track our last known good set of binaries through our deployment pipeline and then choose which one set we want to ship at the end of the release. We know exactly what went into each build and the testing which was performed.

With this approach there are few aspects which are really important:

Knowing what has been committed to each build
Tracking builds between different stages/validation
Visibility into the deployment pipeline

What we ship is choice by our team supported by tracking what is in each build. If more testing and validation is required we communicate with the team to make sure it happens before we release. We closely follow which stories are included in each build.

There is more communication overhead with this approach. The more people involved with each releases the harder it is to agree on what the Last Known Good is at any given point.

The current branch does not need to be perfect. Doing testing on the master branch is okay, but needs to be coordinated. The worst case is master may be broken for short periods. This can be dangerous for less rigorous teams who may leave their code broken for days. It is still highly recommended to use Continuous Integration and keep builds green.

Golden Master

The Golden Master keeping the master branch shippable. Can always release the latest commit.

Other teams focus on keeping their code shippable by ensuring the master branch can always be built and deployed. For each commit, they ensure all integration tests pass thanks to solid Continuous Integration. The majority of the deployment pipeline is automated. More stability may be achieved using extra hardening phases.

Key aspects ensure the code remains stable by:

Avoiding build break at all costs
Extensive preflights
More validation performed prior to merging

Changes requiring manual testing are problematic. They are inherently not “always shippable” once they have been merged. Some teams compensate with even more testing prior to merging to avoid breaking master. Others accept short hardening periods to test all the code which has been committed.

I have worries about this approach and am biased to the Last Known Good. It seems to be running well for teams using it, but can lead to problems due to delayed merges. Since the emphasis is on staying stable changes can be delayed or avoided. Refactoring is harder if teams merge less frequently. Teams need to focus on merging small changes frequently. Another great mitigation is to merge sooner and keeping risky code behind feature flags for later testing. It is imperative to avoid merging ALL THE CHANGES at the end of a release cycle.

Choosing what to ship is easy. Use the latest commit. No need to get fancy picking which build or tracking what has happened.

Risk

All releases have risks. Whether the risks are technological or business-related it does not matter. Every change includes some element of uncertainty. Both Last Known Good and Golden Master strategies address the risks associated with releases but approach the problem differently.

The Last Known Good approach builds in accounting for risk in how you choose which version to use. Picking which builds are good could be a factor of automated testing, selection by testers, voting by a group of people, etc. Business pressure to release something too soon can cause issues to be shipped to production.

Gaps in automation indicating whether or not master is broken reduce the effectiveness of the Golden Master. As with any CI process, they are only as good as the coverage they provide. Risk can be reduced with an extra hardening period or intermediate branches, but this pushes the problem to that location and introduces delays.

Both approaches suffer from issues with the validation used to determine good versions and stop bad versions. If the version choice or validation are incomplete then bad versions will sneak through.

Next Step: Going Continuous

If you take both approaches to the limit you end up at Continuous Delivery and Continuous Deployment. Shipping more frequently and being ready to release at any moment.

With Continuous Delivery it is important to be able to release at any point. What and when to release is chosen based on needs and features. It is essential to do comprehensive testing in development by deploying to production-like environments. The deployment process and configuration should be automated. This is close to our Last Known Good approach with a great emphasis on being deployment ready. The Golden Master can easily be used the same way since every commit should be shippable.

Continuous Deployment takes Continuous Delivery further. Every build which passes automated testing is automatically deployed into production. It is what you would get if after you tested your Golden Master you deployed every passing build into production.

To know gauge how continuous your process is you can use this test from Martin Fowler:

The key test is that a business sponsor could request that the current development version of the software can be deployed into production at a moment’s notice

and nobody would bat an eyelid, let alone panic.

Both our team and others are trying to do Continuous Delivery. We make potential releases all the time and then choose when we want to ship based on our client needs. Earlier this year we accelerated our releases and do full Continuous Deployment into some environments. Releasing more often has been fantastic and lets us react to changes more effectively.

So far our focus in this blog post has been on everything leading up to a final release into production. Improving your release process and shipping continuously is the logical next step.

Don’t stop at being shippable; keep going for Continuous Delivery or Continuous Deployment.

For a more thorough description of each term see this fantastic Stack Overflow question: Continuous Integration vs. Continuous Delivery vs. Continuous Deployment

Why Not Both?

These two options don’t compete with one another. They complement one another. You can happily do them both at the same time. Emphasizing shifts what to prioritize when improving your deployment pipeline.

By applying both approaches together, you increase your odds of having great releases. You can ship more. You will have higher confidence in each release. The closer you get to a Golden Master branch you will have many Last Known Good builds to choose from.

The more thorough your automated validation, the more confidence you have in each build you make. Reduce the overhead and effort for each release. Keep improving your releases and try Continuous Delivery or Continuous Deployment.

Choose the best from each approach and apply it to your deployment pipeline. Then get to work shipping great software.

Safe Ordering For Breaking Changes

2015-10-26T19:51:07-04:00

When you decide to make breaking changes to your API there is a safe easy order to follow. If you don’t you will make consumers of your API very sad. In this post I am going to explain how we update our APIs and slowly introduce changes which otherwise would horribly break them.

You have Version 1.0 live. It is great and you love it! Everyone is using it and telling all their friends about it.

curl http://yourservice.com/api/v1.0/route \
  -X POST --data "<data />" \
  -H "Content-Type: text/xml"

Time goes on and all is well. Out of the blue you have an epiphany. A change to the API which is soooo much better than Version 1.0. JSON instead of XML. You could replace the old API in place, but it would be a breaking change which would hurt all those users of your API.

curl http://yourservice.com/api/v2.0/route \
  -X POST --data "{ 'data': 'booya' }" \
  -H "Content-Type: application/json"

This will be okay. Take a deep breath. Begin your changes in this order:

Update the service to support both Version 1.0 and 2.0
Update all the clients to use Version 2.0
Remove support for Version 1.0

The key to this is step 1. By having both versions live at the same time users can switch to Version B at their leisure. This is key. They are not forced to immediately update all their clients to take the new version. Although Version 2.0 breaks Version 1.0, temporarily supporting both makes the change safe for all the clients.

Steps 2 and 3 can happen as quickly or slowly as you would like. I like to accelerate the process to reduce the number of supported versions in the wild. We have also left older versions alive for extended periods provided the cost of supporting them was very low.

So there you have it, a nice simple order to make breaking changes to your API without breaking your API.

Drawing Ideas Out

2015-10-08T19:42:07-04:00

Listening is important. If you spend all your time talking you will miss good ideas. We have been working closely with another team who is coming up to speed. Instead of helping direct the project I need to shift my efforts to drawing out their good ideas and supporting them.

We had been lined up to work with another team on a project. Due to logistics it was decided the project would be easier if they lead the development. We would assist them and discuss different ideas, but not much else.

No Monopoly on Good Ideas

Prior to the project starting we had spent a great deal of time thinking about the overall problem and potential solutions. The affected code would integrate heavily into what we have done. Over time we have become subject matter experts which has led to strong opinions about how things should be done.

Like many developers I want to solve problems. You know build cool stuff! This normally involves rolling up my sleeves and diving in. I like choosing how to approach different problems and then working with our team to get it done.

However, to succeed on this project I need to put aside my own ideas and listen. Instead of starting with my own opinion, I need to facilitate the conversation so their ideas shine through. My co-worker Daryl put it fantastically, “they have good ideas, we need to help draw them out”.

Arguing for my own ideas would be counterproductive. The goal is not to show off how smart we are or build up my ego. We could hand them over the master plan we would have followed and miss the mark completely. It would kill the project and crush any autonomy the other team has.

The point is to make the project a success and help our clients. Together, we need to focus on shipping great software to meet the client needs.

The Great Ideas Are Waiting

The other team has great developers who are going to do a fantastic job. While we might have a head start in knowing the problem domain, they will be able to come up to speed. We did. We need to give them the space to solve their own problems and to ask questions which will help them come to their own solutions.

Lately they have been wrestling through various challenges. It has been fun discussing potential options with them. Every conversation goes a little deeper. They have been learning a lot and are doing well.

Their solution looks different than how I pictured it in my mind and I think that is fantastic! It shows they are finding their own path.

The great ideas are waiting. Together we can draw them out.

Thanks FTO for being patient with us and the continuing discussions. I cannot wait until V1 ships. Thanks Daryl for being awesome and letting me learn from you. Thank you Travis for gently reminding me I don’t have a monopoly on good ideas and coming up with the phrase.

How Preflights Changed Our World

2015-09-29T18:37:07-04:00

Preflight builds changed our world. They have made master more stable and are an essential part of every pull request. A preflight build or preflight is a build which runs a portion of your CI on a branch being developed. We have introduced preflights into several projects and I can say the results are dramatic. I don’t want to merge my changes without first passing the preflight build.

It has been a while since we started using preflights. Daryl and a few others first introduced then to our bigger projects. Lately we have been trying to apply the same techniques to smaller projects.

Our preflights are similar to our complete CI process in that they build, test, package and deploy the project. Where they differ is how comprehensive they are while testing and the deployment complexity. To speed up the process we run only the most critical portions of the CI process and simplify the deployment. We want the developers to get faster feedback while still providing more thorough validation.

How did we get here?

Last year a small team of solid developers added preflights to a critical project. They made the entire process incredibly simple which I think is why it took off the way it did. The process goes like this:

Creating a pull request automatically starts a preflight build
As the preflight progresses the pull request is updated
When the preflight finishes or fails an email is sent
More preflights can be triggered as needed

We had a CI pipeline, but it would often break due to the number of developers committing to it. It also took hours to run which meant if it broke it would take a long time to get back to normal. This would become even worst if new changes were committed on top of the broken code.

Many developers would instead try to be extremely careful with their changes. They would do a great deal of extra testing and investigation up front. This was time consuming and could not cover all possible changes the developers could make. It still left room from human error and other merge problems.

Preflights allowed us to raise the bar for code before it was merged into master. As soon as the preflights were added people started using them. They were so simple to use and for typical changes would happen automatically. Quickly master became more stable and people started focusing more on the results the preflights were showing.

The started a virtuous cycle, giving the preflights more attention which led to further improvements. Contributors started to iterate on improving what was covered by the preflight tests. Flaky tests became more problematic because they would throw off test results. Over time bad tests were systematically identified and fixed.

Now almost a year later I don’t think I could work on the project without using the preflights. The added confidence and stability from the preflights is dramatic. We can easily iterate on the code and build breaks are much rarer. All developers can benefit from having their changes validated better before merging.

The Next Project

Daryl, who had been on the original team implementing preflights wanted to bring them to our team. We were about to restart a project and planned to change the architecture while keeping the overall behaviour intact. He advocated setting up preflights early in the project and thought they would help keep our code stable. We agreed and I soon set them up.

Our normal workflow was:

Make code, create pull request, merge, run CI, release, repeat.

At first the difference in our behaviour was not pronounced. Preflights slightly shifted the workflow to become:

Make code, create pull request, pass the preflight, merge, run CI, release, repeat.

As development accelerated, the preflights caught breaking changes we had not intended. Defects which would have derailed development were stopped dead in their tracks. Our master became more stable despite our active development.

As I continued to use the preflights I started to experiment. If I was confident in my changes sometimes I would kick off a preflight early to see whether what I had done was valid. The tests were more comprehensive than my local build and I could get better feedback early. This allowed me to work more asynchronously less babysitting builds. The notifications would let me know whether my changes were good or bad.

It all clicked for me when I accidentally merged too soon. I had made some hasty changes and merged prior to the preflight passing. My changes broke master in a way which would have been caught by the preflights. It has been weeks of using the preflight builds and this was the first time I could remember having broken our master.

Thanks to the preflights we rarely broke our master. The preflights made it very clear when changes would cause problems. Instead of relying on humans being careful, we could now let machines protect us from ourselves.

By the Numbers

I feel like this post is less cool without qualifying the exact improvements and how many breakages we prevented. Sadly we have been using preflights long enough and aggressively purge old build history. While I was not able to get a comparison from before the preflights, I was able to look at the statistics on our pull requests for part of September.

We had 43 pull requests including hundreds of commits which triggered over a hundred builds. 98% of pull requests had a passing build with my accidental merge being the one pull request without a passing build. We found 42% of our pull request failed one or more builds and needed more work. In other words, 18/43 pull requests were prevented from breaking our codebase.

Our failing preflights looks quite high without context. I do feel a little bad about these numbers, but given the circumstances and project they may be reasonable. We essentially rewrote the internals of a project while preserving the overall behaviour. We were very active with hundreds of commits across several developers.

I view the preflights for our project as a success. Each of those failures could have broken master and delayed the entire team. Instead failures were stopped before they were merged.

Tradeoffs and Limitations

This might sounds like rainbows and unicorns. Like everything there are tradeoffs and limitations. We learnt several lessons from the process and have more areas to improve.

Our end to end process is slower

The preflights nearly double the amount of time required to go from initial commit to deployed to production. Overlapping preflights with code reviews helps mitigate this. It is great for complex reviews which are longer than the preflight, but for simple changes the extra waiting is annoying.

We have been able to speed up the preflight feedback by running more of the process in parallel. Running in parallel has increased our complexity and led to other challenges.

Limited coverage leaves gaps

We have extended tests which require more hardware or special tools. We cannot use these special assets for every preflight. We also chose to run more important tests in the preflight and leave more comprehensive validation for the existing CI pipeline. This has meant there are gaps in our coverage where issues can sneak in.

We have iteratively improved this process by filling in the larger gaps we find and moving tests into the preflight from the CI. Running more of the build in parallel has allowed us to increase the amount we can validate in a given amount of time.

Harder Recovery

If an error does sneak through the preflight or is merged too soon it is harder to get back to normal. When errors occurred before having preflights developers would often try to fix the change by pushing more changes. With preflights the new fix must also pass a preflight which makes fixing the problem and getting back to green harder.

To counteract this we try to revert the change immediately and bypassing the preflights. In this case we know the code worked before and should continue working without the added changes. We then fix the issue in a separate pull request. Reverting problematic commits is a common practice for CI and is even more important to do when using preflights.

Dependencies add complexity

For the both the initial project and our new project additional dependencies make the preflights more complicated. The ideal would be if all changes could be validated by the preflight. This harder when incorporating third party services or managing external configuration. Coordinating updates and performing them safely is important to not break the preflights. We have had a few breaks caused by updating the configuration too soon or the project being tested changing data badly which broke existing deployments.

Our typical approaches have been to reduce the extra dependencies, keep the contracts relatively stable or version the other components. This is a big area for improvement and challenge for our projects.

Stability is paramount

Intermittent failures make the preflights less effective. The entire system, services and scripts need to be consistent. We want the pass/fail from the preflights to be a clear indication of whether your code should be merged or not. When the preflights randomly fail it is easy to lose trust in their results.

As we continue to iterate on the process we have tried to stabilize any problematic components. A common offender was sensitive integration tests which we have slowly fixed. We have added extra redundancy and monitoring to vital services.

We want to make improvements to preflights safely. This means making small changes which are also validated by the preflights. Any change which cannot be validated by the preflights should be the exception and should be tested thoroughly before being merged.

Summary

Using preflight builds has dramatically made our projects more stable. We have noticed big improvements with our first few projects. We have continued to refine how preflights interact with our processes. They have changed how we work.

I think adding preflights early will be important for new projects. We know CI is essential for all our projects. Preflight builds take this to the next level and find problems before they reach master.

If you are have troubles with stability or want a rock solid master branch then try using preflights.

Thanks to the original team who put together the first set of preflights. They have definitely saved my bacon and I appreciate your effort to get them introduced.

Thanks Daryl for promoting preflight builds within the team. It has made our lives better and kept the code stable.

Programming Practice

2015-09-23T19:34:07-04:00

Lately I have been having fun doing programming problems online. They are a nice way to go over exercises and practice various techniques.

I think it is important to program for fun. Put away your normal work and go hack on something. There is no pressure or deadlines. Programming because you want to.

My coworker wanted some feedback on some problems he had been doing. The site he used was exercism.io. It has a number of problems in different languages for various levels of difficulty. All the problems include a short description and unit tests which your solution should pass. You can see and comment on solutions by other people. The one catch is you must have first solved the problem yourself. I have been getting into some of the problems lately at the cost of my regular blog posts :(.

For a while I was competing with some friends to see who could get further on Project Euler. The site is filled with various math problems which are best solved using programming. Sure you might be able to calculate a few by hand, but the real fun is writing code to find the solution. Some problems can be brute forced whereas others need a more intelligent solution.

I also enjoy reviewing various algorithms and techniques though the regular updates from Coding for Interviews. They review some fundamental data structure or algorithm and provide sample problems. Less raw programming fun and more solid practice.

There you have it. A few handy fun sites to find new challenges:

Go code!

Bonus:

Coding for Interviews has a list of other practice sites and why they like them. I have not seen most of the sites before and think they could be fun! On their is list codewars which offers community curated problems and a ranking system. My brother in-law raves about them and really enjoys their problems.

Parameterized Tests Made Simple

2015-09-17T20:00:07-04:00

I often find my tests look very similar. The behaviour of the tests are exactly the same with different input and output data. Data driven or parameterized tests in NUnit are a great way to combine tests with different values and the same behaviour. In this post I will show you how to use NUnit’s features to create parameterized tests.

If you like this post and want to go deeper, I would like to recommend "NUnit ValueSource: the Complete Tutorial" by Robert Gowland. There are alot more examples which show how to use the various attributes together. This post is just a teaser. Go there when you are done if you want the full story.

In order to demo parameterized test I will be using a simple StringCalculator based on the String Calculator kata. The class, StringCalculator has one method Add which takes in a string containing delimited numbers as input and returns the sum of the numbers.

Written normally each test would call Add with different input and then verify the total. Using parameterized test cases we can easily add new cases which clearly show how the input and the total are related.

If I was to write these tests without using parameters I would need to repeat nearly identical code for each test. The code for the tests would look like this:

using NUnit.Framework;

[TestFixture]
public class SimpleStringCalculatorTests {

    [Test]
    public void Add_EmptyString_ReturnsZero() {
        StringCalculator calculator = new StringCalculator();

        int total = calculator.Add( "" );

        Assert.AreEqual( 0, total );
    }

    [Test]
    public void Add_SingleNumber_ReturnsTheNumber() {
        StringCalculator calculator = new StringCalculator();

        int total = calculator.Add( "1" );

        Assert.AreEqual( 1, total );
    }

    [Test]
    public void Add_TwoNumbers_ReturnsTheTotal() {
        StringCalculator calculator = new StringCalculator();

        int total = calculator.Add( "1,2" );

        Assert.AreEqual( 3, total );
    }
}

We can do better. Thankfully NUnit allows you to create parameterized tests using special attributes. Using these attributes, we can dramatically reduce the duplicate code in the tests.

In order to use these attributes you will need to do the following steps:

Promote the constant values you want to parametize into parameters on the test method
Apply the attributes to define the cases you want to test
…
Profit

Refactoring the tests above my I am left with a single test method:

using NUnit.Framework;

[TestFixture]
public class RefactoredStringCalculatorTests {

    // Warning: This will not run in its current state.
    //          We need to add the attributes first!
    [Test]
    public void Add_SimpleInputs_AddsNumbers( string numbers, int expectedTotal ) {
        StringCalculator calculator = new StringCalculator();

        int total = calculator.Add( numbers );

        Assert.AreEqual( expectedTotal, total );
    }
}

Now we are ready to explore what the different attributes do.

Complete Cases

The first set of attributes, TestCase and TestCaseSource, define complete test cases. You can think of them like rows in a table containing values for each of the test’s parameters. The test framework will call the test method one test case at a time with all the test case’s parameter values.

Both attributes apply to the test method itself. TestCase directly contains values for its test case(s) whereas TestCaseSource refers to another method/type which will supply the values. I will explain the difference more below.

TestCase Attribute

The TestCase attribute is applied directly to a single test and provide values for one test case. If you want multiple cases add more copies of the attribute! The values provided to the TestCase attribute line up with their matching parameter. More options can be configured using named parameters on the attribute, such as the test name or expected exceptions.

The main drawback to this approach is only simple compile time constants can be used as parameters. This simplicity is not all bad. It helps keep the tests focused and avoids having more complicated logic sneak into your test setup.

using NUnit.Framework;

[TestFixture]
public class TestCaseStringCalculatorTests {

    [TestCase( "", 0 )]
    [TestCase( "1", 1 )]
    [TestCase( "1,2", 3 )]
    public void Add_SimpleInputs_AddsNumbers( string numbers, int expectedTotal ) {
        StringCalculator calculator = new StringCalculator();

        int total = calculator.Add( numbers );

        Assert.AreEqual( expectedTotal, total );
    }
}

TestCaseSource Attribute

The TestCaseSource attribute is applied to the test method like the TestCase attribute. This method delegates creating parameter values to other methods, fields or types. Using the TestCaseSource is more flexible and can be used to provide more complicated parameter types. Within the source method you can reuse objects/data for multiple tests. The same source method can also be reused for multiple tests.

To use TestCaseSource you must provide a sourceName and/or sourceType. If only sourceName is used the field/method is assumed to be in the same class as the test.

The type of data provided by the source is very flexible. For simple parameters normal arrays or object[] can be returned from the source. I prefer using IEnumerable<TestCaseData> which has extra options like TestCase for configuring the test such as test name. Review the TestCaseSource documentation for the additional rules for source types.

In this example we will use "AddCases" as our source method which returns IEnumerable<TestCaseData>:

using System.Collections.Generic;
using NUnit.Framework;

[TestFixture]
public class TestCaseSourceStringCalculatorTests {

    private static IEnumerable<TestCaseData> AddCases() {
        yield return new TestCaseData( "", 0 );
        yield return new TestCaseData( "1", 1 );
        yield return new TestCaseData( "1,2", 3 );
    }

    [Test, TestCaseSource( "AddCases" )]
    public void Add_SimpleInputs_AddsNumbers( string numbers, int expectedTotal ) {
        StringCalculator calculator = new StringCalculator();

        int total = calculator.Add( numbers );

        Assert.AreEqual( expectedTotal, total );
    }
}

Single Parameters

Another option is to apply attributes to the test parameters. They can behave like TestCase and TestCaseSource for a single parameter. For tests with multiple parameters the behaviour is more complicated and can be used to create many test cases.

The following attributes are placed on a parameter and control a single parameter’s values:

Whereas the following attributes are applied to the method and define how values from multiple parameters are combined to create complete test cases:

Clear as mud? That is okay! I have a few examples to clarify how to use these attributes.

Values Attribute

This simple example shows using the Values attribute inline with a single parameter. The values from the attribute each create a single test case to be executed. This behaves like TestCase for a single parameter.

using NUnit.Framework;

[TestFixture]
public class ValuesStringCalculatorTests {

    [Test]
    public void Add_NullOrBlank_ReturnsZero(
        [Values( null, "", " ", "\t", "\n" )] string input
    ) {
        StringCalculator calculator = new StringCalculator();

        int total = calculator.Add( input );

        Assert.AreEqual( 0, total );
    }
}

ValueSource Attribute

The ValueSource provides values from a method, or field or type. Just how Values behaves like TestCase, ValueSource behaves like TestCaseSource. In this example I am using the NullOrBlankCases array to determine the values for each test case.

using NUnit.Framework;

[TestFixture]
public class ValueSourceStringCalculatorTests {

    private static string[] NullOrBlankCases = new string[] {
        null,
        "",
        " ",
        "\t",
        "\n"
    };

    [Test]
    public void Add_NullOrBlank_ReturnsZero(
        [ValueSource( "NullOrBlankCases" )] string input
    ) {
        StringCalculator calculator = new StringCalculator();

        int total = calculator.Add( input );

        Assert.AreEqual( 0, total );
    }
}

Range and Random Attributes

Range and Random are two special cases for specifying numbers.

Range generates test cases between a starting point, an end point and optionally the step size between each test case. In the example below, I use Range starting at 2 and going to 10 in steps of 2 to produce the cases 2, 4, 6, 8, 10.

using NUnit.Framework;

[TestFixture]
public class RangeStringCalculatorTests {

    [Test]
    public void Add_EvenNumbersBetweenTwoAndTen_ReturnsTheNumber(
        [Range( 2, 10, 2 )] int number
    ) {
        StringCalculator calculator = new StringCalculator();

        string numbers = number.ToString();
        int total = calculator.Add( numbers );

        Assert.AreEqual( number, total );
    }
}

Random uses random numbers to create multiple test cases. The values can optionally be constrained between two values. Be careful. The random numbers might be very large or very small which can cause your tests to randomly fail. The example below creates 5 tests cases for random numbers between 0 and 10.

using NUnit.Framework;

[TestFixture]
public class RandomStringCalculatorTests {

    [Test]
    public void Add_RandomNumberBetweenZeroAndTen_ReturnsTheNumber(
        [Random( 0, 10, 5 )] int number
    ) {
        StringCalculator calculator = new StringCalculator();

        string numbers = number.ToString();
        int total = calculator.Add( numbers );

        Assert.AreEqual( number, total );
    }
}

Combining Multiple Parameters

Cool! Dealing with a single parameter is really easy. How does it work with multiple parameters? This is where the other attributes are used to control the behaviour for combining the values from all the parameters. They are placed on the method and are mutually exclusive.

The default behaviour is to evaluate all combinations of the parameters. You can specify this behaviour explicitly using the Combinatorial attribute on the method. Be careful, this can generate many tests if you have lots of values or parameters.

You can combine values from the parameters in the order they are declared using the Sequential attribute. This is like treating the values for each parameter as a column in a table. Each row is then forms a single test case. I try to use this option sparingly. Often the parameter values do not line up visually and it can be hard to determine what the cases will be. The example below will have the following cases executed by the attribute:

””, 0
“1”, 1
“1,2”, 3

using NUnit.Framework;

[TestFixture]
public class ParameterStringCalculatorTests {

    private static string[] TestNumbers() {
        return new string[] { "", "1", "1,2" };
    }

    private static int[] TestTotals() {
        return new int[] { 0, 1, 3 };
    }

    [Test, Sequential]
    public void Add_SimpleInputs_AddsNumbers(
        [ValueSource( "TestNumbers" )] string numbers,
        [ValueSource( "TestTotals" )] int expectedTotal
    ) {
        StringCalculator calculator = new StringCalculator();

        int total = calculator.Add( numbers );

        Assert.AreEqual( expectedTotal, total );
    }
}

The last attribute which modifies how the parameters are combined is Pairwise. It uses a heuristic to create cases covering every possible pair of parameters. The intend is to generate fewer cases than Combinatorial while still providing good coverage. I have not used it on a project before. If you have too many tests caused by using Combinatorial you may want to try using this attribute.

Go Test, With Cases!

I hope you enjoyed this overview of the how to create test cases with NUnit. To make it easy to work through the examples I have uploaded a repository to github with sample code for each of the cases. I even included a few bonus examples I made along the way. Enjoy.

Test cases are great fun and can help cut down on duplication in your tests. If you think all your tests all look the same, try using test cases to clean them up.

Investing in your Code

2015-09-10T23:39:07-04:00

Taking code and transforming how it fits together without breaking it is an art. It is also great fun! However, before you start I think you should understand whether your efforts will be worthwhile. This post examines different stages in the code life cycle and how they affect how much you should invest in cleaning up code.

I like having a job. Every day when I go to work I am happy they let me in so I can code up a storm. The important difference between my job and a really kick ass hobby is getting paid. We are a business and staying in business is important for me and our customers.

Taking the time to make deep changes is a good thing. The larger the impact of the change the more important it is to understand why it will be valuable in the long term. What will the impact be? How will the change help your users?

When it comes to refactoring and maintaining code I think this especially tricky. The more entangled the code is the more I want to “fix” it. Beautiful code is something to be admired and even pursued, but only as a means to an end. At the end of the day you still have to ship and support your product.

I think it is important to think about when to update code, how much to change and what the long term benefit will be of what you are doing. Balancing your desire to do the right thing and where to focus your efforts will result in a better product with less effort.

A New Dawn

Evergreen Mountain Lookout Sunset by Michael Matti, used under Creative Commons 2.0 BY-NC

Three categories of code are worth investing: your application’s critical sections, defect filled areas and where you plan on doing work in the future. These categories often overlap and cause the code to be more important. There is a higher probability you will be working on or troubleshooting these areas in the future. Do a favour for future you and leave the code better than you found it.

Every project has features which are more important. Often this will be the reason why the project was started in the first place. The fundamental business process only this project performs or the one page every user loves. Caring for core sections of any project is worthwhile.

Important feature need to work every time. A regression in a critical tool is awful for the user experience. Having great test coverage and simplifying the code around your core features will help keep the code easy to maintain and updates as requirements change.

Larger refactoring exercises are reasonable since improvements will have a bigger impact for your users. Organic growth and tacked on features can lead to areas which do not fully make sense. Restructuring classes to better align and simplify how they interact is justified. In the critical section it is more important to prevent causing issues with your changes. Be careful.

Infamous code riddled with bugs is another good place to invest time in. This is another critical section for all the wrong reasons. The attention it gets is bad and your users will start to notice repeat issues. Defects tend to cluster together and cleaning up the whole area can be a good approach. Reducing complexity which leads to more defects is even better. However, if you don’t plan on supporting the functionality despite the defects this is dead code in disguise and you should move on.

Adding functionality and plan to add more in the future? The amount of time you plan on spending in an area or integrating with it is a good way to decide how much refactoring/cleanup you want to do. Setting up for new changes is a great reason to refactor. After you have implemented new functionality you could consolidate the existing classes.

Unsure?

New Leaves And Old by Mark Robinson, used under Creative Commons 2.0 BY-NC

Not sure what the future holds? If the area you working in is not important or only needs minor changes then spending lots of time cleaning it up is not as useful. Spending a few hours might still make sense, but days or weeks would not. The key is balancing how much time and effort you spend doing it.

What is the best thing to do if you are not sure about how much or what maintenance would be good? Play it safe and focus on smaller improvements. I would focus on the essential improvements: eliminating duplication and improving the testability.

Reducing duplication shrinks the amount of code you need to maintain. Starting with a simple refactoring like “Extracting a Method” is a good start. I recently extracted a method to encapsulate logic which had been copied repeatedly. Now there is one place to fix maintain instead of the many copies.

Find part of the code confusing? Afraid to make changes in an area? Wrapping existing code in tests can help explain what is happening and make other changes safer. Tests provide a safety net for future changes and can often be added fairly easily. Refactoring to make code testable will help loosen heavily coupled classes.

Long Frozen Over

Frozen Trees by Chad Cooper, used under Creative Commons 2.0 BY

Long forgotten code that has been left for dead should be avoided. It is okay for code to be “done” and not expect to change in the future. Any effort is probably wasted or much harder than it needs to be. Often it would be a better decision to walk away.

Clients will see little benefit from the work you do to this unloved code. No matter how much you might want to restructure the classes to perfectly separate concerns it is not worth it.

We recently declared one of our projects as dead. When we first started we wrote a large number of tests through the UI. The tests were great when we first created them, but eventually fell into disrepair as we moved onto other projects. We decided it was not worth dedicating the effort to fix all the tests right now and instead we would add new tests to eventually replace the existing code. For us it would be better to rewrite or remove the tests than maintain them.

If you must update one of these forgotten relics; get in and get out. Don’t make the project worse than it is, but don’t stick around either. Updating dead code is like shuffling deck-chairs on the Titanic. Don’t you have better things to do with your time?

Summary

Before you dive in think about what areas or projects would benefit the most from your improvements. What changes will pay off of the most?

Too many defects? Refactoring for testability and adding more tests could help prevent future defects. Are you repeating the same changes in multiple areas? Reducing duplication could help you. Try browsing the refactoring catalog for more ideas.

The answer will depend on your projects and time-lines. I hope this gave you some new ways to think about when to invest and when to back off.

In summary, I recommend scaling your efforts based on how important the code is and what you plan on doing in the area.

Invest

Critical Code? Keep it clean!
Bugs abound? Squash them down.
Plan on sticking around? Make an investment.

Minor Improvements

Not sure what to do? Stay small.
Focus on the essentials: Reduce duplication and better tests.

Stop

Left for dead? Leave it be.
Do not shuffle the deck-chairs on the Titanic.

Scott MacLellan

The Worst Week of My Life

Before

Terminus

Doubt

Acceptance

Searching

Success

Lessons

Epilogue

Vanilla JS Tetris - Good Luck, Have Fun

How to Self-Host Nancy Without Locking Your DLLs: Shadow Copying

Introduction to Dependency Injection

Dependency Inversion Principle Applied

What About Dependency Injection?

The Benefits

Dependencies For Everyone!

Dependency Injection Containers

Enter the Factories

Poor Man’s Dependency Injection Container

Awesome Containers

More Out of the Box

Connecting the Dots

Further Reading

Footnotes

Thanks

Rock-Solid PowerShell Projects

Consistent Layout

Test Thoroughly

Modules for Everything

Mandatory Continuous Integration

Summary

No Managed-Only Finalizers

What went Wrong

Finding More Finalizers

Learning More

Don’t Do It

Are you your Code Reviews?

Invested

You Are Not Your Code

Caring Professionally

Don't Use A Singleton

Intervention

The Bigger Picture

An Optimization

Not the Tool for the Job

My Time As An Exterminator

Crafting Code with C#

A simple example: A Factory

Hide Internals: A Worker Class

Inheritance: Family Planning

Inheritance: Template Class

All Action: Helper Classes

Manage state: Immutable Classes

Generics: A Different Animal

Conclusion

Do what must be done

Dependency Injection Fun in NancyFx/TinyIoC

Trouble In Paradise

Getting It Working

Going Deeper: The Root Cause

Public Service Announcement

Basic PowerShell Remoting

Being Shippable At All Times

Last Known Good

Golden Master

Risk

Next Step: Going Continuous

Why Not Both?

Safe Ordering For Breaking Changes

Drawing Ideas Out

No Monopoly on Good Ideas

The Great Ideas Are Waiting

How Preflights Changed Our World

How did we get here?

The Next Project

By the Numbers

Tradeoffs and Limitations

Summary

Programming Practice