I recently blogged about the Practical Dependency Injection 101, then DI, testing and mocking. In this post I’ll get into the concept of bootstrapping as it relates to dependency injection.

Constructor Injection

The idea of dependency injection is that classes are defined such that any dependencies on other classes or services, are injected into the class by some external mechanism, as opposed to being “newed up” directly. The most common form of DI is constructor injection, where a class defines a constructor that has as its parameters the external dependencies required by the class.

There are several benefits to this particular method of injection; the most obvious is that in a well designed system the dependencies of a class are clearly visible in the constructor. In the DI 101 post a data provider was defined like this:

public class DevDataProvider : IDataProvider {
    private readonly IIdentService _identService;
    private readonly ILogService _logSvc;
    private static readonly List<Employee> EmployeeStore = new List<Employee>();

    public DevDataProvider(IIdentService identService, ILogService logSvc) {
        if (identService == null) throw new ArgumentNullException("identService");
        if (logSvc == null) throw new ArgumentNullException("logSvc");
        _identService = identService;
        _logSvc = logSvc;
    }

    // Remaining implementation omitted for brevity
}

The constructor is on line 6. From this constructor we can see that the DevDataProvider has dependencies on an IIdentityService and an ILogService. There should be no other dependencies in the class other than to well known, stable libraries like the BCL.

There are other advantages to using constructor injection. Should the list of dependencies get too long, say longer than four parameters, you’ve got a code smell that perhaps the class is doing too much, violating the single responsibility principal.

Bootstrapping

In order to be able to resolve dependencies, the DI container must be configured. This set up is done during the bootstrapping phase. Typically this only needs to be done once, but changes to the container make sense in some scenarios like when a DI container is being used to support extensions or plug-ins. In that case components might be added or removed from the DI container while the app is running. These scenarios are out of scope for this post.

The container may be configured in several ways – Auto configuring, configuration in code, and configuration files (typically XML / app or web.config files). My current favorite DI framework is AutoFac, and I typically configure in code, but different projects will have different demands, so familiarize yourself with the specifics of your selected framework and understand the tradeoffs involved in the different types of configuration. You can even configure the DI container using more than one method – perhaps Auto configuring for the bulk of the registrations, then code or XML for more specific configuration needs.

Bootstrapping a Console Application

Depending on the type of application you’re working on, there are specific places for bootstrapping to take place. The place to do configuration and bootstrapping is sometimes referred to as the composition root (you can read about these concepts in more detail in Mark Seeman’s Dependency Injection book, published by Manning).

In a console application, the static Main() method is a typical place to configure the container. While we rarely write console apps in production (at least I rarely do), the simplicity makes it easy to see the implications of the bootstrapping procedure.

In the following sequence diagram, in step one [1] the Main() entry point is called on the console application. Main() is serving as the composition root. From there a private Bootstrap() methods is called [2] and the DI container is configured. The exact mechanism varies by framework.

Once the container is configured, the main entry point requests that the DI container resolve the App type [3]. The DI container creates whatever dependencies are required by the App [4]. This happens hierarchically; dependencies may themselves have dependencies and so on. The DI container sorts all this out and is also responsible for lifetimes of create objects etc. The DI container can create and return the instance off the App [5]. The Main() function can then pass control to the app [6] which will leverage the injected dependencies [7] to do the real work.

Only Directly Reference the DI Container in the Bootstrapper

This is an important point, and if you get nothing else from this post, understand this.

• The DI container is configured in the composition root (Main() in this case)
• The DI container is used to resolve or build the App
• The app is then run to do the work

Once the app is instantiated, it should have all of its dependencies injected. The app should not have a reference to the DI container! If we allow the app or any of its dependencies to have access to the container, then several bad things happen:

We’ve taken on a dependency to the DI Container itself

Yes its true that the assembly has a dependency on the DI container. But for purposes of this discussion the assembly is not the application. The App class and the services (other classes) it depends on is the application. We don’t want to take a dependency on the DI container in those classes; rather, we should be able to switch to a different DI container if needed and not effect the App and the dependent services.

In any kind of a significant application the app’s classes would be in a different assembly, and services might be scattered across even more assemblies, and those should not have  a dependency on a DI container. They should however be designed and built with the DI pattern in mind – with the dependencies specified in the constructor, with references to abstract types or interfaces, rather than to concrete implementations.

We’re hiding a dependency inside the App

Earlier I mentioned that a benefit of constructor injection is that the dependencies are clearly visible (even documented if you will) in the signature of the constructor. We really don’t want to see lines like this buried in the methods of the classes:

// Anti-pattern - don't use DI container except
// in composition root

var dal = Container.Resolve<IDataAccessService>();

// And defintely don't do this

var dal = new SqlDataAccessService(connectString);

A class that that has these lines buried inside somewhere has hidden dependencies on both the DI container and IDataAccessService (or worse, by using the new keyword directly, on the SqlDataAccessService). These hidden dependencies undermine the benefits of using DI containers at all.

Bootstrapping in other Application Types

Other types of apps have different places for bootstrapping and application roots. Unlike a console app, an ASP.NET MVC 3 application isn’t top-down linear, the application must respond to web requests. It does so by creating instances of controllers, and calling methods on those controllers to respond to web requests.

A controller in MVC3 is like the app was in our console example above. It will be resolved, or created, by the DI container. Controllers are different in that there will likely be several different controllers in an MVC application. Also, we don’t get to resolve a controller and tell it to run right from the composition root, the ASP.NET MVC framework will be receiving web requests and will need to resolve controllers later, after bootstrapping.

In ASP.NET MVC 3 this is accomplished by providing a hook, or a place where we can supply a DI container for ASP.NET MVC 3 to use when creating controllers. The developer configures the DI container, and then wires that container into the MVC framework via an instance of IControllerActivator. In the case of AutoFac, there’s a NuGet package called AutoFac.Mvc3 that includes classes to integrate with MVC3. The implementation details are beyond the scope of this post – just DuckDuckGo AutoFac.Mvc and find a wealth of additional detail. Same goes for WCF, WPF, and Silverlight applications. There are best practices for configuring DI containers for each app type.

DI Unfriendly Application Types

Some application types just do not lend themselves very easily to dependency injection patterns. Classic ASP.NET pops into mind immediately. It was written before Microsoft was as willing to accept OSS, community driven concepts such as DI Containers. A big red flag with ASP.NET is that all subclasses to the Page class (which is what all your ASP.NET pages are) must have a parameterless default constructor. Well there goes parameter injection!

There are other mechanisms for implementing DI patterns in this case, but they’re sub-optimal. Again I’d refer you to Mark Seeman’s Dependency Injection book, which is far and away the best DI book in the .NET space, for advice and examples in dealing with DI unfriendly application types.

In Summary

Hopefully this was helpful in your understanding of a couple of key aspects of using DI containers. Practice a few console applications, and write some tests too. Once you get the idea, move on to more interesting application types. Before long you’ll be shocked you ever wrote applications without some degree of dependency injection. Yea – it’s that good for you.

My last post provided a brief introduction into dependency injection. To review, the example included a data provider for Employee objects, which included a feature to return the object corresponding to the currently logged in user. In the end the following interfaces were defined:

IDataProvider – the function is obvious from the name. One implementation, the DevDataProvider, uses a static List<Employee> as a data store.

IIdentityService – describes a service that supplies the current identity. What current is depends on the implementation of course. A concrete WindowsIdentService defines current as the currently logged in Windows user. The TestIdentService implementation always returned the same username, which is useful for testing as we will see.

ILogService – describes a simple logging service. The ConsoleLogService implementation prints logs to the console.

Dependency Injection &Testing

For this post I’ve added a standard MSTest project and a couple of tests for the data provider. The use of dependency injection patterns in the design of this simple example allows us to easily isolate the code under test.

static IContainer afContainer;

[ClassInitialize]
public static void TestInit(TestContext ctx) {
    var idSvc = A.Fake<IIdentService>();
    A.CallTo(() => idSvc.GetCurrentUserName())
        .Returns("FAKE-ID");

    var bldr = new ContainerBuilder();
    bldr.RegisterInstance(idSvc);
    bldr.RegisterInstance(A.Fake<ILogService>());
    bldr.RegisterType<DevDataProvider>().As<IDataProvider>();
    afContainer = bldr.Build();
}

The test class has a static DI container instance, initialized in the class initializer. I’m using FakeItEasy to create a fake IIdentService at line five. Like six tells the FakeItEasy framework what to return when the GetCurrentUserName() method is called on the fake ident service. Having a fixed response makes testing the data provider a piece of cake.

I then register the fake ident service as well as a fake log service. For the log service, we don’t need to specify any behavior for the methods. The FakeItEasy framework will effectively sink any calls to the methods of the fake log service, which is fine for this test.

Lastly the data provider we want to test is registered with the DI container builder, and then container is built. The tests go on to use the DI container to resolve an instance of the data provider. The DI container will configure the data provider’s dependencies for a log service and an identity service with the fakes we built.

[TestMethod()]
public void GetCurrentEmployeeTest() {
    var e = new Employee { WindowsUsername = "FAKE-ID" };
    var dal = afContainer.Resolve<IDataProvider>();
    dal.AddEmployee(e);
    var result = dal.GetCurrentEmployee();
    Assert.AreEqual(e.WindowsUsername, result.WindowsUsername);
}

This is just a small example of using a DI container in combination with a mock / fake framework for testing. The AutoFac DI container can handle much more complex scenarios than what we’ve thrown at it here. The same is true for the FakeItEasy component. Both of these components are well used, well maintained open source projects. You can find lots of documentation and examples for both. Or you can use any number of other DI containers and mocking frameworks to achieve equivalent results.

The source code for the example is available here, and the blog entry the precedes this one is available here.

In this post we take a look at dependency injection (DI). Target audience is competent .NET developers, C# specifically (but VB’ers who read C# can benefit just as much), who’ve heard of DI but haven’t gotten around to figuring out how it fits in their day to day.

What is Dependency Injection

The first question that we need to address is: What is it that DI does for us? What problem is being solved? DI is about coupling; the degree to which program unit refers to other units. In .NET the units we’re typically worried about are classes, interfaces, components, and assemblies. Dependency injection facilitates reduction these interdependencies. Are DI patterns a silver bullet? Of course not. You can always write bad code regardless of patterns. That being said, if you’re already writing decent code and have good fundamentals, but are not using DI patterns, you’ve got the opportunity to take a leap forward.

How does DI reduce help reduce coupling? The easiest way to describe it is by diving directly into an example.

Example Scenario

We’ll work on a hypothetical in-house app where the Windows AD authenticates employees, and their Windows username is used to index a database with Employee information. It’s pretty common to see stuff like this happening in-house with line of business applications.

The example uses a provider pattern – all the data access will go through a data access provider, allowing us to build a simple provider that stores records in memory during this, our prototype phase. Theoretically we’d replace this as development continued with a provider that leverages persistent storage later.

Here’s the base level example program with no consideration for dependency injection:

class Program {
    static void Main(string[] args) {

        // ** Without using an DI Container approach **

        // Create a new provider aka data access layer
        var dal = new DevDataProvider();

        // New up an employee that's supposed to represent the currently logged in user
        var e = new Employee() {
            WindowsUsername = "thanos\\efvincent",
            EmployeeId = "0001",
            FName = "Eric",
            LName = "Vincent"
        };

        // Add it to the data access layer
        dal.AddEmployee(e);

        // See if the dal can find the current user
        e = dal.GetCurrentEmployee();

        Console.WriteLine(
            "Current logged in person is: {0}", e == null ? "unknown" : e.FName);

        // End
        Console.Write("Press any key...");
        Console.ReadKey(true);

    }
}

public class DevDataProvider {
    private static readonly List<Employee> EmployeeStore = new List<Employee>();

    public Employee GetCurrentEmployee() {
        var emp = EmployeeStore.FirstOrDefault(
            e => e.WindowsUsername.Equals(GetCurrentUserName(), StringComparison.OrdinalIgnoreCase));
        return emp;
    }

    public void AddEmployee(Employee e) {
        EmployeeStore.Add(e);
    }

    public IQueryable<Employee> Employees() {
        return EmployeeStore.AsQueryable();
    }

    private static string GetCurrentUserName() {
        var wu = WindowsIdentity.GetCurrent();
        return wu == null ? string.Empty : wu.Name;
    }
}

public class Employee {
    public string WindowsUsername { get; set; }
    public string EmployeeId { get; set; }
    public string FName { get; set; }
    public string LName { get; set; }
}

In Main() we new up the data access layer, create a new employee, and add it to our store using the data access layer. At line 21 we ask the data access layer to retrieve the employee record for the currently logged in user. Looks pretty typical, so how can IoC help? Let’s look at the coupling here – what classes are dependent on what other classes?

Our main program depends on the DevDataProvider class, and that depends on System.Security to find the Windows username of the currently logged in user. Asking the data access layer to determine the currently logged in user isn’t the best idea, but this is blog post code created to check out dependency injection, so deal with that for the moment.

Why are these dependencies undesirable? First consider how flexible this software is. Or rather, inflexible. We created a “quick” DevDataProvider that stores stuff in a static list. As we continue to build a system, we’d have to refer to DevDataProvider from more and more classes, creating a brittle, tightly coupled system. Replacing DevDataProvider becomes more of a maintenance problem.

Next think about testability. In real life there are unit tests (there should be anyway). One reason why people find excuses not to unit test is because their code is difficult to test. In this example, if we want to test DevDataProvider.GetCurrentEmployee() we have to consider that under the covers it’s calling the Windows API to get the current username. This makes that method harder to than it needs to be.

Step One – Leveraging Interfaces

In this version, we’ve factored out an interface called IDataProvider, and one called IIdentService. The IDataProvider should be pretty obvious – but IIdentService? The idea here is to decouple from the Windows API itself. A developer should understand everywhere that the application makes contact with any external modules, including the operating system, and then consider what the repercussions of that contact are. In this example, coupling to the Windows API to get then logged in username so directly is undesirable. We want to use a service that would supply us with credentials. That way if we’re testing, we can create a fake service that provides a predictable answer, and is therefore easier to test.

Coding to an interface also allows us to radically change the behavior of the service without having to alter its dependencies. If we move to a ASP.NET environment for example, we won’t want to use the current Windows Identity, we may want to use user information from the http context.

// Interface defining an identity service
public interface IIdentService {
    string GetCurrentUserName();
}

// Implementation of an identity service that returns the current
// logged in windows username
public class WindowsIdentService : IIdentService {
    public string GetCurrentUserName() {
        var wu = WindowsIdentity.GetCurrent();
        return wu == null ? string.Empty : wu.Name;
    }
}

// Interface defining a data provider service
public interface IDataProvider {
    Employee GetCurrentEmployee();
    void AddEmployee(Employee e);
    IQueryable<Employee> Employees();
}

// Our development data provider now implements the interface
public class DevDataProvider : IDataProvider {
    private readonly IIdentService _identService;
    private static readonly List<Employee> EmployeeStore = new List<Employee>();

    public DevDataProvider(IIdentService identService) {
        if (identService == null) throw new ArgumentNullException("identService");
        _identService = identService;
    }

    public Employee GetCurrentEmployee() {
        var emp = EmployeeStore.FirstOrDefault(
                        e => e.WindowsUsername.Equals(
                            _identService.GetCurrentUserName(),
                            StringComparison.OrdinalIgnoreCase));
        return emp;
    }

    public void AddEmployee(Employee e) {
        EmployeeStore.Add(e);
    }

    public IQueryable<Employee> Employees() {
        return EmployeeStore.AsQueryable();
    }
}

We’re part of the way to where we need to be. Altering DevDataProvider to depend on the IIdentService interface frees it from a hard dependency on a particular identity implementation. The downside is we’ve made creation of the DevDataProvider a bit more complex, as we need to supply the new instance with an IIdentityService instance.

// Create a new ident service, required for the DAL
IIdentService identSvc = new WindowsIdentService();

// Create a new DAL
IDataProvider dal = new DevDataProvider(identSvc);

The DevDataProvider now takes a constructor parameter of type IIdentService. This is where the injection in dependency injection comes from. DevDataProvider has a dependency, but instead of hard coding it into the definition of DevDataProvider, we inject it. There are different ways of injecting dependencies, but constructor injection is very popular and works well in many, or even most cases.

The complexity of constructing instances increases when we add a simple logging service which logs information or errors messages.

// Interface defining a logging service
public interface ILogService {
    void LogInfo(string msg, params object[] args);
    void LogError(string msg, params object[] args);
}

// Implementation of a console logging service
public class ConsoleLogger : ILogService {
    public void LogInfo(string msg, params object[] args) {
        Console.WriteLine(
            "{0} INFO: {1}", DateTime.Now,
            string.Format(msg, args));
    }

    public void LogError(string msg, params object[] args) {
        Console.WriteLine(
            "{0} ERROR: {1}", DateTime.Now,
            string.Format(msg, args));
    }
}

The ILogService is implemented by a simple console logger. Now both the WindowsIdentService and the DevDataProvider can leverage the logger. They’re both modified to have ILogService instance injected via their respective constructors.

// Implementation of an identity service that returns the current
// logged in windows username
public class WindowsIdentService : IIdentService {
    private readonly ILogService _logSvc;

    public WindowsIdentService(ILogService logSvc) {
        if (logSvc == null) throw new ArgumentNullException("logSvc");
        _logSvc = logSvc;
    }

    public string GetCurrentUserName() {
        var wu = WindowsIdentity.GetCurrent();
        var un = wu == null ? string.Empty : wu.Name;
        _logSvc.LogInfo("Identified current user as: {0}", un);
        return un;
    }
}

// Our development data provider now implements the interface
public class DevDataProvider : IDataProvider {
    private readonly IIdentService _identService;
    private readonly ILogService _logSvc;
    private static readonly List<Employee> EmployeeStore = new List<Employee>();

    public DevDataProvider(IIdentService identService, ILogService logSvc) {
        if (identService == null) throw new ArgumentNullException("identService");
        if (logSvc == null) throw new ArgumentNullException("logSvc");
        _identService = identService;
        _logSvc = logSvc;
    }

    public Employee GetCurrentEmployee() {
        var un = _identService.GetCurrentUserName();
        var emp = EmployeeStore.FirstOrDefault(
                        e => e.WindowsUsername.Equals(un,
                            StringComparison.OrdinalIgnoreCase));
        if (emp == null) _logSvc.LogInfo("Current employee {0} not found", un);
        return emp;
    }

    public void AddEmployee(Employee e) {
        EmployeeStore.Add(e);
        _logSvc.LogInfo("Added employee with id {0}", e.EmployeeId);
    }

    public IQueryable<Employee> Employees() {
        return EmployeeStore.AsQueryable();
    }
}

Now the services are getting more robust and they’re not tightly coupled to each other, they refer only to interfaces of the services they depend on. Main() however, where construction is going on, is getting messy.

static void Main(string[] args) {

    // ** Without using an DI Container approach **

    // Create a new logging service, required for uh.. everything
    ILogService consoleLog = new ConsoleLogger();

    // Create a new ident service, required for the DAL
    IIdentService identSvc = new WindowsIdentService(consoleLog);

    // Create a new DAL
    IDataProvider dal = new DevDataProvider(identSvc, consoleLog);

Finally we’re at the point where we can see the benefit of an dependency injection container. One thing about DI containers is that they’re already written. Several mature, robust, open-source DI containers exist. We’ll use AutoFac, because that’s what I’ve been using lately. We include AutoFac using NuGet.

// Create a singleton dependency injection container
private static readonly IContainer Container = ConfigureContainer();

// Configure the container
static IContainer ConfigureContainer() {
    // This is how AutoFac works. Other DI containers have similar
    // mechanisms for configuring the container
    var bld = new ContainerBuilder();

    // Register the types that implement the interfaces that are required
    // for injection. Note that we have robust control over lifetime, in
    // this case ILogService and IIdentService will be singletons, and
    // IDataProvider will provide a new instance each time it's requested
    bld.RegisterType<ConsoleLogger>().As<ILogService>().SingleInstance();
    bld.RegisterType<WindowsIdentService>().As<IIdentService>().SingleInstance();
    bld.RegisterType<DevDataProvider>().As<IDataProvider>();
    return bld.Build();
}

static void Main(string[] args) {

    // ** Using an IoC Container approach **
    var dal = Container.Resolve<IDataProvider>();

We’ve created a static instance of IContainer (an AutoFac DI container). When we start the app we configure the container, which fundamentally consists of mapping the interfaces to concrete types that will be injected. For example, line 14 specifies that when there’s a need for an instance of ILogService, we will create an instance of ConsoleLogger. It further says that the DI container should use a SingleInstance() of ConsoleLogger. This has the effect of making ConsoleLogger a singleton. In our example, both the WindowsIdentService and the DevDataProvider will be handed the same instance of ConsoleLogger.

The magic happens when we call Container.Resolve<IDataProvider>(). The container determines what concrete class to create, and it looks for any dependencies in the constructor. It will see that to build a DevDataProvider, it needs an ILogService and an IWindowsIdentService, it will recursively resolve those as well.

These are the basics of dependency injection. Even in this simple example, it should be obvious that the components and classes that we can create can be designed with very low coupling and cohesion using this technique. There are anti-patterns for using DI as well. You should endeavor to practice this technique, do research and learn the most effective uses; there’s plenty of material out there.

I hope this example was helpful. Next post I’ll extend this sample by looking at testing, and mock objects, and how DI injection can make testing far easier.

Source code samples can be found at this bitbucket repository.

Update: Follow up post – Dependency Injection, Testing, and Mocking

Two things; unlike many others (geeks and normal humans as well) I’ve been blissfully iPad free. I had an iPhone and still haven’t recovered, so there was no way I was getting a enbiggened* version of Job’s handiwork. So I can’t compare Xoom to the iPad.

I also have no Kindle, Nook, or other dedicated reading device. I had a Nook for 6 days, returned it. Just could not get into the dedicated book reading device. I need a computer that’s a computer, not just a book reader or (in spite of what Apple will tell you) a means to consume media, which is what iPad is.

There are plenty of real reviewers reviewing the Xoom. Go check them out. Mine is the point of view of a non-reviewer and programmer primarily on the Microsoft stack but a fan of other languages and platforms as well (lately Ruby and Python).

Which brings me to the Xoom. Yes, it is also an excellent means of consuming media. It’s got a 16:10 screen that’s apparently a bit bigger than the iPad’s, BUT, it is also a computer. You can program the Xoom without sending Jobs and his crew a licensing fee**. You can give your friends any software you write (yes, software. An “app” is software peeps). The software you write can access the full capabilities of this device. We all know that Android is wide open vs. iOS which is… well which isn’t. And neither is Windows Phone 7 by the way, WP7 is less open than iOS. I still like WP7 better than iOS on the technical front, but lets not get into that.

Operating System

Android is a cool OS, no doubt about that. It’s a Linux platform, not too much hiding that fact. The shell (aka UI) is smooth, gets the job done. It’ll allow you to clutter it up and make it look like crap if you see fit. System menus and utilities are easily accessible. Wireless connectivity is cake to set up, Bluetooth too. And you have programmability – readily available SDKs, full access to the system, excellent documentation to be found, and a great OSS community out there. Thumbs up.

Browser

It’s got a real browser, with tabbed pages, a normal address bar, and even a plugin model. Not all plugins work that work for Chrome, but there are some and the fact that they can be written at all is a very good thing, more will show up. Another ++point for Android acting like a computer instead of a [whatever they’re calling the iPad experience].

The Xoom’s browser scored 218 points on HTML5test.com, that’s better than IE9’s 130, but not as good as any of the other major browsers. Chrome scores 288 for example. Didn’t see an automated way of running the W3C’s tests, but suffice it to say that the Xoom’s browser has pretty decent compliance at this point.

All the web sites I visited looked good. Some sites can be confused by the user-agent and send the mobile version of the site. To fix this, you can nav to about:debug, then under the menu go to Settings… Debug… and set UAString to desktop. Now the user-agent will indicate desktop and you’ll get the full web pages. Oh and ++point for being able to put the browser in debug mode.

Loading pages (per CNet test) is faster on the Xoom than on the iPad2, so that’s good. Everything in my opinion was smooth and fast. Zooming, scrolling, etc. Thumbs up.

Flash

Go to the marketplace, find Flash, install. Done. Now you can use Flash if you really want to. Life is better without it though, Flash can dogeven on a dual core Atom, so there you go*** . Thumbs meh.

Other Stuff

The built in PDF reader is good enough for me. It remembers where you are in each file, which is the killer feature for me in a PDF reader. It’s smooth and fast, snap-zooms appropriately in either portrait or landscape to the correct page width, and has a clear, easy to use index. Thumbs up.

It’s got a book reader too, if you want to plunk money down on an e-book that traps you into a platform. This is Kindle territory, so I’m not buying books like that. I’ll by books from the publisher in PDF format, thanks. So no word on how the books thing is. Thumbs meh.

Rockplayer Light. This is a downloadable video player. Plays all kinds of formats, including MKV. If you know what I’m talking about then this is a ++point for Xoom. If you don’t then never mind. Nothing to see here. Move along.

PlayNow. Stream all kinds of internet vid to your device. Run a proggy on your PC, give it your netflix and hulu accounts. Then stream almost any internet vid to your device. This is a killer app, I use it all the time. I’m on a trial, but I’ll have to pay for this when it expires. Thumbs way up.

Conclusion

This was a crappy review. That’s why I don’t get free stuff. Xoom is cool enough for me to take 15 minutes and jot down some thoughts. Real person, real experiences, real thoughts. The Xoom is a real computer with no keyboard (you can fix that). It’s hackable, quick, and useful right out of the box. It’ll get better with more apps. Now I’ve got billable work to do. Later –e.

______________________

* one of the characters in 4 Lions said this when told the toy AK-47 he was holding while shooting his jihadist video was too small – “lemme move closer to the camera, that’ll embiggen it”. Friggin hilarious. Recommend 4 Lions.

** or Balmer and his crew. See? Fair

*** used dog as a verb. No rules on these interwebs baby!

In a previous post I described how Scott Hanselman’s presentation during PDC 2010 introduced me to several wonderful and interesting technologies that are either just now out or about to be. The first one I’ll discuss is NuGet. There are many excellent introductions to NuGet by Hanselman and others. I’ll see if I can make a meaningful contribution.

NuGet is an add-on to VS2010. It allows us to access packages that have been published either locally or publicly on the web. The main NuGet gallery just went live today at http://nuget.org. These packages make it easy to install, update, and remove libraries and tools in Visual Studio 2010.

Installation

NuGet, like dozens of other useful add-ins, is available through the Visual Studio Extension Manager, reached under Tools… Extension Manager….

From the Extension Manager choose (1) the online gallery, and then (2) search for NuGet. A restart of Visual Studio is required.

The NuGet Package Manager Console

NuGet has two user interfaces. There’s a PowerShell powered console window. This is the UI I was first introduced to and have gravitated towards. The console gives you a more complete sense of control over the process. Open the console by selecting View… Other Windows… Package Manager Console.

The Package Source drop down defaults to the official NuGet package gallery. There are already hundreds of packages out in the gallery, and you can expect it to grow rapidly. You can list the packages in the official library with:

List-Packages –remote

There are a lot of them, and since the console is PowerShell we can take advantage of a PowerShell grid display:

List-Packages –remote | out-GridView

Installing and Removing Packages

Now comes the really cool part. The whole idea of NuGet is to be able to install / add external libraries to your project without having to find them on the web, download a zip or msi file, install it somewhere on your local machine, figure out what the instructions are for incorporating them into your project, and patching it in manually.

Let’s take for example, the latest CTP5 version of Entity Framework, that allows for Code First use of the Entity Framework. EF4 will justify at least one future dedicated post, but we’ll use it here for a quick example. Normally, if you want to try out something like a new version of Entity Framework, there’s downloading, installing, shaving the chicken, bla bla bla that you’ve got to do. Not with NuGet. NuGet can effectively “xcopy deploy” it into our project.

Create a new console application in Visual Studio. From the Package Manager Console, type Install-Package EF and press the tab key. An intellisense window will pop up showing packages that begin with EF.

At the time of this writing, there are five packages. The EFCTP4 ones are obsolete, from the last CTP. For this example, choose EFCodeFirst and press enter. The console displays license info, downloads the EFCodeFirst package, and incorporates it into your project. In the solution explorer, we can see a reference  (1) and a file (2) added to the project.

Checking the properties of the reference to EntityFramework, you can see that it’s not installed to the GAC, it’s local to the solution. NuGet creates a solution level folder called packages. In that folder, in addition to components used as references, is all the data that NuGet needs to install and uninstall packages. All local, without changing any configuration on the machine. More details of the inner workings of NuGet and its packages will come at a later date, plus there’s information out there you can dig up if you’re so inclined.

Using Entity Framework

Now that we’ve got it installed, let’s quickly use EFCodeFirst just to prove it’s working. For this example, you’ll need SQL Server installed (there are other options, but one thing at a time). I’ve got Express which works fine. In another post I’ll show the new SQL Server Compact Edition 4, which coincidentally you can add to a project using NuGet .

EFCodeFirst allows you to build POCO objects and use them to build a DB. Let’s build a quick model. There are a few examples floating around modeling blog posts and comments, so lets not do that. Let’s do (super simple) prescriptions and medications. It’s on my mind because I’m recovering from knee surgery. This code is entered directly into the Program.cs source file for this example.

public class Prescription {
    public int Id { get; set; }
    public string MedName { get; set; }
    public string Directions { get; set; }
    public int Quantity { get; set; }
    public int Refills { get; set; }
    public int RefillsRemaining { get; set; }
    public ICollection<FilledScript> FilledScripts { get; set; }
}

public class FilledScript {
    public int Id { get; set; }
    public Prescription Script { get; set; }
    public DateTime Filled { get; set; }
    public int Doses { get; set; }
}

public class MedContext : DbContext {
    public DbSet<Prescription> Prescriptions { get; set; }
    public DbSet<FilledScript> FilledScripts { get; set; }
}

We have a class for a Prescription, and one for a FilledScript (filled prescription). We give each one an integer Id, which EF will interpret (by convention) as the primary key and identity for each entity. A relation is created between them, again by convention. On the “one” side,  the Prescription has a collection of filled scripts, and on the “many” side, the filled script has a reference to its prescription.

The MedContext class inherits from EF’s DbContext, and pulls the model together. Each DbSet<…> identifies an entity to EF. In this minimal case, that’s all that’s required to define a model using EFCodeFirst. Lets take a look at using it:

static void Main(string[] args) {

    DbDatabase.SetInitializer(new DropCreateDatabaseAlways<MedContext>());

    var ctx = new MedContext();
    var p = new Prescription() {
        MedName = "Vicoden",
        Directions = "One every 4 hours as needed for pain",
        Quantity = 60,
        Refills = 2,
        RefillsRemaining = 1,
        FilledScripts = new[] {
            new FilledScript() {
                Filled = DateTime.Parse("12/28/2010"),
                Doses = 0
            },
            new FilledScript() {
                Filled = DateTime.Parse("1/12/2011"),
                Doses = 48
            }
        }
    };

    ctx.Prescriptions.Add(p);
    ctx.SaveChanges();

    foreach (var script in ctx.Prescriptions) {
        Console.WriteLine("Script for {0}, filled {1} time(s), we have {2} doses on hand",
            script.MedName,
            script.FilledScripts.Count(),
            script.FilledScripts.Sum(fs => fs.Doses));
    }

    Console.Write("\nPress any key...");
    Console.ReadKey(true);
}

For this simple example I’m just adding some data and retrieving it right from the main() function of the console app. Line 3 probably the only line that needs explanation. It’s a call to the static method SetInitializer() on the static DbDatabase class of EF. The parameter is a new instance of DropCreateDatabaseAlways<MedContext>. Creating this initializer with it’s type parameter set to our MedContext tells the EF engine that it should always drop the database and recreate it, every time the app is run. This is part of EF’s new fluent interface. Under almost no circumstance would you actually use this initializer, but it’s handy for testing and playing around. This way you can iterate over your model getting a new database for each run.

The rest of the function adds a Prescription object with two FilledScript objects, and saves it in the context. It then loops through the prescriptions and writes some info to the console about each. No rocket science there.

Where did EF Create the Database?

One cool thing about EFCodeFirst is that you can leave off pretty much all the configuration information. We didn’t tell EF anything about how or where to create the database. It came up with some reasonable defaults. If you open SSMS and connect to your instance of SQL Express, you can see the database it created. Here’s a DB diagram in SSMS from what EF created:

The database it created (1) is named according to the context that it stores, MedContext for us. It created tables (2 & 3) for the entities we created, and to implement the one to many relationship, it added a foreign key to the FilledScripts table linking it to the Prescriptions table. There’s always a DBA to disagree with an auto-generated schema, but in this simple case, you must admit there’s not a whole lot you could do differently. If we select the records out of the database after our run you get the expected results:

All of this was created by convention, and can also be overridden and specified explicitly using either attributes on the classes that make up the model, or by using EF’s fluent API. This barely scratches the surface, but hopefully gets you interested. I’ll cover EF in more depth in a future post, plus there’s plenty of info out there already.

Back to NuGet – Removing a Package

Ok, we’ve seen how easy it is to add a package like EFCodeFirst (and how cool EF itself is). NuGet also allows you to remove a package easily. In the Package Manager Console window, issue the command:

PM> Uninstall-Package EFCodeFirst
Successfully removed 'EFCodeFirst 0.8' from ConsoleApplication2
Successfully uninstalled 'EFCodeFirst 0.8'

NuGet will remove all traces of the package from your project. For EFCodeFirst there’s not much to remove. But other packages are far more complicated, even pulling in prerequisite packages, and modifying the web or app.config. NuGet undoes all these changes when a package is removed.

Try it yourself!

One great thing about NuGet is how painless it makes trying out new things. You’re not fouling up your precious work machine with a bunch of installations. You can create a quick test project, add a bunch of things you’ve always wanted to try, and just blow it all away when you’re done. No evidence you’ve been learning.

Ever want to try Fluent NHibernate but didn’t feel like dealing with all the crap that goes with tracking it down and installing it? How about Ninject, Castle, ELMAH, iTextSharp, Moq, or Prism? They’re all out there. Just NuGet them and start playing / learning.

What else?

I didn’t even mention the wizardy, non-console approach. Right click on your project, and select Add Library Package Reference… and you get a pretty UI that lets you click instead of type. But you’re a coder. You can type.

Publishing your own package. You can create any package you want. Perhaps one that pulls in some of your favorite utility classes. Or perhaps you’ve got some corporate libraries and frameworks that you want to be able to incorporate using NuGet. You can create your own private or a corporate NuGet gallery. Or if you have something worth while, publish it up to the official NuGet gallery, it’s own to the public.

I hope you’ve found this ramble of mine useful. Search around for more NuGet goodness. There are plenty of bloggers way more interesting then me publishing good information.

Happy Coding!

Updated: see the NuGet post

The 2010 PDC was a few months ago. There was plenty of interesting and exciting tech news coming from the conference, but the one session that really sparked something with me was Scott Hanselman’s Presentation, ASP.NET + Packaging + Open Source = Crazy Delicious. Officially, I was mining PDC for Azure and Windows Identity Foundation material for the project I was (and still am) working on. But I love Hanselman’s presentation style; is stuff is always entertaining, stimulating, and packed with information. This one was no exception.

In a nutshell, the presentation described several different new and pre-release technologies mashed up together into a single very technical demo. All of these techs are interesting, and over the last couple of months I’ve dived into several, if not all of them and I’ve learned a whole lot. I plan on writing about these techs, mostly because when I do I end up learning a lot in the process. But also so I can promote some of these in some small way. In the coming [random time period] I’ll be posting on:

NuGet – a free, open source developer focused package management system for the .NET platform intent on simplifying the process of incorporating third party libraries into a .NET application during development. (that’s quoted from the Codeplex page). It’s way cooler then it sounds… you’ll see.

Entity Framework Magic Unicorn Edition – That’s what Hanselman called it, it’s got one of those painful Microsoft names in real life, but Magic Unicorn sounds better. It’s Entity Framework 4, the Code Only Entity Framework. EF without boxes and lines. It’s super cool and you’ll love it. Trust me.

ASP.NET MVC – Not bleeding edge, it’s been around for a couple of years now. Version 3 is in beta right now, and it’s getting tight. Time to take a look. Even if you’re not going to use it at work next week, it’s important to start thinking about web development done differently then ASP.NET (some would argue, correctly).

MVVM Pattern in WPF and Silverlight – Ok this one is not from Hanselman’s talk, but at the same time I’ve been looking at all the other tech I’ve had to dive into some WPF UI work for a project, and I figured I’d see what all the chatter was about with MVVM. I’ve spent so many years server side, I decided to approach UI with a clean slate.

I’m sure there’ll be more. I’m rehabbing from knee replacement surgery, so I have some extra time behind the keyboard. Now – on to NuGet.

Hey that’s an enthusiastic title…

My wife and son (and what the heck, me too) were surprised to see snow falling this morning as we lazed around the fire in the den. Not just some wispy dots, but some decent sized flakes. Not very common in Greensboro, NC the first week of December. There are many years it doesn’t snow at all down here. As a native Long Islander, my default reaction to snow in NC is “yea that’s not REAL snow”. But this kinda is. Very holiday joy, ho ho ho, jingle belly stuff.

So I find myself at my desk reading my RSS feeds and decide to snap a pic with my relatively new Canon S95. My primary camera is a Canon 5Dm2, but the S95 is pretty decent and fits in my pocket. I’ve been strict about taking the S95 with me pretty much whenever I take my wallet with me. Hasn’t resulted in any miracle catches yet, but it’s only a few weeks old.

Looking at the pic I have the same reaction that I usually have with the S95. “Meh…” it just doesn’t feel like a real camera to me. I grab the 5Dm2 that sits within arms reach of my keyboard and go for a quick comparison. To pics, approximately the same field of view. 5Dm2 vs S95. Here are the shots.

This wasn’t going to be much of a “guess which is which” thing, so I’m not ruining anything by saying right off the bat, the S95 is on the left. The color is different, that’s for sure, and this is after trying to get them somewhat close using white balance in Adobe Bridge. The S95 also has a 4:3 aspect ratio – that’s something that can be set, but the other ratios are achieved by cropping this one apparently. A 35mm frame is 3:2, which is what the 5Dm2 is on the right.

I used the S95 and the 5Dm2 as I would normally use them. So for the S95, that mean turning it on, finding the right focal length, adding some exposure compensation (for the darkly lit low contrast white scene). Brace my arm against the window (‘cause I don’t know what shutter speed or ISO this thing will pick), and shoot it. Shot came off at 1/125 sec, ISO 200, f4.9, +0.67 exposure comp. Not bad choices.

With the 5Dm2, I popped on my EF-100mm Macro lens cause it’s super sharp and I love it. I set ISO to 1000, exposure comp to a bit over one, and aperture to 4.5 (coincidentally, same as the S95 chose). Because I wanted it sharp. Shot came off at 1/400 sec. That shutter speed is going to make a difference. Which is why I set it that way. Which is what you do with a DSLR.

So they’re not that different shrunk all the way down to 220px. To see even more of a difference, I’ll crop it down. These crops are of a section in the lower right. They’re not at 100%, I chose a section such that both cameras would have to be reduced for the final output for web. The 5Dm2 clearly reduced more than the S95. Both had the same levels and curves applied, and both had unsharp mask 60%, 1px radius, threshold 2. Both originals where shot RAW (nice that the point and shoot has a RAW mode).

Yep that’s the DSLR vs point and shoot difference in a nutshell, for me. Now point and shoot is necessary sometimes. You want to have a pocketable camera. Just realize what you’re trading for convenience.

I’ve just finished uploading a new in-car video from the Hurricane region PCA Driver’s Ed event at Carolina Motorsports Park in South Carolina. View it embedded here, or click the link below the picture to go to Vimeo.

PCA DE CMP August 2010 from Eric Vincent on Vimeo.

Some notes about the video and data acquisition, for those interested in the gory, geeky details:

· Video was shot on a GoPro HD. Last year I tried a ContourHD, but the picture wasn’t quite as good and the sound was terrible – the GoPro does a better job dealing with the wind noise.

· The data (speedo, Tach, Throttle position) was collected using a PLX 2340 KIWI Wifi. This device interfaces with the OBD-II port of your car (also used by mechanics to read diagnostics), and transmits several different types of metrics in real time over Wi-Fi at with an observed sample rate of about 3 samples / second.

· The data was then recorded using Rev by DevToaster for iPhone. This app is compatible with the PLX, so it can collect data being transmitted by your car. It also adds accelerometer data. There’s GPS too, but the iPhone’s GPS is completely unusable for track purposes. The sampling frequency is too low, and it does some very funky interpolation which in the end renders the GPS data useless. For example, here’s a simple plot (graphed using F#) of one of the runs:

· The data is then exported from the phone as an emailed comma separated value file (.csv).

· A small program I wrote in F# reads the file and transforms the timestamps in to frame numbers (29.97 frames/second), and the data into either rotational data (for the analog gauges) or simple integers (for the digital gauges).

· That data can then be imported into Adobe After Effects. Images of the gauges (drawn from scratch in Adobe Illustrator) are animated into a video overlay using the PLX/Rev data.

· Adobe Premiere then takes the gauges and overlays them on top of the video from the track. There’s some fiddling involved to get the gauge data and video in sync. Premiere then renders the video (20 minutes took 6 hours… not sure if I’m doing that right).

· That rendered video is HD and is pretty big, so I used Microsoft Expression Encoder (Adobe has an encoder too) to re-encode the video into a smaller MP4 format that Vimeo wanted for uploading.

As C# has evolved it has acquired more and more of what some people refer to has functional programming features and constructs. One such concept is the idea that a function is a "first class” value. This is a fancy way of saying that functions are values that can be passed to and returned from other functions. A function that operates on other functions is called a high-order function.

Combinators are high order functions that compose, combine, or otherwise modify functions in useful and interesting ways. These types of operations are not typically seen in C#, but expanding your problem solving toolkit to include these concepts is not only fun and interesting, but can result in new, efficient, robust solutions.

The Timer

Imagine you’ve got a method that you want to put a stopwatch on. For example, in the following code, you need to see how long it’s taking each download to complete, for debugging information.

class Program {

    static string url;

    static void Main(string[] args) {

        url = "http://microsoft.com";
        RetrieveFromWeb();

        url = "http://amazon.com";
        RetrieveFromWeb();

        url = "http://dpreview.com";
        RetrieveFromWeb();

        Console.Write("Press any key...");
        Console.ReadKey(true);
    }

    static void RetrieveFromWeb() {
        System.Net.WebClient wc = new System.Net.WebClient();
        var s = wc.DownloadString(url);
        Console.WriteLine("URL: {0} - string length is {1}", url, s.Length);
    }
}

An ugly solution would be to do this.

static void Main(string[] args) {
    var sw = new System.Diagnostics.Stopwatch();

    sw.Start();
    url = "http://microsoft.com";
    RetrieveFromWeb();
    Console.WriteLine("{0:#,##0}ms", sw.ElapsedMilliseconds);

    sw.Restart();
    url = "http://amazon.com";
    RetrieveFromWeb();
    Console.WriteLine("{0:#,##0}ms", sw.ElapsedMilliseconds);

    sw.Restart();
    url = "http://dpreview.com";
    RetrieveFromWeb();
    Console.WriteLine("{0:#,##0}ms", sw.ElapsedMilliseconds);

    Console.Write("Press any key...");
    Console.ReadKey(true);
}

Try not to get physically ill. You and I both know there’s plenty of code running around in the wild that looks a lot like this. Red flags – you see repeated code – starting, stopping, printing the time. Should we go into the RetrieveFromWeb() method and modify it for timing? Let’s not. Lets try this instead. Step one, let’s define a variable for the method we’re calling. Leaving out the timing code for now, we make this change:

static void Main(string[] args) {

    // Create a delegate, set it to the method of interest

    Action retrieveFromWeb = RetrieveFromWeb;

    url = "http://microsoft.com";
    retrieveFromWeb();

    url = "http://amazon.com";
    retrieveFromWeb();

    url = "http://dpreview.com";
    retrieveFromWeb();

    Console.Write("Press any key...");
    Console.ReadKey(true);
}

I’ve added a local variable of type Action, and set it equal to the call to the method RetrieveFromWeb(), and now we’re calling that method indirectly. It has the same exact effect, the method gets called three times. Only we’ve added a layer of indirection. We do this all the time in OO programming; for example, you might have a Person object, but rather than coding directly to that object, you create an IPerson interface, and code to that, opening up the possibility of mocking the object, decorating it, etc. Similar thing here. Rather than “binding” the call sites directly to the method, we’re binding to a variable that points to the method.

We’re doing this because now we’ve got a value that can be altered or augmented to add additional functionality. This is where a combinator comes in. This is a simple timer combinator:

class Combinators {

    public static Action Time(Action a) {
        return () => {
            var sw = new System.Diagnostics.Stopwatch();
            sw.Start();
            try {
                a();
            } finally {
                Console.WriteLine("{0:#,##0}ms", sw.ElapsedMilliseconds);
            }
        };
    }
}

This combinator takes an Action and returns a new action (aka a new function), that has timing included. At line 8 the parameter action is being called, the timing is what’s added. The only change we have to make to the main method is where the delegate is being defined:

Action retrieveFromWeb = Combinators.Time(RetrieveFromWeb);

The rest of the method stays the same, but now, there’s timing added. This is a super-simple, contrived example, but you should be starting to see what’s possible. Let’s take this a step further. Here’s an example simulating retrieval of information from the database:

static void Main(string[] args) {

    Func<string, Guid> lookupUser = LookupUser;

    var emails = new[] {
        "eric@work.com", "joel@office.com", "cole@school.com",
        "karin@job.com", "haley@home.com" };

    foreach (var em in emails) {
        var id = lookupUser(em);
        Console.WriteLine("user {0} id = {1}", em, id);
    }

    Console.Write("Press any key...");
    Console.ReadKey(true);
}

static Guid LookupUser(string email) {
    // Fake looking up a user in the database
    var rnd = new Random(System.DateTime.Now.Millisecond);
    Thread.Sleep(rnd.Next(250, 2500));
    return Guid.NewGuid();
}

Memoization is the idea that a function can remember the results for given parameters. It’s like caching, but the term is specific to caching function results based on input. Here’s a simple Memoization combinator for C#:

public static Func<A, B> Memoize<A, B>(Func<A, B> fn) {
    var dict = new Dictionary<A, B>();
    return a => {
        B b = default(B);
        if (!dict.TryGetValue(a, out b)) {
            b = fn(a);
            dict.Add(a, b);
        }
        return b;
    };
}

It’s simple, but it demonstrates some very useful and interesting functional programming concepts. First, it’s takes and returns Func<A,B>. This is a delegate with a parameter of type A that returns a type B. This will work for effectively any method with that signature. Next point of interest, a dictionary is created, then the lambda is created and returned. The lambda refers to the dictionary defined outside the lambda. It is said that the dictionary is captured in a closure. It’s not important that you remember the terms, but look over the code and see if the concept is clicking for you. This function will return (effectively) a function, the dictionary is captured by that function. Even when the call to Memoize() goes out of scope, the dict variable will still exist in the returned function. Enough talk. We modify the main program just slightly:

Func<string, Guid> lookupUser = Combinators.Memoize(LookupUser);

The Memoize function will create a new function, one that caches results of the LookupUser function automatically. Nothing else has to change in the program to take advantage of this. Want to be sure that it’s actually working? Time it! The non-memoized LookupUser() has a built in Thread.Sleep(2500), and so takes 2.5sec * number of lookups to run. The memoized version will run almost instantly, so we can prove the memoizer is working by timing. The time combinator we created earlier was for timing Actions. I’ve created a overload of the time combinator that has the signature we need – Func<A,B>:

public static Func<A, B> Time<A, B>(Func<A, B> fn) {
    return a => {
        var sw = new System.Diagnostics.Stopwatch();
        sw.Start();
        try {
            return fn(a);
        } finally {
            Console.WriteLine("{0:#,##0}ms", sw.ElapsedMilliseconds);
        }
    };
}

and added it to the definition of lookUpUser in the main function

Func<string, Guid> lookupUser =
    Combinators.Time(
        Combinators.Memoize<string,Guid>(LookupUser));

Again without modifying the main code, we’ve augmented the method. Throw in some duplicate email addresses to test it out. The real power of these techniques becomes more evident the more you use them. Functional programming developers have been using these are similar techniques for years, and now with the added ability and flexibility of C#, we can employ these patters as well.

I was in one of those curious moods the other day, and decided to check out the API for the service behind my RSS reader, FeedDemon. I like this reader because it keeps my subscriptions and read / unread data synchronized across machines and my iPhone via an app called NetNewsWire.

Well it ends up that Google has gobbled up the service and it’s now Google Reader. Fair enough. It means you can also read your RSS feeds on the web at the Google reader site. Whatevs. It also turns out that Google has not publicized the API for that service yet. Yuck. But there are a couple of people out there (like here and here) who have picked it apart using fiddler or some such thing.

Well to put this meandering story to an end, I was playing with the API and it seems that some of the methods only return JSON. Others return XML, and others still are switchable. Messy. No wonder it’s not public yet. So that brings me to the point of this post… Parsing Json using F#.

Why F#?

I’ve avoided posting about F# to date, even though I love “playing” with it; I feel that posts about C# are more relevant. People are using F#, but I’ve never had the chance to use it on the job, and I don’t know anyone personally who has either. But since it’s mainstream now, what the heck. Plus parsing is one of those tasks that’s right up F#’s alley. If you haven’t worked with a functional language since college (or ever), give it a whirl. It’s refreshingly different from pure, straight, intensely object oriented thinking.

Json, Briefly

Most everyone knows what Json is by now. It’s just a tad more horrifying in real life as the supernatural killer from the Friday the 13th movies with whom it homophonetically shares its name. It’s a text format for representing data that can be evaluated in that wonderfully fast and lose language of the web, JavaScript, resulting in actual JavaScript objects. Read about it from the experts here. Here’s an example of some awesome Json:

{
    "glossary": {
        "title": "example glossary",
        "GlossDiv": {
            "title": "S",
            "GlossList": {
                "GlossEntry": {
                    "ID": "SGML",
                    "SortAs": "SGML",
                    "GlossTerm": "Standard Generalized Markup Language",
                    "Acronym": "SGML",
                    "Abbrev": "ISO 8879:1986",
                    "GlossDef": {
                        "para": "A meta-markup language, used to create markup languages such as DocBook.",
                        "GlossSeeAlso": ["GML", "XML"]
                    },
                    "GlossSee": "markup"
                }
            }
        }
    }
}

The Tokenizer

I’m no parsing / compiler / language expert, but I do know that tokenizing is the first step. This is when we run through the string and create tokens; identifying the open and close braces, the name value pairs, etc.

First thing we want to look at is the Token type, which is a discriminated union. This type defines the logical things that are in the Json string. For example, an open quote, some characters, and a close quote is a string.

type Token =
  | OpenBracket | CloseBracket
  | OpenArray | CloseArray
  | Colon | Comma
  | String of string
  | Number of string

For this simple parser, the tokens are as above, open and close bracket and square bracket (aka array), colon, comma, numbers, and strings. This is a good reason to start with Json, it’s painfully simple. The tokenize function below turns the string into a list of tokens.

let tokenize source = 

  let rec parseString acc = function
    | '\\' :: '"' :: t -> // escaped quote
                          parseString (acc + "\"") t
    | '"' :: t -> // closing quote terminates
                  acc, t
    | c :: t -> // otherwise accumulate
                parseString (acc + (c.ToString())) t
    | _ -> failwith "Malformed string."

  let rec token acc = function
    | (')' :: _) as t -> acc, t // closing paren terminates
    | (':' :: _) as t -> acc, t // colon terminates
    | (',' :: _) as t -> acc, t // comma terminates
    | w :: t when Char.IsWhiteSpace(w) -> acc, t // whitespace terminates
    | [] -> acc, [] // end of list terminates
    | c :: t -> token (acc + (c.ToString())) t // otherwise accumulate chars 

  let rec tokenize' acc = function
    | w :: t when Char.IsWhiteSpace(w) -> tokenize' acc t   // skip whitespace
    | '{' :: t -> tokenize' (OpenBracket :: acc) t
    | '}' :: t -> tokenize' (CloseBracket :: acc) t
    | '[' :: t -> tokenize' (OpenArray :: acc) t
    | ']' :: t -> tokenize' (CloseArray :: acc) t
    | ':' :: t -> tokenize' (Colon :: acc) t
    | ',' :: t -> tokenize' (Comma :: acc) t
    | '"' :: t -> // start of string
      let s, t' = parseString "" t
      tokenize' (Token.String(s) :: acc) t'
    | '-' :: d :: t when Char.IsDigit(d) -> // start of negative number
        let n, t' = token ("-" + d.ToString()) t
        tokenize' (Token.Number(n) :: acc) t'
    | '+' :: d :: t | d :: t when Char.IsDigit(d) -> // start of positive number
        let n, t' = token (d.ToString()) t
        tokenize' (Token.Number(n) :: acc) t'
    | [] -> List.rev acc // end of list terminates
    | _ -> failwith "Tokinzation error"

  tokenize' [] source

We don’t have time for the full treatment of F#. If there’s any interest, I’ll post up some tutorials or at least links to some of the very many existing good tutorials out there already. For now, assume we have an understanding of the syntax, and the logic of tokenizing and parsing is what we’re after here.

One of the first things we see is that tokenize defines a few functions within the function. Great feature of F#, allowing definitions of inner functions; it allows the coder to partition logic without having function explosion.

The driving inner function is tokenize’, whose signature is (Token list –> char list –> Token list). This means it takes a Token list and a char list, and returns a Token list. The first token list (acc) is an accumulator. This list is built up as the procedure calls itself recursively. This is a common pattern in functional languages, to “thread” an accumulator through recursive calls. The second parameter is a char list, or a list of characters from the source string.

We don’t see the char list explicitly because of the use of the  function keyword at like 20, this keyword means that the last parameter (the char list) is implicitly used in a match statement, the cases of which follow after the function keyword. The syntax would be equivalent to:

  let rec tokenize' acc sourceChars =
    match sourceChars with
    | w :: t when Char.IsWhiteSpace(w) -> tokenize' acc t   // skip whitespace
    | '{' :: t -> tokenize' (OpenBracket :: acc) t
    | '}' :: t -> tokenize' (CloseBracket :: acc) t
...

The idea that the sourceChars variable being passed immediately to the match..with construct is so common in F# that the function keyword is used as a contraction, allowing the elimination of what is an unnecessary variable. In many cases, this construct allows for a one line function definition.

Most of the cases in the block matches a character which in turn maps to a token, which is appended to the accumulator (acc : Token list), and passed to a recursive call of tokenize’. This way the tokens are built up (in reverse order) as the string is traversed. This is seen in lines 22-27. Line 21 skips whitespace by calling tokenize’ without adding a token to the accumulator first.

Things are a bit more interesting at lines 28, 31, and 34. Line 28 detects the beginning of a string, and starts a new recursive thread with the parseString function, the signature of which is (string –> char list –> string * char list). So the accumulator is a string, the “work” is being done on a char list, to which is passed our source char list, and the return is a tuple of string * char list, which is our parsed string and the rest of the source char list. One (of I’m sure many) optimizations that could be made is to use a mutable StringBuilder as the accumulator in parseString.

The token function on line 12 does a similar job for non-string literals (numbers in this case, there are no bools in Json).

Lastly, line 37 handles the case where we run out of source characters. The Token list we’ve been accumulating is backwards; we’ve been “appending” to the front of the list all this time. This is a common pattern. At the end, we return the reverse of the list. When the Json at the top is put through the tokenizer, we get this:

> let tk = tokenize source;;

val tk : Token list =
  [OpenBracket; String "glossary"; Colon; OpenBracket; String "title"; Colon;
   String "example glossary"; Comma; String "GlossDiv"; Colon; OpenBracket;
   String "title"; Colon; String "S"; Comma; String "GlossList"; Colon;
   OpenBracket; String "GlossEntry"; Colon; OpenBracket; String "ID"; Colon;
   String "SGML"; Comma; String "SortAs"; Colon; String "SGML"; Comma;
   String "GlossTerm"; Colon; String "Standard Generalized Markup Language";
   Comma; String "Acronym"; Colon; String "SGML"; Comma; String "Abbrev";
   Colon; String "ISO 8879:1986"; Comma; String "GlossDef"; Colon; OpenBracket;
   String "para"; Colon;
   String
     "A meta-markup language, used to create markup languages such as DocBook.";
   Comma; String "GlossSeeAlso"; Colon; OpenArray; String "GML"; Comma;
   String "XML"; CloseArray; CloseBracket; Comma; String "GlossSee"; Colon;
   String "markup"; CloseBracket; CloseBracket; CloseBracket; CloseBracket;
   CloseBracket]

This list of tokens is an intermediate step. From here, we could go in several directions. For my purposes, I wanted to see it as XML. To get there, I created a parser that takes the token list and returns an XElement.

let parseToXml source =
  let map = function
    | Token.Number(n) -> n.ToString()
    | Token.String(x) -> x
    | v -> failwith "Syntax Error, unrecognized token in map()"

  let rec parseValue (acc:XElement) = function
    | OpenBracket :: t ->
      let newElement, t' = parseElement acc t
      newElement, t'
    | OpenArray :: t ->
      let name = acc.Name.LocalName
      if name.EndsWith("ies") && name.Length > 3 then
        let childName = name.Substring(0, name.Length - 3) + "y"
        let newListElement, t' = parseArray childName acc t
        newListElement, t'
      elif name.EndsWith("s") && name.Length > 1 then
        let childName = name.Substring(0, name.Length - 1)
        let newListElement, t' = parseArray childName acc t
        newListElement, t'
      else
        let childName = acc.Name.LocalName
        acc.Name <- XName.Get(childName + "s")
        let newListElement, t' = parseArray childName acc t
        newListElement, t'
    | h :: t ->
      acc.Value <- map(h)
      acc, t
    | _ -> failwith "bad value"

  and parseArray name acc = function
    | Comma :: t -> parseArray name acc t
    | CloseArray :: t ->  acc, t
    | t ->
      let newElement = XElement(XName.Get(name))
      let acc', t' = parseValue newElement t
      acc.Add(acc')
      parseArray name acc t'

  and parseElement (acc : XElement) = function
    | Comma :: t ->
      parseElement acc t
    | Token.String(n) :: Colon :: t ->
      let newElement = XElement(XName.Get(n))
      let v, t' = parseValue newElement t
      acc.Add(v)
      parseElement acc t'
    | CloseBracket :: t ->
      acc, t
    | _ -> failwith "Malformed JSON object"

  let root = XElement(XName.Get("root"))
  let tokens = tokenize source

  match tokens with
    | OpenBracket :: t ->
      let result, t' = parseElement root t
      result
    | _ -> failwith "Json did not begin with an object"

There is still syntax checking happening in the parse step. The tokenizer would have found illegal tokens, but the parse function will find illegal Json structure. It works in a similar way to the tokenizer, except instead of a raw stream of characters, we’ve got a stream of tokens, which is examined one by one and passed to a set of recursive functions which accumulates the XML, in this case as an XElement.

parseToXml defines inner functions map, parseValue, parseArray, and parseElement. It then gets to work by creating a root XElement, and getting the Token list. Json should start with an OpenBracket token. Anything else and we failwith an error (like C#’s throw). When we find that OpenBracket, the recursion begins with a call to parseElement. The root XElement serves as the accumulator, and the Token list is the “work” to be done.

You should be able to see what’s happening if you’re somewhat comfortable with F#. The parseValue function does some extra work to wrap arrays in nodes where the singularity / plurality of the node makes it read a bit better (in English, most of the time). The other point of interest is that the accumulator is a mutable object in parseToXml, unlike the Token list was in tokenize. At lines 37 and 46, the newly created elements are added to the accumulator using the Add(XElement) method, which mutates the accumulator and keeps using it. This is a departure from “purer” functional techniques, but that’s what’s cool about F#, you can make those departures where it makes sense. In this case, leveraging the XElement class of the .NET framework was worth it. Consuming this class from C# is that much easier and more intuitive.

After all is said and done, this is the XML that is emitted.

<root>
  <glossary>
    <title>example glossary</title>
    <GlossDiv>
      <title>S</title>
      <GlossList>
        <GlossEntry>
          <ID>SGML</ID>
          <SortAs>SGML</SortAs>
          <GlossTerm>Standard Generalized Markup Language</GlossTerm>
          <Acronym>SGML</Acronym>
          <Abbrev>ISO 8879:1986</Abbrev>
          <GlossDef>
            <para>A meta-markup language, used to create markup languages such as DocBook.</para>
            <GlossSeeAlsos>
              <GlossSeeAlso>GML</GlossSeeAlso>
              <GlossSeeAlso>XML</GlossSeeAlso>
            </GlossSeeAlsos>
          </GlossDef>
          <GlossSee>markup</GlossSee>
        </GlossEntry>
      </GlossList>
    </GlossDiv>
  </glossary>
</root>

It isn’t the most sophisticated, performant, or I’m sure accurate Json parser out there. But it does the trick for my immediate need, is a good starting point for more sophistication later if needed, and was an excellent exercise in parsing and F#. It was fun to write, and hopefully it stimulates some curiosities. Let me know if you’ve got any questions or comments.