Round the Campfire

Creating composable UI with F# and Prism

2011-10-15T20:17:00.000+01:00

I’ve seen a lot of great demos recently of F# being used to create UI for WPF, Silverlight and Windows Phone. In all cases, this has been a very lightweight UI developed for one specific purpose. This is certainly a valid way to develop user interfaces – no need to overcomplicate, but I wondered if there is also a place for a more “grown up” solution, suitable for building complex applications from the ground up, using F#.

I have recently created a number of full blown user interfaces in C# using Microsoft Patterns & Practices Prism library. This library provides the building blocks for building maintainable yet complex user interfaces for WPF, Silverlight, Windows Phone. This is typically achieved through a central “shell” application, and a number of loosely coupled modules which contain parts of the user interface and other services.

I am going to attempt to write an F# silverlight application using Prism. This series will describe that process, but won’t necessarily go into detail about how Prism applications themselves work. I would highly recommend that anyone new to Prism watch Mike Taulty’s excellent series of videos on this subject, and then refer to the MSDN Developer’s Guide to Microsoft Prism.

Step 1 – Create the Shell Application

The first step will be to create the basic shell application, without any content. This will be a silverlight application, using F#, so ensure you download Microsoft Silverlight 4 Tools for Visual Studio 2010. I also installed the F# Silverlight Application Template so that I could create the initial silverlight application easily.

With the above installed, I created a silverlight app project named FSharpPrismShell and a Web application project, named Web.Host as a very basic host for this single silverlight app.

When I tested this basic setup, I discovered that the web application project, whilst allowing me to add my silverlight application in the “Silverlight Applications” tab of project properties, didn’t actually copy this over to ClientBin for some unknown reason. Instead, I just added a manual copy to the pre-build events:

xcopy /Y /R $(SolutionDir)FSharpPrismShell\Bin\Debug\FSharpPrismShell.xap $(ProjectDir)ClientBin

This allowed me to launch my empty silverlight application correctly in the browser.

Step 2 – Make the Shell Application a Prism Shell

Now we need to add Prism to the silverlight application project. I will do this using NuGet. If you don’t have NuGet yet, you should download and install this (in VS2010 use Tools –> Extension manager, then search for and install NuGet Package Manager).

From the Silverlight application project, right click and select Manage NuGet Packages:

Search the online packages for Prism. This should give the above list, so select and install Prism and Prism.UnityExtensions using this.

The contruction of all the elements required in a Prism application is managed by a Bootstrapper. I now need to change my application initialisation to create and run this Bootstrapper, which in turn will create and initialise the shell view (ie the root silverlight control).

So, I’ll firstly modify the default Page.xaml/MainPage combination to create Shell.xaml and Shell.fs.

<Grid x:Name="LayoutRoot" Background="Red">
    <TextBlock HorizontalAlignment="Center" VerticalAlignment="Bottom">The red area is from the Shell</TextBlock>
</Grid>

namespace FSharpPrismShell
open System
open System.Windows 
open System.Windows.Controls

type Shell() as this =
    inherit UserControl()
    do
        Application.LoadComponent(this, new System.Uri("/FSharpPrismShell;component/Shell.xaml", System.UriKind.Relative))
    let layoutRoot : Grid = downcast this.FindName("LayoutRoot")

    do 
        ()

Note that I have made the background red, and added the textbox to allow us to later see what belongs to the shell, and what belongs to the plug-in module. Next I’ll create the bootstrapper PrismBootStrapper.fs:

namespace FSharpPrismShell
open System
open System.Windows 
open System.Windows.Controls
open Microsoft.Practices.Prism.UnityExtensions
open Microsoft.Practices.Prism.Modularity

type PrismBootStrapper() =
    inherit UnityBootstrapper()

    override this.InitializeShell() =
        base.InitializeShell()
        Application.Current.RootVisual <- downcast this.Shell
        ()

    override this.CreateShell() =
        new Shell() :> DependencyObject

Finally I need to modify Program.fs to initialise and run this bootstrapper, rather than creating the shell directly itself:

let startup () =
    let bootStrapper = new PrismBootStrapper()     
    bootStrapper.Run()

I can now build and run this application. Note: I’m observing another weird issue – I seem to need to rebuild all before running each time, otherwise the browser launches with the previous version. Not sure if this is an issue with cassini or a problem with the pre-build steps.

Therefore, I have a working Prism shell application. The next step is to create a plug-in module for this, and wire the two halves together, so that the plug-in appears within this shell.

Step 3 – Create the Basic Plug-in Module

The most basic plug-in for Prism is a module which exposes a single view to be composed within the shell. This is what we will implement today. Create a second silverlight application project and name this one FSharpPrismModule.

Refactor the default Page.xaml/MainPage combination into MainView.xaml and MainView.fs:

<Grid x:Name="LayoutRoot" Background="Green">
    <TextBlock HorizontalAlignment="Center" VerticalAlignment="Center" FontSize="36">Hello, Modular UI</TextBlock>
    <TextBlock HorizontalAlignment="Center" VerticalAlignment="Bottom">The green area is from the Module</TextBlock>
</Grid>

type MainView() as this =
    inherit UserControl()
    do
        Application.LoadComponent(this, new System.Uri("/FSharpPrismModule;component/MainView.xaml", System.UriKind.Relative))
    let layoutRoot : Grid = downcast this.FindName("LayoutRoot")

    do 
        ()

Now create the discoverable type which implements IModule – this is the core of a Prism module, and will be used to register this module with the shell application later. We’ll call this ModuleDef.fs:

type ModuleDef(regionManager: IRegionManager) =

    interface IModule with
        member this.Initialize() =
            regionManager.RegisterViewWithRegion("ShellContent", typeof<MainView>) 
            |> ignore

What we do here is register a view which Prism will automatically use for any regions defined in the shell named “ShellContent”. Also note that the constructor parameter of type IRegionManager is automatically generated by Prism/Unity when the module is loaded. This is true of any interfaces registered with unity, and is the standard method for providing core services to loaded modules.

This will build and run fine, but we won’t yet see the module for two reasons – it hasn’t been registered with Prism, and we haven’t defined a region in the shell into which its UI should be placed.

Step 4 – Register the Module with the Shell

Firstly we need to update Shell.xaml to add a control which will act as the placeholder, into which Prism will inject the view from our module:

...
xmlns:Regions="http://www.codeplex.com/prism"
...

<Grid x:Name="LayoutRoot" Background="Red">
    <ContentControl Margin="20,20"
                 HorizontalAlignment="Stretch" VerticalAlignment="Stretch"
                 HorizontalContentAlignment="Stretch" VerticalContentAlignment="Stretch"
                 Regions:RegionManager.RegionName="ShellContent" />
    <TextBlock HorizontalAlignment="Center" VerticalAlignment="Bottom">The red area is from the Shell</TextBlock>
</Grid>

Now we need to create a module catalog, which is a xaml file defining the list of modules which Prism should make available as plug-ins. Note that this can also be achieved programmatically within the bootstrapper. We’ll create a file called modules.xaml and include this as content within the shell project.

<Modularity:ModuleCatalog 
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:sys="clr-namespace:System;assembly=mscorlib"
    xmlns:Modularity="clr-namespace:Microsoft.Practices.Prism.Modularity;assembly=Microsoft.Practices.Prism">
    <Modularity:ModuleInfo
        Ref="FSharpPrismModule.xap"
        ModuleName="FSharpPrismModule"
        ModuleType="FSharpPrismModule.ModuleDef, FSharpPrismModule, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null">
    </Modularity:ModuleInfo>
</Modularity:ModuleCatalog >

Note that we are actually including the FSharpPrismModule xap file here, and specifying the type to be loaded within this xap file. Prism works out all the complexity of one xap file (the shell) including another xap file (the module) on our behalf.

In order for this to be found, I will need to update the prebuild step to xcopy this new xap file into the ClientBin directory

xcopy /Y /R $(SolutionDir)FSharpPrismModule\Bin\Debug\FSharpPrismModule.xap $(ProjectDir)ClientBin
xcopy /Y /R $(SolutionDir)FSharpPrismShell\Bin\Debug\FSharpPrismShell.xap $(ProjectDir)ClientBin

This said, I was initially banging my head against a wall when the runtime continued to fail to load FSharpPrismModule.ModuleDef. Eventually (with help from .NET Reflector) I realised that the generated module had the version number 0.0.0.0, so I added an AssemblyInfo.fs file to the module project:

open System.Reflection
open System.Runtime.CompilerServices
open System.Runtime.InteropServices

// The following GUID is for the ID of the typelib if this project is exposed to COM
[<assembly: Guid("06725367-D5EE-4D48-BC6B-827211C133A6")>]
[<assembly: AssemblyVersion("1.0.0.0")>]
[<assembly: AssemblyFileVersion("1.0.0.0")>]

()

In order for the module itself to be loaded, we need to ensure that the bootstrapper knows how to find the modules. In order to achieve this, I added another override to PrismBootStrapper.fs:

override this.CreateModuleCatalog() =
    let catalog = ModuleCatalog.CreateFromXaml(new Uri("modules.xaml", UriKind.Relative))
    catalog :> IModuleCatalog

So finally, Prism can find our module, load it, and determine that it can provide a view for the placeholder control within our shell. If we now run this application, the result will be:

That’s enough Prism for now. What we have is a very simple composed user interface written entirely in F#. It isn’t very useful yet, as we only have one module, we don’t have any additional services, or communication between the shell and the module or between modules. I’ll endeavour to expand this into something more useful in the near future, but for the moment this is a reasonable start.

Source code is available for download from bitbucket.

Creating a no-thrills custom type provider in F# 3.0 – some progress

2011-09-18T12:22:00.000+01:00

I thought I would come at this from a different angle today. Following on from comments on the previous post, I decided it would make sense to try and find out what F#/Visual Studio does when I add a reference to my type provider.

So I have added a separate solution TypeProviderConsumer and opened that in another instance of VS 11. I create an empty F# console app project, and then go back to my type provider solution. Having corrected the omission that Ramon mentioned in a comment on my previous post, my type provider now looks like this:

using System;
using System.Reflection;
using Microsoft.FSharp.Core;
using Microsoft.FSharp.Core.CompilerServices;
    
[Serializable, TypeProvider, CompilationMapping(SourceConstructFlags.ObjectType)]
public class BasicTypeProvider : ITypeProvider, IProvidedNamespace, IDisposable
{
    Type baseType;

    public BasicTypeProvider(TypeProviderConfig config)
    {
        baseType = typeof(StaticType);
    }

    public Type ApplyStaticArguments(Type typeWithoutArguments, string typeNameWithArguments, object[] staticArguments)
    {
        return baseType;
    }

    public System.Linq.Expressions.Expression GetInvokerExpression(System.Reflection.MethodBase syntheticMethodBase, System.Linq.Expressions.ParameterExpression[] parameters)
    {
        return null;
    }

    public IProvidedNamespace[] GetNamespaces()
    {
        return new IProvidedNamespace[] { (IProvidedNamespace) this };
    }

    public System.Reflection.ParameterInfo[] GetStaticParameters(Type typeWithoutArguments)
    {
        return new ParameterInfo[] { };            
    }

    public event EventHandler Invalidate;

    public void Dispose()
    {
    }

    public IProvidedNamespace[] GetNestedNamespaces()
    {
        return new IProvidedNamespace[] { };
    }

    public Type[] GetTypes()
    {
        return new Type[] { baseType };
    }

    public string NamespaceName
    {
        get { return "TestTypeProvider"; }
    }

    public Type ResolveTypeName(string typeName)
    {
        if (string.Equals(typeName, "StaticType"))
        {
            return this.baseType;
        }
        return null;
    }
}

I set breakpoints in every method and property above, then choose Debug, Attach to Process and attach to the other instance of Visual Studio:

Switching back to the other instance, I add a reference to my custom type provider, and bam, we hit the first breakpoint. Awesome. Progress.

Checking out what config gets passed into our constructor:

If I keep hitting F5, it hits the breakpoints in this order:

GetNamespaces()
NamespaceName { get; }
GetTypes()
GetNestedNamespaces()

Then it repeats once more, starting at the constructor.

This is all good news – it means that I’m not missing some strange attribute or other construct. My type provider is being initialised and queried, so I guess at the moment I get the “This is not a provided type” message because I’m not implementing some or all of ITypeProvider or IProvidedNamespace correctly. This is something I can work on by getting my head back into FSharp.Data.TypeProviders. Watch this space.

As an aside – notice the TemporaryFolder property in the config above. If I take a look into the location to which it points, I find a number of generated dlls for the past few days. It turns out these are generated by the Linq to SQL type provider (which I have been testing alongside this investigation) in pairs – one for the type that I generate in my F# code (dbSchema) and another which contains the actual DataContext class and classes for each table in my database. This is good – I understand some more about what is generated under the hood for type providers, and should help me with the next stage of the investigation.

Creating a no-thrills custom type provider in F# 3.0 – attempt 1

2011-09-17T00:29:00.000+01:00

F# 3.0 was released this week at the Microsoft Build conference as part of the Visual Studio 11 Developer Preview. Full details are available on the F# team blog – I recommend you read through this and have a play with the new features – it’s a really feature packed release.

I’m interested specifically in Type Providers, which are a new direction in F#, and provide some of the power of dynamic languages (types generated on the fly to wrap some form of structured data, such as XML from an OData service) but without the performance issues or late binding that come with these. Type Providers are actually an extensibility mechanism for the compiler, which allow for the compiler to query the type provider assembly, which does exactly that – provides types on the fly, which both the compiler and VS Intellisense can make use of. This all happens at compile time, and therefore whilst the code has features with a dynamic flavour, the result remains statically typed.

What I want to do is to create the simplest possible example of a custom type provider. Essentially I would like to create one which provides only one type, and that itself is just a static type within the type provider assembly, no Emit code this time around.

FSharp.Data.TypeProviders

There doesn’t appear to be much (any?) information available yet concerning custom type providers, so my first port of call is to crack open the key F# type providers assembly – FSharp.Data.TypeProviders.dll using Reflector, and see what I can find.

The key touch point appears to be in Microsoft.FSharp.Data.TypeProviders.DesignTime.DataProviders -

[Serializable, TypeProvider, CompilationMapping(SourceConstructFlags.ObjectType)]
public class DataProviders : ITypeProvider, IProvidedNamespace, IDisposable
{
// ...
}

And the key interface here, defined in Microsoft.FSharp.Core.CompilerServices is:

public interface ITypeProvider : IDisposable
{
    // Events
    [CLIEvent]
    event EventHandler Invalidate;

    // Methods
    Type ApplyStaticArguments(Type typeWithoutArguments, string typeNameWithArguments, object[] staticArguments);
    Expression GetInvokerExpression(MethodBase syntheticMethodBase, ParameterExpression[] parameters);
    IProvidedNamespace[] GetNamespaces();
    ParameterInfo[] GetStaticParameters(Type typeWithoutArguments);
}

So I should be able to create an assembly (in C# to start off with) containing a class of my own which implements this above interface, and that should give me a type provider, right?

using System;
using System.Reflection;
using Microsoft.FSharp.Core;
using Microsoft.FSharp.Core.CompilerServices;
    
[Serializable, TypeProvider, CompilationMapping(SourceConstructFlags.ObjectType)]
public class BasicTypeProvider : ITypeProvider, IProvidedNamespace, IDisposable
{
    Type baseType;

    public BasicTypeProvider()
    {
        baseType = typeof(StaticType);
    }

    public Type ApplyStaticArguments(Type typeWithoutArguments, string typeNameWithArguments, object[] staticArguments)
    {
        return baseType;
    }

    public System.Linq.Expressions.Expression GetInvokerExpression(System.Reflection.MethodBase syntheticMethodBase, System.Linq.Expressions.ParameterExpression[] parameters)
    {
        return null;
    }

    public IProvidedNamespace[] GetNamespaces()
    {
        return new IProvidedNamespace[] { (IProvidedNamespace) this };
    }

    public System.Reflection.ParameterInfo[] GetStaticParameters(Type typeWithoutArguments)
    {
        return new ParameterInfo[] { };            
    }

    public event EventHandler Invalidate;

    public void Dispose()
    {
    }

    public IProvidedNamespace[] GetNestedNamespaces()
    {
        return new IProvidedNamespace[] { };
    }

    public Type[] GetTypes()
    {
        return new Type[] { baseType };
    }

    public string NamespaceName
    {
        get { return "RegistryTypeProvider"; }
    }

    public Type ResolveTypeName(string typeName)
    {
        if (string.Equals(typeName, "StaticType"))
        {
            return this.baseType;
        }
        return null;
    }
}

Note that I have included the same attributes as used in the FSharp.Data.TypeProviders assembly, as I assume they’re important, and in all the implementations I just shunt around the StaticType type (which is just a POCO class with a couple of properties, for demo purposes). I don’t know how to handle GetInvokerExpression at the moment, so I’ll return null here, and just hope I get away with it.

Now I build this assembly, and then create a new F# console application project to act as the consumer of this type provider. Add a reference to the type provider assembly and then create a simple use in Program.fs -

open TestTypeProvider

[<Generate>]
type myType = TestTypeProvider.BasicTypeProvider

Sadly this doesn’t work. I get red squigglies under TestTypeProvider.BasicTypeProvider and the message “This is not a provided type. Consider removing the 'Generate' attribute from this type definition.” Thanks. Actually I’d prefer to have a working type provider, if you don’t mind.

If I look back at FSharp.Data.TypeProviders.dll I can see an assembly level attribute which might be important:

... 
[assembly: TypeProviderAssembly]
...

So I add that to my type provider project, rebuild (and looks like I need to remove the type provider assembly reference from my consumer project, and then restart visual studio in order to avoid problems with the type provider assembly being locked).

Nope. “This is not a provided type. Consider removing the 'Generate' attribute from this type definition.”

I’ve now tried many tweaks to the attributes and restarted VS many times but no luck getting around this error message. I think I will take a look at this again in the morning, and dig around inside Reflector to see what else I can discover.

Prime Numbers - Building the EulerMaths library in F#

2010-06-27T19:52:00.000+01:00

Those of you who are doing Project Euler know what I am talking about by the EulerMaths library. There are certain fundamentals which crop up again and again in the questions and it is important to work these into a reusable and performant library.

The first and most obvious of these is the Prime Number generator. Many questions rely on generating a sequence of primes, or on determining whether or not an individual number is a prime. In this post I will step through a number of potential solutions, starting from the most naive, and going to my current state of the art.

Generating a Sequence of Primes - the Naive Solution

The algorithm for this solution is simple. Starting with what we know to be the lowest prime (2), count up to the highest number we want to test (which we call hp) , for each candidate we then test whether or not it is a prime. Lets look at that latter part first:

let isPrime n =
    let rec testAllDivisors d =
        if n = d then true
        else if n%d = 0 then false
                else testAllDivisors (d+1)
    testAllDivisors 2

This is a very common recursive pattern, we have an outer function with the simple parameter - n - the number to test, then define a recursive function with local scope. This recursive function steps through all possible divisors of n, starting from 2. If we find a divisor on the way, the number is not prime, if we reach n, then the number must be prime.

Note that the inner function uses tail recursion - the very last statement is the recursive call, and it performs no calculations with the result. This means the compiler can optimise out stack usage by removing the return address of each successive call to testAllDivisors, which effectively means the last call will return directly to isPrime, not to each of the previous execution of testAllDivisors. This means we can recurse as deep as we need, and will never see a StackOverflowException.

We can test performance of this routine with successively larger primes, and with timed mode on in F# Interactive (use the #time;; command to enable).

Digits	Prime	Time (ms)
5	50069	0
6	509203	2
7	5048423	23
8	50943779	295
9	506977979	2440
10	2147483647	9863

So with primes up to 7 digits or so, the performance isn't too bad for one off tests, but you can see that if we were building a sequence of primes from this, the performance would degrade exponentially. Lets try this now.

let allPrimesUpTo hp =
    let rec testEachPrime n = seq {
        if n <= hp then
            if isPrime n then yield n
            yield! testEachPrime (n+1)
    }
    testEachPrime 2

So this function makes use of our prime test to obtain a sequence of primes up to a given maximum. This uses the recursive sequence generation pattern of yield followed by yield! and the tail recursive call, which is a really nice way of building sequences. When testing the timing of this, don't forget that F# interactive will only iterate over the first 4 elements of the sequence, so you need to ensure that the entire sequence is enumerated when testing out the times. I used something like:

allPrimesUpTo 100000 |> Seq.length

Which forces enumeration of the entire sequence. So lets try that now, using a steadily increasing maximum:

Max	Highest Prime	Time (ms)
1000	997	2
10000	9973	30
100000	99991	2808
500000	499979	52644
1000000	999983	187455

What did I tell you! So this is acceptable for generating sequences of the lower primes, but becomes unusable very quickly. As many Euler problems are designed to weed out poorly optimised algorithms, you won't get very far with this implementation.

Going Parallel

The next option we have is to keep the same brute force approach, but to identify a means to partition the calculations and run in parallel. If we firstly change testEachPrime to support a subrange of the overall range, and yield a sequence within this range:

let rec testEachPrime (f,t) = seq {
    if f <= t then
        if (isPrime f) then yield f
        yield! testEachPrime ((f+1),t)
}

What we also need to do it partition an overall range, of say 2 .. 100000 into a set of subranges which can be used for each parallel task. I make a few assumptions here, the main one being that the overall range will be many times larger than the number of tasks we will run, so an uncomplicated implementation will suffice:

let divideWork f t nd =
    let range = (t+1) - f
    if range < nd then failwith "Range is less than the number of divisions"
    let delta = range/nd
    [| 0 .. nd-1 |] |> Array.map (fun i -> let ti = ((i+1)*delta)-1
                                         ((i*delta)+f),(if i >= (nd-1) then t else ti+f))

This takes the overall range f..t and number of required divisions, returning an array of tuples representing the f..t of each subdivision. Because the division isn't going to be perfect every time, the last pair will be truncated to ensure that its last value will always be the given t. So trying out our example from above we get:

> divideWork 2 100000 16;;
val it : (int * int) [] =
[|(2, 6250); (6251, 12499); (12500, 18748); (18749, 24997); (24998, 31246);
    (31247, 37495); (37496, 43744); (43745, 49993); (49994, 56242);
    (56243, 62491); (62492, 68740); (68741, 74989); (74990, 81238);
    (81239, 87487); (87488, 93736); (93737, 100000)|]

This is all we need now to divide our work up into subtasks, so piecing it together we get:

let allPrimesUpTo hp =
    let getResultsTask (f,t) =
        let rec testEachPrime (f,t) = seq {
            if f <= t then
                if (isPrime f) then yield f
                yield! testEachPrime ((f+1),t)
        }
        testEachPrime (f,t) |> Array.ofSeq

    let ntasks = System.Environment.ProcessorCount * 16

    if hp < (ntasks * 10) then
        getResultsTask (2,hp)
    else
        let taskArgs = divideWork 2 hp ntasks

        let results = 
            Async.Parallel [ for taskNo in 0 .. (ntasks-1) -> async { return getResultsTask taskArgs.[taskNo] } ]
            |> Async.RunSynchronously
        results |> Array.concat

Note first that we package testEachPrime into getResultsTask, which converts the sequence into an array. This is very important, as it ensures that the enumeration (and therefore calculations) are performed whilst running parallel. If you return just a sequence, the Async.Parallel does no work, and it all ends up running sequentially during the final call to Array.concat.

Next, we determine a number of tasks to run based on the CPU (or more specifically the number of cores). I will demonstrate later where the *16 comes in. If the prime we are looking for isn't much lower than the number of tasks we will run, we don't bother with the overhead of scheduling to threads, and instead just run the whole thing synchronously.

So finally, if we do want to run in parallel, we get the array of subranges, then setup and run an async workflow for each task, passing in this range as its parameter. This collates the task results into an array of arrays (which luckily is in the same order as our tasks run), which we can then concat into a single array of results. See Don Syme's Blog for much more detail about how this works.

Determining how many Tasks to run

This will depend on each circumstance, and probably also has a lot to do with the individual processor architectures. My main machine has 8 cores (which is what System.Environment.ProcessorCount returns), but if I run with just 8 tasks, the workload doesn't balance out too well:

You can see the falloff isn't sharp, and some cores are clearly done before others. By gradually increasing the number of tasks, we hit a sweet spot at CPUs * 16, which gives us a more balanced looking load, without too much overhead (and we get the quickest overall times at this level).

This clearly looks much better, with clean ramp up and ramp down on each core. So with this in place, we can repeat our performance test from the previous implementation, and look again at the results, side by side:

Max	Highest Prime	Time (Naive) ms	Time (Parallel, 8 cores) ms	Speed up
1000	997	2	1	x2
10000	9973	30	7	x4
100000	99991	2808	372	x7.5
500000	499979	52644	6947	x7.5
1000000	999983	187455	26016	x7.2

This is great, more or less exactly the speed up we would expect, as there will always be a certain amount of overhead in spinning up threads, and in the concatenation of results once all tasks complete. However, this is only useful if you happen to have a nice big multi-core box, and clearly once we go beyond 10s of millions, we will be needing a cray if we stick with the brute force method.

Do it With a Sieve

There are occasions where the brute force approach is enough. However there is a much more elegant way to generate a sequence of primes, called the Sieve of Eratosthenes - which is explained better than I can, by means of an animated diagram, on the wikipedia entry.

Lets first implement a very functional version of this sieve:

let eratosthenesSieve =
    let allNaturalsFrom2 = Seq.initInfinite ((+)1) |> Seq.filter ((<=)2)
    let rec sieveNext sq = seq {
        let n = sq |> Seq.nth 0
        yield n
        let newSq = sq |> Seq.filter (fun i -> i%n <> 0)
        yield! sieveNext newSq
    }
    sieveNext allNaturalsFrom2

In this implementation we start with an infinite sequence of natural numbers, excluding 1, then pass this into the sieve which yields the first element (which is 2 to start with), and recurses using a new sequence which excludes all members in the originals sequence which are multiples of 2, then does 3, 5, ...

It works great, but look at the times:

> eratosthenesSieve |> Seq.take 100 |> Seq.nth 99;;
Real: 00:00:00.133, CPU: 00:00:00.124, GC gen0: 1, gen1: 0, gen2: 0
val it : int = 541
> eratosthenesSieve |> Seq.take 200 |> Seq.nth 199;;
Real: 00:00:01.782, CPU: 00:00:01.778, GC gen0: 6, gen1: 0, gen2: 0
val it : int = 1223
> eratosthenesSieve |> Seq.take 300 |> Seq.nth 299;;
Real: 00:00:08.443, CPU: 00:00:08.455, GC gen0: 16, gen1: 1, gen2: 0
val it : int = 1987
> eratosthenesSieve |> Seq.take 400 |> Seq.nth 399;;
Real: 00:00:25.824, CPU: 00:00:25.786, GC gen0: 29, gen1: 1, gen2: 0
val it : int = 2741

This clearly is not going to be much use for us. Perhaps if we make the sequences finite, the results will improve?

let eratosthenesSieveFinite hp =
    let allNaturalsFrom2 = seq [2..hp]
    let rec sieveNext sq = seq {
        if (sq |> Seq.length) > 0 then
            let n = sq |> Seq.nth 0
            yield n
            let newSq = sq |> Seq.filter (fun i -> i%n <> 0)
            yield! sieveNext newSq
    }
    sieveNext allNaturalsFrom2

Lets try this with the same sizes of sequence (we use the biggest prime returned above to ensure the sequence sizes are the same:

> eratosthenesSieveFinite 541 |> Seq.take 100 |> Seq.nth 99;;
Real: 00:00:00.387, CPU: 00:00:00.390, GC gen0: 4, gen1: 0, gen2: 0
val it : int = 541
> eratosthenesSieveFinite 1223 |> Seq.take 200 |> Seq.nth 199;;
Real: 00:00:05.262, CPU: 00:00:05.382, GC gen0: 22, gen1: 1, gen2: 1
val it : int = 1223
> eratosthenesSieveFinite 1987 |> Seq.take 300 |> Seq.nth 299;;
Real: 00:00:25.003, CPU: 00:00:25.006, GC gen0: 50, gen1: 3, gen2: 0
val it : int = 1987

Wow, that's MUCH worse. Postulating that dealing with enumerable sequences rather than cold, hard arrays sucks our perf, lets try this same solution, but using arrays instead of sequences. Also, I realise we should stop eliminating once n passes sqrt(hp) so we can add a check for that:

let eratosthenesSieveArrays hp =
    let allNaturalsFrom2 = [|2..hp|]
    let limit = int(sqrt(float(hp)))
    let rec sieveNext sq = [|
        if (sq |> Array.length) > 0 then
            let n = sq.[0]
            if n > limit then
                for n in sq -> n
            else
                yield n
                let newSq = sq |> Array.filter (fun i -> i%n <> 0)
                yield! sieveNext newSq
    |]
    sieveNext allNaturalsFrom2

Which is a vast improvement. Lets merge the new perf results into our performance grid:

Max	Highest Prime	Time (Naive) ms	Time (Parallel, 8 cores) ms	Time (Array Sieve) ms
1000	997	2	1	1
10000	9973	30	7	45
100000	99991	2808	372	176
500000	499979	52644	6947	2164
1000000	999983	187455	26016	2853

So now the functional sieve approach is generally faster than even the 8 core parallel approach. One final approach we should try is to go for a slightly more imperative style - create a mutable array containing all potential primes, and then step through the sieve crossing out elements in the array (setting to 0), finally returning a sequence of all non-zero elements from the array.

let eratosthenesSieveImperative hp =
    let limit = int(sqrt(float(hp)))
    let primes = [| 0 .. hp |]
    primes.[1] <- 0
    let rec iterate n =
        if n < limit then
            [2*n .. n .. hp ] |> Seq.iter (fun i -> primes.[i] <- 0)
            let n = primes |> Seq.filter ((<)n) |> Seq.nth 0
            iterate n
    iterate 2
    primes |> Seq.filter ((<>)0)

This gives the best results so far:

Max	Highest Prime	Time (Naive) ms	Time (Parallel, 8 cores) ms	Time (Array Sieve) ms	Time (Imperative Sieve) ms
1000	997	2	1	1	8
10000	9973	30	7	45	7
100000	99991	2808	372	176	54
500000	499979	52644	6947	2164	267
1000000	999983	187455	26016	2853	616

This goes to show that sometimes you need to sacrifice some of the beauty of functional code to glean that last step in performance. Here we have a mutable array being used for working, but we still follow best practices by not exposing this mutable data to the caller of this function. One thing to bear in mind with this approach though, is that we are creating an in-memory array, so if you go much higher than a few million primes, you will risk running out of memory.

Parallel isPrime

Sometimes all we want to do is check a single number for prime, and especially if this number is very large, we don't want to generate a sequence or run out of memory by using a sieve, so lets use our knowledge about parallelising tasks, and create a parallel version of isPrime:

let isPrimeParallel n =
    let getResultsTask (f,t) =
        let rec testAllDivisors (f,t) =
            if f = t then true
            else if n%f = 0 then false
                 else testAllDivisors (f+1,t)
        testAllDivisors (f,t)

    let ntasks = System.Environment.ProcessorCount * 16

    if n < (ntasks * 10) then
        getResultsTask (2,n)
    else
        let taskArgs = divideWork 2 n ntasks

        let results = 
            Async.Parallel [ for taskNo in 0 .. (ntasks-1) -> async { return getResultsTask taskArgs.[taskNo] } ]
            |> Async.RunSynchronously
        not(results |> Seq.exists ((=)false))

So the logic here is to divide up the problem space again, and run the prime test logic on subsets, collate the results, and only return true if each subtask returned true (i.e. none of them found a divisor). Lets compare the perf of this with the original.

Digits	Prime	Original Time (ms)	Parallel (ms)
5	50069	0	4
6	509203	2	1
7	5048423	23	3
8	50943779	295	45
9	506977979	2440	305
10	2147483647	9863	1285

Conclusion

I wouldn't suggest you need to do this depth of analysis on all problems, but for very critical parts of your Euler toolset, it is clearly worth digging into the options. There are still optimisations we could do here - for example the parallel isPrime could support cancellation, so that all tasks abort as soon as we find a divisor, as we know then that the number is not a prime.

My First F# Post

2010-06-26T00:04:00.003+01:00

I have been experimenting with F# for a couple of years now, occasionally solving problems from Project Euler but have never got around to posting a blog entry on this subject. Recently I have been lucky enough to spend some time with Don Syme and the people of the excellent F#unctional Londoners user group, a number of whom complained that I don't have an F# blog, so I thought I had better put my money where my mouth is and put a post together.

Getting Started

F# is a functional programming language based on OCaml which combines the best of both worlds in terms of the pure functional aspects of ML and pragmatic use of imperative / object oriented language features. F# is now a first class member of the .NET languages and is part of Visual Studio 2010.

If you are new to F#, my recommended training list is as follows:

Firstly, get a taster by following Don's excellent introductory series on Channel 9.
After that, learn what functional programming is all about in Real World Functional Programming by Tomas Petricek and Jon Skeet.
Finally, read Expert F# 2.0 by Don Syme himself, Adam Granicz and Antonio Cisternino which does what it says on the cover.

There are also a number of (proper) blogs on F#, such as Don's (linked above), Tomas Petricek, Robert Pickering, Luca Bolognese and Phillip Trelford.

What's it all About?

I have already worked through over 50 of the puzzles in Project Euler, most of which I solved using F#. This is a fantastic way to improve both your F# and your maths skills (by the way several people have told me they don't do Project Euler because their F#/maths skills aren't good enough - if you think this, you have got it backwards, you will improve both as you work your way through).

Regarding Project Euler - if you have come to this post simply to find the solution, then I feel sorry for you, you have missed the point. I would really recommend that you have a good stab at the solution yourself before reading any further. The sense of achievement as you increment that score one by one is strangely addictive, and if you just copy the work of others, you miss out on all of that.

That said, this post will take you through my solution for Euler 54, which (somewhat unusually) isn't a particularly mathematical problem, but turns out to be a great way to demonstrate some of the features of the F# language. The basic principle of this problem is to compare two hands of cards in the game of poker, and to determine which is the winning hand.

Representing the Cards

The first step is to come up with a representation for the playing cards themselves, as well as some functions for parsing from a plain text description (which is what the Euler problem provides) and for evaluating individual cards. The representation here uses discriminated unions, and is similar to that shown in chapter 3 of Programming F# by Chris Smith.

type Suit =
    | Hearts
    | Diamonds
    | Clubs
    | Spades

type Card =
    | Ace of Suit
    | King of Suit
    | Queen of Suit
    | Jack of Suit
    | Value of int * Suit

These two discriminated unions allow us to represent any of the cards found in a deck of playing cards, eg Ace(Spades) or Value(2,Hearts). Next we need a function to take the string representation of a card (such as "AS", "2H", "TC") and return a value of the relevant discriminated union.

let parseCard (s:string) =
    let parseSuit (c:char) =
        match c with
        | 'H' -> Hearts
        | 'D' -> Diamonds
        | 'C' -> Clubs
        | 'S' -> Spades
        | _ -> failwith "Unknown suit"
    let charVal (c:char) = int(c)-48
    if s.Length <> 2 then failwith "Card should be 2 characters in length"
    let suit = parseSuit s.[1]
    match s.[0] with
    | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9' -> Value(charVal(s.[0]),suit)
    | 'T' -> Value(10,suit)
    | 'J' -> Jack(suit)
    | 'Q' -> Queen(suit)
    | 'K' -> King(suit)
    | 'A' -> Ace(suit)
    | _ -> failwith "Unknown rank"

Talking through this routine, you can see we have a helper function (which is local in scope to the function itself) called parseSuit which takes a single character and attempts to parse this into a suit by using a pattern matching expression. Within the main body of the routine we ensure that the string is 2 characters, no more, no less, get the suit, and then finally pattern match the rank of the card. For the numeric characters we use the tried and tested technique of subtracting the ASCII value of '0' in order to convert the char into a numeric value.

Playing with these functions in the enormously useful F# Interactive REPL window, we can check they work as expected:

> parseCard "KS";;
val it : Card = King Spades
> parseCard "3D";;
val it : Card = Value (3,Diamonds)
> parseCard "ZD";;
System.Exception: Unknown rank
at FSI_0008.parseCard(String s) in C:\Svn\roundthecampfire\trunk\ProjectEuler\EulerHost\Euler54.fs:line 46
at <StartupCode$FSI_0011>.$FSI_0011.main@()
Stopped due to error

And to conclude this section, lets create a function to load the two hands (as a tuple) from a single line of 10 cards as per the supplied poker.txt file, and then try that function on a couple of lines from the file.

let parseHands (sl:string) =
    let parts = sl.Split(' ')
    if parts.Length <> 10 then failwith "Hands must be 10 cards"
    else
        let comb = parts |> Array.map parseCard
        (Array.sub comb 0 5 |> List.ofArray),(Array.sub comb 5 5 |> List.ofArray)

This is our first use of some of the really cool F# higher order functions, such as Array.map and List.ofArray, as well as the forward pipe |> operator. Essentially this function gets an array of the individual 2 char strings, and projects these using Array.map and parseCard into a set of 10 cards. These cards are then split into two hands of 5 cards using Array.sub.

> parseHands "8C TS KC 9H 4S 7D 2S 5D 3S AC";;
val it : Card list * Card list =
([Value (8,Clubs); Value (10,Spades); King Clubs; Value (9,Hearts);
    Value (4,Spades)],
[Value (7,Diamonds); Value (2,Spades); Value (5,Diamonds); Value (3,Spades);
    Ace Clubs])
> parseHands "5C AD 5D AC 9C 7C 5H 8D TD KS";;
val it : Card list * Card list =
([Value (5,Clubs); Ace Diamonds; Value (5,Diamonds); Ace Clubs;
    Value (9,Clubs)],
[Value (7,Clubs); Value (5,Hearts); Value (8,Diamonds); Value (10,Diamonds);
    King Spades])

Functions to Determine the Rank of a Hand

Now that we have our hands of cards, we need a way to determine the highest rank to which a hand belongs, and giving us enough data to be able to compare two hands of the same rank, by checking the highest card which is part of the rank (e.g. a two pairs of Ace,Ace and 9,9 is higher than two pairs of King,Kind and 10,10) or highest cards which aren't part of the rank (e.g. pair of Kings with Ace high is better than pair of Kings with Jack high).

Before we do this, we need a function to return the relative value of cards as per 2,3,4,5,6,7,8,9,T,J,Q,K,A and a function which returns just the suit for a given card (so we can check for flushes).

let valueOf (c:Card) =
    match c with
    | Ace(_) -> 12
    | King(_) -> 11
    | Queen(_) -> 10
    | Jack(_) -> 9
    | Value(n,_) -> n - 2

let suitOf (c:Card) =
    match c with
    | Ace(s) | King(s) | Queen(s) | Jack(s) | Value(_,s) -> s

As you can see, valueOf means nothing in terms of any card game, but allows us to easily see that valueOf(Value(9,Spades)) < valueOf(King(Clubs)) for example.

Now it gets tricky, bear with me through this, hopefully this para and following examples will clarify. I decided to come up with a standard type for each rank matching function. Each function needs to take a list of cards as its input, returning firstly true or false to indicate whether the hand matches that rank, but also returning two arrays of card values each in descending order - the first array is the values which are part of the rank (eg for two pairs, 9s and Aces, the array would be [| 12; 7 |]); and the second array is any cards which remain that are not in the rank (eg for a hand 9H 9C AS TD 2C this matches a pair of 9s with remaining array of [| 12; 8; 0 |]).

So the type of these functions is Card list -> bool * int [] * int [] - we can use this to build a list of the ranking functions from highest (a royal flush) to the lowest (single card high), allowing us to try each in turn until one or both hands matches, and thus compare the hands. First I will define a few more helper functions that we will use in the ranking functions:

let desc n = -n

let allSameSuit (hand:Card list) =
    (hand |> Seq.map suitOf |> Seq.distinct |> Seq.length) = 1

The desc function is used for Seq.sortBy to reverse the order of an integer based sequence simply by negating the input, and allSameSuit is a predicate to determine when all the cards in a given hand belong to just one suit.

MatchHighCard

The lowest rank is X-High, meaning that the hand only contains single cards, and when comparing two hands of this rank, we compare card by card, until one hand has the higher card. Therefore any hand (which doesn't match a higher rank) will match this rank, and all cards in the hand are included in the rank, so appear in the first array, the second array being empty (as there are no remaining cards after those which belong to the rank are removed).

let matchHighCard (hand:Card list) : (bool*int []*int []) =
    true, (hand |> Seq.map valueOf |> Seq.sortBy desc |> Array.ofSeq), [||]

So the implementation is pretty simple, true, followed by the card values in descending order.

MatchTwoPairs

The next example is for two pairs. Here we have two sets of two cards which are in the rank, and a single remaining card.

let matchTwoPairs (hand:Card list) : (bool*int []*int []) =
    let matches =
        hand |> Seq.map valueOf |> Seq.groupBy (fun n -> n)
             |> Seq.map (fun (n,s) -> n,(Seq.length s))
             |> Seq.sortBy (fun (_,l) -> -l) |> Array.ofSeq
    if (Seq.length matches) <> 3 || (matches.[0] |> snd) <> 2 || (matches.[1] |> snd) <> 2 then false,[||],[||]
    else true,([| fst(matches.[0]); fst(matches.[1]) |] |> Array.sortBy desc),[| fst(matches.[2]) |]

This is one of the more complex implementations. We firstly get the card values and group by those values. Seq.groupBy gives us a sequence of tuples, the first value being the card value itself, the second is itself a sequence of each matching value (which is a bit nugatory in this case, as these are just sequences of the same value, hence in the following Seq.map we collapse these into just the number of occurrences). At this point we sort by the sequence length descending (using a custom lambda here as we are dealing with tuple values) and convert to an array for ease of indexing.

You probably need to see an example now. If the original hand was something like "9S JC 2D 2H JS", we get a value sequence of seq [ 7; 9; 0; 0; 9 ]. Following all our mappings we are left with an array of tuples [| (9,2); (0,2); (7,1) |].

Now we can check this array to see if it represents two pairs. It must have three elements, and the first two elements must represent pairs (the snd function gets the second value from the tuple, which is the number of occurrences of the given value). If these do not match, this isn't a rank of Two Pairs, so we return false and empty arrays.

If this is a match, we return true, with an array of two values (the highest of the pairs, then the lowest of the pairs), and finally the single remaining card value.

MatchStraightFlush

This is the last of my examples. A straight flush means the cards are all consecutive values, starting at any value, and they are all of the same suit.

let matchStraightFlush (hand:Card list) : (bool*int []*int []) =
    let values = hand |> List.map valueOf |> List.sort |> Seq.distinct |> Array.ofSeq
    if (Array.length values) <> 5 || (values.[4] - values.[0]) <> 4 || not (allSameSuit hand) then false,[||],[||]
    else true, [|values.[4]|], [||]

First we get the values, sort and remove duplicates (if there are dups, it can't be a straight - we check for this next). So now, if we don't have 5 cards, this isn't a match, or if the highest value - lowest value is not 4, then they are not consecutive, so not a match, or if they are not all the same suit, it isn't a match.

Where we have a match, the value of the highest card in the straight is returned.

Comparing the Hands

Lets define a utility function first. This takes two arrays, comparing item by item until a difference is found, returning negative to indicate the first array is greater, positive if the second array is greater, 0 if they are equal. We will use this to compare cards when two hands are of the same rank.

let compArrays (a:int [], b:int []) =
    if Array.length(a) <> Array.length(b) then failwith "Array lengths differ"
    let rec innerCompArrays (a:int [], b:int [],n) =
        if n = Array.length(a) then 0
        else
            if a.[n] <> b.[n] then b.[n] - a.[n]
            else innerCompArrays(a,b,(n+1))
    innerCompArrays(a,b,0)

Note the array lengths must be equal, and we assume they are both in the same order. We use a simple recursive function to step through item by item until the end of the array is reached, or a difference is found.

Now we can define our ordered list of match functions (note there are more than shown above, other implementations left as an exercise for the reader):

let matchFuncs =
    [   matchRoyalFlush
        matchStraightFlush
        matchFourOfAKind
        matchFullHouse
        matchFlush
        matchStraight
        matchThreeOfAKind
        matchTwoPairs
        matchAPair
        matchHighCard
    ]

We finally have everything we need to do the comparison of two poker hands. We can bring this all together in a single function which itself iterates over the above list of functions until a match is found for one or both hands, then, if needed, comparing the cards in the hand and then remaining cards to find a winner:

type matchFunc = (Card list) -> (bool * int [] * int [])

let compareHands (hands:string) =
    let rec innerCompareHands (handA:Card list, handB:Card list, funcs:matchFunc list) =
        match funcs with
        | [] -> failwith "No Winning Hand!"
        | f::tail ->
            let mtchA,incA,remainA = f(handA)
            let mtchB,incB,remainB = f(handB)
            match mtchA,mtchB with
            | true,false -> 1
            | false,true -> -1
            | true,true ->
                let c = compare incA incB
                if c <> 0 then c
                else compare remainA remainB
            | false,false ->
                innerCompareHands (handA,handB,tail)
    let handA, handB = parseHands(hands)
    if innerCompareHands (handA, handB, matchFuncs) > 0 then 1 else 2

UPDATE: Realised that the compArrays function was redundant, as the F# compare function works as well with lists and arrays as it does with simpler values. Updated the above accordingly.

Here we parse the string containing the two hands into the separate hands, then iterate over this using the inner recursive function. We work through each function in the list, testing both hands, then the inner pattern match checks the two match results side by side to find an outright winner (false,true or true,false) a draw (true,true) in which case we compare first the cards in the matching rank, then the remaining cards if still a draw. If there are no matches for this function, we move to the next rank and test for that. Note that the check for no remaining functions isn't ever going to be exercised as matchHighCard will always match. The final result indicates a win for player 1 or player 2

> compareHands "JH 2D 3S JC AS 7D 3C AH 7C 7S";;
val it : int = 2
> compareHands "TS JS QS KS AS 7D 3C AH 7C 7S";;
val it : int = 1

We can see this works fine - in the first example the second hand (three 7s) beats the first (pair of Jacks), and in the second example, the first hand wins as it is a Royal Flush.

Conclusion

We can see that F# code really lends itself to breaking down a problem like this. We use discriminated unions to represent our model very succinctly. We use pattern matching for parsing and checking card values, we use higher order functions on sequences and arrays to concisely compose our rank matching functions, and we use an array of these functions just as easily as any other data type to build our final comparison routine.

With some work, I could probably make this even more succinct, maybe by using active patterns to replace the recursion over the array of functions with a pattern matching expression instead, but hopefully the above demonstrates that a "good enough" solution has been achieved. In case you are interested, the final result is calculated in 188ms, including reading the file.

Building an OData consumer for Windows Phone 7 Series, Part 2

2010-04-04T12:19:00.004+01:00

In Part 1 I talked about the rough outline of the application – a WP7S app which talks to a publically available OData service (the NerdDinner service) and presents a master detail view of this, with geographical data. We looked at installing the tools, building a WP7S application, and walked around the architecture of the application.

Today I will add the OData service call and implement binding from the master view to the data retrieved, and make sure that this data is rendered in the list in a pleasing manner.

Calling the OData Service

If you haven’t done so already, download the OData client library and unzip the files into a local directory. Add a reference from the NerdDinnerFinder project to System.Data.Services.Client.dll.
Currently there is no documentation available specifically for WP7S, but this blog post by Well Phani describes how to add a service reference using the DataSvcutil.exe tool in the framework 4.0 directory.

Using an Admin command prompt, navigate to the NerdDinnerFinder project directory, then execute this command:

%WINDIR%\Microsoft.NET\Framework\v4.0.30128\DataSvcUtil.exe /?

This gives the following output:

Microsoft (R) DataSvcUtil version 4.0.0.0
Copyright (C) 2008 Microsoft Corporation. All rights reserved.

DataSvcUtil Options
/in:<file>               The file to read the conceptual model from
/out:<file>              The file to write the generated object layer to
/language:CSharp         Generate code using the C# language
/language:VB             Generate code using the VB language
/Version:1.0             Accept CSDL documents tagged with
m:DataServiceVersion=1.0 or lower
/Version:2.0             Accept CSDL documents tagged with
m:DataServiceVersion=2.0 or lower
/DataServiceCollection   Generate collections derived from DataServiceCollection
/uri:<URL>               The URI to read the conceptual model from
/help                    Display the usage message (short form: /?)
/nologo                  Suppress copyright message

Examples
To generate code from a data service.
DataSvcUtil /out:"data.cs" /uri:"http://localhost/data.svc"

To generate code from a conceptual model file.
DataSvcUtil /out:"data.cs" /in:"file.csdl"

To generate code from a entity design model file.
DataSvcUtil /out:"data.cs" /in:"file.edmx"

So using the blog post above as guidance, we can generate our classes directly from the live OData service by using the command":

%WINDIR%\Microsoft.NET\Framework\v4.0.30128\DataSvcUtil.exe
/uri:http://www.nerddinner.com/Services/OData.svc/
/DataServiceCollection /Version:2.0 /out:nerdDinnerService.cs

This creates the file nerdDinnerService.cs which we can now directly include in our project, and take a look at. What we have is a fairly standard looking codegen file with three classes: NerdDinnerEntities, Dinner and RSVP under the NerdDinnerModel namespace. The NerdDinnerEntities class is our entry point, and derives from DataServiceContext in System.Data.Services.Client. The other two classes are our data objects.

Experimenting with the NerdDinnerEntities class, there doesn’t appear to be any obvious ‘Linqy’ way to deal with the data. If I try to directly bind to the Dinners collection it comes back with an opaque NotSupportedException which isn’t much help. Just for the sake of getting things moving, lets use the BeginExecute<T> and EndExecute methods:

private void MainPage_Loaded(object sender, RoutedEventArgs e)
{
    // Reset page transition
    ResetPageTransitionList.Begin();

    context = new NerdDinnerEntities(new Uri(http://www.nerddinner.com/Services/OData.svc));
    context.BeginExecute<Dinner>(new Uri("/Dinners",UriKind.Relative), DinnerReady, null);
}

private void DinnerReady(IAsyncResult asyncResult)
{
    Deployment.Current.Dispatcher.BeginInvoke(
        () =>
        {
            IEnumerable<Dinner> results = context.EndExecute<Dinner>(asyncResult).ToList() as IEnumerable<Dinner>;
            lstDinners.ItemsSource = results;
        }
        );
}

This seems heavy handed, as we have to hand code the relative URI, but it does seem to work. In order to get this to display correctly, we will need to make some changes to the xaml (we should change the name of the list box, and make some changes to the ItemTemplate in order to get the sample listbox to bind to the data we are returning). Replace the entire ListBox element with the following:

<ListBox x:Name="lstDinners" MouseLeftButtonUp="ListBoxOne_MouseLeftButtonUp" 
        Style="{StaticResource PhoneListBox}">
    <ListBox.ItemTemplate>
        <DataTemplate>
            <StackPanel x:Name="DataTemplateStackPanel" Orientation="Vertical">
                <TextBlock Text="{Binding Title}" Style="{StaticResource PhoneTextLargeStyle}" />
                <TextBlock Text="{Binding Description}" Style="{StaticResource PhoneTextNormalStyle}" />
            </StackPanel>
        </DataTemplate>
    </ListBox.ItemTemplate>
</ListBox>

So we change the name, remove the declarative data binding from ItemsSource and update the ItemTemplate with a more simple view onto our Dinner items. If we run this now, we should get a sensible result:

This is our master view, now we need to change the item selection event handler to navigate from this page to the detail page - it already does, but we need to pass in the Id of the selected dinner, so some tweaks are required.

private void ListBoxOne_MouseLeftButtonUp(object sender, MouseButtonEventArgs e)
{
    // Capture selected item data
    selectedItem = (sender as ListBox).SelectedItem as Dinner;

    // Start page transition animation
    PageTransitionList.Begin();
}

private void PageTransitionList_Completed(object sender, EventArgs e)
{
    // Set datacontext of details page to selected listbox item
    NavigationService.Navigate(new Uri(String.Format("/DetailsPage.xaml?id={0}",selectedItem.DinnerID), UriKind.Relative));
}

Note the change to NavigationService.Navigate – by passing in the id as a query string parameter, we can ensure that the navigation will work even with application dehydration/rehydration. With this in place we can make the corresponding change in DetailsPage.xaml.cs and then we are ready to implement the details view.

<Grid x:Name="LayoutRoot" Background="{StaticResource PhoneBackgroundBrush}" d:DataContext="{Binding Items[0]}">
    <Grid.RowDefinitions>
        <RowDefinition Height="Auto"/>
        <RowDefinition Height="*"/>
    </Grid.RowDefinitions>

    <!--TitleGrid is the name of the application and page title-->
    <Grid x:Name="TitleGrid" Grid.Row="0">
        <Grid.Projection>
            <PlaneProjection/>
        </Grid.Projection>
         
        <TextBlock Text="NERD DINNER FINDER" Style="{StaticResource PhoneTextPageTitle1Style}"/>
        <TextBlock Text="{Binding Title}" Style="{StaticResource PhoneTextPageTitle2Style}"/>
    </Grid>

    <!--ContentGrid contains the details-->
    <Grid x:Name="ContentGrid" Grid.Row="1">
        <Grid.Projection>
            <PlaneProjection/>
        </Grid.Projection>

        <StackPanel Orientation="Vertical">
            <TextBlock Text="{Binding Description}" Margin="20,0,20,0" TextWrapping="Wrap" Style="{StaticResource PhoneTextSmallStyle}"/>
            <StackPanel Orientation="Horizontal">
                <TextBlock Text="Hosted by " Style="{StaticResource PhoneTextBodyTextStyle}"/>
                <TextBlock Text="{Binding HostedBy}" Style="{StaticResource PhoneTextBodyTextStyle}"/>
            </StackPanel>
        </StackPanel>
    </Grid>
</Grid>

This ensures that the current item (which is bound to the page itself) is used to populate the various text boxes on this page. The result gets us where we need to be for today:

Round Up

Today we have installed the OData client library, struggled a little bit in understanding the model this exposes (a little documentation here would help a lot), and come up with a 50% elegant solution for bringing in the data we need and binding this within the master view and the detail view.

In the final article, I will try and make use of the geographical data we get back from this service to enable an inset map showing the location of the event on the details page.

Building an OData consumer for Windows Phone 7 Series, Part 1

2010-04-02T22:32:00.004+01:00

Having recently returned from MIX 2010, two of the technologies which stuck in my mind were OData (and specifically the latest incarnation of project Astoria - WCF Data Services) and of course the first CTP of the Windows Phone 7 series SDK.

I am already working on a WP7S application for the marketplace, but I thought it would be an interesting aside to put together a bit of a mash-up application which consumes an OData service, presents a pleasing view over this data using silverlight, and supports viewing location data using Bing maps. I will document the construction of this application in three parts, this being the first.

Finding an OData Service

We could, of course, write our own OData service against the Northwind database or similar, but for the sake of simplicity, I thought it would be better to consume one of the sample services from the OData site. For this demonstration I will use the NerdDinner Service.

As you can see, this is a pretty basic service at the moment, but will suffice for this demonstration. What we would like to do is drill down into the Dinners category, present a list of available dinners, and then provide an interface to drill down into a particular dinner and show the details (including the location).

Getting Started

In order to develop this application you will need to download and install the following components:

The first link will download everything you need to develop for WP7S - Visual Studio 2010 Express Edition, the WP7S Emulator, Silverlight SDK and XNA Game studio. Note that if you already have an edition of VS 2010 installed, it will also install the necessary project templates for that.

Creating the Application

We can make use of existing project templates to get us started quickly. Launch VS2010 and create a new project from Visual C# > Silverlight for Windows Phone > Windows Phone List Application. Name the project & solution NerdDinnerFinder.

This will create a new windows phone project which contains the application (App.xaml) and two silverlight pages (MainPage.xaml and DetailsPage.xaml). If we open up the designer for MainPage.xaml, we can see the design view of the page as it would look on the phone, side by side with the xaml code.

Notice that this has already created us a master/details view in effect, so all we need to do is plumb in the OData service to these xaml pages. If you build and run this application as it stands, it will launch the WP7S emulator (which takes about a minute to load the first time, but thereafter very quick as long as you leave it running).

Examining the Build Output

It's always good to try and ensure we get a deeper understanding of what happens under the hood. Lets take a look at the artifacts produced by this build, and see what they have to show us:

The NerdDinnerFinder.xap file is actually the single file which will be deployed to the device. We can use a trick from the silverlight development toolbox to see what this file contains - simply rename it with a .zip extension and open it in explorer. Within this we can see the same files, and one additional - WMAppManifest.xml.

We therefore have two .png files to store various size of icon imagery for the use of the phone OS, we have the actual dll and pdb for our single project, then two manifest files.

AppManifest.xaml simply describes the application and the binary files of which it is comprised:

<Deployment xmlns="http://schemas.microsoft.com/client/2007/deployment" 
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" 
    EntryPointAssembly="NerdDinnerFinder" 
    EntryPointType="NerdDinnerFinder.App" 
    RuntimeVersion="3.0.40624.0">
    <Deployment.Parts>
        <AssemblyPart x:Name="NerdDinnerFinder" Source="NerdDinnerFinder.dll" />
    </Deployment.Parts>
</Deployment>

The other file WMAppManifest.xml would appear to be the means of defining the resources and capabilities of the app from the point of view of the OS, i.e. as the place where it can get icons and images for the app:

<Deployment xmlns="http://schemas.microsoft.com/windowsphone/2009/deployment" AppPlatformVersion="7.0">
    <App xmlns="" ProductID="{45a8bc14-719e-4920-bce2-57e11d08e8bb}" Title="NerdDinnerFinder" RuntimeType="SilverLight" Version="1.0.0.0" Genre="NormalApp"    Author="" Description="" Publisher="">
        <IconPath IsRelative="true" IsResource="false">ApplicationIcon.png</IconPath>
        <Capabilities>
        </Capabilities>
        <Tasks>
            <DefaultTask Name="_default" PlaceHolderString="Default task"/>
        </Tasks>
        <Tokens>
            <PrimaryToken TokenID="NerdDinnerFinderToken" TaskName="_default">
                <TemplateType5>
                    <BackgroundImageURI IsRelative="true" IsResource="false">Background.png</BackgroundImageURI>
                    <Count>0</Count>
                    <Title>NerdDinnerFinder</Title>
                </TemplateType5>
            </PrimaryToken>
        </Tokens>
    </App>
</Deployment>

I thought maybe we could influence some of the attribute settings under App by modifying the project preferences as follows:

But recompiling using these settings showed no change in the WMAppManifest.xml file. My guess would be this is the file in which the Windows Phone Marketplace team will put all the metadata and code signing stuff when your app gets approved.

Anatomy of the Windows Phone 7 Series DLL

Now we have looked at the contents of the .xap file, lets crack open the NerdDinnerFinder.dll file and see if there is anything unexpected in there:

As we might expect, we have a Page class for the two pages, some viewmodel classes and the App class. We also have all the .xaml files packaged as resources. The TargetFramework assembly level attribute shows “Silverlight,Version=v4.0,Profile=WindowsPhone”. This is slightly strange, as the standard line in MIX was that WP7S is silverlight 3 with some extra bits, whereas this seems to indicate we have silverlight 4, but with some bits taken away…

Structure of the Windows Phone Application

Lets dig around the solution in visual studio now, starting with App.xaml.

App.xaml

This defines the application itself, and is configured in the project settings (see previous section) as the startup object for the solution. The file created by the visual studio template is quite large, as it is used to define a large number of xaml resources for use throughout the application, but starts like so:

<Application 
    x:Class="NerdDinnerFinder.App"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"     
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:system="clr-namespace:System;assembly=mscorlib"
    xmlns:mpc="clr-namespace:Microsoft.Phone.Controls;assembly=Microsoft.Phone.Controls"
    xmlns:phoneNavigation="clr-namespace:Microsoft.Phone.Controls;assembly=Microsoft.Phone.Controls.Navigation">

    <!--RootFrame points to and loads the first page of your application-->
    <Application.RootVisual>
        <phoneNavigation:PhoneApplicationFrame x:Name="RootFrame" Source="/MainPage.xaml"/>
    </Application.RootVisual>

    <!-- Resources for following the Windows Phone design guidelines -->
    <Application.Resources>
        <!--************ THEME RESOURCES ************-->
        <!-- Color Resources -->
        <Color x:Key="PhoneBackgroundColor">#FF1F1F1F</Color>
        <Color x:Key="PhoneContrastForegroundColor">Black</Color>
...
</Application>

Deconstructing this file, we can see there are two main purposes – firstly it defines the root visual, or “entry page” for the application as MainPage.xaml, secondly it defines resources for the rest of the application.

There is a codebehind file, App.xaml.cs, which currently is used to hook up an event handler for all unhandled exceptions.

MainPage.xaml

This is just a standard xaml page, derived from PhoneApplicationPage and defines a simple layout with headings and a list box with an ItemTemplate. The codebehind, MainPage.xaml.cs, defines a transition effect to occur when a list item is clicked, and in the completion handler calls NavigationService.Navigate to move to the detail page.

Summary

So far we have determined what we want to demonstrate with this application, installed the required development tools, created a basic application from the windows phone list application template, and taken a look around this application.

Next up, we will hook up an OData client within this application and bind this to the listbox.

Using DGML in Visual Studio 2010 to visualise a financial model

2010-02-14T20:00:00.000Z

Having just installed the new RC of Visual Studio 2010, and prompted by this video by Chris Lovett and these blog posts by Cameron Skinner it prompted me to experiment with an idea I had a while ago – the ability to generate a diagram showing the calculations and the flow of values for a financial model.

In this experiment I will create a simplistic bond pricer in C#, create a logging mechanism for this model to output values at the various stages, and then implement a means to transform these values into a DGML model file which we can then visualise.

Create a logging mechanism

All we need is a means to generate an XML document containing the Inputs, Simple calculations, and iterated calculations (this is enough to satisfy the requirements of the simple bond pricer)

    public class XmlLogger : ICalcLogger
    {
        public XDocument LoggedValues { get; private set; }

        public XmlLogger()
        {
            LoggedValues = new XDocument();

            LoggedValues.Add(new XElement("Results",
                new XElement("Inputs"),
                new XElement("SingleCalcs"),
                new XElement("IteratedCalcs")));
        }

        public void Save(string filename)
        {
            File.WriteAllText(filename, LoggedValues.ToString());
        }

        public void LogInput(string name, object value)
        {
            LoggedValues.Element("Results").Element("Inputs").Add(
                new XElement("Input",
                    new XAttribute("name", name),
                    new XAttribute("category", "Inputs"),
                    new XAttribute("value", value)));
        }

        public void LogSingleCalc(string name, string category, string formula, object value)
        {
            LoggedValues.Element("Results").Element("SingleCalcs").Add(
                new XElement("SingleCalc",
                    new XAttribute("name", name),
                    new XAttribute("category", category),
                    new XAttribute("formula", formula),
                    new XAttribute("value", value)));
        }

        public void LogIteratedCalc(string name, string category, int iteration, string formula, object value)
        {
            LoggedValues.Element("Results").Element("IteratedCalcs").Add(
                new XElement("IteratedCalc",
                    new XAttribute("name", name),
                    new XAttribute("category", category),
                    new XAttribute("iteration", iteration),
                    new XAttribute("formula", formula),
                    new XAttribute("value", value)));
        }
    }

Create the Bond Pricer model

With the logging in place, we can now create an implementation of the bond pricer which makes use of this logging:

    // very basic clean price calculation based on http://en.wikipedia.org/wiki/Bond_pricing
    public class SimpleBondPricer
    {
        private double F;
        private double C;
        private double r;
        private int T;
        private int n;
        private ICalcLogger logger;

        public SimpleBondPricer(double F, double C, double r, int T, int n, ICalcLogger logger)
        {
            this.F = F;
            this.C = C;
            this.r = r;
            this.T = T;
            this.n = n;
            this.logger = logger;

            logger.LogInput("F", F);
            logger.LogInput("C", C);
            logger.LogInput("r", r);
            logger.LogInput("T", T);
            logger.LogInput("n", n);
        }

        public double CalculatePrice()
        {
            double coupons = CalculateCoupons(logger);
            double discountedFace = CalculatedDiscountedFaceValue(logger);

            double result = discountedFace + coupons;
            logger.LogSingleCalc("Pc", "Results", "Sm+DF", result);
            return result;
        }

        private double CalculatedDiscountedFaceValue(ICalcLogger logger)
        {
            double result = F / Math.Pow(1.0 + r, T);
            logger.LogSingleCalc("DF", "ParCalcs", "F / (1+r)^T", result);
            return result;
        }

        private double CalculateCoupons(ICalcLogger logger)
        {
            int m = n * T;
            logger.LogSingleCalc("m", "CouponCalcs", "n.T", m);

            double u = Math.Pow(1.0 + r, 1.0 / n);
            logger.LogSingleCalc("u", "CouponCalcs", "(1 + r) ^ (1 / n)", u);

            double result = 0.0;
            for (int t = 1; t <= m; t++)
            {
                double iterationResult = C / Math.Pow(u, t);
                logger.LogIteratedCalc("S", "CouponCalcs", t, "C/u^t", iterationResult);
                result += iterationResult;
            }

            logger.LogSingleCalc("Sm", "CouponCalcs", "Sum[t=1..m](C/u^t)", result);
            return result;
        }
    }

So we ensure all inputs are logged once we get them, and each stage in the calculation we log the formula used and the results obtained. So a simple invocation such as:

    var logger = new XmlLogger();

    var bp = new SimpleBondPricer(100.0, 0.08, 0.045, 3, 2, logger);
    double result = bp.CalculatePrice();

    logger.Save(@"C:\BondPricerCalcs.xml");

Will generate the following calculations log:

    <Results>
      <Inputs>
        <Input name="F" category="Inputs" value="100" />
        <Input name="C" category="Inputs" value="0.08" />
        <Input name="r" category="Inputs" value="0.045" />
        <Input name="T" category="Inputs" value="3" />
        <Input name="n" category="Inputs" value="2" />
      </Inputs>
      <SingleCalcs>
        <SingleCalc name="m" category="CouponCalcs" formula="n.T" value="6" />
        <SingleCalc name="u" category="CouponCalcs" formula="(1 + r) ^ (1 / n)" value="1.0222524150130437" />
        <SingleCalc name="Sm" category="CouponCalcs" formula="Sum[t=1..m](C/u^t)" value="0.444727984347155" />
        <SingleCalc name="DF" category="ParCalcs" formula="F / (1+r)^T" value="87.629660405490938" />
        <SingleCalc name="Pc" category="Results" formula="Sm+DF" value="88.074388389838091" />
      </SingleCalcs>
      <IteratedCalcs>
        <IteratedCalc name="S" category="CouponCalcs" iteration="1" formula="C/u^t" value="0.078258558087122954" />
        <IteratedCalc name="S" category="CouponCalcs" iteration="2" formula="C/u^t" value="0.076555023923444987" />
        <IteratedCalc name="S" category="CouponCalcs" iteration="3" formula="C/u^t" value="0.07488857233217508" />
        <IteratedCalc name="S" category="CouponCalcs" iteration="4" formula="C/u^t" value="0.0732583960989904" />
        <IteratedCalc name="S" category="CouponCalcs" iteration="5" formula="C/u^t" value="0.071663705581028789" />
        <IteratedCalc name="S" category="CouponCalcs" iteration="6" formula="C/u^t" value="0.070103728324392739" />
      </IteratedCalcs>
    </Results>

Generating the DGML markup

The XML above contains all we need to create the nodes for our diagram. I considered using XSLT or hand crafting some C# to transform this into DGML, but then thought this is a great use for T4 templates – we just need to iterate over the elements in this XML document, and generate the corresponding DGML nodes.

In VS 2010 I add a new Text Template file to the solution and populate it as follows:

<#@ template language="C#" #>
<#@ output extension="dgml" #>
<#@ assembly name="System.Core" #>
<#@ assembly name="System.Xml" #>
<#@ assembly name="System.Xml.Linq" #>
<#@ import namespace="System.Linq" #>
<#@ import namespace="System.Xml.Linq" #>
<#
    var doc = XDocument.Load(@"C:\BondPricerCalcs.xml");
#>
<?xml version="1.0" encoding="utf-8"?>
<DirectedGraph xmlns="http://schemas.microsoft.com/vs/2009/dgml">
  <Nodes>
<#
    foreach (var element in doc.Element("Results").Element("Inputs").Descendants("Input"))
    {
        string name = element.Attribute("name").Value;
        string category = element.Attribute("category").Value;
        string value = element.Attribute("value").Value;
#>
    <Node Id="<#= name #>" Category="<#= category #>" Description="<#= value #>" Label="<#= name #>" />
<#
    }
#>
<#
    foreach (var element in doc.Element("Results").Element("SingleCalcs").Descendants("SingleCalc"))
    {
        string name = element.Attribute("name").Value;
        string category = element.Attribute("category").Value;
        string formula = element.Attribute("formula").Value;
        string value = element.Attribute("value").Value;
#>
    <Node Id="<#= name #>" Category="<#= category #>" Description="<#= formula #> = <#= value #>" Label="<#= name #>" />
<#
    }
#>
<#
    foreach (var element in doc.Element("Results").Element("IteratedCalcs").Descendants("IteratedCalc"))
    {
        string name = element.Attribute("name").Value;
        string category = element.Attribute("category").Value;
        string iteration = element.Attribute("iteration").Value;
        string formula = element.Attribute("formula").Value;
        string value = element.Attribute("value").Value;
#>
    <Node Id="<#= name #><#= iteration #>" Category="<#= category #>" Description="<#= formula #>[<#= iteration #>] = <#= value #>" Label="<#= name #>[<#= iteration #>]" />
<#
    }
#>
    <Node Id="Coupons" Category="CouponCalcs" Group="Expanded" Label="Coupons" />
  </Nodes>
  <Links>
    <Link Source="DF" Target="Pc" />
    <Link Source="F" Target="DF" />
    <Link Source="m" Target="Sm" />
    <Link Source="n" Target="m" />
    <Link Source="n" Target="u" />
    <Link Source="r" Target="DF" />
    <Link Source="r" Target="u" />
    <Link Source="Sm" Target="Pc" />
    <Link Source="T" Target="DF" />
    <Link Source="T" Target="m" />
    <Link Source="Coupons" Target="Sm" Category="Contains" />
<#
    var m = (from calc in doc.Element("Results").Element("SingleCalcs").Descendants("SingleCalc")
             where calc.Attribute("name").Value == "m"
             select int.Parse(calc.Attribute("value").Value)).First();
    for(int t = 1; t <= m; t++)
    {
#>
    <Link Source="C" Target="S<#= t #>"/>
    <Link Source="u" Target="S<#= t #>"/>
    <Link Source="S<#= t #>" Target="Sm"/>
    <Link Source="Coupons" Target="S<#= t #>" Category="Contains" />
<#
    }
#>
  </Links>
  <Categories>
    <Category Id="Contains" Label="Contains" CanBeDataDriven="False" CanLinkedNodesBeDataDriven="True" IncomingActionLabel="Contained By" IsContainment="True" OutgoingActionLabel="Contains" />
    <Category Id="Inputs" Label="Inputs" IsTag="True" />
    <Category Id="CouponCalcs" Label="Coupon Calcs" IsTag="True" />
    <Category Id="ParCalcs" Label="Par Calcs" IsTag="True" />
    <Category Id="Results" Label="Results" IsTag="True" />
  </Categories>
  <Styles>
    <Style TargetType="Node" GroupLabel="Coupon Calcs" ValueLabel="Has category">
      <Condition Expression="HasCategory('CouponCalcs')" />
      <Setter Property="Background" Value="#FFFFFF00" />
    </Style>
    <Style TargetType="Node" GroupLabel="Par Calcs" ValueLabel="Has category">
      <Condition Expression="HasCategory('ParCalcs')" />
      <Setter Property="Background" Value="Green" />
    </Style>
    <Style TargetType="Node" GroupLabel="Inputs" ValueLabel="Has category">
      <Condition Expression="HasCategory('Inputs')" />
      <Setter Property="Background" Value="White" />
    </Style>
    <Style TargetType="Node" GroupLabel="Results" ValueLabel="Has category">
      <Condition Expression="HasCategory('Results')" />
      <Setter Property="Background" Value="Blue" />
    </Style>
  </Styles>
</DirectedGraph>

So this basically iterates over the XML first to generate all the diagram nodes, then adds all the static links (the ones which don’t change no matter what the model inputs are), and finally the links for the individual coupon calcs. Whenever this file is saved, it will generate a code-behind file as per the output directive, i.e. with a .dgml extension. If you open up this file, we see the DGML source:

<?xml version="1.0" encoding="utf-8"?>
<DirectedGraph xmlns="http://schemas.microsoft.com/vs/2009/dgml">
  <Nodes>
    <Node Id="C" Category="Inputs" Description="0.08" Label="C" />
    <Node Id="Coupons" Category="CouponCalcs" Group="Expanded" Label="Coupons" />
    <Node Id="DF" Category="ParCalcs" Description="F / (1+r)^T = 87.629660405490938" Label="DF" />
    <Node Id="F" Category="Inputs" Description="100" Label="F" />
    <Node Id="Pc" Category="Results" Description="Sm+DF = 88.074388389838091" Label="Pc" />
    <Node Id="S1" Category="CouponCalcs" Description="C/u^t[1] = 0.078258558087122954" Label="S[1]" />
    <Node Id="S2" Category="CouponCalcs" Description="C/u^t[2] = 0.076555023923444987" Label="S[2]" />
    <Node Id="S3" Category="CouponCalcs" Description="C/u^t[3] = 0.07488857233217508" Label="S[3]" />
    <Node Id="S4" Category="CouponCalcs" Description="C/u^t[4] = 0.0732583960989904" Label="S[4]" />
    <Node Id="S5" Category="CouponCalcs" Description="C/u^t[5] = 0.071663705581028789" Label="S[5]" />
    <Node Id="S6" Category="CouponCalcs" Description="C/u^t[6] = 0.070103728324392739" Label="S[6]" />
    <Node Id="Sm" Category="CouponCalcs" Description="Sum[t=1..m](C/u^t) = 0.444727984347155" Label="Sm" />
    <Node Id="T" Category="Inputs" Description="3" Label="T" />
    <Node Id="m" Category="CouponCalcs" Description="n.T = 6" Label="m" />
    <Node Id="n" Category="Inputs" Description="2" Label="n" />
    <Node Id="r" Category="Inputs" Description="0.045" Label="r" />
    <Node Id="u" Category="CouponCalcs" Description="(1 + r) ^ (1 / n) = 1.0222524150130437" Label="u" />
  </Nodes>
  <Links>
    <Link Source="C" Target="S1" />
    <Link Source="C" Target="S2" />
    <Link Source="C" Target="S3" />
    <Link Source="C" Target="S4" />
    <Link Source="C" Target="S5" />
    <Link Source="C" Target="S6" />
    <Link Source="Coupons" Target="S1" Category="Contains" />
    <Link Source="Coupons" Target="S2" Category="Contains" />
    <Link Source="Coupons" Target="S3" Category="Contains" />
    <Link Source="Coupons" Target="S4" Category="Contains" />
    <Link Source="Coupons" Target="S5" Category="Contains" />
    <Link Source="Coupons" Target="S6" Category="Contains" />
    <Link Source="Coupons" Target="Sm" Category="Contains" />
    <Link Source="DF" Target="Pc" />
    <Link Source="F" Target="DF" />
    <Link Source="S1" Target="Sm" />
    <Link Source="S2" Target="Sm" />
    <Link Source="S3" Target="Sm" />
    <Link Source="S4" Target="Sm" />
    <Link Source="S5" Target="Sm" />
    <Link Source="S6" Target="Sm" />
    <Link Source="Sm" Target="Pc" />
    <Link Source="T" Target="DF" />
    <Link Source="T" Target="m" />
    <Link Source="m" Target="Sm" />
    <Link Source="n" Target="m" />
    <Link Source="n" Target="u" />
    <Link Source="r" Target="DF" />
    <Link Source="r" Target="u" />
    <Link Source="u" Target="S1" />
    <Link Source="u" Target="S2" />
    <Link Source="u" Target="S3" />
    <Link Source="u" Target="S4" />
    <Link Source="u" Target="S5" />
    <Link Source="u" Target="S6" />
  </Links>
  <Categories>
    <Category Id="Contains" Label="Contains" CanBeDataDriven="False" CanLinkedNodesBeDataDriven="True" IncomingActionLabel="Contained By" IsContainment="True" OutgoingActionLabel="Contains" />
    <Category Id="CouponCalcs" Label="Coupon Calcs" IsTag="True" />
    <Category Id="Inputs" Label="Inputs" IsTag="True" />
    <Category Id="ParCalcs" Label="Par Calcs" IsTag="True" />
    <Category Id="Results" Label="Results" IsTag="True" />
  </Categories>
  <Properties>
    <Property Id="CanBeDataDriven" Label="CanBeDataDriven" Description="CanBeDataDriven" DataType="System.Boolean" />
    <Property Id="CanLinkedNodesBeDataDriven" Label="CanLinkedNodesBeDataDriven" Description="CanLinkedNodesBeDataDriven" DataType="System.Boolean" />
    <Property Id="Description" Label="Description" Description="Description" DataType="System.String" />
    <Property Id="Group" Label="Group" Description="Display the node as a group" DataType="Microsoft.VisualStudio.Progression.GraphModel.GroupStyle" />
    <Property Id="IncomingActionLabel" Label="IncomingActionLabel" Description="IncomingActionLabel" DataType="System.String" />
    <Property Id="IsContainment" DataType="System.Boolean" />
    <Property Id="IsTag" DataType="System.Boolean" />
    <Property Id="Label" Label="Label" Description="Displayable label of an Annotatable object" DataType="System.String" />
    <Property Id="OutgoingActionLabel" Label="OutgoingActionLabel" Description="OutgoingActionLabel" DataType="System.String" />
  </Properties>
  <Styles>
    <Style TargetType="Node" GroupLabel="Coupon Calcs" ValueLabel="Has category">
      <Condition Expression="HasCategory('CouponCalcs')" />
      <Setter Property="Background" Value="#FFFFFF00" />
    </Style>
    <Style TargetType="Node" GroupLabel="Par Calcs" ValueLabel="Has category">
      <Condition Expression="HasCategory('ParCalcs')" />
      <Setter Property="Background" Value="Green" />
    </Style>
    <Style TargetType="Node" GroupLabel="Inputs" ValueLabel="Has category">
      <Condition Expression="HasCategory('Inputs')" />
      <Setter Property="Background" Value="White" />
    </Style>
    <Style TargetType="Node" GroupLabel="Results" ValueLabel="Has category">
      <Condition Expression="HasCategory('Results')" />
      <Setter Property="Background" Value="Blue" />
    </Style>
  </Styles>
</DirectedGraph>

And we can also view the diagram. Note that we create a collapsible group for the iterated calculations, which could be hierarchical for very complicated models allowing you to visualise an entire set of calculations and drill down as required in order to investigate errors or inconsistencies.

Future Improvements

There is a spillover of model-specific logic in the T4 template. With a bit of thought, we could should be able to create XML loggers which hold all the model specific logic, and related XML format and then generate a T4 template which is model agnostic.

We perform the calculations and logging separately. Instead, we could encode all calculations we perform as expression trees, and walk these trees to not only calculate the values, but also perform the logging (including automatically generating the string representations of each formula). This is way beyond the scope of this article, however.

A concrete example of a reactive application

2010-02-08T20:00:00.000Z

Continuing on from my first play with the Rx framework yesterday, I thought it would be good to come up with a real world example. I thought about rss feeds and stock tickers, but in the end decided on an app with consumes the Twitter REST API.

I will build up an application which can subscribe to a couple of twitter news feeds, polling for new messages on a particular schedule, and merging the results into a single output stream to which we subscribe.

Accessing the Twitter API

I am going to keep this very simple, just access the user_timeline service for a given user, and get the results in XML format, which make it easy for us to project into a Tweet class which we define. I create a class called TwitterAccess with one key static method:

public static Tweet[] GetNewTweets(string username, long lastTweetId)
{
    if(lastTweetId < 1)
        throw new ArgumentOutOfRangeException("lastTweetId", "id must be greater than 0");

    string url = String.Format("http://twitter.com/statuses/user_timeline/{0}.xml?since_id={1}",
                               username.Replace("@", ""), lastTweetId);
    var doc = XDocument.Load(url);

    var items = from s in doc.Descendants("status")
                orderby long.Parse(s.Element("id").Value)
                select new Tweet { Id = long.Parse(s.Element("id").Value),
                                   Time = ParseTwitterDate(s.Element("created_at").Value),
                                   Username = s.Element("user").Element("screen_name").Value,
                                   Status = s.Element("text").Value };
  
    return items.ToArray();
}

This method gets all tweets on the given users timeline whose ID is greater than lastTweetId (although the default limit in twitter is 20 tweets, you will need to specify the count parameter in the rest url if you need more, but this suits our purpose).

The Tweet class just wraps the properties you can see above, and the ParseTwitterDate method just does a DateTime.ParseExact for the specific date format used in twitter.

Wrapping Twitter in an IObservable<T>

Now that we have a means to access twitter, we should create a class which handles the polling of this service and pushes all new tweets out through an IObservable<Tweet> as follows:

public class TweetStream
{
    private long lastTweet = 1;
    private object updateLock = new object();
    private IObservable<Tweet> stream;

    public IObservable<Tweet> Stream
    {
        get { return stream; }
    }

    public TweetStream(string username, int interval)
    {
        var xs = Observable.Interval(TimeSpan.FromSeconds(interval));
        stream = xs.SelectMany(count =>
            {
                Tweet[] items;
                lock (updateLock)
                {
                    items = TwitterAccess.GetNewTweets(username, lastTweet);
                    lastTweet = items.Length > 0 ? items.Last().Id : lastTweet;
                }
                return items.ToObservable();
            });
    }
}

The SelectMany call allows us to define a function which is called each time the interval fires and which itself returns an IObservable. Note that we must use SelectMany instead of Select as SelectMany will automatically flatten all the items in the returned stream into its own stream of Tweets whereas Select would leave you with a stream of IObservables, which will then be difficult to handle.

Getting some output from this stream

I can now update the console app we looked at yesterday in order to create an instance of TweetStream and subscribe to its output, writing this in turn out to the console.

TweetStream bbc = new TweetStream("bbcnews", 120);

var sub = bbc.Stream.Subscribe(i => Console.WriteLine("{0}: {1}: {2}",
                                i.Time.ToString("dd hh:mm"), i.Username, i.Status));

Running this up we get the following output:

07 08:10: bbcnews: A man who was injured during a Premier League football match at Stoke City dies in hospital.  http://
bit.ly/bDjtDM
07 09:00: bbcnews: Worst snow storms to hit eastern US for decades  http://bit.ly/93Jpn9
07 09:00: bbcnews: Iran's leader asks the country's nuclear chief to begin enriching uranium to 20%, in a new challenge
to Western.. http://bit.ly/9AUplq
...
07 02:20: bbcnews: A Scottish pensioner becomes only the seventh person in history to achieve judo's highest rank. http:
//bit.ly/8XRsxv
07 02:40: bbcnews: Scotland take on France at Murrayfield in the third of the opening round of Six Nations fixtures. htt
p://bit.ly/98KHLc

This is very cool – without much code we now have a live twitter feed from a single source, which will automatically bring in new tweets (if available) every two minutes. As a final exercise I will attempt to merge two twitter news feeds into one single stream.

Merging Twitter Feeds

It is a simple matter now to update the above code to create two input feeds and then merge them:

TweetStream bbc = new TweetStream("bbcnews", 120);
TweetStream cnn = new TweetStream("cnn", 120);

var news = bbc.Stream.Merge(cnn.Stream);

var sub = news.Subscribe(i => Console.WriteLine("{0}: {1}: {2}",
                                i.Time.ToString("dd hh:mm"), i.Username, i.Status));

Just by calling the Merge method, the output from the two streams gets combined into a single stream. Note that the tweets from a given stream remain in order, but the relative ordering between the two streams depends on how they are scheduled to execute, the merge doesn’t work based on the feed dates or Ids – this is left as an exercise for the reader.

07 12:00: bbcnews: England and Wales are drawn in the same qualifying group for Euro 2012 in Poland and the Ukraine. htt
p://bit.ly/cCEza2
07 12:00: bbcnews: Home Secretary Alan Johnson rejects the idea that MPs facing expenses charges might avoid trial by ci
ting Parli.. http://bit.ly/9tk3X9
...
07 03:20: bbcnews: A court in Libya dismisses a case against a Swiss businessman who was accused of illegal business act
ivities. http://bit.ly/da0kFc
07 03:20: bbcnews: Police have launched an inquiry after a man's body was found on a main road in rural Stirlingshire. h
ttp://bit.ly/9xHMGp
06 01:06: CNN: Toyota: Apology but no new recall. http://on.cnn.com/cxQQrj
06 01:31: CNN: All military health facilities get Plan B pill. http://on.cnn.com/dbrZox
...
06 11:26: CNN: Tonight 9 ET: Watch the Tea Party Convention's keynote address by Sarah Palin live on CNN, CNN.com and CN
N's iPhone App.
07 11:29: CNN: Residents shovel out from heavy snow as blizzard conditions sweep across the eastern U.S.: http://bit.ly/
cFmAiU
07 11:30: CNN: Steps turn into slopes:  http://bit.ly/cyVbMM

A quick play with Reactive Extensions (Rx)

2010-02-07T20:00:00.005Z

I have finally got around to taking a look at this very interesting framework by DevLabs at Microsoft. What it promises to do is amazing, namely to revolutionize certain types of event driven (i.e. asynchronous) applications by allowing to developer to use the familiar LINQ paradigm to manage multiple streams of incoming asynchronous events.

There are already a fair number of videos about Rx available on Channel 9. In my opinion, this Introduction by Erik Meijer is the best starting point, followed by Wes Dyer’s Hello World equivalent. Following these there are a number of In Depth videos, which are indexed on the Team Blog. I cannot hope to describe Rx and its API anywhere near as well as those guys, so for the rest of this post will assume you have at least skimmed through those intros.

Getting Started

For this quick test, I downloaded the .NET Framework 3.5 SP1 variant and installed it. The install goes to Program Files\Microsoft Reactive Extensions\Redist\DesktopV2 and includes four assemblies and matching help files.
I crack open Visual Studio 2008 SP1 and create a new windows console application (oh yeah, we are pushing the boat out with this demonstration). Add references to the Rx framework assemblies:

Hello World

Lets just get straight in there and create a real hello world application using the Rx framework:

static void Main(string[] args)
{
    var xs = Observable.Return("Hello world");
    var sb = xs.Subscribe(msg => Console.WriteLine(msg));

    Console.WriteLine("Press any key to exit");
    Console.ReadKey();

    sb.Dispose();
}

In this trivial snippet, we call Observable.Return which creates an IObservable<String> whose sole purpose is to push out a single string value and then terminate. In itself this would achieve nothing unless we create something which can subscribe to this stream, so we call Subscribe on this IObservable and pass in a lambda which takes the message content (a string) and writes it to the console. So we get:

This is a start, but is not exactly stretching the capabilities of Rx so far. Over the next few demos, hopefully we can start to see some of the real possibilities.

Counting

The above demo did one thing and then stopped. For real excitement, lets now create an IObservable which just keeps pushing out events forever:

var xs = Observable.Interval(TimeSpan.FromSeconds(1));
var sb = xs.Subscribe(i => Console.WriteLine("{0}: {1}",
DateTime.Now.ToString("mm:ss"), i));

The call to Observable.Interval gives us an IObservable which will sit there pushing events continuously. It starts with an integer value of 0, and after each of the defined intervals it increments this and pushes it out again.

By itself this appears to be a relatively useless feature, but we will see later that in conjunction with some of the more advanced facilities offered by Rx, this can be quite powerful as a means to quantize some other stream of events.

Advanced Counting

For this section, lets first start by creating two separate streams of events with Observable.Interval.

var xs = Observable.Interval(TimeSpan.FromSeconds(1));
var ys = Observable.Interval(TimeSpan.FromMilliseconds(200));

var sb1 = xs.Subscribe(i => Console.WriteLine("{0}: {1} (xs)",
DateTime.Now.ToString("mm:ss"), i));
var sb2 = ys.Subscribe(i => Console.WriteLine("{0}: {1} (ys)",
DateTime.Now.ToString("mm:ss"), i));

As expected, we have two streams outputting results, one ticking five times faster than the other. If we want to synchronize these, we have two options, either throttle each stream so that we output a single event only when both streams have a new event available, or alternatively each time one of the streams pushes a new event we combine that with the most recent event of the other stream. These features are provided by the Zip and the CombineLatest methods respectively.

Zipping Up

One of the nice touches in the videos linked above is the concept of marble diagrams. These do a very good job of visualizing some potentially mind-melting concurrency scenarios, and also do a good job in showing what each of the Rx operators achieve. The simple (no failures) scenario for Zip is as follows:

Each blob represents some event occurring with xs and ys being the input streams in this case. Stream zs is created by zipping xs with ys and combining the two events using the function ‘f’.

var xs = Observable.Interval(TimeSpan.FromSeconds(1));
var ys = Observable.Interval(TimeSpan.FromMilliseconds(200));
var zs = xs.Zip(ys, (x, y) => String.Format("({0},{1})", x, y));

var sb = zs.Subscribe(i => Console.WriteLine("{0}: {1} (zs)",
DateTime.Now.ToString("mm:ss"), i));

In this method we introduce the call to Zip, which combines events from the input streams, and created new events using the lambda. In this case we simply return a string representation of the tuple.

As we can see here, the 200ms stream is effectively throttled back to the speed of the slowest stream. In many cases this may be the desired result, but consider if the two streams were different types of financial market data, one remains up to date whilst the other gets more and more stale, after one hour of this, one of the streams would be feeding data 48 minutes old.

CombineLatest

So to solve this problem, another operator, CombineLatest, steps in. The marble diagram below shows that the same type of function is performed to pair the events, but the means for achieving this is quite different:

In this case, whenever we get an event on either stream, we combine that with the most recent event (or wait until there is one) on the other stream. So once we have data on both streams, the combined event stream pushes events as fast as the fastest of the input streams.

var xs = Observable.Interval(TimeSpan.FromSeconds(1));
var ys = Observable.Interval(TimeSpan.FromMilliseconds(200));
var zs = xs.CombineLatest(ys, (x, y) => String.Format("({0},{1})", x, y));

var sb = zs.Subscribe(i => Console.WriteLine("{0}: {1} (zs)",
DateTime.Now.ToString("mm:ss"), i));

Summary

What I have briefly looked at is just the tip of the iceberg in terms of functionality. There are numerous other methods in the framework for dealing with events, and many ways in which they can be composed to achieve some very powerful results. Also, I have only looked at success scenarios so far, and Rx has much to give in terms of dealing with exceptions and retry policies on particular event streams.