CodeBetter.Com - Stuff you need to Code Better!

Using Latest NUnit Version With MSpec

hdierking — Sun, 08 Nov 2009 20:11:00 GMT

As I’ve been working another blog post about my adventures in learning BDD ala MSpec, I ran into something that I wanted to share as the problem wasn’t immediately obvious (to me anyway).

The problem is that MSpec has a dependency on version 2.4.6.0 of the nunit.framework assembly – and frankly, it’s not really a very big problem. In fact, I didn’t even notice it until now because until I had been running all of my specs exclusively through the Resharper runner (enabled by the MSpec plugin).

** As a side note, using R# with the MSpec plugin absolutely rocks, and while the runner is needed for integrating MSpec with your automated builds, I would totally recommend R# for your regular dev rhythm.

In my project, I originally had some integration tests in the same assembly as my specifications. As a result, I had a direct reference to NUnit (version 2.5.2.9222) in addition to the reference to Machine.Specifications and Machine.Specifications.NUnit (which has a reference to version 2.4.6.0). As a result, the later version of NUnit was being copied to my bin folder and throwing a file load exception for every specification when run using the console runner.

My initial reaction was to simply move all my integration tests to a different assembly and remove the direct NUnit reference in my specifications assembly (remember that referencing Machine.Specifications.NUnit will copy the version of NUnit to the bin folder that MSpec expects. No problem – everything works as you would expect.

However, it seems reasonable to me that, at least in larger projects that are wanting to incorporate specs and not have to re-write all existing tests, that it would be good to simply reconfigure the MSpec console runner to use the latest version of NUnit. This can be done easily using the standard .NET assembly binding redirection feature. Simply create a file named Machine.Specifications.ConsoleRunner.exe.config and drop it into the MSpec folder that contains your console runner (Machine.Specifications.ConsoleRunner.exe). The contents of the file should look like this.

<?xml version="1.0"?>
<configuration>
    <runtime>
      <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
        <dependentAssembly>
          <assemblyIdentity name="nunit.framework"
                            publicKeyToken="96d09a1eb7f44a77"
                            culture="neutral" />
          <bindingRedirect oldVersion="2.4.6.0"
                           newVersion="2.5.2.9222"/>
        </dependentAssembly>
      </assemblyBinding>
    </runtime>
</configuration>

Run the MSpec console again, and voila! Works like a charm.

Astoria, SSIS Adapters and BDD – Code

hdierking — Fri, 06 Nov 2009 19:57:00 GMT

I’m currently working on the post talking about how, in the course of writing my SSIS data source for ADO.NET Data Services, I stumbled into replacing my unit tests with specifications. However, in advance of that, I wanted to go ahead and provide the link to the source code. I would love to get feedback on what’s working and what’s not so that I can get better – and so that ideally, those the random stranger who downloads and looks at it on some idle Tuesday gets better as well.

The code can be found at http://code.msdn.microsoft.com/ssisastoria

December 2-3 London Workshop Update

Dave Laribee — Thu, 05 Nov 2009 15:29:00 GMT

Forgive the self-promotional interruption, but quick housekeeping note:

I've asked the company hosting my workshop in London to reduce the price of my 2-day workshop on December 2-3 aimed at tools and techniques technical leads, folks interested in lean, standard work, etc. to try to get as many people there (it's a group collaboration focused thing). I also want to get as much feedback as I can. If you book before November 15th, the price is £550.

The code is the truth, but it is not the whole truth

Patrick Smacchia — Wed, 04 Nov 2009 13:55:00 GMT

In a recent interview from Grady Booch, co-creator of UML, Grady said:

When Jim, Ivar, and I began our journey that became manifest in the UML, we never intended it to become a programming language. I think that there's a fairly narrow domain for which model-driven development makes sense but that we should return to the roots of the UML, which was to be a language for visualizing, specifying, constructing, and documenting the artifacts of a software-intensive system—in short, a graphical language to help reason about the design of a system as it unfolds. Most diagrams should be thrown away, but there are a few that should be preserved, and in all, one should only use a graphical notation for those things that cannot easily be reasoned about in code. As I've also often said, the code is the truth, but it is not the whole truth, and there are things such as rationale, cross-cutting concerns, and patterns that cannot easily be recovered or seen from code.... These are the things for which a graphical notation adds value, and any such notation should be used only if it has predictive power or reasoning power (meaning, you can ask questions about it).

I especially cannot agree more with:

As I've also often said, the code is the truth, but it is not the whole truth, and there are things such as rationale, cross-cutting concerns, and patterns that cannot easily be recovered or seen from code....

The code is the truth, the code represents the whole asset of a software shop. But code base is a complex asset. Call them patterns, rationales, intentions or decisions, there are many valuable intellectual artifacts buried in a code base that cannot easily be recovered or seen from code.

This is something I realized in 1997 while trying for the first time to demystify a C++ legacy code base. I was angry! What, in 1997, we, the professional developers, haven’t any tooling to visualize code? I remember also, in the architecture department next door, the architects were spending days doing diagrams that had nothing to do with what actually the code was. Putting hands in the code was something considered with low value. It was let for fresh graduated like me.

Hopefully, the result of this period was the seed from where the idea of a tool such as NDepend came to me. Code Metrics, Treemap, Dependency Graph, Dependency Matrix, Instability vs Abstractness diagrams… are natural artifacts to shed light on all these precious intentions buried in the code. I especially like the word intentions that was often used by Bruno Boucard, a former colleague I have great respect for. The idea of a language such as Code Query Language came to me when the need to make all these intentions concrete and verifiable, was clear.

What we do is the result of what we did and who we met, no matter whether it was good or bad experiences. There is no such thing as: today, I gonna find an idea to make my own business. Good ideas come naturally along the way, through what, Paulo Coehlo, would call signs.

Introduction to the Reactive Framework Part II

Matthew.Podwysocki — Tue, 03 Nov 2009 05:28:37 GMT

In my previous post, I talked a little bit about the parts of the Reactive Framework that are coming to the .NET 4 BCL as well as their implementation in F# as part of F# First Class Events. This time, let’s come back to the Reactive Framework itself from which the IObservable<T> and IObserver<T> originated. As you may remember, you can play with the bits of the Reactive Framework from the Silverlight 3 Toolkit.

In the first post in the series, I gave a basic introduction to the basic problems we have with asynchronous and reactive programming. We covered some of the evolution of the .NET framework with asynchronous programming and where the Reactive Framework fits. This time, I’ll pick up where we left off to talk about one of the important ideas behind the Reactive Framework, the duality of Enumerable and Observable.

The Duality of Enumerable & Observable

During Erik’s appearances on Channel 9, an important, yet recurring theme was the mathematical duality of the Enumerable to the Observable. What does that mean exactly though?

We remember from our first post in the series where we talked about the pull (interactive) versus the push (reactive) model. The pull model, represented by the iterator pattern of IEnumerable<T>/IEnumerator<T> states that we must explicitly call a method in order to get each item from our abstracted collection. On the other hand, our push model, represented by the observable pattern of IObservable<T>/IObserver<T> states that we register an interest through a subscription and then items are subsequently handed to us from some abstracted collection.

First, let’s look at the iterator pattern, which is a common interactive programming pattern that’s used quite frequently inside .NET. This consists of two interfaces, the IEnumerable<T> and the IEnumerator<T>. Listed below are some of the highlights of these two interfaces. I cut down what they are to their essential bits.

public interface IEnumerable<T>
{
    IEnumerator<T> GetEnumerator();
} 

public interface IEnumerator<T> : IDisposable
{
    T Current { get; }
    bool MoveNext();
}

The IEnumerable<T> interface exposes a single method which returns an IEnumerator<T> to iterate some object. The IEnumerator<T> gives us the ability to get the current item and determine whether there are more items to iterate.

To think about basic asynchronous programming, we must consider a basic Gang of Four Pattern, the Observer Pattern. This pattern describes where an object called an Observable, maintains a list of its dependent Observer classes and notifies them automatically of any state changes. In Java, this should be rather familiar territory with the java.util.Observer class and the java.util.Observable interfaces. In the Reactive Framework, this approach is no different, as we have an IObservable<T> interface which maintains a list of dependent IObserver<T> interfaces and notifies them automatically of any state changes. Let’s look at the signatures below:

public interface IObserver<T>
{
    void OnCompleted();
    void OnError(Exception exception);
    void OnNext(T value);
} 

public interface IObservable<T>
{
    IDisposable Subscribe(IObserver<T> observer);
}

What you may notice, and what you may have seen on some of Erik Meijer’s Channel 9 videos is that there is a duality between the Enumerable and Observable, given the signature of these interfaces. To explain what I mean, let’s first look at the flow of the Enumerator:

The GetEnumerator function takes no arguments and returns the IEnumerator<T> which has a no-argument function and returns the next item in the list through MoveNext and Current. Now, let’s look at the inverse of it.

The duality of the Enumerable solution is quite simple. We have our observer on the left hand side which has an action function which takes a T argument and returns void. The right hand side is the notification that is sent when something happens. This is all managed by the Observable as it maintains a list of all observers so that it can notify them.

To make this a little more concrete, let mash these two sets of interfaces together in order to dualize. First, let’s start with the IEnumerator<T> interface.

public interface IEnumerator<T> : IDisposable
{
    T Current { get; } // throws exception
    bool MoveNext();
}

What we’ll notice is that we have two items the Current property with a get accessor and the MoveNext method which takes nothing and returns a boolean that indicates if it can move next. You’ll also notice that our Current property also could throw an exception. If we’re in the Java world, we’d explicitly mark this interface with throws Exception, but as C# doesn’t have that, this comment will do. Now, let’s look to dualize this interface.

public interface IDualEnumerator<T>
{
    void SetCurrent(T value | Exception ex);
    void MoveNext(bool canMove);
}

At first glance, all we have to do is flip our Current property to have a setter. Because the Current yielded a value or threw an exception, that’s the way we’ll model it for now. Secondly, our MoveNext will now take a boolean to indicate whether we’re done. Let’s refactor this a little bit more and break the SetCurrent into two distinct operations, one for yielding the next value, and one for throwing an exception.

public interface IDualEnumerator<T>
{
    void Yield(T value);
    void Throw(Exception ex);
    void MoveNext(bool canMove);
}

Now that we have that under control, let’s then move onto our MoveNext method. As you’ll notice, we’ll continue to send in a true until we’re finished pushing values. Instead of calling MoveNext repeatedly, let’s instead call this only once when we’re finished.

public interface IDualEnumerator<T>
{
    void Yield(T value);
    void Throw(Exception ex);
    void Break();
}

What we then end up with is pretty much the signature of our IObserver<T>. By renaming our Yield to OnNext, our Throw to OnException and Break to OnCompleted, we’re left with our original IObserver<T> interface.

public interface IObserver<T>
{   
    void OnNext(T value);
    void OnError(Exception exception);
    void OnCompleted();
}

What about the dual of IEnumerable<T>? Let’s first look at what that interface is.

public interface IEnumerable<T>
{
    IEnumerator<T> GetEnumerator();
}

This interface has a single method called GetEnumerator which returns an IEnumerator<T>. How might we dualize this interface? Well, we already know that the dual of the IEnumerator<T> is IObservable<T>, just as well, we need a way to attach interest in a given dualized Enumerator, so we’ll create a method called Attach. Just as well, when we attach interest, we also need a way to detach our interest for a given observer. That might look like the following.

public interface IDualEnumerable<T>
{
    void Attach(IObserver<T> observer);
    void Detach(IObserver<T> observer);
}

If we look at the Java interfaces for Observer and Observable, these are pretty close in signature. There ultimately is a problem with this scenario in terms of composition. At each level, it needs to remember which things maps to which so that you could undo any of the observers if need be. Not doing so could lead to space leaks and is generally a bad idea. How could we solve this then? Ultimately, we need a way to track our attachment once we call the Attach through some form of a return value. At this point, the tracking is nothing more than needing the ability to detach our given Observer. If you’ll notice in our IEnumerator<T> interface, it inherits the IDisposable interface which gets one thinking. Could we instead, change our Attach method so that when we call it, we return an IDisposable instead which gives us the ability to track our subscription and be able to clean up after we’re done? That’s exactly the approach that the IObservable<T> takes in the final version of the interface.

public interface IObservable<T>
{
    IDisposable Subscribe(IObserver<T> observer);
}

Now that we’ve proven the duality, why does it matter? Because of this duality, the laws that apply to LINQ to Objects now can apply to LINQ to Observables (or Events). And in fact, we can see that through the Observable class which has many of our standard LINQ combinators such as:

Aggregate
First/FirstOrDefault
GroupBy
Join
Last/LastOrDefault
Select
SelectMany
Single/SingleOrDefault
Skip/SkipWhile
Take/TakeWhile
Where
Zip

Because of this duality, we can do such simple things as fire the ProgressChanged event until the RunWorkerCompleted event happens.

var worker = new BackgroundWorker();

var percentage = from progress in worker.ProgressChangedEvent()
                           .Until(worker.RunWorkerCompletedEvent())
                 select progress.EventArgs.ProgressPercentage;

var subscription = percentage.Subscribe(
    p => Console.WriteLine("Percentage Complete: {0}", p));

That’s only scratching the surface of the power of the Reactive Framework. We’ll go in depth in the next part of the series on the combinators. We could go into the fact that the Enumerable solution is really just the lazy list monad and the Observable solution is really just the continuation monad, but that’s for another time as well.

Conclusion

As I’ve stated before, the LiveLabs Reactive Framework gives us the ability to harness reactive programming and treat events as the first class citizens they should have been using LINQ expressions and other standard LINQ combinators. Still, there is much to cover in this series with the standard LINQ combinators, the monadic heritage of this solution and more.

Should I use MEF with an IoC container? - Part 1

Glenn Block — Sun, 01 Nov 2009 06:36:00 GMT

In my post "Should I use MEF for my IoC needs" we took a look at the question of whether or not it is appropriate to use MEF instead of a traditional IoC container. In this post we'll discuss the question of whether or not you should use both an IoC container and MEF in the same application.

MEF and IoC container concerns

As we've mentioned in the past, MEF's primary design in V1 is around open-systems whose components (parts) are dynamically discovered at runtime. IoC containers on the other hand are primarily designed for decoupling a closed (fixed) set of application components in order to improve the maintainability of the system by the engineering team. In other words each is addressing a specific set of concerns. So if that is the case then is it plausable to use both in an application? The answer is Yes. Not only that, but we specifically designed MEF with the idea of integration with other containers in mind. Below will explore one of the key-scenarios for doing this.

Enabling third-party extensibility in an IoC container-dependent application.

You have an existing customer management system that uses an IoC container such as Castle Windsor which you use in order to promote separation-of-concens and testability. The application leverages advanced Windsor facilities for integrating with NHibernate and for applying cross-cutting-concerns (AOP) such as logging, etc. The loosely coupled design of the system has proven to be greatly beneficial and has minimized the friction for developing and deploying the system through several releases.

At some point requests start pouring in from customers want to create their own extensions in order to customize the application in their specific environments. Additionally several ISVs express interest in providing third-party value-adds such as telephony integration, and scheduling. You determine that adding a third-party extensibility model to the application makes sense in order to address this opportunity. Through your research you come across MEF and realize that it provides the infrastructure you need for extensibility. However it does so in a manner that overlaps with many of the functions that Windsor is providing. What to do? Let's explore various options.

a. Migrate the existing code to use MEF and remove the IoC container completely.

One option is to consider ripping out Castle and just using MEF. Technically this sounds reasonable, however after investigation it is not feasible. There are literally thousands of components that are currently registered in the Windsor container. Moving these to MEF involves modfying all of these existing classes in order to make them discoverable by MEF, not to mention refactoring all the places in the application where the container is accessed. Because the changes are so widespread, the likelihood of introducing breakages / bugs in the system is very high. Additionally there are the various Castle facilities such as Dynamic Proxy, and NHibernate integration which MEF does not yet support, which will have to authored by the team. The cost / risk of such changes is too prohibitive, not to mention that it seems like it should be unnecessary.

b. Implement a custom extensibility mechanism built on top of the IoC container?

Another option is to author a home-grown solution. This can be a viable option depending on the needs, and the willingness to author your own solution. For simple cases you can easily facade an extensibility mechanism on top of an IoC. Once you start getting into more advanced features like lazy instantiation, metadata, recomposition and stable composition, you are in for quite a bit of work.

c. Have both the IoC and MEF both present in the same application?

The final option I want to discuss (and the one I recommend for these situations) is to use both. Use the IoC container for addressing internal application decoupling needs, and use MEF for third-party extensibiltiy needs, that is the IoC container for IoCish things and the MEF container for MEFfish things :-)

This path is appealing for several reasons.

It works with existing codebases whether they be applications or frameworks. it does not require any significant rewrite.
It allows leveraging the strengths of each container for the things it does best. For example leveraging MEF's metadata model for discriminating between different extensions vs the dynamic proxy support in Castle.

However, there are several caveats of this approach.

It raises questions such as:
- Which component (part) should live in which container?
- Who owns the lifetime?
- Can a MEF part access services coming from the IoC? How?
- Can the IoC access exports coming from MEF? How?
It increases complexity in that now there are components living in mulitple places.
It increases the potential points of failuire within the application.

All of these issues are managable. In my next post we'll see how.

Getting Your Groove Off

hdierking — Thu, 29 Oct 2009 21:26:00 GMT

Until recently, the only knowledge I had of Groove was that it was some kind of collaboration product that we got as a result of bringing in Ray Ozzie. Then I got a new manager who insisted that our team standardize on Groove for the team document repository - so I got setup on Groove and went on with my life. In general, there’s nothing I found terribly impressive about it, but it wasn’t really getting in my way so no reason to get too upset…

Until my Tortoise SVN icon overlays stopped working.

I ended up spending quite a bit of time this morning trying to figure out exactly why they weren’t working. It was odd – I had no problems on my home machine – but on both of my work machines, same result – no TSVN icons. Finally, @pilotbob pointed me to this article which set off a chain of recollections – most notably that the major difference between my work machines and my home machine had to do with the order that I installed stuff. On my home machine, I installed Office after I got my dev environment setup. However, on my work machine, Office was installed as a part of the default OS install image - and Groove was included in that installation.

So why is this important? Because if you remember from the article, Windows supports only 15 overlay handler slots – and installing Groove puts 5 handlers right there on top of my TSVN handlers – the net result being that my TSVN icon overlays don’t load.

To fix, I simply went into the registry, deleted all of the Groove overlay handlers from HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\ShellIconOverlayIdentifiers

Then rebooted. Voila! TSVN icon overlays come right back up!

Low-Technology

Dave Laribee — Thu, 29 Oct 2009 18:55:00 GMT

I'm completely obsessed with the idea of low-technology:

The term low-technology is a description of those crafts and tools whose inception (typically) predates the Industrial Revolution.

A test for low-technology may be that it can be practiced or fabricated with a minimum of Capital investment by an individual or small group of individuals; and that the knowledge of the practice can be completely comprehended by a single individual, free from increasing specialization and compartmentalization.

An excellent overview of some of the cutting edge efforts in low-technologies can be seen in this TED video with low-technologist Amy Smith. I completely agree with when says, "not all inventions need to be grandiose, complex things... sometimes they can be simple and smart ideas that just help a lot of people."

The second block quote is where I'm coming from. What are those tools in software development that add lots of value without minimal getting started overhead. Pair-programming seems like an excellent example of low technology. Emergent design, test-driven development and value-driven approaches like XP are yet more.

It seems, to me anyway, that these are the things that'll yield the most benefits. Both vendor-supplied tooling and trending topics we hold near and dear, like Domain-Driven Design and DSLs and this fluent madness in general, strike me as more high-technology approaches.

There's nothing wrong with high-technology, of course, but my instinct is that the risk-reward equation for most teams usually favors low-technology. We have, after all, limited bandwidth and attention, so my vote will ever go to what I see as big and easy wins: stuff focused on beefing up the signal of our communication as a team and community against the noise we might get as a software fashion victim.

Anyway, this is a key topic I'll be centering on over the next little while and thought y'all might find this way of looking at our kit, our standard tradecraft at least a little bit interesting.

Eschewing Date Types in our Database

karl — Thu, 29 Oct 2009 13:41:00 GMT

About a month ago, I found myself transforming data for the reporting portion of our application. Like many reporting systems, ours works off of date ranges, with statistics grouped together by the day (and a few by the hour). An early prototype made heavy use of TSQL Date functions (like DateAdd and DatePart). However, I wasn't happy with this solution, my transformations were unnecessarily coupled to SQL Server. I mean, database date functions all do pretty primitive date arithmetic's, surely we can come up with a way to do that ourselves?

I decided to borrow the strategy long-used by the Unix and PHP community (or JavaScript or doubtless countless other platforms/languages) - to store dates as seconds relative to a starting point. Unix Epoch is January 1st, 1970. If you store seconds since then in an unsigned 32 bit integer, you can store up to 136 years. If you only need minute-resolution, you can store over 8 100 years. If you only need day-resolution, then you can store a bit under 12 million years.

Best of all, you can do straight up arithmetic's against these fields to manipulate the dates. If you want to group hits per user for a day, you can do:

insert into DailyHits (UserId, DaysSince, Count)
select UserId, SecondsSince/86400, count(*) 
   from Hits 
   group by UserId, SecondsSince/86400
--86400 is the number of seconds in a day

insert into HourlyHits (UserId, HoursSince, Count)
select UserId, SecondsSince/3600, count(*) 
   from Hits 
   group by UserId, SecondsSince/3600 
--3600 is the number of seconds in a hour

You can also manipulate relative times in C#, to apply as parameters:

public static readonly DateTime Epoch = new DateTime(1970, 1, 1);
public IList GetDailyHits(DateTime from, DateTime to)
{

   command.CommandText = "select * from DailyHits where DaysSince > @From and DaysSince < @To";
   command.Parameters.Add("@From", from.Substract(Epoch).TotalDays);
   command.Parameters.Add("@To", to.Substract(Epoch).TotalDays);
   ...
}

If you are using NHibernate, you can use the IUserType interface to help map DateTimes to their relative counterparts (I think).

The examples above are simplistic, but they do work very well and have a number of benefits - the most significant being great independence from the underlying database.

Astoria, SSIS Adapters and BDD – Take 1

hdierking — Wed, 28 Oct 2009 16:00:17 GMT

Yea I know – 3 things you never thought you would hear mentioned together. And to be really honest, I didn’t start out trying to build this in a BDD fashion – and since I’m totally new to the disciplines of BDD, I may not have been all that successful. But hey, it’s all about the journey, and so I wanted to at least go ahead and start posting about the work I’ve been doing here, partly to share and partly to get your input ways to improve my design/specs.

So here was the initial goal – I was studying ADO.NET Data Services (previously known as Astoria) when I remembered this article that I once read in the Architecture Journal about using RSS as a technique for surfacing data to load into an enterprise data warehouse. That led me to think that it would be a cool opportunity to simultaneously learn Astoria and feed my BI sweet tooth by writing a SSIS data source that could consume an Astoria endpoint. Given the fact that more and more systems are exposing data using syndication, it seemed like creating an SSIS data source was something that could be helpful beyond my simply wanting to play with the technology.

Wait - ADO.NET Data Services what??

Just in case you weren’t 100% sure what ADO.NET Data Services is, it’s technology to expose data and enable operations on data via RESTful operations. You can find out much more about it here and here, but for the purposes of this exercise, here are the 3 things you need to know.

It uses all standard HTTP operations – which means that for getting data, you simply format a HTTP GET request for the service URL plus additional path information and optional querystring arguments to tell the framework how to shape the data.
While there’s not a dependency on the Entity Framework, there is a dependency on the Entity Data Model in that metadata is exposed as CSDL.
Though you have a couple formatting options when getting data, the primary one (and the one used by the data source component) is to get the data as an ATOM feed. The data source component can then project the XML into the appropriate output buffer (or buffers as you’ll see later).

Wait – SSIS what??

Sql Server Integration Services is Microsoft’s ETL tool that ships with SQL Server. You can start learning more about it here, but for the purpose of this exercise, here are the 3 things you need to know.

The unit of execution in SSIS is the package. This is really similar nomenclature to DTS because, not surprisingly, SSIS evolved out of DTS.
Every package contains 2 kinds of logic: 1 “Control Flow” and zero-to-many “Data Flows”. Control flow is meant for larger operations like copying files or running some kind of database maintenance operation (and while interesting, it’s not something I care about for this exercise). The real meat of SSIS (IMHO) happens within a data flow – this is where data is extracted from a source, massaged, and put into a destination. Data flows maintain the entire pipeline of operations between source and destination via in memory buffers.
Data flows use data source components to extract data from the source and add it to the appropriate buffers in the pipeline. Data source components are also used to provide metadata to the SSIS design-time experience so that package designers can use it to clean up and massage the data using transformation components (** for those experienced SSIS folks, it’s worth noting that there is an XML data source component that is a part of the product, and since ADO.NET Data Services returns data as an ATOM feed, is is absolutely possible to consume it directly. However, in order to take this approach, you will need to generate a XSD schema and associate it with the component at design time for the metadata – and every time you change the shape of your data services query, you will have to re-generate and re-associate a new schema. This was one of the primary motivators for me to create a custom data source).

So then, the initial goals were as follows -

Create an SSIS data source component that could consume an ADO.NET Data Services endpoint
Ensure that the component participated in both the design time and runtime experiences
Ensure that the component was able to handle projections of ADO.NET Data services queries that took advantage of the $expand clause

As I got into the development I was pretty quickly able to accomplish the first 2 goals. For example, given the following query configuration..

I was able to get the data shape using ADO.NET Data Service’s $metadata option and expose it back to the designer..

I was even able to run the package for a simple data services query and pipe the results to a flat file..

However, when I started implementing that 3rd goal of managing projections, I quickly realized my quick and dirty project need more thought in its design. This drove me to add a few more goals.

Project should separate the core logic of the component from the SSIS infrastructure – which is heavily reliant on inheritance
Core component logic should be independently testable without requiring a running data services endpoint

Separating the logic out into a different assembly wasn’t really all that painful – just a few keystrokes of Resharper, a couple file moves, a reference added, and I was good to go. However, because this started as a quick and dirty project (I should have known better), I hadn’t designed for data source independence and as such had some larger refactorings to do. And in that process (** keep in mind that this started as an remains primarily a learning exercise for me – I would NEVER recommend taking this kind of “while I’m at it approach” on a real project), I decided that my tests weren’t really where I wanted them and that I had always wanted to get my hands dirty with BDD anyway, so why not? It’s just one more goal.

Tests should be written as specifications

I’ve read from some that BDD is really meant more for testing larger behaviors in business applications. However, it feels pretty good (though initially uncomfortable) at the grain at which I’m using it. For example..

public class when_products_resource_is_queried : with_northwind_CSDL
{
    Establish context = () => queryBuilder.ResourcePath = "Products";

    Because of = () => query = queryBuilder.GetQuery();

    It should_have_8_properties = () => 
        query.DataShape.First().Properties.Count.ShouldEqual(8);

    It should_have_a_single_resource = () => 
        query.DataShape.Count().ShouldEqual(1);

    It should_have_1_key_property = () => 
        query.DataShape.First().Keys.Count.ShouldEqual(1);

    It should_have_correct_query_string = () =>
        query.ToString().ShouldEqual(string.Format(
            "{0}/Products", Constants.SERVICE_URI));
}

Ok – so those were the goals as they evolved. Going forward, my plan is to dig into the logic design and specifications, then talk specifically about projections, then finally talk about how this stuff works with SSIS (and how I would like to see the SSIS API simplified). Sorry to not get into all that tonight, but as my friend Glenn has advised me countless times, if you wait to blog until everything is nice and tidy and completed, you may never blog again.

The Most Productive Programming Language

Dave Laribee — Wed, 28 Oct 2009 00:08:00 GMT

The most productive and useful programming language you know, by a long shot, is English or Hindi or Portuguese or Spanish or French or whatever human language your team uses. Good communication skills are way more valuable to you and your customers than picking up a new language every year. That is, while there is value in the item on the right, I value the item on the left more. Sound familiar?

Developers that name things well and program with a respect for human languages create more maintainable and easier to follow code. Being able to communicate with a customer, analyst and/or product owner can save you hour upon hour of effort and keep your solution simple. No programming language no matter how syntactically minimal or internal DSL friendly can ever come close to this.

Hierarchical Namespace Component

Patrick Smacchia — Mon, 26 Oct 2009 14:49:00 GMT

Modeling components of a code base with namespaces is a much better idea than modeling them with assemblies. I wrote about the namespaces vs. assemblies debate in the past (Control components dependencies to gain clear architecture). The fact is that namespace is a logical artifact while assembly is a physical artifact. Of course, creating assemblies to partition parts of the code that at runtime physically don’t live within the same AppDomain is a good thing. I explained that in Advices on partitioning code through .NET assemblies. Unfortunately, the vast majority of .NET teams use assemblies instead of namespaces to partition logically their code. They end up with hundreds of Visual Studio projects:

Hundreds of VS projects organized through dozens of VS solutions is something hard to manage.
These hundreds of projects takes a lot of extra time to be compiled (because of the repetitive tier assemblies parsing). By extra time I mean that the compilation takes minutes instead of seconds!
And finally, loading all these assemblies at runtime slow down consequently the startup time of applications. Indeed, doing so multiply the unit overhead that the CLR needs to load an assembly. The overhead is mainly due to CAS security checks, cryptographic signing check, and IO file read.

In order to model logical components, the logical nature of namespace is a preemptive advantage over the physical nature of assembly. However namespace have a second awesome characteristic: they can be declared hierarchically. So namespaces can be use to modelize a hierarchy of components. Today I would like to ramble on some consequences and caveat of organizing hierarchically components.

Theory: Tree vs. Flat view

This hierarchical way of componentizing code is something that we took account early in the NDepend development. At any time, the user can choose to display namespaces through tree (i.e through a hierarchy)…

…or to display flat namespaces:

Personally I prefer the tree option. After all, why do developers naturally partition the code? Because of the need to organize code and API through some smaller chunks. Smaller chunks are naturally easier to develop, understand, maintain, test and debug. This is the famous divide and conquer tenet.

But when the code base is consequently growing developers are facing a dilemma:

Smaller components are easier to be managed one by one. But this implies to deal with hundreds of components. As a consequence the big picture becomes messy.
Keeping a small number of components force to have large components. While the big picture remain understandable, larger components are harder to manage.

This is why developers apply recursively the divide and conquer idea. As a result, they obtain a hierarchical organization of code.

Practice: Tree vs. Flat view

These 2 dependency graphs illustrate well the point. The first one is made of the 52 namespaces declared in the mscorlib (this is the flat view).

The second one is made of the 16 root namespaces of the DLL mscorlib (this is the root tree view). The root tree view present less information than the flat view, but at least it is readable.

Moreover the root tree view is a good start to dig into the code and browse nested namespaces inter-dependency:

The same phenomenon appears with the Dependency Matrix. The root tree view (first matrix) is a better way to browse the code than the flat view (second matrix).

Component Hierarchy and Dependency Cycle

Avoiding cycles between components is an essential point to care for, in order to keep a code base maintainable. I detailed that in the post: Maintainability, Learnability, Component, Layering. When components are hierarchically organized, some non-trivial cycles can appear. Let's look at a small example.

If we look at the dependency between the 3 following namespaces we won’t find any cycles:

namespace ParentA.Child1 {
   public class ClassA1 { private ParentB.ClassB m_Field; }
}
namespace ParentA.Child2 {
   public class ClassA2 { }
}
namespace ParentB {
   public class ClassB { private ParentA.Child2.ClassA2 m_Field; }
}

There is no cycle, this is clear from the flat dependency view,

This is what I called a non-trivial cycle because a cycle exists while there are no cycles between namespaces. The fact is that RootA.* is not considered as a namespace because it doesn’t contain directly any class. In our internal NDepend terminology we named RootA.* a namespace container. Technically your code is entangled and contain a cycle even if physically there is no cycle between namespaces!

Hierarchical components constitute a powerful way to partition code but be aware that some non-trivial entangling cases might appear. In a next post, I’ll expose a cool way to use hierarchical componentization effectively in the real world.

Something you can (not) do with your (old) MSDN disks

pvanooijen — Mon, 26 Oct 2009 12:52:00 GMT

Time to renew my MSDN subscription. I don’t know about you but I’m still very pleased with the MSDN program.

In the old pre DVD days it used to be quite a problem what to do with the enormous piles of disks you received over and over again. Especially those things you never use. Anyone interested in a Finish version of Exchange server ? A Greek version of Windows 98 ? Since DVD times the piles has gotten a little smaller, but still. Googling around what to do with all these disks pops up an old rant of mine on this.

Being a biologist with a huge interest in aquarium fish (see my W7 betta rant) and an “old goat-wool-socks” environmental concern I thought I had a good use. Digital disk are very reflective, using them as a reflector for the light in my aquarium would give the disks a new life and would increase the effectiveness of fish tank lighting. So I tried.

This is what the result looked like in less than a year.

Moist has crept between the reflective (aluminum) layer and the (polycarbonate) disk itself and completely oxidized the metal. Resulting in a surface which reflects far less than the ground surface itself. Lesson learned: this does not work.

What you can do today with your disks is what I already asked for in my original rant: don’t get them. MSDN subscriptions now offer a download only option. You are free to download what you need (and what’s in your license). For many a product this was what I did already, downloading got me the product long before the disk finally arrived by mail. So if you want to stop wasting resources too: do switch to this subscription type. The only wish I have left is to save a little on money. Not producing and distributing the disks saves MS in the costs. I would like a small share of that in the form of a reduction on my MSDN fee. Fair deal ?

Postgresql - Day 2

karl — Sun, 25 Oct 2009 01:40:00 GMT

My migration from SQL Server to PostgreSQL for an upcoming project continues. On day 1 I managed to get the application up and running quickly, thanks to NHibernate. Ayende was kind enough to post some advice to minimize the already-few problems I ran into (I've since implemented the ISQLExceptionConverter and am really happy about it).

Today I had a bigger challenge - making the OLTP to OLAP transformation script work. I knew this would be more difficult because I'm using straight up SQL (instead of HQL), as well as a couple instances of SQLBulkCopy. So far, most of the transformation is done in sql - there's very little data being pulled down and processed by in code. Luckily, I have a huge and thorough integration test, which populates the database with a couple days worth of data, executes the transformation script in steps (as it would be in real life), and then validates the results. Essentially my test figures out the results in C# (by looping over the entities and aggregating the results every which way) and compares it to what the transformation scripts does.

Aside from a couple syntactical differences, such as having used [ ] to escape the count column, 1 column ambiguity that PostgreSQL couldn't resolve (weird), and having used insert {table} instead of insert into {table}, I only ran into 2 major problem.

The first was my usage of SqlBulkCopy which is tied to SQL Server's bcp utility. I was only using it to bulk-load the data for my integration test, but it did make a huge difference, and I wanted to keep the same performance if possible. It turns out that PostgreSQL supports a COPY from STDIN command, which the .NET Npgsql library exposes as an NpgsqlCopyIn class. Although it didn't perform quite as well (it still rocked), I much prefer the implementation. You essentially write a CSV to a stream - much nicer than having to transform my objects to datatables. Here's the reusable code that I used:

public static void BulkCopy<T>(IEnumerable<T> items, Func<T, object[]> callback, string tableName, string columns)
{
    var sql = string.Format("copy {0} {1} from stdin with delimiter '|'", tableName, columns >> string.Empty);
    using (var connection = new NpgsqlConnection(_connectionString))
    {
        connection.Open();                
        using (var command = new NpgsqlCommand(sql, connection))                
        {                    
            var copy = new NpgsqlCopyIn(command, connection);                    
            try
            {
                copy.Start();                        
                foreach (var item in items)
                {
                    var data = SerializeData(callback(item));
                    var raw = Encoding.UTF8.GetBytes(string.Concat(data, "\n"));
                    copy.CopyStream.Write(raw, 0, raw.Length);
                }
            }
            catch (Exception)
            {
                copy.Cancel("Undo copy");
                throw;
            }
            finally
            {                        
                if (copy.CopyStream != null)
                {                            
                    copy.CopyStream.Close();
                }
                copy.End();
            }
        }
    }
}
private static string SerializeData(object[] data)
{
    var sb = new StringBuilder();
    foreach(var d in data)
    {
        if (d == null)
        {
            sb.Append("\\N");
        }
        else if (d is DateTime)
        {
            sb.Append(((DateTime) d).ToString("yyyy-MM-dd HH:mm:ss"));
        }
        else if (d is Enum)
        {
            sb.Append(((Enum) d).ToString("d"));
        }
        else
        {
            sb.Append(d.ToString());
        }
        sb.Append("|");
    }
    return sb.Remove(sb.Length - 1, 1).ToString();
}

You use it like so:

var users = GetAllUsers();
Func<User, object[]> callback = u => new object[] { p.Id, p.Email, p.UserName, p.Password };
BulkCopy(users, callback, "Users", null);

If you want to specify the columns, say because the Id is an auto incrementing field, simply do:

var users = GetAllUsers();
Func<User, object[]> callback = u => new object[] {  p.Email, p.UserName, p.Password };
BulkCopy(users, callback, "Users", "(Email, UserName, Password)");

The code essentially turns each entity in your collection into a serialized string, which gets written to the stream.

The other problem I had was that one of my most complex queries was really slow. The query had joins, joining a subselect in the where clause. SQL Server didn't have a problem with it, PostgreSQL did. I wasn't too surprised because past experience with MySQL had taught me that such queries could be problematic. I saw a few statements hinting that PostgreSQL 8.4 (I'm running 8.3) might fix it. I spent some time trying to speed it up, had some success, and then simply decided to go about it a different way. While I think its very likely that I wasn't doing it properly, there's no denying that SQL Server handled this better.

I did time the execution of my integration test, using the same code. This was run on identically pathetic machines - an Atom 330 with 4gigs of ram and a 5400RMP hard drive. Postgresql 8.3 on Ubuntu server 9.04 and SQL Server 2008 on Windows Server 2008 R2. Without using any bulk copy, Postgresql was slightly faster. With their respective bulk copy, SQL Server was slightly faster. I'm not giving numbers because the results were really close, seconds apart when everything took minutes. Also, I do think the test was useful; however, the code was single threaded which means it didn't provide any insight on how the databases behave when under a more typical kind of load. My goal was really just to make sure I wasn't making a huge mistake, and it doesn't look, or feel, like I am.

Machine.Migrations changes

aaronjensen — Sat, 24 Oct 2009 21:56:00 GMT

Jacob first introduced Machine.Migrations over a year ago. Since then, it's been a solid part of our process and we're up to nearly 500 migrations with it on one project.

Recently, I finally got around to making some changes I've been wanting to make and I wanted to call'em out:

We split the repostiory out to its own on github. I did this similar to the way I did it for MSpec some time ago. Again, the advantage here is that change logs are localized to the project rather than the entire machine overarching project.
I added complaints if you have multiple migrations with the same number. This silently caused problems before as one of the migrations would not get applied. Now it just yells and dies like a good app. Of course, this is not as important because of the next change.

Next I added support for timestamped migrations. This helps immensely on active projects with multiple developers. You don't have to deal with communicating migration numbers to the team when you add them, you just generate a new one and merge it when you feel like it. We actually had half of our topic on Campfire dedicated to our current migration number. The other half was who owed how many pushups for breaking the build.

Here's the rake task we use to generate a new migration:

namespace :new do
  task :migration do
    raise "usage: rake new:migration name=\"Your migration name\"" unless ENV.include?('name')

    name = ENV['name']

    filepath = "db/migrate/#{Time.now.strftime('%Y%m%d%H%M%S')}_#{name.gsub(/ /,'_')}.cs"

    text = File.read("db/migrate/template.cs")
    File.open(filepath, 'w') { |file|
      file.puts text.gsub(/\$MigrationName\$/,"#{name.gsub(/ /,'_')}")
    }
  end
end

Usage is simple, just create a template.cs in the directory (we use db/migrate) and then type

rake new:migration name="this is my migration name"

Example template:

using System;
using System.Collections.Generic;
using System.Text;
using Machine.Migrations;

public class $MigrationName$ : SimpleMigration
{
  public override void Up()
  {
    throw new System.NotImplementedException();
  }

  public override void Down()
  {
    throw new System.NotImplementedException();
  }
}

Finally, I added a command line runner. As we've grown more and more sick of msbuild and more and more fond of rake, we've wanted to slowly all but eliminate our dependency on MSBuild. The command line tool is quite simple:

Machine.Migrations
Copyright (C) 2007, 2008, 2009 Machine Project

  c, connection-string  Required. The connection string to the database to migrate
  t, to                 Applies or unapplies migrations to get to the specified migration
  u, up                 Applies migrations to the latest
  s, scope              The scope?
  d, directory          Required. Directory containing the migrations
  v, compiler-version   Version of the compiler to use
  debug                 Show Diagnostics
  r, references         Assemblies to reference while building migrations separated by commas
  t, timeout            Default command timeout for migrations
  ?, help               Display this help screen

The only two flags you usually need are -c and -d. Just pass in the connection string and the location of your migrations. Be sure to quote them: migrate.exe -c "my connection string" -d "my migration directory".

Hopefully these changes help others. Grab the latest here and just msbuild the sln to build it.

I do wonder though, what other active .NET migration projects are people using and how do you like them?

Map control confusion

pvanooijen — Thu, 22 Oct 2009 15:30:00 GMT

When it comes to including geographical information in your web-application you have to make two main decisions. The first one is which API you are going to use, the second one is who is going to provide the geographical data. These two are not always completely independent.

A lot of websites use google maps. The geographical information is supplied by google and the only way to get to that information is by using the Google Javascript API. When it comes to performance Javascript is far from ideal, more sophisticated applications which try to add a lot of information to a map are quite likely to hit a script timeout. The other downside of this approach is the (lack of) smoothness of the display. Panning or zooming a map results in a lot of image blur.

Both downsides can be overcome by using Silverlight. The DeepEarth project was the first to use deepzoom which results in a smooth display when scanning or zooming. The strong typed compiled C# Silverlight code working with the map has not only a speed advantage over Javascript, it’s running async resulting in an app which not only has a smooth look but also a smooth feel.

You can build amazing apps with that. This is a part of the MVP map created by DeepEarth evangelist SoulSolutions

When it comes to where and how to get the geographical data, which can be satellite images, road maps or routing information, confusion starts. The DeepEarth project offered a choice of map providers. Ranging from OpenStreetmap to Yahoo maps and Microsoft Live Earth. Google maps were not included, the reason is the one I mentioned. Nobody so far has managed to create a hack to get Google Map data without leaving JS.

But the most confusing data-provider was and is Microsoft. The first versions of DeepEarth offered plain Live maps. In later versions you were supposed to use LiveMaps webservices. The usage of these services was by far free. There was a free developer test account available which resulted in map images all decorated with the word “staging”. For production you had to buy a license. To get a proper deal for that always got stuck somewhere in the pipeline. At the same time the old versions using plain Live maps kept working. At the moment I’m writing this seems to be history already.

LiveMaps have become Bing maps. At the same time Microsoft started developing their own Silverlight map control. The good thing is that it is a marvel to work with. The performance is even better than DeepEarth. The API just shines, a lot of functionality is present straight out of the box. All the extra work I had to do to get my app running in Deep Earth could be replaced by just hooking into some events. The architecture is great as well. It is by no means just limited to using Bing Maps data, you can even bypass that completely using the control as a blank working surface with a Latitude-Longitude coordinate system. A part of the architecture is a clear and easy way to use any map provider you can imagine, or build yourself. There are a lot of good stories on the web on working with the control.

But, as announced, there is still confusion. First there is the name. It is Microsoft Virtual Earth Silverlight Map Control. I have no problem with that, do you like Bing ? I can’t get used to that. More serious is the source of map data. The control, now in CTP, displays full MS map data without any need to apply for a (test-) license. No staging, no problem. The control itself has a Token property, according to the documentation:

Gets or sets the token to use for service requests.

It contains an empty string. How and when to use it is not clear at all. When it comes to production will it be needed to use the control? Or will it only be needed to get actual MS map data? Which data ? Just routing information ? Maps? Satellite images ? I just cannot find out. In case anybody can shed some light on this, I would appreciate that very much. Building software using the control is great, but what are price and conditions my customer will have to deal with?

Migrating to Postgresql with my friend NHibernate

karl — Thu, 22 Oct 2009 13:23:00 GMT

Last night I decided to accelerate our plan to migrate from SQL Server to PostgreSQL and gave it a quick attempt to see what the work involved was. I was hoping that due to our usage of NHibernate, and by having meticulously written database agonistic code the migration would be cake.

Turns out I was right and although there were a couple bugs along the way, the process took less than 30 minutes. Today I want to talk about the small issues we faced in migrating our code, as well as provide an overview of what I've learned about PostgreSQL so far.

First though, why migrate? I want to avoid getting rantish [in this post], but the cost of ownership for SQL Server isn't remotely close to being competitive. SQL Express is great, provided you can live with (or are willing to) its numerous constraints. Bizspark (I only bring it up because I feel someone will mention it in a comment) is a subprime mortgage - it lulls you into a sense of free only to nail you with licensing cost down the road. I'm sure someone can quote some obscure feature that might be needed by some system, but for the rest of us, the only reason to use SQL Server is familiarity. Familiarity is a pragmatic and reasonable reason to pick a technology, but only for so long.

Secondly, why PostgresQL? Well, I've used sqlite and MySQL extensively in the past (including submitting patches to their .NET drivers). However, MySQL is currently in a weird state (the sun/oracle thing) and I need ACID compliance and my experience with InnoDB has been average.

Thirdly, its worth pointing out that the system being migrated has a comprehensive set of unit tests, including a number tests which hit the DB. While I generally consider unit tests a design-tool, there's no doubt that they provided a significant sanity check.

So, what problems did I run into?

Given the use of NHibernate, the application side of things was migrated quiet easily. However, I did have to modify a few things. This is possibly because of flaws in my code (vs limitations of NHibernate) - hopefully someone will point such flaws out.

Unique Constraints

I use unique indexes to ensure that things like user emails are unique throughout the system. However, I haven't yet figured out a way to catch such exceptions in a db-agnostic manner. My code looked something like:

try
{
    Session.SaveOrUpdate(user);
    transaction.Commit();
    return true;
}
catch (GenericADOException ex)
{
    transaction.Rollback();
    var sql = ex.InnerException as SqlException;
    if (sql != null && sql.Number == 2601)
    {
        return false;
    }
    throw;
}
catch (Exception)
{
    transaction.Rollback();
    throw;
}

Obviously this code isn't going to work with the PostgreSQL driver. I keep thinking there *is* a way to handle this across multiple databases, but haven't found the answer yet.

Count - Long vs Int

When I fetch the number of rows for a query - more often than not to do paging - i do a select count(*) with a UniqueResult<int>(). However, PostgreSQL's count(*) returns a long, while SQL Server returns an int (I've run into this before, since sqlite also returns a long). This causes a cast exception. Again, I'm sure there's a simple solution to this inconsistency.

Bit and Booleans

There was one specific query which tried to retrieve all active users. The IsActive field was declared as a bit in SQL Server (and a boolean in PostgreSQL). The following query didn't work in PostgresQL "from User where IsActive = 1". Why? because 1 isn't a valid boolean, and you'll get a database error that an integer can't be converted to a boolean. I had to change the query to use '1' (in single quotes) or one of the other valid boolean values for true.

Connection String

Its also worth mentioning that the connection string example listed here, didn't work for me using the latest Npgsql drivers. Instead of Initial Catalog, I had to use the Database key.

Aside from those little issues, the code was up in running in about 10 minutes. Most of my time was actually spent getting to know PostgreSQL. Here's what I've learned so far.

Learning PostgreSQL

PostgreSQL doesn't consider two null values to be the same, meaning you can put a unique constraint on a nullable column without any problems. In SQL Server you need a Nullbuster column to achieve this same result (its a neat trick, but its neater not having to use it).
PostgreSQL's uniqueidentifier type is called UUID, a bit is a boolean, and a datetime is a timestamp.
All of PostgreSQL's text types are unicode by default (so there is no nvarchar you simply use varchar and get the same benefits)
PostgreSQL's handling of auto incrementing (identity) is pretty different. Instead of declaring a column as int not null identity(1, 1) primary key, you define it as serial not null primary key. "serial" is just some syntactical sugar around what actually happens - PostgreSQL generates a sequence table for that column, and sets the columns default value to the sequence's next value. Of course, identity(1, 1) is just some syntactical sugar around however SQL Server implements the feature - its just more transparent in PostgreSQL.

Hopefully as the weeks go by, I'll have more information to share about the migration, and what I've learned.

Being Business Focused: Risk

drusellers — Thu, 22 Oct 2009 11:08:26 GMT

'Being Business Focused' is going to explore various business concepts and how we can use them to explain the bottom line impact of our decisions.

Don't forget we learned roi = (fv-iv)/iv

Now I want to talk about another fundamental concept at the bank and how it can affect overall ROI. Today's word is 'Risk'. In the world of finance risk has a very real and measurable impact. You can see that impact very easily in the interest rates charged for loans. The riskier the loan the higher the interest rate. There is a financial model that we can use to discuss the different elements of risk and its impact on the cost of financing a project. The specific one I want to talk about is the 'Capital Asset Pricing Model' (CAPM). I like this model because it is simple and demonstrates a few things that I think are interesting about risk.

Ok, so what is CAPM? CAPM states that the expected return (Ri) is equal to the risk free rate (Rf) plus the beta coefficient (Beta) multiplied by the risk premium, where the risk premium is the market risk rate (Rm) minus the risk free rate.

Ri = Rf + Beta*(Rm - Rf)

So what does all that mean? Risk free price/rate is the price for the THING if there is no risk involved. The market price/rate is the price for the THING on the street, this price is usually higher than the risk free rate because it includes a 'premium' to account for the risk involved, ie "You mean I might lose my money, then I want a bigger payout!" The beta coefficient I can only explain as the volatility in the market, if the market is making wild swings then any risk you take will be magnified by it and therefore affects the expected return.

This next section is not entirely accurate and is used only to show how playing with the risk levers can change the ROI. NOTE: You probably won't use the model directly, but understanding the different components of this model has been helpful for me when talking to the business.

When we start to talk about buying and selling money, my brain starts to hurt. I am going to buy money...with money? owwwww... What I have noticed is that the smart guys with the finance knowledge talk about the price of money as the rate that you borrow it at, the money borrowed is called capital, and the CAPM helps us analyze the rate. To show its impact on ROI I am going to substitute it for the initial value since it kinda represents the price we borrow money at.

WARNING: Dru is getting jiggy with the math

roi = (fv - capm)/capm

Now that we have a game plan, lets plug some numbers into CAPM and see what happens. In the financial world government bonds tend to be the benchmark of risk free money, so lets say that gov bonds are going for 0.05, and the same bond offered by your company is going for 0.07 (these are totally made up numbers), and for the beta lets assume a 1 (so that we can ignore it for now, hit me up on email for more info).

0.05 + 1 * (0.07-0.05)
0.05 + 1 * 0.02
0.05 + 0.02
0.7

We can see now that we have a risk premium of 0.02, and that if we can find a way to reduce that risk premium it will have a direct affect on our ROI calculation above. Like the financial world anything we do to try and reduce risk is going to cost us something, so its nice to know that your cost (increase initial value) of the mitigation is less than its return (future value). Let's say that something happens and we can now offer that same bond at 0.06, with negligible impact on initial value.

0.05 + 1 * (0.06-0.05)
0.05 + 1 * 0.01
0.05 + 0.01
0.06

So we see the price lowered. Not a big surprise since we were able to magically lower it, but we did it by targeting the risk premium and we can now see its impacts on ROI. By lowering CAPM in our ROI calculation we have increased ROI (lowering the bottom number and increasing the top number).

Before

roi = (x - 0.07)/0.07

After

roi = (x - 0.06)/0.06

Ok, so what are some examples of developers lowering risk? As developers we do many things to reduce risk in our projects. I know one thing that I like to do is put the risky parts of my code base behind an interface. That way I contain the risk of that section to just that section. In financial terms I am limiting my exposure to the risk. Maybe the risk is that the business hasn't decided something, like which payment gateway to use. When building a system that kind of unknown brings an implementation risk to the project, but we can effectively nullify a large chunk of that by simply coding against an interface that we will implement once the business has decided.

So what is the take away? Lowering risk can lower your effective initial investment, which will then increase your return on investment.

I hope that you find this helpful

-d

Building an iPhone app

aaronjensen — Wed, 21 Oct 2009 20:12:00 GMT

Today we launched PicTranslator, a free Picture Translator for the iPhone. Building it was quite an adventure. I wanted to share some of the technologies we used, you may be surprised.

On the iPhone:

iPhone SDK: This one's obvious. Unfortunately, MonoTouch wasn't out when we started to develop. It looks rather promising.
To be perfectly honest though, Objective-C is pretty cool. Especially once you figure out it's memory management scheme. It's kind of fun to be back in pointer land again. Brings back fond memories. Oh, and though I did miss vim, xcode is a pretty darn good little editor.
json-framework: This is a pretty slick JSON parser for the iPhone. Hand parsing JSON in obj-C would not have been fun. This made it easy. I'm pretty sure I followed this tutorial to get it up and running.
ASIHttpRequest: Nice Http framework, helpful for doing asynchronous requests and sending and receing files easily.
We also used a few random snippets found around the web. Here's one on rounded corners. Also stackoverflow was invaluable.

On the server:

nServiceBus: NSB is fantastic. It allows us to scale to any size machine or any number of machines. I'm realizing more and more than most reasonable size applications should leverage messaging in one way or another. We're using MSMQ as the transport.
ASP.NET MVC: My web framework of choice.
MSpec: Gotta test it somehow right, why not use my own framework.
Topshelf: Best way I know of to create services in .NET. Makes it easy to run as a console app or a service. Love it.
log4net: Another no brainer. log4net is how you log in .NET. Well, most of you any ways :).

Not a bad stack, eh? Any surprises? Oh, and if you have an iPhone, be sure to give PicTranslator a shot, we'd love your feedback.

Interesting findings in the diff between .NET Fx v4 Beta1 and Beta2

Patrick Smacchia — Wed, 21 Oct 2009 14:31:00 GMT

Normal 0 21 false false false FR X-NONE X-NONE

I compared with NDepend .NET Fx v4.0 Beta1 with the new.NET Fx v4.0 Beta2 just released. I enumerated below some random interesting finds. Don’t hesitate to ping me back if I’ve missed some cool public API things.

To do the comparison yourself, you need to have kept a copy of beta1 assemblies as I explained in this post: Analyze and explore the evolution of .NET Framework by yourself

A lot of work has been done since the Beta1, there are 92K methods modified or added!

I take a chance to show a preview screenshot of a great upcoming NDepend feature concerning the comparison mode: the possibility to disassemble (with Reflector) and compare implementations of a modified method. With this feature one can fine-grain control evolution of a codebase from assemblies only.

Some new core types whose names are pretty explicit:

Some new core public methods that seem useful to me.

Some new WPF types. None of them contains IL instructions, meaning that they are DataObjects (declared with auto properties), enumerations, delegates or interfaces:

A new System.Numerics assembly has been created. This assembly now contains the class BigInteger (formerly defined in System). This assembly also contains 3 new interesting classes: Complex, DoubleULong and NumericsHelpers.

The assembly System.Caching seems to have been renamed System.Runtime.Caching.

A new assembly System.Device that seems to only expose so far classes to modelize a Geo Localization (GPS) device API in the namespace System.Device.Location. It is pretty surprising actually, it looks like the .NET Fx is now modelizing some domains! It is like if future versions of .NET Fx will come with a System.Financial assembly with Account, Bid and Stock classes?!

System.Activities.* assemblies have been merged to give birth to a new System.Activities.Presentation assembly that seems to contain all previous assemblies code, now organized through namespaces.

Advanced Selenium RC logging with MSpec

aaronjensen — Tue, 20 Oct 2009 01:30:00 GMT

Today I pushed some new features to MSpec to allow you to see this in MSpec’s html report when you get Selenium RC failures:

This is very similar to ruby’s selenium client’s rspec report. You can see a real example of that here.

So how do you get this? It’s pretty easy. All you need to do is grab and build the latest MSpec, add a reference to Machine.Specifications.SeleniumSupport.dll and add a class like the following to your assembly that contains your selenium specs in it:

  public class SeleniumSupport : SeleniumResultSupplementer
  {
    protected override string ImagesPath
    {
      get { return GetTempPath(); }
    }

    protected override DefaultSelenium Selenium
    {
      get { return ...; } // Replace with the DefaultSelenium in use 
                          // by the current running spec
    }

    static string tempPath;

    private static string GetTempPath()
    {
      if (string.IsNullOrEmpty(tempPath))
      {
        tempPath = Path.Combine(Path.GetTempPath(), "selenium");

        if (Directory.Exists(tempPath))
        {
          Directory.Delete(tempPath, true);
        }

        Directory.CreateDirectory(tempPath);
      }

      return tempPath;
    }
  }

That's pretty much it. Like I said, this is brand new, just got it up today so please let me know if you have any problems with it. You can do so on twitter or the machine google group. Enjoy!

F# October 2009 CTP/Beta2 – F# + Rx Together At Last

Matthew.Podwysocki — Mon, 19 Oct 2009 22:37:24 GMT

Lately, I’ve been covering a lot of F# First Class Events as well as the Reactive Framework which has been leading up to a head as it were. It has been announced through Channel 9 that .NET 4 will include the two major interfaces from the Reactive Framework, the IObservable<T> and IObserver<T>. Today, as Don Syme announces with the release of the F# October 2009 CTP and F# for Visual Studio 2010 Beta 2, those two interfaces, the IObservable<T> and IObserver<T> have been integrated into F# First Class Events.

What does that mean and what are the implications?

Mixing in Observables

In the previous version of F#, you may remember that we dealt primary with the IEvent interface and if you needed the ability to subscribe and unsubscribe, you would use the IDelegateEvent interface. Let’s look at how this was implemented in the past.

type IDelegateEvent<'Delegate when 'Delegate :> System.Delegate > =
  abstract AddHandler: handler:'Delegate -> unit
  abstract RemoveHandler: handler:'Delegate -> unit 

type IEvent<'Delegate,'Args when 'Delegate : delegate<'Args,unit> and 
                                 'Delegate :> System.Delegate > =
  abstract Add: callback:('Args -> unit) -> unit
  inherit IDelegateEvent<'Delegate>

As you’ll note, we had the ability to add and remove delegate handlers using the AddHandler and RemoveHandler methods with our IDelegateEvent interface and our ability to add a handler via the Add method in the IEvent interface. Both of these gave us some pretty rich options when dealing with first class events. When the Reactive Framework interfaces were to be folded in to .NET 4.0, it was only a matter of time before the F# team looked on how to integrate the idea of Observables and F# First Class Events. WIthin the Visual Studio 2010 Beta 2 download, you’ll find that F# does in fact reference the System.IObservable<T> and System.IObserver<T> interfaces. If you are still using Visual Studio 2008, these interfaces have also been included as well so that you can take advantage of them. Now if we are to look at how F# First Class Events work now, let’s go over the inheritance chain.

type IObserver<'T> =
  abstract OnNext : value : 'T -> unit
  abstract OnError : error : exn -> unit
  abstract OnCompleted : unit -> unit

type IObservable<'T> =
  abstract Subscribe : observer : IObserver<'T> -> System.IDisposable
  
type IDelegateEvent<'Delegate when 'Delegate :> System.Delegate > =
  abstract AddHandler: handler:'Delegate -> unit
  abstract RemoveHandler: handler:'Delegate -> unit 

type IEvent<'Delegate,'Args when 'Delegate : delegate<'Args,unit> and 
                                 'Delegate :> System.Delegate > =
  inherit IDelegateEvent<'Delegate>
  inherit IObservable<'Args>

What you’ll now notice is that we have in code, defined our IObserver<T> and IObservable<T> which either come from the .NET 4.0 base class library or in F# itself for Visual Studio 2008, and then we have our IEvent<TDel,T> inheriting both the IDelegateEvent<TDel> and the IObservable<T> interfaces. This means that when we’re dealing with First Class Events in F#, we can go back and forth from using the IEvent<TDel,T> and IObservable<T> interfaces, based upon our needs. So, what does that buy us?

Exploring Observables

Much like the Event module in F# defines combinators for F# First Class Events, the Observable module defines the standard combinators for using the IObservable<T> interface. Much like the Event module, the Observable module includes add, filter, map, partition, merge, choose, and scan. Let’s go over these briefly about what each one is about.

add
Create an observer which permanently subscribes to the given observable and which calls the given function for each observation.
subscribe
Create an observer which subscribes to the given observable and which calls the given function for each observation.
map
Return an observable which transforms the observations of the source by the given function. The transformation function is executed once for each subscribed observer. The returned object also propagates error observations arising from the source and completes when the source completes.
filter
Return an observable which filters the observations of the source by the given function. The observable will see only those observations for which the predicate returns true. The predicate is executed once for each subscribed observer. The returned object also propagates error observations arising from the source and completes when the source completes.
scan
Return an observables which, for each observer, allocates an item of state and applies the given accumulating function to successive values arising from the input. The returned object will trigger observations for each computed state value, excluding the initial value. The returned object propagates all errors arising from the source and completes when the source completes.
choose
Return an observable which chooses a projection of observations from the source using the given function. The returned object will trigger observations x for which the splitter returns Some x. The returned object also propagates all errors arising from the source and completes when the source completes.
partition
Return two observables which partition the observations of the source by the given function. The first will trigger observations for those values for which the predicate returns true. The second will trigger observations for those values where the predicate returns false. The predicate is executed once for each subscribed observer. Both also propagate all error observations arising from the source and each completes when the source completes.
split
Return two observables which split the observations of the source by the given function. The first will trigger observations x for which the splitter returns Choice1Of2 x. The second will trigger observations y for which the splitter returns Choice2Of2 y. The splitter is executed once for each subscribed observer. Both also propagate error observations arising from the source and each completes when the source completes.
merge

Return an observable for the merged observations from the sources. The returned object propagates success and error values arising from either source and completes when both the sources have completed.

You’ll notice one difference from the Event module to the Observable module is the inclusion of the subscribe function. This allows us to subscribe to an observable, which in turn will return an IDisposable. This IDisposable, as I’ve covered before, allows us to nicely tell the system when we’re no longer interested in receiving notifications instead of having to tell the observer when we want to attach and detach. We’ll cover more of this in a future post on the Introduction to the Reactive Framework series.

What does this change buy us exactly? Well, now we can treat observables much the same as we would for F# first class events. Now we can take an example from a previous post and show how we can create two observables based upon whether the X and Y coordinate of the mouse is in a certain location as we drag it across our form.

// Create form
let form = new Form(Visible=true, TopMost=true, Text="Event Sample")

// Create under and over for X and Y coordinates
let (overEvent, underEvent) =
  form.MouseDown
  |> Observable.merge form.MouseMove
  |> Observable.filter (fun args -> args.Button = MouseButtons.Left)
  |> Observable.map (fun args -> (args.X, args.Y))
  |> Observable.partition (fun (x, y) -> x > 100 && y > 100)

// Subscribe to each
let overSubscription =
  overEvent 
  |> Observable.subscribe (fun (x, y) -> printfn "Over (%d, %d)" x y)
let underSubscription =
  underEvent 
  |> Observable.subscribe (fun (x, y) -> printfn "Under (%d, %d)" x y)
  
// Much later, clean up
overSubscription.Dispose()
underSubscription.Dispose()

As you can see from this code, we’re treating our Observables no differently than we would our F# First Class Events. With the inheritance chain of the IEvent from the IObservable, we get seamless integration between the two. Unlike C#, we don’t need to create extension methods to turn our events into observables and instead get that for free.

Conclusion

What you will notice is that we lack many of the combinators that the Reactive Framework has built in such as Until/WaitUntil, Take/TakeWhile, Skip/SkipWhile among others for which we’ll cover in the Introduction to the Reactive Framework series. Many of these aren’t hard to implement and we’ll get the added benefit of exception management that the Observables give us.

It’s pretty interesting to see how F# has evolved from release to release. This release for Visual Studio 2010 Beta 2 and the F# October 2009 CTP have some nice changes as noted in the release notes. The integration of the IObservable/IObserver is not to be overlooked as one of them. Download them today and give the team feedback! Now back to our Observables series.

Time for a New Computer

karl — Sun, 18 Oct 2009 21:25:00 GMT

A couple weeks ago I bought a new computer. While my existing computer still had a lot of life in it, I was eager to laverage the power of the solid state drives (SSD) - specifically Intel's 2nd generation X25. I figured that since I'd be updating the hard drive and installing a new OS (Win7), then there'd really be no better time to upgrade everything else.

You can see a video of the computer opening and building the NHibernate solution: http://www.youtube.com/watch?v=kPdRwS3vEbE

A lot has been said about how fast SSDs are, and its all true. People who do straight up $/gb comparisons with normal mechanical hard drives are missing the point. Of course, if you can't afford it, you can't afford it, but if you are in the market for a new computer, I'd strongly suggest that you see if you can't get it to fit into your budget.

The core parts of my computer are:

Intel Core i7 860
G.SKILL DDR3-1600 4x2GB
Noctua NH-U12P SE2
Gigabyte P55-UD4P ATX
Intel X25-M 160GB G2
Western Digital Caviar Black 640GB
Corsair TX650W 650W ATX
2x Scythe Ultra Kaze 120MM 1000RPM
Antec P183 ATX

Intel Core i7 860
The new Lynnfield processors from Intel fit the LGA1156 socket. You'll likely be interested in the i7 860 or the i5 750. Both are quad cores, but the i7 supports hyperthreading. Benchmarks have shown that hyperthreading will improve performance from ~0-30%...with 10% often being cited. Given that the i7 860 costs more than 10%, the i5 750 is definitely a better value - and getting it instead of the i7 860 is a great way to fit an SSD into your build.

On a side note, you might also be interested in the i7 920, which is based on the LGA1366 platform. The difference between the two is that the LGA1366 supports tripple channel memory and has more bandwidth to the pci-e slots. Only the most memory intensive applications have shown to benefit at all from triple channel memory. However, if you're a heavy gamer and plan on doing SLI/Crossfire (2+ video cards), the LGA1366 and i7920 is probably for you.

Both types of chip have something called Turbo Mode - that is they detect when cores are being underutilized and speed up the working cores. The overall effect is that the chips all stay within the same thermal envelope while allowing single threaded applications run faster.

G.SKILL DDR3-1600 4x2GB
RAM is really cheap, so there's no reason not to do 4-8gb nowadays. My old computer had 8gb also, and although I only managed to pass the 5gb barrier a couple times (that I noticed), I still think its money well spent. Do note that today's systems are running on DDR3 RAM, and, as a consequence, DDR2 RAM is starting to go up in price. Do be mindful of the RAM's voltage. Many DDR3 modules are currently rated above the spec'd 1.5V.

Noctua NH-U12P SE2
I bought an after-market cooler for my CPU because I like silent computers, and I like to overclock a little. Since I bought the CPU the day of their release, I had limited choice, and while I'm happy with the performance, there's likely cheaper and/or better solutions currently available.

The i7 860 runs at 2.8ghz, with a maximum turbo mode of 3.46ghz. The maximum junction temperature (Tj. Max) for the chip is 100C - which is when things start to shut off, definitely something we want to stay away from. With this cooler, and my RAM which is rated at 1600mhz, I easily got the CPU running at 3.5ghz. My idle temperature is 40C, and my load temperature is 60C. My definition of "easy" is that I clicked a couple settings in the bios, didn't have to adjust the voltage to any components, and the system is rock solid. There's a lot more that could be done, but overall, this is good enough for me.

Gigabyte P55-UD4P ATX
Motherboards are very similar these days - especially since more and more features are being directly integrated into the CPU. The motherboard manufactures are trying all sorts of trick to differentiate themselves, but unless you are a power user just get a brand that you trust with the features that you like, compare the prices to make sure its reasonable, and buy it.

There are a couple things that you do need to be careful of. First of all, uATX is becoming more and more popular, so don't accidentally buy one of those (unless that's what you want). Also, motherboard manufactures pay a licensing fee to Nvidia to include SLI support on their motherboards. If you plan on doing SLI, make sure your motherboard supports it. If you don't, you can save $10 by buying a "lower" version (they don't pay a licensing to AMD to Crossfire support, so most motherboards will support that).

Intel X25-M 160GB G2
Whether you pick up a 160GB or an 80GB it really doesn't matter - both are plenty to store Windows, VS.NET, Office, a couple other core apps and a couple games. What is important is that you get this drive. It is as far beyond other SSDs as SSDs are beyond mechanical drives. It destroys everything else when looking at random reads and writes. Other SSDs are faster when doing sequential reads and writes, but that is meaningless to 95% of how your computer operates day to day.

The G2 will also (at some point in the near future) support the powerful TRIM command, which Windows 7 supports, via firmware upgrade. This is a command that Windows passes onto the SSD controller to tell it that a data block is no longer in use. Without it, SSDs slow down over time (Intel drives less than others).

Also of interest, Windows 7 does a handful of optimization tricks when it detects an SSD. It disables defragmentation and other features like ReadyBoost. I've also disabled Windows Indexing on the drive.

Western Digital Caviar Black 640GB
Although I'm running a NAS for backups, and file storage (mp3s, movies, pictures), I do feel a little claustrophobic with 160gb. Therefore I added another drive to my system to store files and applications which aren't performance sensitive. There's a lot of choice here...I've had great luck with WD.

Corsair TX650W 650W ATX
A lot of people pick overly powerful power supplies, the truth is that the 650W I got has plenty of headroom, and could easily power a dual video-card setup. Make sure that the efficiency of the PSU is clearly listed. At the very least you want a 80 PLUS Bronze rating.

2x Scythe Ultra Kaze 120MM 1000RPM
Just a couple case fans to keep everything cool. These puppies run at a very low 1000RPM, which means you can't hear them.

Antec P183 ATX
There are a lot of cases to chose from, they vary in price and looks greatly. I'm largely interested in tool-less access (thumb screws), compartmentalization, and good cable management.

Video Card
I didn't get a new video card, but I will be buying an ATI 5850 once 3rd party manufactures implement their own cooling solution to provide quieter operations. If you don't play games at all, any video card will do - provided that it has the necessary outputs, and can do your monitor's native resolution in 2d (and I bet they _all_ can). If you only play a little, an ATI 4870 or 5750 is currently a good buy. If you need something powerful right now, look at the 5850 or 5870.

Message Granularity

drusellers — Sun, 18 Oct 2009 00:39:00 GMT

So, how big is a message? What all should you include in a message? What if you are in a bigger company, how do you roll messaging out across an enterprise? Honestly, I am still learning here too, but here are some rules of thumb I have picked up along the way:

You want more than one uber message for the whole company
You want a canonical message model (this means that messages aren't tied to any one application)
- This one I am still not 100% sure on, but I am pretty sure its correct
Each message should be a distinct unit or event
It should represent a business transaction as an atomic, self-contained and self describing unit of work (don't pass primary keys around)
Don't worry about the size of the message unless you have to. I don't even think about it anymore. The size (bytes) of the message should be considered as it can affect how fast your message can go across the wire. If this level of speed isn't a concern, then I wouldn't worry about it. ie if you can wait 0.10 of a second or more than I wouldn't sweat it.

Fine grained messages create more overhead from a management perspective so each one needs to earn its place.

Governance
- Build a way to track your messages. Even if its in excel.
- Once a message is deployed, be careful about upgrades. I try to follow Udi's advice that instead of modifying the message I make a new one.
- Make sure that only one system / department owns the message. This will make changes a bit clearer to see who can initiate them.

Links:
http://www.eaipatterns.com/CanonicalDataModel.html
http://fuzzypanic.blogspot.com/2005/04/canonical-message-format-and-xml.html
http://fuzzypanic.blogspot.com/2006/05/eda-lessons-learned-canonical-message.html

[ANN] DC ALT.NET 10/22/2009 – MongoDB with David James

Matthew.Podwysocki — Fri, 16 Oct 2009 03:38:45 GMT

This month DC ALT.NET will once again move into the not often covered subjects in the .NET world, in covering MongoDB with David James. Stay tuned to the mailing list for up to date details. There has been a bit of discussion around non relational databases lumped under the name of NoSQL, especially given the upcoming NoSQLEast conference. This meeting, we'll dive into MongoDB, one of the NoSQL solutions.

MongoDB is an open source document-oriented database written in the C++ programming language. Like other document-oriented databases, MongoDB is designed for problems without heavy transactional requirements that aren't easily solved by traditional relational databases, including problems which require the database to span many servers. K. Scott Allen has a great writeup on how to use MongoDB from C# that you can find here and is a great read. And yes, there is even LINQ support as well. Come join us and find out what NoSQL alternatives can do for you.

Also, we'll have a hacking session for those interested, so be sure to install MongoDB beforehand. The presentation will be mainly in Ruby and JavaScript, but there are C# options available as well.

About David James

David James is a Web Developer for Sunlight Labs. Prior to joining Sunlight, David founded CommunityGoals, a socially minded startup, and Collective Insight, a technology consultancy. He has worn the hats of entrepreneur, analyst, software developer, and consultant while working in fields including software, energy, and hardware engineering. David has a Master of Public Affairs degree from the LBJ School of Public Affairs, a liberal arts BA degree and an electrical and computer engineering BS degree from The University of Texas at Austin. He enjoys building tools that advance causes such as open government, community development, and environmental and energy policy. David also enjoys playing the piano and cycling and is sometimes known for coaxing friends into sprinting for city limit signs.

Date/Time: 10/22/2009 – 7-9PM

Location:
The Motley Fool
2000 Duke Street
Alexandria, VA 22314