CodeBetter.Com - Stuff you need to Code Better!

Introduction to the Reactive Framework Part IV

Matthew.Podwysocki — Sun, 15 Nov 2009 23:54:11 GMT

In the previous post in this series, we covered how to turn .NET events into first class values through IObservable instances. By doing so, we were able to do much more interesting things than just subscribe and unsubscribe, instead we were able to create a mouse drag event with little effort through composition. In this post, let’s look at going from push to pull by turning collections into IObservable<T> instances.

Let’s get caught up to where we are today:

From IEnumerables To Observables

As you may remember from the first post and second post in this series, we covered going from IEnumerable<T> to IObservable<T> and the duality between the two. But, how can we switch back and forth between them? Remember that we can create an IObservable from a couple of angles. Let’s first create one by using the chaining together Empty and StartsWith combinators on the Observable class to create an IObservable<T>. Note that this is not the publicly available release of the Reactive Framework, so bear with any name changes.

var observable = Observable.Empty<int>()
    .StartWith(1)
    .StartWith(2)
    .StartWith(3);

This allows me to start out with an empty IObservable<int> and then pre-append the value to the front of the instance, so in this case it would render 3, 2, 1. Next, let’s look at how we might generate a sequence IObservable from 1 to 10.

var observable = Observable.Generate(
    1,                        // Initial value
    value => value <= 10,    // Predicate
    value => value,            // Selector
    value => value + 1);    // Iterator
observable.Subscribe(Console.WriteLine);

In this instance, we supply an initial value, then a predicate to determine when to stop, a result selector, and then finally an iterator. There are plenty of overloads as well on the Generate to suit your needs when creating IObservable<T> values. But, what about taking existing IEnumerable<T> values and converting them to IObservable<T> values? One could use the the Create function which allows us to create arbitrary IObservable<T> instances.

var enumerable = Enumerable.Range(1, 10);
var observable = Observable.Create<int>(observer =>
{
    try
    {
        foreach (var item in enumerable)
            observer.OnNext(item);
        observer.OnCompleted();
    }
    catch (Exception exception)
    {
        observer.OnError(exception);
    }

    return () => { };
});
observable.Subscribe(
    value => Console.WriteLine(value),
    exn   => Console.WriteLine(exn.ToString()));

What we’re doing in the above code is using the Create method which gives us an IObserver<T> that we can post values to. In this case, we’re iterating through a collection and for each item, we’re calling OnNext. At the end of the iteration, we call the OnCompleted method which indicates that there are no more values. If there were some sort of exception when iterating the values, it would be caught and the exception posted to the observer. We can then decide how to handle it during our Subscribe method call, if at all. Finally, we return an Action which is to be called once we unsubscribe from the given IObserver<T>, which in this instance, we do nothing. That’s all fine and good, but there is an easier way of course, by using the ToObservable extension method.

var enumerable = Enumerable.Range(1, 10);
var observable = enumerable.ToObservable();
observable.Subscribe(Console.WriteLine);

Going from IEnumerable<T> to IObservable<T> as you an see is quite easy. Just as well, going back is easy as well through either of two methods, one being calling the ToEnumerable extension method, and the other being the GetEnumerator method.

// To Observable
var enumerable = Enumerable.Range(1, 10);
var observable = enumerable.ToObservable();

// And back again
var enumerableAgain = observable.ToEnumerable();
var enumerator = observable.GetEnumerator();

This ultimately leaves us in control of when we want to switch between the pull and push model, depending on our circumstances. It’s sort of like a buy one get one free in that respect. What makes this interesting is to take a collection and spread it out to multiple instances, for example reading in a file and then you could have multiple subscriptions each process the data in turn. Those possibilities are endless if we want to do certain actions concurrently.

var text = File.ReadAllLines("DocumentToProcess.txt");
var textObservable = text.AsObservable();

// Process concurrently
var textLetterSub = text.Subscribe(CountLetters);
var textWordSub = text.Subscribe(CountWords);
var textVowelSub = text.Subscribe(CountVowels);

There are also more things to come when dealing with collections that we’ll get into in later posts in this series, but this is enough for now.

Conclusion

As I’ve stated before, the LiveLabs Reactive Framework gives us the ability to harness reactive programming and treat events as the first class citizens they should have been using LINQ expressions and other standard LINQ combinators. Still, there is much to cover in this series with the standard LINQ combinators, the monadic heritage of this solution as well as turning Tasks and Asynchronous Methods into Observables.

Code Coverage: What Is It Good For?

Dave Laribee — Fri, 13 Nov 2009 20:06:00 GMT

Absolutely nothing. Or so the story goes...

I had a mini-exchange with my pal Glenn Block last night on Twitter. He was recycling the somewhat old meme that code coverage is a cockamamie metric that, as he puts it, "[code coverage is] a false security blanket on quality".

Measurements (other than net profit, of course) are merely indicators. They give us insight into what we're doing, red flags that help us know when to follow up on and drill into with more qualitative and hands on tools.This is, more-or-less, the idea behind Autonomation or "automation with a human touch."

For example: while code coverage isn't proof that our testing efforts will yield higher maintainability, it does tell us a teams commitment to a test practice.

I tend to believe the most interesting indicators are composite metrics. What happens when we start thinking about combining code coverage with cyclomatic complexity? I'd like to read a (hopefully) inverse relationship over time, coverage rising while complexity falls, as a good indicator that we're doing simple design.

The important thing to remember about metrics is they are only single purpose tools which, correctly applied, gift us with little tidbits of insight. It's up to you to find meaningful and creative uses for the various metrics and their various combinations. Once you start doing that, measurements like code coverage can certainly help you achieve your immediate goals.

Introduction to the Reactive Framework Part III

Matthew.Podwysocki — Fri, 13 Nov 2009 04:06:55 GMT

In my previous post in the Introduction to the Reactive Framework series, I discussed the duality between the IEnumerable/IEnumerator and IObservable/IObserver and how that enables us to generalize the ideas from LINQ to Objects to LINQ to Observables/Events. After the last post, I was asked how we go from events to Observables, so let’s tackle that in this post.

Before we get started, let’s get caught up to where we are today:

From Events...

Let’s start out by looking at the way we’d usually do event-based programming in .NET. First, if we have the need to pass data as arguments to our event, we’ll need to create a class which inherits from EventArgs, and in this case we have a class called SomeEventArgs. This class holds nothing more than just a simple object value.

public class SomeEventArgs : EventArgs {
    public object Value { get; private set; }

    public SomeEventArgs(object value) {
        Value = value
    }
}

Then we’d have a class which exposes an event with an EventHandler of our SomeEventArgs. At some point along the way we’d have a method which invokes our event, keeping in mind that we need to check for nulls before invoking.

public class SomeClass
{
    public event EventHandler<SomeEventArgs> SomeEvent;
    public event EventHandler<SomeEventArgs> OtherEvent;
    
    public void InvokeEvent(object value) {
        var handler = SomeEvent;
        if(handler != null)
            handler(this, new SomeEventArgs(value));
    }
    // Other event invoking code here
}

Now to do something useful with it, you can subscribe and unsubscribe by using the += and –= operators. In this case, I actually declared a method called SomeHandler which will react to the event arguments that I give it.

var someClass = new SomeClass();

// Add handler
someClass.SomeEvent += SomeHandler;

// Remove handler
someClass.SomeEvent -= SomeHandler;

Our SomeHandler method must follow the signature with the sender as an object and the EventArgs derived class, whatever it might be. At this point we react to anything that we’re subscribed to in this method.

private void SomeHandler(object sender, SomeEventArgs e) {
    // Some handler code
}

The problem with this solution is that it isn’t quite compositional. What do I mean by that? What if I want to combine this SomeEvent firing with another event? Or if I want to ensure that an event gets posted on a certain synchronization context? Or if I want to throttle the firing of an event to a certain threshold? All of those are a tad more difficult using standard out of the box .NET events, but, we can with Observables. So, how do we get there?

To Observables

Unlike F#, we don’t get Observables for free when we use events, so we have a little work ahead of us. In order for us to go from events to Observables , we simply need to use the Observable.FromEvent method. There are several flavors that we can use. The first option is to use a little “magic string” juice to wrap our event in an IObservable. This method takes the object to bind for the event and the event name as a string.

public static class EventExtensions {
    public static IObservable<Event<SomeEventArgs>> GetSomeEvent(
        this SomeClass someClass) {
    
    return Observable.FromEvent<SomeEventArgs>(
        someClass, "SomeEvent");
}

This method then returns to us an IObservable<Event<SomeEventArgs>> which allows us to get access to the EventArgs and the Sender through their respective properties on the Event class. Personally, I’m not a fan of magic strings at all and if I can get away with not using them, all the better. Luckily, there are other options. The next option is to use Action<EventHandler<EventArgs>> arguments for both the subscribe and unsubscribe. Keep in mind that this only works for those events which are declared to use EventHandler<TEventArgs>.

public static class EventExtensions {
    public static IObservable<Event<SomeEventArgs>> GetSomeEvent(
        this SomeClass someClass) {
    
    return Observable.FromEvent<SomeEventArgs>(
        h => someClass.SomeEvent += h,
        h => someClass.SomeEvent -= h);
}

If your given event doesn’t have the signature of EventHandler<TEventArgs>, you can still take advantage of the Reactive Framework as well. In this instance, let’s extend the WPF IInputElement to wrap the MouseMove event as an IObservable<Event<MouseEventArgs>>.

public static IObservable<Event<MouseEventArgs>> GetMouseMove(
    this IInputElement inputElement)
{
    return Observable.FromEvent(
        (EventHandler<MouseEventArgs> h) => new MouseEventHandler(h),
        h => inputElement.MouseMove += h,
        h => inputElement.MouseMove -= h);
}

What this does for us behind the scenes is create an IObservable/IObserver to handle the events as they happen. Each invocation of the event causes the OnNext to be fired with the Event<TEventArgs> argument. In this case, the OnComplete and OnException are not implemented as part of the IObserver instance.

An Example

Let’s create a little example of drawing on a canvas in WPF. In order to do that, we’ll need not only the MouseMove extension as we have above, but also MouseLeftButtonDown and MouseLeftButtonUp. Each of those extension methods should follow the same pattern as above in terms of needing the extra parameter to convert the EventHandler<MouseButtonEventArgs> to a MouseButtonEventHandler.

First, we need to capture position of the mouse during our mouse moves. After that, we’ll need to zip together the mouse move and another mouse move that is one ahead in order to capture a delta. Finally, we’ll capture the mouse downs and the mouse deltas until the mouse ups happen. Listed below is code to make that happen.

var mouseMoves = from mm in mainCanvas.GetMouseMove()
                 let location = mm.EventArgs.GetPosition(mainCanvas)
                 select new { location.X, location.Y};
                 
var mouseDiffs = mouseMoves
    .Skip(1)
    .Zip(mouseMoves, (l, r) => new {X1 = l.X, Y1 = l.Y, X2 = r.X, Y2 = r.Y});

var mouseDrag = from _  in mainCanvas.GetMouseLeftButtonDown()
                from md in mouseDiffs.Until(
                    mainCanvas.GetMouseLeftButtonUp())
                select md;

The code should read rather straight forward as we attach to the mainCanvas object in order to draw. The interesting functions are the mouseDiffs and the mouseDrag. The mouseDiffs takes the mouseMoves and then skips 1 ahead and zips it with another mouseMoves, and projects to hold both left and right values. The mouseDrag function takes events from left mouse button down and then from mouseDiffs until the left mouse button up Observable happens.

Now that we have our mouseDrag, what can we do with it? Well, let’s go ahead and draw on our canvas:

var mouseSub = mouseDrag.Subscribe(item =>
{
    var line = new Line
                  {
                      Stroke = Brushes.LightSteelBlue,
                      X1 = item.X1,
                      X2 = item.X2,
                      Y1 = item.Y1,
                      Y2 = item.Y2,
                      StrokeThickness = 5
                  };
    mainCanvas.Children.Add(line);
});

Our mouseSub subscription creates a new line with our coordinates from zipped mouseDrag Observable and then adds it to our canvas. Consider how little code was required to make this happen, and try doing that with standard .NET events. Quite frankly, this kind of solution makes event-based programming using .NET events expressive enough to be usable.

Conclusion

Astoria, SSIS Adapters and BDD – On BDD and MSpec

hdierking — Fri, 13 Nov 2009 01:04:00 GMT

Let me preface this post by saying that it is about my (early) experiences practicing BDD in the context of a specific specification framework – MSpec. There are plenty of folks out there who are much more authoritative on this subject than I. And if you happen to be one of those people, I would love to get your feedback on what I’ve done here and how I could do it better.

In developing a SSIS data source to talk to an ADO.NET Data Services endpoint, shifting my approach from a basic TDD style to specifications was not originally part of the plan. However, as I got deeper into development, I felt as though I actually wanted a bit more rigor in terms of semantic restrictions (e.g. arrange/act/assert) than I was getting out of the box from my unit test framework. MSpec had been on my radar for a little while (I was introduced to both this framework and BDD in general at ALT.NET), so I took a quick look - and after staring oddly for a moment at the =()=> syntax decided that it provided the kind of semantic consistency I was really looking for. Additionally, the fact that it could plug into the ReSharper unit test runner (with underscores removed from boxcar casing no less!) made it quick and very easy to get up and going on the journey writing specifications rather than tests. Going through the specification writing exercise on this project has me convinced at this point that as someone else once said, BDD is TDD done right. I certainly plan to keep pursuing specifications over traditional unit tests in future efforts.

So where to start?

I’m not going to spend time in this post talking about how to use MSpec – there’s already lots of good guidance out there. That said, because the syntax initially looks a bit funky, I will say 2 quick things that are helpful to keep in mind when you first look at specifications built on top of MSpec.

Everything in your specification (test) class is a field. And all of the work gets down via delegate fields which are of types with names like “Establish”, “Because”, and “It”
Regarding the =()=> syntax, you are probably already comfortable with lambda expressions, so you can ignore the => on the right - and you can get the assignment operator on the left. Therefore, that just leaves the empty parenthetical – and this just means that your delegate (lambda) is for a function that has no arguments. There, all better?

I already had some unit tests written – though some of them were more of integration tests and revealed some pretty serious problems in my design – but we’ll deal more with those topics later. So my starting point was to try and re-write some of my existing tests as specifications. For example, take the following:

[Test]
public void Given_Null_URI_Throw_Exception() {
    try {
        new DataServicesQuery(_dataServicesQueryBuilder, null, "Suppliers");
    }
    catch(ArgumentNullException ex) {
        Assert.AreEqual("serviceUri", ex.ParamName);
    }
}

Re-writing this as a specification resulted in the following:

public class when_service_uri_is_null : with_dummy_resource_builder_and_reader
{
    static NullReferenceException lastException;

    Because service_uri_is_null = () => {
        try {
            queryBuilder.ServiceURI = null;
            queryBuilder.ResourcePath = "Products";
            queryBuilder.GetQuery();
        }
        catch (NullReferenceException ex) {
            lastException = ex;
        }
    };

    It should_have_specific_error_message = 
        () => lastException.Message.ShouldEqual("Service URI must be provided");

    It should_raise_an_exception = 
        () => lastException.ShouldNotBeNull();
}

A couple things to point out here.

First and foremost, there is a more rigid semantic that is enforced by both convention and even by the type system. The convention is that classes follow a “when X [with Y]” pattern (I found that this class name should map very tightly to the instructions contained in the Establish and/or Because expressions – if it does not, you may have a problem in your design). Also, assertion (“It” expressions) delegates should follow a naming convention whereby “should …” is followed by a description of the assertion. MSpec also uses the type system to enforces this convention in how the delegate types are named (e.g. – “Because”, “It”).
All of the NUnit attributes identifying test fixtures and tests are gone. MSpec is smart enough to figure out what is a test vs. what is a regular old class based on the MSpec delegates that make up the class. When looked at through the NUnit lens, the specification class remains the test fixture, and each “It” delegate instance becomes an NUnit test.
Notice the with_dummy_resource_builder_and_reader class. As with unit tests, inheritance can be a great way to provide shared context to multiple test fixtures. This base class is responsible for 2 things: 1) exposing a few data fields for its child classes, and 2) providing its own Establish delegate to setup those data fields.
For me, this format of specifications seemed to yield another benefit – without explicit effort. That benefit was more tests with each test having a much better definition around its responsibility.

This particular type of test – one where you’re expecting an exception – was one where I wasn’t 100% confident in my approach. However, I didn’t find any baked in support from MSpec – and frankly, for the framework to provide a feature similar to NUnit’s ExpectedExceptionAttribute doesn’t really make a lot of sense to me for writing specifications anyway. You can see above the approach that I took – simply trap the exception and then run a couple tests over the exception to assert various characteristics of it. If there’s a better way to go about this, I would love to hear it.

After I started getting the hang of writing specifications by rewriting my tests, I began venturing into new parts of my application – and this is where I really started realizing just how much ceremony exists in traditional unit testing frameworks compared to something like MSpec. Because mostly everything in MSpec is a field, there’s no need to provide any kind of method body, etc. after declaring the title of the specification. So, for example, when I wanted to write specifications around executing a query, I was able to quickly capture the specs as follows:

public class when_query_is_executed_for_products_with_expanded_suppliers
     : with_query_shape_factory
{
    It should_have_2_result_sets;
    It should_have_a_result_set_named_products;
    It should_have_8_columns_in_products_result_set;
    It should_have_77_rows_in_products_result_set;
    It should_have_a_result_set_named_suppliers;
    It should_have_13_columns_in_suppliers_result_set;
    It should_have_77_rows_in_suppliers_result_set;
}

Remember, “It” is simply a type and “should_have_2_result_sets” is simply a field declaration – no need to initialize it with anything just yet. When the MSpec runner is executing, it checks to see whether the field has been initialized – if it hasn’t, as in this case, it flags it as if you used NUnit’s IgnoreAttribute.

Finally, I mentioned earlier in this post about using ReSharper to run MSpec specifications. While I would probably still be interested in MSpec if this integration didn’t exist, the fact that it was there removed any question or hesitation. We talk a lot about the readability of tests, and of tests being the spec – and this is, based on what I’ve experienced so far, where specification frameworks like MSpec really shine. Bringing that goodness directly into Visual Studio ala ReSharper mades the feedback loop even tighter, and really made for a happy development process.

If you’ve read this far and you have no clue what this project is that I’m talking about, see the overview here. If you want to download the code and look at it yourself, check it out at http://code.msdn.microsoft.com/ssisastoria

Switching virtual networks the easy way

pvanooijen — Thu, 12 Nov 2009 10:40:00 GMT

For my laptop I am using server 2008 as OS. The main reason is Hyper V, for every different project I can fire up one or more appropriate virtual servers. For the details on setting up such an environment see this post. On the advantages of using Server 2008 R2 see this post. Traveling around the laptop meets a lot of different networks, from the Office domain of which it is a member to some kind of wireless in the wild open. Sometimes I want my virtual servers to see and be seen by the outer world, sometimes I don’t.

You can do a lot of network configuration in Hyper-V, in this post I described a rather baroque configuration where all kinds of network connectivity can be switched on and off. Which worked well but is far more complicated than needed. By now I have found out that switching networks in Hyper –V can be as simple as this:

Create a virtual network for every adapter on the laptop.

My laptop has an internal (fixed IP address) adapter and an external USB adapter (depending on DHCP). The Hyper-V virtual network manager does not allow you (yet ?) to pick Wireless or dial-up adapters for a virtual network.

Assign each virtual machine just one network adapter.
Select the active adapter as needed.

The good thing is that this can be done while the virtual machine is running. Changing the network is instantaneous. Just like pulling the cable and inserting another one. So much simpler than all the previous multi network fiddling.

[ANN] DC ALT.NET – 11/24/2009 “Stop Talking, Start Teaching” with Jeff Casimir

Matthew.Podwysocki — Tue, 10 Nov 2009 19:48:14 GMT

As a group, we’ve been focusing a lot on technology on such things as JavaScript, MongoDB and so on. This month, I’d like to take a step back as a group and instead focus on some soft skills. When I started this group back in 2007, I had the idea that this group would help others not only become better technologists in a wide array of technologies, but also better teachers as well. On November 24th, we have Jeff Casimir to help us understand how to become a better teacher with his presentation “Stop Talking, Start Teaching: 5 Rules for Successful Presentations and Demos”. Listed below are the details:

Stop Talking, Start Teaching: 5 Rules for Successful Presentations & Demos

Some of the most important work we do as programmers involves pitching ideas. Whether it's inside the team, searching for funders, or running demos for potential clients, we all need to present ideas and cultivate an audience. The past decade has seen the rise of the presentation as an important communication medium, but few people do it well. There are many resources on how to make good looking slides, but we need to focus on the method, structure, and content.

We don't need to be "speakers" – we need to be teachers. In this session we'll explore the five most essential rules to successful teaching and how they apply to you.

Attendees will, by the end of the session, have a framework for designing and critiquing their work and that of others. This new understanding will make them better teammates and more successful salespeople.

About Jeff Casimir

After majoring in Computer Systems Engineering I joined Teach for America and began a career in education. I taught high school Computer Science for four years before moving into school administration. As a Vice Principal I was responsible for evaluating and hiring new teachers, observing and coaching existing teachers, and defining much of the school's academic process. Most of all, I hate boring presentations. I want these smart, experienced people to Stop Talking, and Start Teaching.

Hope to see you there!

Design Conundrum for StoryTeller -- Opinions Wanted

Jeremy D. Miller — Mon, 09 Nov 2009 21:31:00 GMT

I made the preview release of StoryTeller a couple months ago as soon as I got some sort of "intelligent editor" functionality going in the WPF client so you very rarely ever had to expose your tender eyes to the horrors of the underlying Xml storage. Fine, except for one little problem: the UX and usability suck (and I know because my team uses it and I'm irritated with it when I use it. Hooray for dogfooding). It's also butt ugly.

Flash forward to the past couple months. My team at Dovetail is putting the finishing touches on our online rules engine editor feature in our product (I'm blogging about it soon). We basically use StoryTeller as our "DSL" engine, but we built a completely web based editor for the StoryTeller grammars and integrated that into our web application. I had originally envisioned this module as being written in Silverlight because of the complexity of the dynamic layout and behavior. Instead, we decided to build the UI out with client side JavaScript. We do generate HTML templates on the server representing the DSL grammars that embed behavioral directives with class attributes and copious usage of the jQuery Metadata plugin, but all the behavior of the UI is accomplished through client side scripting. My team might not completely agree, but frankly I'm thrilled with how it's gone. The UI looks pretty good and the screen behavior was relatively easy to create with the HTML/JavaScript/CSS/jQuery combination. We were even able to TDD quite a bit of the JavaScript development using QUnit.

So here's my dilemma, I'm committed to making StoryTeller rock and I absolutely need to replace the existing test editor functionality. At this point I have two options -- either write a new version using WPF that closely mimics the improved usability that we built at Dovetail or stick with JavaScript/jQuery/HTML and embed a WebBrowser control into StoryTeller's WPF client (which I already do). Here are the facts as I see them and I'm obviously biased at the moment toward the HTML/JavaScript model -- but I'm looking for other people's opinions, especially about WPF:

Pro-WPF

It's an excuse/reason to learn a lot more about WPF for me. I think I'd finally have to dig into data templates, styling, and attached behaviors to do this well.
Embedding a WebBrowser control into a WPF app and letting more COM stuff in the door is kloogey and occasionally error prone. The integration is odd
Might run faster
Richer editing controls

Pro-HTML/JavaScript/jQuery

The amazing jQuery ecosystem of plugins and support. At a minimum, I'd be using the jQuery UI sortable/draggable stuff, jQuery Validation, and jQuery Metadata. I went looking today and I just don't see any substantial OSS community around WPF other than more copycat MVVM frameworks than you can shake a stick at.
I'm much more familiar with the jQuery way of doing things and HTML
I've got a successful working prototype to start from
Many of the mechanics are much simpler with jQuery than with WPF. For example, I want users to be able to reorder parts of the StoryTeller specification by dragging elements around. This is just a matter of some declarative class names in HTML with jQuery, but the WPF equivalent looks exceptionally nasty.
I actually like coding with JavaScript and the dynamic features make it relatively easy to extend the model with less ceremony than a static typed language
The feedback cycle is faster with JavaScript than C#
I'd really love to be able to allow users to embed the StoryTeller editor in their own applications, and the HTML/JavaScript model has much more reach than a WPF or even a Silverlight model would.
I think this model is actually much easier to test than the WPF model. It's hard to write automated tests against WPF without something special like TypeMock that can get around how the WPF team sealed so many of the routed event classes or an expensive 3rd party tool that screen scrapes.
Truth be told, I don't really like WPF. It's weird and has more "oddball-ness" to it than any other technology I've ever used. There are parts of it that I love (the layout system comes to mind), and others that drive me batty (XAML's runaway verbosity, binding expressions, routed events). I don't hate it on first sight the way I did WebForms back in the day, but something about it just doesn't agree with me.

Call it a Wash

I've got JetBrains RubyMine for the JavaScript editing, so the tooling support is not really an advantage for the WPF/C# solution anymore
qUnit isn't that hard to use, so I'm not sure that the C# code is any easier to test and I have a recipe for integrating qUnit tests into an automated build anyway.

Anyway, it's been interesting. Now, what do you think?

On being Culture Aware

Patrick Smacchia — Mon, 09 Nov 2009 14:26:00 GMT

Normal 0 21 false false false FR X-NONE X-NONE

A typical development pitfall comes from forgetting about the culture setup of the machine on which the code is executed in production (typical it works on my machine mistake). Imagine that a program running on a French machine stores a float as the string "3,14". The float value cannot be restored from the same program running on a US machine. It would expect the string to be "3.14" with a dot, not a coma.

To prevent such mistake in the NDepend code I created the following CQL rule. Basically, it forces the code base to use only Parse() and ToString() methods that accept a parameter of type IFormatProvider. As far as I know, the problem occurs on DateTime, float (Single), double and decimal.

// <Name>Float and Date Parsing must be culture aware</Name>

WARN IF Count > 0 IN SELECT METHODS

FROM TYPES "System.DateTime",

"System.Single",

"System.Double",

"System.Decimal"

// The NameLike CQL clause operates on the signature

// "methodName(type1,type2...typeN)"

WHERE (NameLike "Parse\(" OR NameLike "ToString\(") AND

!NameLike "IFormatProvider"

With this rule set up, in the future we won’t forget about caring for culture when converting from/to string. Previously, I already wrote a post concerning this Nurture Knowledge Database agile state in mind.

FxCop has a similar rule named Specify IFormatProvider. We are inclined to append this CQL rule to the set of default CQL rules, what is your opinion?

Using Latest NUnit Version With MSpec

hdierking — Sun, 08 Nov 2009 20:11:00 GMT

As I’ve been working another blog post about my adventures in learning BDD ala MSpec, I ran into something that I wanted to share as the problem wasn’t immediately obvious (to me anyway).

The problem is that MSpec has a dependency on version 2.4.6.0 of the nunit.framework assembly – and frankly, it’s not really a very big problem. In fact, I didn’t even notice it until now because I had been running all of my specs exclusively through the Resharper runner (enabled by the MSpec plugin).

** As a side note, using R# with the MSpec plugin absolutely rocks, and while the runner is needed for integrating MSpec with your automated builds, I would totally recommend R# for your regular dev rhythm.

In my project, I originally had some integration tests in the same assembly as my specifications. As a result, I had a direct reference to NUnit (version 2.5.2.9222) in addition to the reference to Machine.Specifications and Machine.Specifications.NUnit (which has a reference to version 2.4.6.0). As a result, the later version of NUnit was being copied to my bin folder and throwing a file load exception for every specification when run using the console runner.

My initial reaction was to simply move all my integration tests to a different assembly and remove the direct NUnit reference in my specifications assembly (remember that referencing Machine.Specifications.NUnit will copy the version of NUnit to the bin folder that MSpec expects. No problem – everything works as you would expect.

However, it seems reasonable to me that, at least in larger projects that are wanting to incorporate specs and not have to re-write all existing tests, that it would be good to simply reconfigure the MSpec console runner to use the latest version of NUnit. This can be done easily using the standard .NET assembly binding redirection feature. Simply create a file named Machine.Specifications.ConsoleRunner.exe.config and drop it into the MSpec folder that contains your console runner (Machine.Specifications.ConsoleRunner.exe). The contents of the file should look like this.

<?xml version="1.0"?>
<configuration>
    <runtime>
      <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
        <dependentAssembly>
          <assemblyIdentity name="nunit.framework"
                            publicKeyToken="96d09a1eb7f44a77"
                            culture="neutral" />
          <bindingRedirect oldVersion="2.4.6.0"
                           newVersion="2.5.2.9222"/>
        </dependentAssembly>
      </assemblyBinding>
    </runtime>
</configuration>

Run the MSpec console again, and voila! Works like a charm.

Astoria, SSIS Adapters and BDD – Code

hdierking — Fri, 06 Nov 2009 19:57:00 GMT

I’m currently working on the post talking about how, in the course of writing my SSIS data source for ADO.NET Data Services, I stumbled into replacing my unit tests with specifications. However, in advance of that, I wanted to go ahead and provide the link to the source code. I would love to get feedback on what’s working and what’s not so that I can get better – and so that ideally, those the random stranger who downloads and looks at it on some idle Tuesday gets better as well.

The code can be found at http://code.msdn.microsoft.com/ssisastoria

December 2-3 London Workshop Update

Dave Laribee — Thu, 05 Nov 2009 15:29:00 GMT

Forgive the self-promotional interruption, but quick housekeeping note:

I've asked the company hosting my workshop in London to reduce the price of my 2-day workshop on December 2-3 aimed at tools and techniques technical leads, folks interested in lean, standard work, etc. to try to get as many people there (it's a group collaboration focused thing). I also want to get as much feedback as I can. If you book before November 15th, the price is £550.

The code is the truth, but it is not the whole truth

Patrick Smacchia — Wed, 04 Nov 2009 13:55:00 GMT

In a recent interview from Grady Booch, co-creator of UML, Grady said:

When Jim, Ivar, and I began our journey that became manifest in the UML, we never intended it to become a programming language. I think that there's a fairly narrow domain for which model-driven development makes sense but that we should return to the roots of the UML, which was to be a language for visualizing, specifying, constructing, and documenting the artifacts of a software-intensive system—in short, a graphical language to help reason about the design of a system as it unfolds. Most diagrams should be thrown away, but there are a few that should be preserved, and in all, one should only use a graphical notation for those things that cannot easily be reasoned about in code. As I've also often said, the code is the truth, but it is not the whole truth, and there are things such as rationale, cross-cutting concerns, and patterns that cannot easily be recovered or seen from code.... These are the things for which a graphical notation adds value, and any such notation should be used only if it has predictive power or reasoning power (meaning, you can ask questions about it).

I especially cannot agree more with:

As I've also often said, the code is the truth, but it is not the whole truth, and there are things such as rationale, cross-cutting concerns, and patterns that cannot easily be recovered or seen from code....

The code is the truth, the code represents the whole asset of a software shop. But code base is a complex asset. Call them patterns, rationales, intentions or decisions, there are many valuable intellectual artifacts buried in a code base that cannot easily be recovered or seen from code.

This is something I realized in 1997 while trying for the first time to demystify a C++ legacy code base. I was angry! What, in 1997, we, the professional developers, haven’t any tooling to visualize code? I remember also, in the architecture department next door, the architects were spending days doing diagrams that had nothing to do with what actually the code was. Putting hands in the code was something considered with low value. It was let for fresh graduated like me.

Hopefully, the result of this period was the seed from where the idea of a tool such as NDepend came to me. Code Metrics, Treemap, Dependency Graph, Dependency Matrix, Instability vs Abstractness diagrams… are natural artifacts to shed light on all these precious intentions buried in the code. I especially like the word intentions that was often used by Bruno Boucard, a former colleague I have great respect for. The idea of a language such as Code Query Language came to me when the need to make all these intentions concrete and verifiable, was clear.

What we do is the result of what we did and who we met, no matter whether it was good or bad experiences. There is no such thing as: today, I gonna find an idea to make my own business. Good ideas come naturally along the way, through what, Paulo Coehlo, would call signs.

Introduction to the Reactive Framework Part II

Matthew.Podwysocki — Tue, 03 Nov 2009 05:28:37 GMT

In my previous post, I talked a little bit about the parts of the Reactive Framework that are coming to the .NET 4 BCL as well as their implementation in F# as part of F# First Class Events. This time, let’s come back to the Reactive Framework itself from which the IObservable<T> and IObserver<T> originated. As you may remember, you can play with the bits of the Reactive Framework from the Silverlight 3 Toolkit.

In the first post in the series, I gave a basic introduction to the basic problems we have with asynchronous and reactive programming. We covered some of the evolution of the .NET framework with asynchronous programming and where the Reactive Framework fits. This time, I’ll pick up where we left off to talk about one of the important ideas behind the Reactive Framework, the duality of Enumerable and Observable.

The Duality of Enumerable & Observable

During Erik’s appearances on Channel 9, an important, yet recurring theme was the mathematical duality of the Enumerable to the Observable. What does that mean exactly though?

We remember from our first post in the series where we talked about the pull (interactive) versus the push (reactive) model. The pull model, represented by the iterator pattern of IEnumerable<T>/IEnumerator<T> states that we must explicitly call a method in order to get each item from our abstracted collection. On the other hand, our push model, represented by the observable pattern of IObservable<T>/IObserver<T> states that we register an interest through a subscription and then items are subsequently handed to us from some abstracted collection.

First, let’s look at the iterator pattern, which is a common interactive programming pattern that’s used quite frequently inside .NET. This consists of two interfaces, the IEnumerable<T> and the IEnumerator<T>. Listed below are some of the highlights of these two interfaces. I cut down what they are to their essential bits.

public interface IEnumerable<T>
{
    IEnumerator<T> GetEnumerator();
} 

public interface IEnumerator<T> : IDisposable
{
    T Current { get; }
    bool MoveNext();
}

The IEnumerable<T> interface exposes a single method which returns an IEnumerator<T> to iterate some object. The IEnumerator<T> gives us the ability to get the current item and determine whether there are more items to iterate.

To think about basic asynchronous programming, we must consider a basic Gang of Four Pattern, the Observer Pattern. This pattern describes where an object called an Observable, maintains a list of its dependent Observer classes and notifies them automatically of any state changes. In Java, this should be rather familiar territory with the java.util.Observer class and the java.util.Observable interfaces. In the Reactive Framework, this approach is no different, as we have an IObservable<T> interface which maintains a list of dependent IObserver<T> interfaces and notifies them automatically of any state changes. Let’s look at the signatures below:

public interface IObserver<T>
{
    void OnCompleted();
    void OnError(Exception exception);
    void OnNext(T value);
} 

public interface IObservable<T>
{
    IDisposable Subscribe(IObserver<T> observer);
}

What you may notice, and what you may have seen on some of Erik Meijer’s Channel 9 videos is that there is a duality between the Enumerable and Observable, given the signature of these interfaces. To explain what I mean, let’s first look at the flow of the Enumerator:

The GetEnumerator function takes no arguments and returns the IEnumerator<T> which has a no-argument function and returns the next item in the list through MoveNext and Current. Now, let’s look at the inverse of it.

The duality of the Enumerable solution is quite simple. We have our observer on the left hand side which has an action function which takes a T argument and returns void. The right hand side is the notification that is sent when something happens. This is all managed by the Observable as it maintains a list of all observers so that it can notify them.

To make this a little more concrete, let mash these two sets of interfaces together in order to dualize. First, let’s start with the IEnumerator<T> interface.

public interface IEnumerator<T> : IDisposable
{
    T Current { get; } // throws exception
    bool MoveNext();
}

What we’ll notice is that we have two items the Current property with a get accessor and the MoveNext method which takes nothing and returns a boolean that indicates if it can move next. You’ll also notice that our Current property also could throw an exception. If we’re in the Java world, we’d explicitly mark this interface with throws Exception, but as C# doesn’t have that, this comment will do. Now, let’s look to dualize this interface.

public interface IDualEnumerator<T>
{
    void SetCurrent(T value | Exception ex);
    void MoveNext(bool canMove);
}

At first glance, all we have to do is flip our Current property to have a setter. Because the Current yielded a value or threw an exception, that’s the way we’ll model it for now. Secondly, our MoveNext will now take a boolean to indicate whether we’re done. Let’s refactor this a little bit more and break the SetCurrent into two distinct operations, one for yielding the next value, and one for throwing an exception.

public interface IDualEnumerator<T>
{
    void Yield(T value);
    void Throw(Exception ex);
    void MoveNext(bool canMove);
}

Now that we have that under control, let’s then move onto our MoveNext method. As you’ll notice, we’ll continue to send in a true until we’re finished pushing values. Instead of calling MoveNext repeatedly, let’s instead call this only once when we’re finished.

public interface IDualEnumerator<T>
{
    void Yield(T value);
    void Throw(Exception ex);
    void Break();
}

What we then end up with is pretty much the signature of our IObserver<T>. By renaming our Yield to OnNext, our Throw to OnException and Break to OnCompleted, we’re left with our original IObserver<T> interface.

public interface IObserver<T>
{   
    void OnNext(T value);
    void OnError(Exception exception);
    void OnCompleted();
}

What about the dual of IEnumerable<T>? Let’s first look at what that interface is.

public interface IEnumerable<T>
{
    IEnumerator<T> GetEnumerator();
}

This interface has a single method called GetEnumerator which returns an IEnumerator<T>. How might we dualize this interface? Well, we already know that the dual of the IEnumerator<T> is IObservable<T>, just as well, we need a way to attach interest in a given dualized Enumerator, so we’ll create a method called Attach. Just as well, when we attach interest, we also need a way to detach our interest for a given observer. That might look like the following.

public interface IDualEnumerable<T>
{
    void Attach(IObserver<T> observer);
    void Detach(IObserver<T> observer);
}

If we look at the Java interfaces for Observer and Observable, these are pretty close in signature. There ultimately is a problem with this scenario in terms of composition. At each level, it needs to remember which things maps to which so that you could undo any of the observers if need be. Not doing so could lead to space leaks and is generally a bad idea. How could we solve this then? Ultimately, we need a way to track our attachment once we call the Attach through some form of a return value. At this point, the tracking is nothing more than needing the ability to detach our given Observer. If you’ll notice in our IEnumerator<T> interface, it inherits the IDisposable interface which gets one thinking. Could we instead, change our Attach method so that when we call it, we return an IDisposable instead which gives us the ability to track our subscription and be able to clean up after we’re done? That’s exactly the approach that the IObservable<T> takes in the final version of the interface.

public interface IObservable<T>
{
    IDisposable Subscribe(IObserver<T> observer);
}

Now that we’ve proven the duality, why does it matter? Because of this duality, the laws that apply to LINQ to Objects now can apply to LINQ to Observables (or Events). And in fact, we can see that through the Observable class which has many of our standard LINQ combinators such as:

Aggregate
First/FirstOrDefault
GroupBy
Join
Last/LastOrDefault
Select
SelectMany
Single/SingleOrDefault
Skip/SkipWhile
Take/TakeWhile
Where
Zip

Because of this duality, we can do such simple things as fire the ProgressChanged event until the RunWorkerCompleted event happens.

var worker = new BackgroundWorker();

var percentage = from progress in worker.ProgressChangedEvent()
                           .Until(worker.RunWorkerCompletedEvent())
                 select progress.EventArgs.ProgressPercentage;

var subscription = percentage.Subscribe(
    p => Console.WriteLine("Percentage Complete: {0}", p));

That’s only scratching the surface of the power of the Reactive Framework. We’ll go in depth in the next part of the series on the combinators. We could go into the fact that the Enumerable solution is really just the lazy list monad and the Observable solution is really just the continuation monad, but that’s for another time as well.

Conclusion

Should I use MEF with an IoC container? - Part 1

Glenn Block — Sun, 01 Nov 2009 06:36:00 GMT

In my post "Should I use MEF for my IoC needs" we took a look at the question of whether or not it is appropriate to use MEF instead of a traditional IoC container. In this post we'll discuss the question of whether or not you should use both an IoC container and MEF in the same application.

MEF and IoC container concerns

As we've mentioned in the past, MEF's primary design in V1 is around open-systems whose components (parts) are dynamically discovered at runtime. IoC containers on the other hand are primarily designed for decoupling a closed (fixed) set of application components in order to improve the maintainability of the system by the engineering team. In other words each is addressing a specific set of concerns. So if that is the case then is it plausable to use both in an application? The answer is Yes. Not only that, but we specifically designed MEF with the idea of integration with other containers in mind. Below will explore one of the key-scenarios for doing this.

Enabling third-party extensibility in an IoC container-dependent application.

You have an existing customer management system that uses an IoC container such as Castle Windsor which you use in order to promote separation-of-concens and testability. The application leverages advanced Windsor facilities for integrating with NHibernate and for applying cross-cutting-concerns (AOP) such as logging, etc. The loosely coupled design of the system has proven to be greatly beneficial and has minimized the friction for developing and deploying the system through several releases.

At some point requests start pouring in from customers want to create their own extensions in order to customize the application in their specific environments. Additionally several ISVs express interest in providing third-party value-adds such as telephony integration, and scheduling. You determine that adding a third-party extensibility model to the application makes sense in order to address this opportunity. Through your research you come across MEF and realize that it provides the infrastructure you need for extensibility. However it does so in a manner that overlaps with many of the functions that Windsor is providing. What to do? Let's explore various options.

a. Migrate the existing code to use MEF and remove the IoC container completely.

One option is to consider ripping out Castle and just using MEF. Technically this sounds reasonable, however after investigation it is not feasible. There are literally thousands of components that are currently registered in the Windsor container. Moving these to MEF involves modfying all of these existing classes in order to make them discoverable by MEF, not to mention refactoring all the places in the application where the container is accessed. Because the changes are so widespread, the likelihood of introducing breakages / bugs in the system is very high. Additionally there are the various Castle facilities such as Dynamic Proxy, and NHibernate integration which MEF does not yet support, which will have to authored by the team. The cost / risk of such changes is too prohibitive, not to mention that it seems like it should be unnecessary.

b. Implement a custom extensibility mechanism built on top of the IoC container?

Another option is to author a home-grown solution. This can be a viable option depending on the needs, and the willingness to author your own solution. For simple cases you can easily facade an extensibility mechanism on top of an IoC. Once you start getting into more advanced features like lazy instantiation, metadata, recomposition and stable composition, you are in for quite a bit of work.

c. Have both the IoC and MEF both present in the same application?

The final option I want to discuss (and the one I recommend for these situations) is to use both. Use the IoC container for addressing internal application decoupling needs, and use MEF for third-party extensibiltiy needs, that is the IoC container for IoCish things and the MEF container for MEFfish things :-)

This path is appealing for several reasons.

It works with existing codebases whether they be applications or frameworks. it does not require any significant rewrite.
It allows leveraging the strengths of each container for the things it does best. For example leveraging MEF's metadata model for discriminating between different extensions vs the dynamic proxy support in Castle.

However, there are several caveats of this approach.

It raises questions such as:
- Which component (part) should live in which container?
- Who owns the lifetime?
- Can a MEF part access services coming from the IoC? How?
- Can the IoC access exports coming from MEF? How?
It increases complexity in that now there are components living in mulitple places.
It increases the potential points of failuire within the application.

All of these issues are managable. In my next post we'll see how.

Getting Your Groove Off

hdierking — Thu, 29 Oct 2009 21:26:00 GMT

Until recently, the only knowledge I had of Groove was that it was some kind of collaboration product that we got as a result of bringing in Ray Ozzie. Then I got a new manager who insisted that our team standardize on Groove for the team document repository - so I got setup on Groove and went on with my life. In general, there’s nothing I found terribly impressive about it, but it wasn’t really getting in my way so no reason to get too upset…

Until my Tortoise SVN icon overlays stopped working.

I ended up spending quite a bit of time this morning trying to figure out exactly why they weren’t working. It was odd – I had no problems on my home machine – but on both of my work machines, same result – no TSVN icons. Finally, @pilotbob pointed me to this article which set off a chain of recollections – most notably that the major difference between my work machines and my home machine had to do with the order that I installed stuff. On my home machine, I installed Office after I got my dev environment setup. However, on my work machine, Office was installed as a part of the default OS install image - and Groove was included in that installation.

So why is this important? Because if you remember from the article, Windows supports only 15 overlay handler slots – and installing Groove puts 5 handlers right there on top of my TSVN handlers – the net result being that my TSVN icon overlays don’t load.

To fix, I simply went into the registry, deleted all of the Groove overlay handlers from HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\ShellIconOverlayIdentifiers

Then rebooted. Voila! TSVN icon overlays come right back up!

Low-Technology

Dave Laribee — Thu, 29 Oct 2009 18:55:00 GMT

I'm completely obsessed with the idea of low-technology:

The term low-technology is a description of those crafts and tools whose inception (typically) predates the Industrial Revolution.

A test for low-technology may be that it can be practiced or fabricated with a minimum of Capital investment by an individual or small group of individuals; and that the knowledge of the practice can be completely comprehended by a single individual, free from increasing specialization and compartmentalization.

An excellent overview of some of the cutting edge efforts in low-technologies can be seen in this TED video with low-technologist Amy Smith. I completely agree with when says, "not all inventions need to be grandiose, complex things... sometimes they can be simple and smart ideas that just help a lot of people."

The second block quote is where I'm coming from. What are those tools in software development that add lots of value without minimal getting started overhead. Pair-programming seems like an excellent example of low technology. Emergent design, test-driven development and value-driven approaches like XP are yet more.

It seems, to me anyway, that these are the things that'll yield the most benefits. Both vendor-supplied tooling and trending topics we hold near and dear, like Domain-Driven Design and DSLs and this fluent madness in general, strike me as more high-technology approaches.

There's nothing wrong with high-technology, of course, but my instinct is that the risk-reward equation for most teams usually favors low-technology. We have, after all, limited bandwidth and attention, so my vote will ever go to what I see as big and easy wins: stuff focused on beefing up the signal of our communication as a team and community against the noise we might get as a software fashion victim.

Anyway, this is a key topic I'll be centering on over the next little while and thought y'all might find this way of looking at our kit, our standard tradecraft at least a little bit interesting.

Eschewing Date Types in our Database

karl — Thu, 29 Oct 2009 13:41:00 GMT

About a month ago, I found myself transforming data for the reporting portion of our application. Like many reporting systems, ours works off of date ranges, with statistics grouped together by the day (and a few by the hour). An early prototype made heavy use of TSQL Date functions (like DateAdd and DatePart). However, I wasn't happy with this solution, my transformations were unnecessarily coupled to SQL Server. I mean, database date functions all do pretty primitive date arithmetic's, surely we can come up with a way to do that ourselves?

I decided to borrow the strategy long-used by the Unix and PHP community (or JavaScript or doubtless countless other platforms/languages) - to store dates as seconds relative to a starting point. Unix Epoch is January 1st, 1970. If you store seconds since then in an unsigned 32 bit integer, you can store up to 136 years. If you only need minute-resolution, you can store over 8 100 years. If you only need day-resolution, then you can store a bit under 12 million years.

Best of all, you can do straight up arithmetic's against these fields to manipulate the dates. If you want to group hits per user for a day, you can do:

insert into DailyHits (UserId, DaysSince, Count)
select UserId, SecondsSince/86400, count(*) 
   from Hits 
   group by UserId, SecondsSince/86400
--86400 is the number of seconds in a day

insert into HourlyHits (UserId, HoursSince, Count)
select UserId, SecondsSince/3600, count(*) 
   from Hits 
   group by UserId, SecondsSince/3600 
--3600 is the number of seconds in a hour

You can also manipulate relative times in C#, to apply as parameters:

public static readonly DateTime Epoch = new DateTime(1970, 1, 1);
public IList GetDailyHits(DateTime from, DateTime to)
{

   command.CommandText = "select * from DailyHits where DaysSince > @From and DaysSince < @To";
   command.Parameters.Add("@From", from.Substract(Epoch).TotalDays);
   command.Parameters.Add("@To", to.Substract(Epoch).TotalDays);
   ...
}

If you are using NHibernate, you can use the IUserType interface to help map DateTimes to their relative counterparts (I think).

The examples above are simplistic, but they do work very well and have a number of benefits - the most significant being great independence from the underlying database.

Astoria, SSIS Adapters and BDD – Take 1

hdierking — Wed, 28 Oct 2009 16:00:17 GMT

Yea I know – 3 things you never thought you would hear mentioned together. And to be really honest, I didn’t start out trying to build this in a BDD fashion – and since I’m totally new to the disciplines of BDD, I may not have been all that successful. But hey, it’s all about the journey, and so I wanted to at least go ahead and start posting about the work I’ve been doing here, partly to share and partly to get your input ways to improve my design/specs.

So here was the initial goal – I was studying ADO.NET Data Services (previously known as Astoria) when I remembered this article that I once read in the Architecture Journal about using RSS as a technique for surfacing data to load into an enterprise data warehouse. That led me to think that it would be a cool opportunity to simultaneously learn Astoria and feed my BI sweet tooth by writing a SSIS data source that could consume an Astoria endpoint. Given the fact that more and more systems are exposing data using syndication, it seemed like creating an SSIS data source was something that could be helpful beyond my simply wanting to play with the technology.

Wait - ADO.NET Data Services what??

Just in case you weren’t 100% sure what ADO.NET Data Services is, it’s technology to expose data and enable operations on data via RESTful operations. You can find out much more about it here and here, but for the purposes of this exercise, here are the 3 things you need to know.

It uses all standard HTTP operations – which means that for getting data, you simply format a HTTP GET request for the service URL plus additional path information and optional querystring arguments to tell the framework how to shape the data.
While there’s not a dependency on the Entity Framework, there is a dependency on the Entity Data Model in that metadata is exposed as CSDL.
Though you have a couple formatting options when getting data, the primary one (and the one used by the data source component) is to get the data as an ATOM feed. The data source component can then project the XML into the appropriate output buffer (or buffers as you’ll see later).

Wait – SSIS what??

Sql Server Integration Services is Microsoft’s ETL tool that ships with SQL Server. You can start learning more about it here, but for the purpose of this exercise, here are the 3 things you need to know.

The unit of execution in SSIS is the package. This is really similar nomenclature to DTS because, not surprisingly, SSIS evolved out of DTS.
Every package contains 2 kinds of logic: 1 “Control Flow” and zero-to-many “Data Flows”. Control flow is meant for larger operations like copying files or running some kind of database maintenance operation (and while interesting, it’s not something I care about for this exercise). The real meat of SSIS (IMHO) happens within a data flow – this is where data is extracted from a source, massaged, and put into a destination. Data flows maintain the entire pipeline of operations between source and destination via in memory buffers.
Data flows use data source components to extract data from the source and add it to the appropriate buffers in the pipeline. Data source components are also used to provide metadata to the SSIS design-time experience so that package designers can use it to clean up and massage the data using transformation components (** for those experienced SSIS folks, it’s worth noting that there is an XML data source component that is a part of the product, and since ADO.NET Data Services returns data as an ATOM feed, is is absolutely possible to consume it directly. However, in order to take this approach, you will need to generate a XSD schema and associate it with the component at design time for the metadata – and every time you change the shape of your data services query, you will have to re-generate and re-associate a new schema. This was one of the primary motivators for me to create a custom data source).

So then, the initial goals were as follows -

Create an SSIS data source component that could consume an ADO.NET Data Services endpoint
Ensure that the component participated in both the design time and runtime experiences
Ensure that the component was able to handle projections of ADO.NET Data services queries that took advantage of the $expand clause

As I got into the development I was pretty quickly able to accomplish the first 2 goals. For example, given the following query configuration..

I was able to get the data shape using ADO.NET Data Service’s $metadata option and expose it back to the designer..

I was even able to run the package for a simple data services query and pipe the results to a flat file..

However, when I started implementing that 3rd goal of managing projections, I quickly realized my quick and dirty project need more thought in its design. This drove me to add a few more goals.

Project should separate the core logic of the component from the SSIS infrastructure – which is heavily reliant on inheritance
Core component logic should be independently testable without requiring a running data services endpoint

Separating the logic out into a different assembly wasn’t really all that painful – just a few keystrokes of Resharper, a couple file moves, a reference added, and I was good to go. However, because this started as a quick and dirty project (I should have known better), I hadn’t designed for data source independence and as such had some larger refactorings to do. And in that process (** keep in mind that this started as an remains primarily a learning exercise for me – I would NEVER recommend taking this kind of “while I’m at it approach” on a real project), I decided that my tests weren’t really where I wanted them and that I had always wanted to get my hands dirty with BDD anyway, so why not? It’s just one more goal.

Tests should be written as specifications

I’ve read from some that BDD is really meant more for testing larger behaviors in business applications. However, it feels pretty good (though initially uncomfortable) at the grain at which I’m using it. For example..

public class when_products_resource_is_queried : with_northwind_CSDL
{
    Establish context = () => queryBuilder.ResourcePath = "Products";

    Because of = () => query = queryBuilder.GetQuery();

    It should_have_8_properties = () => 
        query.DataShape.First().Properties.Count.ShouldEqual(8);

    It should_have_a_single_resource = () => 
        query.DataShape.Count().ShouldEqual(1);

    It should_have_1_key_property = () => 
        query.DataShape.First().Keys.Count.ShouldEqual(1);

    It should_have_correct_query_string = () =>
        query.ToString().ShouldEqual(string.Format(
            "{0}/Products", Constants.SERVICE_URI));
}

Ok – so those were the goals as they evolved. Going forward, my plan is to dig into the logic design and specifications, then talk specifically about projections, then finally talk about how this stuff works with SSIS (and how I would like to see the SSIS API simplified). Sorry to not get into all that tonight, but as my friend Glenn has advised me countless times, if you wait to blog until everything is nice and tidy and completed, you may never blog again.

The Most Productive Programming Language

Dave Laribee — Wed, 28 Oct 2009 00:08:00 GMT

The most productive and useful programming language you know, by a long shot, is English or Hindi or Portuguese or Spanish or French or whatever human language your team uses. Good communication skills are way more valuable to you and your customers than picking up a new language every year. That is, while there is value in the item on the right, I value the item on the left more. Sound familiar?

Developers that name things well and program with a respect for human languages create more maintainable and easier to follow code. Being able to communicate with a customer, analyst and/or product owner can save you hour upon hour of effort and keep your solution simple. No programming language no matter how syntactically minimal or internal DSL friendly can ever come close to this.

Hierarchical Namespace Component

Patrick Smacchia — Mon, 26 Oct 2009 14:49:00 GMT

Modeling components of a code base with namespaces is a much better idea than modeling them with assemblies. I wrote about the namespaces vs. assemblies debate in the past (Control components dependencies to gain clear architecture). The fact is that namespace is a logical artifact while assembly is a physical artifact. Of course, creating assemblies to partition parts of the code that at runtime physically don’t live within the same AppDomain is a good thing. I explained that in Advices on partitioning code through .NET assemblies. Unfortunately, the vast majority of .NET teams use assemblies instead of namespaces to partition logically their code. They end up with hundreds of Visual Studio projects:

Hundreds of VS projects organized through dozens of VS solutions is something hard to manage.
These hundreds of projects takes a lot of extra time to be compiled (because of the repetitive tier assemblies parsing). By extra time I mean that the compilation takes minutes instead of seconds!
And finally, loading all these assemblies at runtime slow down consequently the startup time of applications. Indeed, doing so multiply the unit overhead that the CLR needs to load an assembly. The overhead is mainly due to CAS security checks, cryptographic signing check, and IO file read.

In order to model logical components, the logical nature of namespace is a preemptive advantage over the physical nature of assembly. However namespace have a second awesome characteristic: they can be declared hierarchically. So namespaces can be use to modelize a hierarchy of components. Today I would like to ramble on some consequences and caveat of organizing hierarchically components.

Theory: Tree vs. Flat view

This hierarchical way of componentizing code is something that we took account early in the NDepend development. At any time, the user can choose to display namespaces through tree (i.e through a hierarchy)…

…or to display flat namespaces:

Personally I prefer the tree option. After all, why do developers naturally partition the code? Because of the need to organize code and API through some smaller chunks. Smaller chunks are naturally easier to develop, understand, maintain, test and debug. This is the famous divide and conquer tenet.

But when the code base is consequently growing developers are facing a dilemma:

Smaller components are easier to be managed one by one. But this implies to deal with hundreds of components. As a consequence the big picture becomes messy.
Keeping a small number of components force to have large components. While the big picture remain understandable, larger components are harder to manage.

This is why developers apply recursively the divide and conquer idea. As a result, they obtain a hierarchical organization of code.

Practice: Tree vs. Flat view

These 2 dependency graphs illustrate well the point. The first one is made of the 52 namespaces declared in the mscorlib (this is the flat view).

The second one is made of the 16 root namespaces of the DLL mscorlib (this is the root tree view). The root tree view present less information than the flat view, but at least it is readable.

Moreover the root tree view is a good start to dig into the code and browse nested namespaces inter-dependency:

The same phenomenon appears with the Dependency Matrix. The root tree view (first matrix) is a better way to browse the code than the flat view (second matrix).

Component Hierarchy and Dependency Cycle

Avoiding cycles between components is an essential point to care for, in order to keep a code base maintainable. I detailed that in the post: Maintainability, Learnability, Component, Layering. When components are hierarchically organized, some non-trivial cycles can appear. Let's look at a small example.

If we look at the dependency between the 3 following namespaces we won’t find any cycles:

namespace ParentA.Child1 {
   public class ClassA1 { private ParentB.ClassB m_Field; }
}
namespace ParentA.Child2 {
   public class ClassA2 { }
}
namespace ParentB {
   public class ClassB { private ParentA.Child2.ClassA2 m_Field; }
}

There is no cycle, this is clear from the flat dependency view,

This is what I called a non-trivial cycle because a cycle exists while there are no cycles between namespaces. The fact is that RootA.* is not considered as a namespace because it doesn’t contain directly any class. In our internal NDepend terminology we named RootA.* a namespace container. Technically your code is entangled and contain a cycle even if physically there is no cycle between namespaces!

Hierarchical components constitute a powerful way to partition code but be aware that some non-trivial entangling cases might appear. In a next post, I’ll expose a cool way to use hierarchical componentization effectively in the real world.

Something you can (not) do with your (old) MSDN disks

pvanooijen — Mon, 26 Oct 2009 12:52:00 GMT

Time to renew my MSDN subscription. I don’t know about you but I’m still very pleased with the MSDN program.

In the old pre DVD days it used to be quite a problem what to do with the enormous piles of disks you received over and over again. Especially those things you never use. Anyone interested in a Finish version of Exchange server ? A Greek version of Windows 98 ? Since DVD times the piles has gotten a little smaller, but still. Googling around what to do with all these disks pops up an old rant of mine on this.

Being a biologist with a huge interest in aquarium fish (see my W7 betta rant) and an “old goat-wool-socks” environmental concern I thought I had a good use. Digital disk are very reflective, using them as a reflector for the light in my aquarium would give the disks a new life and would increase the effectiveness of fish tank lighting. So I tried.

This is what the result looked like in less than a year.

Moist has crept between the reflective (aluminum) layer and the (polycarbonate) disk itself and completely oxidized the metal. Resulting in a surface which reflects far less than the ground surface itself. Lesson learned: this does not work.

What you can do today with your disks is what I already asked for in my original rant: don’t get them. MSDN subscriptions now offer a download only option. You are free to download what you need (and what’s in your license). For many a product this was what I did already, downloading got me the product long before the disk finally arrived by mail. So if you want to stop wasting resources too: do switch to this subscription type. The only wish I have left is to save a little on money. Not producing and distributing the disks saves MS in the costs. I would like a small share of that in the form of a reduction on my MSDN fee. Fair deal ?

Postgresql - Day 2

karl — Sun, 25 Oct 2009 01:40:00 GMT

My migration from SQL Server to PostgreSQL for an upcoming project continues. On day 1 I managed to get the application up and running quickly, thanks to NHibernate. Ayende was kind enough to post some advice to minimize the already-few problems I ran into (I've since implemented the ISQLExceptionConverter and am really happy about it).

Today I had a bigger challenge - making the OLTP to OLAP transformation script work. I knew this would be more difficult because I'm using straight up SQL (instead of HQL), as well as a couple instances of SQLBulkCopy. So far, most of the transformation is done in sql - there's very little data being pulled down and processed by in code. Luckily, I have a huge and thorough integration test, which populates the database with a couple days worth of data, executes the transformation script in steps (as it would be in real life), and then validates the results. Essentially my test figures out the results in C# (by looping over the entities and aggregating the results every which way) and compares it to what the transformation scripts does.

Aside from a couple syntactical differences, such as having used [ ] to escape the count column, 1 column ambiguity that PostgreSQL couldn't resolve (weird), and having used insert {table} instead of insert into {table}, I only ran into 2 major problem.

The first was my usage of SqlBulkCopy which is tied to SQL Server's bcp utility. I was only using it to bulk-load the data for my integration test, but it did make a huge difference, and I wanted to keep the same performance if possible. It turns out that PostgreSQL supports a COPY from STDIN command, which the .NET Npgsql library exposes as an NpgsqlCopyIn class. Although it didn't perform quite as well (it still rocked), I much prefer the implementation. You essentially write a CSV to a stream - much nicer than having to transform my objects to datatables. Here's the reusable code that I used:

public static void BulkCopy<T>(IEnumerable<T> items, Func<T, object[]> callback, string tableName, string columns)
{
    var sql = string.Format("copy {0} {1} from stdin with delimiter '|'", tableName, columns >> string.Empty);
    using (var connection = new NpgsqlConnection(_connectionString))
    {
        connection.Open();                
        using (var command = new NpgsqlCommand(sql, connection))                
        {                    
            var copy = new NpgsqlCopyIn(command, connection);                    
            try
            {
                copy.Start();                        
                foreach (var item in items)
                {
                    var data = SerializeData(callback(item));
                    var raw = Encoding.UTF8.GetBytes(string.Concat(data, "\n"));
                    copy.CopyStream.Write(raw, 0, raw.Length);
                }
            }
            catch (Exception)
            {
                copy.Cancel("Undo copy");
                throw;
            }
            finally
            {                        
                if (copy.CopyStream != null)
                {                            
                    copy.CopyStream.Close();
                }
                copy.End();
            }
        }
    }
}
private static string SerializeData(object[] data)
{
    var sb = new StringBuilder();
    foreach(var d in data)
    {
        if (d == null)
        {
            sb.Append("\\N");
        }
        else if (d is DateTime)
        {
            sb.Append(((DateTime) d).ToString("yyyy-MM-dd HH:mm:ss"));
        }
        else if (d is Enum)
        {
            sb.Append(((Enum) d).ToString("d"));
        }
        else
        {
            sb.Append(d.ToString());
        }
        sb.Append("|");
    }
    return sb.Remove(sb.Length - 1, 1).ToString();
}

You use it like so:

var users = GetAllUsers();
Func<User, object[]> callback = u => new object[] { p.Id, p.Email, p.UserName, p.Password };
BulkCopy(users, callback, "Users", null);

If you want to specify the columns, say because the Id is an auto incrementing field, simply do:

var users = GetAllUsers();
Func<User, object[]> callback = u => new object[] {  p.Email, p.UserName, p.Password };
BulkCopy(users, callback, "Users", "(Email, UserName, Password)");

The code essentially turns each entity in your collection into a serialized string, which gets written to the stream.

The other problem I had was that one of my most complex queries was really slow. The query had joins, joining a subselect in the where clause. SQL Server didn't have a problem with it, PostgreSQL did. I wasn't too surprised because past experience with MySQL had taught me that such queries could be problematic. I saw a few statements hinting that PostgreSQL 8.4 (I'm running 8.3) might fix it. I spent some time trying to speed it up, had some success, and then simply decided to go about it a different way. While I think its very likely that I wasn't doing it properly, there's no denying that SQL Server handled this better.

I did time the execution of my integration test, using the same code. This was run on identically pathetic machines - an Atom 330 with 4gigs of ram and a 5400RMP hard drive. Postgresql 8.3 on Ubuntu server 9.04 and SQL Server 2008 on Windows Server 2008 R2. Without using any bulk copy, Postgresql was slightly faster. With their respective bulk copy, SQL Server was slightly faster. I'm not giving numbers because the results were really close, seconds apart when everything took minutes. Also, I do think the test was useful; however, the code was single threaded which means it didn't provide any insight on how the databases behave when under a more typical kind of load. My goal was really just to make sure I wasn't making a huge mistake, and it doesn't look, or feel, like I am.

Machine.Migrations changes

aaronjensen — Sat, 24 Oct 2009 21:56:00 GMT

Jacob first introduced Machine.Migrations over a year ago. Since then, it's been a solid part of our process and we're up to nearly 500 migrations with it on one project.

Recently, I finally got around to making some changes I've been wanting to make and I wanted to call'em out:

We split the repostiory out to its own on github. I did this similar to the way I did it for MSpec some time ago. Again, the advantage here is that change logs are localized to the project rather than the entire machine overarching project.
I added complaints if you have multiple migrations with the same number. This silently caused problems before as one of the migrations would not get applied. Now it just yells and dies like a good app. Of course, this is not as important because of the next change.

Next I added support for timestamped migrations. This helps immensely on active projects with multiple developers. You don't have to deal with communicating migration numbers to the team when you add them, you just generate a new one and merge it when you feel like it. We actually had half of our topic on Campfire dedicated to our current migration number. The other half was who owed how many pushups for breaking the build.

Here's the rake task we use to generate a new migration:

namespace :new do
  task :migration do
    raise "usage: rake new:migration name=\"Your migration name\"" unless ENV.include?('name')

    name = ENV['name']

    filepath = "db/migrate/#{Time.now.strftime('%Y%m%d%H%M%S')}_#{name.gsub(/ /,'_')}.cs"

    text = File.read("db/migrate/template.cs")
    File.open(filepath, 'w') { |file|
      file.puts text.gsub(/\$MigrationName\$/,"#{name.gsub(/ /,'_')}")
    }
  end
end

Usage is simple, just create a template.cs in the directory (we use db/migrate) and then type

rake new:migration name="this is my migration name"

Example template:

using System;
using System.Collections.Generic;
using System.Text;
using Machine.Migrations;

public class $MigrationName$ : SimpleMigration
{
  public override void Up()
  {
    throw new System.NotImplementedException();
  }

  public override void Down()
  {
    throw new System.NotImplementedException();
  }
}

Finally, I added a command line runner. As we've grown more and more sick of msbuild and more and more fond of rake, we've wanted to slowly all but eliminate our dependency on MSBuild. The command line tool is quite simple:

Machine.Migrations
Copyright (C) 2007, 2008, 2009 Machine Project

  c, connection-string  Required. The connection string to the database to migrate
  t, to                 Applies or unapplies migrations to get to the specified migration
  u, up                 Applies migrations to the latest
  s, scope              The scope?
  d, directory          Required. Directory containing the migrations
  v, compiler-version   Version of the compiler to use
  debug                 Show Diagnostics
  r, references         Assemblies to reference while building migrations separated by commas
  t, timeout            Default command timeout for migrations
  ?, help               Display this help screen

The only two flags you usually need are -c and -d. Just pass in the connection string and the location of your migrations. Be sure to quote them: migrate.exe -c "my connection string" -d "my migration directory".

Hopefully these changes help others. Grab the latest here and just msbuild the sln to build it.

I do wonder though, what other active .NET migration projects are people using and how do you like them?

Map control confusion

pvanooijen — Thu, 22 Oct 2009 15:30:00 GMT

When it comes to including geographical information in your web-application you have to make two main decisions. The first one is which API you are going to use, the second one is who is going to provide the geographical data. These two are not always completely independent.

A lot of websites use google maps. The geographical information is supplied by google and the only way to get to that information is by using the Google Javascript API. When it comes to performance Javascript is far from ideal, more sophisticated applications which try to add a lot of information to a map are quite likely to hit a script timeout. The other downside of this approach is the (lack of) smoothness of the display. Panning or zooming a map results in a lot of image blur.

Both downsides can be overcome by using Silverlight. The DeepEarth project was the first to use deepzoom which results in a smooth display when scanning or zooming. The strong typed compiled C# Silverlight code working with the map has not only a speed advantage over Javascript, it’s running async resulting in an app which not only has a smooth look but also a smooth feel.

You can build amazing apps with that. This is a part of the MVP map created by DeepEarth evangelist SoulSolutions

When it comes to where and how to get the geographical data, which can be satellite images, road maps or routing information, confusion starts. The DeepEarth project offered a choice of map providers. Ranging from OpenStreetmap to Yahoo maps and Microsoft Live Earth. Google maps were not included, the reason is the one I mentioned. Nobody so far has managed to create a hack to get Google Map data without leaving JS.

But the most confusing data-provider was and is Microsoft. The first versions of DeepEarth offered plain Live maps. In later versions you were supposed to use LiveMaps webservices. The usage of these services was by far free. There was a free developer test account available which resulted in map images all decorated with the word “staging”. For production you had to buy a license. To get a proper deal for that always got stuck somewhere in the pipeline. At the same time the old versions using plain Live maps kept working. At the moment I’m writing this seems to be history already.

LiveMaps have become Bing maps. At the same time Microsoft started developing their own Silverlight map control. The good thing is that it is a marvel to work with. The performance is even better than DeepEarth. The API just shines, a lot of functionality is present straight out of the box. All the extra work I had to do to get my app running in Deep Earth could be replaced by just hooking into some events. The architecture is great as well. It is by no means just limited to using Bing Maps data, you can even bypass that completely using the control as a blank working surface with a Latitude-Longitude coordinate system. A part of the architecture is a clear and easy way to use any map provider you can imagine, or build yourself. There are a lot of good stories on the web on working with the control.

But, as announced, there is still confusion. First there is the name. It is Microsoft Virtual Earth Silverlight Map Control. I have no problem with that, do you like Bing ? I can’t get used to that. More serious is the source of map data. The control, now in CTP, displays full MS map data without any need to apply for a (test-) license. No staging, no problem. The control itself has a Token property, according to the documentation:

Gets or sets the token to use for service requests.

It contains an empty string. How and when to use it is not clear at all. When it comes to production will it be needed to use the control? Or will it only be needed to get actual MS map data? Which data ? Just routing information ? Maps? Satellite images ? I just cannot find out. In case anybody can shed some light on this, I would appreciate that very much. Building software using the control is great, but what are price and conditions my customer will have to deal with?

Migrating to Postgresql with my friend NHibernate

karl — Thu, 22 Oct 2009 13:23:00 GMT

Last night I decided to accelerate our plan to migrate from SQL Server to PostgreSQL and gave it a quick attempt to see what the work involved was. I was hoping that due to our usage of NHibernate, and by having meticulously written database agonistic code the migration would be cake.

Turns out I was right and although there were a couple bugs along the way, the process took less than 30 minutes. Today I want to talk about the small issues we faced in migrating our code, as well as provide an overview of what I've learned about PostgreSQL so far.

First though, why migrate? I want to avoid getting rantish [in this post], but the cost of ownership for SQL Server isn't remotely close to being competitive. SQL Express is great, provided you can live with (or are willing to) its numerous constraints. Bizspark (I only bring it up because I feel someone will mention it in a comment) is a subprime mortgage - it lulls you into a sense of free only to nail you with licensing cost down the road. I'm sure someone can quote some obscure feature that might be needed by some system, but for the rest of us, the only reason to use SQL Server is familiarity. Familiarity is a pragmatic and reasonable reason to pick a technology, but only for so long.

Secondly, why PostgresQL? Well, I've used sqlite and MySQL extensively in the past (including submitting patches to their .NET drivers). However, MySQL is currently in a weird state (the sun/oracle thing) and I need ACID compliance and my experience with InnoDB has been average.

Thirdly, its worth pointing out that the system being migrated has a comprehensive set of unit tests, including a number tests which hit the DB. While I generally consider unit tests a design-tool, there's no doubt that they provided a significant sanity check.

So, what problems did I run into?

Given the use of NHibernate, the application side of things was migrated quiet easily. However, I did have to modify a few things. This is possibly because of flaws in my code (vs limitations of NHibernate) - hopefully someone will point such flaws out.

Unique Constraints

I use unique indexes to ensure that things like user emails are unique throughout the system. However, I haven't yet figured out a way to catch such exceptions in a db-agnostic manner. My code looked something like:

try
{
    Session.SaveOrUpdate(user);
    transaction.Commit();
    return true;
}
catch (GenericADOException ex)
{
    transaction.Rollback();
    var sql = ex.InnerException as SqlException;
    if (sql != null && sql.Number == 2601)
    {
        return false;
    }
    throw;
}
catch (Exception)
{
    transaction.Rollback();
    throw;
}

Obviously this code isn't going to work with the PostgreSQL driver. I keep thinking there *is* a way to handle this across multiple databases, but haven't found the answer yet.

Count - Long vs Int

When I fetch the number of rows for a query - more often than not to do paging - i do a select count(*) with a UniqueResult<int>(). However, PostgreSQL's count(*) returns a long, while SQL Server returns an int (I've run into this before, since sqlite also returns a long). This causes a cast exception. Again, I'm sure there's a simple solution to this inconsistency.

Bit and Booleans

There was one specific query which tried to retrieve all active users. The IsActive field was declared as a bit in SQL Server (and a boolean in PostgreSQL). The following query didn't work in PostgresQL "from User where IsActive = 1". Why? because 1 isn't a valid boolean, and you'll get a database error that an integer can't be converted to a boolean. I had to change the query to use '1' (in single quotes) or one of the other valid boolean values for true.

Connection String

Its also worth mentioning that the connection string example listed here, didn't work for me using the latest Npgsql drivers. Instead of Initial Catalog, I had to use the Database key.

Aside from those little issues, the code was up in running in about 10 minutes. Most of my time was actually spent getting to know PostgreSQL. Here's what I've learned so far.

Learning PostgreSQL

PostgreSQL doesn't consider two null values to be the same, meaning you can put a unique constraint on a nullable column without any problems. In SQL Server you need a Nullbuster column to achieve this same result (its a neat trick, but its neater not having to use it).
PostgreSQL's uniqueidentifier type is called UUID, a bit is a boolean, and a datetime is a timestamp.
All of PostgreSQL's text types are unicode by default (so there is no nvarchar you simply use varchar and get the same benefits)
PostgreSQL's handling of auto incrementing (identity) is pretty different. Instead of declaring a column as int not null identity(1, 1) primary key, you define it as serial not null primary key. "serial" is just some syntactical sugar around what actually happens - PostgreSQL generates a sequence table for that column, and sets the columns default value to the sequence's next value. Of course, identity(1, 1) is just some syntactical sugar around however SQL Server implements the feature - its just more transparent in PostgreSQL.

Hopefully as the weeks go by, I'll have more information to share about the migration, and what I've learned.