Jérôme Laban

WCF Streamed Transfers, IIS6 and IIS7 HTTP KeepAlive

Jay — Fri, 10 Jul 2009 21:18:00 -0700

A while back, I was working on a client issue where I was having some kind of unusual socket exception from a WCF client connecting to an IIS6 hosted WCF service.

To get a long story short, if you're using the .NET 3.5 WCF streamed transfer on IIS6 and making a lot of transfers in a small time, disable the KeepAlive feature on your web site. The performance will be lower, but it will last longer (without a client support call).

Still here with me ? :) If you have a bit more time to read, here some detail about what I found on this issue...

The setup is pretty simple : A WCF client that is sending a stream over a WCF service that has the transferMode set to streamed. This allows the transfer of a lot of information using genuine streaming, which means that the client writes to a System.IO.Stream instance, and the server reads from an other System.IO.Stream, and the data does not need to be transferred all at once, like in a "normal" SOAP communication. I'm using the required basicHttpBinding for both ends.

The strange thing is that after having made more than 15000 requests to transfer streams, I was receivig this exception :

System.ServiceModel.CommunicationException: Could not connect to http://server/streamtest/StreamServiceTest.Service1.svc.
TCP error code 10048: Only one usage of each socket address (protocol/network address/port) is normally permitted 10.0.0.1:80.
---> System.Net.WebException: Unable to connect to the remote server
---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted 10.0.0.1:80

This is a rather common issue, which is mostly found where an application tries to bind to a TCP port but cannot do so, either because it is already being bound to an other application or because it does not use the SO_REUSEADDR socket option and the port was closed very recently.

What is rather unusual is that this exception is raised on the client side and not on the server side !

After a few netstat -an, I found out that an awful lot of sockets were lingering with the following state :

TCP the.client:50819 the.server:80 TIME_WAIT

There were something like 15000 lines of this, with incrementing numbers for the local port. This state is normal, it's meant to be that way, but it's generally more found lingering on a server, much less on a client.

That could mean only one thing, considering that IIS6.0 is an HTTP/1.1 compliant web server: WCF is requesting that the connection to be closed at after a streamed transfer.

Wireshark being my friend, I started looking up at the content of the dialog between IIS6 and my client application :

POST /streamtest/StreamServiceTest.Service1.svc HTTP/1.1
MIME-Version: 1.0
Content-Type: multipart/related; type="application/xop+xml";start="<http://tempuri.org/0>";boundary="uuid:41d2cf74-aaa6-4a80-a6c4-0ec37692a437+id=1";start-info="text/xml"
SOAPAction: "http://tempuri.org/IService1/Operation1"
Host: the.server
Transfer-Encoding: chunked
Expect: 100-continue
Connection: Keep-Alive

The server answers this :

HTTP/1.1 100 Continue

Then the stream transfer takes place, gets the SOAP response, then at the end :

HTTP/1.1 200 OK
Date: Sat, 11 Jul 2009 01:40:16 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Connection: close
MIME-Version: 1.0
Transfer-Encoding: chunked
Cache-Control: private

I quickly found out that IIS6.0, or the WCF handler is forcing the connection to close on this last request. That's not particularly unusual, since a server may explictly deny an HTTP client to keep alive the connection.

What's even more unusual is that out of luck by trying to deactivate the IIS6.0 keep alive setting on my web site, I noticed that all the connections were properly closed on the client...!

I tried analysing a bit deeper the dialog between the client and the server, and I noticed two differences :

The content of the final answer of the IIS contains two "Connection: close" headers, which could mean one by the WCF handler, and one by IIS itself. I'm not sure if repeating headers is forbidden in the RFC, I'd have to read it again to be sure.
It looks like the order of the FIN/ACK, ACK packets is a bit different, but I'm not sure either where that stands. Both the client and the server are sending FIN packets to the other side, probably the result of calling Socket.Close().

But then I found out something even stranger : It all works on IIS7 ! And the best of all, the KeepAlive status is honored by the web server. That obviously means that the global performance of the web service is better on IIS7 than it is on IIS6, since there is only one connection opened for all my 15000 calls, which is rather good. Too bad my client cannot switch to IIS7 for now...

It also seems that the WCF client is not behaving the same way it does with IIS6, because at the TCP level only the client is sending a TCP FIN packet and the server is not when the keep alive is disabled.

I think I'll be posting this on Microsoft Connect soon, but I'm not sure where the problem lies, whether it is in IIS6, the WCF client or the WCF server handler, but there is definitely an issue here.

Working with Bill Graziano's ClearTrace to optimize SQL queries

Jay — Wed, 27 May 2009 19:37:00 -0700

Cet article est disponible en francais.

After listening to the RunAs Radio show #103 with Bill Graziano, I decided to give a try to his tool, ClearTrace, an SQL trace analysis tool.

It turns out that I’m in an SQL optimization spree recently, and the project I’m working on had an complete sequence of operations that took more than 24 hours to complete. Analyzing traces with the SQL profiler can be time consuming - needle in a haystack consuming - particularly when the log is over 7GB in size, in that case.

Finding small queries that are executed thousands of times is rather hard to track, and finding proper candidates for optimization is a bit complex. You don't want to spend time optimizing a query that has a small impact.

This is where Bill’s tool come into play. Give it a trace file, the tool analyses it and gives you aggregate information about what takes the most CPU/Duration/Read/Write. Pick your victim.

After a few hours and a few runs of ClearTrace to find which stored procedure needed rework, I found a bunch of store procedure that were executed thousands of time and that were using a lot of cumulative I/O. After optimizing these procedures, the whole process that took more than 24 hours is now down to about 7 hours.

Nothing magic here, the key is to find what to optimize on long running processes executing millions of queries. Bill’s tool does that perfectly !

On a side note, at first ClearTrace threw an out of memory exception after trying to import my big trace file. Turns out that after exporting it with Reflector and debugging the code, I spotted a small refactoring issue that Bill fixed very quickly. Thanks Bill !

As Carl Franklin says in .NET Rocks' “Better Know a Framework”, learn it, use it, love it !

A C# Traverse extension method, with an F# detour

Jay — Sun, 17 May 2009 09:10:00 -0700

Cet article est disponible en Français.

The Traverse extension method in C#

Occasionally, you'll come across data structures that take the form of single linked lists, like for instance the MethodInfo class and its GetBaseDefinition method.

Let's say for a virtual method you want, for a specific type, discover which overriden method in the hierarchy is marked with a specific attribute. I assume in this example that the expected attribute is not inheritable.

You could implement it like this :

    private static MethodInfo GetTaggedMethod(MethodInfo info)
    {
        MethodInfo ret = null;

        do
        {
            var attr = info.GetCustomAttributes(typeof(MyAttribute), false) as MyAttribute[];

            if (attr.Length != 0)
                return info;

            ret = info;

            info = info.GetBaseDefinition();
        }
        while (ret != info);

        return null;
    }

This method has two states variables and a loop, which makes it a bit harder to stabilize. This is a method that could easily be expressed as a LINQ query, but (as far as I know) there is no way to make a enumeration of a data structure which is part of a linked list.

To be able to do this, which is "traverse" a list of objects of the same type that are linked from one to the next, an extension method containing a generic iterator can be written like this :

    public static class Extensions
    {
        public static IEnumerable<T> Traverse<T>(this T source, Func<T, T> next)
        {
            while (source != null)
            {
                yield return source;
                source = next(source);
            }
        }
    }

This is a really simple iterator method, which calls a method to get the next element using the current element and stops if the next value is null.

It can be used easily like this, using the GetBaseDefinition example :

var methodInfo = typeof(Dummy).GetMethod("Foo");

IEnumerable<MethodInfo> methods = methodInfo.Traverse(m => m != m.GetBaseDefinition() ? m.GetBaseDefinition() : null);

Just to be precise, the lambda is not exactly perfect, as it is calling GetBaseDefinition twice. It can definitely be optimised a bit.

Anyway, to go back at the first example, the GetTaggedMethod function can be written as a single LINQ query, using the Traverse extension :

    private static MethodInfo GetTaggedMethod(MethodInfo info)
    {
        var methods = from m in methodInfo.Traverse(m => m != m.GetBaseDefinition() ? m.GetBaseDefinition() : null)
                      let attributes = m.GetCustomAttributes(typeof(MyAttribute), false)
                      where attributes.Length != 0
                      select m;

        return methods.FirstOrDefault();
    }

I, for one, find this code more readable... But this is a question of taste :)

Nonetheless, the MethodInfo linked-list is not the perfect example, because the end of the chain is not a null reference but rather the same method we're testing. Most of the time, a chain will end with a null, which is why the Traverse method uses null to end the enumeration. I've been using this method to perform queries on a hierarchy of objects that have parent objects of the same type, and the parent's root set to null. It has proven to be quite useful and concise when used in a LINQ query.

An F# Detour

As I was here, I also tried to find out what an F# version of this code would be. So, with the help of recursive functions, I came up with this :

    let rec traverse(m, n) =
       let next = n(m)
       if next = null then
         [m]
       else
       [m] @ traverse(next, n)

The interesting part here is that F# does not require to specify any type. "m" is actually an object, and "n" a (obj -> obj) function, but returns a list of objects. And it's used like this :

    let testMethod = typeof<Dummy>.GetMethod("Foo")

    for m in traverse(testMethod, fun x -> if x = x.GetBaseDefinition() then null else x.GetBaseDefinition()) do
       Printf.printfn "%s.%s" m.DeclaringType.Name m.Name

Actually, the F# traverse method is not exactly like the C# traverse method, because it is not an extension method, and it is not lazily evaluated. It is also a bit more verbose, mainly because I did not find an equivalent of the ternary operator "?:".

After digging a bit in the F# language spec, I found out it exists an somehow equivalent to the yield keyword. It is used like this :

    let rec traverse(m, n) =
       seq {
       let next = n(m)
       if next = null then
           yield m
       else
           yield m
           yield! traverse(next, n)
   }

It is used the same way, but the return value is not a list anymore but a sequence.

I also find interesting that F# is able to return tuples out of the box, and for my attribute lookup, I'd have the method and I'll also have the attribute instance that has been found. Umbrella also defines tuples useable from C#, but it's an addon.

F# is getting more and more interesting as I dig into its features and capabilities...

Hyper-V Virtual Machine Mover 1.0.2.0

Jay — Sun, 10 May 2009 15:20:00 -0700

Thanks to the guys at Lakewood Communications, I've updated my tool to move Hyper-V Virtual Machines to allow VMs that do not have any snapshots.

Version 1.0.2.0 also has a minor fix to try guessing the original VM path to replace it correctly in the configuration file. That would mean that you could have detached a VM and could not have attached it back.

On the Win2008 R2 front now the RC is out, for the time being since I don't have any spare hardware to test it on, I don't know if it works or if it still needed. If you do have tested my tool on this OS, please let me know.

Download the latest version here.

SharePoint : The database connection string is not available. (0xc0041228)

Jay — Tue, 31 Mar 2009 20:21:00 -0700

A Sharepoint Services 3.0 setup I'm managing had a few issues lately, and I had to bring back up an old version of the system. The original setup had a Search Server Express 2008 installed, and the backup I restored did not, even though I had the databases for it.

After reinstalling everything that was needed, and having the Search Server properly indexing content, I kept having a lot of messages like "The database connection string is not available." in the event log, and "Your search cannot be completed because of a service error." in the search tool in any Sharepoint site. I had the content database properly associated with the correct indexer.

I did not notice at first that the service named "Windows SharePoint Services Search" was not started, and when I tried to start it, I had a nice "The handle is invalid." error message... Not very helpful.

A few posts around the web were suggesting to stop that service, then restart it. One suggested to check the user account of that service, which was "Network Service" for me. I changed it to the same domain account that the "Windows SharePoint Services Timer" service is using. At this point, the service was starting properly, but I was still having the "The database connection string is not available." message.

In the "Services on Server", I tried stopping the "Windows SharePoint Services Search" service, (telling me that it was going to delete the index content), which succeeded. But trying to restart the service gave me an error saying that the database already had content, and that I had to create a new one.

I did create a new database, but the service would still not start, this time giving an other error message that I enjoy so much : "An unknown problem occured".

I went back to some forum posts, and I came across a command to "initialize" the service from the command line with STSADM :

stsadm -o spsearch -action start -farmserviceaccount [domain_account] -farmservicepassword [domain_account_password]

Which at first gave me this :

The specified database has an incorrect collation. Rebuild the database with the Latin1_General_CI_AS_KS_WS collation or create a new database.

I did re-create the database with the proper collation, then ran stsadm again and it gave me this :

The Windows SharePoint Services Search service was successfully started.

Hurray ! That did the trick, and indeed, my searches in any Sharepoint sites were not returing any error. I just had to wait for the service to refresh its index, and my search was running again !

This is a long and verbose post, but I hope this will help someone with this cryptic message...

Hyper-V Virtual Machine Mover and Hyper-V Server

Jay — Sat, 28 Mar 2009 14:47:00 -0700

I've had some hardware trouble lately, with hard-drives failing with some VMs on these, and my Hyper-V Mover tool hase saved me a great deal of time.

I've had some time to improve it and this time, it is possible to attach and detach VMs from remote machines, and particularly those that are on Hyper-V Server machines.

I've created a page for this tool, the Hyper-V Virtual Machine Mover, version 1.0.1.0.

Still no sources available but they'll be available on CodePlex soon.

Google Transit and Montreal's STM

Jay — Sun, 15 Mar 2009 21:53:00 -0700

A while ago, the Montréal's STM transit system announced that they were now supported by Google Transit.

While it is possible to trace proper routes, Google's having the same problem as I do, which is that the STM is updating schedules per trimester. And since it's the STM that is providing the data and that it's not been updated since the 1st of January 2009, schedules have been incorrect ever since.

To be perfectly fair, I did not update the schedules in my application since that time too by lack of time to create a proper update procedure, but I'm not paid for that either...

Now that I've given it some thoughts, I'm now streamlining the schedule updates stops after stops as long as they are out of date. Previously, I updated the database all at once, but this does not scale... Now the updates are progressive, which is far more manageable for me.

Anyway, now there may be a simple message saying that the displayed schedule is outdated, which is better than trusting the time and blaming the STM for no reason :)

A tool to move an Hyper-V Virtual Machine without exporting it

Jay — Fri, 20 Feb 2009 20:26:00 -0700

Cet article est disponible en francais.

Hyper-V is a wonderful tool, and provides great performance and stability. But on the administration side, available tools are a bit scare, and even though most general operations are available, some are a bit hard to use. One can guess that this will improve in Windows Server 2008 R2.

But for now, the administration tools do not provide any mean to import a VM that has not been previously exported. Exporting a VM can only be done on the original host server while it is still running. In the case of a crashed server, exporting a VM becomes a bit more complex.

Some techniques do exist, here and there, that explain by means of mklink and icacls, how to recreate symbolic links and file permissions for the VM configuration files. But that stays a particularly complex task, mostly because all files must be modified, and a specific order must be respected for all modifications. And this is especially true for the case of a running host server.

After having dug in Hyper-V symlinks and WMI interface, I created a GUI tool that allows to attach and detach VM that have not been previously epoxted.

Some thougts on this tool :

A VM can only be detached if it is in the "Saved" or "Stopped" state.
It is not necessary to stop the Hyper-V service and all modifications are detected live by the service.
A VM can only be imported if it contains at least on HDD on the IDE 0 controller.
All the VM files must be under the same directory, HDD and snapshots.
All files that are modified are backed-up next to the original files; All other files are not modified nor moved.
.NET 3.5 must be installed.

I'll provide the sources for this tool in a near future, as well as a console version.

Of course, there will be bugs, and do not hesitate to report them to me. I may not be able to do anything about it because it is a tool that performs a operation that is (I assume) not supported by Microsoft.

You can download the tool here.

Using Multiple Where Clauses in a LINQ Query

Jay — Sat, 06 Dec 2008 15:32:00 -0700

Cet article est disponible en francais.

After writing my previous article where I needed to intercept exceptions in a LINQ Query, I found out that it is possible to specify multiple where clauses in a LINQ Query.

Here is the query :

var q = from file in Directory.GetFiles(@"C:\Windows\Microsoft.NET\Framework\v2.0.50727", "*.dll")
        let asm = file.TryWith(f => Assembly.LoadFile(f))
        where asm != null
        let types = asm.TryWith(a => a.GetTypes(), (Exception e) => new Type[0])
        where types.Any()
        select new { asm, types };

This query is able to find assemblies for which it is possible to list types. The point of using multiple Where clauses is to avoir evaluating chunks of a query if previous chunks can prevent it. By the way, the TryWith around the Assembly.GetTypes() is there to intercept exceptions raised when loading types, in case dependencies would not be available at the moment of the enumeration.

A useful LINQ trick to remember !

F#, TryWith, Maybe and Umbrella

Jay — Sat, 06 Dec 2008 13:49:00 -0700

Cet article est disponible en Français.

I've thrown myself a bit in the discovery of F#, and even though I do not intend to make it my first language, I intend to use techniques and features found in it and try to port them into C#. New additions in C# 3.0 make it a good target for functional concepts.

There seem to be a consensus for the fact that F# is not a multi-purpose language, as C# is also not, for instance with the writing of parallel code. C# is not a perfect language for this, but F# seems to be. At the opposite, F# does not seem to be a language of choice for writing GUI code. For my part, and considering that F# if not really official, reusing concepts will be enough for now.

TryWith Extension

Using F#, I had to write this:

   let AllValidAssemblies = [
        for file in Directory.GetFiles(@"C:\Windows\Microsoft.NET\Framework\v2.0.50727", "*.dll") ->
        System.Reflection.Assembly.LoadFile(file)
    ]

This code creates a list of assemblies that can be loaded in the current AppDomain. There is however an issue with the invocation of the Assembly.LoadFile method, because it raises an exception when the file is not loadable for some reason. This is a non-modifiable behavior, even though we would like to return a null instead of an exception.

To work around this, there is a feature in F# that can do this :

    let EnumAllTypes = [
        for file in Directory.GetFiles(@"C:\Windows\Microsoft.NET\Framework\v2.0.50727", "*.dll") ->
        try System.Reflection.Assembly.LoadFile(file) with _ -> null
    ]

The point of the try/with block is to transform any exception into a null reference.

To transpose the creation of this list in C# with a LINQ query, the same problem arises. We must intercept the exception raised by LoadFile and convert it to a null reference.

Here is the equivalent in C#, without the exception handling :

    var q = from file in Directory.GetFiles(@"C:\Windows\Microsoft.NET\Framework\v2.0.50727", "*.dll")
            let asm = Assembly.LoadFile(file)
            select asm;

When LoadFile raises an exception, the execution of the request is interrupted, which is a problem.

        {
                                        }



                        {
                                        }



Methods can be of a great value here, and even though a normal method could do the trick, we can write this :
public static class Extensions
    public static TResult TryWith<TInstance, TResult>(this TInstance instance, Func<TInstance, TResult> action)
    {
       try {
          return action(instance);
       }
       catch {
          return default(TResult);
       }
    }
ms,geneva" size="2"> behind this method is to reproduce the behavior of the try/with F# construct. With this method, we can update the LINQ query into this :
var q = from file in Directory.GetFiles(@"C:\Windows\Microsoft.NET\Framework\v2.0.50727", "*.dll")
         let asm = file.TryWith(f => Assembly.LoadFile(f))
         select asm;
ms,geneva" size="2">This is creates the same list the F# code does, with null references for assemblies that could not be loaded.
method can be overloaded to be a bit more flexible, like calling a method for a specific exception : public static TResult TryWith<TInstance, TResult, TException>(
       this TInstance instance, Func<TInstance, TResult> action,
       Func<TException, TResult> exceptionHandler
    )
    where TException : Exception
    try {
       return action(instance);
    }
    catch (TException e) {
       return exceptionHandler(e);
    }
    catch {
       return default(TResult);
    }
ms,geneva" size="2">By the way, there is an interesting bug with this code. If we execute this :
string value = null;
var res = value.TryWith(s => s.ToString(), (Exception e) => "Exception");
ms,geneva" size="2">The behavior is different depending on whether it is executed with or without the debugger with the x86 runtime. It seems that the code generator "forgets" to add the handler for the TException typed exception, which is annoying. This is not a big bug, mainly because it only appears when the x86 debugger is present. With the x64 runtime debugger, there is no problem though. For those interesting in this, the bug is on Connect.

Maybe Extension

I also added recently in Umbrella an extension named Maybe, which has a behavior rather similar to TryWith, but without the exceptions :

    public static TResult Maybe<TResult, TInstance>(
         this TInstance instance,
         Func<TInstance, TResult> function)
    {
       return instance == null ? default(TResult) : function(instance);
    }

The point of this method is to be able to execute code if the original value is not null. For instance :

object instance = null;
Console.WriteLine("{0}", instance.Maybe(o => o.GetType());

This allows the evaluation of the GetType call, only if "instance" is not null. With a method call like this, it is possible to write an "if" block, with inside a LINQ query, this because a bit more complex.

The idea for the code is not new and is similar to the functional Monad concept. It has been covered numerous times, and an implementation more in line with F# can be found on Matthew Podwysocki's Blog.

Pollution ?

When we're talking about pollution linked to Extension Methods, we're talking about Intellisense pollution. We can quickly find ourselves dealing with a bunch of extensions that are useful in the context of the current code, which renders Intellisense unusable. With Umbrella, these two extensions are somehow polluting all types, because they are generic without constraints.

Although these are only two very generic extensions, this can apply to almost any block of code, but they could find themselves better placed in an Umbrella Extension Point, rather than directly on every types.

We could have this :

object instance = null;
Console.WriteLine("{0}", instance.Control().Maybe(o => o.GetType());

But I have some issues with this approach : To be able to create an extension point the Control methd has to create an instance of the IExtensionPoint. This add a new instantiation during the execution, although the lambda also creates itself a hidden instance, we're counting anymore... There is also the fact that it lengthens the code line, but is only aesthetics. We'll see what pops out ...

Anyway, it is interesting to see the impact the learning a new language has on the style of writing code with another language that one's been using for a long time...

Local Variables in Lambda Expressions

Jay — Thu, 20 Nov 2008 19:58:00 -0700

Cet article est disponible en français.

After a quick chat with Eric Lippert about a post on the use in lambda expressions of a local variable declared in a foreach loop, Eric pointed me out that this piece of code :

    int a = 0;
    Action action = () => Console.WriteLine(a);
    action();

Is actually not expanded by the compiler to this code :

    [CompilerGenerated]
    private sealed class <>c__DisplayClass1
    {
       public int a;

       public void <Main>b__0()
       {
          Console.WriteLine(this.a);
       }
    }

    void Main()
    {
     int a = 0;

       var display = new <>c__DisplayClass1();
       display.a = a;

       var action = new Action(display.<Main>b__0);

   action();
    }

I made the assumption that a local variable was simply copied in the "DisplayClass" if it is not used after the creation of the lambda, which is not the case.

If we take this slightly different sample :

    int a = 0;
    Action action = () => Console.WriteLine(a);
    a = 42;
    action();

My assumption would have made this code display "0". This is correct because lambda expressions (and anonymous methods) "capture" the variable and not the value; The execution must display 42.

Actually, this latter piece of code is expanded like this:

    var display = new <>c__DisplayClass1();

    display.a = 0;

    var action = new Action(display.<Main>b__0);

    display.a = 42;

    action();

We can see that in fact, the variable that was previously local, on the stack, has been "promoted" as a field memberr of the "Display Class". This means that all references to this "local" variable, inside or outside of the lambda, are replaced to point to the current instanc e of the "DisplayClass".

This is quite simple actually, but we can feel that the "magic" behind the C# 3.0 syntactic sugar is the result of a lot of thinking !

I will end this post by a big thanks to Eric Lippert, who took the time to answer me, even though he's probably under heavy load with the developement of C# 4.0. (With the contravariance of generics, yey !)

Lambda Expression and ForEach loops

Jay — Tue, 18 Nov 2008 20:58:00 -0700

Cet article est disponible en français.

To enhance the performances of a type serializer, and to use a small extension that I recently wrote for Umbrella I stumbled upon an interesting small "Side Effect" seen when creating lambda expressions inside a foreach loop.

Let's take this simple piece of code :

    var actionList = new List<Func<int>>();

    foreach (var value in Enumerable.Range(0, 10))
    {
       actionList.Add(() => value);
    }

    actionList.ForEach(func => Console.Write("{0} ", func()));

Which outputs this :

9 9 9 9 9 9 9 9 9 9

Which is, of course, what we could have expected.

Lambda expression have the ability to use variables that are in the scope when they are declared. This makes them very interesting, but to properly use them, it is best to understand how they are "materialized" by the compiler.

Like a lot of features of C#, like the using, foreach, iterators or lock, lambdas are syntactic sugar destined to simplify the writing of code that is most of the time pretty verbose. It is possible to write the expanded code for these keyswords in C#.

Let's take this other piece of code :

int a = 0;
Action action = () => Console.WriteLine(a);
action();

The lambda expression is "materialized" by the C# compiler under the form of a "Display Class", that allows the storage of the local variable "a" :

    [CompilerGenerated]
    private sealed class <>c__DisplayClass1
    {
       public int a;

       public void <Main>b__0()
       {
          Console.WriteLine(this.a);
       }
    }

We can see that the indentifiers for the generated class are not valid in C#, but are valide from the CLR point of view. We can also see that the local variable used during the declaration is present as a member variable, in the class that contains the code of the lambda expression. The compiler will then write this to create an instance of the lambda expression :

    int a = 0;

    var display = new <>c__DisplayClass1();
    display.a = a;

    Action action = new Action(display.<Main>b__0);

    action();

There also, this is not valid C#.

But then, what happens for the foreach case so that the content of the variable is repeated ?

If we analyze the first code sample generated by the compiler with Reflector, there is nothing much fancy to see with the C# visualizer :

    List<Func<int>> actionList = new List<Func<int>>();
    using (IEnumerator<int> CS$5$0000 = Enumerable.Range(0, 10).GetEnumerator())
    {
       while (CS$5$0000.MoveNext())
       {
          int value = CS$5$0000.Current;
          actionList.Add(delegate {
             return value;
          });
       }
    }

The lambda expression is represented as an anonymous method, which is a synonym of lambda, but that does not explain the behavior.

We must look at the generated IL to understand the behavior, and this is the correct C# code that is generated :

    List<Func<int>> actionList = new List<Func<int>>();
    using (IEnumerator<int> CS$5$0000 = Enumerable.Range(0, 10).GetEnumerator())
    {
       var myLambda = new <>c__DisplayClass4();

       while (CS$5$0000.MoveNext())
       {
          int value = CS$5$0000.Current;

          myLambda.value = value;

          actionList.Add(new Func<int>(myLambda.b_0));
       }
    }

We can easily see what the problem is : The instance of the class containing the lambda is created only once, and reused many times to assign a new value for each iteration. This explains why the execution of all the lambdas return the last enumerated value, because they all refer to the same instance of the "DisplayClass" type.

However, if we write the code this way :

    foreach (var value in Enumerable.Range(0, 10))
    {
       int myValue = value;
       actionList.Add(() => myValue);
    }

The behavior changes, and this time, each lambda has the correct value.

From the compiler point of view, the creation of a new instance for the container class must be the consequence of the creation of a new variable. For the case of a ForEach statement, this is not the case and the variable is treated as created only once, then reused..

So, from the compiler point of view, a ForEach loop is expanded like this :

    using (var it = Enumerable.Range(0, 10).GetEnumerator())
    {
       int value;

       while (it.MoveNext())
       {
          value = it.Current;
          actionList2.Add(() => value);
       }
    }

This is probably a question of interpretation, but I wasn't exactly expecting this...

So, one must pay attention to the way local variables are used in lambda expressions, depending on their declaration location.

I'll explain in a later post why I did have to use lamda expressions in a ForEach loop.

Working with Umbrella in .NET 3.5

Jay — Sat, 08 Nov 2008 16:06:00 -0700

If you've been using .NET 3.5, and the new features that are provided by C# 3.0, and especially LINQ, you must have wondered why, oh why, there isn't an extension named ForEach on an IEnumerable<T>.

Well, I still haven't figured that out, though it appears that it might have something to do with the fact that by nature an Action<T> is most of the time not "pure", which means that it modifies some states, somewhere. I can't remember where I've found that explanation, though it "might" make sense from a functional point of view.

But you may wonder why, then, is there a ForEach method on List<T> ? Well, I don't know why there is that kind of consistency issue, but I know there is one library that somehow tries to fix this :

Umbrella

This library fills the gaps left by the BCL, and adds a whole bunch of new extension methods that eases the use of .NET 3.5 classes. This library is not a framework though, mainly because you don't need to redesign your whole application to use it.

A few weeks ago during a presentation of Umbrella at the .NET User Group of Montreal, the creators Francois Tanguay and Erik Renaud from nVentive, were asked the question "Where do we start ?". This is a tough question to answer, because of the nature of umbrella, which "plugs" itself everywhere it can.

Using Umbrella

Let’s see a simple example:

    Enumerable
        .Range(20, 30)
        .ForEach(
            (index, value) => "Value #{0}={1}".InvariantCultureFormat(index, value)
        );

As you can see, using Umbrella requires most of the time the use of lambda expressions. Here, this example is an answer to the question "How can I get the index of the enumerated value in a foreach statement ?".

Another extension provided by Umbrella is the Remove method on an ICollection by specifying a predicated to select elements to be removed.

var values = new List<int>(Enumerable.Range(1, 10));
values.Remove(v => v > 4);

Sure, you can write some code that does this, but you'll have to create a temporary list to store elements to be removed, then remove them from the collection.

There is also an extension on IDictionary<> that simplifies the use of TryGetValue which requires an “out” argument, which is particularly annoying to write. Umbrella provides a TryGetValueOrDefault that either gives you the value for the key you requested, or default(TValue) if the key is not found.

The Action.ToDisposable() extension method

One particularly interesting extension is the ability to encapsulate an Action delegate into an anonymous disposable instance. Let’s say that we need to profile the duration of a particular scope of code. We would need to write something like this :

    var w = Stopwatch.StartNew();
    for (int i = 0; i < 1000; i++) { }
    Console.WriteLine(w.Elapsed);

We would have to write the enclosing code to time for each portion of code we would need to profile. You could of course write a class that would do this on purpose :

    class ProfileScope : IDisposable
    {
        Stopwatch w = Stopwatch.StartNew();
        void IDisposable.Dispose()
        {
            Console.WriteLine(w.Elapsed);
        }
    }

And it’s being used like with a “using” statement :

    using (new ProfileScope())
    {
        for (int i = 0; i < 1000; i++)
        {
        }
    }

This time, timing a block of code is somehow easier to write, however, there is a way to simplify the writing of the writing of the utility class by doing this:

    static IDisposable CreateProfileScope()
    {
        var w = Stopwatch.StartNew();
        return Actions.Create(() => Console.WriteLine(w.Elapsed)).ToDisposable();
    }

We can use here the ToDisposable extension to be able to call the specified action in the dispose method of some disposable class.

The point of this code is to avoid exposing the inner details of the profiling code, and just expose a known and generic way for executing something at the end of a scope.

Extension Points

There is one drawback when using extension methods: They tend to “pollute” the original type. By pollute I mean that they appear in the IntelliSense window. Because it is easy to add extension methods, you end up having hundreds of new methods on a type, which make IntelliSense much less useable.

The Umbrella guys came up with the idea of Extension Points, which are a way to group extension methods the namespaces do for types.

This is somehow a hack from a code perspective, but is rather ingenious from a usability perspective. For instance, extending the serialization applies to every type that exists, and placing all serialization extension methods directly on System.Object is not a good idea.

Extensions points are used like this:

    using (MemoryStream stream = new MemoryStream())
    {
        int a = 0;
        a.Serialization().Binary(stream);
        // ...
    }

Serialization is an extension method placed on every object that returns an SerializationExtensionPoint instance, and the “Binary” method extends that type instead of System.Object.

It is important to remember that Umbrella is for most of its code is not "rocket science", as it probably contains code that you may already have partially developed for your own project. It's only the amount of small but useful utility methods that Umbrella provides that makes it worth using.

I’ll be writing a few other posts about Umbrella; there are a lot of extensions in there that are great time savers.

Speaker Idol Montreal

Jay — Tue, 04 Nov 2008 06:56:00 -0700

Yesterday, Monday 3rd, was the day of Speaker Idol Montreal.

I'd like to thank every attendee for being here and listening all of these great presentations ! Ten minutes is a format that's a bit rough though, one's got to learn every word to be said to make a good point during that kind of presentation length. (And don't rely on the computer that's available there... You might end up not being able to use the keyboard :) )

Anyway, if you were interested by the subject of my presentation, you may find a more thorough explanation on this blog post about DataBinding in WPF. By the way, you may also be interested in this post about DataBinding Application Settings.

See you next time in the .NET Montreal User Group !

Montréal Bus Stop Locator Site and the iPhone

Jay — Sat, 01 Nov 2008 11:13:00 -0700

Cet article est disponible en francais ici.

I've updated my utility WebSite to look a bit better on the iPhone (Thanks, Mr Sanx).

Now, the next buses schedule are a bit easier to read and past schedules are visually easier to leave out.

There are some other features that are in the oven, I hope to push them online soon enough :)

The geolocation on the iPhone would be a plus, but unfortunately, Google Gears is not available on the iPhone. There seem to be some sort of extension named weblocate:// to do this from a web application, but that seems a bit shady...

Anyway, you can visit the Web Site here from your Windows Mobile, iPhone, or desktop computer at this address :

http://jaylee.org/stm

Here's an example of what it's looking now :