Jon Skeet: Coding Blog

Book Review: Async in C# 5.0

skeet — Mon, 20 May 2013 23:14:02 GMT

Resources:

Amazon, Barnes and Noble, Play Books
The book's web site (O'Reilly) – downloads, errata etc

A while ago I was attending one of the Developer, Developer, Developer conference in Reading, and I heard Alex Davies give a talk about actors and async. He mentioned that he was in the process of writing a short book for O'Reilly about async in C# 5, and I offered to review it for him. Many months later (sorry Alex!) I'm finally getting round to it.

Disclaimer: The review copy was given to me for free, and equally the book is arguably a competitor of the upcoming 3rd edition of C# in Depth from the view of readers who already own the 2nd edition... so you could say I'm biased in both directions. Hopefully they cancel out.

This is a book purely on async. It's not a general C# book, and it doesn't even cover the tiny non-async features in C# 5. It's all about asynchrony. As you'd expect, it's therefore pretty short (92 pages) and can comfortably be consumed in a single session. Alex's writing style is informal and easy to read. Of course the topic of the book is anything but simple, so even though you may read the whole book in one go first time, that doesn't mean you're likely to fully internalize it straight away. The book is divided into 15 short chapters, so you can revisit specific areas as and when you need to.

Aside

I've been writing and speaking about async for about two and a half years now. I've tried various ways of explaining it, and I'm pretty sure it's one of those awkward concepts which really just needs to click eventually. I've had some mails from people for whom my explanation was the one to do the trick... and other mails from folks who only "got it" after seeing another perspective. I'd encourage anyone learning about async to read a variety of books, articles, blog posts and so on. I don't even think it's a matter of finding the single "right" explanation for you – it's a matter of letting them all percolate.

The book covers all the topics you'd expect it to:

Why asynchrony is important
Drawbacks of library-only approaches
How async/await behaves in general
Threading and synchronization contexts
Exceptions
Different code contexts (ASP.NET, WinRT, regular UI apps)
How async code is compiled

Additionally there are brief sections on unit testing, parallelism and actors. Personally I'd have preferred the actors part to be omitted, with more discussion on the testing side – particularly in terms of how to write deterministic asynchronous tests. However, I know that Alex is a big fan of actors, so I can forgive a little self-indulgence on that front.

There's one area where I'm not sure I agree with the advice in the book: exceptions. Alex repeatedly gives the advice that you shouldn't let exceptions go unobserved. I used to go along with that almost without thinking – but now I'm not so sure. There are definitely cases where that definitely is the case, but I'm not as comfortable with the global advice as I used to be. I'll try to put my thoughts in order on this front and blog about this separately at a later date.

That aside, this is a good, pragmatic book. To be honest, I suspect no book on async is going to go into quite as many details as the PFX team blog, and that's probably a good thing. But "Async in C# 5.0" is a very good starting point for anyone wanting to get to grips with async, and I in no way begrudge any potential C# in Depth 3rd edition sales I may lose by saying so ;)

New tool to play with: SemanticMerge

skeet — Thu, 18 Apr 2013 21:38:23 GMT

A little while ago I was contacted about a new merge tool from the company behind PlasticSCM. (I haven't used Plastic myself, but I'd heard of it.) My initial reaction was that I wasn't interested in anything which required me to learn yet another source control system, but SemanticMerge is independent of PlasticSCM.

My interested was piqued when I learned that SemanticMerge is actually built on Roslyn. I don't generally care much about implementation details, but I'm looking out for uses of Roslyn outside Microsoft, partly to see if I can gain any inspiration for using it myself. Between the name and the implementation detail, it should be fairly obvious that this is a tool which understands changes in C# source code rather better than a plain text diff.

I've had SemanticMerge installed on one of my laptops for a month or so now. Unfortunately it's getting pretty light use – my main coding outside work is on Noda Time, and as I perform the vast majority of the commits, the only time I really need to perform merges is when I've forgotten to push a commit from one machine before switching to another. I've used it for diffs though, and it seems to be doing the right thing there – showing which members have been added, moved, changed etc.

I don't believe it can currently support multi-file changes – for example, spotting that a lot of changes are all due to a rename operation – but even if it doesn't right now, I suspect that may be worked on in the future. And of course, the merge functionality is the main point.

SemanticMerge is now in beta, so pop over to the web site and give it a try.

The Open-Closed Principle, in review

skeet — Fri, 15 Mar 2013 09:22:01 GMT

Background

I've been to a few talks on SOLID before. Most of the principles seem pretty reasonable to me – but I've never "got" the open-closed principle (OCP from here on). At CodeMash this year, I mentioned this to the wonderful Cori Drew, who said that she'd been at a user group talk where she felt it was explained well. She mailed me a link to the user group video, which I finally managed to get round to watching last week. (The OCP part is at around 1 hour 20.)

Unfortunately I still wasn't satisfied, so I thought I'd try to hit up the relevant literature. Obviously there are umpteen guides to OCP, but I decided to start with Wikipedia, and go from there. I mentioned my continuing disappointment on Twitter, and the conversation got lively. Uncle Bob Martin (one of the two "canonical sources" for OCP) wrote a follow-up blog post, and I decided it would be worth writing one of my own, too, which you're now reading.

I should say up-front that in some senses this blog post isn't so much about the details of the open-closed principle, as about the importance of careful choice of terminology at all levels. As we'll see later, when it comes to the "true" meaning of OCP, I'm pretty much with Uncle Bob: it's motherhood and apple pie. But I believe that meaning is much more clearly stated in various other principles, and that OCP as the expression of an idea is doing more harm than good.

Reading material

So what is it? (Part 1 – high level)

This is where it gets interesting. You see, there appear to be several different interpretation of the principle – some only subtly distinct, others seemingly almost unrelated. Even without looking anything up, I knew an expanded version of the name:

Modules should be open for extension and closed for modification.

The version quoted in Wikipedia and in Uncle Bob's paper actually uses "Software entities (classes, modules, functions, etc.)" instead of modules, but I'm not sure that really helps. Now I'm not naïve enough to expect everything in a principle to be clear just from the title, but I do expect some light to be shed. In this case, unfortunately I'm none the wiser. "Open" and "closed" sound permissive and restrictive respectively, but without very concrete ideas about what "extension" and "modification" mean, it's hard to tell much more.

Fair enough – so we read on to the next level. Unfortunately I don't have Bertrand Meyer's "Object-Oriented Software Construction" book (which I take to be the original), but Uncle Bob's paper is freely available. Wikipedia's summary of Meyer's version is:

The idea was that once completed, the implementation of a class could only be modified to correct errors; new or changed features would require that a different class be created. That class could reuse coding from the original class through inheritance. The derived subclass might or might not have the same interface as the original class.

Meyer's definition advocates implementation inheritance. Implementation can be reused through inheritance but interface specifications need not be. The existing implementation is closed to modifications, and new implementations need not implement the existing interface.

And Uncle Bob's high level description is:

Modules that conform to the open-closed principle have two primary attributes.

They are "Open For Extension". This means that the behavior of the module can be extended. That we can make the module behave in new and different ways as the requirements of the application change, or to meet the needs of new applications.

They are "Closed for Modiﬁcation". The source code of such a module is inviolate. No one is allowed to make source code changes to it.

I immediately took a dislike to both of these descriptions. Both of them specifically say that the source code can't be changed, and the description of Meyer's approach to "make a change by extending a class" feels like a ghastly abuse of inheritance to me... and goes firmly against my (continued) belief in Josh Bloch's advice of "design for inheritance or prohibit it" – where in the majority of cases, designing a class for inheritance involves an awful lot of work for little gain. Designing an interface (or pure abstract class) still involves work, but with fewer restrictions and risks.

Craig Larman's article uses the term "closed" in a much more reasonable way, to my mind:

Also, the phrase "closed with respect to X" means that clients are not affected if X changes.

When I say "more reasonable way" I mean in terms of how I want to write code... not in terms of the use of the word "closed". This is simply not how the word "closed" is used elsewhere in my experience. In the rare cases where "closed" is used with "to", it's usually in terms of what's not allowed in: "This bar is closed to under 18s" for example. Indeed, that's how I read "closed to modification" and that appears to be backed up by the two quotes which say that once a class is complete, the source code cannot be changed.

Likewise the meaning of "open for extension" seems unusual to me. I'd argue that the intuitive meaning is "can be extended" – where the use of the term "extended" certainly nods towards inheritance, even if it's not intended meaning. If the idea is "we can make the module behave differently" – as Uncle Bob's description suggests – then "open for extension" is a very odd way of expressing that idea. I'd even argue that in the example given later, it's not the "open" module that behaves differently – it's the combination of the module and its collaborators, acting as a unified program, which behaves differently after some aspects are modified.

So what is it? (Part 2 – more detail)

Reading on through the rest of Uncle Bob's paper, the ideas become much more familiar. There's a reasonable example of a function which is asked to draw a collection of shapes: the bad code is aware of all the types of shape available, and handles each one separately. The good code uses an abstraction where each shape (Circle, Square) knows how to draw itself and inherits from a common base class (Shape). Great stuff... but what's that got to do with what was described above? How are the concepts of "open" and "closed" clarified?

The answer is that they're not. The word "open" doesn't occur anywhere in the rest of the text, other than as part of the term "open-closed principle" or as a label for "open client". While it's perhaps rather easier to see this in hindsight, I suspect that any time a section which is meant to clarify a concept doesn't use some of the key words used to describe the concept in a nutshell, that description should be treated as suspect.

The word "closed" appears more often, but only in terms of "closed against" which is never actually defined. (Is "closed against" the same as "closed for"?) Without Craig Larman's explanation, sentences like this make little sense to me:

The function DrawAllShapes does not conform to the open-closed principle because it cannot be closed against new kinds of shapes.

Even Craig's explanation feels somewhat at odds with Uncle Bob's usage, as it talks about clients being affected. This is another of the issues I have with the original two descriptions: they talk about a single module being open/closed, whereas we're dealing with abstractions where there are naturally at least two pieces of code involved (and usually three). Craig's description of changes in one module not affecting clients is describing a relationship – which is a far more useful way of approaching things. Even thinking about the shape example, I'm getting increasingly confused about exactly what's open and what's closed. It feels to me like it's neither the concrete shape classes nor the shape-drawing code which is open or closed – it's the interface between the two; the abstract Shape class. After all, these statements seem reasonable:

The Shape class is open for extension: there can be many different concrete subclasses, and code which only depends on the Shape class doesn't need to know about them in order to use them when they are presented as shapes.
The Shape class is closed for modification: no existing functions can be removed (as they may be relied on by existing clients) and no new pure virtual functions can be added (as they will not be implemented by existing subclasses).

It's still not how I'd choose to express it, but at least it feels like it makes sense in very concrete terms. It doesn't work well with how Uncle Bob uses the term "closed" though, so I still think I may be on a different page when it comes to that meaning. (Uncle Bob does also make the point that any significant program isn't going to adhere to the principle completely in every part of the code – but in order to judge where it's appropriate to be closed, I do really need to understand what being closed means.)

Just to make it crystal clear, other than the use of the word "closed," the low-level description of what's good and what's bad, and why, is absolutely fine. I really have no problems with it. As I said at the start, the idea being expressed makes perfect sense. It just doesn't work (for me) when expressed in the terms used at a higher level.

Protected Variation

By contrast, let's look at a closely related idea which I hadn't actually heard about before I started all this research: protected variation. This name was apparently coined by Alistair Cockburn, and Craig Larman either quotes or paraphrases this as:

Identify points of predicted variation and create a stable interface around them.

Now that's a description I can immediately identify with. Every single word of it makes sense to me, even without reading any more of Craig's article. (I have read the rest, obviously, and I'd encourage others to do so.) This goes back to Josh Bloch's "design for inheritance or prohibit it" motto: identifying points of predicted variation is hard, and it's necessary in order to create a stable interface which is neither too constrictive for implementations nor too woolly for clients. With class inheritance there's the additional concern of interactions within a class hierarchy when a virtual method is called.

So in Uncle Bob's Shape example, all there is is a point of predicted variation: how the shape is drawn. PV suggests the converse as well – that as well as points of predicted variation, there may be points which will not vary. That's inherent in the API to some extent – every shape must be capable of drawing itself with no further information (the Draw method has no parameters) but it could also be extended to non-virtual aspects. For example, we could decide that every shape has one (and only one) colour, which will not change during its lifetime. That can be implemented in the Shape class itself – with no predicted variation, there's no need of polymorphism.

Of course, the costs of incorrectly predicting variation can be high: if you predict more variation than is actually warranted, you waste effort on over-engineering. If you predict less variation than is required, you usually end up either having to change quite a lot of code (if it's all under your control) or having to come up with an "extended" interface. There's the other aspect of shirking responsibility on this predicted variation to some extent, by making some parts "optional" – that's like saying, "We know implementations will vary here in an incompatible way, but we're not going to try to deal with it in the API. Good luck!" This usually arises in collection APIs, around mutating operations which may or may not be valid (based on whether the collection is mutable or not).

Not only is PV easy to understand – it's easy to remember for its comedy value, at least if you're a fan of The Hitchhiker's Guide to the Galaxy. Remember Vroomfondel and Majikthise, the philosophers who invaded Cruxwan University just as Deep Thought was about to announce the answer to Life, the Universe, and Everything? Even though they were arguing with programmers, it sounds like they were actually the ones with software engineering experience:

"I'll tell you what the problem is mate," said Majikthise, "demarcation, that's the problem!"

[...]

"That's right!" shouted Vroomfondel, "we demand rigidly defined areas of doubt and uncertainty!"

That sounds like a pretty good alternative description of Protected Variation to me.

Conclusion

So, that's what I don't like about OCP. The name, and the broad description – both of which I believe to be unhelpful, and poorly understood. (While I've obviously considered the possibility that I'm the only one who finds it confusing, I've heard enough variation in the explanations of it to suggest that I'm really not the only one.)

That sounds like a triviality, but I think it's rather important. I suspect that OCP has been at least mentioned in passing in thousands if not tends of thousands of user groups and conferences. The purpose of such gatherings is largely for communication of ideas – and when a sound idea is poorly expressed, an opportunity is wasted. I suspect that any time Uncle Bob has personally presented it in detail, the idea has sunk in appropriately – possibly after some initial confusion about the terminology. But what about all the misinterpretations and "glancing blows" where OCP is only mentioned as a good thing that clearly everyone wants to adhere to, with no explanation beyond the obscure ones described in part one above? How many times did they shed more confusion than light?

I believe more people are familiar with Uncle Bob's work on OCP than Bertrand Meyer's. Further, I suspect that if Bertrand Meyer hadn't already introduced the name and brief description, Uncle Bob may well have come up with far more descriptive ones himself, and the world would have been a better place. Fortunately, we do have a better name and description for a concept which is at least very closely related. (I'm not going to claim PV and OCP are identical, but close enough for a lot of uses.)

Ultimately, words matter – particularly when it comes to single sentence descriptions which act as soundbytes; shorthand for communicating a complex idea. It's not about whether the more complex idea can be understood after carefully reading thorough explanations. It's about whether the shorthand conveys the essence of the idea in a clear way. On that front, I believe the open-closed principle fails – which is why I'd love to see it retired in favour of more accessible ones.

Note for new readers

I suspect this post may end up being read more widely than most of my blog entries. If you're going to leave a comment, please be aware that the CAPTCHA doesn't work on Chrome. I'm aware of this, but can't fix it myself. If you right-click on the broken image and select "open in new tab" you should get a working image. Apologies for the inconvenience.

C# in Depth 3rd edition available for early access, plus a discount code…

skeet — Sat, 16 Feb 2013 14:31:54 GMT

Readers who follow me on Twitter or Google+ know this already, but…

The third edition of C# in Depth is now available for early access from its page on the Manning website. I’ve been given a special discount code which expires at midnight EST on February 17th, so be quick if you want to use it – it gives 50% off either version. The code is “csharpsk”.

It’s likely that we’ll have a separate (permanent) discount for readers who already own the second edition, but the details of that haven’t been decided yet.

Just to be clear, the third edition is largely the second edition plus the changes to cover C# 5 – I haven’t done as much rewriting as I did for the second edition, mostly because I was already pretty happy with the second edition :) Obviously the largest (by far) feature in C# 5 is async/await, which is covered in detail in the new chapter 15.

Fun with Object and Collection Initializers

skeet — Thu, 14 Feb 2013 22:59:00 GMT

Gosh it feels like a long time since I’ve blogged – particularly since I’ve blogged anything really C#-language-related.

At some point I want to blog about my two CodeMash 2013 sessions (making the C# compiler/team cry, and learning lessons about API design from the Spice Girls) but those will take significant time – so here’s a quick post about object and collection initializers instead. Two interesting little oddities...

Is it an object initializer? Is it a collection initializer? No, it’s a syntax error!

The first part came out of a real life situation – FakeDateTimeZoneSource, if you want to look at the complete context.

Basically, I have a class designed to help test time zone-sensitive code. As ever, I like to create immutable objects, so I have a builder class. That builder class has various properties which we’d like to be able to set, and we’d also like to be able to provide it with the time zones it supports, as simply as possible. For the zones-only use case (where the other properties can just be defaulted) I want to support code like this:

var source = new FakeDateTimeZoneSource.Builder
{
    CreateZone("x"),
    CreateZone("y"),
    CreateZone("a"),
    CreateZone("b")
}.Build();

(CreateZone is just a method to create an arbitrary time zone with the given name.)

To achieve this, I made the Builder implement IEnumerable<DateTimeZone>, and created an Add method. (In this case the IEnumerable<> implementation actually works; in another case I've used explicit interface implementation and made the GetEnumerator() method throw NotSupportedException, as it's really not meant to be called in either case.)

So far, so good. The collection initializer worked perfectly as normal. But what about when we want to set some other properties? Without any time zones, that's fine:

var source = new FakeDateTimeZoneSource.Builder
{
VersionId = "foo"
}.Build();

But how could we set VersionId and add some zones? This doesn't work:

var invalid = new FakeDateTimeZoneSource.Builder
{
    VersionId = "foo",
    CreateZone("x"),
    CreateZone("y")
}.Build();

That's neither a valid object initializer (the second part doesn't specify a field or property) nor a valid collection initializer (the first part does set a property).

In the end, I had to expose an IList<DateTimeZone> property:

var valid = new FakeDateTimeZoneSource.Builder
{
VersionId = "foo",
Zones = { CreateZone("x"), CreateZone("y") }
}.Build();

An alternative would have been to expose a propert of type Builder which just returned itself - the same code would have been valid, but it would have been distinctly odd, and allowed some really spurious code.

I'm happy with the result in terms of the flexibility for clients - but the class design feels a bit messy, and I wouldn't have wanted to expose this for the "production" assembly of Noda Time.

Describing all of this to a colleague gave rise to the following rather sillier observation...

Is it an object initializer? Is it a collection initializer? (Parenthetically speaking...)

In a lot of C# code, an assignment expression is just a normal expression. That means there's potentially room for ambiguity, in exactly the same kind of situation as above - when sometimes we want a collection initializer, and sometimes we want an object initializer. Consider this sample class:

using System;
using System.Collections;

class Weird : IEnumerable
{
    public string Foo { get; set; }

    private int count;
    public int Count { get { return count; } }

    public void Add(string x)
    {
        count++;
    }

    IEnumerator IEnumerable.GetEnumerator()
    {
        throw new NotSupportedException();
    }
}

As you can see, it doesn't actually remember anything passed to the Add method, but it does remember how many times we've called it.

Now let's try using Weird in two ways which only differ in terms of parentheses. First up, no parentheses:

string Foo = "x";
Weird weird = new Weird { Foo = "y" };

Console.WriteLine(Foo);         // x
Console.WriteLine(weird.Foo);   // y
Console.WriteLine(weird.Count); // 0

Okay, so it's odd having a local variable called Foo, but we're basically fine. This is an object initializer, and it's setting the Foo property within the new Weird instance. Now let's add a pair of parentheses:

string Foo = "x";
Weird weird = new Weird { (Foo = "y") };

Console.WriteLine(Foo);         // y
Console.WriteLine(weird.Foo);   // Nothing (null)
Console.WriteLine(weird.Count); // 1

Just adding those parenthese turn the object initializer into a collection initializer, whose sole item is the result of the assignment operator - which is the value which has now been assigned to Foo.

Needless to say, I don't recommend using this approach in real code...

Stack Overflow question checklist

skeet — Sat, 24 Nov 2012 09:22:28 GMT

My earlier post on how to write a good question is pretty long, and I suspect that even when I refer people to it, often they don't bother reading it. So here's a short list of questions to check after you've written a question (and to think about before you write the question):

Have you done some research before asking the question? ¹

Have you explained what you've already tried to solve your problem?

Have you specified which language and platform you're using, including version number where relevant?

If your question includes code, have you written it as a short but complete program? ²

If your question includes code, have you checked that it's correctly formatted? ³

If your code doesn't compile, have you included the exact compiler error?

If your question doesn't include code, are you sure it shouldn't?

If your program throws an exception, have you included the exception, with both the message and the stack trace?

If your program produces different results to what you expected, have you stated what you expected, why you expected it, and the actual results?

If your question is related to anything locale-specific (languages, time zones) have you stated the relevant information about your system (e.g. your current time zone)?

Have you checked that your question looks reasonable in terms of formatting?

Have you checked the spelling and grammar to the best of your ability? ⁴

Have you read the whole question to yourself carefully, to make sure it makes sense and contains enough information for someone coming to it without any of the context that you already know?

If the answer to any of these questions is "no" you should take the time to fix up your question before posting. I realize this may seem like a lot of effort, but it will help you to get a useful answer as quickly as possible. Don't forget that you're basically asking other people to help you out of the goodness of their heart - it's up to you to do all you can to make that as simple as possible.

¹ If you went from "something's not working" to "asking a question" in less than 10 minutes, you probably haven't done enough research.

² Ideally anyone answering the question should be able to copy your code, paste it into a text editor, compile it, run it, and observe the problem. Console applications are good for this - unless your question is directly about a user interface aspect, prefer to write a short console app. Remove anything not directly related to your question, but keep it complete enough to run.

³ Try to avoid code which makes users scroll horizontally. You may well need to change how you split lines from how you have it in your IDE. Take the time to make it as clear as possible for those trying to help you.

⁴ I realize that English isn't the first language for many Stack Overflow users. We're not looking for perfection - just some effort. If you know your English isn't good, see if a colleague or friend can help you with your question before you post it.

Noda Time v1.0 released

skeet — Wed, 07 Nov 2012 23:36:44 GMT

Go get Noda Time 1.0!

Today is the end of the longest release cycle I've been personally involved in. On November 5th 2009, I announced my intention to write a port of Joda Time for .NET. The next day, Noda Time was born - with a lofty (foolhardy) set of targets.

Near the end of a talk *about* Noda Time this evening, I released Noda Time 1.0.0.

It's taken three years, but I'm immensely proud of what we've managed to achieve. We're far from "done" but I believe we're already significantly ahead of most other date/time APIs I've seen in terms of providing a clean API which reduces *incidental* complexity while highlighting the *inherent* complexity of the domain. (This is a theme I'm becoming dogmatic about on various fronts.)

There's more to do - I can't see myself considering Noda Time to be "done" any time soon - but hopefully now we've got a stable release, we can start to build user momentum.

One point I raised at the DotNetDevNet presentation tonight was that there's a definite benefit (in my very biased view) in just *looking into* Noda Time:

If you can't use it in your production code, use it when prototyping
If you can't use it in your prototype code, play with it in personal projects
If you can't use it in personal projects, read the user guide to understand the concepts

I hope that simply looking at the various types that Noda Time providers will give you more insight into how you should be thinking about date and time handling in your code. While the BCL API has a lot of flaws, you can work around most of them if you make it crystal clear what your data means at every step. The type system will leave that largely ambiguous, but there's nothing to stop you from naming your variables descriptively, and adding appropriate
comments.

Of course, I would far prefer it if you'd start using Noda Time and raising issues on how to make it better. Spread the word.

Oh, and if anyone from the BCL team is reading this and would like to include something like Noda Time into .NET 5 as a "next generation" date/time, I'd be *really* interested in talking to you :)

How can I enumerate thee? Let me count the ways...

skeet — Sun, 30 Sep 2012 18:58:21 GMT

This weekend, I was writing some demo code for the async chapter of C# in Depth - the idea was to decompile a simple asynchronous method and see what happened. I received quite a surprise during this, in a way which had nothing to do with asynchrony.

Given that at execution time, text refers to an instance of System.String, and assuming nothing in the body of the loop captures the ch variable, how would you expect the following loop to be compiled?

foreach (char ch in text)
{
// Body here
}

Before today, I could think of four answers depending on the compile-time type of text, assuming it compiled at all. One of those answers is if text is declared to be dynamic, which I'm not going to go into here. Let's stick with static typing.

If `text` is declared as `IEnumerable`...

In this case, the compiler can only use the non-generic IEnumerator interface, and I'd expect the code to be roughly equivalent to this:

IEnumerator iterator = text.GetEnumerator();
try
{
    while (iterator.MoveNext())
    {
        char ch = (char) iterator.Current;
        // Body here
    }
}
finally
{
    IDisposable disposable = iterator as IDisposable;
    if (disposable != null)
    {
        disposable.Dispose();
    }
}

Note how the disposal of the iterator has to be conditional, as IEnumerator doesn't extend IDisposable.

If `text` is declared as `IEnumerable<char>`...

Here, we don't need any execution time casting, and the disposal can be unconditional:

IEnumerator<char> iterator = text.GetEnumerator();
try
{
    while (iterator.MoveNext())
    {
        char ch = iterator.Current;
        // Body here
    }
}
finally
{
    iterator.Dispose();
}

If `text` is declared as `string`...

Now things get interesting. System.String implements IEnumerable<char> using explicit interface implementation, and exposes a separate public GetEnumerator() method which is declared to return a CharEnumerator.

Usually when I find a type doing this sort of thing, it's for the sake of efficiency, to reduce heap allocations. For example, List<T>.GetEnumerator returns a List<T>.Enumerator which is struct with the appropriate iteration members. This means if you use foreach over an expression of type List<T>, the iterator can stay on the stack in most cases, saving object allocation and garbage collection.

In this case, however, I suspect CharEnumerator was introduced (way back in .NET 1.0) to avoid having to box each character in the string. This was one reason for foreach handling to be based on types obeying the enumerable pattern, as well as there being support through the normal interfaces. It strikes me that it could still have been a structure in the same way as for List<T>, but maybe that wasn't considered as an option.

Anyway, it means that I would have expected the code to be compiled like this, even back to C# 1:

CharEnumerator iterator = text.GetEnumerator();
try
{
    while (iterator.MoveNext())
    {
        char ch = iterator.Current;
        // Body here
    }
}
finally
{
    iterator.Dispose();
}

What really happens when `text` is declared as `string`...

(This is the bit that surprised me.)

So far, I've been assuming that the C# compiler doesn't have any special knowledge about strings, when it comes to iteration. I knew it did for arrays, but that's all. The actual result - under the C# 5 compiler, at least - is to use the Length property and the indexer directly:

int index = 0;
while (index < text.Length)
{
    char ch = text[index];
    index++;
    // Body here
}

There's no heap allocation, and no Dispose call. If the variable in question can change its value within the loop (e.g. if it's a field, or a captured variable, or there's an assignment to it within the body) then a copy is made of the variable value (just a reference, of course) first, so that all member access is performed on the same object.

Conclusion

So, there we go. There's nothing particularly mind-blowing here - certainly nothing to affect your programming style, unless you were deliberately avoiding using foreach on strings "because it's slow." It's still a good lesson in not assuming you know what the compiler is going to do though... so long as the results are as expected, I'm very happy for them to put extra smarts in, even if it does mean having to change my C# in Depth sample code a bit...

Stack Overflow and personal emails

skeet — Wed, 22 Aug 2012 17:51:24 GMT

This post is partly meant to be a general announcement, and partly meant to be something I can point people at in the future (rather than writing a short version of this on each email).

These days, I get at least a few emails practically every day along the lines of:

"I saw you on Stack Overflow, and would like you to answer this development question for me..."

It's clear that the author:

Is aware of Stack Overflow
Is aware that Stack Overflow is a site for development Q&A
Is aware that I answer questions on Stack Overflow

... and yet they believe that the right way of getting me to answer a question is by emailing it to me directly. Sometimes it's a link to a Stack Overflow question, sometimes it's the question asked directly in email.

In the early days of Stack Overflow, this wasn't too bad. I'd get maybe one email like this a week. Nowadays, it's simply too much.

If you have a question worthy of Stack Overflow, ask it on Stack Overflow. If you've been banned from asking questions due to asking too many low-quality ones before, then I'm unlikely to enjoy answering your questions by email - learn what makes a good question instead, and edit your existing questions.

If you've already asked the question on Stack Overflow, you should consider why you think it's more worthy of my attention than everyone else's questions. You should also consider what would happen if everyone who would like me to answer a question decided to email me.

Of course in some cases it's appropriate. If you've already asked a question, written it as well as you can, waited a while to see if you get any answers naturally, and if it's in an area that you know I'm particularly experienced in (read: the C# language, basically) then that's fine. If your question is about something from C# in Depth - a snippet which doesn't work or some text you don't understand, for example - then it's entirely appropriate to mail me directly.

Basically, ask yourself whether you think I will actually welcome the email. Is it about something you know I'm specifically interested in? Or are you just trying to get more attention to a question, somewhat like jumping a queue?

I'm aware that it's possible this post makes me look either like a grumpy curmudgeon or (worse) like an egocentric pseudo-celebrity. The truth is I'm just like everyone else, with very little time on my hands - time I'd like to spend as usefully and fairly as possible.

The future of "C# in Depth"

skeet — Sat, 04 Aug 2012 08:19:19 GMT

I'm getting fairly frequent questions - mostly on Twitter - about whether there's going to be a third edition of C# in Depth. I figure it's worth answering it once in some detail rather than repeatedly in 140 characters ;)

I'm currently writing a couple of new chapters covering the new features in C# 5 - primarily async, of course. The current "plan" is that these will be added to the existing 2nd edition to create a 3rd edition. There will be minimal changes to the existing text of the 2nd edition - basically going over the errata and editing a few places which ought to mention C# 5 early. (In particular the changes to how foreach loop variables are captured.)

So there will definitely be new chapters. I'm hoping there'll be a full new print (and ebook of course) edition, but no contracts have been signed yet. I'm hoping that the new chapters will be provided free electronically to anyone who's already got the ebook of the 2nd edition - but we'll see. Oh, and I don't have any timelines at the moment. Work is more demanding than it was when I was writing the first and second editions, but obviously I'll try to get the job done at a reasonable pace. (Writing about async in a way which is both accessible and accurate is really tricky, by the way.)

Of course when I've finished those, I've got two other C# books I want to be writing... when I'm not working on Noda Time, Tekpub screencasts etc...

Update

I had a question on Twitter around the "two other C# books". I don't want to go into too many details - partly because they're very likely to change - but my intention is to write "C# from Scratch" and "C# in Style". The first would be for complete beginners; the second wouldn't go into "how things work" so much as "how to use the language most effectively." (Yes, competition for Effective C#.) One possibility is that both would be donationware, at least in ebook form, ideally with community involvement in terms of public comments.

I'm hoping that both will use the same codebase as an extended example, where "From Scratch" would explain what the code does, and "In Style" would explain why I chose that approach. Oh, and "From Scratch" would use unit testing as a teaching tool wherever possible, attempting to convey the idea that it's something every self-respecting dev does :)

The perils of conditional mutability

skeet — Sun, 06 May 2012 13:58:45 GMT

This morning I was wrestling with trying to make some Noda Time unit tests faster. For some reason, the continuous integration host we're using is really slow at loading resources under .NET 4. The unit tests which run in 10 seconds on my home laptop take over three hours on the continuous integration system. Taking stack traces at regular intervals showed the problem was with the NodaFormatInfo constructor, which reads some resources.

I may look into streamlining the resource access later, but before we get to that point, I wanted to try to reduce the number of times we call that constructor in the first place. NodaFormatInfo is meant to be cached, so I wouldn't have expected thousands of instances to be created - but it's only cached when the System.Globalization.CultureInfo it's based on is read-only. This is where the problems start...

CultureInfo is conditionally mutable (not an official term, just one I've coined for the purposes of this post). You can ask whether or not it's read-only with the IsReadOnly property, and obviously if it's read-only you can't change it. Additionally, CultureInfo is composed of other conditionally mutable objects - DateTimeFormatInfo, NumberFormatInfo etc. There's a static ReadOnly method on CultureInfo to create a read-only wrapper around a mutable CultureInfo. It's not clearly documented whether that's expected to take a deep copy (so that callers can really rely on it not changing) or whether it's expected to reflect any further changes made to the culture info it's based on. To go in the other direction, you can call Clone on a CultureInfo to create a mutable copy of any existing culture.

Further complications are introduced by the properties on the composite objects - we have properties such as DateTimeFormatInfo.MonthNames which returns a string array. Remember, arrays are always mutable. So it's really important to know whether the array reference returned from the property refers to a copy of the underlying data, or whether it refers to the array that's used internally by the type. Obviously for read-only DateTimeFormatInfo objects, I'd expect a copy to be returned - but for a mutable DateTimeFormatInfo, it would potentially make sense to return the underlying array reference. Unfortunately, the documentation doesn't make this clear - but in practice, it always returns a copy. If you need to change the month names, you need to clone the array, mutate the clone, and then set the MonthNames property.

All of this makes CultureInfo hard to work with. The caching decision earlier on only really works if a "read-only" culture genuinely won't change behind the scenes. The type system gives you no help to catch subtle bugs at compile-time. Making any of this robust but efficient (in terms of taking minimal copies) is tricky to say the least.

Not only does it make it hard to work with from a client's point of view, but apparently it's hard to implement correctly too...

First bug: Mono's invariant culture isn't terribly invariant...

(Broken in 2.10.8; apparently fixed later.)

I discovered this while getting Noda Time's unit tests to pass on Mono. Unfortunately there are some I've had to effectively disable at the moment, due to deficiencies in Mono (some of which are being fixed, of course).

Here's a program which builds a clone of the invariant culture, changes the clone's genitive month names, and then prints out the first non-genitive name from the plain invariant culture:

using System;
using System.Globalization;

class Test
{
    static void Main()
    {
        CultureInfo clone = (CultureInfo) CultureInfo.InvariantCulture.Clone();
        // Note: not even deliberately changing MonthNames for this culture!
        clone.DateTimeFormat.MonthGenitiveNames[0] = "Changed";

        // Prints Changed
        Console.WriteLine(CultureInfo.InvariantCulture.DateTimeFormat.MonthNames[0]);
    }
}

I believe this bug is really due to the lack of support for genitive month names in Mono at the moment - the MonthGenitiveNames property always just returns a reference to the month names for the invariant culture - without taking a copy first. (It always returns the invariant culture's month names, even if you're using a different culture entirely.) The code above shows an "innocent" attempt to change a mutable clone - but in reality we could have used any culture (including an immutable one) to make the change.

Note that in the .NET implementation, the change would only have been to a copy of the underlying data, so even the clone wouldn't have reflected change anywhere.

Second bug: ReadOnly losing changes

The second bug is the one I found this morning. It looks like it's fixed in .NET 4, but it's present in .NET 3.5, which is where it bit me this morning. When you try to make a read-only wrapper around a mutated culture, some of the properties are preserved... but some aren't:

using System;
using System.Globalization;

class Test
{
    static void Main()
    {
        CultureInfo clone = (CultureInfo) CultureInfo.InvariantCulture.Clone();
        clone.DateTimeFormat.AMDesignator = "ChangedAm";

        // The array is recreated on each call to MonthNames, so changing the
        // value within the array itself doesn't work :(
        string[] months = (string[]) clone.DateTimeFormat.MonthNames;
        months[0] = "ChangedMonth";
        clone.DateTimeFormat.MonthNames = months;

        CultureInfo readOnlyCopy = CultureInfo.ReadOnly(clone);
        Console.WriteLine(clone.DateTimeFormat.AMDesignator); // ChangedAm
        Console.WriteLine(clone.DateTimeFormat.MonthNames[0]); // ChangedMonth

        Console.WriteLine(readOnlyCopy.DateTimeFormat.AMDesignator); // ChangedAm
        Console.WriteLine(readOnlyCopy.DateTimeFormat.MonthNames[0]); // January (!)
    }
}

I don't know what's going on here. In the end I just left the test code using the mutable clone, having added a comment explaining why it wasn't created a read-only wrapper.

I've experimented with a few different options here, including setting the MonthNames property on the clone after creating the wrapper. No joy - I simply can't make the new month names stick in the read-only copy. <sigh>

Conclusion

I've been frustrated by the approach we've taken to cultures in Noda Time for a while. I haven't worked out exactly what we should do about it yet, so it's likely to stay somewhat annoying for v1, but I may well revisit it significantly for v2. Unfortunately, there's nothing I can do about CultureInfo itself.

What I would have preferred in all of this is the builder pattern: make CultureInfo, DateTimeFormatInfo etc all immutable, but give each of them mutable builder types, with the ability to create a mutable builder based on an existing immutable instance, and obviously to create a new immutable instance from a builder. That would make all kinds of things simpler - including caching.

For the moment though, I hope we can all learn lessons from this - or have old lessons reinforced, at least:

Making a single type behave in different ways based on different "modes" makes it hard to use correctly. (Yes, this is the same first conclusion as with DateTime in the previous post. Funny, that.)
Immutability has to be deep to be meaningful: it's not much use having a supposedly read-only object which composes a StringBuilder...
Arrays should be considered somewhat harmful. If you're going to return an array from a method, make sure you document whether this is a copy of the underlying data, or a "live" reference. (The latter should be very rare, particularly for a public API.) The exception here is if you return an empty array - that's effectively immutable, so you can always return it with no problems.
The builder pattern rocks - use it!

In my next post I'll try to get back to the TimeZoneInfo oddities I mentioned last time.

More fun with DateTime

skeet — Wed, 02 May 2012 20:12:43 GMT

(Note that this is deliberately not posted in the Noda Time blog. I reckon it's of wider interest from a design perspective, and I won't be posting any of the equivalent Noda Time code. I'll just say now that we don't have this sort of craziness in Noda Time, and leave it at that...)

A few weeks ago, I was answering a Stack Overflow question when I noticed an operation around dates and times which should have been losing information apparently not doing so. I investigated further, and discovered some "interesting" aspects of both DateTime and TimeZoneInfo. In an effort to keep this post down to a readable length (at least for most readers; certain WebDriver developers who shall remain nameless have probably given up by now already) I'll save the TimeZoneInfo bits for another post.

Background: daylight saving transitions and ambiguous times

There's one piece of inherent date/time complexity you'll need to understand for this post to make sense: sometimes, a local date/time occurs twice. For the purposes of this post, I'm going to assume you're in the UK time zone. On October 28th 2012, at 2am local time (1am UTC), UK clocks will go back to 1am local time. So 1:20am local time occurs twice - once at 12:20am UTC (in daylight saving time, BST), and once at 1:20am UTC (in standard time, GMT).

If you want to run any of the code in this post and you're not in the UK, please adjust the dates and times used to a similar ambiguity for when your clocks go back. If you happen to be in a time zone which doesn't observe daylight savings, I'm afraid you'll have to adjust your system time zone in order to see the effect for yourself.

DateTime.Kind and conversions

As you may already know, as of .NET 2.0, DateTime has a Kind property, of type DateTimeKind - an enum with the following values:

Local: The DateTime is considered to be in the system time zone. Not an arbitrary "local time in some time zone", but in the specific current system time zone.
Utc: The DateTime is considered to be in UTC (corollary: it always unambiguously represents an instant in time)
Unspecified: This means different things in different contexts, but it's a sort of "don't know" kind; this is closer to "local time in some time zone" which is represented as LocalDateTime in Noda Time.

DateTime provides three methods to convert between the kinds:

ToUniversalTime: if the original kind is Local or Unspecified, convert it from local time to universal time in the system time zone. If the original kind is Utc, this is a no-op.
ToLocalTime: if the original kind is Utc or Unspecified, convert it from UTC to local time. If the original kind is Local, this is a no-op.
SpecifyKind: keep the existing date/time, but just change the kind. (So 7am stays as 7am, but it changes the meaning of that 7am effectively.)

(Prior to .NET 2.0, ToUniversalTime and ToLocalTime were already present, but always assumed the original value needed conversion - so if you called x.ToLocalTime().ToLocalTime().ToLocalTime() the result would probably end up with the appropriate offset from UTC being applied three times!)

Of course, none of these methods change the existing value - DateTime is immutable, and a value type - instead, they return a new value.

DateTime's Deep Dark Secret

(The code in this section is presented in several chunks, but it forms a single complete piece of code - later chunks refer to variables in earlier chunks. Put it all together in a Main method to run it.)

Armed with the information in the previous sections, we should be able to make DateTime lose data. If we start with 12:20am UTC and 1:20am UTC on October 28th as DateTimes with a kind of Utc, when we convert them to local time (on a system in the UK time zone) we should get 1:20am in both cases due to the daylight saving transition. Indeed, that works:

// Start with different UTC values around a DST transition
var original1 = new DateTime(2012, 10, 28, 0, 20, 0, DateTimeKind.Utc);
var original2 = new DateTime(2012, 10, 28, 1, 20, 0, DateTimeKind.Utc);

// Convert to local time
var local1 = original1.ToLocalTime();
var local2 = original2.ToLocalTime();

// Result is the same for both values. Information loss?
var expected = new DateTime(2012, 10, 28, 1, 20, 0, DateTimeKind.Local);
Console.WriteLine(local1 == expected); // True
Console.WriteLine(local2 == expected); // True
Console.WriteLine(local1 == local2); // True

If we've started with two different values, applied the same operation to both, and ended up with equal values, then we must have lost information, right? That doesn't mean that operation is "bad" any more than "dividing by 2" is bad. You ought to be aware of that information loss, that's all.

So, we ought to be able to demonstrate that information loss further by converting back from local time to universal time. Here we have the opposite problem: from our local time of 1:20am, we have two valid universal times we could convert to - either 12:20am UTC or 1:20am UTC. Both answers would be correct - they are universal times at which the local time would be 1:20am. So which one will get picked? Well... here's the surprising bit:

// Convert back to UTC
var roundTrip1 = local1.ToUniversalTime();
var roundTrip2 = local2.ToUniversalTime();

// Values round-trip correctly! Information has been recovered...
Console.WriteLine(roundTrip1 == original1); // True
Console.WriteLine(roundTrip2 == original2); // True
Console.WriteLine(roundTrip1 == roundTrip2); // False

Somehow, each of the local values knows which universal value it came from. The The information has been recovered, so the reverse conversion round-trips each value back to its original one. How is that possible?

It turns out that DateTime actually has four potential kinds: Local, Utc, Unspecified, and "local but treat it as the earlier option when resolving ambiguity". A DateTime is really just a 64-bit number of ticks, but because the range of DateTime is only January 1st 0001 to December 31st 9999. That range can be represented in 62 bits, leaving 2 bits "spare" to represent the kind. 2 bits gives 4 possible values... the three documented ones and the shadowy extra one.

Through experimentation, I've discovered that the kind is preserved if you perform arithmetic on the value, too... so if you go to another "fall back" DST transition such as October 30th 2011, the ambiguity resolution works the same way as before:

var local3 = local1.AddYears(-1).AddDays(2);
var local4 = local2.AddYears(-1).AddDays(2);
Console.WriteLine(local3.ToUniversalTime().Hour); // 0
Console.WriteLine(local4.ToUniversalTime().Hour); // 1

If you use DateTime.SpecifyKind with DateTimeKind.Local, however, it goes back to the "normal" kind, even though it looks like it should be a no-op:

// Should be a no-op?
var local5 = DateTime.SpecifyKind(local1, local1.Kind);
Console.WriteLine(local5.ToUniversalTime().Hour); // 1

Is this correct behaviour? Or should it be a no-op, just like calling ToLocalTime on a "local" DateTime is? (Yes, I've checked - that doesn't lose the information.) It's hard to say, really, as this whole business appears to be undocumented... at least, I haven't seen anything in MSDN about it. (Please add a link in the comments if you find something. The behaviour actually goes against what's documented, as far as I can tell.)

I haven't looked into whether various forms of serialization preserve values like this faithfully, by the way - but you'd have to work hard to reproduce it in non-framework code. You can't explicitly construct a DateTime with the "extra" kind; the only ways I know of to create such a value are via a conversion to local time or through arithmetic on a value which already has the kind. (Admittedly if you're serializing a DateTime with a Kind of Local, you're already on potentially shaky ground, given that you could be deserializing it on a machine with a different system time zone.)

Unkind comparisons

I've misled you a little, I have to admit. In the code above, when I compared the "expected" value with the results of the first conversions, I deliberately specified DateTimeKind.Local in the constructor call. After all, that's the kind we do expect. Well, yes - but I then printed the result of comparing this value with local1 and local2... and those comparisons would have been the same regardless of the kind I'd specified in the constructor.

All comparisons between DateTimes ignore the Kind property. It's not just restricted to equality. So for example, consider this comparison:

// In June: Local time is UTC+1, so 8am UTC is 9am local
var dt1 = new DateTime(2012, 6, 1, 8, 0, 0, DateTimeKind.Utc);
var dt2 = new DateTime(2012, 6, 1, 8, 30, 0, DateTimeKind.Local);
Console.WriteLine(dt1 < dt2); // True

When viewed in terms of "what instants in time do these both represent?" the answer here is wrong - when you convert both values into the same time zone (in either direction), dt1 occurs after dt2. But a simple look at the properties tells a different story. In practice, I suspect that most comparisons between DateTime values of different kinds involve code which is at best sloppy and is quite possibly broken in a meaningful way.

Of course, if you bring Kind=Unspecified into the picture, it becomes impossible to compare meaningfully in a kind-sensitive way. Is 12am UTC before or after 1am Unspecified? It depends what time zone you later use.

To be clear, it is a hard-to-resolve issue, and one that we don't do terribly well at in Noda Time at the moment for ZonedDateTime. (And even with just LocalDateTime you've got issues between calendars.) This is a situation where providing separate Comparer<T> implementations works nicely - so you can explicitly say what kind of comparison you want.

Conclusions

There's more fun to be had with a similar situation when we look at TimeZoneInfo, but for now, a few lessons:

Giving a type different "modes" which make it mean fairly significantly different things is likely to cause headaches
Keeping one of those modes secret (and preventing users from even constructing a value in that mode directly) leads to even more fun and games
If two instances of your type are considered "equal" but behave differently, you should at least consider whether there's something smelly going on
There's always more fun to be had with DateTime...

Type initializer circular dependencies

skeet — Sat, 07 Apr 2012 09:35:45 GMT

To some readers, the title of this post may induce nightmarish recollections of late-night debugging sessions. To others it may be simply the epitome of jargon. Just to break the jargon down a bit:

Type initializer: the code executed to initialize the static variables of a class, and the static constructor
Circular dependency: two bits of code which depend on each other - in this case, two classes whose type initializers each require that the other class is initialized

A quick example of the kind of problem I'm talking about would be helpful here. What would you expect this code to print?

using System;

class Test
{
    static void Main()
    {
        Console.WriteLine(First.Beta);
    }
}

class First
{
    public static readonly int Alpha = 5;
    public static readonly int Beta = Second.Gamma;
}

class Second
{
    public static readonly int Gamma = First.Alpha;
}

Of course, without even glancing at the specification, any expectations are pretty irrelevant. Here's what the spec (section 10.5.5.1 of the C# 4 version):

The static field variable initializers of a class correspond to a sequence of assignments that are executed in the textual order in which they appear in the class declaration. If a static constructor (§10.12) exists in the class, execution of the static field initializers occurs immediately prior to executing that static constructor. Otherwise, the static field initializers are executed at an implementation-dependent time prior to the first use of a static field of that class.

In addition to the language specification, the CLI specification gives more details about type initialization in the face of circular dependencies and multiple threads. I won't post the details here, but the gist of it is:

Type initialization acts like a lock, to prevent more than one thread from initializing a type
If the CLI notices that type A needs to be initialized in order to make progress, but it's already in the process of initializing type A in the same thread, it continues as if the type were already initialized.

So here's what you might expect to happen:

Initialize Test: no further action required
Start running Main
Start initializing First (as we need First.Beta)
Set First.Alpha to 5
Start initializing Second (as we need Second.Gamma)
Set Second.Gamma to First.Alpha (5)
End initializing Second
Set First.Beta to Second.Gamma (5)
End initializing First
Print 5

Here's what actually happens - on my box, running .NET 4.5 beta. (I know that type initialization changed for .NET 4, for example. I don't know of any changes for .NET 4.5, but I'm not going to claim it's impossible.)

Initialize Test: no further action required
Start running Main
Start initializing First (as we need First.Beta)
Start initializing Second (we will need Second.Gamma)
Set Second.Gamma to First.Alpha (0)
End initializing Second
Set First.Alpha to 5
Set First.Beta to Second.Gamma (0)
End initializing First
Print 0

Step 5 is the interesting one here. We know that we need First to be initialized, in order to get First.Alpha, but this thread is already initializing First (we started in step 3) so we just access First.Alpha and hope that it's got the right value. As it happens, that variable initializer hasn't been executed yet. Oops.

(One subtlety here is that I could have declared all these variables as constants instead using "const" which would have avoided all these problems.)

Back in the real world...

Hopefully that example makes it clear why circular dependencies in type initializers are nasty. They're hard to spot, hard to debug, and hard to test. Pretty much your classic Heisenbug, really. It's important to note that if the program above had happened to initialize Second first (to access a different variable, for example) we could have ended up with a different result. In particular, it's easy to get into a situation where running all your unit tests can cause a failure - but if you run just the failing test, it passes.

One way of avoiding all of this is never to use any type initializers for anything, of course. In many cases that's exactly the right solution - but often there are natural uses, particularly for well-known values such as Encoding.Utf8, TimeZoneInfo.Utc and the like. Note that in both of those cases they are static properties, but I would expect them to be backed by static fields. I'm somewhat ambivalent between using public static readonly fields and public static get-only properties - but as we'll see later, there's a definite advantage to using properties.

Noda Time has quite a few values like this - partly because so many of its types are immutable. It makes sense to create a single UTC time zone, a single ISO calendar system, a single "pattern" (text formatter/parser) for each of a variety of common cases. In addition to the publicly visible values, there are various static variables used internally, mostly for caching purposes. All of this definitely adds complexity - and makes it harder to test - but the performance benefits can be significant.

Unfortunately, a lot of these values end up with fairly natural circular dependencies - as I discovered just recently, where adding a new static field caused all kinds of breakage. I was able to fix the immediate cause, but it left me concerned about the integrity of the code. I'd fixed the one failure I knew about - but what about any others?

Testing type initialization

One of the biggest issues with type initialization is the order-sensitivity - combined with the way that once a type has been initialized once, that's it for that AppDomain. As I showed earlier, it's possible that initializing types in one particular order causes a problem, but a different order won't.

I've decided that for Noda Time at least, I want to be reasonably sure that type initialization circularity isn't going to bite me. So I want to validate that no type initializers form cycles, whatever order the types are initialized in. Logically if we can detect a cycle starting with one type, we ought to be able to detect it starting with any of the other types in that cycle - but I'm sufficiently concerned about weird corner cases that I'd rather just take a brute force approach.

So, as a rough plan:

Start with an empty set of dependencies
For each type in the target assembly:
- Create a new AppDomain
- Load the target assembly into it
- Force the type to be initialized
- Take a stack trace at the start of each type initializer and record any dependencies
Look for cycles in the complete set of dependencies

Note that we'll never spot a cycle within any single AppDomain, due to the way that type initialization works. We have to put together the results for multiple initialization sequences to try to find a cycle.

A description of the code would probably be harder to follow than the code itself, but the code is relatively long - I've included it at the end of this post to avoid intefering with the narrative flow. For more up-to-date versions in the future, look at the Noda Time repository.

This isn't a terribly nice solution, for various reasons:

Creating a new AppDomain and loading assemblies into it from a unit test runner isn't as simple as it might be. My code doesn't currently work with NCrunch; I don't know how it finds its assemblies yet. When I've fixed that, I'm sure other test runners would still be broken. Likewise I've had to explicitly filter types to get TeamCity (the continuous integration system Noda Time uses) to work properly. Currently, you'd need to edit the test code to change the filters. (It's possible that there are better ways of filtering, of course.)
It relies on each type within the production code which has an "interesting" type initializer to have a line like this:
private static readonly int TypeInitializationChecking = NodaTime.Utility.TypeInitializationChecker.RecordInitializationStart();
Not only does the previous line need to be added to the production code - it clearly gets executed each time, and takes up heap space per type. It's only 4 bytes for each type involved, and it does no real work when we're not testing, but it's a nuisance anyway. I could use preprocessor directives to remove the code from non-debug or non-test-targeted builds, but that would look even messier.
It only picks up cycles which occur when running the version of .NET the tests happen to execute on. Given that there are ordering changes for different versions, I wouldn't like to claim this is 100% bullet-proof. Likewise if there are only cycles when you're running in some specific cultures (or other environmental features), it won't necessarily pick those up.
I've deliberately not tried to make the testing code thread-safe. That's fine in Noda Time - I don't have any asynchronous operations or new threads in Noda Time at all - but other code may need to make this more robust.

So with all these caveats, is it still worth it? Absolutely: it's already found bugs which have now been fixed.

In fact, the test didn't get as far as reporting cycles to start with - it turned out that if you initialized one particular type first, the type initializer would fail with a NullReferenceException. Ouch! Once I'd fixed that, there were still quite a few problems to fix. Somewhat coincidentally, fixing them improved the design too - although the user-visible API didn't change at all.

Fixing type initializer cycles

In the past, I've occasionally "fixed" type initialization ordering problems by simply moving fields around. The cycles still existed, but I figured out how to make them harmless. I can say now that this approach does not scale, and is more effort than it's worth. The code ends up being brittle, hard to think about, and once you've got more than a couple of types involved it's really error-prone, at least for my brain. It's much better to break the cycle completely. To this end, I've ended up using a fairly simple technique to defer initialization of static variables. It's a poor-man's Lazy<T>, to some extent - but I'd rather not have to write Lazy<T> myself, and we're currently targeting .NET 3.5...

Basically, instead of exposing a public static readonly field which creates the cycle, you expose a public static readonly property - which returns an internal static readonly field in a nested, private static class. We still get the nice thread-safe once-only initialization of a type initializer, but the nested type won't be initialized until it needs to be. (In theory it could be initialized earlier, but a static constructor would ensure it isn't.) So long as nothing within the rest of the type initializer for the containing class uses that property, we avoid the cycle.

So instead of this:

// Requires Bar to be initialized - if Bar also requires Foo to be
// initialized, we have a problem...
public static readonly Foo SimpleFoo = new Foo(Bar.Zero);

We might have:

public static readonly Foo SimpleFoo { get { return Constants.SimpleFoo; } }

private static class Constants
{
    private static readonly int TypeInitializationChecking = NodaTime.Utility.TypeInitializationChecker.RecordInitializationStart();

    // This requires both Foo and Bar to be initialized, but that's okay
    // so long as neither of them require Foo.Constants to be initialized.
    // (The unit test would spot that.)
    internal static readonly Foo SimpleFoo = new Foo(Bar.Zero);
}

I'm currently undecided about whether to include static constructors in these classes to ensure lazy initialization. If the type initializer for Foo triggered the initializer of Foo.Constants, we'd be back to square one... but adding static constructors into each of these nested classes sounds like a bit of a pain. The nested classes should call into the type initialization checking as well, to validate they don't cause any problems themselves.

Conclusion

I have to say, part of me really doesn't like either the testing code or the workaround. Both smack of being clever, which is never a good thing. It's definitely worth considering whether you could actually just get rid of the type initializer (or part of it) entirely, avoiding maintaining so much static state. It would be nice to be able to detect these type initializer cycles without running anything, simply using static analysis - I'm going to see whether NDepend could do that when I get a chance. The workaround doesn't feel as neat as Lazy<T>, which is really what's called for here - but I don't trust myself to implement it correctly and efficiently myself.

So while both are somewhat hacky, they're better than the alternative: buggy code. That's what I'm ashamed to say I had in Noda Time, and I don't think I'd ever have spotted all the cycles by inspection. It's worth a try on your own code - see whether you've got problems lurking...

Appendix: Testing code

As promised earlier, here's the code for the production and test classes.

TypeInitializationChecker

This is in NodaTime.dll itself.

internal sealed class TypeInitializationChecker : MarshalByRefObject
{
    private static List<Dependency> dependencies = null;

    private static readonly MethodInfo EntryMethod = typeof(TypeInitializationChecker).GetMethod("FindDependencies");

    internal static int RecordInitializationStart()
    {
        if (dependencies == null)
        {
            return 0;
        }
        Type previousType = null;
        foreach (var frame in new StackTrace().GetFrames())
        {
            var method = frame.GetMethod();
            if (method == EntryMethod)
            {
                break;
            }
            var declaringType = method.DeclaringType;
            if (method == declaringType.TypeInitializer)
            {
                if (previousType != null)
                {
                    dependencies.Add(new Dependency(declaringType, previousType));
                }
                previousType = declaringType;
            }
        }
        return 0;
    }

    /// <summary>
    /// Invoked from the unit tests, this finds the dependency chain for a single type
    /// by invoking its type initializer.
    /// </summary>
    public Dependency[] FindDependencies(string name)
    {
        dependencies = new List<Dependency>();
        Type type = typeof(TypeInitializationChecker).Assembly.GetType(name, true);
        RuntimeHelpers.RunClassConstructor(type.TypeHandle);
        return dependencies.ToArray();
    }

    /// <summary>
    /// A simple from/to tuple, which can be marshaled across AppDomains.
    /// </summary>
    internal sealed class Dependency : MarshalByRefObject
    {
        public string From { get; private set; }
        public string To { get; private set; }
        internal Dependency(Type from, Type to)
        {
            From = from.FullName;
            To = to.FullName;
        }
    }
}

TypeInitializationTest

This is within NodaTime.Test:

[TestFixture]
public class TypeInitializationTest
{
    [Test]
    public void BuildInitializerLoops()
    {
        Assembly assembly = typeof(TypeInitializationChecker).Assembly;
        var dependencies = new List<TypeInitializationChecker.Dependency>();
        // Test each type in a new AppDomain - we want to see what happens where each type is initialized first.
        // Note: Namespace prefix check is present to get this to survive in test runners which
        // inject extra types. (Seen with JetBrains.Profiler.Core.Instrumentation.DataOnStack.)
        foreach (var type in assembly.GetTypes().Where(t => t.FullName.StartsWith("NodaTime")))
        {
            // Note: this won't be enough to load the assembly in all test runners. In particular, it fails in
            // NCrunch at the moment.
            AppDomainSetup setup = new AppDomainSetup { ApplicationBase = AppDomain.CurrentDomain.BaseDirectory };
            AppDomain domain = AppDomain.CreateDomain("InitializationTest" + type.Name, AppDomain.CurrentDomain.Evidence, setup);
            var helper = (TypeInitializationChecker)domain.CreateInstanceAndUnwrap(assembly.FullName,
                typeof(TypeInitializationChecker).FullName);
            dependencies.AddRange(helper.FindDependencies(type.FullName));
        }
        var lookup = dependencies.ToLookup(d => d.From, d => d.To);
        // This is less efficient than it might be, but I'm aiming for simplicity: starting at each type
        // which has a dependency, can we make a cycle?
        // See Tarjan's Algorithm in Wikipedia for ways this could be made more efficient.
        // http://en.wikipedia.org/wiki/Tarjan's_strongly_connected_components_algorithm
        foreach (var group in lookup)
        {
            Stack<string> path = new Stack<string>();
            CheckForCycles(group.Key, path, lookup);
        }
    }

    private static void CheckForCycles(string next, Stack<string> path, ILookup<string, string> dependencyLookup)
    {
        if (path.Contains(next))
        {
            Assert.Fail("Type initializer cycle: {0}-{1}", string.Join("-", path.Reverse().ToArray()), next);
        }
        path.Push(next);
        foreach (var candidate in dependencyLookup[next].Distinct())
        {
            CheckForCycles(candidate, path, dependencyLookup);
        }
        path.Pop();
    }
}

Diagnosing weird problems - a Stack Overflow case study

skeet — Fri, 16 Mar 2012 06:31:00 GMT

Earlier, I came across this Stack Overflow question. I solved it, tweeted it, but then thought it would serve as a useful case study into the mental processes I go through when trying to solve a problem - whether that's on Stack Overflow, at work, or at home.

It's definitely worth reading the original question, but the executive summary is:

When I compute the checksum/hash of c:\Windows\System32\Calc.exe using various tools and algorithms, those tools all give the same answer for each algorithm. When I try doing the same thing in Java, I get different results. What's going on?

Now to start with, I'd like to shower a bit of praise on the author:

The post came with a short but utterly complete program to demonstrate the problem
The comments in the program showed the expected values and the actual values
The code was mostly pretty clean (clean enough for an SO post anyway)

In short, it had pretty much everything I ask for in a question. Yay! Additionally, the result seemed to be strange. The chances of any one of Java's hashing algorithms being broken seem pretty slim, but four of them? Okay, now you've got me interested.

Reproducing the problem

Unless I can spot the error immediately, I usually try to reproduce the problem in a Stack Overflow post with a short but complete program. In this case, the program was already provided, so it was just a matter of copy/paste/compile/run. This one had the additional tweak that it was comparing the results of Java with the results of other tools, so I had to get hold of an MD5 sum tool first. (I chose to start with MD5 for no particular reason.) I happened to pick this one, but it didn't really seem it would make much difference. (As it happens, that was an incorrect assumption, but hey...)

I ran md5sums on c:\Windows\System32\calc.exe, and got the same result as the poster. Handy.

I then ran the Java code, and again got the same result as the poster: step complete, we have a discrepancy between at least one tool (and MD5 isn't that hard to get right) and Java.

Looking for obvious problems

The code has four main areas:

Reading a file
Updating digests
Converting bytes to hex
Storing and displaying results

Of these, all of the first three have common and fairly simple error modes. For example:

Failure to use the return value from InputStream.read()
Failure to update the digests using only the relevant portion of the data buffer
Failure to cope with Java's signed bytes

The code for storing and displaying results seemed solid enough to ignore to start with, and brief inspection suggested that the first two failure modes had been avoided. While the hex code didn't have any obvious problems either, it made sense to check it. I simply printed the result of hard-coding the "known good" CRC-32 value:

System.out.println(toHex(new byte[] {
(byte) 0x8D, (byte) 0x8F, (byte) 0x5F, (byte) 0x8E
}));

That produced the right result, so I ruled out that part of the code too. Even if it had errors in some cases, we know it's capable of producing the right string for one of the values we know we should be returning, so it can't be getting that value.

Initial checks around the file

I'm always suspicious of stream-handling code - or rather, I know how easily it can hide subtle bugs. Even though it looked okay, I thought I'd check - so I added a simple total to the code so I could see how many bytes had been hashed. I also removed all the hash algorithms other than MD5, just for simplicity:

MessageDigest md5 = MessageDigest.getInstance("MD5");

FileInputStream fis = new FileInputStream(file);
byte data[] = new byte[size];
int len = 0;
int total = 0;
while ((len = fis.read(data)) != -1) {
md5.update(data, 0, len);
total += len;
}
fis.close();
System.out.println("Total bytes read: " + total);

results.put(md5.getAlgorithm(), toHex(md5.digest()));

It's worth noting that I haven't tried to fix up bits of the code which I know I would change if I were actually doing a code review:

The stream isn't being closed in a finally block, so we'll have a dangling resource (until GC) if an IOException is thrown
The initial value of len is never read, and can be removed

Neither of these matters in terms of the problem at hand, and closing the file "properly" would make the sample code more complicated. (For the sake of just a short sample program, I'd be tempted to remove it entirely.)

The result showed the number of bytes being read as the command prompt did when I ran "dir c:\Windows\System32\Calc.exe" - so again, everything looks like it's okay.

Desperate times call for stupid measures

Just on a whim, I decided to copy Calc.exe to a local folder (the current directory) and retry. After all, accessing a file in a system folder might have some odd notions applied to it. It's hard to work out what, but... there's nothing to lose just by trying a simple test. If it can rule anything out, and you've got no better ideas, you might as well try even the daftest idea.

I modified the source code to use the freshly-copied file, and it gave the same result. Hmm.

I then reran md5sums on the copied file... and it gave the same result as Java. In other words, running "md5sums c:\Windows\System32\Calc.exe" gave one result, but "md5sums CopyOfCalc.exe" gave a different result. At this point we've moved from Java looking like it's behaving weirdly to md5sums looking suspicious.

Proving the root cause

At this point we're sort of done - we've basically proved that the Java code produces the right hash for whatever data it's given, but we're left with the question of what's happening on the file system. I had a hunch that it might be something to do with x86 vs x64 code (all of this was running on a 64-bit version of Windows 7) - so how do we test that assumption?

I don't know if there's a simple way of running an x86 version of the JVM, but I do know how to switch .NET code between x86 and x64 - you can do that for an assembly at build time. C# also makes the hashing and hex conversion simple, so I was able to knock up a very small app to show the problem:

using System;
using System.IO;
using System.Security.Cryptography;

class Test
{
    static void Main()
    {
        using (var md5 = MD5.Create())
        {
            string path = "c:/Windows/System32/Calc.exe";
            var bytes = md5.ComputeHash(File.ReadAllBytes(path));
            Console.WriteLine(BitConverter.ToString(bytes));
        }
    }
}

(For a larger file I'd have used File.OpenRead instead, but then I'd have wanted to close the stream afterwards. Somehow it wasn't worth correcting the existing possible handle leak in the Java code, but I didn't want to write leaky code myself. So instead I've got code which reads the whole file into memory unnecessarily... <sigh>)

You can choose the architecture to build against (usually AnyCPU, x86 or x64 - though it's interesting to see that "arm" is also an option under .NET 4.5, for obvious reasons) either from Visual Studio or using the "/platform" command line option. This doesn't change the IL emitted (as far as I'm aware) but it's used for interop with native code - and in the case of executables, it also determines the version of the CLR which is used.

Building and running in x86 mode gave the same answer as the original "assumed to be good" tools; building and running in x64 mode gave the same answer as Java.

Explaining the root cause

At this point we've proved that the file system gives different results depending on whether you access it from a 64-bit process or a 32-bit process. The final piece of the puzzle was to find some something to explain why that happens. With all the evidence about what's happening, it was now easy to search for more information, and I found this article giving satisfactory details. Basically, there are two different copies of the system executables on a 64 bit system: x86 ones which run under the 32-bit emulator, and x64 ones. They're actually in different directories, but when a process opens a file in \Windows\System32, the copy which matches the architecture of the process is used. It's almost as if the \Windows\System32 directory is a symlink which changes target depending on the current process.

A Stack Overflow comment on my answer gave a final nugget that this is called the "File System Redirector".

Conclusion

Debugging sessions often feel a bit like this - particularly if you're like me, and only resort to real debugging after unit testing has failed. It's a matter of looking under all kinds of rocks, trying anything and everything, but keeping track of everything you try. At the end of process you should be able to explain every result you've seen, in an ideal world. (You may not be able to give evidence of the actual chain of events, but you should be able to construct a plausible chain of events which concurs with your theory.)

Be aware of areas which can often lead to problems, and check those first, gradually reducing the scope of "stuff you don't understand" to a manageable level, until it disappears completely.

Eduasync 20: Changes between the VS11 Preview and the Visual Studio 11 Beta

skeet — Tue, 06 Mar 2012 19:54:54 GMT

A while I ago I blogged about what had changed under the hood of async between the CTP and the VS11 Preview. Well, now that the VS11 Beta is out, it's time to do it all again...

Note that the code in this post is in the Eduasync codebase, under a different solution (Eduasync VS11.sln). Many of the old existing projects won't compile with VS11 beta, but I'd rather leave them as they are for posterity, showing the evolution of the feature.

Stephen Toub has an excellent blog post covering some of this, so while I'll mention things he's covered, I won't go into much detail about them. Let's start off there though...

(EDIT: Stephen has also mailed me with some corrections, which I've edited in - mostly without indication, as the post has been up for less than seven hours, and it'll make for a better reading experience.)

Awaiter pattern changes

The awaiter pattern is now not just a pattern. The IsCompleted property and GetResult method are still "loose" but OnCompleted is now part of an interface: INotifyCompletion. Awaiters have to implement INotifyCompleted, but may also implement ICriticalNotifyCompletion and its UnsafeOnCompleted method.

The OnCompleted method is just as it was before, and needs to flow the execuction context; the UnsafeOnCompleted method is simpler, as it doesn't need to flow the execution context. All of this only matters if you're implementing your own awaiters, of course. (More details in Stephen's blog post. I've found this area somewhat confusing, so please do read his post carefully!)

Skeleton method changes

Just as I have previously, I'm using the (entirely unofficial) term "skeleton method" to mean the very short method created by the compiler with the same signature as an async method: this is the entry point to the async method, effectively, which creates and starts the state machine containing all the real logic.

There are two changes I've noticed in the skeleton method. Firstly, for some reason the state machine state numbers have changed. Whereas previously a state number of 0 meant "initial or running", positive values meant "between calls, or navigating back to the await expression" and -1 meant "finished", now -1 means "initial or running", non-negative means "between calls, or navigating back to await expression" and -2 means "finished". It's not clear why this change has been made, given that it requires an extra assignment at the start of every skeleton method (to set the state to -1).

More importantly, the skeleton method no longer calls MoveNext directly on the state machine that it's built. Instead, it calls Start<TStateMachine> on the AsyncTaskMethodBuilder<T> (or whichever method builder it's using). It passes the state machine by reference (presumably for efficiency), and TStateMachine is constrained to implement the now-public-and-in-mscorlib IAsyncStateMachine interface. I'll come back to the relationship between the state machine and the builder later on.

Task caching

(Code is in project 30: TaskCaching)

It's possible for an async method to complete entirely synchronously. In this situation, the result is known before the method returns, and the task returned by the method is already in the RanToCompletionState. If two tasks have already run to completion with the same value, they can be (apparently) regarded as equivalent... so the beta now caches a task in this situation, for some types and values. (Apparently the preview cached too, but I hadn't noticed, and the beta caches more.) According to my experiments and some comments:

For int, tasks with values -1 to 8 inclusive are cached
For bool, both values (true and false)
For char, byte, sbyte, short, ushort, uint, long, ulong, IntPtr and UIntPtr tasks with value 0 (or '\0') are cached
For reference types, null is cached
For other types, no tasks are cached

EDIT: After they've completed, tasks are normally immutable except for disposal - the cached tasks are tweaked slightly to make disposal a no-op.

State machine interface changes

In the VS11 preview release, each state machine implemented an interface, but that interface was internal to the generated assembly, and contained a single method (SetMoveNextDelegate). It's now a public interface with two methods:

Personally I'm not keen on the naming of "MoveNext" - I can't help but feel that if we didn't have the "naming baggage" of IEnumerator and the fact that at least early on, the code generator was very similar to that used for iterator blocks, we'd have something different. (It is moving to the next state of the state machine, but it still doesn't quite feel right.) I'd favour something like "ContinueExecution". However, it doesn't matter - it obviously does what you'd expect, and you're not going to be calling this yourself.

SetStateMachine is a stranger beast. The documentation states:

Configures the state machine with a heap-allocated replica.

... which says almost nothing, really. The implementation is always simple, just like SetMoveNextDelegate was, although this time it delegates to the builder for the real work (a common theme, as we'll see):

void IAsyncStateMachine.SetStateMachine(IAsyncStateMachine param0)
{
this.<>t__builder.SetStateMachine(param0);
}

Now AsyncTaskMethodBuilder.SetStateMachine is also documented pretty sparsely:

Associates the builder with the specified state machine.

Again, no real help. However, we'll see that it's the builder which is responsible for calling OnContinue now, and as it can call MoveNext on an IStateMachine, it makes sense to tell it which state machine it's associated with... but can't it do that directly?

Well, not quite. The problem (as I understand it) is around when boxing occurs. We initially create the state machine on the stack, and it contains the builder. (Both are structs.) That's fine until we need a continuation, but then we've got to be able to get back to the current state later, after the current stack frame has been blown away. So we need to box the state machine. That will create a copy of the current builder (within the box). We need the builder within the boxed state machine to contain a reference to the same box. So the order has to be:

Box the state machine
Tell the state machine about the boxed reference
The state machine tells its nested builder about the boxed reference

Back when the state machine was in charge of the boxing, this went via the delegate: the act of creating the box was implicit when creating the delegate, and then casting the delegate target to the interface type allowed a reference to the newly-created delegate to be set within the copy. This is similar, but using the builder instead. It's hard to follow, but of course it's not going to matter.

State machine field changes

There are various kinds of fields in the state machine:

Those corresponding with local variables and parameters in the async method
The state
The field(s) associated with awaiters
(In the preview/beta) The field associated with the "current execution stack" at the point of an await expression
(In the CTP) An unused "disposing" field

Of these, I believe only the awaiters have actually changed, but before we talk about that, let's revisit local variables.

Local variable hoisting

I've just noticed that the local variables are only hoisted to fields when its scope contains an await expressions, but in that case all local variables of that scope are hoisted, whether or not they're used "across" awaits. It would be possible to hoist only those which need to be maintained between executions, but then you wouldn't be able to see the others when debugging, which would be somewhat confusing. Likewise local variables of the same type which are never propagated across the same await could be aliased. For example, consider this async method:

static async Task<int> M(Random rng)
{
    int x = rng.Next(1000);
    int y = x + rng.Next(1000);

    await Task.Yield();

    int z = y + rng.Next(1000);

    await Task.Yield();
    return z;
}

If the compiler could be confident you didn't need to debug through this code, it could make do with one field of type "Random" and one field of type "int" - x can be a completely local variable in MoveNext (it's not used between two awaits) and y and z can be aliased (we never need the value of y after we've first written to z).

Local variable aliasing probably isn't particularly useful for "normal" methods as the JIT may be able to do it (so long as you don't have a debugger attached, potentially) but in this case we expect the state machine to be boxed at some point, so potentially does make a difference (while the stack is typically reasonably small, you could have a lot of outstanding async methods in some scenarios). Maybe in a future release, the C# compiler could have an aggressive optimization mode around this, to be turned on explicitly. (I don't think it should be a particularly high priority, mind you.)

Awaiter fields

(Code is in project 31, AwaiterFields.)

Awaiter fields have changed a bit over the course of async's history.

In the CTPs (all of them, I believe) each await expression had its own awaiter field in the state machine, with a type corresponding to the declared awaiter type from the awaitable. (Remember that the awaitable is the thing you await, such as a task, and the awaiter is what you get back from calling GetAwaiter on the awaitable).

In the VS11 Preview, there was always a single awaiter field of type object. From what I saw, it was usually populated with a single-element array containing an awaiter. For value type awaiters (i.e. where the awaiter is a struct) this is somewhat similar to boxing, but maintaining strong typing, so calls to IsCompleted etc can still be made. It's possible that reference type awaiters were stored without the array creation, as it would serve no purpose. (I don't have any machines with just the preview installed to verify this.)

In the Beta, we have a mixture. If there are any reference type awaiters, they all end up being stored in a single field of type object, which is then cast back to the actual type when required. (Don't forget that only one awaiter can be "active" at a time, which makes this possible.) This includes awaiters of an interface type - it's only the compile-time type declared as the return type of the GetAwaiter method of the awaitable which is important.

If any of the awaiter types are value types, each of these types gets its own field. So there might be a TaskAwaiter<int> field and a TaskAwaiter<string> field, for example. However, there can still be "sharing" going on: if there are multiple await expressions all of the same value type awaiter, they will all share a single field. (This all feels a little like the JITting of generics, but it's somewhat coincidental.)

MoveNext method changes

(Code is in project 32, BetaStateMachine)

As I've mentioned earlier, the builder is now responsible for a lot more of the work than it was in earlier versions. The majority of the code remains the same as far as I can tell, in terms of handling branching, evaluating expressions with multiple await expressions and so on. The code in the source repository shows what the complete state machine looks like, but for the sake of clarity, I'll just focus on a single snippet. If we have an await expression like this:

await x;

then the state machine code in the VS11 Preview would look something like this:

localTaskAwaiter = x.GetAwaiter();
if (localTaskAwaiter.IsCompleted)
{
    goto AwaitCompleted;
}
this.state = 1;
TaskAwaiter[] awaiterArray = { localTaskAwaiter };
this.awaiter = awaiterArray;
Action continuation = this.MoveNextDelegate;
if (continuation == null)
{
    Task<int> task = this.builder.Task;
    continuation = MoveNext;
    ((IStateMachine) continuation.Target).SetMoveNextDelegate(continuation);
}
awaiterArray[0].OnCompleted(continuation);
...
return;

(That's just setting up the await, of course - there's then the bit where the result is fetched, but that's less interesting. There's also the matter of doFinallyBodies.)

In the VS11 Beta, it's something this instead (for an awaiter type "AwaiterType" which implements ICriticalNotifyCompletion, in a state machine of type ThisStateMachine).

localAwaiter = x.GetAwaiter();
if (!localAwaiter.IsCompleted)
{
    state = 0;
    awaiterField = localAwaiter;
    builder.AwaitUnsafeOnCompleted<AwaiterType, ThisStateMachine>(ref localAwaiter, ref this);
    doFinallyBodies = false;
    return;
}

If the awaiter type only implements INotifyCompletion, it calls AwaitOnCompleted instead. Note how the calls are generic (but both type variables are constrained to implement appropriate interfaces) which avoids boxing.

The call to the builder will call back to the state machine's SetStateMachine method if this is the first awaiter that hasn't already completed within this execution of the async method. So that handles the section which checked for the continuation being null in the first block of code. Most of the rest of the change is explained by the difference in awaiter types, and obviously AwaitOnCompleted/AwaitUnsafeOnCompleted also ends up calling into OnCompleted on the awaiter itself.

Mutable value type awaiters

(Code is in project 32, MutableAwaiters)

One subtle difference which really shouldn't hurt people but is fun to explore is what happens if you have an awaiter which is a mutable value type. Due to the way awaiters were carefully handled pre-beta, mutations which were conducted as part of OnCompleted would still be visible in GetResult. That's not the case in the beta (as Stephen mentions in his blog post). Mind you, it doesn't mean that all mutations will be ignored... just ones in OnCompleted. A mutation from IsCompleted is still visible, as shown here:

public struct MutableAwaiter : INotifyCompletion
{
     private string message;

     public MutableAwaiter(string message)
     {
         this.message = message;
     }

     public bool IsCompleted
     {
         get
         {
             message = "Set in IsCompleted";
             return false;
         }
     }

     public void OnCompleted(Action action)
     {
         message = "Set in OnCompleted";
         // Ick! Completes inline. Never mind, it's only a demo...
         action();
     }

     public string GetResult()
     {
         return message;
     }
}

What would you expect to be returned from this awaiter? You can verify that all three members are called... but "Set in IsCompleted" is returned. That's because IsCompleted is called before the awaiter value is copied into a field within the state machine. Even though the state machine passes the awaiter by reference, it's passing the local variable, which is of course a separate variable from the field.

I'm absolutely not suggesting that you should rely on any of this behaviour. If you really need to be able to mutate your awaiter, make it a reference type.

Conclusion

The main changes in the Beta are around the interactions between the AsyncTaskMethodBuilder (et al) and the state machine, including the new interfaces for awaiters. There's been quite a bit of optimization, although I still see room for a bit more:

When there's only a single kind of reference type awaiter, the field for storing it could be of that type rather than of type object, removing the need for an execution-time cast
The "stack" variable could be removed in some cases, and made into a specific type in many others
With appropriate optimization flags, local variables which aren't used await expressions could stay local to the state machine instead of being hoisted, and hoisted variables could be aliased in some cases.

One thing which concerns me slightly is how the C# language specification is going to change - the addition of the new interfaces is definitely going to mean more complexity from this previously "tidy" feature. I'm sure it's worth it for the sake of efficiency and the like, but part of me sighs at every added tweak.

So, is this now close to the finished version of async? Only time will tell. I haven't checked whether dynamic awaitables have finally been introduced... if they have, I'll put that in the next post.

Subtleties in API design - member placement

skeet — Wed, 29 Feb 2012 17:27:25 GMT

Noda Time is nearing v1.0, which means I'm spending more time writing documentation than code. It also means reviewing the APIs we've got with a critical eye - whether that's removing extraneous members, adding useful ones, or moving things around. (In particular, writing documentation often suggests where a change would make calling code read more naturally.)

This post is about one particular section of the API, and the choices available. Although I do go into some detail around the specific calls involved, that's just for context... the underlying choices are ones which could be faced when designing any API. I've rarely spent as much time thinking about API decisions as I have with Noda Time, so hopefully this will prove interesting to you even if you really don't care about Noda Time itself as a project.

Introduction: time zones, local date/times and zoned date/times - oh my!

(Okay, so that's not quite as snappy as the Judy Garland version, but hey...)

The area of API we're going to focus on is time zones, and converting between "local" date/time values and "zoned" ones. The three types involved are:

LocalDateTime: a "local" date and time, with no specific time zone. So, something like "7:30 in the evening on February 27th 2012". This means different instants in time in different time zones, of course: if you're arranging a meeting, it's good enough when the attendees are in the same time zone, but not good enough if you're meeting with someone on the other side of the world. (A LocalDateTime also has an associated calendar system, but I'm not going to talk about that much in this post.)
DateTimeZone: a time zone. At its core, this maps any "instant" in time to an offset - the difference between UTC and local time in that time zone. The offset changes over time, typically (but not always) due to daylight saving changes.
ZonedDateTime: a date and time in a particular time zone, with an offset from UTC to avoid ambiguity in some cases (and for efficiency). This identifies a specific instant in time (simply by subtracting the offset from the local date/time). Conceptually this is equivalent to just maintaining the "instant" value, the time zone, and the calendar system - but it's generally cleaner to think of it as a "local" value with an offset from UTC.

If those brief descriptions don't make sense for you at the moment (this sort of thing is quite hard to describe concisely and precisely) you may want to see whether the Noda Time user guide "concepts" page helps.

The API task: mapping from { LocalDateTime, DateTimeZone } to ZonedDateTime

It's easy to get from a ZonedDateTime to a LocalDateTime - you can just use the LocalDateTime property. The difficult bit is the other way round. We obviously want to be able to create a ZonedDateTime from the combination of a LocalDateTime and a DateTimeZone, but the question is where to put this functionality. Three options suggest themselves:

A static method (or constructor) in ZonedDateTime which takes both the time zone and the local date/time as arguments
An instance method on LocalDateTime which just takes the time zone as an argument
An instance method on DateTimeZone which just takes the local date/time as an argument

It gets more complicated though - we're not really talking about one operation here, but potentially several. Although the mapping from instant to offset is unambiguous in DateTimeZone, the mapping from LocalDateTime to offset is not straightforward. There can be 0, 1 or 2 possible results. For example, in the America/Los_Angeles time zone the clocks go forward from 2am to 3am on Sunday March 11th 2012, and go back from 2am to 1am on Sunday 4th November 2012. That means:

The mapping from local date/time to offset at 7.30pm on February 27th 2012 is unambiguous: it's definitely -8 hours (L.A. is 8 hours behind UTC).
The mapping at 2.30am on March 11th 2012 is impossible: at 2am the clocks were put forward to 3am, so 2.30am simply never occurs.
The mapping at 2.30am on November 4th 2012 is ambiguous: it happens once before the clocks go back at 3am, and once afterwards. The offset is either -7 or -8 hours, depending on which occurrence you mean.

When mapping a local time to "global" time, this is something you should really think about. Most APIs obscure the issue, but one of the purposes of Noda Time is to force developers to think about issues which they should be aware of. This one is particularly insidious in that it's the kind of problem which is much more likely to arise when you're asleep than during daylight hours - so it's unlikely to be found during manual testing. (Ditto the day of week - most time zones have daylight saving transitions early on a Sunday morning.)

So, Noda Time exposes four ways of mapping a LocalDateTime and DateTimeZone to a ZonedDateTime:

Exact: if there's a single mapping, return it. Otherwise, throw an exception.
Earlier: if there's a single mapping, return it. If there's more than one, return the earlier one. If the time is skipped, throw an exception.
Later: if there's a single mapping, return it. If there's more than one, return the later one. If the time is skipped, throw an exception.
All information: find out all the information relevant to mapping the given local value - how many matches there are, what they would be, what the time zone information is for each mapping, etc. The caller can then do what they want.

Options available

The question is how we expose these operations. Let's look at some options, then discuss the pros and cons.

Option 1: methods on LocalDateTime

A lot of Noda Time is designed to be "fluent" so it makes a certain amount of sense to be able to take a LocalDateTime, perform some arithmetic on it, then convert it to a ZonedDateTime, then (say) format it. So we could have something like:

var zoned = local.InZone(zone); // implicitly exact
var zoned = local.InZoneOrEarlier(zone);
var zoned = local.InZoneOrLater(zone);
var mapping = local.MapToZone(zone);

Option 2: methods on DateTimeZone

All the calculations involved are really about the time zone - the local date/time value is just a simple value as far as most of this is concerned. So we can put the methods on DateTimeZone instead:

var zoned = zone.AtExactly(local);
var zoned = zone.AtEarlier(local);
var zoned = zone.AtLater(local);
var mapping = zone.MapLocalDateTime(local);

Option 3: methods (or constructors) on ZonedDateTime

Maybe we consider the two inputs to be fairly equal, but the result type is more important:

var zoned = ZonedDateTime.FromLocal(zone, local);
var zoned = ZonedDateTime.FromLocalOrEarlier(zone, local);
var zoned = ZonedDateTime.FromLocalOrLater(zone, local);
var mapping = ZoneLocalMapping.FromLocal(local)

(I'm not terribly happy about the names here; there could be better ones of course.)

Variation a: any of the above options, but with an enum for ambiguity resolution

We don't really need four methods on any of these APIs; the first three only differ by how they handle ambiguity (the situation where a particular local date/time occurs twice). We could use an enum to represent that choice instead:

var zoned = local.InZone(zone, ZoneAmbiguityResolver.Error);
var zoned = local.InZone(zone, ZoneAmbiguityResolver.Earlier);
var zoned = local.InZone(zone, ZoneAmbiguityResolver.Later);

(Or a "smart enum" with behaviour, if we wanted. A normal class type with methods etc, but only a restricted set of valid values.)

Variation b: always go via the mapping result

Given that we already have the idea of getting the full mapping results, we can reduce the API to just one method to return the mapping information, and then put extra methods on that:

var zoned = local.MapInZone(zone).SingleMatch;
var zoned = local.MapInZone(zone).SingleOrEarlier;
var zoned = local.MapInZone(zone).SingleOrLater;

(Again, the names aren't fixed in stone, and the second part could be methods instead of properties if we wanted.)

Variation c: return a sequence of results

If we return a sequence with 0, 1 or 2 ZonedDateTime values, the user can just use LINQ to get the one they want. Again, this can apply wherever we decide to put the method:

var zoned = zone.At(local).Single();
var zoned = zone.At(local).First();
var zoned = zone.At(local).Last();

So, it looks like we effectively have two mostly-orthogonal decisions here:

Where to "start" the conversion - the target type for the method call
How to represent the multiple options

We'll consider them separately.

Regarding the "source" type

To start with, I'll reveal my bias: the existing implementation is option 2 (four methods on DateTimeZone). This was after a small amount of debate on the Noda Time mailing list, and this was the most relevant bit of the discussion:

Me (before going with the current implementation):

It feels a little odd to me to use the zone as the principal class here - just in terms of usability. It makes total sense in terms of the logic, but I tend to think of having a LocalDateTime first, and then converting that to use a particular zone - it's not an operation which feels like it acts on the zone itself.

David N:

I actually feel the opposite: asking a DateTimeZone how a particular LocalDateTime would be represented in that zone feels natural, while asking the LocalDateTime how it would be represented in a zone feels odd. The zone is a first-class entity, with identity and behavior; the LocalDateTime is just a set of values. Why would the LocalDateTime be expected to know how it is represented in a particular zone?

Even though I replied to that in a "maybe" kind of tone, the argument basically convinced me. The trouble is, a colleague was then surprised when he read the documentation around calendar arithmetic and conversions. Surprising users is pretty much a cardinal sin when it comes to API design - and although in this case it was the relatively harmless sort of surprise ("I can't find the member I want in A; oh, it turns out it's in B") rather than a behavioural surprise ("I thought it would do X, but instead it does Y") it's still bad news. I should reveal my colleague's bias too - he has experience of Joda Time, where the relevant call is LocalDateTime.toDateTime(DateTimeZone). (There are calls in DateTimeZone itself, but they're lower-level.)

We've discussed this a fair amount (leading to this blog post) and so far we've concluded that it really depends on how you think about time zones. As a Noda Time user, would you consider them to be rich objects with complex behaviour, or would you think of them as mere "tokens" to be passed around and used without much direct interaction? The two ways of viewing the type aren't necessarily in conflict - I've deliberately designed CalendarSystem to hide its (fairly ugly) innards. There are a few public instance members, but most are internal. But what about time zones?

There's an argument to be made for educating Noda Time users to think about time zones as more complex beasts than just tokens, and I'm happy to do that in other areas (such as choosing which type to use in the first place) but here it feels like it's one step too far. On the other hand, I don't want to stifle users who are thinking of DateTimeZone in that way. In the mailing list thread, David also expressed a dislike for the approach of including functionality in multiple places - and to a certain extent I agree (one of the things I dislike about its API is that it lets you do just about anything with anything)... but in this case it feels like it's justified.

Regardless of how you're thinking about DateTimeZone, it's more likely that you're going to want to use a LocalDateTime value which is the result of some other expression, and then apply some "constant" zone to it, then potentially keep going. If you think about a LINQ-style pipeline of operations, the part that varies in the conversion is much more likely to be the LocalDateTime than the time zone. As such, a method on LocalDateTime allows for a more fluent calling style:

var zoned = parseResult.Value
.PlusMonths(1)
.InZone(LondonTimeZone);

versus:

var zoned = LondonTimeZone.AtExactly(parseResult.Value.PlusMonths(1));

Or to keep the code order the same as the execution order:

var local = parseResult.Value.PlusMonths(1);
var zoned = LondonTimeZone.AtExactly(local);

Obviously the effects become more noticeable the more operations you perform. Personally I'm happy with either the first or third approach - although it's worth being aware that either of the first two have the advantage of being one expression, and therefore easy to use when assigning a static readonly field or something similar.

I'm reasonably happy with having one method on each type, or possibly two (MapLocalDateTime and At*) on DateTimeZone and one (just InZone) on LocalDateTime. I really don't like the idea of having four methods on DateTimeZone and three methods on LocalDateTime. So, let's consider the different variations which cut down the number of methods required.

Expressing "exactly," "earlier," and "later" in one method

This is essentially a discussion of the "variations" above. Just to recap, the possibilities we've come up with are:

Add another parameter to the method to indicate how to handle ambiguities (or impossibilities) - just return a ZonedDateTime
Return a value of a different type (e.g. ZoneLocalMapping) which can be used to get at all the information you could want
Return a sequence of possible ZonedDateTime values, expecting the caller to probably use LINQ's First/Last/Single/FirstOrDefault etc to get at the value they want

The last of these is the only one which gives an easy way of handling the extreme corner case of a local time occurring more than twice - for example, a time zone which goes back one hour at 2am (to 1am) and then goes back another two hours at 3am. I think it's reasonable to dismiss this corner case; however mad time zones can be, I haven't seen anything quite that crazy yet.

At the time of the original discussion, Noda Time was targeting .NET 2.0, which was one reason for not going with the final option here - we couldn't guarantee that LINQ would be available. Now, Noda Time is targeting .NET 3.5 in order to use TimeZoneInfo, but it still doesn't feel like an ideal fit:

Returning a sequence doesn't give information about (say) the two zone intervals "surrounding" a gap
A sequence may be surprising to users who expect just a single value
The exceptions thrown by First, Single etc when their expectations aren't met are very domain-neutral; we can give more information
FirstOrDefault will return the default value for ZonedDateTime in the case of ambiguity. That would be unfortunate, as ZonedDateTime is a value type, and its default value is actually somewhat unhelpful. (It has a null calendar system, effectively. There's not a lot we can do about this, but that's a post for another day.) We could make it a sequence of Nullable<ZonedDateTime> and guarantee that any values in it are actually non-null, but that's really straining things.

Putting these together, there are enough negative points to this idea that I'm happy to rule it out. But what about the first two?

The first has the advantage that the caller only needs to make a single method call, effectively passing in a "magic token" (the ambiguity resolver) which they don't really need to understand. On the other hand, if they want more information, they'll have to call a different method - and I'm not really sure we want to encourage too much of this "magic token" behaviour.

The second has three disadvantages, all fairly slight:

The user may initially expect the result of a method mapping a LocalDateTime to a ZonedDateTime to be a ZonedDateTime... what's this extra intermediate result doing? This is "only" a matter of user education, and it's pretty discoverable. It's an extra concept the user needs to understand, but it's a good concept to understand.
Calling two methods or a method and a property (e.g. zone.MapLocalDateTime(localDateTime).Earlier) may end up being slightly more long-winded than a single method call. I can't get excited about this.
We have to allocate an extra object for the mapping, even when we know it's unique. Usually, this object will become eligible for garbage collection immediately. We could make it a struct, but I don't think it's a natural value type - I'd rather trust that allocating objects in gen0 is pretty cheap.

With the second method, we can replace all the existing methods in DateTimeZone with a single one (or rather, just remove the AtXyz methods, leaving MapLocalDateTime). We can then create pleasantly-named methods on ZoneLocalMapping (which isn't quite right for this purpose at the moment).

Conclusion

This has been an interesting thought experiment for me, and it's suggested some changes I will be applying before v1. We'll see how they pan out. If you want to follow them, look for relevant source code changes.

The important points I've been thinking about are:

What would a new user expect to be available? If they haven't read any documentation, what are they likely to try?
What should the user know about? Where there are important decisions to make, how can we provide guidance?
What would an experienced user (who is already thinking about the Noda Time concepts in the way that we want to encourage) expect to be available?
Where does the balance lie between providing a "too crowded" API (with lots of different ways of achieving the same thing) and a "sparse" API (where there's always one way of achieving a goal, but it may not be the most convenient one for your situation)
How does our choice fit in with other technologies? For example, the final "variation" seems like it plays nicely with LINQ at first - but a few subtleties make it a worse fit than it might seem.
How does this affect performance? (Performance is important in Noda Time - but there would have to be a significant performance problem for me to deviate from an elegant solution.)

So, any other thoughts? Did we miss some options? What other factors should we have taken into consideration?

Currying vs partial function application

skeet — Mon, 30 Jan 2012 18:32:06 GMT

This is a slightly odd post, and before you read it you should probably put yourself into one of three buckets:

Someone who doesn't care too much about functional programming, and finds higher order functions tricky: feel free to skip this post entirely.
Someone who knows all about functional programming, and already knows the difference between currying and partial function application: please read this post carefully and post comments about any inaccuracies you find. (Yes, the CAPTCHA is broken on Chrome; sorry.)
Someone who doesn't know much about functional programming yet, but is interested to learn more: please take this post with a pinch of salt, and read the comments carefully. Read other articles by more experienced developers for more information.

Basically, I've been aware for a while that some people use the terms currying and partial function application somewhat interchangably, when they shouldn't. It's one of those topics (like monads) which I feel I understand to some extent, and I've decided that the best way of making sure I understand it is to try to write about it. If it helps the topic become more accessible to other developers, so much the better.

This post contains no Haskell

Almost every explanation I've ever seen of either topic has given examples in a "proper" functional language, typically Haskell. I have absolutely nothing against Haskell, but I typically find it easier to understand examples in a programming language I understand. I also find it much easier to write examples in a program language I understand, so all the examples in this post are going to be in C#. In fact, it's all available in a single file - that includes all of the examples, admittedly with a few variables renamed. Just compile and run.

C# isn't really a functional language - I know just about enough to understand that delegates aren't really a proper substitute for first class functions. However, they're good enough to demonstrate the principles involved.

While it's possible to demonstrate currying and partial function application using a function (method) taking a very small number of parameters, I've chosen to use 3 for clarity. Although my methods to perform the currying and partial function application will be generic (so all the types of parameters and return value are arbitrary) I'm using a simple function for demonstration purposes:

static string SampleFunction(int a, int b, int c)
{
return string.Format("a={0}; b={1}; c={2}", a, b, c);
}

So far, so simple. There's nothing tricky about that method, so don't look for anything surprising.

What's it all about?

Both currying and partial function application are about converting one sort of function to another. We'll use delegates as an approximation to functions, so if we want to treat the method SampleFunction as a value, we can write:

Func<int, int, int, string> function = SampleFunction;

This single line is useful for two reasons:

Assigning the value to a variable hammers home the point that it really is a value. A delegate instance is an object much like any other, and the value of the function variable is a reference just like any other.
Method group conversions (using just the name of the method as a way of creating a delegate) doesn't work terribly nicely with type inference when calling a generic method.

We can already call the function using three arguments with no further work:

string result = function(1, 2, 3);

Or equivalently:

string result = function.Invoke(1, 2, 3);

(The C# compiler just converts the shorthand of the first form to the second. The IL emitted will be the same.)

That's fine if we've got all the arguments available at the same time, but what if we haven't? To give a concrete (if somewhat contrived) example, suppose we have a logging function with three parameters (source, severity, message) and within a single class (which I'll call BusinessLogic for the moment) we always want to use the same value for the "source" parameter, and we'd like to be able to log easily everywhere in the class specifying just the severity and message. We have a few options:

Create an adapter class which takes the log function (or more generally a logging object) and the "source" value in its constructor, stashes both in instance variables, and exposes a method with two parameters. That method just delegates to the stashed logger, using the source it's remembered to supply the first argument to the three-parameter method. In BusinessLogic we create an instance of the adapter class, and stash a reference in an instance variable - then just call the two-parameter method everywhere we need to. This is probably overkill if we only need the adapter from BusinessLogic, but it's reusable... so long as we're trying to adapt the same logging function.
Store the original logger in our BusinessLogic class, but create a helper method with two parameters. This can hard-code the value used for the "source" parameter in one place (the helper method). If we need to do this in several places, it gets annoying.
Use a more general functional programming approach - probably partial function application in this case.

I'm deliberately ignoring the discrepancy between holding a reference to "the logger" and holding a reference to "the logging function". Obviously there's a significant difference if we need to use more than one function from the logging class, but for the purposes of thinking about currying and partial function application, we'll just think of "a logger" as "a function taking three parameters" (like our sample function).

Now that I've given the slightly-real-world concrete example for a bit of motivation, I'm going to ignore it for the rest of the post, and just talk about our sample function. I don't want to write a whole BusinessLogic class which pretends to do something useful; I'm sure you can perform the appropriate mental conversion from "the sample function" to "something I might actually want to do".

Partial Function Application

The purpose of partial function application is to take a function with N parameters and a value for one of those parameters, and return a function with N-1 parameters, such that calling the result will assemble all the required values appropriately (the 1 argument given to the partial application operation itself, and the N-1 arguments given to the returned function). So in code form, these two calls should be equivalent for our 3-parameter method:

// Normal call
string result1 = function(1, 2, 3);

// Call via partial application
Func<int, int, string> partialFunction = ApplyPartial(function, 1);
string result2 = partialFunction(2, 3);

In this case I've implemented partial application with a single parameter, and chosen the first one - you could write an ApplyPartial method which took more arguments to apply, or which used them somewhere else in the final function execution. I believe that picking off parameters one at a time, from the start, is the most conventional approach.

Thanks to anonymous functions (a lambda expression in this case, but an anonymous method wouldn't be much more verbose), the implementation of ApplyPartial is simple:

static Func<T2, T3, TResult> ApplyPartial<T1, T2, T3, TResult>
(Func<T1, T2, T3, TResult> function, T1 arg1)
{
return (b, c) => function(arg1, b, c);
}

The generics make the method look more complicated than it really is. Note that the lack of higher order types in C# means that you'd need a method like this for every delegate you wanted to use - if you wanted a version for a function which started with four parameters, you'd need an ApplyPartial<T1, T2, T3, T4, TResult> method etc. You'd probably also want a parallel set of methods for the Action delegate family.

The final thing to note is that assuming we had all of these methods, we could perform partial function application again - even potentially down to a parameterless function if we wanted, like this:

Func<int, int, string> partial1 = ApplyPartial(function, 1);
Func<int, string> partial2 = ApplyPartial(partial1, 2);
Func<string> partial3 = ApplyPartial(partial2, 3);
string result = partial3();

Again, only the final line would actually invoke the original function.

Okay, so that's partial function application. That's relatively straightforward. Currying is slightly harder to get your head round, in my view.

Currying

Whereas partial function application converts a function with N parameters into a function with N-1 parameters by applying one argument, currying effectively decomposes the function into functions taking a single parameter. We don't pass any arguments into the Curry method itself:

Curry(f) returns a function f1 such that...
f1(a) returns a function f2 such that...
f2(b) returns a function f3 such that...
f3(c) invokes f(a, b, c)

(Again, note that this is specific to our three-parameter function - but hopefully it's obvious how it would extend to other signatures.)

To give our "equivalence" example again, we can write:

// Normal call
string result1 = function(1, 2, 3);

// Call via currying
Func<int, Func<int, Func<int, string>>> f1 = Curry(function);
Func<int, Func<int, string>> f2 = f1(1);
Func<int, string> f3 = f2(2);
string result2 = f3(3);

// Or to do make all the calls together...
var curried = Curry(function);
string result3 = curried(1)(2)(3);

The difference between the latter examples shows one reason why functional languages often have good type inference and compact representations of function types: that declaration of f1 is pretty fearsome.

Now that we know what the Curry method is meant to do, it's actually surprisingly simple to implement. Indeed, all we need to do is translate the bullet points above into lambda expressions. It's a thing of beauty:

static Func<T1, Func<T2, Func<T3, TResult>>> Curry<T1, T2, T3, TResult>
(Func<T1, T2, T3, TResult> function)
{
return a => b => c => function(a, b, c);
}

If you want to add some brackets to make it clearer for you, feel free - personally I think it just adds clutter. Either way, we get what we want. (It's worth thinking about how annoying it would be to write that without lambda expressions or anonymous methods. Not hard, just annoying.)

So that's currying. I think. Maybe.

Conclusion

I can't say I've ever knowingly used currying, whereas I suspect some bits of Noda Time's text parsing effectively use partial functional application. (If anyone really wants me to dig in and check, I'll do so.)

I really hope I've got the difference between them right here - it feels right, in that the two are clearly related, but also quite distinct. Now that we've reached the end, think about how that difference manifests itself when there are only two parameters, and hopefully you'll see why I chose to use three :)

My gut feeling is that currying is a more useful concept in an academic context, whereas partial functional application is more useful in practice. However, that's the gut feeling of someone who hasn't really used a functional language in anger. If I ever really get round to using F#, maybe I'll do a follow-up post. For now, I'm hoping that my trusty smart readers can provide useful thoughts for others.

Eduasync part 19: ordering by completion, ahead of time...

skeet — Mon, 16 Jan 2012 22:22:43 GMT

Today's post involves the MagicOrdering project in source control (project 28).

When I wrote part 16 of Eduasync, showing composition in the form of majority voting, one reader mailed me a really interesting suggestion. We don't really need to wait for any of the tasks to complete on each iteration of the loop - we only need to wait for the next task to complete. Now that sounds impossible - sure, it's great if we know the completion order of the tasks, but half the point of asynchrony is that many things can be happening at once, and we don't know when they'll complete. However, it's not as silly as it sounds.

If you give me a collection of tasks, I'll give you back another collection of tasks which will return the same results - but I'll order them so that the first returned task will have the same result as whichever of your original tasks completes first, and the second returned task will have the same result as whichever of your original tasks completes second, and so on. They won't be the same tasks as you gave me, reordered - but they'll be tasks with the same results. I'll propagate cancellation, exceptions and so on.

It still sounds impossible... until you realize that I don't have to associate one of my returned tasks with one of your original tasks until it has completed. Before anything has completed, all the tasks look the same. The trick is that as soon as I see one of your tasks complete, I can fetch the result and propagate it to the first of the tasks I've returned to you, using TaskCompletionSource<T>. When the second of your tasks completes, I propagate the result to the second of the returned tasks, etc. This is all quite easy using Task<T>.ContinueWith - barring a few caveats I'll mention later on.

Once we've built a method to do this, we can then really easily build a method which is the async equivalent of Parallel.ForEach (and indeed you could write multiple methods for the various overloads). This will execute a specific action on each task in turn, as it completes... it's like repeatedly calling Task.WhenAny, but we only actually need to wait for one task at a time, because we know that the first task in our "completion ordered" collection will be the first one to complete (duh).

Show me the code!

Enough description - let's look at how we'll demonstrate both methods, and then how we implement them.

private static async Task PrintDelayedRandomTasksAsync()
{
    Random rng = new Random();
    var values = Enumerable.Range(0, 10).Select(_ => rng.Next(3000)).ToList();
    Console.WriteLine("Initial order: {0}", string.Join(" ", values));

    var tasks = values.Select(DelayAsync);

    var ordered = OrderByCompletion(tasks);

    Console.WriteLine("In order of completion:");
    await ForEach(ordered, Console.WriteLine);
}

/// <summary>
/// Returns a task which delays (asynchronously) by the given number of milliseconds,
/// then return that same number back.
/// </summary>
private static async Task<int> DelayAsync(int delayMillis)
{
    await TaskEx.Delay(delayMillis);
    return delayMillis;
}

The idea is that we're going to create 10 tasks which each just wait for some random period of time, and return the same time period back. We'll create them in any old order - but obviously they should complete in (at least roughly) the same order as the returned numbers.

Once we've created the collection of tasks, we'll call OrderByCompletion to create a second collection of tasks, returning the same results but this time in completion order - so ordered.ElementAt(0) will be the first task to complete, for example.

Finally, we call ForEach and pass in the ordered task collection, along with Console.WriteLine as the action to take with each value. We await the resulting Task to mimic blocking until the foreach loop has finished. Note that we could make this a non-async method and just return the task returned by ForEach, given that that's our only await expression and it's right at the end of the method. This would be marginally faster, too - there's no need to build an extra state machine. See Stephen Toub's article about async performance for more information.

ForEach

I'd like to get ForEach out of the way first, as it's so simple: it's literally just iterating over the tasks, awaiting them and propagating the result to the action. We get the "return a task which will wait until we've finished" for free by virtue of making it an async method.

/// <summary>
/// Executes the given action on each of the tasks in turn, in the order of
/// the sequence. The action is passed the result of each task.
/// </summary>
private static async Task ForEach<T>(IEnumerable<Task<T>> tasks, Action<T> action)
{
    foreach (var task in tasks)
    {
        T value = await task;
        action(value);
    }
}

Simple, right? Let's get onto the meat...

OrderByCompletion

This is the tricky bit, and I've actually split it into two methods to make it slightly easier to comprehend. The PropagateResult method feels like it could be useful in other composition methods, too.

The basic plan is:

Copy the input tasks to a list: we need to work out how many there are and iterate over them, so let's make sure we only iterate once
Create a collection of TaskCompletionSource<T> references, one for each input task. Note that we're not associating any particular input task with any particular completion source - we just need the same number of them
Declare an integer to keep track of "the next available completion source"
Attach a continuation to each input task which will be increment the counter we've just declared, and propagate the just-completed task's status
Return a view onto the collection of TaskCompletionSource<T> values, projecting each one to its Task property

Once you're happy with the idea, the implementation isn't too surprising (although it is quite long):

/// <summary>
/// Returns a sequence of tasks which will be observed to complete with the same set
/// of results as the given input tasks, but in the order in which the original tasks complete.
/// </summary>
private static IEnumerable<Task<T>> OrderByCompletion<T>(IEnumerable<Task<T>> inputTasks)
{
    // Copy the input so we know it'll be stable, and we don't evaluate it twice
    var inputTaskList = inputTasks.ToList();

    // Could use Enumerable.Range here, if we wanted...
    var completionSourceList = new List<TaskCompletionSource<T>>(inputTaskList.Count);
    for (int i = 0; i < inputTaskList.Count; i++)
    {
        completionSourceList.Add(new TaskCompletionSource<T>());
    }

    // At any one time, this is "the index of the box we've just filled".
    // It would be nice to make it nextIndex and start with 0, but Interlocked.Increment
    // returns the incremented value...
    int prevIndex = -1;

    // We don't have to create this outside the loop, but it makes it clearer
    // that the continuation is the same for all tasks.
    Action<Task<T>> continuation = completedTask =>
    {
        int index = Interlocked.Increment(ref prevIndex);
        var source = completionSourceList[index];
        PropagateResult(completedTask, source);
    };

    foreach (var inputTask in inputTaskList)
    {
        // TODO: Work out whether TaskScheduler.Default is really the right one to use.
        inputTask.ContinueWith(continuation,
                               CancellationToken.None,
                               TaskContinuationOptions.ExecuteSynchronously,
                               TaskScheduler.Default);
    }

    return completionSourceList.Select(source => source.Task);
}

/// <summary>
/// Propagates the status of the given task (which must be completed) to a task completion source
/// (which should not be).
/// </summary>
private static void PropagateResult<T>(Task<T> completedTask,
    TaskCompletionSource<T> completionSource)
{
    switch (completedTask.Status)
    {
        case TaskStatus.Canceled:
            completionSource.TrySetCanceled();
            break;
        case TaskStatus.Faulted:
            completionSource.TrySetException(completedTask.Exception.InnerExceptions);
            break;
        case TaskStatus.RanToCompletion:
            completionSource.TrySetResult(completedTask.Result);
            break;
        default:
            // TODO: Work out whether this is really appropriate. Could set
            // an exception in the completion source, of course...
            throw new ArgumentException("Task was not completed");
    }
}

You'll notice there are a couple of TODO comments there. The exception in PropagateResult really shouldn't happen - the continuation shouldn't be called when the task hasn't completed. I still need to think carefully about how tasks should propagate exceptions though.

The arguments to ContinueWith are more tricky: working through my TimeMachine class and some unit tests with Bill Wagner last week showed just how little I know about how SynchronizationContext, the task awaiters, task schedulers, and TaskContinuationOptions.ExecuteSynchronously all interact. I would definitely need to look into that more deeply before TimeMachine was really ready for heavy use... which means you should probably be looking at the TPL in more depth too.

Conclusion

Sure enough, when you run the code, the results appear in order, as the tasks complete. Here's one sample of the output:

Initial order: 335 468 1842 1991 2512 2603 270 2854 1972 1327
In order of completion:
270
335
468
1327
1842
1972
1991
2512
2603
2854

TODOs aside, the code in this post is remarkable (which I can say with modesty, as I've only refactored it from the code sent to me by another reader and Stephen Toub). It makes me smile every time I think about the seemingly-impossible job it accomplishes. I suspect this approach could be useful in any number of composition blocks - it's definitely one to remember.

Coding in the style of Glee

skeet — Sun, 15 Jan 2012 20:07:55 GMT

As previously mentioned, at CodeMash 2012 I gave a very silly Pecha Kucha talk entitled "Coding in the style of Glee". The video is on YouTube, or can be seen embedded below:

(There's also another YouTube video from a different angle.)

This post gives the 20 slides (which were just text; no fancy pictures unlike my competitors) and what I meant to say about them. (Edited very slightly to remove a couple of CodeMash-specific in-jokes.) Don't forget that each slide was only up for 20 seconds.

Coding in the style of Glee

As you may know, I’m from the UK, and it’s wonderful to be here. This is my first US conference, so it’s great to be in the country which has shared with the world its most marvellous cultural output: the Fox show, Glee.

At first I watched it just for surface story – but now I know better – I know that really, the songs are all about the culture and practice of coding.

(It isn't easy) Bein' Green

When I started coding, it was on a ZX Spectrum, in Basic. It was hard, but the computer came with a great manual. I later learned C from a ringbinder of some course or other – and entirely failed to understand half the language. Of course, this was before Stack Overflow, when it was really hard being a newbie – where could you turn for information?

Getting to know you

Over time I became semi-competent in C, with the help of friends. But learning is a constant process, of course – getting to know new languages and platforms is just part of a good dev's life every day.

Learning itself is a skill – how similar it is to getting to know small children, I leave to your imagination.

Man in the Mirror

Glee doesn’t just talk about the coding experience, of course – it talks about specific technologies. This Michael Jackson song is talking about reflection, of course. Although the idea wasn’t new in Java, it was new to me – and now it would be almost unthinkable to come up with a new platform which didn’t let you find out about what was in the code.

Bridge over Troubled Water

Another technology covered beautifully by Glee is the interop. We’re in a big world, and we always need to talk to other systems. Whether it’s over JNI (heaven help you), P/Invoke, SOAP, REST – whatever, I hope next time you connect to another system, you’ll hear this haunting Simon and Garfunkel melody in the background.

I will survive

And who could forget persistence frameworks. I’m not sure whether Gloria Gaynor had Hibernate and the Entity Framework in mind when she sang this, but I’m utterly convinced that the Glee writers did. When you submit your data, it’s just got to survive – what else would you want?

You can't always get what you want

We’d all like perfect specifications, reliable libraries, ideal languages, etc – but that’s just not going to happen. It’s possible that of course you won’t get what you need – even if you try real hard. But hey, you might just.

Lean on me (or Agile on me)

(I didn't actually have notes written for this one. Copied from the video.)

Glee sympathizes with you - but it also have a bit of an answer: lean on me. Lean and agile development, so we can adapt to constantly changing specifications, and eventually we will have something that is useful. Maybe nothing like what we initially envisaged, but it will be something useful.

Losing my Religion

Of course, we don’t always stay in love with a platform. I’d like to dedicate this slight to Enterprise Java. Fortunately I never had to deal with Enterprise Java Beans, but I “enjoyed” enough other J2EE APIs to make me yearn for a world without BeanProcessorFactoryFactories.

Anything Goes

Now I’m pretty conservative – only in terms of coding, mind you. I’m a statically typed language guy. But Glee celebrates dynamic languages too – languages where really, anything goes until you try to execute it. Even though I haven’t gone down the dynamic route myself much, it’s important that we all welcome the diversity of languages and platforms we have.

Get Happy

Along with the rise of dynamic languages, we seem to have seen a rise of happy developers. We’ve always had enthusiastic developers, but there’s a certain air about your typical Ruby on Rails developer which feels new to me. Again, I’m not a Ruby fan myself – but it’s always nice to see other happy people, and maybe one day I’ll see the light.

Bust your Windows

I don’t know what I can say about this song. Do the Glee writers have it in for Microsoft? I don’t remember “Bust your OSX” or “Bust your Linux” for example. Only Windows is targeted here.

The Safety Dance

One big change for me since joining Google is increased awareness of the need for redundancy – the intricate dance we need to perform to create a service which will stay up no matter what. Where redundancy is a dirty word in most of industry, as developers we celebrate it – and will do anything we can to avoid…

The Only Exception

… a single point of failure.

(Yes, that really is all I'd prepared for that slide. Hence the need for improvisation.)

Telephone

(From video.) Glee celebrates the rise of phone apps. Who these days could be unaware of the importance of the development of mobile applications? And obviously, we can credit the iPhone for that, but since the iPhone, and just smart phone apps, we've also started...

U Can't Touch This

(From video.) Tablets! And touch screen devices of all kinds. So Windows 8 - very touch-based, and sooner or later we're all going to have to get with it. I don't do UIs, I've never done a touch UI in my life, I have no idea how it works. But clearly it's one of the ways forward.

Forget You

As smart phones and tablets become more ubiquitous and more bound to us as people, privacy has become more important. Glee gave us a timely reminder of the reverse of the persistence early on: we need to be able to forget about users, as well as remember them.

(A)waiting for a girl like you

(From video.) I'd like to leave on an up-note, so: I'm clearly very, very excited (really, really excited) about C# 5 and its await keyword so I ask you - I beg you - be excited about development. And always bear in mind your users.

My life would suck without you

Users rule our world. Can’t live without them, can’t shoot ‘em.

Celebrate – we do stuff to make users really happy! This is awesome! We should be thrilled!

(Even for enterprise apps, we’re doing useful stuff. Honest.)

Don't stop believing

(From video.) So to sum up: have fun, keep learning, really, really enjoy what you're doing, and... don't stop!

CodeMash 2012 report

skeet — Sun, 15 Jan 2012 19:38:42 GMT

I'm nearly home - on a bus back from Heathrow airport to Reading - returning from CodeMash 2012. This was my first US conference, and I had a wonderful time. It was pretty densely packed in terms of presenting / recording for me:

I presented two back-to-back sessions jointly with Bill Wagner, on async. These went down really well (particularly Bill's genius idea of using the Doctor Who quote about time being a "big ball of wibbly-wobbly, timey-wimey stuff") and were great fun to give. Bill's a class act, and I think we got the balance between use and underpinnings about right.
I recorded a podcast with Scott Hanselman (we were going to record two, but the first one ended up being longer than expected)
I presented a talk on "C#'s Greatest Mistakes" which ended up being somewhere between a discussion on language design, and a demonstration of surprising "features" of C#. It overran by 15 minutes without me coming close to running out of things to say, but hopefully it was useful. It was a somewhat rambly session, but at least I warned folks of that up-front. It would be nice to be able to present the same sort of material in a really "tight" way, but I'm just not sure how to.
I gave a 20x20 "Pecha Kucha" talk called "Coding in the style of Glee" as the silliest topic I could come up with on short notice. This was absolutely terrifying and extremely silly. I only came third in the contest (and the winner, Leon, was simply phenomenal) but I was happy that I'd only embarrassed myself about as much as I'd expected to. The YouTube video of this is already up, and I'll write a blog post with the slide titles and what I was trying to say in them :)

Unfortunately due to last minute async prep and desperately trying to cobble together slides for the Glee talk, I didn't have time to actually attend as many talks as I'd have liked. Even though I was present for the whole of the Scala Koans session in the PreCompilr on Wednesday, I found myself next to Bruce Eckel, and ended up chatting with him for most of the time. It would have been a bit of a waste not to, really. (And at least some of that talking was Scala-related...) I also watched the whole of the SignalR presentation by Brady Gaster - where I was apparently the only person in the room with an aversion to "dynamic" in C# 4. That made me the butt of a few jokes, but not too many for comfort.

In terms of C#-related talks, I went to the first half of Dustin Campbell's Roslyn session, but was somewhat distracted by putting together Glee slides and had to leave half way through to hand them in. My final session of the conference was Bill Wagner's "Stunt coding in C# - I dare you to try this at home" which was excellent, and a very fitting end to the conference for me.

Highlights of the conference for me:

Messing with Bill Wagner's code at the end of not just our joint async talk but also his Stunt Coding talk. I've never before asked a presented whether they mind me just stealing the keyboard, but I was confident that Bill (and the attendees) would get a kick out of it - and the code was nicer afterwards :)
Meeting so many people... some that I've met before (I hadn't seen Ted Neward since I gate-crashed a party at his house after the MVP 2005 Summit), some I'd met virtually but not physically before (like Bill) and there loads of other folks I'd never known at all before - including Cori Drew. Cori simply seemed to pop up everywhere - I swear she had about 20 clones at CodeMash. (She also recorded the video of the Glee talk, and it's her laughter you can hear - thanks very much, Cori!) Everyone at the conference was incredibly friendly, and I was really touched by how many people said on the last day that they'd appreciated me making the long trip.
Confounding Dustin Campbell and Kevin Pilch-Bisson with my evil generic overloading puzzle. Just to be clear, these are two seriously smart guys and this was a friendly over-lunch challenge. It's always a privilege to meet more of the team responsible for C# and Visual Studio.
The number of families who came - this is something I've never seen at other conferences, and it really made a difference in terms of the atmosphere of the non-dev bits. It was fabulous to see the kids in the water park, for example. Even out of just attendees, there was a greater proportion of women at CodeMash than at other conferences I've been to - obviously still vastly outnumbered by the men, but it was nice to see some improvement on that front.

This will probably be my only international conference for 2012, so it's a good job that it was so wonderfully organized. I really hope I have the chance to attend next year too. Many thanks to everyone who helped make it such a special conference.

Jon Skeet: Coding Blog

Book Review: Async in C# 5.0

New tool to play with: SemanticMerge

The Open-Closed Principle, in review

Background

Reading material

So what is it? (Part 1 – high level)

So what is it? (Part 2 – more detail)

Protected Variation

Conclusion

Note for new readers

C# in Depth 3rd edition available for early access, plus a discount code…

Fun with Object and Collection Initializers

Is it an object initializer? Is it a collection initializer? No, it’s a syntax error!

Is it an object initializer? Is it a collection initializer? (Parenthetically speaking...)

Stack Overflow question checklist

Noda Time v1.0 released

How can I enumerate thee? Let me count the ways...

If text is declared as IEnumerable...

If text is declared as IEnumerable<char>...

If text is declared as string...

What really happens when text is declared as string...

Conclusion

Stack Overflow and personal emails

The future of "C# in Depth"

The perils of conditional mutability

First bug: Mono's invariant culture isn't terribly invariant...

Second bug: ReadOnly losing changes

Conclusion

More fun with DateTime

Background: daylight saving transitions and ambiguous times

DateTime.Kind and conversions

DateTime's Deep Dark Secret

Unkind comparisons

Conclusions

Type initializer circular dependencies

Back in the real world...

Testing type initialization

Fixing type initializer cycles

Conclusion

Appendix: Testing code

TypeInitializationChecker

TypeInitializationTest

Diagnosing weird problems - a Stack Overflow case study

Reproducing the problem

Initial checks around the file

Desperate times call for stupid measures

Proving the root cause

Explaining the root cause

Conclusion

Eduasync 20: Changes between the VS11 Preview and the Visual Studio 11 Beta

Awaiter pattern changes

Skeleton method changes

Task caching

State machine interface changes

State machine field changes

Local variable hoisting

Awaiter fields

MoveNext method changes

Mutable value type awaiters

Conclusion

Subtleties in API design - member placement

Introduction: time zones, local date/times and zoned date/times - oh my!

The API task: mapping from { LocalDateTime, DateTimeZone } to ZonedDateTime

Options available

Option 1: methods on LocalDateTime

Option 2: methods on DateTimeZone

Option 3: methods (or constructors) on ZonedDateTime

Variation a: any of the above options, but with an enum for ambiguity resolution

Variation b: always go via the mapping result

Variation c: return a sequence of results

Regarding the "source" type

Expressing "exactly," "earlier," and "later" in one method

Conclusion

Currying vs partial function application

This post contains no Haskell

What's it all about?

Partial Function Application

Currying

Conclusion

If `text` is declared as `IEnumerable`...

If `text` is declared as `IEnumerable<char>`...

If `text` is declared as `string`...

What really happens when `text` is declared as `string`...