Nito Programming

Out for a Bit

2011-12-01T00:46:00.001-05:00

There are several things that I was planning to do over the last week or two:

Collect my slides, notes, and demos for my "Thread is Dead" talk recently given at GRDevDay.
Update Nito.AsyncEx to support Silverlight 5 and possibly also Windows Phone.
Finish my "command line parsing" series of blog posts, and start a new series looking at "async in the real world" - essentially my "Thread is Dead" talk broken up into a couple dozen posts.

Unfortunately, sometimes the unexpected happens. My two-year-old son has been diagnosed with Leukemia, and I am writing this far from home in his hospital room.

I do still plan to do all of the things listed above, but they won't get done as soon as I was hoping. My apologies especially to the GRDevDay people!

Virtualizing Two Machines over Thanksgiving

2011-11-26T15:28:00.000-05:00

Like many "computer people," I do a lot of admin work for friends and familiy. Over the last few years, I've worked with my church to get them out of the dark ages of computing. The process is almost complete; I only have one more machine to replace, and then they will all be 64-bit dual-core 4GB systems running Pro editions of Windows. Next year I hope to (finally) put in a domain.

It turns out that two of the old machines have some outdated software that's critical to weekly operations. I'm working on replacements for the software, but in the meantime, the old machines were just sitting around, taking up space in the church office.

I decided to try to virtualize these machines on Friday (the day after Thanksgiving). This blog entry is just a "lessons learned" from this adventure.

The Challenge, and the Plan

The old "server" (XP Home with 192 MB RAM) and the old office machine (XP Home with 256 MB RAM) both needed virtualization. Due to the way the weekly process is done in the office, the old server would have to be virtualized onto the new server, and the old office machine would have to be virtualized onto the new office machine.

I'm most familiar with VMWare products (particularly VMWare Workstation), and I highly recommend them. However, I wanted to see if it was possible to virtualize these machines without incurring a licensing cost. My budget at Landmark Baptist isn't comparable to most IT departments. ;) So, I decided to try Hyper-V or Virtual PC, falling back on VirtualBox if necessary (it wasn't).

The server was the first machine to be replaced, so unfortunately at this point the new server has the most outdated hardware/OS. It's running Server 2008 but without Hyper-V... or even CPU virtualization support. :( Furthermore, according to what I've read, Hyper-V doesn't support USB, which IMO is a significant limitation (and a showstopper for the old "server").

So, I decided to try using Virtual PC for both virtual machines. The new office machine runs Win7 Pro, which is fully supported by the current version of Virtual PC ("Windows Virtual PC"). I was a bit apprehensive about the new server; Server 2008 isn't an officially supported platform for the previous version of Virtual PC ("Microsoft Virtual PC 2007 SP1"), but it turned out to work fine. Microsoft still has Virtual PC 2007 available for download, and SP1 added support for machines without virtualization hardware (which is just what I needed).

One limitation with Virtual PC is that it can only handle 127 GB hard drives. In my case, both machines had hard drives much smaller than that, so it wasn't a problem.

The plan at this point was to virtualize each machine to a different version of Virtual PC (running on different OSes and hardware). We'll see how well this worked in a moment, but first I'll mention the tool which kicked off this whole adventure.

Systems Internals has a great tool called disk2vhd, which can create a virtual disk from a physical disk - even storing the virtual disk image on the physical disk it's imaging, while the physical disk is running the OS running disk2vhd. If you think about it, that's pretty cool.

Disk2vhd can take quite a while (i.e., 8-10 hours) to run, so I tried to make my plan where it would run overnight. Once I have the machines in a VHD image, I should be able to create a Virtual PC machine using that for a hard drive. VirtualBox also supports VHD, so my fallback would be ready just in case.

There are several articles on the Internet where others have successfully converted a physical XP machine to a virtual PC on Windows 7. The steps are straightforward: Create a disk image using disk2vhd; copy the image to the host PC; set up a new virtual machine in Virtual PC; re-activate Windows on the virtual machine; and install Integration Components/Services.

One final note: during my preparations, I discovered that XP can run into a stop 0x7B when backing up to a disk image and restoring on different hardware (which is very similar to what I'm doing with disk2vhd). The steps to fix this are in KB314082. I did not run into this issue, but I'm including it here for others who may.

On Wednesday (the day before Thanksgiving), I had done all the research and established my plan. I downloaded disk2vhd, VirtualBox, and both versions of Virtual PC onto my USB drive and left for Petoskey. That night, I started both machines running disk2vhd and went over to my Mom's for Thanksgiving.

A Snag: OEM OS

I popped in to check the status on Thursday morning. The server disk2vhd failed; my external USB drive had a faulty power adapter and it had shorted out overnight. So I restarted it with my other USB drive, and turned my attention to the office machine.

I had noticed on the disk2vhd download page that OEM OS licenses prevent virtualization. Turns out the office machine was XP Home OEM. The VHD came out fine, but it was not possible to re-activate Windows on the virtual machine. I did have a spare XP Home Retail key, but apparently you can't activate an OEM install with a Retail key. I also tried the original OEM key, but that didn't work since it's keyed to the BIOS which is different in a virtual machine.

Re-installing the OS was out of the question (if I actually had the install media for the outdated programs, I would have installed them on the new machine and we wouldn't need to virtualize in the first place). In desperation, I searched online for any way to convert OEM to Retail in-place. Most of the articles recommended running a repair from a different CD, but that seemed hokey to me (how would that affect updates already installed?).

Finally, I discovered the Product Key Update Tool. I ran it on the old office machine, converting it from OEM to Retail, and then re-started disk2vhd. This time, I ran disk2vhd with the output disk image going directly over the network to the new host PC; this worked just fine and I highly recommend it.

During my searching, I also discovered sysprep. The Product Key Update Tool changes the old key to a new key; whereas Sysprep removes the existing key, requiring the user to type it in the next time the computer boots. I used the Update Tool, but Sysprep would probably also work.

Another Snag: Remote Control

I was hoping to do most of the work on Friday from the comfort of my Mom's living room, eating Thanksgiving leftovers and watching the kids play with their uncles. Unfortunately, I could not get mouse capture to work at all remotely before Integration Services were installed.

It doesn't appear to be possible to set up a new virtual machine remotely. At least not using LogMeIn, which is my remote control software of choice; in the past I've used pcAnywhere, UltraVNC, and Windows Live Mesh, but I've now settled solidly on LogMeIn.

I also tried to LogMeIn into another computer and Remote Desktop to the Virtual PC host; however, the mouse capture was still funky (the scale was messed up). Once I got Integration Services installed, remotely controlling a host PC worked fine.

So, I ended up having to physically be present for the initial virtual machine setup, which was disappointing.

Another Snag: Networking

When I brought up the old "server" as a virtual machine on the new server, the networking didn't work. Since I only had the Windows Activation UI available, it wasn't possible to diagnose. By default, Virtual PC will share the host's network card (using a network switch in software). The new server had a static IP, but this shouldn't have caused a problem. When I switched it to use NAT (a network router in software), the problem went away.

I've always used NAT for my VMWare virtual machines, so this was a natural step.

Issue: Slow Initial Boot

The "server" disk2vhd process never finished. I'm not entirely sure why; the disk file was approximately the correct size, but disk2vhd never completed. Eventually I just exited the program and decided to try to use the file anyway.

When starting the old server as a virtual machine for the first time, it took about an hour to get from initial startup, through Windows activation, and to the desktop. Virtual PC was pegging the CPU the entire time. I'm unsure of the reason for this; the host PC does not have virtualization hardware, the vhd could be incomplete, the vhd is dynamic, ...

Once I installed Integration Components and rebooted, the CPU problems disappeared. I can't say whether the resolution was due to the installation or the rebooting.

Issue: Integration Components on XP Home

The virtualized XP office machine is running on Windows Virtual PC under a Windows 7 Pro host. Normally, this situation allows a really neat trick: you can set up a program on the virtual machine so it looks like a program on the host, with its own Start menu entry, running in a regular window instead of a full virtual machine desktop, etc.

Unfortunately, that does not work if the virtual machine is XP Home. Apparently, the Integration Components use RDP (Remote Desktop) for that functionality. The auto-login feature is also not available.

Conclusion

The project was completed, though it took longer than I expected. I'll find out next week if everything works sufficiently on the virtualized machines.

Lessons learned:

You cannot virtualize an OEM install. You have to change it to a Retail install first, using the Product Key Update Tool.
Disk2vhd can target a vhd image over the network.
You must be physically present to set up the virtual machines, at least until the point that Integration Services are installed.
If you're having problems getting the virtual machine on the network, try using NAT.
Some Integration Components features do not work if the guest is XP Home.

Option Parsing: Case Sensitivity

2011-10-13T12:30:00.000-04:00

By default, all option parsing is case-sensitive:

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option("name", 'n')]
    public string Name { get; private set; }
  }

  static int Main(string[] args)
  {
    try
    {
      var options = OptionParser.Parse<Options>();
      Console.WriteLine("Name: " + options.Name);
      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe /name Bob
Name: Bob

> CommandLineParsingTest.exe /Name Bob
Unknown option  Name  in parameter  /Name

This is normal for Unix users, but Windows users expect case-insensitivity. You can pass your own StringComparer to the Parse method to support case-insensitivity:

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option("name", 'n')]
    public string Name { get; private set; }
  }

  static int Main(string[] args)
  {
    try
    {
      var options = OptionParser.Parse<Options>(stringComparer:StringComparer.CurrentCultureIgnoreCase);
      Console.WriteLine("Name: " + options.Name);
      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe /name Bob
Name: Bob

> CommandLineParsingTest.exe /Name Bob
Name: Bob

Option Parsing; Positional Arguments

2011-10-06T12:30:00.000-04:00

"Positional arguments" are any arguments not associated with an option. When using the Nito.KitchenSink option parsing library, positional arguments must come after any options and their arguments.

Individual Positional Arguments

You can use the PositionalArgumentAttribute to specify positional arguments in your options class. This attribute takes a single integral parameter, the 0-based index of the positional argument.

Positional arguments support the entire range of parsing possibilities, including SimpleParserAttribute.

This example uses a regular Level option along with a Name positional parameter.

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option('l')]
    public int? Level { get; set; }

    [PositionalArgument(0)]
    public string Name { get; set; }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine("Level: " + options.Level);
      Console.WriteLine("Name: " + options.Name);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
Level:
Name:

> CommandLineParsingTest.exe Bob
Level:
Name: Bob

> CommandLineParsingTest.exe -l 13
Level: 13
Name:

> CommandLineParsingTest.exe -l 13 Bob
Level: 13
Name: Bob

> CommandLineParsingTest.exe Bob -l 13
Unknown parameter  -l

The last test above shows that positional arguments must come after all regular options.

If you need to pass a positional argument that starts with a dash (-) or forward slash (/), you can pass the special option "--", which forces all remaining command-line arguments to be interpreted as positional arguments:

> CommandLineParsingTest.exe -Negative
Unknown option  N  in parameter  -Negative

> CommandLineParsingTest.exe -- -Negative
Level:
Name: -Negative

The Positional Argument Collection

Every options class must have one property that can receive "extra" positional arguments. Extra positional arguments are any positional arguments after those defined by PositionalArgumentAttribute.

Most programs do not need this functionality, so the OptionArgumentsBase class provides a simple collection called AdditionalArguments. By default, OptionArgumentsBase.Validate will throw an UnknownOptionException if any positional arguments end up in that collection.

A program may make use of the AdditionalArguments collection by overriding Validate:

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [PositionalArgument(0)]
    public string Name { get; set; }

    public override void Validate()
    {
    }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine("Name: " + options.Name);
      Console.WriteLine("ArgList: " + string.Join(", ", options.AdditionalArguments));

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
Name:
ArgList:

> CommandLineParsingTest.exe Bob
Name: Bob
ArgList:

> CommandLineParsingTest.exe Bob 17
Name: Bob
ArgList: 17

> CommandLineParsingTest.exe Bob -l 13
Name: Bob
ArgList: -l, 13

> CommandLineParsingTest.exe -- Bob
Name: Bob
ArgList:

Alternatively, an options class may provide its own collection, marked with the PositionalArgumentsAttribute (note the plural "Arguments"). When it does this, the options class may not derive from OptionArgumentsBase; rather, it should implement the IOptionArguments interface.

The property does not have to be List<string> (which is used by OptionArgumentsBase). The only requirements on the collection is that it only have one method named Add which takes a single parameter. The parameter does not have to be string; it can be any type, and the standard parsing rules apply.

This means that PositionalArguments can be placed on a property of dictionary type, as long as a matching parser is provided.

Here's an example of a program taking any number of integer parameters:

class Program
{
  private sealed class Options : IOptionArguments
  {
    public Options()
    {
      this.Integers = new List<int>();
    }

    [PositionalArguments]
    public List<int> Integers { get; private set; }

    public void Validate()
    {
    }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine("Integers: " + string.Join(", ", options.Integers));

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
Integers:

> CommandLineParsingTest.exe 13
Integers: 13

> CommandLineParsingTest.exe 13 7
Integers: 13, 7

> CommandLineParsingTest.exe 13 7 Bob
Could not parse  Bob  as Int32

Rx and Async

2011-09-16T09:00:00.000-04:00

I saw some rather shocking tweets yesterday from the BUILD conference:

The author of that original tweet followed up with a blog post with some interesting Rx-related quotes from Anders Hejlsberg: "I don't know if we've decided [whether Rx will be included in future versions of .NET]." and "Personally I've found the stuff we've done with async allows you to do a lot more [than Rx]." (emphasis mine).

Interesting. I tried to take a listen for myself, but the Channel9 live interview was no longer available. Note that these remarks were made during a live interview, and were not part of a presentation; I'm hoping that Anders just answered off the cuff and didn't mean it.

One reason I found those quotes controversial is because parallel programming (TPL/PLINQ), background operations (async/await), and asynchronous streams (Rx) all address different problems. In particular, Async only supports background operations and does not support asynchronous streams. Rx supports both, but Async will become the default solution for background operations because it's easier to use than Rx.

So, I agree with Anders that Async is easier to use, but I totally disagree that Async is more powerful. Rx can do everything Async can do, and can do some things that Async can't do.

It comes down to the difference between asynchronous operations and asynchronous events. An asynchronous operation is something that my program can start, and it will complete some time later. An asynchronous event stream is something that is happening all the time independent of my program; it can subscribe and unsubscribe, but does not cause the events. This is an important distinction if you consider an event stream that produces in quick bursts (e.g., mouse movement); Rx allows collating all of those events, but an async-based solution may miss some (because it has to restart the operation each time it completes).

Historically, asynchronous events have been a blind spot for Microsoft. Consider a condensed history of asynchronous support:

Asynchronous Programming Model (APM). In the beginning, there was only IAsyncResult (Begin/End). The APM was everywhere, even baked into delegate types. The thing to note about APM is that it is purely an asynchronous operation; no asynchronous events are supported. The program starts the operation, which has a single point of completion.
Event-Based Asynchronous Pattern (EAP). Way back in .NET 2.0, the EAP was introduced. EAP works by capturing the current SynchronizationContext and then raising events on that context. This was the first asynchronous pattern that supported both asynchronous operations and asynchronous events. Unfortunately, the documentation assumed that EAP objects are only implementing asynchronous operations, and completely ignored the EAP support for asynchronous events. In addition, the most famous EAP implementation (BackgroundWorker) was just an asynchronous operation. However, the Nito.Async library included some helpers for EAP components, and included sample socket components using EAP in an asynchronous event fashion.
Rx. Supporting .NET 3.5 and up, the Rx libraries are all about asynchronous events (and they also support asynchronous operations, which are just a singleton asynchronous event). Rx is also more powerful than EAP because it has a very flexible execution context, while EAP ties everything through a single SynchronizationContext. However, the learning curve for Rx is steep.
Async/await and the Task-Based Asynchronous Pattern (TAP). These extensions to the language allow for a very natural and easy way to deal with asynchronous operations, but they do not support asynchronous events.

In terms of power and flexibility, TAP is approximately equivalent to APM (less powerful than EAP and Rx). The only reason it's a step forward is because it is so easy to learn and use. Some simple programs may use only TAP, but other programs will need both TAP and Rx.

Rx is a very welcome (and necessary) addition to our toolset. Async does not and can not replace it.

(P.S. All of this - and much more - is covered in my "Thread is Dead" talk, which has been submitted for consideration at a couple of conferences in the next few months. I'll update this space when it's accepted.)

Nito.AsyncEx Available

2011-09-09T08:43:00.001-04:00

Nito.AsyncEx is now available.

Just as the Nito.Async library helps you work with the Event-Based Asynchronous Pattern (and its underlying concepts such as SynchronizationContext), the Nito.AsyncEx library helps you work with the Task-Based Asynchronous Pattern (and its underlying concepts such as Task).

The Async CTP "Why Do the Keywords Work THAT Way" Unofficial FAQ

2011-09-01T12:30:00.004-04:00

There's a lot of interest in the Async CTP, with good reason. The Async CTP will make asynchronous programming much, much easier than it has ever been. It's somewhat less powerful but much easier to learn than Rx.

The Async CTP introduces two new keywords, async and await. Asynchronous methods (or lambda expressions) must return void, Task, or Task<TResult>.

This post is not an introduction to the Async CTP; there's plenty of tutorial resources available out there. This post is an attempt to bring together the answers to a few common questions that programmers have when they start using the Async CTP.

Inferring the Return Type

When returning a value from an async method, the method body returns the value directly, but the method itself is declared as returning a Task<TResult>. There is a bit of "disconnect" when you declare a method returning one type and have to return another type:

// Actual syntax
public async Task<int> GetValue()
{
  await TaskEx.Delay(100);
  return 13; // Return type is "int", not "Task<int>"
}

Question: Why can't I write this:

// Hypothetical syntax
public async int GetValue()
{
  await TaskEx.Delay(100);
  return 13; // Return type is "int"
}

Consider: How will the method signature look to callers? Async methods that return a value must have an actual result type of Task<TResult>. So GetValue will show up in IntelliSense as returning Task<TResult> (this would also be true for the object browser, Reflector, etc).

Inferring the return type was considered during the initial design, but the team concluded that the keeping the "disconnect" within the async method was better than spreading the "disconnect" throughout the code base. The "disconnect" is still there, but it's smaller than it could be. The consensus is that a consistent method signature is preferred.

Consider: There is a difference between async void and async Task.

An async Task method is just like any other asynchronous operation, only without a return value. An async void method acts as a "top-level" asynchronous operation. An async Task method may be composed into other async methods using await. An async void method may be used as an event handler. An async void method also has another important property: in an ASP.NET context, it informs the web server that the page is not completed until it returns (see my MSDN article for more information on how this works).

Inferring the return type would remove the distinction between async void and async Task; either all async methods would be async void (preventing composability), or they would all be async Task (preventing them from being event handlers, and requiring an alternative solution for ASP.NET support).

Async Return

There is still a "disconnect" between the method declaration return type and the method body return type. Another option that has been suggested is to add a keyword to return to indicate that the value given to return is not really what's being returned, e.g.:

// Hypothetical syntax
public async Task<int> GetValue()
{
  await TaskEx.Delay(100);
  async return 13; // "async return" means the value will be wrapped in a Task
}

Consider: Converting large amounts of code from synchronous to asynchronous.

The async return keyword was also considered, but it wasn't compelling enough. This is particularly true when converting a lot of synchronous code to asynchronous code (which will be common over the next few years); forcing people to add async to every return statement just seemed like "needless busy-work." It's easier to get used to the "disconnect".

Inferring "async"

The async keyword must be applied to a method that makes use of await. However, it also gives a warning if it is applied to a method that does not make use of await.

Question: Why can't async be inferred based on the presence of await:

// Hypothetical syntax
public Task<int> GetValue()
{
  // The presence of "await" implies that this is an "async" method.
  await TaskEx.Delay(100);
  return 13;
}

Consider: Backwards compatibility and code readability.

Eric Lippert has the definitive post on the subject. It's also been discussed in blog comments, Channel9, and forums.

To summarize, a single-word await keyword would be too big of a breaking change. The choice was between a multi-word await (e.g., await for) or a keyword on the method (async) that would enable the await keyword just within that method. Explicitly marking methods async is easier for both humans and computers to parse, so they decided to go with the async/await pair.

Inferring "await"

Question: Since it makes sense to explicitly include async (see above), why can't await be inferred based on the presence of async:

// Hypothetical syntax
public async Task<int> GetValue()
{
  // "await" is implied, since this is an "async" method.
  TaskEx.Delay(100);
  return 13;
}

Consider: Parallel composition of asynchronous operations.

At first glance, inferring await appears to simplify basic asynchronous operations. As long as all waiting is done in serial (i.e., one operation is awaited, then another, and then another), this works fine. However, it falls apart when one considers parallel composition.

Parallel composition in the Async CTP is done using TaskEx.WhenAny and TaskEx.WhenAll methods. Here's a simple example which starts two operations immediately and asynchronously waits for both of them to complete:

// Actual syntax
public async Task<int> GetValue()
{
  // Asynchronously retrieve two partial values.
  // Note that these are *not* awaited at this time.
  Task<int> part1 = GetValuePart1();
  Task<int> part2 = GetValuePart2();

  // Wait for both values to arrive.
  await TaskEx.WhenAll(part1, part2);

  // Calculate our result.
  int value1 = await part1; // Does not actually wait.
  int value2 = await part2; // Does not actually wait.
  return value1 + value2;
}

In order to do parallel composition, we must have the ability to say we're not going to await an expression.

Option Parsing: Validation

2011-08-25T12:30:00.001-04:00

The option parsing pipeline consists of three steps: lexing, parsing, and validation. So far, we've only talked about the first two steps; today we'll look at validation.

Option argument classes must derive from IOptionArguments, which only has one method:

/// <summary>
/// An arguments class, which uses option attributes on its properties.
/// </summary>
public interface IOptionArguments
{
  /// <summary>
  /// Validates the arguments by throwing <see cref="OptionParsingException"/> errors as necessary.
  /// </summary>
  void Validate();
}

The Validate method should do any validation, and throw an exception if the option argument class properties are not acceptable. The OptionArgumentsBase type includes an implementation of Validate that just does some basic validation (we'll cover it in detail next week). This method may be overridden in derived classes.

Validating Option Values

It's possible to include any logic you need in the Validate method. This example forces an option value to be in the range [0, 3]:

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option("level", 'l')]
    public int Level { get; set; }

    public override void Validate()
    {
      base.Validate();
      if (this.Level < 0 || this.Level > 3)
        throw new OptionParsingException.OptionArgumentException("Level must be in the range [0, 3].");
    }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine("Level: " + options.Level);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
Level: 0

> CommandLineParsingTest.exe -l 3
Level: 3

> CommandLineParsingTest.exe -l 4
Level must be in the range [0, 3].

Other option parsing libraries do validation using various attributes (e.g., the example above would use a [RangeAttribute]). However, using a Validate method is both simpler and more powerful.

Required Options

It's possible to use validation to require an option.

Please note: The technique described here is controversial! In general, people who have designed many command-line interfaces do not recommend required options (at the very least, the terminology is confusing: it's a required optional parameter). Usually, a required option is better represented as a positional argument or a subcommand (both of which will be covered in later blog posts). Consider carefully before using required options.

The example below requires a level to be specified:

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option("level", 'l')]
    public int? Level { get; set; }

    public override void Validate()
    {
      base.Validate();
      if (this.Level == null)
        throw new OptionParsingException.OptionArgumentException("Level must be specified.");
    }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine("Level: " + options.Level);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
Level must be specified.

> CommandLineParsingTest.exe -l 4
Level: 4

> CommandLineParsingTest.exe -l 0
Level: 0

To reiterate, people with much more experience than I recommend against using "required options". They recommend positional arguments or subcommands instead.

Option Parsing: Boolean Options

2011-08-18T12:30:00.005-04:00

Options as Flags

Most options require an option argument. Some options take an optional argument. Then there are the options that take no argument at all. These are the "flag" options - the option value is either set or unset.

Options with no arguments may only be defined on boolean properties. Consider this program, which defines two options (a and b) that do not take arguments, and a third option (c) which takes a required argument:

class ProgramtO
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option('a', Argument = OptionArgument.None)]
    public bool A { get; set; }

    [Option('b', Argument = OptionArgument.None)]
    public bool B { get; set; }

    [Option('c')]
    public bool C { get; set; }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine("A: " + options.A);
      Console.WriteLine("B: " + options.B);
      Console.WriteLine("C: " + options.C);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
A: False
B: False
C: False

> CommandLineParsingTest.exe -a -b
A: True
B: True
C: False

> CommandLineParsingTest.exe -c
Missing argument for option  c

> CommandLineParsingTest.exe -c true
A: False
B: False
C: True

Short Option Runs

Arguments that do not take arguments may be combined on the command line into a "short option run." A short option run must use the short names of the options; it cannot use the long names.

> CommandLineParsingTest.exe -ab
A: True
B: True
C: False

There is no way to pass an argument to an option in a short option run.

> CommandLineParsingTest.exe -ac true
Option  c  cannot be in a short option run (because it takes an argument) in parameter  -ac

> CommandLineParsingTest.exe -ac=true
Invalid parameter  -ac=true

This is a deliberate departure from the behavior of GNU's getopt. Short option runs with arguments are not readable and may cause compatibility problems when the options change.

Inverse Aliases

Some programs prefer the ability to specify an "on" and an "off" version for the same option. This can be easily done by having the boolean properties share a single backing value, with the "off" version inverting its value. These are very similar to aliases, except that they mean the opposite instead of the same.

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    public Options()
    {
      this.B = true;
    }

    [Option("a", Argument = OptionArgument.None)]
    public bool A { get; set; }

    [Option("no-a", Argument = OptionArgument.None)]
    public bool NoA
    {
      get { return !this.A; }
      set { this.A = !value; }
    }

    [Option("b", Argument = OptionArgument.None)]
    public bool B { get; set; }

    [Option("no-b", Argument = OptionArgument.None)]
    public bool NoB
    {
      get { return !this.B; }
      set { this.B = !value; }
    }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine("A: " + options.A);
      Console.WriteLine("B: " + options.B);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
A: False
B: True

> CommandLineParsingTest.exe /a
A: True
B: True

> CommandLineParsingTest.exe /b
A: False
B: True

> CommandLineParsingTest.exe /no-a
A: False
B: True

> CommandLineParsingTest.exe /no-b
A: False
B: False

> CommandLineParsingTest.exe /a /no-a
A: False
B: True

The last example shows that the default overwrite behavior of options produces the expected result: when there are multiple conflicting options on a command line, the last one wins.

Note that the options in this sample do not have short names. They are allowed to have short names, but options with inverse aliases do not usually have short names.

Option Parsing: Argument Parsing

2011-08-11T12:30:00.020-04:00

This is going to be an in-depth post on how argument parsing works in the Nito.KitchenSink.OptionParsing library, and a couple of ways the parsing can be modified.

General Option Argument Parsing Rules

First, a reminder about terminology; in this example, the "v" is the short option name, and the "3" is the option argument:

> CommandLineTest.exe -v 3

Also remember that an option argument may be required for an option, or it may be optional. If you need a refresher, read the earlier post options with optional arguments.

Required option arguments are allowed to begin with a dash (-) or forward-slash (/), but optional option arguments are not. To start an optional option argument with these characters, specify the argument using a full-colon (:) or equals sign (=).

Consider this example program, which just takes two string arguments, one required and one optional:

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option("required", 'r')]
    public string RequiredValue { get; set; }

    [Option("optional", 'o', Argument = OptionArgument.Optional)]
    public string OptionalValue { get; set; }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      if (options.RequiredValue != null)
        Console.WriteLine("Required Value: " + options.RequiredValue);
      if (options.OptionalValue != null)
        Console.WriteLine("Optional Value: " + options.OptionalValue);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe -r a -o b
Required Value: a
Optional Value: b

> CommandLineParsingTest.exe -r /a -o b
Required Value: /a
Optional Value: b

> CommandLineParsingTest.exe -r a -o /b
Unknown option  b  in parameter  /b

> CommandLineParsingTest.exe -o "/b"
Unknown option  b  in parameter  /b

> CommandLineParsingTest.exe -o:/b
Optional Value: /b

> CommandLineParsingTest.exe -o=/b
Optional Value: /b

Note that placing the argument in double-quotes does not allow the argument to start with a dash or forward-slash.

Reminder: the command shell has its own set of reserved characters (&, |, (, ), <, >, and ^). These can be escaped using ^, or they can be wrapped in double-quotes. Command shell escapes are described in more detail in the post on command-line lexing.

Implementing a Simple Argument Parser

Parsing an argument option is done in two steps. The first step is to parse that portion of the command line as a string, using the rules above. The second step is to parse the string into an instance of the corresponding property type on the option arguments class. Since the examples above used a property type of string, there was no processing during the second step.

It is possible to use only a part of the option parsing pipeline to get options and their arguments as strings. Pass a sequence of OptionDefinition instancess and a command line into the parser; the result is a sequence of Option instances (where each argument is typed as string). Details of these types will be covered in a future blog post.

The option parsing library uses a collection of "simple parsers" to convert from a string to a known type. By default, the simple parser collection understands how to parse bool; signed and unsigned 8-bit, 16-bit, 32-bit, and 64-bit integers; BigInteger; single and double-precision floating point; decimal; Guid; TimeSpan; DateTime; and DateTimeOffset. Strings, enumerations and nullable types are treated specially: strings are never parsed, enumerations use Enum.Parse, and nullable types are supported if their corresponding non-nullable types are supported. The built-in parsers all use the standard TryParse methods.

Say, for example, we wanted to accept an argument of type Complex. The Complex type is not included in the default simple parser collection (in fact, it does not even have a Parse or TryParse method!).

If we try to add it to our program, then whatever we pass as the argument value will just fail to parse:

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option("value", 'v')]
    public Complex? Value { get; set; }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      if (options.Value != null)
        Console.WriteLine("Value: " + options.Value);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe -v (3,5)
Could not parse  (3,5)  as Complex

We can create a parser for the Complex type by implementing ISimpleParser. This interface only has two members: the type of the result and a TryParse method.

Once we've implemented our special parser, we need to pass it to the Parse method. To do this, we create a SimpleParserCollection, add our special parser, and pass the collection to the Parse method.

Our solution now looks like this:

class Program
{
  private sealed class ComplexParser : ISimpleParser
  {
    public Type ResultType
    {
      get { return typeof(Complex); }
    }

    public object TryParse(string value)
    {
      // Match the following pattern: '(' double ',' double ')'
      if (value.Length < 5 || value[0] != '(' || value[value.Length - 1] != ')')
        return null;
      var components = value.Substring(1, value.Length - 2).Split(',');
      if (components.Length != 2)
        return null;
      double real, imaginary;
      if (!double.TryParse(components[0], out real))
        return null;
      if (!double.TryParse(components[1], out imaginary))
        return null;
      return new Complex(real, imaginary);
    }
  }


  private sealed class Options : OptionArgumentsBase
  {
    [Option("value", 'v')]
    public Complex? Value { get; set; }
  }

  static int Main()
  {
    try
    {
      var parsers = new SimpleParserCollection();
      parsers.Add(new ComplexParser());
      var options = OptionParser.Parse<Options>(parserCollection: parsers);

      if (options.Value != null)
        Console.WriteLine("Value: " + options.Value);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe -v (3,5)
Value: (3, 5)

We added a custom parser to the collection, and the option parsing library now understands how to parse a new type. We could add any number of Complex properties, and they would all use the new parser.

This is a powerful extension point, but what if we want to modify the way an extisting type is parsed?

Replacing a Simple Argument Parser

The default parsers in a simple parser collection only use the basic TryParse methods, which may not be exactly what is needed. SimpleParserCollection.Add will actually replace the parser for a given type if there is already a parser for that type.

We'll use uint for our example. We want to allow either decimal numbers or hexadecimal numbers prefixed by "0x". System.UInt32.TryParse(string) does not accept hexadecimal numbers:

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option("value", 'v')]
    public uint? Value { get; set; }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      if (options.Value != null)
        Console.WriteLine("Value: " + options.Value);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe -v 11
Value: 11

> CommandLineParsingTest.exe -v 0x11
Could not parse  0x11  as UInt32

Just like the last example, we'll implement our own parser, and we'll add it to the parser collection (replacing the default parser).

class Program
{
  private sealed class UInt32HexParser : ISimpleParser
  {
    public Type ResultType
    {
      get { return typeof(uint); }
    }

    public object TryParse(string value)
    {
      uint ret;
      if (value.StartsWith("0x"))
      {
        if (!uint.TryParse(value.Substring(2), NumberStyles.AllowHexSpecifier, CultureInfo.InvariantCulture, out ret))
          return null;
        return ret;
      }

      if (!uint.TryParse(value, out ret))
        return null;
      return ret;
    }
  }

  private sealed class Options : OptionArgumentsBase
  {
    [Option("value", 'v')]
    public uint? Value { get; set; }
  }

  static int Main()
  {
    try
    {
      var parsers = new SimpleParserCollection();
      parsers.Add(new UInt32HexParser());
      var options = OptionParser.Parse<Options>(parserCollection: parsers);

      if (options.Value != null)
        Console.WriteLine("Value: " + options.Value);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe -v 11
Value: 11

> CommandLineParsingTest.exe -v 0x11
Value: 17

Our program now allows decimal or hexadecimal values for all uint argument values.

These custom parsers can be written for any type, including types specific for your program. The only type they won't work on is string, since the simple parser collection just passes string values straight through.

Overriding the Simple Argument Parser

The examples so far have implemented a custom parser and added it to the parser collection. This changes the parsing behavior for every property of that type. Sometimes we just want to apply a parser to a single property; this can be done by using the SimpleParserAttribute.

This example defines a hex parser (without the "0x" prefix) and then uses it for only one of its properties:

class Program
{
  private sealed class UInt32HexParser : ISimpleParser
  {
    public Type ResultType
    {
      get { return typeof(uint); }
    }

    public object TryParse(string value)
    {
      uint ret;
      if (!uint.TryParse(value, NumberStyles.AllowHexSpecifier, CultureInfo.InvariantCulture, out ret))
        return null;
      return ret;
    }
  }

  private sealed class Options : OptionArgumentsBase
  {
    [Option("hex-value", 'h')]
    [SimpleParser(typeof(UInt32HexParser))]
    public uint? HexValue { get; set; }

    [Option("dec-value", 'd')]
    public uint? DecValue { get; set; }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      if (options.HexValue != null)
        Console.WriteLine("HexValue: " + options.HexValue);
      if (options.DecValue != null)
        Console.WriteLine("DecValue: " + options.DecValue);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe -h 11 -d 11
HexValue: 17
DecValue: 11

Custom Parsers for Multiple Argument Values

Revisiting the problem of multiple argument values, we can use a custom parser for a cleaner solution. This example "sequence parser" uses the default simple parser for int types, which is easier to deal with than int.TryParse:

class Program
{
  private sealed class Int32SequenceParser : ISimpleParser
  {
    public Type ResultType
    {
      get { return typeof(IEnumerable<int>); }
    }

    public object TryParse(string value)
    {
      var values = value.Split(',');
      ISimpleParser defaultParser = new DefaultSimpleParser<int>();
      var result = values.Select(x => defaultParser.TryParse(x));
      if (result.Any(x => x == null))
        return null;
      return result.Cast<int>();
    }
  }

  private sealed class Options : OptionArgumentsBase
  {
    [Option("values", 'v')]
    [SimpleParser(typeof(Int32SequenceParser))]
    public IEnumerable<int> Values { get; set; }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      if (options.Values != null)
        Console.WriteLine("Values: " + string.Join(" ", options.Values));

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe

> CommandLineParsingTest.exe -v 2,3,5,7
Values: 2 3 5 7

> CommandLineParsingTest.exe -v 2,3a,5
Could not parse  2,3a,5  as IEnumerable

> CommandLineParsingTest.exe -v 2,3 -v 5,7
Values: 5 7

The last example above shows that the default behavior of the actual property setter is still overwrite, not append. If you want to allow appending sequences, you'll need to change the setter to append each sequence to an internal collection.

Option Parsing: Preventing Multiple Argument Values

2011-08-04T12:30:00.012-04:00

When dealing with multiple argument values, there are four basic behaviors: overwrite, append, prevent, and ignore.

Last week's post contained a few examples of the append behavior, which is supported by having the property setter place the values into a backing list.

The default behavior in the Nito.KitchenSink option parsing library is to overwrite previous values. In other words, options coming later on the command line may "override" options earlier on the command line. Consider this example:

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option("level", 'l')]
    public int? Level { get; set; }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine("Level: " + options.Level);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
Level:

> CommandLineParsingTest.exe -l 3
Level: 3

> CommandLineParsingTest.exe -l 3 -l 9
Level: 9

This is the default behavior, and is probably what users expect. However, for some options, the prevent or ignore behaviors may make sense.

The prevent and ignore behaviors are closely related. Like last week's post, these behaviors are implemented by placing special code in the property setter.

The prevent behavior can be implemented by having a nullable backing value, and throwing from the setter if it is already set. The only tricky part is choosing the exception to throw from the setter; I recommend throwing an exception derived from OptionParsingException, since that indicates a usage error. Any exception thrown from a property setter will be wrapped in an OptionParsingException.OptionArgumentException (in versions 1.1.2 and newer).

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    private int? level;

    [Option("level", 'l')]
    public int? Level
    {
      get
      {
        return this.level;
      }

      set
      {
        if (this.level.HasValue)
          throw new OptionParsingException.OptionArgumentException("The value may only be specified once.");
        this.level = value;
      }
    }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine("Level: " + options.Level);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
Level:

> CommandLineParsingTest.exe -l 3
Level: 3

> CommandLineParsingTest.exe -l 3 -l 9
The value may only be specified once.

Likewise, the ignore behavior can be implemented by having a nullable backing value, and ignoring the setter if it is already set:

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    private int? level;

    [Option("level", 'l')]
    public int? Level
    {
      get
      {
        return this.level;
      }

      set
      {
        if (!this.level.HasValue)
          this.level = value;
      }
    }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine("Level: " + options.Level);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
Level:

> CommandLineParsingTest.exe -l 3
Level: 3

> CommandLineParsingTest.exe -l 3 -l 9
Level: 3

Note that the ignore behavior may confuse users; most command-line programs use overwrite behavior, which is the default.

Option Parsing: Allowing Multiple Argument Values

2011-07-28T12:30:00.007-04:00

Some options need to take a sequence of argument values. There are several ways to accomplish this using the Nito.KitchenSink Option Parsing library.

Enumeration Flags

If the option values are a series of enumerated flags, then the built-in enumeration parser will handle multiple values automatically:

class Program
{
  [Flags]
  private enum FavoriteThings
  {
    None = 0x0,
    Mittens = 0x1,
    Kittens = 0x2,
    Snowflakes = 0x4,
  }

  private sealed class Options : OptionArgumentsBase
  {
    [Option("favorite-things", 'f')]
    public FavoriteThings FavoriteThings { get; set; }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine(options.FavoriteThings);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
None

> CommandLineParsingTest.exe /favorite-things Mittens
Mittens

> CommandLineParsingTest.exe /favorite-things Mittens,Kittens
Mittens, Kittens

> CommandLineParsingTest.exe /favorite-things "Mittens, Snowflakes"
Mittens, Snowflakes

> CommandLineParsingTest.exe /favorite-things DogBites
Could not parse  DogBites  as FavoriteThings

Using a Property Setter for Individual Values

The example above works well enough for enumerations, but not all arguments are that simple. In these situations, we can take advantage of the fact that arguments are applied to the options class by property setters.

The following example allows multiple individual values for an argument. As each argument value is set, it is saved into a collection of values.

Note that using a property setter in this fashion is not a good OOP practice; however, the adverse design affects are contained within the options class.

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    public Options()
    {
      Numbers = new List<int>();
    }

    public List<int> Numbers { get; private set; }

    [Option("number", 'n')]
    public int NumberOption
    {
      set
      {
        Numbers.Add(value);
      }
    }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine(string.Join(", ", options.Numbers));

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe


> CommandLineParsingTest.exe -n 3
3

> CommandLineParsingTest.exe -n 3 -n 6
3, 6

> CommandLineParsingTest.exe -n 3,6
Could not parse  3,6  as Int32

Note that the last test failed; the options class above only allows multiple individual arguments, not a group of values.

Using a Property Setter for Grouped Values

In this case, we want to be able to pass a sequence of values (delimited somehow) as a single argument, and have them interpreted as multiple individual values.

We can again take advantage of the property setter hack, but we have to do our own parsing of the delimited value. We will use a property type of string to prevent automatic parsing.

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    public Options()
    {
      Numbers = new List<int>();
    }

    public List<int> Numbers { get; private set; }

    [Option("number", 'n')]
    public string NumberOption
    {
      set
      {
        // Note: this example uses poor error handling!
        //  We *should* use TryParse and throw OptionParsingException.
        Numbers.AddRange(value.Split(';').Select(x => int.Parse(x)));
      }
    }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine(string.Join(", ", options.Numbers));

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe -n 3 -n 6
3, 6

> CommandLineParsingTest.exe -n 3;6
3, 6

This works, but still feels a bit "hackish". We're out of time for today, but in a few weeks we'll revisit this problem when we talk about custom argument parsers.

System.Threading.Timer Constructor and Garbage Collection

2011-07-21T12:30:00.005-04:00

This week, we take a break from the option parsing posts to bring you an interesting corner case from the BCL.

The System.Threading.Timer constructor has several overloads; all except one take a state parameter which is passed to the TimerCallback delegate when the timer fires.

It turns out that this state parameter (and the TimerCallback delegate) have an interesting effect on garbage collection: if neither of them reference the System.Threading.Timer object, it may be garbage collected, causing it to stop. This is because both the TimerCallback delegate and the state parameter are wrapped into a GCHandle. If neither of them reference the timer object, it may be eligible for GC, freeing the GCHandle from its finalizer.

The single-parameter constructor does not suffer from this problem, because it passes this for the state (not null). Most real-world usage of System.Threading.Timer either references the timer from the callback or uses the timer for the state, so this interesting garbage collection behavior will probably not be noticed.

This blog post was prompted by my own question on Stack Overflow.

Option Parsing: Options with Optional Arguments

2011-07-14T12:30:00.006-04:00

All of the examples so far have illustrated options with required arguments; that is, if the option is passed, it must be followed by an argument. It's also possible to define an option that takes an optional argument:

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option("with-extreme-prejudice", 'p', OptionArgument.Optional)]
    public int? PrejudiceLevel { get; set; }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      if (options.PrejudiceLevel.HasValue)
        Console.WriteLine("Extreme Prejudice specified: " + options.PrejudiceLevel.Value);
      else
        Console.WriteLine("Regular prejudice will do.");

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
Regular prejudice will do.

> CommandLineParsingTest.exe -p 3
Extreme Prejudice specified: 3

> CommandLineParsingTest.exe -p
Regular prejudice will do.

The last example above illustrates the problem with options that take optional arguments: there isn't an easy way to determine whether the option was passed without an argument or the option was not passed at all. In both of these cases, the property is left at the default value (null in this case).

The solution is to use the OptionPresent attribute, as such:

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option("with-extreme-prejudice", 'p', OptionArgument.Optional)]
    public int? PrejudiceLevel { get; set; }

    [OptionPresent('p')]
    public bool PrejudiceLevelWasSpecified { get; set; }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse();

      if (!options.PrejudiceLevelWasSpecified)
        Console.WriteLine("Regular prejudice will do.");
      else if (options.PrejudiceLevel.HasValue)
        Console.WriteLine("Extreme Prejudice specified: " + options.PrejudiceLevel.Value);
      else
        Console.WriteLine("Extreme Prejudice specified.");

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
Regular prejudice will do.

> CommandLineParsingTest.exe -p 3
Extreme Prejudice specified: 3

> CommandLineParsingTest.exe -p
Extreme Prejudice specified.

It is now possible to distinguish all possibilities. The OptionPresent example above uses the short option name, but this attribute also works with long names.

Option Parsing: Default and Nullable Argument Values

2011-07-07T12:30:00.006-04:00

Most of our examples so far have already dealt with options taking arguments, because most options in the real world do take arguments. Today we'll start looking at option arguments in depth.

The option pipeline post laid out the steps taken when using an Option Arguments class:

The Option Arguments class is default-constructed.
Attributes of properties on the class are used to produce a collection of option definitions.
The command line is parsed, setting properties on the Option Arguments instance.

We'll take advantage of these steps to handle several common scenarios.

Default Values

Default argument values are set in the default constructor:

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    public Options()
    {
      // Set default values.
      Quality = 3;
    }

    [Option("level", 'l')]
    public int Level { get; set; }

    [Option("quality")]
    public int Quality { get; set; }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine("Level: " + options.Level);
      Console.WriteLine("Quality: " + options.Quality);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
Level: 0
Quality: 3

> CommandLineParsingTest.exe /level 7
Level: 7
Quality: 3

> CommandLineParsingTest.exe /quality 4
Level: 0
Quality: 4

Nullable Values

There are some situations where a "default value" doesn't make sense for an option; you need to know whether there was a value passed, and what the value is (if it was passed). In this situation, you can use a nullable value type for your property:

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option("level", 'l')]
    public int? Level { get; set; }

    [Option("name", 'n')]
    public string Name { get; set; }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      if (options.Level.HasValue)
        Console.WriteLine("Level: " + options.Level.Value);
      else
        Console.WriteLine("Level not specified.");
      if (options.Name != null)
        Console.WriteLine("Name: " + options.Name);
      else
        Console.WriteLine("Name not specified.");

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
Level not specified.
Name not specified.

> CommandLineParsingTest.exe /level 3
Level: 3
Name not specified.

> CommandLineParsingTest.exe /level 0
Level: 0
Name not specified.

> CommandLineParsingTest.exe /name Bob
Level not specified.
Name: Bob

> CommandLineParsingTest.exe /name ""
Level not specified.
Name:

Option Parsing: Option Names

2011-06-30T12:30:00.008-04:00

An option may have a long name, a short name, or both. "Short names" are just single characters, while "long names" are strings. Option names may not contain the special characters : or =.

Commonly-used options should have both a long name and a short name. The short name enables faster typing on the command line, while the long name enables self-documenting command lines (for use in script and batch files). Normally, the short name is the first character of the long name, but this is not required.

Less-common options should have just a long name; this avoids polluting the short name namespace.

using System;
using Nito.KitchenSink.OptionParsing;

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option("level", 'l')]
    public int Level { get; set; }

    [Option("priority")]
    public int Priority { get; set; }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();
      
      Console.WriteLine("Level: " + options.Level);
      Console.WriteLine("Priority: " + options.Priority);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe
Level: 0
Priority: 0

> CommandLineParsingTest.exe /l 3
Level: 3
Priority: 0

> CommandLineParsingTest.exe /level 3
Level: 3
Priority: 0

> CommandLineParsingTest.exe /priority 1
Level: 0
Priority: 1

> CommandLineParsingTest.exe /p 1
Unknown option  p  in parameter  /p

Normally, options do not have just a short name without a long name, but you can do it if you want do.

Multiple Long and Short Names

Options may have "aliases" (multiple long and/or short names). The easiest way to add aliases is to have separate properties on your Option Arguments class that refer to the same underlying field.

The following example shows one alias that is used to change an old option "level" into a more descriptive option "frob-level", marking the old option as obsolete. Another alias "frobbing-level" is also added, which is just a regular alias (without any options marked obsolete).

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    // The old "level" option, now obsolete and made into an alias for "frob-level".
    [Option("level", 'l')]
    [Obsolete]
    public int Level
    {
      get { return FrobLevel; }
      set
      {
        Console.Error.WriteLine("Warning: The --level option is obsolete; use --frob-level instead.");
        FrobLevel = value;
      }
    }

    [Option("frob-level")]
    public int FrobLevel { get; set; }

    // Another alias for "frob-level"; this one is not obsolete.
    [Option("frobbing-level")]
    public int FrobbingLevel
    {
      get { return FrobLevel; }
      set { FrobLevel = value; }
    }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine(options.FrobLevel);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

> CommandLineParsingTest.exe /level 4
Warning: The --level option is obsolete; use --frob-level instead.
4

> CommandLineParsingTest.exe /frob-level 4
4

> CommandLineParsingTest.exe /frobbing-level 6
6

Abbreviated Option Names

Some programs support abbreviated option names; for example, the option "pack" may be abbreviated as "pa" or "p" (assuming there is no other option that starts with "pa" or "p", respectively). However, this causes backwards compatibility issues; for example, an updated version of the program may introduce an option named "push", and any scripts that used the abbreviated option "p" then become ambiguous. For this reason, Nito.KitchenSink.OptionParsing does not include automatic support for abbreviated option names. If you need abbreviated option names, you may use explicit aliases to achieve the same effect.

> CommandLineParsingTest.exe /frob 6
Unknown option  frob  in parameter  /frob

Option Parsing: Error Handling

2011-06-23T12:30:00.009-04:00

The Nito.KitchenSink Option Parsing Library wraps all option parsing errors into an exception derived from Nito.KitchenSink.OptionParsing.OptionParsingException. There are three more specific exception types (InvalidParameterException, OptionArgumentException, and UnknownOptionException), but they are seldomly needed.

All steps of the option parsing pipeline should only throw exceptions derived from OptionParsingException. In particular, this is true for custom validation (which will be described in detail in a future post).

The following example program shows how option parsing errors should be handled in a console application:

using System;
using Nito.KitchenSink.OptionParsing;

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option("level", 'l')]
    public int Level { get; set; }

    public static int Usage()
    {
      // Standard console size:
      //                      [                                                                                ]
      Console.Error.WriteLine("Usage: CommandLineOptionTest <arguments>");
      Console.Error.WriteLine("  -l, --level=LEVEL        level at which to operate");
      return 2;
    }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();
      
      // Program logic
      Console.WriteLine(options.Level);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return Options.Usage();
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

First, the Options class is declared, which defines the options our program takes. It also exposes a Usage method, which displays command-line usage information. Usage writes its information to Console.Error and returns an error code.

The Nito.KitchenSink.OptionParsing library does not attempt to write the Usage method for you automatically. Other option parsing libraries have attempted this, but the results are (IMHO) less than ideal.

The program's Main method returns an int, and contains a top-level try/catch. The try block parses the options, performs its requested task (in this case, the program just writes the Level option to the console), and then returns 0 (meaning "success").

If there is an option parsing exception, then the exception message is written to Console.Error, usage information is displayed, and an error code is returned.

If there is some other (unexpected) exception (during option parsing or program logic), then the entire exception (including the call stack) is written to Console.Error and an error code is returned.

Notes

For console programs, a return value of 0 indicates success and any other return value usually indicates an error. I used two different error codes in the example above, but they could just as easily be a single error code because distinguishing usage errors is not normally useful.

An options class does not have to include a Usage method; I just usually put it there so it's along with the class that defines the options. In future blog posts, I'll post example code that skips the Usage method to avoid distractions, but it should be included in real-world code.

Option Parsing: Lexing

2011-06-16T12:30:00.018-04:00

The first step in parsing a command line is lexing, which converts a single string (the command line) into a sequence of strings (individual options and/or arguments). Actually, the very first step takes place before the program even runs: the command shell has its own simple lexer.

Command Shell Escaping and Quoting

The information in this section is derived from the TechNet articles Command shell overview (webcite) and The Windows NT Command Shell (webcite).

The command shell has these special characters: &, |, (, ), <, >, and ^. There are two ways to pass these special characters on the command line: escaping and quoting.

The ^ character is the shell escape character. You may prefix any of the special shell characters with that escape character, and the special shell character will be passed to the program (without the escape character).

The command shell also supports quoting; special characters may be passed within a pair of double-quotes. In this case, the special characters are passed to the program along with the surrounding quotes.

The shell escaping and quoting appears to be a simple algorithm: escaped characters (including normal characters) are passed through directly, and each (non-escaped) double-quote either starts or ends a quoted string. Consider the outputs from this example program:

static void Main(string[] args)
{
  Console.WriteLine(Environment.CommandLine);
}

> CommandLineParsingTest.exe ^^ "^"
CommandLineParsingTest.exe ^ "^"

> CommandLineParsingTest.exe ^"^"
CommandLineParsingTest.exe ""

> CommandLineParsingTest.exe ^""
CommandLineParsingTest.exe ""

> CommandLineParsingTest.exe "^"^"
CommandLineParsingTest.exe "^""

> CommandLineParsingTest.exe "^"^^"
CommandLineParsingTest.exe "^"^"

> CommandLineParsingTest.exe "^^
CommandLineParsingTest.exe "^^

Shell escaping and quoting are applied to every process by the Command Shell; there is no way to opt out of this behavior. After the command shell does its own escaping and quoting, the command line is passed to the program.

Default .NET Lexing

The command line is split up into a list of process arguments by the .NET runtime. The algorithm is described in the documentation for Environment.GetCommandLineArgs. The same results (except for the process name) are also passed as the single argument to the Main method, if present.

The .NET lexing also uses a combination of escaping and quoting, but it has some surprising results because escaping is allowed inside quoting. The escape character is \, and the quote character is the double-quote.

Each non-escaped double-quote starts or ends a quoted string, just like command shell quoting. However, unlike command shell quoting, escaping is allowed within quoted strings. The .NET lexing also allows two consecutive double-quotes inside a quoted string to represent a single double-quote. Consider the outputs from this example program:

static void Main(string[] args)
{
  foreach (var arg in args)
    Console.WriteLine(arg);
}

> CommandLineParsingTest.exe "a"
a

> CommandLineParsingTest.exe \"a"
"a

> CommandLineParsingTest.exe \"a
"a

> CommandLineParsingTest.exe "a\"
a"

> CommandLineParsingTest.exe "a\\"
a\

> CommandLineParsingTest.exe a \"
a
"

> CommandLineParsingTest.exe "a \\"
a \

> CommandLineParsingTest.exe "a\"b"
a"b

> CommandLineParsingTest.exe "a""b"
a"b

> CommandLineParsingTest.exe a "" """"
a

"

This lexing behavior is particularly problematic when passing directories. Since directories may contain spaces, they should be wrapped with quotes. However, if the directory ends with a backslash, the closing quote will be escaped:

> CommandLineParsingTest.exe "c:\my path"
c:\my path

> CommandLineParsingTest.exe "c:\my path\"
c:\my path"

This is a rather serious limitation of the default .NET lexer. It is possible to write your own replacement lexer using a different algorithm. This lexer would take the process command line as input and produce a sequence of strings.

The Nito.KitchenSink.OptionParser library does not have a lexer of its own, but it will accept a sequence of strings as input into its parsing methods. If no sequence of strings is passed to a parsing method, then the method will use the process' command line lexed with the default .NET lexer.

Option Parsing: The Option Parsing Pipeline

2011-06-09T12:30:00.000-04:00

There are three main phases during option parsing:

Lexing
Parsing
Validation

The Lexing phase deals with the escaping and quoting of special characters and splitting the command line string into a sequence of strings. The Parsing phase evaluates the sequence of strings from the lexing phase, and interprets them as options and arguments; this includes parsing arguments as necessary, e.g., converting a string argument "3" into the numeric argument value 3. The Validation phase determines if the options and arguments represent a valid command for the program to perform.

The Nito.KitchenSink.OptionParser library does not have a lexer, but does have a parser and hooks for validation. The easiest way to use the library is by calling a single method:

var options = OptionParser.Parse<MyOptionArguments>();

This single method wraps all the phases of the option parsing pipeline:

The command line for the process is lexed using the default .NET lexing.
The option and argument definitions are inferred from properties and attributes on the MyOptionArguments type.
These definitions are used to parse the lexed command line, saving the results into properties on a default-constructed MyOptionsArguments object.
Validation is performed on the MyOptionsArguments object, which is then returned.

Future posts will show how each of these steps may be configured (or replaced).

How to Run Processes Remotely

2011-06-02T12:30:00.001-04:00

Today I'm going to delve deeply into something I discovered many years ago (c. 2003). It's an interesting little trick that hopefully no one will ever have to use.

When a process running on one computer needs to perform some operation on another computer, the common solution is to actually have two processes that use interprocess communication. The one process sends its commands to the other process, which executes them on behalf of the first process. Normally, one must install a server on one computer and a client on the other. So, if someone needs to perform an operation on another computer, then that computer must already have the software installed.

However, there is a way to send a program to a remote computer and run it, without having any special existing software on the target machine. This approach doesn't work in every situation, but it's useful to know. The command line programs in the famous PSTools suite use the approach documented here to "inject" copies of themselves onto remote computers; this allows a simple form of remote administration. The white paper PsExec Internals (webcite) includes the specific details for PsExec.

Step 1: Establish an Authenticated Connection

About Connections

A user session on one computer may have network connections to other computers. One common example is network drives; each network drive is a connection to another computer. Network connections may also exist without mapping a drive letter.

Network connections may be examined and modified using the Windows Networking (WNet) API or the net command. Unfortunately, there are no .NET wrappers for this API in the BCL.

About Authentication

Each network connection has to be authenticated, but there are situations where this happens automatically. When you map a network drive using Explorer, by default Windows will use your local logon to attempt to log onto the remote machine, and if it's accepted, you won't actually get prompted for credentials. This is particularly common in Domain environments.

The net use command allows you to display current connections to other computers, and add or remove those connections.

Authentication Quirks

Microsoft made the design decision that any number of network connections may exist between two different computers, but that the same credentials must be used for all those connections. You may use different credentials for connections to two different servers, but all connections to the same server must use the same credentials. According to a rather dated KB106211 (webcite), this is done "for security purposes." The newer KB183366 (webcite) documents the limitation in more detail, but does not give a reason.

If you do attempt to use different credentials for different connections to the same server, you'll get a 1219 error: "Multiple connections to a server or shared resource by the same user, using more than one user name, are not allowed. Disconnect all previous connections to the server or shared resource and try again." I've also seen this error when Explorer tries to auto-reconnect its mapped drives and it gets confused; it appears to happen more commonly on wireless networks when resuming from a low-power state.

There's a "greybeard" trick used to get around this limitation: connect to the IP address instead of the hostname (or, if you want more work, set up multiple hostnames for that server). The logic behind "the same server" appears to be just a string comparison. This workaround has been documented in KB938120 (webcite).

There are some notable situations where it's not possible to establish an authenticated connection:

If the target (server) machine is running a client OS "Home" edition (e.g., XP Home, Vista Home Basic, Vista Home Premium, Windows 7 Home Basic, Windows 7 Home Premium), then no authenticated connections are possible.
If the target (server) machine is running a client OS "Professional" edition (e.g., XP Professional, Vista Business/Enterprise/Ultimate, Windows 7 Professional/Enterprise/Ultimate), then that machine must either be a member of a domain or turn off "simple file sharing" to support authenticated connections.

Note that if you're working in a domain enviroment, Everything Just Works. For the rest of us, we have to turn off "simple file sharing."

If the server is running a Home edition, or if it is not connected to a domain and is using simple file sharing, then it does not support authenticated connections. Instead, every incoming network connection is authenticated with the Guest account; see KB300489 (webcite).

Another non-authenticated approach is to use null sessions, which are truly anonymous. This means they work even if the Guest account is disabled. Null sessions are disabled by default and considered a security risk.

To send a program to a remote computer, you'll need an authenticated connection. A Guest authentication (or null session) is insufficient.

Common Shares

There are some hidden network shares for Windows systems. They are recreated automatically on reboot if they've been deleted.

Hidden shares are not shown in the normal GUI, but they can be displayed by the command net share.

The standard hidden share names that are important to us are:

IPC$ - An share that is used only for authentication.
ADMIN$ - The equivalent of %SYSTEMROOT% (usually "C:\Windows").

You can create your own hidden shares: KB314984 (webcite). You can also prevent the automatic creation of the standard hidden shares: KB954422 (webcite), but this may cause lots of problems: KB842715 (webcite).

With all of that background information, our first step is to actually establish the authenticated connection to \\computer\IPC$. The other steps are quite simple in comparison!

Step 2: Copy the Program to the Target

Just copy the program to \\computer\ADMIN$, right into the Windows directory. I recommend renaming the file during the copy to a unique name, to avoid conflicts. You don't need to explicitly establish a network connection to \\computer\ADMIN$; the existing connection to \\computer\IPC$ will be your authentication.

Step 3: Register and Execute the Program

This step makes use of the little-known fact that Win32 services may be installed remotely. The service configuration API can be used to install the service on the remote computer and then start it.

The .NET ServiceController class does expose remote control of services (starting, stopping, etc), but it does not expose remote installation of services.

Step 4: Securely Communicate

Once the service is running on the remote computer, it is simple matter to communicate with the original process and carry out its instructions. It's not quite as simple to do so in a secure manner, though; strongly consider encrypting all network communication and using impersonation in the service.

Also remember that - as a service - you are limited in what you can do.

Enjoy!

There aren't too many good use cases for this technique. Remote administration is one, as demonstrated by the PsTools suite from Microsoft TechNet Systems Internals.

Another possible application is to inject an installer for remote control software, such as VNC or pcAnywhere. This could be useful in the rare case where a computer is physically inaccessible.

Getting the ObjectContext from an EntityObject

2011-05-26T12:30:00.000-04:00

There are a few situations where it's useful to get an ObjectContext from an EntityObject. Note that in general I do not recommend a design that depends on this; there doesn't appear to be an easy way to do this using code first in EF 4.1 (using the DbContext API). That said, either of the solutions in this blog post will work when using the ObjectContext API.

The most common solution for this problem is from a 2009 Microsoft blog post by Alex James (webcite). Unfortunately, that solution has several limitations (including the requirement that the entities must have relations to other entities). Both of the solutions below do not have these limitations.

We use an example entity container named NorthwindEntites, derived from ObjectContext. To this we will add a factory method FromEntity(EntityObject entity), which retrieves the NorthwindEntities instance to which that entity is attached, or null if the entity is detached.

Solution 1: Dynamic

The idea behind this solution is to add a property to the entity type that points to its own ObjectContext. It's possible to do this by modifying the code-generating template file, but it's also possible to just add the property to each entity type manually and use dynamic duck typing to access it.

The modified NorthwindEntities uses OnContextCreated to hook into its constructor and set up event handlers to respond whenever an entity is added to or removed from this context. Each event handler uses dynamic duck typing to access an "ObjectContext" property on the entity; if no such property exists, the entity is ignored.

using System.ComponentModel;
using System.Data.Objects.DataClasses;
using Microsoft.CSharp.RuntimeBinder;

namespace WindowsFormsApplication1
{
  public partial class NorthwindEntities
  {
    partial void OnContextCreated()
    {
      this.ObjectMaterialized += (_, e) =>
      {
        try
        {
          dynamic entity = e.Entity;
          entity.ObjectContext = this;
        }
        catch (RuntimeBinderException)
        {
        }
      };
      this.ObjectStateManager.ObjectStateManagerChanged += (_, e) =>
      {
        if (e.Action == CollectionChangeAction.Add)
        {
          try
          {
            dynamic entity = e.Element;
            entity.ObjectContext = this;
          }
          catch (RuntimeBinderException)
          {
          }
        }
        else if (e.Action == CollectionChangeAction.Remove)
        {
          try
          {
            dynamic entity = e.Element;
            entity.ObjectContext = null;
          }
          catch (RuntimeBinderException)
          {
          }
        }
      };
    }

    /// <summary>
    /// Gets the object context for the entity. Returns <c>null</c> if the entity is detached or does not define an <c>ObjectContext</c> property.
    /// </summary>
    /// <param name="entity">The entity for which to return the object context.</param>
    public static NorthwindEntities FromEntity(EntityObject entity)
    {
      try
      {
        dynamic dynamicEntity = entity;
        return dynamicEntity.ObjectContext;
      }
      catch (RuntimeBinderException)
      {
        return null;
      }
    }
  }
}

The disadvantage to this approach is that you have to add an ObjectContext property to each entity type, like this:

namespace WindowsFormsApplication1
{
  public partial class Order
  {
    /// <summary> 
    /// Gets or sets the context for this entity.
    ///  This should not be set by end-user code; this property will be set
    ///  automatically as entities are created or added,
    ///  and will be set to <c>null</c> as entities are detached.
    /// </summary> 
    internal NorthwindEntities ObjectContext { get; set; }
  }
}

Alternatively, you could modify the creation template. Either way, it's a fair amount of work.

Solution 2: Connected Properties

The Connected Properties library may be used to "attach" properties to entity objects at run-time. This means it's no longer necessary to add the ObjectContext property on each entity type.

This modified NorthwindEntities uses the same hooks as the one above, but it uses connected properties instead of dynamic duck typing:

using System.ComponentModel;
using System.Data.Objects.DataClasses;
using Nito.ConnectedProperties;
using Nito.ConnectedProperties.Implicit;

namespace WindowsFormsApplication1
{
  public partial class NorthwindEntities
  {
    /// <summary>
    /// The object context connected property type.
    /// </summary>
    private struct ObjectContextProperty { }

    /// <summary>
    /// Gets the object context connected property for a specified carrier object.
    /// </summary>
    /// <param name="entity">The carrier object.</param>
    /// <returns>The connected property.</returns>
    private static IConnectibleProperty<NorthwindEntities> ObjectContext(object entity)
    {
      return entity.GetConnectedProperty<NorthwindEntities, ObjectContextProperty>();
    }

    /// <summary>
    /// Handles post-construction event.
    /// </summary>
    partial void OnContextCreated()
    {
      this.ObjectMaterialized += (_, e) =>
      {
        ObjectContext(e.Entity).Set(this);
      };

      this.ObjectStateManager.ObjectStateManagerChanged += (_, e) =>
      {
        if (e.Action == CollectionChangeAction.Add)
        {
          ObjectContext(e.Element).Set(this);
        }
        else if (e.Action == CollectionChangeAction.Remove)
        {
          ObjectContext(e.Element).Set(null);
        }
      };
    }

    /// <summary>
    /// Gets the object context for the entity. Returns <c>null</c> if the entity is detached.
    /// </summary>
    /// <param name="entity">The entity for which to return the object context.</param>
    public static NorthwindEntities FromEntity(EntityObject entity)
    {
      return ObjectContext(entity).GetOrConnect(null);
    }
  }
}

The disadvantage of this approach is that you do need to take a dependency on the Connected Properties library, but I think that's a reasonable tradeoff.

This post was inspired by a recent SO question.

Signed and Unsigned Comparisons in C, C#, and T-SQL

2011-05-19T12:30:00.007-04:00

As noted earlier, I've been doing a lot of firmware development recently (in C). A long-standing rule of C (and C++) is to convert signed int values to unsigned int values if both are used in a comparison; this is what the standard specifies. It's also traditional to issue a warning (the famous comparison between signed and unsigned integer expressions warning) because such code is usually a mistake.

In C, the expression ( (unsigned)-1 == -1 ) will compile (with a warning), and have a value of 1 (true). C# and T-SQL both handle that situation very differently.

In C#, the expression ( unchecked((uint)-1) == -1 ) will compile without warning and have a value of false. What actually happens (webcite) is that the unsigned value of 0xFFFFFFFF and the signed value of -1 are both converted to long values and then compared.

The C# behavior does makes sense. C# continues the C/C++ tradition of implicitly promoting to int for any integral binary operation (e.g., two byte values added together are converted to ints before the addition). However, the additional implicit promotion to long was a bit of a surprise. This is probably due to 64-bit math becoming more commonplace - not common enough to have an implicit promotion to long for every binary operation, but enough to add the implicit promotion to long when necessary.

In SQL Server's TSQL, the expression select 'true' where -1 = 4294967295; will return an empty result set (meaning -1 is not considered equivalent to 4294967295). Interestingly, it's doing the same kind of promotion as C# (webcite). In this case, the value 4294967295 is typed as a bigint (a 64-bit signed integer).

The C# behavior can be seen when inspecting T-SQL statements generated by Linq to Entites. In my case, I was using code such as this to look up an item by serial number (an unsigned value):

using (var context = new MyEntities())
{
    uint serialNumber = 0xFFFFFFFF;
    var item = context.Items.Where(x => x.SerialNumber == serialNumber).SingleOrDefault();
}

The generated T-SQL statement (for SQL Server Compact Edition) was like this:

SELECT TOP (2) [Extent1].[ID] AS [ID], [Extent1].[SerialNumber] AS [SerialNumber] FROM [Items] AS [Extent1]  WHERE ( CAST( [Extent1].[SerialNumber] AS bigint)) = @p__linq__0

p__linq__0 had a DbType of Int64. Note that the generated T-SQL includes an explicit cast to bigint because the C# language implicitly inserted a cast to long in the expression x.SerialNumber == serialNumber.

Lesson learned: be careful of mixing signed and unsigned types. The warnings that existed in C/C++ are not necessarily present in C# (or T-SQL).

Managed Services and UIs

2011-05-12T12:05:00.001-04:00

One common question that I've seen is how to display a UI from a service.

The answer is: "don't".

Usually, when someone asks this question, the correct solution is to change the application from a service to a background application run whenever a user logs in (e.g., from the Startup folder), possibly with a tray icon. Occasionally, this isn't possible, and the correct solution in that case is to split the application into two applications: a service without a UI, and a UI front-end (which may be a backround application run automatically).

Unfortunately, some people try to push forward with the "service with a UI" approach. This is doomed to fail.

Inevitable Failure

There are two hurdles to displaying a UI from a service; the first is architectural, and the second is technical.

The architectural hurdle is simply that displaying a UI from a service just doesn't make sense. A Win32 service is a program that runs (or can be run) any time the computer is running, regardless of whether or not there is a user logged in. It doesn't make much sense to talk about "displaying a UI" if there isn't a user to show it to. Also consider multi-user (terminal server or fast-user-switching) computers: which user would see the UI?

The technical hurdle is a bit more complex. To summarize: services which display UIs are a security risk.

The Win32 windows messaging system was designed without security in mind. Before you get too mad at Microsoft, remember that the Internet (including email, TCP/IP, and HTTP) were designed without security, too. Back then, it was hard enough just to get it working, without worrying about someone deliberately trying to destroy it. Most security on the Internet today is due to wrapping the original insecure protocols in an encrypted, authenticated stream (SSL/TLS).

In fact, when I first got on the Internet, the common instructions for setting up an email server explicitly stated that it should be set up as an open relay so that anyone could send email through it. Then someone invented spam. The instructions have since been revised.

Similarly, in the early days, Windows had no need for security. In early versions of Windows, multitasking was non-preemptive, so any program could effortlessly cause a denial-of-service attack. Furthermore, each program had direct access to hardware, and causing a complete system crash was trivial.

These days, the situation is much improved. On modern OSes (not including the 9x line), a user-mode program simply cannot crash the system; it can only crash itself. With almost every new OS, Microsoft has enhanced security by trusting programs less (e.g., User Account Control).

One attack vector for malicious programs is a privilege escalation. This is a way for an untrusted program to trick the OS into trusting it more. One privilege escalation attack that has been discovered is called a shatter attack. This "shatter attack" is based on Win32 message passing.

In response, Microsoft made two changes starting in Vista: User Interface Privilege Isolation (see New UAC Technologies for Windows Vista (webcite)); and Session 0 Isolation (see this Application Compatibility blog post (webcite) or this Word document).

User Interface Privilege Isolation is a simple system where less-trusted programs (such as Internet Explorer) are limited in which Win32 messages they may send to more-trusted programs (such as services). This doesn't prevent services from having UIs, but may trip up programmers if they try to communicate with their service via message passing.

Session 0 Isolation is more surprising to most programmers, simply because most programmers are not aware of desktops or window stations. The following MSDN resources provide a good intro to the concept:

In essence, the older versions of Windows (XP and earlier) would run services in the same session as the first user that logged on to the physical computer, and services displaying UIs would be seen by that user. Newer versions of Windows (Vista and later) run services in their own special session (Session 0), which has its own window station and desktop completely independent from anything the user sees.

Naturally, this broke a lot of existing services, so Microsoft implemented a couple of workarounds. One is the "Interactive Service" flag, which would allow a service to display a UI. Another is the Interactive Service Detection Service, which is a special service that detects dialogs on Session 0 and notifies the user (if any) of them.

It is possible to set the Interactive Service flag when installing a service (not through the regular .NET Framework APIs; you have to p/Invoke for it). That is a horrible hack and should never, ever be applied to a new application. Even with that flag, some notification systems may not work as expected (see the end of this blog post from the security team (webcite)).

Remember that the Interactive Service flag is a backwards compatibility hack that weakens overall system security and may be removed from Windows vNext. Similarly, the Interactive Service Detection Service comes with this disclaimer: "This support might be removed from a future Windows release, at which time all applications and drivers must handle Session 0 isolation properly."

So - while it is possible to hack together a service with a UI today - you'd only be setting yourself up for failure in the future.

Simple and Easy Entity Framework SQL Tracing

2011-05-05T12:30:00.000-04:00

There's an easy way to add tracing to an application, but Entity Framework brings some special challenges. ObjectQuery.ToTraceString does allow tracing of SQL SELECT commands, but there's no built-in way to trace database updates.

However, there is an Entity Framework Tracing Provider that allows this. Follow the quick-start instructions on the home page, and you'll be off in no time!

Here's a few tests using SQL Server Compact Edition to access the Northwind sample database. This code:

using (var db = new NorthwindContext())
{
    MessageBox.Show(db.Orders.Count(x => x.Order_Date < DateTime.Now).ToString());
}

will result in this trace:

EntityFramework.NorthwindEntities Information: 0 : Executing 1: SELECT [GroupBy1].[A1] AS [C1] FROM ( SELECT COUNT(1) AS [A1] FROM [Orders] AS [Extent1] WHERE [Extent1].[Order Date] < ( CAST( GetDate() AS datetime)) ) AS [GroupBy1]
EntityFramework.NorthwindEntities Information: 0 : Finished 1 in 00:00:00.0466592: [DbDataReader(C1:Int)]

Note that the total time taken by the query is included in the finishing trace. Another interesting tidbit is that DateTime.Now is not evaluated on the client side; rather, the SQL statement includes a call to GetDate.

Here's some code that deletes an order:

using (var db = new NorthwindContext())
{
    db.Orders.DeleteObject(db.Orders.OrderBy(x => x.Order_Date).First());
    db.SaveChanges();
}

resulting in this trace:

EntityFramework.NorthwindEntities Information: 0 : Executing 2: SELECT TOP (1) [Extent1].[Order ID] AS [Order ID], [Extent1].[Customer ID] AS [Customer ID], [Extent1].[Employee ID] AS [Employee ID], [Extent1].[Ship Name] AS [Ship Name], [Extent1].[Ship Address] AS [Ship Address], [Extent1].[Ship City] AS [Ship City], [Extent1].[Ship Region] AS [Ship Region], [Extent1].[Ship Postal Code] AS [Ship Postal Code], [Extent1].[Ship Country] AS [Ship Country], [Extent1].[Ship Via] AS [Ship Via], [Extent1].[Order Date] AS [Order Date], [Extent1].[Required Date] AS [Required Date], [Extent1].[Shipped Date] AS [Shipped Date], [Extent1].[Freight] AS [Freight] FROM [Orders] AS [Extent1] ORDER BY [Extent1].[Order Date] ASC
EntityFramework.NorthwindEntities Information: 0 : Finished 2 in 00:00:00.0027257: [DbDataReader(Order ID:Int, Customer ID:NVarChar, Employee ID:Int, Ship Name:NVarChar, Ship Address:NVarChar, Ship City:NVarChar, Ship Region:NVarChar, Ship Postal Code:NVarChar, Ship Country:NVarChar, Ship Via:Int, Order Date:DateTime, Required Date:DateTime, Shipped Date:DateTime, Freight:Money)]
EntityFramework.NorthwindEntities Information: 0 : Executing 3: delete [Orders] where ([Order ID] = @0) { @0=[Int32,0,Input]10000 }
EntityFramework.NorthwindEntities Information: 0 : Finished 3 in 00:00:00.0482807: 1 rows affected

As expected, the first command executes a single-row SELECT, followed by a DELETE that affects a single row. Note the use of the parameterized deletion query.

Unfortunately, the Entity Framework Tracing Provider does not support everything; in particular, direct database commands (e.g., ExecuteStoreCommand) are not supported.

Unmanaged Blocking

2011-04-28T17:53:00.000-04:00

Managed code should never block in an unmanaged function, if it can possibly help it. As a general rule, wait functions (such as WaitForMultipleObjects) should never be p/Invoked from managed code.

The MSDN document Reliability Best Practices (webcite) states "do not block indefinitely in unmanaged code." Specifically, "blocking using a Win32 synchronization primitive is a clear example of something we cannot allow" because "a blocked thread prevents the CLR from unloading the AppDomain, at least without doing some extremely unsafe operations." As a general rule, they suggest that any function blocking more than 10 seconds will require special CLR support! In other words, if you're doing unmanaged blocking for that long, you'll have to write your own .NET runtime host.

The legendary Chris Brumme has a good blog entry on Managed Blocking (webcite). He enumerates several reasons why unmanaged blocking is inappropriate:

The CLR loses control of the thread. This is the same reason covered in the MSDN article above.
Managed blocking will do message pumping (in the right way) while blocked. This is necessary for STA threads (including UI threads as well as threads doing STA COM interop). Mr. Blumme has another classic classic blog entry: Apartments and Pumping in the CLR (webcite) that delves in-depth into this issue, and is probably the most complex blog post in existence.
The CLR collects information about managed threads, including how often and how long they block; this information is used (among other things) for making the ThreadPool more efficient. Unmanaged blocking prevents the CLR from gathering this information.
(The fourth reason from the blog post - hiding of platform differences - is no longer applicable, since the Windows 9x line is no longer supported by modern .NET applications).

Joe Duffy, in a very interesting post on Hooking CLR Calls with SynchronizationContext (webcite), talks about using a custom SynchronizationContext implementation to receive notifications about managed blocking. These types of notifications simply won't work if a managed thread does unmanaged blocking.

Closely related to Joe Duffy's post above is the fact that some CLR hosts take special action when managed threads block. In particular, SQL Server makes use of that information. Any host that is based on fibers instead of threads would also require that information (AutoCAD is the only such host that I'm aware of). Again, unmanaged blocking would prevent these hosts from working as expected.

WaitHandle is the key to unusual managed blocking situations. You can wait on any or all of a series of handles, or even derive from the WaitHandle class itself if you have a Win32 synchronization primitive not already wrapped by the BCL.

The bottom line is: avoid unmanaged blocking.