ILoggable

C# 3.0 language features vs. Java's OS community

noreply@blogger.com (ether) — Mon, 06 Oct 2008 16:31:00 +0000

I've been in a constant struggle for a number of years and it's a battle between wanting to program in C# but envying the variety of open source projects available on the JVM.

And this isn't so much a windows/linux thing, because mono vs. native .NET is really the least of my problems. The mono team has done an amazing job letting me write code without worry in Visual Studio and deploy on linux. With mono 2.0 just out, i have no code left that doesn't just work when i copy the dll's over. The fact that i do a lot of .NET on linux is not a pain point for me.

But when it comes to open source libraries, the eco-system on java is just so much richer. Think of an API or protocol and google it for java and you'll find it. Do the same for .NET and your chances are much smaller. And if you find it, chances are it's a port of Jfoo to Nfoo. So you could say, I've got library envy.

So why not just go java? Well, there are a number of C# language features that many might accuse of being "sugar", but i find them incredibly useful to express my intent in code. Here are the things that keep me in C# 3.0:

Delegates, especially anonymous delegates. For concurrent programming, they are essential
Events. Sure, they are just more sugar on top of delegates, but for async, having the ability to easily declare, subscribe and fire events are a god sent.
Properties. Public fields can't be part of an interface and if you ever need to attach observation to the value change, then fields are a dead end. And seriously, getFoo()/setFoo() is busy work for the compiler. It's less readable and more error prone.
Lambda expressions. Yes, i want a taste of functional with my imperative
LINQ. And i don't mean LINQ to SQL. This isn't about databases, this is dealing with collections, doing map, reduce, filter and projection. Yeah, you can loop by hand, but it's so much less expressive or readable.
var. Type inference is the way to go for static type languages. Less clutter, still strongly typed. You can get most of the benefits of dynamic languages with type inference and not have to give up type safety. I hope that we get closer to duck typing tho.
Dead easy interop with native processes. No, my VM is not some abstract world that shan't know about its environment. It's just a better way to code, so let me do machine specific things without making me pay penance in blood for it.
A language with much more innovation. Ugh, I know MS made innovation a dirty word in their anti-trust battles, but it's really true for C#. C#2.0's best features were aped by Java 5 and C#3.0 is once again miles ahead in new capabilities. People might argue whether that's a good thing, but it can't be disputed that C# is innovating much more rapidly.

So there you go, I'm a C# fan boy. I admit it. But still, i got this library envy. What's a guy to do? I see two possible paths, C# on the JVM or java libs on mono. Both are mostly possible, but I really need to do some spikes to know where the pain starts with the two options.

C# on the JVM: Mainsoft Grasshopper

Author in C#, cross-compile IL into java byte code. Mainsoft is apparently collaborating with the mono guys, so generally what makes it into mono in terms of language features will make it into Grasshopper. Need to take some of my projects and see what happens when i try this.

Java on C#: IKVM

Compile java code into .NET IL using IKVM. This is dependent on the state and compatibility of IKVM and Java. Need to take some popular java open source projects and see what happens when i try to build them using IKVM.

Either sounds like inviting a life of debugging edge cases that no one else cares about. But I don't see a better option. Am I overlooking some other path? Should i stop whining and just go java? Is the OS situation not nearly as bad on .NET as i make it out to be and I can just continue as is? Things that keep me up at night.

Synchronicity Kills

noreply@blogger.com (ether) — Sat, 04 Oct 2008 07:34:00 +0000

Finally got around to starting my blogging about the technology being built for notify.me. It's pretty funny that both my Mindtouch and my notify.me coding revolve around heterogenous, asynchronous programming to achieve highly scalable concurrency. Since i started working on notify.me before I started at Mindtouch and dug deep into Dream, the approaches are currently rather different, although I think i can bring lessons learned from either to the other as things progress.

IEnumerable.ForEach()

noreply@blogger.com (ether) — Fri, 03 Oct 2008 04:44:00 +0000

I keep wanting to do ForEach() on a collection, and noticed that it was inconsistent whether that extension method was available or not. I always figured that i wasn't importing the right namespace, and blamed ReSharper for not being smart enough to figure out what namespace I was missing. So I finally googled the problem only to find this. Turns out ForEach() isn't on IEnumerable, only on Array and List. Meh. But thanks to Dave, i now have it in my core assembly.

A case for TDD

noreply@blogger.com (ether) — Thu, 02 Oct 2008 04:45:00 +0000

I know, it's been rather quiet here lately. I've just been slammed with coding, so writing things up is falling behind. In addition, my blogging time's going to be split between techblog.notify.me and www.mindtouch.com/blog. And I'm behind on both of those as well. I should have some fun Dream and DekiScript stuff for the mindtouch blog and some asynchronous programming for the notify.me blog soon. As soon as i can get myself to stop coding again.

So what makes me stop coding for a minute to babble on? It's just a quick case studio of why TDD is important.

I've been an Inversion of Control/Dependency Injection for about a year and a half, and while I've eased my way into it, I'm pretty much at the "an interface for every class" stage of having everything abstracted so i can easily mock things. But here and there, I take in third party assemblies for my projects. And most of the time, they are not well interfaced. And generally I try to create a facade that is interfaced, so i can test my interaction in isolation. But depending on how many secondary classes their code uses, sometimes my facade gets lazy leaving places i can't mock.

Now, i'm pretty religious about test coverage, but i do have holes where my facade leaves untestable bits. And this is where TDD shows it's worth. Because when a feature is added or a refactor happens, almost with 100% certainty, the bugs that manage to get into production are in the code that doesn't have test coverage.

The lesson here is that the time saved in not building a properly mockable facade, thereby torpedoing my testability, is repaid manyfold in debugging later as bugs make it into production. meh.

OMFG AppleTV, seriously? Wipe yourself clean, why don't you?

noreply@blogger.com (ether) — Mon, 14 Jul 2008 02:08:00 +0000

Ok, here's a feature of AppleTV that can only have it's origin in some devils bargain with Content industry: Once you associate an iTunes instance with AppleTV for sync, you better not ever want to change it, because in the blink of an eye, you will staring a freshly wiped HD. See, syncing means that whatever is on your PC/Mac is also on your AppleTV. So if you remove it from the PC, it disappears from your AppleTV. Ok, fine, a bit stupid a way to do content management on the AppleTV, but maybe i'll buy that it simplifies things, since all you have to do it keep things organized in AppleTV and all is good. But here' the kicker, you ever change your AppleTV association to another iTunes, all is wiped of the AppleTV after one quick dialog. And no, oh, it's only hidden in case you re-associate because you realize that was not what you meant to do. No, you now get to re-synch 150GB back to your AppleTV. All for a second of misreading a dialog and clicking the wrong button. Wow! Really? That's great UI simplification. User makes mistake, gets to sit around for a couple of hours while AppleTV re-synch. Any you cannot tell me that the brilliant minds at Apple came up with this scheme because it was just the most intuitive UI they could think of. No, this is all about the content industry being deathly afraid that someone might take their AppleTV to a friends house, copy all his content and then take it back home. As if this assinine sync behavior prevents something like that. Grrr.. This is just stupid. 8 out of 450 items synced.

The riddle of the disappearing WPF databinding

noreply@blogger.com (ether) — Wed, 02 Jul 2008 20:56:00 +0000

I'm currently on a custom control that has a bunch of panels slaved to each other via databinding. And I ran into a bug where moving an element around would suddenly break the sync with the other panels. Nothing in the Output about data-binding failing, so i inspected it with Snoop to see that my data-binding had just up and gone away. So I tracked down the suspect code and monitored the data source with Mole as i stepped through and saw that the data-binding went away right after I set the dependency property.

Yeah, not really magic at all. A quick look at the Xaml showed that I had left this particular binding as default (i.e. OneWay). And if you have a OneWay binding and manually set the dependency property, the binding gets overwritten and goes away. Switching the binding to Mode=TwoWay restored my sync.

WPF Custom Panel layout and Dependency Properties

noreply@blogger.com (ether) — Wed, 02 Jul 2008 05:28:00 +0000

Just a quick note I learned the hard way when creating custom panels in WPF: If your panel has dependency properties or attached properties on its children that affect measure and/or arrange, calling InvalidateMeasure or InvalidateArrange won't necessarily do the trick. For that matter, calling these methods isn't even necessary. Instead use the FrameworkPropertyMetadata Metadata class to set appropriate FrameworkPropertyMetadataOptions.

So if you have a Dependency Property on your panel that affects the measure or arrange of its children, make sure it has FrameworkPropertyMetadataOptions.AffectsMeasure and/or FrameworkPropertyMetadataOptions.AffectsArrange set.

Similarily, if your panel has an attached property for its children, whose modification affects how that child is laid out, set FrameworkPropertyMetadataOptions.AffectsParentMeasure and/or FrameworkPropertyMetadataOptions.AffectsParentArrange.

Now layout and arrange are properly invalidated and called for you.

XP 64, IIS 6 and ASP.NET

noreply@blogger.com (ether) — Wed, 19 Mar 2008 16:13:00 +0000

My current dev machine is running XP 64, which is a first for me. In the default setup IIS was not installed, so I went through Add/Remove Programs and installed it, which gave me IIS 6. This in turn has several tabs for ASP.NET, but try as you might none of these are what actually turns on ASP.NET and you just end up with mysterious 404s on a application enabled directory that's configured just like the working ASP.NET on your other machine.

Well, it turns out that ASP.NET (even though it shows up in the Properties tabs) is not installed by default and if you go to Web Service Extensions you won't see it there. So next, track down aspnet_regiis which is in the Framework directory and run

aspnet_regiis -i

Then go back to IIS Manager -> Web Service Extensions where ASP.NET should now be an available extension. Enable it and finally ASP.NET works.

Sharing code between Silverlight and .NET

noreply@blogger.com (ether) — Wed, 12 Mar 2008 06:12:00 +0000

Currently, a Silverlight Class Library cannot be loaded by the server side project and vice versa. This despite there being very close parity in the BCL on either side. Now, I agree that sharing actual business logic between client and server is a bit of an edge case, but when it comes to data interchange, having a single codebase for you DTOs would be very useful.

Luckily, with a bit of trickery, this can be accomplished, although it would be nice if Visual Studio could just do this for you. I've done this on two different projects, once creating DTOs that were then serialized using the JSON DataContract serializer and other other time using the normal XmlSerializer.

The basic trick is this: Two separate projects can point at the same source code files (and even the integrated source control providers will play along with this game). However, playing by Visual studio rules you can't just create two projects in the same directory because the wizards treats ProjectName == DirectoryName. Here's how you get around this:

Create a new server side Class Library, say Server.Dto.
Create a new Silverlight Class Library, say Silverlight.Dto.
Clean out that default Class.cs and update both to have the same default namespace Foo.Dto. Your solution should look something like this:
Now, right-click on Silverlight.Dto and choose Remove
Close Visual Studio
Rename the Silverlight.Dto/Properties directory to Silverlight.Dto/Silverlight.Properties
Copy Silverlight.Properties and Silverlight.Dto.csproj to the Server.Dto directory
Now, and this is the tricky bit, open Silverlight.Dto.csproj in some text-editor (notepad works, but don't try to use Visual studio for this) and change the line
```
<Compile Include="Properties\AssemblyInfo.cs" />
```
to
```
<Compile Include="Silverlight.Properties\AssemblyInfo.cs" />
```
Re-open your solution, right-click on the Solution in the Solution Explorer and choose Add -> Existing Project..., browse to Server.Dto and select Silverlight.Dto.csproj
Add your first shared class FooDto.cs to Server.Dto (if you add it on the Silverlight side, you need to clean up a bunch of using statements).
Select Silverlight.Dto and click the Show All Files button at the top of the panel. You should see FooDto.cs in Silverlight.Dto now
Right-click FooDto.cs and choose Include in Project, like this:
Voila, you now have two assemblies referencing the same code so that you can use the same code on both client and server sides. Just repeat the last couple of steps for every new class you want to share.

I know you could use a Web Service from within Silverlight and would automatically generate you the proxy on the client side. And I'd recommend that if your payload is dynamic. However, if your payload is generated occasionally by a server side program or periodic service, this methods lets you create Dto's, serialized in your favorite manner that can be consumed as static files with WebClient.

Epoch DateTime conversion Extension Methods

noreply@blogger.com (ether) — Sat, 08 Mar 2008 19:42:00 +0000

Interop with unix often requires dealing with Epoch or Unix time, or the number of seconds since January 1st, 1970 UTC. And if you throw java in the mix, then epoch becomes milliseconds, since they define System.currentTimeMillis() as the number milliseconds since the Epoch. Anyway, i figured, this was the perfect use of Extension methods. Well, almost... There's still the issue that getting a DateTime object from seconds requires a static method added to DateTime, which extension methods do not currently support. This means that instead of

DateTime utcDateTime = DateTime.FromEpochSeconds(seconds)

we have to be content with

DateTime utcDateTime = DateTimeEx.FromEpochSeconds(seconds)

Now i also added extensions on long and int but, really, that falls into the realm of stupid extension method tricks. I left them in there, only because they are part of DateTimeEx, therefore will only be available if the appropriate namespace is included and so is at least tangentially relevant in the current scope. Well, that's my rationalization, at least. With this extra extension method you now can do

DateTime utcDateTime = 1205003848.DateTimeFromEpochSeconds();

The one thing to be aware of with all these helpers is that it always deals with UTC time, i.e. the DateTime that you convert to epoch time needs to have a DateTimeKind that is not Unspecified. Conversely, the DateTime you get back is UTC and if you want to deal with it with localtime, you need to call ToLocalTime() on it first.

Anyway, here's the class:

using System;

namespace Droog.DateTime
{
  public static class DateTimeEx
  {
    public static DateTime FromEpochMilliseconds(long milliseconds)
    {
      return new DateTime(1970, 1, 1,0,0,0,DateTimeKind.Utc).AddMilliseconds(milliseconds);
    }

    public static DateTime FromEpochSeconds(int seconds)
    {
      return FromEpochMilliseconds((long)seconds * 1000);
    }

    public static DateTime DateTimeFromEpochMilliSeconds(this long milliseconds)
    {
      return FromEpochMilliseconds(milliseconds);
    }

    public static DateTime DateTimeFromEpochSeconds(this int seconds)
    {
      return FromEpochSeconds(seconds);
    }

    public static int ToEpochSeconds(this DateTime dt)
    {
      return (int)(ToEpochMilliseconds(dt)/1000);
    }

    public static long ToEpochMilliseconds(this DateTime dt)
    {
      return (long)(dt.ToUniversalTime() - new DateTime(1970, 1, 1)).TotalMilliseconds;
    }
  }
}

Synergy

noreply@blogger.com (ether) — Wed, 05 Mar 2008 06:05:00 +0000

I used to run x2vnc and win2vnc back in the MP3 days to let me control my Windows and linux boxen. Later I used the same setup with my old MacBook 15" and a Linux box. The other day, my friend n8 posted about his Synergy setup, which came perfectly timed. I just started a new gig at Bunkspeed and I'm using a dedicated desktop for dev instead of my MacBook Pro, but i don't want my Mac to be wasted. So i set up Synergy on both my home and work desktops and have the Mac on a stand running two Synergy clients (only one of which ever finds a server to connect to. This setup rules!

Update: Don't know if this is something i'll find a way around, but apparently logging on to the VPN killed the connectivity between my desktop and mac :(

Update 2: Ok, as simple as going into the Advanced config and telling it what local IP to listen for connections on. All good again

VS2k5 & WPF

noreply@blogger.com (ether) — Mon, 03 Mar 2008 21:30:00 +0000

After having tried a bunch of different iterations to get the WPF tools installed for VS2k5, here's what finally worked"

Install .NET 3.0
Install Visual Studio Tools for WCF/WPF
And only then install .NET 3.0 SP1

Talk about annoying dependencies.

Use .NET/Mono, it's the environmentally friendly choice

noreply@blogger.com (ether) — Wed, 27 Feb 2008 00:17:00 +0000

I just read what's got to be my favorite Miguel De Icaza quote to date:

In a world that is increasingly green, it is a waste of perfectly healthy computer cycles to interpret your code when you can use an optimizing JIT compiler to run your code.

I'm going to presume that's with tongue firmly implanted in cheek.

Addicted to .Net 3.5

noreply@blogger.com (ether) — Mon, 18 Feb 2008 22:43:00 +0000

Outside of LINQ, i thought that 3.5 was a lot of cool but not vital syntactic sugar. This weekend marks the first time since November that I fired up VS.NET 2k5 to build an app targeting the 2.0 framework and I was amazed how much I'd already come to rely on that sugar. Now this might be seen as an invalidation of my preference of explicit, verbose syntax versus the terse syntax of many scripting languages. I'd like to point out that terseness and expressiveness in sytnax are two separate things. Syntactic sugar that let's me express my action in more concise code that easily conveys the meaning is not the same thing as using a terse vocabulary to keep the typing down and requiring memorization of abbreviated keywords to understand the code. Anyway, here are the parts of 3.5 I've missed more than once this weekend.

Extension Methods on IEnumerable

The plethora of extension methods on all things IEnumerable is largely due to LINQ, but the To* methods have become just a basic part of my vocubulary. Taking the Values of a Dictionary into an Array of the same type now is seems just painfully verbose without the ToArray() method. Compare

FileInfo[] f = new FileInfo[files.Count];
files.Values.CopyTo(f,0);
return f;

with

return files.Values.ToArray();

Object/List initializer syntax

Now this I really thought of as frivolous. However, using objects that use DTOs as their initializer/storage, initialization does become rather awkward without loops or long constructors:

      FileInfoData[] remoteFileData = new FileInfoData[]
      {
        new FileInfoData(),
        new FileInfoData(),
        new FileInfoData()
      };
      remoteFileData[0].name = "test1.mpg";
      remoteFileData[1].name = "test2.mpg";
      remoteFileData[2].name = "test10.mpg";
      FileInfo[] remoteFiles = new FileInfo[]
      {
        new FileInfo(remoteFileData[0]),
        new FileInfo(remoteFileData[1]),
        new FileInfo(remoteFileData[2]),
      };

versus

      FileInfo[] remoteFiles = new FileInfo[]
      {
        new FileInfo(new FileInfoData()
        {
          Name = "test1.mpg"
        }),
        new FileInfo(new FileInfoData()
        {
          Name = "test2.mpg"
        }),
        new FileInfo(new FileInfoData()
        {
          Name = "test10.mpg"
        }),
      };

Anonymous delegates are a pain and hard to read

I needed to pass in a delegate to a function as a factory callback. Perfect scenario for a nice concise lambda. But I was in 2.0, so that meant defining a delegate and anonymous delegate syntax resulting ing

  public delegate ILocalFileSystemManager LocalCreateDelegate(string localPath, string extension);

  public class FileSystemManagerFactory
  {
    public FileSystemManagerFactory( LocalCreateDelegate localFactory )
    {
      this.localFactory = localFactory;
    }
  }

FileSystemManagerFactory factory = new FileSystemManagerFactory(
  delegate(string localPath, string extension)
  {
    return new MockLocalFileSystemManager(localPath, extension);
  });

instead of

  public FileSystemManagerFactory( Func<string,string,ILocalFileSystemManager> localFactory )
  {
    this.localFactory = localFactory;
  }

FileSystemManagerFactory factory = new FileSystemManagerFactory(
  (localPath, extension)
  =>
  return new MockLocalFileSystemManager(localPath, extension);
  );

Automatic Properties

If there is one feature of C# (well and java as well) that is the most code generated, it's getters and setters. I've never liked how code generation tools created those for me, since i liked having my private members in one place and Properties in another. So i've been typing them out for years. But with C# 3.0, we got automatic properties. The two patterns, read/write properties and read-only properties are oft repeated like this

string readwriteMember;
string readonlyMember;

public string ReadWrite
{
  get { return readwriteMember; }
  set { readwriteMember = value; }
}

public string ReadOnly { get { return readonlyMember; }

Not a ton of code, but certainly takes more time to write than

public string ReadWrite { get; set; }

public string ReadOnly { get; private set; }

Ho hum.. I'm in 2.0, so i'll have to deal, but I certainly hope that 3.5 has a fast pick-up rate on the client (on the server, I can still control my environment).

LINQ: Immutability vs. Deferred execution

noreply@blogger.com (ether) — Fri, 01 Feb 2008 18:05:00 +0000

The last couple of nights I've been playing with some Linq to Sql and a whole lot of Linq to Objects and I have to say where coming up with complex Regular Expressions used to be one of my favorite puzzles, coming up with complex projections and transformations through Linq is quickly taking its place. Simple Linq is well documented, but when it comes to aggregation, it's a lot sparser. I expect to write more of that up once I feel more comfortable with the syntax.

In the meantime, I wanted to write up some non-obvious observation about deferred execution with Linq. Considering the gotchas with lambdas, it's easy to extend the lessons learned to linq, since it is after all deferred execution. But what's different with Linq is that, while execution is deferred, the expression tree built via a query is also immutable. I came across this trying to do some simple query re-use.

Let's start with a simple DTO:

public class Order
{
  public Order(int id, int val, bool buyOrder)
  {
    Id = id;
    Value = val;
    IsBuyOrder = buyOrder;
  }
  public int Id { get; set; }
  public int Value { get; set; }
  public bool IsBuyOrder { get; set; }
}

And a set of this data:

Order[] orders = new Order[]
{
  new Order(1,2,true),
  new Order(2,2,false),
  new Order(3,4,true),
  new Order(4,4,false),
  new Order(5,6,true),
  new Order(6,6,false),
};

Let's split those into buy and sell orders:

var buyOrders = from order in orders
          where order.IsBuyOrder
          select order;

var sellOrders = from order in orders
                 where !order.IsBuyOrder
                 select order;

If we want to find the buy and the sell order with a value of 2, you'd think we could write one query and re-use it for both of those queries. Since both queries results in IEnumerable<Order>, how about we define a query source and assign the value of either above query.

IEnumerable<Order> orders2 = null;

var orderAtTwo = from order in orders2
                 where order.Value == 2
                 select order;

orders2 = buyOrders;
int buyOrderId = orderAtTwo.First().Id;

orders2 = sellOrders;
int sellOrderId = orderAtTwo.First().Id;

Console.WriteLine("buy Id: {0}, sell Id: {1}", buyOrderId, sellOrderId);

Since the query is deferred until we call .First() on it, that seems like a reasonable syntax. Except this will result in an System.ArgumentNullException because our query grabbed a reference to orders2 at query definition, even though the query won't be executed until later. Giving orders2 a new value does not change the original reference in the immutable expression tree.

A way around this is to replace the actual contents of orders2. However, for us to do that, we have to turn it into the query source into a collection first.

orders2.Clear();
orders2.AddRange(buyOrders);
int buyOrderId = orderAtTwo.First().Id;

orders2.Clear();
orders2.AddRange(sellOrders);
int sellOrderId = orderAtTwo.First().Id;

Console.WriteLine("buy Id: {0}, sell Id: {1}", buyOrderId, sellOrderId);

This gives us the expected

buy Id: 1, sell Id: 2

Let's put aside the awkwardness of clearing out a list and stuffing data back in, this code has another unfortunate sideeffect. .AddRange() actually executes the query passed to it, so we execute our buy and sell queries to populate orders2 and then execute orderAtTwo twice against those collections. The beauty of linq is that if you create a query from a query, your not running multiple queries, but building a more complex query to be executed. So, what we really want is query "re-use" that results in single expression trees at execution time.

To achieve this, we need to move the shared query into a separate method such as:

private IEnumerable<Order> GetTwo(IEnumerable<Order> source)
{
  return from order in source
         where order.Value == 2
         select order;
}

and the code becomes:

int buyOrderId = GetTwo(buyOrders).First().Id;
int sellOrderId = GetTwo(sellOrders).First().Id;

Console.WriteLine("buy Id: {0}, sell Id: {1}", buyOrderId, sellOrderId);

This gives the same output as above, and we're only running two queries, each against the original collection. The method call means that we don't get to re-use an expression tree, since it builds a new one, combining the expression tree passed to it with the one it builds itself.

Struct's via Automatic Properties can be tricky

noreply@blogger.com (ether) — Tue, 15 Jan 2008 23:39:00 +0000

Here's a bit of code i just got through debugging...

public Point Point { get; private set; }

public void Offset(Point origin)
{
  Point.Offset(-origin.X, -origin.Y);
}

Can you tell what's wrong here? Let's just say that the Offset won't take. structs are value types, which means anytime you pass one around you get a new copy. So far so good. And that means when you expose a struct value via a property, the accessing party always is looking at a copy. Again, fine, after all when you do a set you change the stored value and if you need to manipulate it, you just manipulate the actual struct.

Enter Automatic properties and you might forget about this last detail and not realize that you never get access to the underlying value, even from within the class. I.e. when I call Point.Offset, i'm calling it on the copy that was passed to me and the resulting value is immediately thrown away. So i just went back to using the property to facade a private Point, which i can now manipulate inside of Offset. Duh.

More Deferred Execution Fun: foreach and delegation scope

noreply@blogger.com (ether) — Fri, 11 Jan 2008 20:50:00 +0000

This is closely related to my last post on deferred execution gotchas and its basically more "if you inline delegated code, you may easily overlook scope side-effects". This time it's about dealing with foreach and using the local each item for deferred execution.

public void SpawnActions()
{
  foreach (ActionContext context in contexts)
  {
    int id = context.Id;
    Action<int> callback =  
      (workerNumber) =>
      {
        Console.WriteLine("{0} Id: {1}/{2}", workerNumber, id, context.Id);
      };
    ThreadPool.QueueUserWorkItem(new WaitCallback(FutureExecute), callback);
  }
}

public void FutureExecute(object state)
{
  int id = worker++;
  Action<int> callback = state as Action<int>;
  Thread.Sleep(500);
  callback(id);
}

The output looks like this:

0 Id: 0/9
1 Id: 1/9
2 Id: 2/9
3 Id: 3/9
4 Id: 4/9
5 Id: 5/9
6 Id: 6/9
7 Id: 7/9
8 Id: 8/9
9 Id: 9/9

So while the foreach scope variable context is kept alive for the deferred execution, it turns out that foreach re-uses the variable on each pass through the loop and therefore when the Action is later executed, each one has a reference to the last context. So what we need to do is create a true local variable for context so that the lambda's scope can hold on to the reference we want.

  foreach (ActionContext context in contexts)
  {
    int id = context.Id;
    // locally scoped variable
    ActionContext c2 = context;
    Action<int> callback =  
      (workerNumber) =>
      {
        Console.WriteLine("{0} Id: {1}/{2}", workerNumber, id, c2.Id);
      };
    ThreadPool.QueueUserWorkItem(new WaitCallback(FutureExecute), callback);
  }

And now our results are a bit more what we expected:

0 Id: 0/0
1 Id: 1/1
2 Id: 2/2
3 Id: 3/3
4 Id: 4/4
5 Id: 5/5
6 Id: 6/6
7 Id: 7/7
8 Id: 8/8
9 Id: 9/9

Copying a Silverlight User Control from one project to another

noreply@blogger.com (ether) — Thu, 03 Jan 2008 22:17:00 +0000

Here's something I tracked down with no help from error messages:

When you copy a user control in Silverlight 1.1 from one project to another the Xaml that the control loads will have it's Build Action set to SilverlightPage. When you then run your project and try to create an instance of that control you'll get the ever so informative AG_E_INVALID_ARGUMENT. All you need to do to fix it, is set the Build Action to Embedded Resource again. Tada!

I love declarative definition of UI with behavior wired to static code. But man, at its current state, the debugging support for it just isn't there. I mean it's bad enough that having strings in the declarative side to link to actions that won't be updated by normal refactoring, nor will they show up as references, but at this state Xaml brings the worst part of scripting languages to compile-time checked coding:

Vague runtime errors without a stacktrace

Bah.

XP on Bootcamp & VMWare Fusion, take 2

noreply@blogger.com (ether) — Thu, 03 Jan 2008 16:45:00 +0000

Time to re-install XP

My setup until yesterday was Bootcamp partion that was running as a VM using VMWare Fusion. The Bootcamp partition was set up as FAT32, because I NTFS came up as readonly when mounted under Mac OS.

Then I started up another VM (Fedora Core 7, although i have since noticed it's not what OS you run, but just a second VM). Maybe it's not enough memory, maybe it's a VM running from bootcamp plus a VM running from a disk image, but while it had worked previously, this time, it locked up my Mac hard. I finally had to hard boot the Mac. When I got back into th VM, I noticed things were broken. Now, I've had to hard boot XP many times and I've never seen this. An indeterminate number of files were corrupted. I noticed one XML file that halfway through turned into binary garbage, so I assume that the other systems failing were suffering from similar corruption. Basically it was hosed, because there was no way to determine what had been corrupted. Time to re-install.

Bootcamp & VMware install

Looking at my post from last time, it was clearly written with the frazzled recollection of a day of trying to make things work, since i once again ran into problems. This time I'm making sure I write the resolution down.

Imaged the old partition, because you always realize that you forgot to back up some vital file.
Removed the old partition and created a new one (using the Bootcamp tool), this time large enough for more than just fallback use (since it's used daily under VMWare). This creates a FAT32 partition.
Inserted my install disk and Bootcamp restarted to fire up the windows install. Important note here, this has to be a real XP install disk. I first tried to use my MSDN DVD with the chooser for picking what OS to install. However, the Mac keyboard doesn't seem to work when you get to the menu. So I used an XP Pro w/ SP2 CD and rebooted. Now the installer ran just fine
Formatted the partition using NTFS. Even if you use FAT32, re-format, don't use the Bootcamp formatted partition. At least for me, using that prepared partition didn't work and created an unbootable image. I know NTFS cannot be written to by MacOS, but it doesn't really matter, since once I boot it as a VM, i can always transfer files via loopback file sharing. I'm going NTFS to get a journaling file system. Theoretically that should prevent the corruption I got last time.
After the install completes, pop in the Mac OS disk and let it install the Bootcamp utilities. This gives you full support for the Macbook Pro hardware.
Activate Windows and reboot into Mac OS X.
Fire up VMWare Fusion. Bootcamp partition should be listed as a VM. Since I previously had a bootcamp partition, I had to go into Library::VMWare Fusion::Application Support::Virtual Machines and remove the old Bootcamp partition folder. Fusion will then do its magic and prep the Bootcamp partition to run as a VMWare Image.
Activate Windows again. That should be the last time you have to do it.

Let's hope this doesn't turn into a bi-monthly process :)

The dangers of deferred execution

noreply@blogger.com (ether) — Sun, 30 Dec 2007 21:39:00 +0000

I recently wrote about Action & Func, which along with Lambda expression let you do easy inline callbacks like this:

Utility.ActionDownloader.Download(
  Configuration.GetAssetUri(dto.Url),
  (Downloader d) =>
  {
    FloatContainer c = (FloatContainer)XamlReader.Load(d.ResponseText);
    c.Initialize(dto);
  });

i.e. I can call a downloader and inline pass it a bit of code to execute once the download completes. But the catch of course is that looking at the code, and following the usual visual tracing of flow hides the fact that c.Initialize(dto) doesn't get called until some asynchronous time in the future. Now, that's always been a side-effect of delegates, but until they became anonymous and inline, the visual deception of code that looks like it's in the current flow scope but isn't wasn't there.

What happened was that I needed my main routine to execute some code after FloatContainer was initialized, and by habit i created an Initialized event on FloatContainer. Of course this was superfluous, since my lambda expression called the synchronous Initialize, i.e my action could be placed inline after that call to c.Initialize(dto) and be guaranteed to be called after initialization had completed.

This scenario just meant I created some superfluous code. However, I'm sure as I use lambda expression more, there will be more pitfalls of writing code that doesn't consider that its execution time is unknown, as is the state of the objects tied to the scope of the expression.

This last bit about objects tied to the expression scope is especially tricky and I think we will see some help in terms of Immutable concepts weaving their way into C# 3.x or 4.0, as the whole functional aspect of lambda expressions really work best when dealing with objects that cannot change state. Eric Lippert's been laying the groundwork in a number of posts on the subject and while he constantly disclaims that his ponderings are not a roadmap for C#, I am still going to assume that his interest and recognition of the subject of Immutables will have some impact in a future revision of the language. Well, I at least hope it does.

Software Activation vs. Virtualization, Part 3

noreply@blogger.com (ether) — Fri, 14 Dec 2007 18:50:00 +0000

Part of an ongoing saga.

Rebooted back into VMWare Fusion and yeah, Illustrator Activation was indeed screwed there as well. Office 2007 too, but at least it just let's me reactivate (no doubt noting me as a repeat offender somewhere). So I called Adobe and was told that "it's a sensitive piece of software". No it's not. Illustrator can take any beating you give it.. It's the "anti-piracy" crap that's sensitive. I got an "emergency activation code" to get it going again and was advised to Deactivate before i switch VM setups and then re-activate after the reboot. OMFG. Seriously, just give me USB dongle if you are so sensitive about it. That would be inifintely more convenient.

Dug around the net a bit and it seems that if i fake my mac address to be the same between boot camp and the VM boot, it'll not invalidate my activation. Might try that next. Of course, the same board i found that on also noted that if I just got a crack for my legally purchased product, all troubles would be gone as well. Yes, once again, anti-piracy crap is not stopping pirates but legitimate customers. You'd figure someone might have spotted the pattern here, but may those DRM-colored glasses filter reality a bit too well.

Action & Func: Never write another delegate

noreply@blogger.com (ether) — Fri, 14 Dec 2007 06:39:00 +0000

With lambda expressions in C#, the Func generic delegate and it's variations have been getting a lot of attention. So naturally, you might think that the lambda syntax is just a shortcut for creating anonymous delegates, whether they return values or not.

First let's look at the evolution of delegates from 1.1 to now. Delegates, simply are the method equivalent of function pointers. They let you pass a method call as an argument for later execution. The cool thing (and a garbage collection pitfall) is that a delegate creates a lexical closure, i.e. the delegate carries with it the object that the method gets called on. For garbage collection this means that a delegate prevents an object from being collection. That's why it's important to unsubscribe from those events you subscribed to.

But I digress. Let's define a delegate that returns an Integer and a method that matches that delegate:

delegate int IntProducerDelegate();

public int x = 0;
public int IntProducer()
{
  return x++;
}

With the original .NET 1.0 syntax we'd create the delegate like this:

IntProducerDelegate p1 = new IntProducerDelegate(IntProducer);

Now we can call p1() and get an integer back, and since it's closure, each time we call p1() the originating objects x increases as does our return value.

Then, in .Net 2.0 we got anonymous delegates.

IntProducerDelegate p2 = delegate { return IntProducer(); };

// or with IntProducer's action inlined...
IntProducerDelegate p3 = delegate { return x++; };

This got rid of the need to create a method just to pass along a closure that manipulated our object at a later time. The other thing that anonymous delegates re-inforce is that delegates just care about signature. IntProducerDelegate can get assigned any delegate that takes no argument and returns an int. That sounds like a perfect scenario for a delegate and in .NET 3.5, we got just that, a set of generic delegates called Func. Using Func, we quickly get to our lambda expression replacing the original delegate syntax like this:

// create a new Func delegate just like the IntProducerDelegate
IntProducerDelegate p3 = new Func<int>(IntProducer);

// which means that we don't need IntProducerDelegate at all anymore
Func<int> p4 = delegate { return x++; };

// and the anonymous delegate can also be shorthanded with a lambda expression
Func<int> p5 = () => { return x++; };
// which says, given that we take no argument "()", execute and return the following "return x++;"

However, before there ever was Func, .Net 2.0 introduced the generic delegate Action, which is a natural counterpart to Func, encapsulating a method that does not return anything. Following through the example of the producer, we'll create a consumer like:

delegate void IntConsumerDelegate(int i);

public void IntConsumer(int i)
{
  Console.WriteLine("The number is {0}", i);
}

Now following the same evolution of syntax we get this:

IntConsumerDelegate c1 = new IntConsumerDelegate(IntConsumer);

IntConsumerDelegate c2 = new Action<int>(IntConsumer);

Action<int> c3 = delegate(int i) { Console.WriteLine("The number is {0}", i); };

Action<int> c4 = (i) => { Console.WriteLine("The number is {0}", i); };

So lambda syntax can be used to create either a Func or an Action. And that also means that we never have to explicitly need to create another delegate, being able to use a variation of these two generic delegates as our arsenal for storing lambda expressions of all kinds.

Software Activation vs. Virtualization (and multiple PC ownership)

noreply@blogger.com (ether) — Thu, 13 Dec 2007 18:12:00 +0000

Just as Virtualization is finally becoming a useful technology, everybody and their uncle has decided that software activation is the new hot way to stop theft. Of course, like all anti-piracy tools, the paying customers get screwed, because the pirates have already patched their copies to not require activation. Bravo! You know i'd prefer friggin USB dongles to this big brother activation business.

I've talked about these problems before, but I've got more fun with the VM vs. bootcamp image activation troubles. I just got Adobe CS3 and for a programmer with occasional Photoshop/Illustrator needs, that's a pretty serious expense. I mean it costs me more than MSDN and gets used a fraction of the time. But I need it. And forget that I have three different computers I use at different times and I really ought to be able to install my purchased software on all of these machines, since I, the owner of the license, will never be using two computers at once. But that's a whole other story.

Back to the re-activation on hardware change business... I've been running Windows under VMware for the last couple of weeks, but for the Illustrator work I need to do right now, it was a bit sluggish. No problem, reboot into Bootcamp! Mind you, this isn't a differnt install of Windows. This is the same physical disk partition, but booted natively vs. via VMware. What happens? Illustrator bitches about activation, as does office, because it saw the hardware change. Let me guess, when i reboot in the virtual machine it'll bitch yet again. Sooner or later it'll just shut me down as a serial offender. Thanks! Way to reward my purchase.

A case for XML

noreply@blogger.com (ether) — Mon, 10 Dec 2007 00:38:00 +0000

XML gets maligned a lot. It's enterprisey, bloated, overly complex, etc. And the abuses visited upon it, like trying to express flow control or whole DSLs in it or being proposed as some sort of panacea for all interop problems only compound this perception. But as long as you treat it as what it is, data storage, I generally can find little justification to use something else. Not because it's the best, but because it's everywhere.

If you are your own consumer and you want a more efficient data storage, just go binary already. If you're not, then I bet your data consumers are just tickled that they have to add another parser to their repository of data ingestors. Jim Clark probably put it best when he said:

"For the payload format, XML has to be the mainstay, not because it's technically wonderful, but because of the extraordinary breadth of adoption that it has succeeded in achieving. This is where the JSON (or YAML) folks are really missing the point by proudly pointing to the technical advantages of their format: any damn fool could produce a better data format than XML."

Ok, I won't get religious on the subject, but mostly wanted to give a couple of examples, where the abilities and the adoption of XML have been a godsend for me. All this does assume you have a mature XML infrastructure. If you're dealing with XML via SAX or even are doing the parsing and writing by hand, then you are in a world of hurt, I admit. But unless it's a memory constraint there really is no reason to do that. Virtually every language has an XML DOM lib at this point.

I love namespaces

One feature a lot of people usually point to when they decry XML to me is namespaces. They can be tricky, i admit, and a lot of consumers of XML don't handle them right, causing problems. Like Blend puking on namespaces that weren't apparently hardcoded into its parser. But very simply, namespaces let you annotate an existing data format without messing with it.

<somedata droog:meta="some info about somedata">
  <droog:metablock>And a whole block of extra data</droog:metablock>
</somedata>

Here's the scenario. I get data in XML and need to reference metadata for processing further down the pipeline. I could have ingested the XML and then written out my own data format. But that would mean I'd have to also do the reverse if I wanted to pass the data along or return it after some modifications and I have to define yet another data format. By creating my own namespace, I am able to annotate the existing data without affecting the source schema and I can simply strip out my namespace when passing the processed data along to someone else. Every data format should be so versatile.

Transformation, Part 1: Templating

When writing webapps, there are literally dozens of templating engines and there's constantly new ones emerging. I chose to learn XSLT some years back because I liked how Cocoon and AxKit handled web pages. Just create your data in XML and then transform it using XSLT according to the delivery needs. So far, nothing especially unique compared to other templating engines. Except unlike most engines, it didn't rely on some program creating the data and then invoking the templating code. XSLT works with dynamic Apps as easily as with static XML or third party XML without having.

Since those web site roots, I've had need for email templating and data transformation in .NET projects and was able to leverage the same XSLT knowledge. That means I don't have to pick up yet another tool to do a familiar task just a little differently.

What's the file format?

When I first started playing with Xaml, I was taking Live For Speed geometry data and wanted to render it in WPF and Silverlight. Sure, I had to learn the syntax of the geometry constructs, but I didn't have to worry about figuring out the data format. I just used the more than familiar XmlDocument and was able to concentrate on geometry, not file formats.

Transformation, Part 2: Rewriting

Currently I'm working with Xaml again for a Silverlight project. My problem was that I had data visualization in Xaml format (coming out of Illustrator), as well as associated metadata (a database of context data) and I needed to attach the metadata to the geometry, along with behavior. Since the first two are output from other tools I needed a process that could be automated. One way would be to walk the Visual tree once loaded, create a parallel hierarchy of objects containing the metadata and behavior and attach their behavior to the visual tree. But i'd rather have the data do this for itself.

<Canvas x:Name="rolloverContainer_1" Width="100" Height="100">
  <!-- Some geometry data -->
</Canvas>

<!-- becomes -->

<droog:RolloverContainer x:Name="rolloverContainer_1" Width="100" Height="100">
  <!-- Some geometry data -->
</droog:RolloverContainer>

So I created custom controls that subclassed the geometry content containers. I then created a post-processing script that simply loaded the Xaml into the DOM and rewrote the geometry containers as the appropriate custom controls using object naming as an identifying convention. Now the wiring happens automatically at load, courtesy of Silverlight. Again, no special parser required, just using the same XmlDocument class I've used for years.

And finally, Serialization

I use XML serialization for over the wire transfers as well as data and configuration storage. In all cases, it lets me simply define my DTOs and use them as part of my object hierarchy without ever having to worry about persistence. I just save my object graph by serializing it to XML and rebuild the graph by deserializing the stream again.

I admit that this last bit does depend on some language dependent plumbing that's not all that standard. In .NET, it's built in and let's me mark in my objects with attributes. In Java, I use Simple for the same effect. Without this attribute driven mark up, I'd have to walk the DOM and build m objects by hand, which would be painful.

Sure, for data, binary serialization would be cheaper and more compact, but that misses the other benefits I get for free. The data can be ingested and produced by a wide variety of other platforms, I can manually edit it, or easily build tools for editing and generation, without any specialized coding.

For my Silverlight project, I'm currently using JSON as my serialization layer between client and server, since there currently is no XmlSerializer or even XmlDocument in Silverlight 1.1. It, too, was painless to generate and ingest and, admittedly, much more compact. But I then I added this bit to my DTO:

List<IContentContainer> Containers = new List<IContentContainer>();

It serialized just fine, but then on the other end it complained about there not being a no-argument constructor for IContentContainer. Ho Hum. Easily enough worked around for now, but I will be switching back to XML for this once Silverlight 2.0 fleshes out the framework. Worst case, I'll have to build XmlSerializerLitem, or something like that, myself.

All in all, XML has allowed me to do a lot of data related work without having to constantly worry about yet another file format, or parser. It's really not about being the best format, but about it virtually being everywhere and being supported with a mature toolchain across the vast majority of programming environment and that pays a lot of dividents, imho.

Searching a Tree of Objects with Linq

noreply@blogger.com (ether) — Wed, 28 Nov 2007 19:22:00 +0000

I've finally had legitimate use for LINQ to Objects, not just to make the syntax cleaner, but also to simplify the underlying code and provide me a lot of flexibility without significant work.

The scenario

I have a tree of objects that have both a type and a name. The name is unique, the Type is not. The interface is this:

public interface INode
{
  int Id { get; }
  string Name { get; set; }
  string Type { get; set; }
  List<INode> Children { get; }
}

I want to be able to find a single named Node in the tree and I want to be able to retrieve a collection of all nodes for a particular type. The searchable interface could be expressed as this:

public interface ISearchableNode : INode
{
  INode FindByName(string name);
  IEnumerable<INode> FindByType(string name);
}

Both require me to walk the tree and examine each node, so clearly I just want to have one walk routine and generically evaluate the node in question. In C# 2.0 parlance, that means I could pass an anonymous delegate into my generic find routine and have it recursively iterate through all the children. I also pass along a resultset to be populated.

The signature for the evaluation delegate looks like this:

delegate bool FindDelegate(INode node);

but since I'm using C# 3.0 (i.e. .NET 3.5) I can use lambda expressions to avoid creating a delegate and simplify my syntax. Instead of FindDelegate, I can simply use Func<INode,bool>:

// Instead of this:
private void Find(INode node, List<INode> resultSet, FindDelegate findDelegate);
// called like this for a Name search:
Find(this, resultSet, delegate(INode node) { return node.Name == name; });

// I can use this:
private void Find(INode node, List<INode> resultSet, Func<INode, bool> f)
// called like this:
Find(this, resultSet, node => node.Name == name);

Thus giving me the following implementation for ISearchableNode:



public INode FindByName(string name)
{
  List<INode> resultSet = new List<INode>();
  Find(this, resultSet, x => x.Name == name);
  return resultSet.FirstOrDefault();
}

public IEnumerable<INode> FindByType(string type)
{
  List<INode> resultSet = new List<INode>();
  Find(this, resultSet, x => x.Type == type);
  return (IEnumerable<INode>)resultSet;
}

private void Find(INode node, List<INode> resultSet, Func<INode, bool> f)
{
  if (f(node))
  {
    resultSet.Add(node);
  }
  foreach (INode child in node.Children)
  {
    Find(child, resultSet, f);
  }
}



Problem solved, move on... Well, except there is significant room for improvement. Here are the two main issues that ought to be resolved:


Syntax is limited to two types of searches and exposing the generic find makes for an ugly syntax. It would be much nicer if queries to the tree could be expressed in LINQ syntax.
It's also inefficient for the Name search, since I'm walking the entire tree, even if the first node matched the criteria.


LINQ to Hierarchical Data

In order to use LINQ to objects, I need to either create a custom query provider or implement IEnumerable. The latter is significantly simpler and could be expressed using the following interface:



public interface IQueryableNode : IEnumerable<INode>
{
}


Ok, ok, I don't even need an interface, I could just implement IEnumerable<INode>... But what does that actually mean? In the simplest sense, I'm iterating over the node's children, however, I also with do descend into the children's children and so on. So a simple foreach won't do. I could just do the same tree walking with a resultset as I did above and return the Enumerator of the resulting list to implement the interface, but C# 2.0 introduced a much more useful way to implement non-linear Iterators, i.e. the yield keyword. Instead of building a list to be interated over, yield let's the iterating code return values as they are found, which means it can be used for recursive iteration. Thus the GetEnumerator is implemented simply as follows



#region IEnumerable<Node> Members
public IEnumerator<INode> GetEnumerator()
{
  yield return this;
  foreach (Node child in Children)
  {
    foreach (Node subchild in child)
    {
      yield return subchild;
    }
  }
}
#endregion

#region IEnumerable Members
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{
  return this.GetEnumerator();
}
#endregion

Nice and simple and ready for LINQ.

Searching for all Nodes of a type becomes



var allBar = from n in x
                where n.Type == "bar"
                select n;
foreach (Node n in allBar)
{
  // do something with that node
}


and the search for a specifically named node becomes



INode node = (from n in x
              where n.Name == "i"
              select n).FirstOrDefault();



But the real benefit of this approach is that I don't have hard-coded search methods, but can express much more complex queries in a very natural syntax without any extra code on the Node.

Deferred execution

As it turns out, using yield for the recursive iteration also solved the second issue. As yield returns values as it encounters them during iteration, the search doesn't happen until the query is executed. And one of the side effects of LINQ syntax is that creating a query does not execute it until the result set is iterated over. Therefore, FirstOrDefault() actually short-circuits the query as soon as the first match (and in case of Name, it's going to be the only match) is hit.