Developer Journey

Stupid Cartoon Bears

Kevin Hazzard — Thu, 24 Feb 2011 05:01:00 GMT

Rahm Emanuel's use of a famous Yogi Berra quote during a recent Chicago Mayoral debate got me thinking about unfair history has been to arguably the best catcher in the history of baseball.

You don't have to agree with me that Yogi Berra was the best catcher in baseball history. Johnny Bench was a great catcher but there are many statisticians on my side who say that Yogi was better. And statistics never lie, right? Combined with my own well-conceived prejudices, I feel quite comfortable with my claim.

I was having a conversation with a twenty-something fellow at work recently when I mentioned Yogi Berra after quoting him in context. The youngster, ostensibly a baseball follower if not a fan, asked me if I was quoting that "stupid cartoon bear". It was funny until I realized he wasn't joking. Now close your eyes for a moment and imagine if you:

Were arguably the best catcher in the history of baseball
Played in and/or coached 18 All-Star games and 13 World Series
Played for the Yankees and Mets and won the American League MVP award 3 times over 19 years of play
Went on to coach the Yankees and Mets for 20 more years
Were inducted in to the Baseball Hall of Fame
Were elected by fans as 1 of 2 catchers on the Major League Baseball All-Century Team (guess who else?)

and your entire career was lost to generations of baseball fans because of a stupid cartoon bear. The Yogi Berra quote that seems most appropriate here: "If the world was perfect, it wouldn't be." True, Yogi. So true.

Free Alternatives to Reflector

Kevin Hazzard — Wed, 23 Feb 2011 12:00:00 GMT

Thanks to my friend Mike Etheridge for this news bit. A few days ago, I blogged about Red Gate's decision to make Reflector a commercial-only product. In other words, there will be no free version of Reflector when version 7 is released in late February of 2011. The developer community responded quickly.

On 17 February, about a week after my blog post concerning Red Gate's decision, JetBrains announced that they will be bundling a decompiler with ReSharper 6 after it launches. JetBrains hasn't released ReSharper 6 yet but you can get it via their Early Access Program nightly builds if you want to check it out. The decompiler looks quite good, at least the ways in which it's connected to the rest of the ReSharper suite: very well thought out and with the attention to detail that we've come to expect from the folks at JetBrains. The really good news from the announcement is that the decompiler will be unbundled and made available for free sometime after ReSharper 6 launches.

Another free tool called ILSpy was announced by Daniel Grunwald and David Srbecky on the SharpDevelop Wiki on 22 February. They claim that ILSpy development began after Red Gate's announcement just a few weeks ago which is somewhat amazing to me given the richness of what ILSpy can do in this initial release. A quick peek at the source code to ILSpy on github shows that they are using the Cecil (pronounced see-sil) decompiler from the mono project which explains why they've been able to achieve such a steep initial trajectory with ILSpy. I just hope that Daniel and David can find a way to voluntarily monetize their open source project in ways that Lutz Roeder (the inventor of Reflector) and Red Gate never did. A simple "Donate via PayPal" link in the About dialog would go a long way to making this tool free and technically excellent perpetually.

Richmond Code Camp XI

Kevin Hazzard — Mon, 14 Feb 2011 13:00:00 GMT

Richmond Code Camp XI will be held on May 21, 2011. If you are interested in speaking at RCC XI, complete and submit this form. You can also register as an attendee here. Spread the word and direct people here for more information, please. Submitted talks should be geared to 75 minute-long slots. We expect to offer 50 of these talks during this one-day event.

Richmond Code Camp is Microsoft-leaning because it is, after all, a .NET Code Camp. However, we will accept talks that are not Microsoft-oriented, especially if they compare and contrast other technologies against Microsoft’s platform in an objective way. Talks on security, language trends and techniques, emerging technologies, software development methodologies and communication frameworks are always popular, regardless of vendor. For whatever reason, we seem to be perpetually in need of more and deeper database-themed talks in particular, especially around database architecture, data modeling and business intelligence. We are also open to great talks on IT Pro (system administration & networking) topics, too.

For speakers, here are some tips for getting your talk selected:

Give it a twist: something unique that will both attract campers and promises to hold their interest for 75 minutes
Use a short, concise title that has marketing punch; don't get too cute when naming your talk
Make it relevant to work that people need to accomplish or to skills that people want to develop
In the abstract, sell your talk as conversational and code-focused
In the abstract, also let people know why you are qualified to be speaking about the topic
People don’t read bios as deeply as they do the abstracts so sell yourself in as few words as possible

This is our 11th Code Camp in Richmond as the name implies and we are expect attendance to be over 400 for this one-day, free event. Free WiFi, great parking, free breakfast, lunch and snacks. Great giveaways from our contributors. And we are known for hosting a really fun, totally gratis speaker dinner on the Friday evening before the Code Camp. Don't miss that! We expect to have our speakers and topics selected by April 1st so get your talks submitted soon. Directions to the campus and the selected topics list will be available on the official Richmond Code Camp site after April 1st.

Please remember to tell a few friends and colleagues about the Richmond Code Camp. Our number one complaint is that there are too many great talks. Bring tech-savvy friends and co-workers to divide and conquer! Thanks.

Red Gate's Decision to Commercialize Reflector

Kevin Hazzard — Wed, 09 Feb 2011 22:30:00 GMT

Red Gate's Co-CEO Neil Davidson announced in February 2011 that they would be making version 7 of the venerable Reflector product commercial-only. My thoughts are captured in this blog post and I hope that you'll take the opportunity to offer your insights and opinions as well.

When Neil made the announcement, there was a lot of traffic on the Twittersphere about what the decision means to the .NET developer community. Most of what I saw was fairly negative. It seems that #2 on Scott Hanselman's Big Ten Life and Work-Changing Utilities was presumed by many to be a perpetually free utility with perpetual updates to deal with language changes, Framework revisions and CLR changes, of course. Reflector also appeared prominently in James Avery's MSDN Magazine article entitled Ten Must-Have Tools That Every Developer Should Download Now. In that article, James says of his example, "it does not represent what I consider to be the best use of .NET Reflector, which is to examine .NET Framework assemblies and methods" which brings me around to my opinion on the subject.

Since 2008, Microsoft has opened up much of the .NET Framework Class Library (FCL) source via the Microsoft Symbol Server. Setting up the symbol server in Visual Studio lets you literally step into Microsoft's code for much of the FCL. Of course, not everything's available but I've stepped into Microsoft's code hundreds, maybe thousands of times over the past few years and found what I needed at least 99 percent of the time. In 2007, if James was correct that the best use of Reflector is to examine (really to reverse engineer) .NET Framework assemblies and methods, how important is Reflector in a world where Microsoft's Framework code is simple to obtain, especially in the context of a debugger where parameters, results and side-effects can be evaluated as they occur? My opinion is that Reflector isn't all that useful anymore.

My co-author of Metaprogramming in .NET, Jason Bock, and I were discussing the impact of Red Gate's decision on our book. Jason summed up the impact on him pretty well when he said to me, "To be honest, I really don't use it that much anymore. Decompiling code isn't something I need to do a lot of." That's pretty telling coming from a hard-core metaprogrammer like Jason. If he doesn't need Reflector, who does? At my company, CapTech Consulting, there was a lively discussion of the topic on one of our internal Microsoft-oriented mailing lists. Darrell Norton, a fellow Microsoft MVP and all-around genius, quickly reached the same conclusion that I did. When I can step into most of Microsoft's code, why do I need to disassemble it? David Catherman, another of my colleagues also offered up a tool called Mole by Karl Shifflett as an alternative to Reflector. I haven't used it yet but it looks very useful. I wish I had time to run down all the cool stuff I hear about. I'll put Mole in position 487 of the cool things I need to check out for now.

In closing, I should say that my reckoning about Reflector's usefulness has nothing to do with the company Red Gate. Anyone who knows me knows that I just love Red Gate products. I use them at work every day and honestly, I just can't imagine doing my job without them. My work would be half as fulfilling and take me ten times longer to do it if it weren't for the SQL Toolbelt and the .NET Developer Bundle products. When my clients see me using these tools, they're typically shocked with the kind of productivity I can achieve. They often buy the tools for their own staffs and I don't even get any kind of kick-back from Red Gate. Note to self... Besides being thoroughly conceived from a design perspective, the Red Gate tools are rock solid performers on the job, too. And when I have a problem, the online support forums are a good way to get the information I need. The Red Gate support staff are really good at monitoring the forums and responding with thorough and helpful responses. By making Reflector 7 a fully commercial product, perhaps we'll see that famous Red Gate support and attention-to-detail start to shine through with features that make it more than just a good tool for disassembling Microsoft's Framework assemblies. For $35 US, it will most likely be a bargain, given Red Gate's track record. That's my opinion, anyhow. And if you don't want to spend the $35, try Mole or set up the Microsoft Symbol Server in Visual Studio to see what kind of utility you get out of those solutions.

Using T4 to Generate SQL Merge Scripts

Kevin Hazzard — Tue, 08 Feb 2011 17:30:00 GMT

In the forthcoming book that Jason Bock and I are working on called Metaprogramming in .NET, we cover a lot of code generation techniques. Unfortunately, we weren’t able to fit a full chapter into the first version of the book concerning Microsoft’s Text Template Transformation Toolkit (T4) so I’ll dedicate some blog posts to the topic instead. In this post, I’ll show you how to use T4 to generate MERGE scripts in T-SQL.

The following T4 script requires that the Microsoft SQL Server 2008 (or 2005) SQL Management Objects (SMO) assemblies are installed in the Global Assembly Cache (GAC). It also assumes that I have the Northwind database installed in my “(local)” instance of SQL server and that there’s another instances called ReplicatedNorthwind. The idea here is that I’m going to use a T4 script to generate T-SQL code that uses the MERGE statement to “upsert” data from the Northwind Products table to the ReplicatedNorthwind Products table. Changed records will be updated. New records will be inserted and those records deleted from the source will also be deleted from the target. T-SQL’s MERGE statement wraps all this functionality up into a one logical operation:

First, let’s look at the T4 script. If you're reading this directly on my blog (not via syndication), you can double-click the code and copy it with Ctrl-C:

<#@ template language="C#" #>
<#@ output extension="SQL"#>
<#@ assembly name="Microsoft.SqlServer.Smo" #>
<#@ assembly name="Microsoft.SqlServer.ConnectionInfo" #>
<#@ assembly name="Microsoft.SqlServer.Management.Sdk.Sfc" #>
<#@ import namespace="Microsoft.SqlServer.Management.Smo" #>
<#
  string serverName = "(local)";
  string sourceDbName = "Northwind";
  string targetDbName = "ReplicatedNorthwind";
  string schemaName = "dbo";
  string entityName = "Products";

  Server server = new Server(serverName);
  Database targetDatabase = new Database(server, targetDbName);
  Table targetTable = new Table(targetDatabase, entityName, schemaName);
  targetTable.Refresh();

  int index = 0;
#>
BEGIN TRANSACTION;
BEGIN TRY
MERGE [<#= targetDbName #>].[<#= schemaName #>].[<#= entityName #>] AS [Target]
  USING [<#= sourceDbName #>].[<#= schemaName #>].[<#= entityName #>] AS [Source]
    ON
    (
      /* join on the primary key column(s) */
<#
  PushIndent("\t\t\t");
  index = 0;
  foreach (Column column in targetTable.Columns)
  {
    if (column.InPrimaryKey)
    {
      if (index > 0)
        Write("AND ");
      WriteLine("[Target].[" + column.Name + "] = [Source].[" + column.Name + "]");
      index++;
    }
  }
  PopIndent();
#>
    )
  WHEN MATCHED
    AND
    (
      /* and compare all non-key column(s) for inequality */
<#
  PushIndent("\t\t\t");
  // This compares all the values to look for update-ability
  // which can slow down the merge. If you have an update date
  // or something like it in your schema, you can check for its
  // existence here and use that to make the merge efficient.
  index = 0;
  foreach (Column column in targetTable.Columns)
  {
    if (!column.InPrimaryKey)
    {
      if (index > 0)
        Write("OR ");
      WriteLine("[Target].[" + column.Name + "] <> [Source].[" + column.Name + "]");
      index++;
    }
  }
  PopIndent();
#>
    )
    THEN UPDATE SET
<#
  PushIndent("\t\t\t");
  index = 0;
  foreach (Column column in targetTable.Columns)
  {
    if (!column.InPrimaryKey)
    {
      if (index > 0)
        Write(", ");
      WriteLine("[Target].[" + column.Name + "] = [Source].[" + column.Name + "]");
      index++;
    }
  }
  PopIndent();
#>
  WHEN NOT MATCHED BY TARGET
    THEN INSERT
    (
<#
  PushIndent("\t\t\t");
  index = 0;
  foreach (Column column in targetTable.Columns)
  {
    if (index > 0)
      Write(", ");
    WriteLine("[" + column.Name + "]");
    index++;
  }
  PopIndent();
#>
    )
    VALUES
    (
<#
  PushIndent("\t\t\t");
  index = 0;
  foreach (Column column in targetTable.Columns)
  {
    if (index > 0)
      Write(", ");
    WriteLine("[Source].[" + column.Name + "]");
    index++;
  }
  PopIndent();
#>
    )
  WHEN NOT MATCHED BY SOURCE
    /* are you sure you want to delete? */
    /* this could be converted to a logical delete */
    /* by using an update on the target instead */
    THEN DELETE
  OUTPUT
    $action AS [Operation]
<#
  PushIndent("\t\t");
  foreach (Column column in targetTable.Columns)
  {
    if (column.InPrimaryKey)
      WriteLine(", Inserted.[" + column.Name + "] AS [Merged" + column.Name + "]");
  }
  PopIndent();
#>
<#
  PushIndent("\t\t");
  index = 0;
  foreach (Column column in targetTable.Columns)
  {
    if (index > 0)
      WriteLine("");
    if (column.InPrimaryKey)
      Write(", Deleted.[" + column.Name + "] AS [Deleted" + column.Name + "]");
  }
  WriteLine(";");
  PopIndent();
#>
END TRY
BEGIN CATCH
  IF @@TRANCOUNT > 0
  BEGIN
    ROLLBACK TRANSACTION;
    PRINT N'The transaction was rolled back because the merge failed.';
  END
  PRINT N'Error number = ' + CONVERT(NCHAR(10),ERROR_NUMBER());
  PRINT N'Error severity = ' + CONVERT(NCHAR(10),ERROR_SEVERITY());
  PRINT N'Error state = ' + CONVERT(NCHAR(10),ERROR_STATE());
  PRINT N'Error procedure = ' + ERROR_PROCEDURE();
  PRINT N'Error line = ' + CONVERT(NCHAR(10),ERROR_LINE());
  PRINT N'Error message = ' + ERROR_MESSAGE();
END CATCH

IF @@TRANCOUNT > 0
BEGIN
  COMMIT TRANSACTION;
  PRINT N'The transaction was committed with no errors.';
END

In the pre-amble, I specified that C# will be the scripting language and that we’re going to emit SQL code from it. Next, some assembly directives are used to load referenced assemblies into the script generator’s context. The three (3) SMO assemblies referenced here are expected to be in the GAC but they could optionally be placed in the script engine’s private assembly path, too. In a future post, I’ll show you how to resolve assemblies in a much more predictable way. For now, let’s depend on the GAC.

Next comes an import directive. This is akin to the Import keyword in Visual Basic or the using declaration in C#. Notice that I said using declaration, not using statement because in C# the using keyword can be used in both ways. A using declaration in C# is used to import the objects from a specified namespace. And that’s what I’m asking T4’s scripting engine to do here. I want to import the Microsoft.SqlServer.Management.Smo namespace for use throughout the script that follows.

The remainder of the script is T-SQL code with bits of C# script sprinkled about using <# (or <#=) and #> tags. T4’s scripting engine will compile and execute the script from top to bottom, injecting literal T-SQL text and evaluating the C# expressions and injecting their output as well. The output, which shows up under the TT file in Visual Studio’s Solution Explorer looks like this:

BEGIN TRANSACTION;
BEGIN TRY
MERGE [ReplicatedNorthwind].[dbo].[Products] AS [Target]
  USING [Northwind].[dbo].[Products] AS [Source]
    ON
    (
      /* join on the primary key column(s) */
      [Target].[ProductID] = [Source].[ProductID]
    )
  WHEN MATCHED
    AND
    (
      /* and compare all non-key column(s) for inequality */
      [Target].[ProductName] <> [Source].[ProductName]
      OR [Target].[SupplierID] <> [Source].[SupplierID]
      OR [Target].[CategoryID] <> [Source].[CategoryID]
      OR [Target].[QuantityPerUnit] <> [Source].[QuantityPerUnit]
      OR [Target].[UnitPrice] <> [Source].[UnitPrice]
      OR [Target].[UnitsInStock] <> [Source].[UnitsInStock]
      OR [Target].[UnitsOnOrder] <> [Source].[UnitsOnOrder]
      OR [Target].[ReorderLevel] <> [Source].[ReorderLevel]
      OR [Target].[Discontinued] <> [Source].[Discontinued]
    )
    THEN UPDATE SET
      [Target].[ProductName] = [Source].[ProductName]
      , [Target].[SupplierID] = [Source].[SupplierID]
      , [Target].[CategoryID] = [Source].[CategoryID]
      , [Target].[QuantityPerUnit] = [Source].[QuantityPerUnit]
      , [Target].[UnitPrice] = [Source].[UnitPrice]
      , [Target].[UnitsInStock] = [Source].[UnitsInStock]
      , [Target].[UnitsOnOrder] = [Source].[UnitsOnOrder]
      , [Target].[ReorderLevel] = [Source].[ReorderLevel]
      , [Target].[Discontinued] = [Source].[Discontinued]
  WHEN NOT MATCHED BY TARGET
    THEN INSERT
    (
      [ProductID]
      , [ProductName]
      , [SupplierID]
      , [CategoryID]
      , [QuantityPerUnit]
      , [UnitPrice]
      , [UnitsInStock]
      , [UnitsOnOrder]
      , [ReorderLevel]
      , [Discontinued]
    )
    VALUES
    (
      [Source].[ProductID]
      , [Source].[ProductName]
      , [Source].[SupplierID]
      , [Source].[CategoryID]
      , [Source].[QuantityPerUnit]
      , [Source].[UnitPrice]
      , [Source].[UnitsInStock]
      , [Source].[UnitsOnOrder]
      , [Source].[ReorderLevel]
      , [Source].[Discontinued]
    )
  WHEN NOT MATCHED BY SOURCE
    /* are you sure you want to delete? */
    /* this could be converted to a logical delete */
    /* by using an update on the target instead */
    THEN DELETE
  OUTPUT
    $action AS [Operation]
    , Inserted.[ProductID] AS [MergedProductID]
    , Deleted.[ProductID] AS [DeletedProductID];
END TRY
BEGIN CATCH
  IF @@TRANCOUNT > 0
  BEGIN
    ROLLBACK TRANSACTION;
    PRINT N'The transaction was rolled back because the merge failed.';
  END
  PRINT N'Error number = ' + CONVERT(NCHAR(10),ERROR_NUMBER());
  PRINT N'Error severity = ' + CONVERT(NCHAR(10),ERROR_SEVERITY());
  PRINT N'Error state = ' + CONVERT(NCHAR(10),ERROR_STATE());
  PRINT N'Error procedure = ' + ERROR_PROCEDURE();
  PRINT N'Error line = ' + CONVERT(NCHAR(10),ERROR_LINE());
  PRINT N'Error message = ' + ERROR_MESSAGE();
END CATCH

IF @@TRANCOUNT > 0
BEGIN
  COMMIT TRANSACTION;
  PRINT N'The transaction was committed with no errors.';
END

If you have a Northwind database locally and another database locally named ReplicatedNorthwind that contains a Products table with the correct schema, the resulting SQL script will merge changes that occur in the Northwind Products table to the other table. Try inserting a product, updating a product and deleting a product in the source Products table and running the merge script. You’ll see that all three (3) changes are detected and executed for you to synchronize the tables. If you don’t have Northwind databases handy, try modifying the T4 script to work with two other databases of your choosing instead. The only real requirement is that the target table’s schema has to be a subset of the source table’s schema to function correctly.

I won’t dig into all the particulars of how T-SQL MERGE works. It’s a fairly clean syntax and there are plenty of resources on the net for learning about that. What I wanted to make clear in this post is how simple the code generation process is when using T4. You can see how simple it is to iterate over the targetTable.Columns collection and evaluate each Column object’s InPrimaryKey property to build up the MERGE statement.

In my next post concerning the T4 code generator, I’ll show you how to pass parameters to the T4 script by implementing the ITextTemplatingEngineHost and ITextTemplatingSessionHost interfaces. In that post, I'll build one metaprogramming technique on another to create a customized script generator that can generically MERGE one entire database to another.

Metaprogramming in the .NET Micro Framework

Kevin Hazzard — Sun, 06 Feb 2011 04:45:00 GMT

Metaprogramming capability has been in the .NET Framework since day 1. So it’s no surprise that the .NET Micro Framework has some basic metaprogramming capabilities, too. Most of the introductory examples you see for working in the .NET Micro Framework involve blinking an LED. So, why not blink out “Hello world” on an LED in Morse code? And because we can, why not do it in metaprogramming style?

The basic metaprogamming functionality that we’ll take advantage of in this little application is the multicast delegate. Morse code letters can be expressed as a series of dots and dashes. The dots are short signals that are 1/3 the length of the dashes. And there are rules for intra-letter spacing, intra-word spacing and extra-word spacing. The word “HELLO” might be expressed as four dots or “....” for H, a single dot “.” for E, the sequence “.-..” for L and so on. With the appropriate spacing in between, of course. The goal is to blink the LED on my Netduino to emit the correct Morse code sequence for a string.

private Hashtable _letters = new Hashtable();
public Morse(OutputPort led)
{
  _letters.Add(' ', "*");
  _letters.Add('A', ".-");
  _letters.Add('B', "-...");
  _letters.Add('C', "-.-.");
  // remainder omitted for brevity

The first step is to insert each dot-dash sequence into a dictionary keyed by each letter of the alphabet. The .NET Micro Framework 4.1 doesn’t support generic collection classes so we have to use an old-fashioned Hashtable. That’s OK because I’ll take advantage of the weak typing in the Hashtable to demonstrate memoizing my dynamic Morse code sequences in the next section. Now that I’ve got the letters and their dot-dash sequences in a dictionary, now I need some functions for the dot, dash and space parts.

private int dotLength = 200;
private void dash()
{
  _led.Write(true);
  Thread.Sleep(dotLength * 3);
  _led.Write(false);
}

private void dot()
{
  _led.Write(true);
  Thread.Sleep(dotLength);
  _led.Write(false);
}

private void intraletterspace()
{
  Thread.Sleep(dotLength);
}

private void intrawordspace()
{
  Thread.Sleep(dotLength * 3);
}

private void extrawordspace()
{
  // this is actually 7 but we'll borrow
  // the intraword space from the end of
  // the last word
  Thread.Sleep(dotLength * 4);
}

public delegate void seq();

Notice the seq delegate type declared here? The various dot, dash and space functions shown here will be assembled into a seq delegate depending on their formulations found in the _letters Hashtable. The function to do the conversion from string to multicast delegate looks like:

private seq ConvertToDelegate(string letterSeq)
{
  seq del = null;
  bool addIntraWordSpace = true;
  foreach (char c in letterSeq.ToCharArray())
  {
    switch (c)
    {
      default:
        break;
      case '*':
        del += extrawordspace;
        addIntraWordSpace = false;
        break;
      case '.':
        if (del == null)
          del = dot;
        else
        {
          del += intraletterspace;
          del += dot;
        }
        break;
      case '-':
        if (del == null)
          del = dash;
        else
        {
          del += intraletterspace;
          del += dash;
        }
        break;
    }
  }
  if (addIntraWordSpace)
    del += intrawordspace;
  return del;
}

The conversion function evaluates each dot, dash or space in the source string and assembles a seq delegate that invokes each of the appropriate functions in order. Isn’t metaprogramming fun? We’re actually treating each symbol in the string like an instruction in a little programming language and converting them into function calls. The delegate type, which is really a MulticastDelegate from the Framework, does all the hard work. When we use the += operator to add a new function call, the MulticastDelegate adds a new function to the list of those to be invoked. The last step is to invoke the sequences as letters are encountered in a source string like “Hello world”.

public void Emit(string text)
{
  if (text == null)
    return;
  text = text.Trim().ToUpper();
  if (text.Length == 0)
    return;

  foreach (char c in text.ToCharArray())
  {
    char actual = c;
    if (!_letters.Contains(c))
      actual = ' ';
    object s = _letters[actual];
    if (s is string)
    {
      // memoize the multicast delegate for each letter
      _letters[actual] = s = ConvertToDelegate(s as string);
    }
    (s as seq)();
  }
}

The code is simple enough. It just iterates over the characters in an input string, converts the associated sequence characters to seq delegates and invokes them. However, like any good metaprogramming application, we always look for chances to optimize the code generation steps. They are the most expensive so caching the results of our little “dot-dash compiler” is really important. The weak typing of the Hashtable makes this easy enough. When a new letter is encountered during emission of the phrase, it’s compiled into a seq delegate and cached in the Hashtable in place of the original string (source code) that was used to create it. A bit of reflection is used to determine whether or not the object in the value part of the Hashtable is a string (my dot-dash style source code) or a seq delegate that can actually be invoked to flash the LED on my Netduino. Once a letter has been compiled, it never has to be compiled again while the application is running.

The single C# source code file for this fun, little application can be downloaded here.

Common Table Expression Recursion

Kevin Hazzard — Wed, 02 Feb 2011 03:00:00 GMT

I like using recursion when it helps to make an algorithm clearer. That's not always the case though. For many developers, recursion is difficult to understand. So I'm always on the hunt for code samples that make recursion easier to understand. One good example of this can be found in traversing the foriegn keys in a database using a SQL Server Common Table Expression (CTE).

When I load data into a database, I'd like to know the order I must use to load the tables. Some tables will be at Level Zero (0). Those are the ones that have no dependencies on other tables. They're typically type tables that contain reference data and they can be loaded at any time because they have no foreign keys in them. The next level, Level One (1), are those tables that depend on the Level Zero (0) tables. And so on, and so on. If I knew the load order, my Extract, Transform and Load (ETL) packages could be scripted to run level by level starting at Level Zero (0). A Common Table Expression (CTE) can be used to find that load order using recursion. Let's look at the code first, then I'll describe it.

WITH
[key_info] AS
(
  SELECT
    OBJECT_SCHEMA_NAME([sfk].[parent_object_id]) AS [from_schema]
    , OBJECT_NAME([sfk].[parent_object_id]) AS [from_table]
    , OBJECT_SCHEMA_NAME([sfk].[referenced_object_id]) AS [to_schema]
    , OBJECT_NAME([sfk].[referenced_object_id]) AS [to_table]
  FROM [sys].[foreign_keys] AS [sfk]
  WHERE [sfk].[parent_object_id] <> [sfk].[referenced_object_id]
)
, [level_info] AS 
(
  -- the anchor part
  SELECT
    [ss].[name] AS [SchemaName]
    , [st].[name] AS [TableName]
    , 0 AS [ReferenceLevel]
  FROM [sys].[tables] AS [st]
  INNER JOIN [sys].[schemas] AS [ss]
    ON [st].[schema_id] = [ss].[schema_id]
  LEFT OUTER JOIN [key_info] AS [ki]
    ON [ss].[name] = [ki].[from_schema]
      AND [st].[name] = [ki].[from_table]
  WHERE [ki].[from_schema] IS NULL

  UNION ALL

  -- the recursive part
  SELECT
    [ki].[from_schema]
    , [ki].[from_table]
    , [li].[ReferenceLevel] + 1
  FROM [key_info] AS [ki]
  INNER JOIN [level_info] AS [li]
    ON [ki].[to_schema] = [li].[SchemaName]
      AND [ki].[to_table] = [li].[TableName]
)
SELECT
  [li].[SchemaName]
  , [li].[TableName]
  , MAX([li].[ReferenceLevel]) AS [LoadLevel]
FROM [level_info] AS [li]
GROUP BY [li].[SchemaName], [li].[TableName]
ORDER BY MAX([li].[ReferenceLevel]), [li].[TableName];

The WITH statement kicks off the CTE by establishing two data elements that will be used later on. The first one is called [key_info] and it's just a dump of the foreign keys in the database, capturing parent (from) tables to referenced (to) tables. Care must be taken not to include those keys that refer back to the same table. We don't need them for discovering the load levels and they will cause the recursion to become infinite. More on that in a moment.

The [key_info] part of the WITH statement isn't recursive but the next part named [level_info] is. You can tell it's a recursive CTE by finding the UNION ALL statement in the middle of the [level_info] definition. The part of the query before the UNION ALL is called the anchor part. The part following it is the recursive part. The anchor part of [level_info] builds a list of schemas and tables then LEFT OUTER JOINs them to the [key_info] using the parent (from) side of the keys. The trick to understanding this is in realizing (A) that it's a LEFT OUTER JOIN to the key data and that (B) it uses a WHERE clause that includes only those tables with NULL keys on the parent (from) side of the LEFT OUTER JOIN. In other words, I'm including only those tables that have no foreign key references. Also notice how the [ReferenceLevel] is hard-coded to zero (0) in this anchor part? These are our Level Zero (0) tables: the ones that can be loaded at any time because they don't depend on anything else.

After the anchor part establishes the Level Zero (0) tables, the recursive part below the UNION ALL finishes the job. SQL will execute the recursive part over and over again until no new data is unioned in to the results. That's how it knows when to stop so it's important to make sure that you write the recursive part in such a way that it will actually finish. Had we included self-referencing (same table) keys in the [key_info], we'd have had a real problem getting the recursive part to finish.

The recursive part of this query will finish because the referenced (to) values in the [key_info] are joined with the schema and table combinations in [level_info] from the last recursion pass (or the anchor results if it's the first call to the recursive part, of course). Joining on the referenced (to) side of the key is important because we want to find all the tables already in the [level_info] list that have references "to" them. Then, in the [level_info] data in this pass, we capture the parent (from) data on the matches and increment the [ReferenceLevel] to one greater than the last known value. This could result in the same schema/table combination being included in the recursive results more than once, potentially at different levels. But that's OK. We take care of the duplicates in the last part of the query.

Following the CTE that does all the recursive work is the consumption part of the query. It can be a SELECT statement or an INSERT that selects from the CTE as its source. In this case, we simply select out the data, grouping by the schema and table name and picking off the MAX([ReferenceLevel]). It's important to pick the maximum reference level that was recorded for any table because it may refererence tables at varying load levels. As an example, imagine a table named C that references another table A directly using a foreign key. Table C may also reference a table named B that in turn references table A. In our ungrouped [level_info] results, Table A would appear twice: once as a Level One (1) table and again as a Level Two (2) table. We need to select the Level Two (2) row to make sure that we don't try to load table C before table B is fully loaded; hence the use of the MAX function in the final SELECT statement.

Running this query on the AdventureWorks database, for example, shows that there are eight (8) layers of data to be loaded into this database. In Level Zero (0), we find basic lookup tables like [Person].[AddressType] and [Sales].[SalesReason]. Tables like that don't depend on anything else so they can be updated first. At the other end of the spectrum are the tables at Level Seven (7). These are highly dependent tables like [Sales].[SalesOrderDetails] and [Sales].[SalesOrderHeaderSalesReason]. And, of course, there are dozens of tables in the six (6) layers in between. Try this query on your own databases to see what it reveals about the organization of your data. Missing foreign keys might become more apparent to you by examining the results.

Hopefully, this query helps you to wrap your brain around recursion a bit better, too. Enjoy.

Server Side JavaScript

Kevin Hazzard — Wed, 19 Jan 2011 12:00:00 GMT

I’m going to warn you that this is a rant. I just can’t take it anymore. Microsoft is a great company but there are some things going on at Microsoft these days that are very, very unsettling, in my opinion. I am going to address one of those issues in as graceful and as courteous a manner as I can muster. At a minimum, I promise not to go all Bellware on them. Here it is.

JavaScript is arguably the most popular programming language on planet Earth. I’m not sure what the Alpha Centaurians code in but it probably looks a lot like JavaScript, too. JavaScript isn’t perfect but it’s deliciously dynamic and declarative and just as functional as most people really desire in a general purpose programming language. I totally appreciate the great things that Microsoft is doing on the client side to make JavaScript faster, more standards (meaning jQuery) compliant and to provide better tooling. But don’t we deserve the power and flexibility of JavaScript on the server side, too?

Back in the day, I coded using Server Side JavaScript (SSJS) on Netscape Web Servers running on Solaris. It wasn’t that great an experience as I recall. It was OK but the development tool support was pretty weak and the debugging story was just awful. But JavaScript has matured a lot over the past 15 years since control was handed over to ECMA. Development environment support for JavaScript is very good now, even from Microsoft which seemed to be doing everything possible to make the JavaScript experience for developers and end-users poor until a few years ago. After so much noise, I suppose they couldn't ignore us any longer. And it’s a good thing, too. Without a fast, highly-compliant scripting language in Internet Explorer 9, that browser version would be doomed to be Microsoft’s last, in my opinion. Microsoft really does get it now about JavaScript in the browser. Hiring someone like Rey Bango is a testament to their understanding. Microsoft simply must have an excellent JavaScript story to tell with IE9. It’s got to pass all the benchmarks and compatibility tests with the highest possible marks. And the Visual Studio integration has to be tight, especially with respect to debugging. I have no doubt that Microsoft will do everything that they need to do to make JavaScript a first-class citizen within Internet Explorer from now on.

Microsoft’s JavaScript story on the server side isn’t all that rosy, however. In fact, other than a nascent, very limited version that showed up in some DLR alpha bits, there is no SSJS story from Microsoft. My repeated attempts to get Microsoft programming language product managers to engage in meaningful conversation about SSJS have fallen on deaf ears. They usually just stare at me when I say that we really need a good SSJS story as if I’m speaking Alpha Centaurian. They usually nod and say something like, “Yeah, well that’s nice but um… I need to go. My little brother got his arm stuck in the microwave so my mom had to take him to the hospital… And my grandmother freaked out and hijacked a busload of penguins. So, it’s kind of a family crisis. Buh-bye.”

Unfortunately, this is pretty much in line with the dynamic language story that we’re seeing from Microsoft across the board. The IronPython implementation got just up to the point of near perfection and BOOM! They dropped it. I think it’s somewhat noble of Microsoft to turn support of IronPython and IronRuby over to the community as they did. They didn’t just abandon it. But let’s be honest. You don’t give control away to the community if you’re invested. The community will do good things with both of the Iron languages but some problems require a company at Microsoft’s scale to get the mindshare they need to succeed. Microsoft’s decision to give up (for all intents and purposes) on IronPython and IronRuby is just a bad business decision, in my opinion, no matter what other irons (pardon the pun) Microsoft has in the fire. I know that Microsoft has other priorities but they made a huge mistake within the developer community with those decisions.

When I first met Harry Pierson of Microsoft a few years ago, his enthusiasm for the Python language rubbed off on me. I already loved Python from having worked around the RocketMail guys many years before when I was at Intel (before the Yahoo! acquisition of RocketMail). I could tell after swapping ideas about IronPython and the DLR around for a few hours that Harry and I were kindred spirits. Harry’s love for the Python language is totally infectious. I’ve always had a penchant for dynamic languages and Python, in particular, really rung my bell, so to speak. Python’s syntax is so light and clean and fluent. There’s little or no ceremony required to get truly wonderful work done with a few lines of code. And the standard library is very rich: “Batteries Included” we say in the Python world. Best of all, Python has excellent support for my other language love affair: functional programming. So much so that when I started working with the F# language, it felt Pythonic to me. It’s no wonder that a couple of years later, Luke Hoban (from Microsoft’s F# team) revealed to me that the F# programming language was deliberately designed to attract Python developers. From a statically-typed versus dynamically-typed perspective, these two languages couldn’t be more in opposition to one another. But the feel of working in F# is authentically Pythonic. And according to Luke, that feeling was embedded there quite deliberately.

Recently, I’ve been thinking about the state of things in the language world. My one biggest regret is failing to tell Microsoft how important dynamic languages are to me. I blogged about IronPython a lot. I gave dozens of user group and Code Camp presentations on integrating static and dynamic languages using .NET technologies. I even have a book coming out this year called Metaprogramming in .NET from Manning that contains a hundred plus pages on the Dynamic Language Runtime and all the great metaprogramming things that can be done with it. The DLR is firmly in place in the .NET Framework so I’m not worried about that. But seeing what’s happened to IronPython and IronRuby over the past few months, I lay awake at night sometimes wondering what else I could have done to make influential fellows like Scott Guthrie and Scott Hanselman and Somasegar understand why they should be showing their support publicly and within Microsoft for dynamic languages. I honestly feel like I failed my developer colleagues by not being a better spokesperson for Python and Ruby on .NET. Moreover, my grief about the subject spills over into other languages that should have been. Those that could have been. Server Side JavaScript is chief among them.

My reasons for wanting Server Side JavaScript running on the DLR are pretty simple. Today, I write in a multitude of “languages”: C#, F#, T-SQL, PL/SQL, JavaScript, CSS, XHTML, XSLT and a handful of other XML dialects that come in handy on occasion. It’s all too much really. As I use each of these “tools”, I ponder why they fit well for solving particular problems. Each of these languages is appropriately domain-specific which then leads me to wonder why I work in so many domains. Setting that question aside for a moment, I can’t help but think that if all the genius floating about in Microsoft’s Building 41 were aimed at creating a world-class Server Side JavaScript implementation on the DLR, we could rest our heads on our pillows at night enjoying a common language between at least two of our important domains.

I imagine a world where I might write JavaScript to render an MVC application, for example. The tooling in the environment I envision is so tight that emitting JavaScript to run on either side of the “pipe” is simply a matter of choosing the correct namespace. Better still, the JavaScript I’m thinking of has a runtime engine that decides where the code should execute. On a slower connection, the engine might decide to remote some of the code to the client side to provide a better user experience. On faster connections, it might decide to generate more AJAX callbacks from the client and run more of the code on the server to speed up the application or to enhance security. Emitting minified or pre-compiled versions of types and functions in my library would also be totally invisible to me. It would be automatic. Intrinsic. Creating markup from this system would be completely integrated, too. Calling an application a web app or a desktop client would be a thing of the past. There would only be apps. Making mental distinctions about where or how code must run would seem totally foreign to developers in this environment. Code in this dream becomes pure intent: homogenized, universal and sufficiently abstracted from any implementation constraints to become properly focused on solving problems first and foremost. List comprehension and query comprehension would eventually give way to a fully declarative intent realization model, arguably the goal of any universal computer. I did say this was a dream, didn’t I?

I know it may sound crazy. Some of the things I envision probably won’t be achieved in my lifetime. But think for a moment about the kinds of innovations that would come out of the process if the languages team at Microsoft utterly embraced JavaScript on the server side. Think about the developer experience they would craft in Visual Studio. It would be no less than stunning. Think about the editor and debugger and designer experiences. Imagine how IIS would evolve to support such a unified JavaScript execution model. Ponder for a moment how the JavaScript language itself might evolve if Microsoft put all of their mental and marketplace muscle behind it as both a server and client language of choice. Over time, languages like Python and Ruby would actually suffer from that kind of dedication as developers flocked to JavaScript as the common language on both sides of the average Internet connection. Silverlight, as much as I admire it for what it needs to be in 2011, would simply evaporate if my vision were to become real. And in that sort of scenario, Silverlight’s retirement wouldn’t be seen as a bad thing at all. Rather, it would just be a natural part of the evolution toward a better, more unified web for everyone.

Justin Etheredge turned me onto node.js at CodeMash 2011. It does evented I/O on Google’s V8 JavaScript Engine. Extremely cool stuff for building web services and definitely a step in the right direction, in my opinion. Google is a great company but I wish it were my trusted friend and partner Microsoft taking this step with me. Say what you will about Microsoft but my corporate and government clients trust them. And that’s why I’ve invested so much of my career in making Microsoft technologies successful. If I have to start migrating toward other vendors, I will do so. But I lament that decision in my heart of hearts. I can only hope that some of you who read this will agree with me and echo my sentiments in your own way.

Exploring LINQPad Talk at RDNUG January 2011

Kevin Hazzard — Mon, 17 Jan 2011 16:39:00 GMT

Here are the slides from my Exploring LINQ with LINQPad talk at Richmond .NET User Group on January 6, 2011.

UPDATE 1/26: Here is the code from the presentation.

Embedding IronPython Talk at CodeMash 2011

Kevin Hazzard — Mon, 17 Jan 2011 16:26:00 GMT

Here are the slides and code from my embedding IronPython talk at CodeMash 2011 entitled “Embedding IronPython as Your Application’s Scripting Language”. I’ve given versions of this talk over the past couple of years but this one’s up-to-date with all of the latest information from the Dynamic Language Runtime (DLR) hosting specification.