Ian Picknell - Under the Covers

Execute Procedure for XML Auto

2011-11-23T07:41:00.000Z

Introduction

For several years now, SQL Server has had the ability to convert the results of an arbitrary SELECT statement into XML by appending the FOR XML AUTO clause. For example, whilst:

  SELECT TOP 10 * FROM sys.types

generates a standard rowset:

name	system_type_id	user_type_id	schema_id	principal_id	max_length	precision	scale	collation_name	is_nullable	is_user_defined	is_assembly_type	default_object_id	rule_object_id	is_table_type
image	34	34	4	NULL	16	0	0	NULL	1	0	0	0	0	0
text	35	35	4	NULL	16	0	0	Latin1_General_CI_AS	1	0	0	0	0	0
uniqueidentifier	36	36	4	NULL	16	0	0	NULL	1	0	0	0	0	0
date	40	40	4	NULL	3	10	0	NULL	1	0	0	0	0	0
time	41	41	4	NULL	5	16	7	NULL	1	0	0	0	0	0
datetime2	42	42	4	NULL	8	27	7	NULL	1	0	0	0	0	0
datetimeoffset	43	43	4	NULL	10	34	7	NULL	1	0	0	0	0	0
tinyint	48	48	4	NULL	1	3	0	NULL	1	0	0	0	0	0
smallint	52	52	4	NULL	2	5	0	NULL	1	0	0	0	0	0
int	56	56	4	NULL	4	10	0	NULL	1	0	0	0	0	0

simply appending a FOR XML clause (in this case FOR XML AUTO):

  SELECT TOP 10 * FROM sys.types FOR XML AUTO

generates an XML fragment instead:

XML_F52E2B61-18A1-11d1-B105-00805F49916B
<sys.types name="image" system_type_id="34" user_type_id="34" schema_id="4" max_length="16" precision="0" scale="0" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="text" system_type_id="35" user_type_id="35" schema_id="4" max_length="16" precision="0" scale="0" collation_name="Latin1_General_CI_AS" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="uniqueidentifier" system_type_id="36" user_type_id="36" schema_id="4" max_length="16" precision="0" scale="0" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="date" system_type_id="40" user_type_id="40" schema_id="4" max_length="3" precision="10" scale="0" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="time" system_type_id="41" user_type_id="41" schema_id="4" max_length="5" precision="16" scale="7" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="datetime2" system_type_id="42" user_type_id="42" schema_id="4" max_length="8" precision="27" scale="7" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="datetimeoffset" system_type_id="43" user_type_id="43" schema_id="4" max_length="10" precision="34" scale="7" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="tinyint" system_type_id="48" user_type_id="48" schema_id="4" max_length="1" precision="3" scale="0" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="smallint" system_type_id="52" user_type_id="52" schema_id="4" max_length="2" precision="5"

XML_F52E2B61-18A1-11d1-B105-00805F49916B

<sys.types name="image" system_type_id="34" user_type_id="34" schema_id="4" max_length="16" precision="0" scale="0" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="text" system_type_id="35" user_type_id="35" schema_id="4" max_length="16" precision="0" scale="0" collation_name="Latin1_General_CI_AS" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="uniqueidentifier" system_type_id="36" user_type_id="36" schema_id="4" max_length="16" precision="0" scale="0" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="date" system_type_id="40" user_type_id="40" schema_id="4" max_length="3" precision="10" scale="0" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="time" system_type_id="41" user_type_id="41" schema_id="4" max_length="5" precision="16" scale="7" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="datetime2" system_type_id="42" user_type_id="42" schema_id="4" max_length="8" precision="27" scale="7" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="datetimeoffset" system_type_id="43" user_type_id="43" schema_id="4" max_length="10" precision="34" scale="7" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="tinyint" system_type_id="48" user_type_id="48" schema_id="4" max_length="1" precision="3" scale="0" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0"/><sys.types name="smallint" system_type_id="52" user_type_id="52" schema_id="4" max_length="2" precision="5"

Clicking on the hyperlink shows you the full XML fragment in all its glory:

<sys.types name="image" system_type_id="34" user_type_id="34" schema_id="4" max_length="16" precision="0" scale="0" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0" />
<sys.types name="text" system_type_id="35" user_type_id="35" schema_id="4" max_length="16" precision="0" scale="0" collation_name="Latin1_General_CI_AS" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0" />
<sys.types name="uniqueidentifier" system_type_id="36" user_type_id="36" schema_id="4" max_length="16" precision="0" scale="0" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0" />
<sys.types name="date" system_type_id="40" user_type_id="40" schema_id="4" max_length="3" precision="10" scale="0" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0" />
<sys.types name="time" system_type_id="41" user_type_id="41" schema_id="4" max_length="5" precision="16" scale="7" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0" />
<sys.types name="datetime2" system_type_id="42" user_type_id="42" schema_id="4" max_length="8" precision="27" scale="7" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0" />
<sys.types name="datetimeoffset" system_type_id="43" user_type_id="43" schema_id="4" max_length="10" precision="34" scale="7" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0" />
<sys.types name="tinyint" system_type_id="48" user_type_id="48" schema_id="4" max_length="1" precision="3" scale="0" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0" />
<sys.types name="smallint" system_type_id="52" user_type_id="52" schema_id="4" max_length="2" precision="5" scale="0" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0" />
<sys.types name="int" system_type_id="56" user_type_id="56" schema_id="4" max_length="4" precision="10" scale="0" is_nullable="1" is_user_defined="0" is_assembly_type="0" default_object_id="0" rule_object_id="0" is_table_type="0" />

Note that I've referred to this as an XML fragment as it contains no root element. We can add one if we like by modifying the FOR XML AUTO clause, but I want to keep this as vanilla as possible.

Now comes the tricky part - image the data you want to return as XML isn't available directly from a table or view, but is selected by a stored procedure. How you you select that that data as XML?

Well, one option is simply to update the stored procedure such that it selects XML by adding a FOR XML clause within the procedure itself. But that'll break existing callers of the stored procedure.

You could copy/paste the stored procedure into a new one, and alter the new one to select XML. But that creates a maintenance burden as every time one stored procedure it changed, someone needs to remember to change the other one too to keep them in sync.

You could wrap one stored procedure in another. So your new stored procedures creates a temporary table, or perhaps a table variable, and does a INSERT INTO ... EXEC to populate the table with the results of the actual stored procedure, then selects the results from the temporary table with a FOR XML clause. For a lot of people, this is the option that'll work best. But it relies on the 'outer' stored procedure knowing the exact structure of the rowset selected by the 'inner' stored procedure, which may change at some point in the future. It's a solution which only works for the one stored procedure it's been coded for.

Isn't there a more flexible, generic solution? Yes there is...

Managed Stored Procedures

Since SQL Server 2005 we've been able to create stored procedures in managed code (i.e. in .NET) rather than just in Transact-SQL. By creating the 'outer' stored procedure mentioned above in managed code, we can have it operate in an entirely generic manner. This means we can call any existing stored procedure, without modifying it, and have the results returned to us as XML. Here's how:

1. In Visual Studio (I'm using 2010, but 2008 or even 2005 should be fine) create a new Class Library project (I called mine SqlServerClr) and re-name the default Class1.cs to StoredProcedures.cs.

2. Use Project Properties to ensure the Target Framework is .NET Framework 2.0.

3. Replace the contents of the StoredProcedures class with:

using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using Microsoft.SqlServer.Server;

namespace SqlServerClr
{
  /// <summary>
  /// Servers as a container for Common Language Runtime (CLR) stored procedures.
  /// </summary>
  public static partial class StoredProcedures
  {
    /// <summary>
    /// Mimics the effect of a FOR XML AUTO clause having been applied to a stored procedure.
    /// </summary>
    /// <param name="procedureName">The name of the stored procedure to execute.</param>
    /// <param name="parameters">Any parameters which should be passed to the procedure.</param>
    /// <remarks>
    /// <para>
    /// The statement executed by this procedure is "EXECUTE [<i>procedureName</i>] 
    /// <i>parameters</i>"; the text in <i>parameters</i> should be formatted accordingly.
    /// For example, "'07 Jun 1969'" or "1234, 'Spa%'".
    /// </para>
    /// <para>
    /// This code is vulnerable to a SQL injection attack and should not be used in situations where
    /// either <i>procedureName</i> or <i>parameters</i> are supplied by end-users without thorough
    /// input validation.
    /// </para>
    /// </remarks>
    public static void ExecuteProcedureForXmlAuto(string procedureName, string parameters)
    {
      // validate our inputs
      if (string.IsNullOrEmpty(procedureName))
      {
        throw new ArgumentNullException("procedureName");
      }
      if (procedureName.IndexOfAny(new[] {'[',']','"'}) != -1)
      {
        throw new ArgumentException("The name of the stored procedure must not be delimited.", "procedureName");
      }

      // establish a connection to the server within which stored procedure is running
      using (SqlConnection sqlConnection = new SqlConnection("context connection=true"))
      {
        sqlConnection.Open();
        // create a memory stream into which we can write our XML
        using (System.IO.MemoryStream memoryStream = new System.IO.MemoryStream())
        {
          // we'll need an XmlTextWriter with which to write into the memory stream
          System.Xml.XmlTextWriter xmlTextWriter = new System.Xml.XmlTextWriter(memoryStream, System.Text.Encoding.Unicode);

          // create the command we're going to execute
          string commandText = string.Format("EXECUTE [{0}] {1}", procedureName, parameters);
          using (SqlCommand sqlCommand = new SqlCommand(commandText, sqlConnection))
          {
            // execute the command, obtaining a SqlDataReader with which to read the results
            using (SqlDataReader sqlDataReader = sqlCommand.ExecuteReader())
            {
              // load the contents of the SqlDataReader, as XML, into a MemoryStream
              while (sqlDataReader.Read())
              {
                xmlTextWriter.WriteStartElement(procedureName);
                for (int ordinal = 0; ordinal < sqlDataReader.FieldCount; ordinal++)
                {
                  // omit null values from the output
                  if (sqlDataReader.IsDBNull(ordinal) == false)
                  {
                    string attributeValue;
                    switch (sqlDataReader.GetDataTypeName(ordinal))
                    {
                      case "datetime":
                        // we can't use "s", the ISO 8601 format, as it omits the three-digit
                        // seconds fraction which FOR XML AUTO uses
                        attributeValue = sqlDataReader.GetDateTime(ordinal).ToString("yyyy-MM-ddTHH:mm:ss.fff", System.Globalization.CultureInfo.CurrentCulture);
                        break;
                      case "float":
                        // use a exponential format, with 15 digits after the decimal point
                        attributeValue = sqlDataReader.GetDouble(ordinal).ToString("e15", System.Globalization.CultureInfo.CurrentCulture);
                        break;
                      case "bit":
                        attributeValue = sqlDataReader.GetBoolean(ordinal) ? "1" : "0";
                        break;
                      default:
                        attributeValue = sqlDataReader.GetValue(ordinal).ToString().Trim();
                        break;
                    }
                    xmlTextWriter.WriteAttributeString(sqlDataReader.GetName(ordinal), attributeValue);
                  }
                }
                xmlTextWriter.WriteEndElement();
              }
              xmlTextWriter.Flush();
              memoryStream.Position = 0L;
            }
          }

          // send a row of meta-data back to SQL Server
          string outputColumnName = "XML_" + System.Guid.NewGuid().ToString().ToUpper(System.Globalization.CultureInfo.CurrentCulture);
          SqlDataRecord sqlDataRecord = new SqlDataRecord(new SqlMetaData(outputColumnName, SqlDbType.Xml));
          SqlContext.Pipe.SendResultsStart(sqlDataRecord);

          // now send the XML itself in a single row
          SqlXml sqlXml = new SqlXml(memoryStream);
          sqlDataRecord.SetSqlXml(0, sqlXml);
          SqlContext.Pipe.SendResultsRow(sqlDataRecord);

          // end the sending of results back to SQL Server
          SqlContext.Pipe.SendResultsEnd();
        }
      }
    }
  }
}

4. Build the project.

5. Fire up SQL Server Management Studio (or whatever Microsoft have re-named it to by the time you read this).

6. Open a new Query window, and select the database in which you wish the CLR stored procedure to be created.

7. Enable the CLR for your instance of SQL Server as follows (it doesn't matter if it's already enabled):

EXEC sp_configure 'clr enabled', '1'
GO
RECONFIGURE
GO

8. Load the assembly into SQL Server:

CREATE ASSEMBLY SqlServerClrAssembly
FROM 'C:\Users\Ian.Picknell\Documents\Blog\SQL Server\Execute Procedure For XML Auto\SqlServerClr\bin\Debug\SqlServerClr.dll'
WITH PERMISSION_SET = SAFE
GO

9. Provide SQL Server with details of the stored procedure within the assembly (specifically its name and parameters).

CREATE PROCEDURE dbo.ExecuteProcedureForXmlAuto
(
  @ProcedureName NVARCHAR(128),
  @Parameters    NVARCHAR(4000) = NULL
)
AS EXTERNAL NAME SqlServerClrAssembly.[SqlServerClr.StoredProcedures].ExecuteProcedureForXmlAuto;
GO

Note that the format of the 'AS' clause in this context is AS EXTERNAL NAME assembly_name.class_name.method_name. The class_name component must be a valid SQL Server identifier - so you'll almost always have to use delimters here.

10. Call your target stored procedure 'through' the ExecuteProcedureForXmlAuto wrapper. For example:

EXEC dbo.ExecuteProcedureForXmlAuto 'sp_configure'
GO

The above statement produced the following:

XML_880017D4-254D-4287-9B9E-496D10CA0C82
<sp_configure name="allow updates" minimum="0" maximum="1" config_value="0" run_value="0" /><sp_configure name="backup compression default" minimum="0" maximum="1" config_value="0" run_value="0" /><sp_configure name="clr enabled" minimum="0" maximum="1" config_value="1" run_value="1" /><sp_configure name="cross db ownership chaining" minimum="0" maximum="1" config_value="0" run_value="0" /><sp_configure name="default language" minimum="0" maximum="9999" config_value="0" run_value="0" /><sp_configure name="filestream access level" minimum="0" maximum="2" config_value="0" run_value="0" /><sp_configure name="max text repl size (B)" minimum="-1" maximum="2147483647" config_value="65536" run_value="65536" /><sp_configure name="nested triggers" minimum="0" maximum="1" config_value="1" run_value="1" /><sp_configure name="remote access" minimum="0" maximum="1" config_value="1" run_value="1" /><sp_configure name="remote admin connections" minimum="0" maximum="1" config_value="0" run_value="0" /><sp_configure name="remote login timeout (s)" minimum="0" maximum="2147483647" config_value="20" run_value="20" /><sp_configure name="remote proc trans" minimum="0" maximum="1" config_value="0" run_value="0" /><sp_configure name="remote query timeout (s)" minimum="0" maximum="2147483647" config_value="600" run_value="600" /><sp_configure name="server trigger recursion" minimum="0" maximum="1" config_value="1" run_value="1" /><sp_configure name="show advanced options" minimum="0" maximum="1" config_value="0" run_value="0" /><sp_configure name="user options" minimum="0" maximum="32767" config_value="0" run_value="0" />

XML_880017D4-254D-4287-9B9E-496D10CA0C82

<sp_configure name="allow updates" minimum="0" maximum="1" config_value="0" run_value="0" /><sp_configure name="backup compression default" minimum="0" maximum="1" config_value="0" run_value="0" /><sp_configure name="clr enabled" minimum="0" maximum="1" config_value="1" run_value="1" /><sp_configure name="cross db ownership chaining" minimum="0" maximum="1" config_value="0" run_value="0" /><sp_configure name="default language" minimum="0" maximum="9999" config_value="0" run_value="0" /><sp_configure name="filestream access level" minimum="0" maximum="2" config_value="0" run_value="0" /><sp_configure name="max text repl size (B)" minimum="-1" maximum="2147483647" config_value="65536" run_value="65536" /><sp_configure name="nested triggers" minimum="0" maximum="1" config_value="1" run_value="1" /><sp_configure name="remote access" minimum="0" maximum="1" config_value="1" run_value="1" /><sp_configure name="remote admin connections" minimum="0" maximum="1" config_value="0" run_value="0" /><sp_configure name="remote login timeout (s)" minimum="0" maximum="2147483647" config_value="20" run_value="20" /><sp_configure name="remote proc trans" minimum="0" maximum="1" config_value="0" run_value="0" /><sp_configure name="remote query timeout (s)" minimum="0" maximum="2147483647" config_value="600" run_value="600" /><sp_configure name="server trigger recursion" minimum="0" maximum="1" config_value="1" run_value="1" /><sp_configure name="show advanced options" minimum="0" maximum="1" config_value="0" run_value="0" /><sp_configure name="user options" minimum="0" maximum="32767" config_value="0" run_value="0" />

Clicking the hyperlink then gives you XML which is (to my eye) identical to that produced by a genuine FOR XML AUTO clause:

<sp_configure name="allow updates" minimum="0" maximum="1" config_value="0" run_value="0" />
<sp_configure name="backup compression default" minimum="0" maximum="1" config_value="0" run_value="0" />
<sp_configure name="clr enabled" minimum="0" maximum="1" config_value="1" run_value="1" />
<sp_configure name="cross db ownership chaining" minimum="0" maximum="1" config_value="0" run_value="0" />
<sp_configure name="default language" minimum="0" maximum="9999" config_value="0" run_value="0" />
<sp_configure name="filestream access level" minimum="0" maximum="2" config_value="0" run_value="0" />
<sp_configure name="max text repl size (B)" minimum="-1" maximum="2147483647" config_value="65536" run_value="65536" />
<sp_configure name="nested triggers" minimum="0" maximum="1" config_value="1" run_value="1" />
<sp_configure name="remote access" minimum="0" maximum="1" config_value="1" run_value="1" />
<sp_configure name="remote admin connections" minimum="0" maximum="1" config_value="0" run_value="0" />
<sp_configure name="remote login timeout (s)" minimum="0" maximum="2147483647" config_value="20" run_value="20" />
<sp_configure name="remote proc trans" minimum="0" maximum="1" config_value="0" run_value="0" />
<sp_configure name="remote query timeout (s)" minimum="0" maximum="2147483647" config_value="600" run_value="600" />
<sp_configure name="server trigger recursion" minimum="0" maximum="1" config_value="1" run_value="1" />
<sp_configure name="show advanced options" minimum="0" maximum="1" config_value="0" run_value="0" />
<sp_configure name="user options" minimum="0" maximum="32767" config_value="0" run_value="0" />

Conclusion

There probably aren't a huge number of scenarios where you have an existing stored procedure which produces a rowset, but now need to get XML from it instead. I originally wrote a version of the above CLR stored procedure for use with BizTalk Server 2006, which required that data be returned from SQL Server as XML and my brief was that the messaging application I was developing must be capable of sourcing data from any existing stored procedure. ExecuteProcedureForXmlAuto fit the bill perfectly.

Implementing IDispatchMessageInspector

2011-03-29T23:08:00.001+01:00

Introduction

Have you ever needed to change the contract used by WCF service, but were held back by the impact it would have on existing clients? Sure, you could just ask your clients to update their service references and tweak the code which calls your service, but that requires that all your clients (of which there might be many) to do a code release on the same day you release your new server - not ideal.

One way to resolve this would be to leave the existing contract alone, but to create a second modified contract exposed via a new endpoint. That way clients could upgrade to the new contract (and the new endpoint) at their leisure. This will probably create some redundancy in the server-side code, as you'd have very similar code at both endpoints, although you could probably refactor any common code such that it was shared. There's also a risk that this approach would lead to end-point proliferation over time as you make various tweaks to the contract. So, is there an an alternative?

One approach which I used recently, and which is the subject of this post, is to intercept the calls which use the old contract and transform them into a call to the new contract on-the-fly. WCF has an extensibility interface designed just for this purpose - IDispatchMessageInspector. Note that, despite its name, implementing this interface allows us to both inspect and modify inbound and outbound messages.

To demonstrate this approach I'm going to:

create a WCF Service exposing a particular interface,
create a client which consumes this service,
modify the service in a manner which is not backwards compatible,
show that the client can no longer consume the service,
implement IDispatchMessageInspector in the service to modify calls on the fly, and
show that the client can now consume the service again.

Creating the WCF Service

I'm not going to go through this step-by-step. If you're interested in implementing IDispatchMessageInspector to modify WCF messages on the fly, then I'm sure you'll have created a WCF service or two in your time. So I'll just be providing source code listing of the various files:

IEmployeeService (the service contract)
EmployeeService (an implementation of that contract)
Employee (a data contract exposed via the service)
Server (contains the console application's entry point and service host provider)

IEmployeeService

  using System.ServiceModel;

  namespace WcfService
  {
    [ServiceContract]
    public interface IEmployeeService
    {
      [OperationContract]
      void SaveEmployee(Employee employee);
    }
  }

EmployeeService

  using System;

  namespace WcfService
  {
    public class EmployeeService : IEmployeeService
    {
      public void SaveEmployee(Employee employee)
      {
        Console.WriteLine("Saved {0}", employee);
      }
    }
  }

Employee

  using System;
  using System.Runtime.Serialization;

  namespace WcfService
  {
    [DataContract]
    public class Employee
    {
      [DataMember]
      public string Name { get; set; }

      [DataMember]
      public DateTime DateOfBirth { get; set; }

      [DataMember]
      public char Gender { get; set; }

      public override string ToString()
      {
        return string.Format("{0} (DOB:{1} Gender:{2})", Name, DateOfBirth.ToShortDateString(), Gender);
      }
    }
  }

Server

  using System;
  using System.ServiceModel;
  using System.ServiceModel.Description;

  namespace WcfService
  {
    class Server
    {
      static void Main()
      {
        // create a host and add an end-point to it
        ServiceHost serviceHost = new ServiceHost(typeof(EmployeeService));
        ContractDescription contractDescription = ContractDescription.GetContract(typeof(IEmployeeService), typeof(EmployeeService));
        NetTcpBinding netTcpBinding = new NetTcpBinding();
        EndpointAddress endpointAddress = new EndpointAddress("net.tcp://localhost:8001/EmployeeService");
        ServiceEndpoint serviceEndpoint = new ServiceEndpoint(contractDescription, netTcpBinding, endpointAddress);
        serviceHost.Description.Endpoints.Add(serviceEndpoint);

        // add a meta-data end-point
        ServiceMetadataBehavior serviceMetadataBehavior = new ServiceMetadataBehavior();
        serviceMetadataBehavior.HttpGetEnabled = true;
        serviceMetadataBehavior.HttpGetUrl = new Uri("http://localhost:8002/EmployeeService/mex");
        serviceHost.Description.Behaviors.Add(serviceMetadataBehavior);

        // open the host
        serviceHost.Open();
        Console.WriteLine("Press ENTER to terminate service.");

        // close the host once the user hits ENTER
        Console.ReadLine();
        serviceHost.Close();
      }
    }
  }

Okay, so that the server-side code sorted. As you can see it's all fairly basic stuff: the service merely accepts an Employee into the SaveEmployee method of its IEmployeeService and writes details of that Employee to the console.

Creating the WCF Client

The client is just as trivial. It consists of a single Client class (in addition to all the auto-generated Service Reference stuff and App.config) which creates an instance of the proxy Employee class and asks the service to save it.

Client

  using System;
  using WcfClient.EmployeeServiceProxy;

  namespace WcfClient
  {
    class Client
    {
      static void Main()
      {
        EmployeeServiceClient employeeServiceClient = new EmployeeServiceClient();
        employeeServiceClient.Open();
  
        Employee employee = new Employee
        {
          Name = "Jane Armstrong",
          DateOfBirth = new DateTime(1969, 7, 12),
          Gender = 'F'
        };

        try
        {
          Console.WriteLine("Saving {0} (DOB:{1} Gender:{2})", employee.Name, employee.DateOfBirth.ToShortDateString(), employee.Gender);
          employeeServiceClient.SaveEmployee(employee);
          employeeServiceClient.Close();
        }
        catch (Exception ex)
        {
          Console.WriteLine(ex);
          employeeServiceClient.Abort();
        }

        Console.WriteLine("Press ENTER to terminate client.");
        Console.ReadLine();
      }
    }
  }

I generated the EmployeeServiceProxy class and App.config simply by using the Add Service Reference dialog, pointing it at the meta-data end-point of the service, http://localhost:8002/EmployeeService/mex.

Testing the Client and Service

Okay, now we're ready to show how this client and service interact. They're both console applications, so you can just start them both up from the command-line. Let's start the service first:

  > WcfService\bin\debug\WcfService.exe
  Press ENTER to terminate service.

and then the client:

  > WcfClient\bin\Debug\WcfClient.exe
  Saving Jane Armstrong (DOB:12/07/1969 Gender:F)
  Press ENTER to terminate client.

If you glance back at the service, you'll notice that it now displays:

Saved Jane Armstrong (DOB:12/07/1969 Gender:F)

So, our extremely simple WCF client/service application works fine. Now let's break it.

Changing the Contract

In an ideal world, WCF contracts would be immutable. But life's not like that. Requirements change. In my scenario I'm going to imagine that the client/service above underwent a code review and someone said: "You're using a char for Gender? Hmmm. That allows a client to send a Gender of 'X' or some other invalid value. Perhaps an enumeration might be better?" The problem is, you've already deployed you client and have dozens of people using it. You really don't want to have to re-deploy it.

Before we look at how implementing IDispatchMessageInspector can solve this issue, let's change the contract (on the service-side only) and see what happens when a client calls it.

We're going to need a Gender enumeration:

  namespace WcfService
  {
    public enum Gender
    {
      Male,
      Female
    }
  }

And we're going to need to change Employee to use this enumeration:

  using System;
  using System.Runtime.Serialization;

  namespace WcfService
  {
    [DataContract]
    public class Employee
    {
      ...

      [DataMember]
      public Gender Gender { get; set; }
      
      ...
    }
  }

Once we've re-compiled the service, we can fire it up:

  > WcfService\bin\Debug\WcfService.exe
  Press ENTER to terminate service.

and watch what happens when the client (which is still sending a char Gender) tries to save an employee.

  > WcfClient\bin\Debug\WcfClient.exe
  Saving Jane Armstrong (DOB:12/07/1969 Gender:F)
  System.ServiceModel.FaultException: The formatter threw an exception while tryin
  g to deserialize the message: There was an error while trying to deserialize par
  ameter http://tempuri.org/:employee. The InnerException message was 'Invalid enu
  m value '70' cannot be deserialized into type 'WcfService.Gender'. Ensure that t
  he necessary enum values are present and are marked with EnumMemberAttribute att
  ribute if the type has DataContractAttribute attribute.'.  Please see InnerExcep
  tion for more details.

Exactly as expected, the client can't pass a char as the service is expecting a member of the Gender enumeration. (In case you're wondering, the enum value 70 in the message above comes from the ASCII value for the letter 'F'.)

Enter IDispatchMessageInspector

WCF allows us to both inspect and modify messages after they've been received by the service but, crucially, before they've been forwarded to an operation. This is achieved by providing a class which implements IDispatchMessageInspector and adding it as a service behaviour. Adding it as a service behaviour can be achieved in code or via a configuration file. As my service is code-only so far, I'm going to stick with that approach.

A skeleton IDispatchMessageInspector implementation would look like this:

  using System;
  using System.IO;
  using System.ServiceModel;
  using System.ServiceModel.Channels;
  using System.ServiceModel.Dispatcher;
  using System.Xml;
  using System.Xml.XPath;
  using System.Xml.Xsl;

  namespace WcfService
  {
    public class MessageInspector : IDispatchMessageInspector
    {
      public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext)
      {
        // inspect and/or modify the request

        // return any object which will be passed to BeforeSendReply to correlate requests and replies
        return null;
      }
  
      public void BeforeSendReply(ref  Message reply, object correlationState)
      {
        // inspect and/or modify the reply
      }
    }
  }

(In case you're wondering about all those redundant using statements - we'll need them later when we provide some real implementations.)

The object you return in AfterReceivedRequest for a given request will be passed to BeforeSendReply when processing the matching reply. I've used this in the past to optionally transform the response - i.e. AfterReceiveRequest inspects the request and establishes whether it came from a legacy client (in our example, that would be one which is passing a char Gender) and returns true/false so BeforeSendReply knows whether it needs to transform the reply such that it can be processed by a legacy client (in our example, the reply hasn't changed, so there's no need to do anything in BeforeSendReply).

You'll notice that the request parameter passed to AfterReceiveRequest, and the reply parameter passed to BeforeSendRepy, are passed by reference. This is because the body of a Message object is immutable - it can't be modified. If you want to modify the message (as we do) you need to create a new Message object with the message you want to use instead of the original and assign that to the request (or reply) parameter. One other wrinkle is that the body of a Message object can only be read once. So if you read it in AfterReceiveRequest (just to log its contents to a file, for example) then it can't be read again by the rest of the WCF stack. Fun, eh? We'll ignore this issue for the moment (don't worry, we'll get back to it) and plough on. If you're just doing a straight transformation this isn't a problem anyway: you read the Message body (once) and create a new Message containing the body you want to be passed to the receiving operation.

A Message object essentially represents a SOAP message and, like a SOAP message, contains both headers and a body. In order for us to write some code which transforms the message, we're going to need to see what the message looks like as it arrives from the client. So we need an IDispatchMessageInspector implementation to inspect the message. We can simply use the template above, and change the AfterReceiveRequest method to read:

   public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext)
   {
     // write the request to the console (only works for small messages)
     Console.WriteLine(request.ToString());

     // we don't need to pass a correlation state
     return null;
   }

This code breaks one of the rules I mentioned above whereby a message can only be read once - it seems that ToString is allowed to ready the message, up to a point. Although I've not seen it documented anywhere I believe that there must be a small internal buffer which is used by the ToString method such that it can output a representation of the string without consuming its contents. For a larger message, the output produced by ToString would contain elipses (...) to indicate the message body.

To cause our MessageInspector to be invoked, we need to create a new class which implememnts IEndpointBehavior and add our MessageInspector to the list of IMessageInspectors when its ApplyDispatchBehavior method is called:

  using System.ServiceModel.Channels;
  using System.ServiceModel.Description;
  using System.ServiceModel.Dispatcher;

  namespace WcfService
  {
    public class EndpointBehaviour : IEndpointBehavior
    {
      public void AddBindingParameters(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters)
      {
        // do nothing
      }

      public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
      {
        // do nothing
      }

      public void ApplyDispatchBehavior(ServiceEndpoint endpoint, EndpointDispatcher endpointDispatcher)
      {
        endpointDispatcher.DispatchRuntime.MessageInspectors.Add(new MessageInspector());
      }

      public void Validate(ServiceEndpoint endpoint)
      {
        // do nothing
      }
    }
  }

To cause our EndpointBehaviour to be used, we need to add it to the list of end-point behaviours by adding the following line to Server.cs, immediately after the serviceEndpoint has been initialised:

  serviceEndpoint.Behaviors.Add(new EndpointBehaviour());

When we start the WcfService and WcfClient this time, the WcfClient will still fail (we haven't corrected the actual problem yet) but at least we'll see in the WcfService what the SOAP message sent by the WcfClient was:

Press ENTER to terminate service.
<s:Envelope xmlns:a="http://www.w3.org/2005/08/addressing" xmlns:s="http://www.w
3.org/2003/05/soap-envelope">
  <s:Header>
    <a:Action s:mustUnderstand="1">http://tempuri.org/IEmployeeService/SaveEmplo
yee</a:Action>
    <a:MessageID>urn:uuid:796c06e4-03b9-4012-899c-935199d5e81e</a:MessageID>
    <a:ReplyTo>
      <a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address>
    </a:ReplyTo>
    <a:To s:mustUnderstand="1">net.tcp://localhost:8001/EmployeeService</a:To>
  </s:Header>
  <s:Body>
    <SaveEmployee xmlns="http://tempuri.org/">
      <employee xmlns:b="http://schemas.datacontract.org/2004/07/WcfService" xml
ns:i="http://www.w3.org/2001/XMLSchema-instance">
        <b:DateOfBirth>1969-07-12T00:00:00</b:DateOfBirth>
        <b:Gender>70</b:Gender>
        <b:Name>Jane Armstrong</b:Name>
      </employee>
    </SaveEmployee>
  </s:Body>
</s:Envelope>

The change we want to make to this Message is trivial: we simply need to change <b:Gender>70</b:Gender> to <b:Gender>Female</b:Gender>. Now, we could use XSLT to do this (which is what I'd typically do if the transformation required were non-trivial), but let's just use an XmlDocument for the modification.

So, we'll modify of the AfterReceiveRequest so it reads:

  public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext)
  {
    // copy the contents of the original message's body into a MemoryStream
    MemoryStream memoryStream = new MemoryStream();
    XmlDictionaryWriter xmlDictionaryWriter = XmlDictionaryWriter.CreateTextWriter(memoryStream);
    request.WriteBodyContents(xmlDictionaryWriter);
    xmlDictionaryWriter.Flush();
    
    // write the contents of the MemoryStream to the Console
    memoryStream.Position = 0L;
    Console.WriteLine(new StreamReader(memoryStream).ReadToEnd());
    
    // load the stream into an XmlDocument
    memoryStream.Position = 0L;
    XmlDocument xmlDocument = new XmlDocument();
    xmlDocument.Load(memoryStream);
    
    // register the namespaces we'll need to xpath to the <gender> node
    XmlNamespaceManager xmlNamespaceManager = new XmlNamespaceManager(xmlDocument.NameTable);
    xmlNamespaceManager.AddNamespace("t", "http://tempuri.org/");
    xmlNamespaceManager.AddNamespace("b", "http://schemas.datacontract.org/2004/07/WcfService");
    
    // modify the <gender> node
    XmlNode genderNode = xmlDocument.SelectSingleNode("/t:SaveEmployee/t:employee/b:Gender", xmlNamespaceManager);
    if (genderNode != null)
    {
      genderNode.InnerText = genderNode.InnerText == "70" ? "Female" : "Male";
    }
    
    // copy the modified XmlDocument back into the MemoryStream
    memoryStream.Position = 0L;
    xmlDocument.Save(memoryStream);
    
    // write the contents of the MemoryStream to the Console
    memoryStream.Position = 0L;
    Console.WriteLine(new StreamReader(memoryStream).ReadToEnd());
    
    // create a new Message from the MemoryStream (copying the original's properties and headers)
    memoryStream.Position = 0L;
    XmlReader xmlReader = XmlReader.Create(memoryStream);
    Message modifiedRequest = Message.CreateMessage(request.Headers.MessageVersion, request.Headers.Action, xmlReader);
    modifiedRequest.Properties.CopyProperties(request.Properties);
    modifiedRequest.Headers.CopyHeadersFrom(request);
    
    // re-assign the request parameter to our new modified request
    request = modifiedRequest;
    
    // return
    return null;
  }

If you're like me you'll hate this code. Let's count the number of copies of the message are required: there's obviously the original message (one), then we copy it into a MemoryStream (two), then we copy that into an XmlDocument (three); once we've modified it, we copy it back into a MemoryStream (four) and finally into a new Message (five). Really? We need to have five copies of the message? In reality there we only four as we re-used the MemoryStream. Also, the final Message we create is passed XmlReader which wraps the MemoryStream - the Message doesn't have a separate copy of this data, it simply reads it from the MemoryStream via the XmlReader when necessarily, not at construction time. So there are actually only three copies the message in memory at any one time. But that's still two too many but my count. In this example the message is clearly tiny, but imagine if it weren't. Those extra copies of the message could really hurt scalability.

Before we look at alternative ways to code this, let's just see if it actually works first. So we'll fire up WcfService:

Press ENTER to terminate service.

And then WcfClient:

Saving Jane Armstrong (DOB:12/07/1969 Gender:F)
Press ENTER to terminate client.

Note that the client didn't experience an exception this time. Glancing back at the WcfService we now see:

Press ENTER to terminate service.
<SaveEmployee xmlns="http://tempuri.org/"><employee xmlns:b="http://schemas.data
contract.org/2004/07/WcfService" xmlns:i="http://www.w3.org/2001/XMLSchema-insta
nce"><b:DateOfBirth>1969-07-12T00:00:00</b:DateOfBirth><b:Gender>70</b:Gender><b
:Name>Jane Armstrong</b:Name></employee></SaveEmployee>
<SaveEmployee xmlns="http://tempuri.org/">
  <employee xmlns:b="http://schemas.datacontract.org/2004/07/WcfService" xmlns:i
="http://www.w3.org/2001/XMLSchema-instance">
    <b:DateOfBirth>1969-07-12T00:00:00</b:DateOfBirth>
    <b:Gender>Female</b:Gender>
    <b:Name>Jane Armstrong</b:Name>
  </employee>
</SaveEmployee>
Saved Jane Armstrong (DOB:12/07/1969 Gender:Female)

(Yours won't be in colour - I've colour-coded the output to explain what you're seeing). The orange XML is the original message received by WcfService. You'll note that Gender is 70 (the ASCII value for 'F'). The green XML is once we've translate it; after translation the Gender is Female. You'll also note that the Saved message shows that the deserialised Employee object contains the correct value for Gender.

Alternate Implementations

So, are there any better solutions? Well, you might try using the XPathNavigator returned by the MessageBuffer's CreateNavigator() method:

   public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext)
   {
    // copy the original request into a buffer which we can read and modify
    MessageBuffer messageBuffer = request.CreateBufferedCopy(Int32.MaxValue);

    // do the necessary prep work to navigate over the request
    XPathNavigator xpathNagivator = messageBuffer.CreateNavigator();
    XmlNamespaceManager xmlNamespaceManager = new XmlNamespaceManager(xpathNagivator.NameTable);
    xmlNamespaceManager.AddNamespace("s", "http://www.w3.org/2003/05/soap-envelope");
    xmlNamespaceManager.AddNamespace("t", "http://tempuri.org/");
    xmlNamespaceManager.AddNamespace("b", "http://schemas.datacontract.org/2004/07/WcfService");
     
    // modify the <gender> node
    XPathNavigator genderNode = xpathNagivator.SelectSingleNode("/s:Envelope/s:Body/t:SaveEmployee/t:employee/b:Gender", xmlNamespaceManager);
    if (genderNode != null)
    {
      genderNode.SetValue(genderNode.Value == "70" ? "Female" : "Male");
    }

    // re-assign the request parameter to a new Message built from our modified buffer
    request = messageBuffer.CreateMessage();

    // tidy up and return
    messageBuffer.Close();
    return null;
  }

Unfortunately, the XPathNavigator created is read-only (its CanEdit property is set to false) so the call to its SetValue method throws a NotSupportedException.

What about an XSLT-based approach? How would that work? Let's start by creating the XSLT itself. We can use the orange XML above as our sample input, and the green XML above as out expected output. The XSLT is pretty simple, as you'd expect.

   <?xml version="1.0" encoding="utf-8"?>
   <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:t="http://tempuri.org/" xmlns:b="http://schemas.datacontract.org/2004/07/WcfService">
     <xsl:output method="xml" indent="yes"/>

     <!-- transform the Gender node -->
     <xsl:template match="t:SaveEmployee/t:employee/b:Gender/text()">
       <xsl:choose>
         <xsl:when test=".='70'">Female</xsl:when>
         <xsl:otherwise>Male</xsl:otherwise>
       </xsl:choose>
     </xsl:template>

     <!-- simply copy all other nodes as-is -->
     <xsl:template match="@* | node()">
       <xsl:copy>
         <xsl:apply-templates select="@* | node()"/>
       </xsl:copy>
     </xsl:template>
   </xsl:stylesheet>

We need to add this file (I called mine TransformRequest.xslt) to the WcfService project and set its Build Action to Embedded Resource. We won't want to load a compile this XSLT every time a request comes in, so we'll place that logic in the MessageInspector's constructor:

  private XslCompiledTransform _compiledTransform;

  public MessageInspector()
  {
    string transformResourceName = string.Format("{0}.{1}", GetType().Namespace, TransformFileName);
    using (Stream stream = GetType().Assembly.GetManifestResourceStream(transformResourceName))
    {
      if (stream == null)
      {
        throw new Exception(string.Format("Unable to load the resource named '{0}' - is it embedded within the the '{1}' assembly?", transformResourceName, GetType().Assembly.FullName));
      }
      _compiledTransform = new XslCompiledTransform();
      _compiledTransform.Load(XmlReader.Create(stream));
    }
  }

Once we have the XSLT in memory, using it within a revised AfterReceiveRequest becomes quite simple.

  public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext)
  {
    // transform the original request, via the XSLT, into a new MemoryStream
    MemoryStream memoryStream = new MemoryStream();
    _compiledTransform.Transform(request.GetReaderAtBodyContents(), null, memoryStream);

    // create a new Message from the MemoryStream (copying the original's properties and headers)
    memoryStream.Position = 0L;
    XmlReader xmlReader = XmlReader.Create(memoryStream);
    Message modifiedRequest = Message.CreateMessage(request.Headers.MessageVersion, request.Headers.Action, xmlReader);
    modifiedRequest.Properties.CopyProperties(request.Properties);
    modifiedRequest.Headers.CopyHeadersFrom(request);

    // re-assign the request parameter to our new modified request and return
    request = modifiedRequest;
    return null;
  }

You note that there are now only two copies of the request message - the original message and the revised messages stored in the MemoryStream. As was noted before, although the Message object we create is passed an XmlReader which wraps the MemoryStream - the Message doesn't have a separate copy of the data, it simply reads it from the MemoryStream via the XmlReader when necessarily.

Although using XSLT might be overkill for this simple requirement, I actually prefer it. It nicely separates the act of performing the transformation (the XSLT) from the plumbing necessary to hook it up to WCF (the AfterReceiveRequest method).

Conclusion

IDispatchMessageInspector isn't something you'd design into an appliction at the outset - you should really aim to get your WCF contract fixed before you start allowing clients to connect. But as a get-out-of-jail-free card when you've no choice but to change the interface, it's invaluable.

Detecting Month-Ends in JavaScript

2010-11-23T08:00:00.000Z

Background

I came across the following problem many years ago whilst doing some work for a financial services company. They had some code (which I don't think was written by me, but my memory is a little hazy on the subject) which attempted to establish whether the current date was the last date in a month. The code was written in JavaScript.

The normal way of performing such a test is to:

(a) establish what month the current date falls within,
(b) add a single day to the current date,
(c) establish what month the date resulting from (b) falls within, and
(d) compare values (a) and (c) - if they are different then the current date is indeed the last date in a month.

Steps (a) and (c) are easy - JavaScripts's Date object provides a getMonth() method which returns the zero-based month number (i.e. 0=January, 1=February, etc).

Step (b) isn't so easy as JavaScript's Data object does not provide any date arithmetic operations per se. That it, it does not provide an addDays() method or any equivalent thereof.

Well, the value actually stored internally by the Data object is the number of milliseconds between the date/time being represented and midnight (GMT) on 1st January 1970. And we can both read this value via the getTime() method, and write this value via the single-parametered constructor respectively. That is, I can obtain the internally stored value of a Date object called now via ms = now.getTime(); and I can create a Date object from this value via then = new Date(ms);. As we know how many milliseconds there are in a day (1000*60*60*24), we can add a day's worth of milliseconds to a given date to add one day to it - which is actually what step (b) requires.

So, putting it all together we get:

  var dateToday = new Date();
  var dateTomorrow = new Date(dateToday.getTime() + (1000 * 60 * 60 * 24));

  if (dateToday.getMonth() == dateTomorrow.getMonth())
    alert("NOT last day of month");
  else
    alert("IS last day of month");

So, has anyone spotted the floor in this logic? What if I told you the logic would fail (in the UK) on the last day of October 2004, 2010 and 2021? Yes - you've guess it - it'd to do with British Summer Time (BST) ending and the UK reverting to Grenwich Mean Time (GMT). At 2:00am on the last Sunday in October the clocks are put back to 1:00am. On such days there are therefore 25 hours (midnight to midnight). If the last Sunday in October happens to be the last day of the month (i.e. in 2004, 2010 and 2021) then the above logic will claim that it's not the last day of the month.

Such a bug can clearly lie dormant for many, many years before it's encountered.

The Solution

The solution is easy. The code shouldn't have used getMonth() to retrieve the current month, as that retrieves the month according to local time. To ignore the effect of the clocks changing we want to use getUTCMonth() instead. Problem solved. Or is it?

Remember that the Date object stores a GMT-relative value. So when it retrieves the date from the current system clock (as in date = new Date()) or has its value set via constructor arguments (as in date = new Date(2010, 9, 31)) it will always convert that to GMT before storing it. So if the current date/time happened to be midnight (00:00) on 31 October 2010 - which is still within British Summer Time - what's actually stored is 23:00 on 30 October 2010 - because that is the same date/time expressed in terms of GMT. So when we add 24 hours to this date we get 23:00 on 31 October 2010. As the month still has't changed, even a version of the above code which uses getUTCMonth() will report that 31 October 2010 is not the last day of the month as long as the time component is less that 01:00.

So a getUTCMonth-version of the above code will work, except for the one hour between 00:00 and 01:00 on the 31st October if that date is a Sunday. So for the fifteen years between 1st January 2005 and 31st December 2020 there's only a single hour during which it will fail. That asmuses me. It's the sort of really rare bug that would almost certainly go unnoticed during testing but could well cause problems (depending upon the end-of-month processing the test was intended to trigger).

An Actual Solution

There are probably lots of solutions involving the use of GMT (rather than local) time throughout. But the simplest approach is to realise that we really don't care about the time, but it's the time which is getting us into trouble. So let's change it.

Simply adding the following line into the original code, immediately after dateToday is initialised fixes things nicely:

  dateToday.setHours(12, 0, 0, 0);

Although this might seem a little like cheating, it works well. And that's good enough for me.

Selectively Overriding XML Serialisation Attributes

2010-10-28T18:09:00.000+01:00

Introduction

As I mentioned in my last post, although you can override XML serialisation attributes by passing an XmlAttributeOverrides instance to an XmlSerializer, the attributes you provide for a given type and member replace all the existing XML serialisation attributes - you can't simply tweak one or two and leave the rest intact.

If you're thinking that type and members only tend to have one or two serialistion attributes, then take a look at this set attributes from an auto-generated EquityDerivativeBase class:

[System.Xml.Serialization.XmlIncludeAttribute(typeof(EquityDerivativeShortFormBase))]
[System.Xml.Serialization.XmlIncludeAttribute(typeof(EquityOptionTransactionSupplement))]
[System.Xml.Serialization.XmlIncludeAttribute(typeof(BrokerEquityOption))]
[System.Xml.Serialization.XmlIncludeAttribute(typeof(EquityDerivativeLongFormBase))]
[System.Xml.Serialization.XmlIncludeAttribute(typeof(EquityOption))]
[System.Xml.Serialization.XmlIncludeAttribute(typeof(EquityForward))]
[System.CodeDom.Compiler.GeneratedCodeAttribute("xsd", "4.0.30319.1")]
[System.SerializableAttribute()]
[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.ComponentModel.DesignerCategoryAttribute("code")]
[System.Xml.Serialization.XmlTypeAttribute(Namespace="http://www.fpml.org/FpML-5/confirmation")]

Let's suppose I want to use XmlAttributeOverrides to alter the value of the XmlTypeAttribute at run-rime, to place the element in a different namespace. Well I can. But the XmlAttributeOverrides instance I supply is used to replace all the existing attributes. So I lose each of the XmlIncludeAttribute attributes which define the classes which use this class as a base class.

Book and Genre classes (with Xml Attributes)

To demonstrate how to override the attributes selectively I'm going to use the same Book class as in my last post to demonstrate selectively overriding these attributes. I've added a lot more attributes to the members of the Book class to demonstrate that they all get retained.

[XmlType(TypeName="Book")]
[XmlRoot("book", Namespace="http://tempuri.org")]
public class Book
{
  [XmlIgnore]
  public int InternalId { get; set; }

  [XmlElement("title")]
  public string Title { get; set; }

  [DefaultValue("Anonymous")]
  [XmlArray("authors")]
  [XmlArrayItem("author")]
  public string[] Authors { get; set; }

  [XmlElement("isbn13")]
  public string Isbn13 { get; set; }
 
  [XmlText]
  public string Extract { get; set; }

  [XmlAttribute("genre")]
  public Genre Genre { get; set; }

  [XmlNamespaceDeclarations]
  public XmlSerializerNamespaces XmlNamespaces { get; set; }

  [XmlAnyAttribute]
  public XmlAttribute[] OtherAttributes { get; set; }

  [XmlAnyElement]
  public XmlElement[] OtherElements { get; set; }

  public Book()
  {
    XmlNamespaces = new XmlSerializerNamespaces();
    XmlNamespaces.Add("ns", "http://tempuri.org");
  }
}

public enum Genre
{
  [XmlEnum("unknown")]
  Unknown,
  [XmlEnum("autobiography")]
  Autobiography,
  [XmlEnum("computing-text")]
  ComputingText,
  [XmlEnum("classic")]
  Classic
}

Solution

The solution is to copy all the existing attributes into an XmlAttributeOverrides instance (modifying them as they're copied), and then apply the XmlAttributeOverrides to the XmlSerializer. That way the XmlAttributeOverrides object retains all of the original attributes (with the exception of any changes made in transit). Let me show you what I mean:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.IO;
using System.Reflection;
using System.Xml;
using System.Xml.Serialization;

namespace ConsoleApplication
{
  public class Program
  {
    static void Main(string[] args)
    {
      // create some Books
      Book[] books = new Book[]
      {
        new Book { InternalId=1, Title="The Road Ahead", Authors=new string[] {"Bill Gates"}, Isbn13="978-0670859139", Genre=Genre.Autobiography },
        new Book { InternalId=2, Title="Beowulf", Authors=new string[] {"Anonymous"}, Isbn13="978-1588278296", Extract="That grim spirit was called Grendel,\nFamous waste-wanderer that held the moors\nFen and fastness; the land of the race of monsters\nThe unhappy creature occupied for a while\nAfter the Creator had condemned them.", Genre=Genre.Classic },
        new Book { InternalId=3, Title="The C Programming Language (2nd Edition)", Authors=newstring[] {"Brian W Kernighan","Dennis M Ritchie"}, Isbn13="978-0131103627", Genre=Genre.ComputingText },
      };

      // copy the existing attributes into an XmlAttributeOverrides instance (providing an
      // Action which tweaks the attributes for Book.Isbn13)
      XmlAttributeOverrides xmlAttributeOverrides = GetXmlAttributeOverrides(new Type[] { typeof(Book), typeof(Genre) },
        (type, memberName, xmlAttributes) =>
        {
          if (type == typeof(Book) && memberName == "Isbn13")
          {
            // remove the sttribute which specifies an element named "isbn13"
            xmlAttributes.XmlElements.Clear();
            // and add an attribute which specifies an attribute named "isbn"
            xmlAttributes.XmlAttribute = new XmlAttributeAttribute("isbn");
          }
        });

      // serialise the books into a MemoryStream (using the overrides)
      XmlSerializer xmlSerializer = new XmlSerializer(typeof(Book[]), xmlAttributeOverrides);
      MemoryStream memoryStream = new MemoryStream();
      xmlSerializer.Serialize(memoryStream, books);

      // write the contents of the MemoryStream to the Console
      memoryStream.Position = 0L;
      StreamReader streamReader = new StreamReader(memoryStream);
      Console.WriteLine(streamReader.ReadToEnd());

      // wait for user to hit ENTER
      Console.ReadLine();
    }

    private static XmlAttributeOverrides GetXmlAttributeOverrides(Type[] types, Action<Type, string, XmlAttributes> tweakAttributesAction)
    {
      XmlAttributeOverrides xmlAttributeOverrides = new XmlAttributeOverrides();
      XmlAttributes xmlAttributes;

       foreach (Type type in types)
       {
        // get the Type's attributes first
        xmlAttributes = GetXmlAttributes(type.GetCustomAttributes(false));
        xmlAttributeOverrides.Add(type, xmlAttributes);

        // then iterate over the members, checking those attributes too
        if (type.IsEnum)
        {
          foreach (FieldInfo fieldInfo in type.GetFields(BindingFlags.Static | BindingFlags.Public))
          {
            xmlAttributes = GetXmlAttributes(fieldInfo.GetCustomAttributes(false));
            tweakAttributesAction(type, fieldInfo.Name, xmlAttributes);
            xmlAttributeOverrides.Add(type, fieldInfo.Name, xmlAttributes);
          }
        }
        else
        {
          foreach (PropertyInfo propertyInfo in type.GetProperties())
          {
            xmlAttributes = GetXmlAttributes(propertyInfo.GetCustomAttributes(false));
            tweakAttributesAction(type, propertyInfo.Name, xmlAttributes);
            xmlAttributeOverrides.Add(type, propertyInfo.Name, xmlAttributes);
          }
        }
      }

      return xmlAttributeOverrides;
    }

    private static XmlAttributes GetXmlAttributes(object[] attributes)
    {
      XmlAttributes xmlAttributes =new XmlAttributes();

      Dictionary<Type, Action<object>> dictionary = new Dictionary<Type, Action<object>>()
      {
        { typeof(XmlAnyAttributeAttribute), attribute => xmlAttributes.XmlAnyAttribute = attribute as XmlAnyAttributeAttribute },
        { typeof(XmlAnyElementAttribute), attribute => xmlAttributes.XmlAnyElements.Add(attribute as XmlAnyElementAttribute) },
        { typeof(XmlArrayAttribute), attribute => xmlAttributes.XmlArray = attribute as XmlArrayAttribute },
        { typeof(XmlArrayItemAttribute), attribute => xmlAttributes.XmlArrayItems.Add(attribute as XmlArrayItemAttribute) },
        { typeof(XmlAttributeAttribute), attribute => xmlAttributes.XmlAttribute = attribute as XmlAttributeAttribute },
        { typeof(DefaultValueAttribute), attribute => xmlAttributes.XmlDefaultValue = (attribute as DefaultValueAttribute).Value },
        { typeof(XmlElementAttribute), attribute => xmlAttributes.XmlElements.Add(attribute as XmlElementAttribute) },
        { typeof(XmlEnumAttribute), attribute => xmlAttributes.XmlEnum = attribute as XmlEnumAttribute },
        { typeof(XmlIgnoreAttribute), attribute => xmlAttributes.XmlIgnore = true },
        { typeof(XmlNamespaceDeclarationsAttribute), attribute => xmlAttributes.Xmlns = true },
        { typeof(XmlRootAttribute), attribute => xmlAttributes.XmlRoot = attribute as XmlRootAttribute },
        { typeof(XmlTextAttribute), attribute => xmlAttributes.XmlText = attribute as XmlTextAttribute },
        { typeof(XmlTypeAttribute), attribute => xmlAttributes.XmlType = attribute as XmlTypeAttribute },
      };

      foreach (object attribute in attributes)
      {
        // establish what type of attribute we're dealing with
        Type type = attribute.GetType();
        // if the attribute is one of those supported by XmlAttributes
        if (dictionary.ContainsKey(type))
        {
          // lookup the appropriate action and invoke it
          dictionary[type](attribute);
        }
      }

      return xmlAttributes;
    }
}

You'll notice that I use an Action to tweak the existing attributes prior to copying them into the XmlAttributeOverrides object, rather than simpy copying them all into the XmlAttributeOverrides object and then tweaking them there. This is because you can only add attributes to XmlAttributeOverrides - you can't remove them or modify them.

The above code produced the following output. Note that the Isbn13 property has been serialised into an attribute named isbn, contrary to what the attributes applied statically request. Also note that all other attributes have been honoured.

<?xml version="1.0"?>
<ArrayOfBook xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Book xmlns:ns="http://tempuri.org" isbn="978-0670859139" genre="autobiography">
    <ns:title>The Road Ahead</ns:title>
    <ns:authors>
      <ns:author>Bill Gates</ns:author>
    </ns:authors>
  </Book>
  <Book xmlns:ns="http://tempuri.org" isbn="978-1588278296" genre="classic">
    <ns:title>Beowulf</ns:title>
    <ns:authors>
      <ns:author>Anonymous</ns:author>
    </ns:authors>That grim spirit was called Grendel,
Famous waste-wanderer that held the moors
Fen and fastness; the land of the race of monsters
The unhappy creature occupied for a while
After the Creator had condemned them.</Book>
  <Book xmlns:ns="http://tempuri.org" isbn="978-0131103627" genre="computing-text">
    <ns:title>The C Programming Language (2nd Edition)</ns:title>
    <ns:authors>
      <ns:author>Brian W Kernighan</ns:author>
      <ns:author>Dennis M Ritchie</ns:author>
    </ns:authors>
  </Book>
</ArrayOfBook>

Summary

So, to selectively override XML serialisation attributes, simply create an XmlAttributeOverrides which contains all the attributes you want to apply - even if you have to copy most of them from the attributes applied statically.

Default Values Don't Get Serialised

2010-10-17T13:20:00.000+01:00

The Problem

I came across some unusual behaviour during XML Serialisation recently and thought I'd share both the problem and my solution to it.

To demonstrate the behaviour I've created a small console application which serialises two Book objects to the Console. I've marked the default value of the Author property as "Anonymous", which seems reasonable enough. The second of the two Book objects specifically has its Author property set to "Anonymous", which it doesn't really need to have - as that's the default value.

Here's the code:

using System;
using System.ComponentModel;
using System.IO;
using System.Xml.Serialization;

namespace ConsoleApplication
{
  public class Program
  {
    static void Main(string[] args)
    {
      // create some Books
      Book[] books = new Book[]
      {
        new Book { Title = "The Road Ahead", Author = "Bill Gates", Isbn13 = "978-0670859139" },
        new Book { Title = "Beowulf", Author = "Anonymous", Isbn13 = "978-1588278296" },
      };

      // serialise it into a MemoryStream
      XmlSerializer xmlSerializer = new XmlSerializer(typeof(Book[]));
      MemoryStream memoryStream = new MemoryStream();
      xmlSerializer.Serialize(memoryStream, books);
    
      // write the contents of the MemoryStream to the Console
      memoryStream.Position = 0L;
      StreamReader streamReader = new StreamReader(memoryStream);
      Console.WriteLine(streamReader.ReadToEnd());

      // wait for user to hit ENTER
      Console.ReadLine();
    }
  }

  public class Book
  {
    [XmlElement("title")]
    public string Title { get; set; }

    [XmlElement("author")]
    [DefaultValue("Anonymous")]
    public string Author { get; set; }
    
    [XmlElement("isbn13")]
    public string Isbn13 { get; set; }
  }
}

When I run the above, something odd happens. The Author property is not serialised for the second book, as you'll see below. It turns out that this is by design. The theory is that if the default value has been used, then there's no need to explicitly output it - a consumer of the data will simply see that the element is missing and infer the default value.

<?xml version="1.0"?>
<ArrayOfBook xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Book>
    <title>The Road Ahead</title>
    <author>Bill Gates</author>
    <isbn13>978-0670859139</isbn13>
  </Book>
  <Book>
    <title>Beowulf</title>
    <isbn13>978-1588278296</isbn13>
  </Book>
</ArrayOfBook>

The problem, obviously, is that this approach assumes that consumer is aware that a default value exists. Can you see any indication from the serialised XML that a default value exists, or that an element has been suppressed because it's value was equal to the default value? No - that information is hidden from the consumer. We're simply left to hope that the consumer is aware of this default value.

The Solution

Well, the obvious solution is to simply remove the Default attribute from the Author property. But what if you can't? It's perfectly possible that you're serialising an object to which you do not have the source code. In my case I was serialising an object where the source code had been auto-generated from an series of XSDs - and was 43,600 lines long, and contained 72 default constraints. If I can avoid editing auto-generated code I always will. So how?

Well, although I'd never used it before, there's an overload to the XmlSerializer which allows you to provide a series of overrides for the serialisation attributes. Unfortunately, it overrides all of the serialisation attributes for a given type and member - you can't just say "use this Default attribute, but leave all other attributes alone". This means we need to re-state the XmlElement attribute to specify a name of "author" (otherwise you'll get the default of "Author"). The call to the XmlSerializer's constructor thus becomes:

      XmlAttributes bookAuthorXmlAttributes = new XmlAttributes();
      bookAuthorXmlAttributes.XmlElements.Add(new XmlElementAttribute("author"));
      XmlAttributeOverrides xmlAttributeOverrides = new XmlAttributeOverrides();
      xmlAttributeOverrides.Add(typeof(Book), "Author",  bookAuthorXmlAttributes);
      XmlSerializer xmlSerializer = new XmlSerializer(typeof(Book[]), xmlAttributeOverrides);

The output this time is:

<?xml version="1.0"?>
<ArrayOfBook xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Book>
    <title>The Road Ahead</title>
    <author>Bill Gates</author>
    <isbn13>978-0670859139</isbn13>
  </Book>
  <Book>
    <title>Beowulf</title>
    <author>Anonymous</author>
    <isbn13>978-1588278296</isbn13>
  </Book>
</ArrayOfBook>

Perfect. But it's a shame we've had to re-state the XmlElement attribute isn't it? It would be better if we could inspect the attributes which were already present and add all of these in the XmlAttributeOverrides, with the exception of the Default attribute we're trying to remove. That'll be the subject of my next post.

Validating Integers in SQL Server

2010-04-10T18:57:00.000+01:00

Introduction

Hypothetical scenario: Data has been loaded into a SQL Server table from a CSV file; each column in the table has been defined as VARCHAR(255) to prevent the data load job from failing if one or more values are not of the expected data type. You're job is to copy data from this 'raw' table into one where the columns are strongly typed (i.e. INT, DATETIME, DECIMAL, etc). The requirement is to pull through those rows which are entirely valid, but to ignore those rows where any value in the VARCHAR column cannot be successfully converted to the target data type. Actually, if rows which are only partially valid could be copied across too, with invalid values being set to NULL in the target table, then that would be even better.

The requirement seems simple enough, but how would you do this? It's actually quite tricky to get right. So I'm going to make the scenario even simpler - you only have to support INT data types. Your job is simply to make sure that the supplied VARCHAR can be converted to an INT before you actual attempt to do so.

ISNUMERIC

You first port-of-call might be to use ISNUMERIC to establish whether a source value is numeric. This sounds reasonable enough.

SELECT ISNUMERIC('9876543210')

The above statement returns 1 - the value '9876543210' is numeric. But is it an INT? No - it's too large. The documentation for ISNUMERIC says that it returns 1 if the supplied character column can be converted to at least one of the numeric data types. So that's not particularly useful if we're trying to establish whether a value is a valid INT or not.

ARITHABORT

So you decide simply to convert the value, ignoring arithmetic overflows using SET ARITHABORT OFF. Let's try:

SET ARITHABORT OFF
SELECT CONVERT(INT, '9876543210')

Msg 248, Level 16, State 1, Line 2
The conversion of the varchar value '9876543210' overflowed an int column. Maximum integer value exceeded.

Then you remember that setting ARITHABORT OFF has no effect unless you also set ANSI_WARNINGS OFF. So you try again:

SET ARITHABORT OFF
SET ANSI_WARNINGS OFF
SELECT CONVERT(INT, '9876543210')

Perfect - this selects the value NULL, which can be easily tested for.

User-Defined Function

So we wrap this test into a user-defined function so it can be easily re-used.

CREATE FUNCTION dbo.IsINT
(
  @Input VARCHAR(255)
)
RETURNS BIT
AS
BEGIN
  SET ARITHABORT OFF
  SET ANSI_WARNINGS OFF
  RETURN CASE WHEN CONVERT(INT, @Input) IS NULL THEN 0 ELSE 1 END
END
GO

Msg 443, Level 16, State 15, Procedure IsINT, Line 8
Invalid use of side-effecting or time-dependent operator in 'SET OPTION OFF' within a function.
Msg 443, Level 16, State 15, Procedure IsINT, Line 9
Invalid use of side-effecting or time-dependent operator in 'SET OPTION OFF' within a function.

We can't use SET option OFF within a function? That's a shame. Okay, so we'll have to require that the caller does this. But the least we should do is check that they have.

CREATE FUNCTION dbo.IsINT
(
  @Input VARCHAR(255)
)
RETURNS BIT
AS
BEGIN
  IF SESSIONPROPERTY('ARITHABORT') = 1
    RAISERROR (N'This function requires that ARITHABORT be set to OFF before it is called.', 16, 1)

  IF SESSIONPROPERTY('ANSI_WARNINGS') = 1
    RAISERROR (N'This function requires that ANSI_WARNINGS be set to OFF before it is called.', 16, 1)

  RETURN CASE WHEN CONVERT(INT, @Input) IS NULL THEN 0 ELSE 1 END
END

Msg 443, Level 16, State 14, Procedure IsINT, Line 9
Invalid use of side-effecting or time-dependent operator in 'RAISERROR' within a function.
Msg 443, Level 16, State 14, Procedure IsINT, Line 12
Invalid use of side-effecting or time-dependent operator in 'RAISERROR' within a function.

Nope. Can't do that either. We'll just have to be content with returning NULL if the options haven't been set correctly and hoping the caller has read the documentation for our function.

CREATE FUNCTION dbo.IsINT
(
  @Input VARCHAR(255)
)
RETURNS BIT
AS
-- Note to callers: ARITHABORT and ANSI_WARNINGS must be set to OFF
-- before calling; if not, this function will always return NULL.
BEGIN
  IF SESSIONPROPERTY('ARITHABORT') = 1
    RETURN NULL

  IF SESSIONPROPERTY('ANSI_WARNINGS') = 1
    RETURN NULL

  RETURN CASE WHEN CONVERT(INT, @Input) IS NULL THEN 0 ELSE 1 END
END

Let's try our function with something which is clearly a valid integer:

SELECT dbo.IsINT('12345')

Ooops - forgot to set ARITHABORT and ANSI_WARNINGS off so it returned NULL. Good job it warned me. Let's try again.

SET ARITHABORT OFF
SET ANSI_WARNINGS OFF
SELECT dbo.IsINT('12345')

That worked (it returned 1). Let's throw a few more valid integers at it:

SELECT dbo.IsINT('2147483647')
SELECT dbo.IsINT('0')
SELECT dbo.IsINT('-0')
SELECT dbo.IsINT('-') -- treated as 0
SELECT dbo.IsINT('-2147483648')

Yup - they all returned 1. How about a couple which are out-of-range:

SELECT dbo.IsINT('2147483648')
SELECT dbo.IsINT('-2147483649')

Both these returned 0. Finally, let's check to see what happens when we pass a non-integer number or a non-numeric value.

SELECT dbo.IsINT('1.23')

Msg 245, Level 16, State 1, Line 1
Conversion failed when converting the varchar value '1.23' to data type int.

SELECT dbo.IsINT('N/A')

Msg 245, Level 16, State 1, Line 1
Conversion failed when converting the varchar value 'N/A' to data type int.

Not so good. It seems that ARITHABORT can only be used to suppress errors of an arithmetic nature, not all conversion errors.

Numeric, but not Integer

We can cater for non-numerics by using ISNUMERIC within our function:

  RETURN CASE WHEN ISNUMERIC(@Input) = 0 OR CONVERT(INT, @Input) IS NULL THEN 0 ELSE 1 END

But how are we going to cope with '1.23' without raising an error? I guess we could convert the input to a FLOAT first (on the assumption that all genuine numerics will convert to a FLOAT) and then check to see if there are any digits to the right of the decimal place. Or we could take a look for a '.' within the input string. We might need to be a little careful as I'm sure there are some locales which use a character other than '.' as the decimal symbol, although I'm not sure what support SQL Server has for these. A quick check in Control Panel shows that the decimal symbol is indeed configurable within Windows itself, with the default in France being ','. I can't see any support for this within SQL Server though so perhaps that's a red herring.

So let's go with the simple option of looking for a '.' within the input string; if we find one we know the input cannot be an integer. (I'm ignoring the question of whether or not "1.0" should be considered an integer for the moment.)

Our RETURN statement thus becomes:

  RETURN CASE WHEN CHARINDEX('.', @Input) != 0 OR ISNUMERIC(@Input) = 0 OR CONVERT(INT, @Input) IS NULL THEN 0 ELSE 1 END

Surely we must have tested for everything now: we only even attempt the conversion if the input is devoid of a '.' and is considered to be numeric, and then we ignore the result of any arithmetic overflow. Job done.

Exponential Notation

Hang on a moment. What about integers expressed in exponential form? What happens if we try our IsINT function against '1E5'. It doesn't contain a '.', is numeric, and shouldn't cause an overflow. Let's see:

SELECT dbo.IsINT('1E5')

Msg 245, Level 16, State 1, Line 1
Conversion failed when converting the varchar value '1E5' to data type int.

Now 1E5 is certainly an integer. It's just 100000 expressed in exponential form. But the requirement was to "to make sure that the supplied VARCHAR can be converted to an INT". So the fact that 1E5 is an integer is irrelevant - we only care about whether or not it can be converted to an INT. It cannot. In fact, this little 'Get Out of Jail Free' card allows us to ignore the issue of whether or not "1.0" is an integer or not too - CONVERT chokes on it, and that's all that counts.

So, it's looks like we're going to have to handle this as a special case. Using the same approach we used when checking for a '.' seems the right thing to do. Actually, seeing as we'll have to check for '.', 'E' and 'e' we might as well use LIKE.

Solution

The final solution we end up with is therefore:

CREATE FUNCTION dbo.IsINT
(
  @Input VARCHAR(255)
)
RETURNS BIT
AS
-- Note to callers: ARITHABORT and ANSI_WARNINGS must be set to OFF
-- before calling; if not this function will always return NULL.
BEGIN
  IF SESSIONPROPERTY('ARITHABORT') = 1
    RETURN NULL

  IF SESSIONPROPERTY('ANSI_WARNINGS') = 1
    RETURN NULL

  RETURN CASE WHEN @Input LIKE '%[.Ee]%' OR ISNUMERIC(@Input) = 0 OR CONVERT(INT, @Input) IS NULL THEN 0 ELSE 1 END
END

That's quite a lot of effort just to establish whether a VARCHAR can convert into an INT. Imagine how much more effort it would be if we needed to support DATETIME types too, with all their variations, or DECIMALs with precision and scale to account for. And even now, I don't know for sure that IsINT won't thrown an exception one day when someone passes something to it which I haven't allowed for.

So, the next time someone suggests you just load data as VARCHARs and do the conversion to a string type within SQL Server, I hope you'll at least think twice before saying what a good idea that is.

Post Script

In case any of you are thinking that the easy option would simply be to wrap the attempted conversion in a TRY CATCH block, you're out of luck. SQL Server doesn't support the use of exception handling within a user-defined function. And clearly you don't want to attempt to convert an entire set of data within a TRY CATCH block as a single conversion failure will drop you into the CATCH block, whilst we want good data to survive. The only way I can see to use the TRY CATCH approach is to CURSOR over the set of data being converted. Okay for small sets of data perhaps, but not for anything larger.

Strong Name Storage

2010-03-31T21:43:00.002+01:00

Introduction

In Adding a Strong Name to a Third-Party Assembly I described how you can add a strong name to a third-party assembly by disassembling that assembly into IL using ILDASM and then re-assemble it with ILASM, specifying a strong name key.

I never really liked that approach. It seemed like using a sledge-hammer to crack a nut: just because we want to add a few bytes (i.e. the strong name) into an assembly, we have to break the entire assembly apart and put it back together again?

It always struck me as odd that the Strong Name Tool (SN.exe) provided no support for adding a strong name signature to a assembly, unless one already existed or the assembly had at least been delayed signed at compilation stage. Is adding a few bytes into the assembly to accommodate the strong name signature really that hard?

So for the last couple of weeks I've been studying both:

These documents describe the physical structure of .NET assemblies. Using the information contained therein, I've been able to establish just how a strong name is stored within an assembly. It's not pretty.

Portable Executable / Common Object File Format

A PE/COFF file (which all Windows EXEs/DLLs are) contains a series of headers followed by a series of sections. Each section must being on a boundary defined by the File Alignment field within one of the headers, which indicates that each section begins on either a 512-byte or 4096-byte boundary. We are only really interested in the section named ".text" as it is this section which contains the Strong Name Signature and the CLI Meta Data.

CLR Runtime Header / CLI Header

Within one of the headers exists a Data Directory which references various tables and strings within the image. The 14th entry in the Data Directory (counting from 0) contains the VirtualAddress and Size of the a header referred to variously as the CLR Runtime Header or CLI Header. I'll use the term CLI Header for the sake of consistency.

The VirtualAddress of the CLI Header is the address of the first byte of the header relative to the image base when the section is loaded into memory. As we're dealing with the file itself, not an in-memory image, we need to refer to the virtual address of the containing section and the file offset of the first byte of the section to establish the file offset of the CLI Header.

Strong Name Signature

The strong name itself is referenced by two fields within the CLI Header - StrongNameSignatureRVA and StrongNameSignatureSize. In a weak-named assembly both of these fields are set to 0. In a strong-named assembly they contain the relative virtual address (i.e. the offset relative to the start of the containing section) and size (in bytes) of the strong name signature. We could attempt to derive the strong name ourselves, but there seems little point. All we really need to do is set each byte in the strong name signature to 0x00 as that will represent a delay-signed assembly. We can then have the Strong Name Tool (SN.exe) perform the actual signing.

Given that each section is padded to a 512-byte (or 4096-byte) boundary anyway, it's possible that we can simply point StrongNameSignatureRVA to some location within this 'padding' area and make that the strong name signature. But it's also quite likely that there will be insufficient free space within the section to accommodate the strong name, so the section itself may need to be lengthened. Obviously this will mean the sections which follow it will need to be moved, and all the offsets within the headers will need to be updated to reference their new locations.

Public Key

Although we may leave the Strong Name Signature itself full of 0x00 bytes, we can't get away without setting the Public Key. Establishing where to locate the Strong Name Signature above was fairly easy - the location of the public key is a little more tricky.

First, we have to reference the MetaDataRVA and MetaDataSize fields within the CLI Header to establish the location of the CLI Meta Data within the image. The CLI Meta Data consists of a header plus five streams of data. One of these streams, #Blob (the Blob Heap), contains the 160-byte public key. Unfortunately, entries within the Blob Heap are not named - they are referenced by their offset relative to the start of the stream - so we can't just add a new entry to this stream and be done with it. The entry which refers to the public key within the Blob Heap in within a separate steam, #~, which consists of a series of tables with varying structures.

The simplest .NET application you can think of will contain the following tables: Module, TypeRef, TypeDef, MethodDef, MemberRef, CustomAttribute, Assembly and AssemblyRef. The reference to the public key is stored within the Assembly table. But to even find the Assembly table you need to seek past all the tables preceding it (the list above is in order, so Assembly is one of the last tables).

To establish the location of the Assembly table you need to know which tables precede it (the list above is a minimum set, there may be many other tables in a given assembly). This list of tables is readily available. You also need to know how many rows are in each table. This too, is readily available. Finally, you need to know how wide each row is. Establishing this is not so straight-forward. For example, the Extends column in the TypeDef table contains an index into the TypeDef, TypeRef, or TypeSpec tables. This index will occupy 2 or 4 bytes depending upon the number of rows in those tables. Such an index is referred to in the specification as a coded index "using 2 bytes if the maximum number of rows of tables t₀, ...t_n-1, is less than 2^{(16 – (log n))}, and using 4 bytes otherwise". In this example n is 3 (as the index may reference one of 3 separate tables).

Method Implementations

One other thing we need to handle is the fact that the RVA field in reach row of the MethodDef table contains the relative virtual address of that method's implementation. Once we've inserted a zeroed-out strong name signature and a public key and into the file, these method implementations will now be at different locations so we'll have to update the RVAs too. A similar issue occurs with the RVA column in the FieldRVA table.

Summary

Given all the above, it's no surprise that the Strong Name tool opted not to support this scenario. I don't give up easily, so I'm going to pursue my goal of creating a tool which performs all the magic necessary to add a strong name where the Strong Name Tool cannot. But I suspect I won't achieve this any time soon.

Launching a ClickOnce Application

2010-03-12T07:38:00.001Z

Introduction

If you're anything like the me, you won’t be content with knowing that browsing to http://MyServer/MyVirtualDir/MyApplication.application downloads, installs and launches MyApplication. You'll want to know how this is achieved. This post goes under the covers of ClickOnce to show you how it's done. Well, at least how some of it is done.

As this post makes quite a lot of references to registry entries, I'll use the standard abbreviations HKCU and HKCR to represent the HKEY_CURRENT_USER and HKEY_CLASSES_ROOT hives respectively.

Retrieving the Deployment Manifest

What happens when you browse to http://MyServer/MyVirtualDir/MyApplication.application?

The first thing to realise is that no 'magic' is happening behind the scenes. When you send an HTTP GET request to a web server (Internet Information Services, IIS, for example) the web server will typically react in one of two ways. It will either establish that your request must be forwarded to a component on the server for processing (a 'handler' in IIS terminology) or will establish that the request represents a static file which should simply have its contents returned.

IIS stores details of handlers, and the requests which are forwarded to each handler, in its metabase. Handlers can be defined in terms of a script map (potentially in combination with a managed handler) or a module mapping. For example:

Requests matching the pattern *.asp are forwarded, via a script map, to the ISAPI module %windir%\system32\inetsrv\asp.dll.
Requests matching the pattern *.aspx are forwarded, via a script map, to the ISAPI module %windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll and to the managed handler System.Web.UI.PageHandlerFactory.
Requests matching the pattern *.shtml are forwarded, via a module mapping, to the pre-registered ServerSideIncludeModule module.

You can view the current script maps via IIS Manager or by issuing the following statement at the command line whilst in your C:\Inetpub\AdminScripts folder:

cscript adsutil.vbs get W3SVC/ScriptMaps

There is no static map, managed handler, or module mapping for an *.application request. Such requests are picked up by the StaticFileModule module which matches * requests (i.e. requests not processed elsewhere).

Once IIS has established that the request represents a static file which will simply have its contents returned, it looks up the MIME type which it should place in the Content-Type header of the HTTP response. This, and not the name of the file requested, is to be used by the browser to establish what type of data is present within the body of the HTTP response.

IIS stores a map from file extension to MIME type within its metabase. To see the default MIME map for your IIS instance use IIS Manager or issue the following statement at the command line whilst in your C:\Inetpub\AdminScripts folder:

cscript adsutil.vbs enum /MIMEMAP

The MIME map reports than the .application extension maps to a MIME type of application/x-ms-application. This value is therefore placed in the Content-Type HTTP response header. The HTTP response body simply contains the unprocessed contents of the .application file itself.

Upon receiving the HTTP response, Internet Explorer looks up the file extension which should be assumed, given the MIME type specified in the header HTTP response. If this seems a little odd to you, you are not alone. I believe it stems from the fact that within Windows it is file extensions, not MIME types, which are associated with applications. As the HTTP standard defines that the MIME type should be used to determine the type of data, Windows has no real choice but to map the MIME type onto a file extension, to then map from a file extension to an application.

Internet Explorer looks up application/x-ms-application first in the user-specific MIME database at HKCU\Software\Classes\MIME\Database\Content Type and, if it fails to find an entry there, the machine-specific MIME database at HKCR\MIME\Database\Content Type. It reads the Extension sub-key and establishes that the MIME type application/x-ms-application maps to the .application extension. Internet Explorer then saves the file into its Temporary Internet Files folder (typically located at %USERPROFILE%\AppData\Local\Microsoft\Windows\Temporary Internet Files, %USERPROFILE%\Local Settings\Temporary Internet Files or %SYSTEMROOT%\Temp\Temporary Internet Files). Of course, you won't be able to see the file in Windows Explorer as Internet Explorer does its usual cloak-and-dagger trick of preventing you access to files on your own PC. When I tested this, the received file was placed within a hidden system folder called KEH38PJB which was within a hidden system folder called Content.IE5 (I tested on Internet Explorer 8.0) beneath Temporary Internet Files. The file name will be derived from the requested URL, albeit with the name modified to ensure uniqueness and to apply the .application file extension. For example, suppose you create an ASPX page called Default.aspx and have it return an HTTP response with the MIME type "application/x-ms-application" (as I have done) you'll end up with a file beneath Temporary Internet Files called Default[1].application. This again re-enforces the fact that it's the MIME type which drives this process, not the file name extension as stored on the server.

Opening the Deployment Manifest

Once the deployment manifest has been stored in the Temporary Internet Files folder, Internet Explorer then attempts to establish how it should handle the file with the (assumed and actual) .application extension. It checks the user-specific file types at HKCU\Software\Classes and, if it fails to find an .application sub-key there, checks for machine-specific file types at HKCR. Via this means it establishes that the .application extension denotes an Application.Manifest file. It then uses this information to check the user-specific HKCU\Software\Classes\Application.Manifest and machine-specific HKCR\Application.Manifest keys to establish the CLSID of a library which handles Application.Manifest files and yields the result {98af66e4-aa41-4226-b80f-0b1a8f34eeb4}. Finally, it looks up this CLSID in the user-specific HKCU\Software\Classes\CLSID\{98af66e4-aa41-4226-b80f-0b1a8f34eeb4} and machine-specific HKCR\CLSID\{98af66e4-aa41-4226-b80f-0b1a8f34eeb4} paths to establish that .application files are handled by C:\WINDOWS\system32\DFshim.dll.

This is where things start to get a little complicated. I said earlier that no 'magic' was happening behind the scenes. Well, whilst that was true for the process of retrieving the deployment manifest it most certainly is not true of the process of actually launching the ClickOnce application once the deployment manifest has been retrieved and the handler, DFshim.dll, has been identified.

DFshim.dll is described in the registry as the 'Manifest mime handler' although its file properties describe it as the 'Application Deployment Support Library'. It implements the Internet Explorer MIME handler COM interface. It is a native 32-bit DLL, written in Microsoft Visual C++ 2005, and was installed into C:\Windows\system32 when the .NET Framework 2.0 was installed.

DFshim.dll has a hard-coded reference to DFdll.dll, which it locates by checking the values of HKLM\SOFTWARE\Microsoft\.NETFramework\InstallRoot (typically C:\Windows\Microsoft.NET\Framework\) and the keys beneath HKLM\SOFTWARE\Microsoft\.NETFramework\Policy\AppPatch (typically v2.0.50727, even if a later version of the Framework is installed). DFdll.dll too is a native 32-bit DLL written in Microsoft Visual C++ 2005.

DFdll.dll uses COM services exposed by DFsvc.exe, which is also located in the .NET Framework folder. DFsvc.exe is a standard .NET MSIL assembly. DFsvc contains a single Main method (within the System.Deployment.Application namespace) which simply calls the internal System.Deployment.Application.DFServiceEntryPoint.Initialize method within System.Deployment.dll. The Initialize method registers System.Deployment.Application.DeploymentServiceCom with COM via System.Runtime.InteropServices.RegistrationServices (i.e. it performs the equivalent of CoRegisterClassObject in COM) using the CLSID {33246f92-d56f-4e34-837a-9a49bfc91df3}. This is the means by which its services are made available to DFdll.dll.

The COM service exposed by System.Deployment.Application.DeploymentServiceCom delegates the majority of its methods to other non-ComVisible classes within the System.Deployment.Application namespace. For example, the public ActivateDeployment method calls ActivateDeployment on a new System.Deployment.Application.ApplicationActivator instance, the public CheckForDeploymentUpdate method calls CheckForDeploymentUpdate on System.Deployment.Application.SubscriptionStore, etc.

It is clear that the vast majority of the work surrounding the actual installation and launch of the ClickOnce application is undertaken by classes within the System.Deployment namespace, hosted within DFsvc.exe. DFshim.dll and DFdll.dll appear to primarily be responsible for arbitrating between the COM-based world of Internet Explorer and the .NET-based world of ClickOnce.

Summary

Despite appearances, the Web server has very little involvement in the process of launching a ClickOnce application: it simply serves-up the contents of the deployment manifest along with the appropriate MIME type. The real fun happens on the client.

Serialising Enumerations with WCF

2010-03-02T07:55:00.001Z

Introduction

There are occasions when the exceptions thrown by Windows Communication Foundation (WCF) are clear and unambiguous. There are also occassions where they are not. In this post I'll describe a real-world problem which a colleague encountered recently together which the steps which you might reasonably perform to diagnose it.

The problem manifested itself in a System.ServiceModel.CommunicationException being thrown with nothing particularly useful in the Exception message itself.

System.ServiceModel.CommunicationException: An error occurred while receiving th
e HTTP response to http://localhost:8731/Design_Time_Addresses/WcfServiceLibrary
/Service1/. This could be due to the service endpoint binding not using the HTTP
 protocol. This could also be due to an HTTP request context being aborted by th
e server (possibly due to the service shutting down). See server logs for more d
etails. ---> System.Net.WebException: The underlying connection was closed: An u
nexpected error occurred on a receive. ---> System.IO.IOException: Unable to rea
d data from the transport connection: An existing connection was forcibly closed
 by the remote host. ---> System.Net.Sockets.SocketException: An existing connec
tion was forcibly closed by the remote host

Clearly something very bad happened. But the WCF service itself was still up and running, so one of the theories put forward in the message above "possibly due to the service shutting down" can be quickly dismissed. I'm sure it's obvious, but the above message was generated by my mock-up of the problem rather than the original problem itself.

Mock-up of Original Problem

Let me quickly run through the mock-up of the problem which I put together. It consists of three projects: WcfServiceLibrary contains a single WCF service called Service1, WcfConsoleHost provides a host for Service1 in the form of a Console Application, and WcfClient consumes Service1.

WcfServiceLibrary consists of a Service1 class which implements an IService1 interface. Here's the interface:

using System;
using System.Runtime.Serialization;
using System.ServiceModel;

namespace WcfServiceLibrary
{
    public enum Gender
    {
        Male = 1,
        Female = 2
    }

    [DataContract]
    public class MyCompositeType
    {
        [DataMember]
        public bool MyBool
        {
            get;
            set;
        }

        [DataMember]
        public string MyString
        {
            get;
            set;
        }

        [DataMember]
        public Gender MyEnum
        {
            get;
            set;
        }
    }

    [ServiceContract]
    public interface IService1
    {
        [OperationContract]
        MyCompositeType GetMyCompositeType(bool initialised);
    }
}

And the class:

using System;

namespace WcfServiceLibrary
{
    public class Service1 : IService1
    {
        public MyCompositeType GetMyCompositeType(bool initialised)
        {
            MyCompositeType myCompositeObject = new MyCompositeType();

            if (initialised)
            {
                myCompositeObject.MyBool = true;
                myCompositeObject.MyString = "Hello World";
                myCompositeObject.MyEnum = Gender.Male;
            }

            return myCompositeObject;
        }
    }
}

WcfConsoleHost just contains a Program class which hosts Service1:

using System;
using System.ServiceModel;

namespace WcfConsoleHost
{
    class Program
    {
        static void Main(string[]args)
        {
            try
            {
                ServiceHost serviceHost = new ServiceHost(typeof(WcfServiceLibrary.Service1));
                serviceHost.Open();

                foreach (System.ServiceModel.Description.ServiceEndpoint serviceEndPoint in serviceHost.Description.Endpoints)
                {
                    Console.WriteLine("Listening at {0}...", serviceEndPoint.Address.ToString());
                }
            }
            catch (System.Exception ex)
            {
                System.Console.WriteLine(ex.ToString());
            }

            Console.ReadLine();
        }
    }
}

Finally, we have WcfClient which again just consist of a Program class. Upon execution, WcfClient simply retrieves an initialised instance of MyCompositeType via WCF and writes its details to the console.

using System;

namespace WcfClient
{
    class Program
    {
        static void Main(string[]args)
        {
            try
            {
                DisplayMyCompositeObject(true);
            }
            catch (System.Exception ex)
            {
                Console.WriteLine(ex);
            }

            Console.ReadLine();
        }

        static void DisplayMyCompositeObject(bool initialised)
        {
            Service1Reference.MyCompositeType myCompositeObject = GetMyCompositeType(initialised);

            Console.WriteLine("MyBool={0}", myCompositeObject.MyBool);
            Console.WriteLine("MyString={0}", myCompositeObject.MyString);
            Console.WriteLine("MyString={0}", myCompositeObject.MyEnum);
            Console.WriteLine();
        }

        static Service1Reference.MyCompositeType GetMyCompositeType(bool initialised)
        {
            Service1Reference.Service1Client service1Client = null;
            Service1Reference.MyCompositeType myCompositeObject = null;

            try
            {
                service1Client = new Service1Reference.Service1Client();
                myCompositeObject = service1Client.GetMyCompositeType(initialised);
                service1Client.Close();
            }
            catch
            {
                if(service1Client != null)
                {
                    service1Client.Abort();
                }
                throw;
            }

            return myCompositeObject;
        }
    }
}

For the sake of completeness, I'll also show you the configuration files I'm using. First, the server-side configuration file in WcfConsoleHost:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.web>
    <compilation debug="true" />
  </system.web>
  <system.serviceModel>
    <services>
      <service name="WcfServiceLibrary.Service1" behaviorConfiguration="WcfServiceLibrary.Service1Behavior">
        <host>
          <baseAddresses>
            <add baseAddress="http://localhost:8731/Design_Time_Addresses/WcfServiceLibrary/Service1/"/>
          </baseAddresses>
        </host>
        <endpoint address="" binding="basicHttpBinding" contract="WcfServiceLibrary.IService1">
          <identity>
            <dns value="localhost"/>
          </identity>
        </endpoint>
        <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange"/>
      </service>
    </services>
    <behaviors>
      <serviceBehaviors>
        <behavior name="WcfServiceLibrary.Service1Behavior">
          <serviceMetadata httpGetEnabled="true"/>
          <serviceDebug includeExceptionDetailInFaults="true" />
        </behavior>
      </serviceBehaviors>
    </behaviors>
  </system.serviceModel>
</configuration>

And the client-side configuration file in WcfClient:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.serviceModel>
    <bindings>
      <basicHttpBinding>
        <binding name="BasicHttpBinding_IService1" closeTimeout="00:01:00"
          openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
          allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
          maxBufferSize="65536" maxBufferPoolSize="524288" maxReceivedMessageSize="65536"
          messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"
          useDefaultWebProxy="true">
          <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
            maxBytesPerRead="4096" maxNameTableCharCount="16384" />
          <security mode="None">
            <transport clientCredentialType="None" proxyCredentialType="None" realm="" />
            <message clientCredentialType="UserName" algorithmSuite="Default" />
          </security>
        </binding>
      </basicHttpBinding>
    </bindings>
    <client>
      <endpoint address="http://localhost:8731/Design_Time_Addresses/WcfServiceLibrary/Service1/"
        binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_IService1"
        contract="Service1Reference.IService1" name="BasicHttpBinding_IService1" />
    </client>
  </system.serviceModel>
</configuration>

Testing the Application

With all this in place we can start the WcfConsoleHost:

Listening at http://localhost:8731/Design_Time_Addresses/WcfServiceLibrary/Servi
ce1/...
Listening at http://localhost:8731/Design_Time_Addresses/WcfServiceLibrary/Servi
ce1/mex...

Firing up WcfClient the gives us a nice simple:

MyBool=True
MyString=Hello World
MyString=Male

So what's the problem? Well the problem is that if we pass false to WcfClient.Program.DisplayMyCompositeObject (and hence to WcfClient.Program.GetMyCompositeType and hence to WcfServiceLibrary.Service1.GetMyCompositeType), to ask that an uninitialised instance of MyCompositeObject be returned, we get the following output:

System.ServiceModel.CommunicationException: An error occurred while receiving th
e HTTP response to http://localhost:8731/Design_Time_Addresses/WcfServiceLibrary
/Service1/. This could be due to the service endpoint binding not using the HTTP
 protocol. This could also be due to an HTTP request context being aborted by th
e server (possibly due to the service shutting down). See server logs for more d
etails. ---> System.Net.WebException: The underlying connection was closed: An u
nexpected error occurred on a receive. ---> System.IO.IOException: Unable to rea
d data from the transport connection: An existing connection was forcibly closed
 by the remote host. ---> .Net.Sockets.SocketException: An existing connec
tion was forcibly closed by the remote host
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size,
 SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 s
ize)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 s
ize)
   at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetriev
edStream, Boolean probeRead)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.GetResponse()
   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpCha
nnelRequest.WaitForReply(TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebExc
eption(WebException webException, HttpWebRequest request, HttpAbortReason abortR
eason)
   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpCha
nnelRequest.WaitForReply(TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeS
pan timeout)
   at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message messag
e, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean on
eway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan tim
eout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean on
eway, ProxyOperationRuntime operation, Object[] ins, Object[] outs)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCall
Message methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage req
Msg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgDa
ta, Int32 type)
   at WcfClient.Service1Reference.IService1.GetMyCompositeType(Boolean initialis
ed)
   at WcfClient.Service1Reference.Service1Client.GetMyCompositeType(Boolean init
ialised) in C:\Users\Ian.Picknell\Documents\Blog\WCF\Serialising Enumerations wi
th WCF\WcfClient\Service References\Service1Reference\Reference.cs:line 140
   at WcfClient.Program.GetMyCompositeType(Boolean initialised) in C:\Users\Ian.
Picknell\Documents\Blog\WCF\Serialising Enumerations with WCF\WcfClient\Program.
cs:line 48
   at WcfClient.Program.DisplayMyCompositeObject(Boolean initialised) in C:\User
s\Ian.Picknell\Documents\Blog\WCF\Serialising Enumerations with WCF\WcfClient\Pr
ogram.cs:line 23
   at WcfClient.Program.Main(String[] args) in C:\Users\Ian.Picknell\Documents\B
log\WCF\Serialising Enumerations with WCF\WcfClient\Program.cs:line 11

I've highlighted what I believe to be the first exception thrown. One thing you may have noticed in the server-side configuration file earlier in the post was that I'm asking for exception details to be included within faults, which is generally a good idea during development. The inner-most exception (according to the client) was .Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host.

Diagnosing the Problem

Lets enable logging on both client and server to obtain more details. We update both configuration files to add the following within the <system.serviceModel> element:

<diagnostics>
  <messageLogging logEntireMessage="true" logMalformedMessages="true" logMessagesAtServiceLevel="true" logMessagesAtTransportLevel="true" />
</diagnostics>

We then direct the messages to an appropriate listener by adding a <system.diagnostics> element directly within the <configuration> element.

<system.diagnostics>
  <sources>
    <source name="System.ServiceModel" switchValue="Information, ActivityTracing" propagateActivity="true" >
      <listeners>
        <add name="xml"/>
      </listeners>
    </source>
    <source name="System.ServiceModel.MessageLogging">
      <listeners>
        <add name="xml"/>
      </listeners>
    </source>
  </sources>
  <sharedListeners>
    <add initializeData="WcfClient.svclog" type="System.Diagnostics.XmlWriterTraceListener" name="xml"/>
  </sharedListeners>
</system.diagnostics>

The only difference between the new entries in the client-side and server-side configuration files is the name of the file to which diagnostic messages will be written: WcfClient.svclog and WcfConsoleHost.svclog respectively.

Now we can both re-start WcfConsoleHost and WcfClient and re-produce the error. Having done this we need to check the logs for any additional information. So we fire-up Microsoft Service Trace Viewer, within the Microsoft Windows SDK and open the client-side log file, WcfClient.svclog.

In the Activity tab on the far left we immediately notice that one of the activities is highlighted in red. This is a 'Process action' activity for 'http://tempuri.org/IService1/GetMyCompositeType'. Clicking on the activity populates the panel in the top-right, which has a 'Throwing an exception' message highlighted in red. Clicking this text populates the panel in the bottom-right with details of the exception. Studying the exception doesn't really provide any additional information - it simply re-iterates that An existing connection was forcibly closed by the remote host. Checking the messages and activities immediately prior to the failure provides no additional insight.

Let's check the server-side log file. After all, if the client is saying that the server terminated the connection then hopefully the server is aware that it did this and knows why. So we open WcfConsoleHost.svclog with Microsoft Service Trace Viewer.

We again notice in the Activity tab on the far left that one of the activities is highlighted in red. This is a 'Process action' activity for 'http://tempuri.org/IService1/GetMyCompositeType'. Clicking on the activity populates the panel in the top-right, which has a 'Throwing an exception' message highlighted in red. This all sounds worrying familiar so far. Clicking on this text populates the panel in the bottom-right with details of the exception. But the exception is different to that experience by the client. The exception message is:

There was an error while trying to serialize parameter http://tempuri.org/:GetMyCompositeTypeResult. The InnerException message was 'Enum value '0' is invalid for type 'WcfServiceLibrary.Gender' and cannot be serialized. Ensure that the necessary enum values are present and are marked with EnumMemberAttribute attribute if the type has DataContractAttribute attribute.'.  Please see InnerException for more details.

Of course. Silly me. The Gender enumeration has values 1 and 2 (for Male and Female respectively) but as it's uninitialised it'll have the value 0 which simply isn't valid. It's a shame that WCF doesn't propagate the above exception to the client, especially as I've set includeExceptionDetailInFaults to true in my server-side configuration file. But at least the message is plain and simple once we switch on server-side logging.

Moral of the story? Always ensure that your enumerations support 0, even if you map that to something generic like Unknown or Undefined. You'll still have to handle these values at some point, but at least WCF won't thrown exceptions when attempting to pass them across the wire.

Faking a Strong Name

2010-02-23T07:50:00.000Z

Introduction

Once upon a time a strong name was just that - strong. If you'd referenced an assembly with a strong name within your application you could be sure that that's the assembly you were going to get a run-time. Well, even that was never true. An assembly re-direct could cause you to be delivered a different version but it would always have to have the same name and public key token, and hence must have originated from the same source. They key thing is that no-one could slip your application a fake assembly and pass it off as the real thing.

Well, it turns out that now they can. You simply place your fake assembly in the GAC folder within which the real version of the assembly would logically reside. The name of the folder is used to establish its version and public key token. Don't believe me? Read on...

Scenario

The scenario is based upon a situation I encountered a few years ago. ThirdPartySecurity contains a class called DirectoryServices which is essentially a wrapper around System.DirectoryServices - it allows the caller to easily performs some look-ups against Active Directory ('who are the members of this group', that kind of thing). ConsoleApplication has a reference to ThirdPartySecurity and simply acts as a test harness. MockDirectoryServices is exactly that - a mocked-up implementation of System.DirectoryServices. It is not referenced by anything.

He's the source code for the third-party DirectoryServices in its entirety. This is the actual source code of the assembly which was performing this job on a project I worked on. The code is not mine. I did not write it. I've left the completely alone (despite the urge to perform some major re-factoring), with the exception that I changed the namespace to protect the innocent and have changed the catch block in GetUsersInGroup such that it re-throws an exception rather than logging and swallowing it.

using System;
using System.DirectoryServices;

namespace ThirdPartySecurity
{
    /// <summary>
    /// Summary description for ActiveDirectory.
    /// </summary>
    public class DirectoryServices
    {
        public DirectoryServices()
        {
        }

        #region public methods

        /// <summary>
        /// Takes an Active Directory group name and returns an array of user names that
        /// belong to that single group.
        /// </summary>
        /// <param name="groupName"></param>
        /// <returns></returns>
        /// <remarks>If there are no groups return a null array.  If there are no users in the group, return an empty array.</remarks>
        public string[] GetUsersInGroup (string groupName)
        {
            //If no Users in the group, return an empty array.
            string [] result = new string[0];

            try
            {
                DirectorySearcher ds = new DirectorySearcher();
                ds.SearchRoot = new DirectoryEntry();   // start searching from local domain

                if (groupName.IndexOf("\\") > 0)
                    groupName = groupName.Substring(groupName.IndexOf("\\")+1); //Take out the Domain 

                ds.Filter = BuildLDAPFilter("group",groupName);

                SearchResultCollection src = ds.FindAll();
                DirectoryEntry objGroupEntry;

                int i = 0;
                if (src.Count == 1) //Must match to a single group
                {
                    foreach(SearchResult objResult in src) //for the 1st and only group
                    {
                        objGroupEntry = objResult.GetDirectoryEntry();
                        string[] members = new string[objGroupEntry.Properties["member"].Count];
                        foreach(object objMember in objGroupEntry.Properties["member"])
                        {
                            string memberDetails = objMember.ToString();
                            //strip out CN= from start of string and get name up to first comma delimiter
                            members[i] = (memberDetails.Substring(3,memberDetails.IndexOf(",")-3));
                            i++;
                        }
                        result = members;
                    }
                }
                else //If no groups return a Null Array
                {
                    result = null;
                }

                src.Dispose();
                ds.Dispose();
            }
            catch (System.Exception ex)
            {
                // NOTE:the original version of this catch block logged the exception via the
                //      Enterprise Library and then swallowed it; we'll simply re-throw it
                //Framework.EnterpriseLibrary.Log(String.Format("GetUsersInGroup(\"{0}\")failed (Active Directory is probably unavailable); the following exception hasbeen silently ignored.\n\n{1}", groupName, ex),Framework.Severity.Warning);
                throw;
            }

            return result;
        }


        /// <summary>
        /// Takes a user name in an Active Directory store and returns the display name for that user.
        /// </summary>
        /// <param name="userName"></param>
        /// <returns></returns>
        public string GetDisplayName (string userName)
        {
            return GetPropertyFromUser(userName, "displayname");
        }


        /// <summary>
        /// /// Takes a user name in an Active Directory store and returns the email address for that user.
        /// </summary>
        /// <param name="userName"></param>
        /// <returns></returns>
        public string GetEmailAddress (string userName)
        {
            return GetPropertyFromUser(userName, "mail");
        }

        #endregion

        #region private methods

        /// <summary>
        /// Forms a filter string for the search in LDAP Format
        /// </summary>
        /// <param name="objectCategory"></param>
        /// <param name="filter"></param>
        /// <returns></returns>
        private string BuildLDAPFilter(string objectCategory, string filter)
        {
            String result;
            result = String.Format("(&(objectCategory={0})(name={1}))", objectCategory, filter);
            return result;
        }


        /// <summary>
        /// /// Takes a user name in an Active Directory store and returns a property specified by the caller for that user.
        /// </summary>
        /// <param name="userName">The user to search for the given <b>propertyName</b></param>
        /// <param name="propertyName">The property of the <b>userName</b> to be returned.</param>
        /// <returns></returns>
        /// <remarks>If the user is not found returns null, if the property does not exist return empty string</remarks>
        private string GetPropertyFromUser(string userName, string propertyName)
        {
            string result = null;

            DirectorySearcher ds = new DirectorySearcher();
            ds.SearchRoot = new DirectoryEntry();   // start searching from local domain
            ds.Filter = BuildLDAPFilter("user",userName);
            SearchResultCollection src = ds.FindAll();

            //TODO: Reject the search if more than 1 user returned
            if (src.Count == 1)
            {
                result = string.Empty; //If the property does not exist return an empty string
                foreach (SearchResult srcResult in src)
                    result = srcResult.Properties[propertyName][0].ToString();
            }

            return result;
        }


        #endregion
    }
}

Problem

The problem I had on this project was that I didn't have access to Active Directory when I was developing off-line on my laptop. Of course, the DirectoryServices class above could be modified in many ways to work around this: the most obvious way would be to have it implement an IDirectoryServices interface which could also be implemented by a new class called MockDirectoryServices, with some entry in the configuration file defining which implementation would be used at runtime. But that would rely upon having access to the source code, and I'm trying to show you a technique which will work even if you only have access to the assembly. (Hence why I called this assembly ThirdPartySecurity - to help us imagine that the source code is unavailable).

I can demonstrate the problem via the simple ConsoleApplication which makes use of ThirdPartySecurity to enumerate the members of a specific group.

using System;

namespace ConsoleApplication
{
    class Program
    {
        static void Main(string[]args)
        {
            try
            {
                (new Program()).Run();
            }
            catch (System.Exception ex)
            {
                Console.WriteLine(ex);
            }

            // ask our AppDomain what DirectoryServices-related assemblies are loaded
            Console.WriteLine("\nDirectoryServices-related assemblies loaded into the current AppDomain are:");
            foreach (System.Reflection.Assembly assembly in AppDomain.CurrentDomain.GetAssemblies())
            {
                if (assembly.GetName().Name.Contains("DirectoryServices"))
                {
                    Console.WriteLine("\t{0} (from {1})", assembly.FullName,assembly.CodeBase);
                }
            }

            Console.WriteLine("\nPress ENTER to exit");
            Console.ReadLine();
        }

        void Run()
        {
            // create an instance of the third-party DirectoryServices wrapper
            ThirdPartySecurity.DirectoryServices directoryServices = new ThirdPartySecurity.DirectoryServices();

            // define which group we'll be seeking members of
            string groupName = "MYDOMAIN\\MyGroup";
            Console.WriteLine("The members of the group \"{0}\"are:", groupName);

            // ask the third-party DirectoryServices wrapper for the members of the specified group
            string[] memberNames = directoryServices.GetUsersInGroup(groupName);

            // output the results
            foreach (string memberName in memberNames)
            {
                Console.WriteLine("\t{0}", memberName);
            }
        }
    }
}

Obviously ConsoleApplication has a reference to ThirdPartySecurity and ThirdPartySecurity has a reference to System.DirectoryServices. The reason I included code to display assemblies within the AppDomain containing the text 'DirectoryServices' in their name will become clear later on.

If I run this code on my laptop, which is currently disconnected and has no access to Active Directory, I get the following output. I've highlighted key items within the output.

The members of the group "MYDOMAIN\MyGroup"are:
System.Runtime.InteropServices.COMException (0x8007054B): The specified domain e
ither does not exist or could not be contacted.

   at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)
   at System.DirectoryServices.DirectoryEntry.Bind()
   at System.DirectoryServices.DirectoryEntry.get_AdsObject()
   at System.DirectoryServices.PropertyValueCollection.PopulateList()
   at System.DirectoryServices.PropertyValueCollection..ctor(DirectoryEntry entr
y, String propertyName)
   at System.DirectoryServices.PropertyCollection.get_Item(String propertyName)
   at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)
   at System.DirectoryServices.DirectoryEntry.Bind()
   at System.DirectoryServices.DirectoryEntry.get_AdsObject()
   at System.DirectoryServices.DirectorySearcher.FindAll(Boolean findMoreThanOne
)
   at System.DirectoryServices.DirectorySearcher.FindAll()
   at ThirdPartySecurity.DirectoryServices.GetUsersInGroup(String groupName) in
C:\Users\Ian.Picknell\Documents\Blog\Strong Names\Faking a Strong Name\ThirdPart
ySecurity\DirectoryServices.cs:line 72
   at ConsoleApplication.Program.Run() in C:\Users\Ian.Picknell\Documents\Blog\S
trong Names\Faking a Strong Name\ConsoleApplication\Program.cs:line 42
   at ConsoleApplication.Program.Main(String[] args) in C:\Users\Ian.Picknell\Do
cuments\Blog\Strong Names\Faking a Strong Name\ConsoleApplication\Program.cs:lin
e 11

DirectoryServices-related assemblies loaded into the current AppDomain are:
        System.DirectoryServices, Version=2.0.0.0, Culture=neutral, PublicKeyTok
en=b03f5f7f11d50a3a (from file:///C:/Windows/assembly/GAC_MSIL/System.DirectoryS
ervices/2.0.0.0__b03f5f7f11d50a3a/System.DirectoryServices.dll)

Press ENTER to exit

Solution

The solution is to use a mock implementation of System.DirectoryServices and then persuade ThirdPartySecurity to use that instead - despite the fact that it holds a reference to the strong-named real System.DirectoryServices. I highlighted the version number and public key tokens in the output above because they'll change in a moment.

Let's take a look at the mocked-up System.DirectoryServices. You'll note that I've had to make sure that the methods exist on the same classes that they would in the real System.DirectoryServices. For example, when ThirdPartySecurity calls Dispose on the real DirectorySearcher it's actually calling the Dispose method on System.ComponentModel.Component because that's what the real DirectorySearcher inherits from. So our fake version does too. By contrast, the real PropertyCollection has no base class - it handles all the collection logic itself; so our mock version needs to ensure that properties like Count are available directly on the PropertyCollection class and are not simply inherited from the base Dictionary.

namespace System.DirectoryServices
{
    public class DirectorySearcher : System.ComponentModel.Component
    {
        public DirectoryEntry SearchRoot { get; set; }
        public string Filter { get; set; }
 
        public SearchResultCollection FindAll()
        {
            SearchResultCollection searchResultCollection = new SearchResultCollection();
            SearchResult searchResult = new SearchResult();
            searchResultCollection.Add(searchResult);
            return searchResultCollection;
        }
    }

    public class DirectoryEntry
    {
        private PropertyCollection _properties;

        public PropertyCollection Properties
        {
            get { return _properties; }
            set { _properties = value; }
        }
    }

    public class SearchResult
    {
        public DirectoryEntry GetDirectoryEntry()
        {
            DirectoryEntry directoryEntry = new DirectoryEntry();
            directoryEntry.Properties = new PropertyCollection();
            PropertyValueCollection members = new PropertyValueCollection();
            ((System.Collections.IList)members).Add(@"CN=MYDOMAIN\ian.picknell,otherstuff");
            ((System.Collections.IList)members).Add(@"CN=MYDOMAIN\jane.armstrong,otherstuff");
            directoryEntry.Properties["member"] = members;
            return directoryEntry;
        }
    }

    public class SearchResultCollection : System.Collections.ArrayList, IDisposable
    {
        public void Dispose()
        {
        }
    }

    public class PropertyCollection : System.Collections.Generic.Dictionary<string, PropertyValueCollection>
    {
        new public int Count
        {
            get { return base.Count; }
        }

        new public PropertyValueCollection this[string key]
        {
            get { return base[key]; }
            set { base[key] = value; }
        }
    }

    public class PropertyValueCollection : System.Collections.CollectionBase
    {
    }
}

We need to make sure this is compiled to an assembly named System.DirectoryServices via the Assembly Name property on the Application tab of Project Properties. We also need to make sure that it has a strong name - I just assigned it a new key pair via the Signing tab of Project Properties.

Okay, so how to be persuade ThirdPartySecurity to use this mock implementation of System.DirectoryServices rather than the real thing?

Magic

The first thing we need to do is add a configuration file for ConsoleApplication to redirect requests for System.DirectoryServices from version 2.0.0.0 to some other version - I'll choose 2.1.0.0.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="System.DirectoryServices"
                          publicKeyToken="b03f5f7f11d50a3a"
                          culture="neutral" />
        <bindingRedirect oldVersion="2.0.0.0"
                         newVersion="2.1.0.0" />
      </dependentAssembly>
    </assemblyBinding>
  </runtime>
</configuration>

This just uses a standard assembly re-direction. As I mentioned earlier, such re-directions only allow the version number to be changed - you can't change the assembly name or public key token.

To persuade .NET that our MockDirectoryServices is actually the real System.DirectoryServices simply create the appropriate folder in the GAC and drop it in there:

> MD %SystemRoot%\assembly\GAC_MSIL\System.DirectoryServices\2.1.0.0__b03f5f7f11d50a3a
> COPY System.DirectoryServices.dll %SystemRoot%\assembly\GAC_MSIL\System.DirectoryServices\2.1.0.0__b03f5f7f11d50a3a

So we're copying our System.DirectoryServices (actually generated from the MockDirectoryServices project), which has a version number of 1.0.0.0 (because we didn't change it) and some random public key token and dropping it in the folder which should contain version 2.1.0.0 of System.DirectoryServices, signed with Microsoft's key pair (which has a public key token of b03f5f7f11d50a3a). In other words, we're lying.

When we run ConsoleApplication this time, we get a much more healthy result - even with no Active Directory in sight. Again, I've highlighted some interesting elements in the output.

The members of the group "MYDOMAIN\MyGroup"are:
        MYDOMAIN\ian.picknell
        MYDOMAIN\jane.armstrong

DirectoryServices-related assemblies loaded into the current AppDomain are:
        System.DirectoryServices, Version=1.0.0.0, Culture=neutral, PublicKeyTok
en=7c6ae5d5059e81d9 (from file:///C:/Windows/assembly/GAC_MSIL/System.DirectoryS
ervices/2.1.0.0__b03f5f7f11d50a3a/System.DirectoryServices.dll)

Press ENTER to exit

You'll note that our AppDomain knows that it's dealing with version 1.0.0.0 of an assembly which has a public key token of eeab6744580125d7, which is clearly not the version ThirdPartySecurity references. But it doesn't seem to care. It was loaded from a folder within the GAC where version 2.1.0.0 with a public key token of b03f5f7f11d50a3a was expected to exist, and that seems good enough for it.

So, if you ever find yourself needing to provide a mock implementation of an assembly which has a strong name, now you know how.

Ian Picknell - Under the Covers

Execute Procedure for XML Auto

Introduction

Managed Stored Procedures

Conclusion

See Also

Implementing IDispatchMessageInspector

Introduction

Creating the WCF Service

IEmployeeService

EmployeeService

Employee

Server

Creating the WCF Client

Client

Testing the Client and Service

Changing the Contract

Enter IDispatchMessageInspector

Alternate Implementations

Conclusion

See Also

Detecting Month-Ends in JavaScript

Background

The Solution

An Actual Solution

Selectively Overriding XML Serialisation Attributes

Introduction

Book and Genre classes (with Xml Attributes)

Solution

Summary

See Also

Default Values Don't Get Serialised

The Problem

The Solution

See Also

Validating Integers in SQL Server

Introduction

ISNUMERIC

ARITHABORT

User-Defined Function

Numeric, but not Integer

Exponential Notation

Solution

Post Script

Strong Name Storage

Introduction

Portable Executable / Common Object File Format

CLR Runtime Header / CLI Header

Strong Name Signature

Public Key

Method Implementations

Summary

See Also

Launching a ClickOnce Application

Introduction

Retrieving the Deployment Manifest

Opening the Deployment Manifest

Summary

Serialising Enumerations with WCF

Introduction

Mock-up of Original Problem

Testing the Application

Diagnosing the Problem

See Also

Faking a Strong Name

Introduction

Scenario

Problem

Solution

Magic

See Also