The set of project item templates has been extended to support preprocessed templates and generators. Here is what you will see when you add a new item to a Visual Basic project.

This list contains familiar project item templates, such as Template, Generator and Unit Test. Script is the project item template that was previously named File. It is a traditional .tt file that is meant to include other, partial .tt files such as Template and Generator. Hopefully the new name Script makes more sense than File.

The list of project item templates for Visual C# projects is similar and also includes ready-to-use code generators written in C#. It is the same with the exception of the Enum SQL View, which at this time works only in Visual Studio 2008.

Preprocessed Templates and Generators

Preprocessed Template and Preprocessed Generator project item templates, which are similar to the traditional Template and Generator respectively, were added take advantage of the new capability offered by Visual Studio 2010 to preprocess .tt files and compile T4-based code generators in .NET assemblies.

When you add a Preprocessed Template to your project, two files get created - a preprocessed .tt file and a partial .cs or .vb file similar to the ones below.

Template.tt (C#)

<#@ template language="C#" inherits="T4Toolbox.Template" #>

Template.partial.cs

namespace ClassLibrary1
{
    using System;
    using T4Toolbox;

    public partial class Template1
    {
        protected override void Validate()
        {
            this.Warning("Template properties have not been validated");
        }
    }
}

Template.tt (Visual Basic)

<#@ template language="VB" inherits="T4Toolbox.Template" #>

Template.partial.vb

Imports T4Toolbox

Partial Public Class Template1
    Inherits Template

    Protected Overrides Sub Validate()
        Me.Warning("Template properties have not been validated")
    End Sub

End Class

For preprocessed .tt files, T4 doesn’t generate actual output; instead it generates the code generation template itself. Here is a cleaned-up version of what you will see if you open the files generated by TextTemplatingFilePreprocessor.

Template.cs

namespace ClassLibrary1
{
    using System;
    using System.CodeDom.Compiler;

    #line 1 "C:\…\Template1.tt"
    [GeneratedCodeAttribute("Microsoft.VisualStudio.TextTemplating", "10.0.0.0")]
    public partial class Template1 : T4Toolbox.Template
    {
        public override string TransformText()
        {
            return this.GenerationEnvironment.ToString();
        }
    }

    #line default
    #line hidden
}

Template.vb

Imports System
Imports System.CodeDom.Compiler

<GeneratedCodeAttribute("Microsoft.VisualStudio.TextTemplating", "10.0.0.0")> _
Partial Public Class Template1
    Inherits T4Toolbox.Template
    Public Overrides Function TransformText() As String
        Return Me.GenerationEnvironment.ToString
    End Function
End Class

Together, Template and Template.partial files define a single class that descends from T4Toolbox.Template which I assume is already familiar to the reader. In a traditional T4 Toolbox Template, you would place all logic in a single .tt file, using a mix of text blocks and class feature blocks. With a preprocessed template, you will typically want to use the .tt file only for the logic that requires text templating. Non-templating logic, such as validation, properties, utility methods, etc. can be placed in the partial .cs/.vb file. This helps to reduce complexity of .tt files by isolating “presentation” code that requires text templating in its own file where it is not mixed with the class definition and model access code.

Despite of what their name implies, Preprocessed Generator project items are not .tt files. A traditional Generator is typically defined in a single class feature block of a .tt file, does not use any text templating features and uses one or more traditional Templates to generate output files. Similarly, a Preprocessed Generator is a simple .cs/.vb file that contains a Generator class definition and uses one or more Preprocessed Templates to generate output files. Here is what it looks like.

C#

namespace ClassLibrary1
{
    using System;
    using T4Toolbox;

    public class Generator1 : Generator
    {
        protected override void RunCore()
        {

        }

        protected override void Validate()
        {
            this.Warning("Generator properties have not been validated");
        }
    }
}

Visual Basic

Imports T4Toolbox

Public Class Generator1
    Inherits Generator

    Protected Overrides Sub RunCore()

    End Sub

    Protected Overrides Sub Validate()
        Me.Warning("Generator properties have not been validated")
    End Sub

End Class

Just like in a traditional generator, you would typically define several properties and override RunCore and Validate methods.

T4Toolbox.Template Class Has a Breaking Change

In order to be compatible with template preprocessing in Visual Studio 2010, interface and implementation of the T4Toolbox.Template class had to change. TransformText method used to have a sealed implementation that called the abstract RenderCore method you had to override in the descendants. Now the TransformText method itself is abstract and serves the purpose of RenderCore which has been removed.

This is a breaking change for any existing traditional Template classes you may have. Here is an example of a Template implementation you had before.

C#

<#+
public class Template2 : Template
{
    public override void RenderCore()
    {
        // Text blocks and code generation logic
        // …
    }
}
#>

Visual Basic

<#+
Public Class Template2
    Inherits Template

    Public Overrides Sub RenderCore()
        ‘ Text blocks and code generation logic
        ‘ …
    End Sub

End Class
#>

And here is an example of a Template implementation you have to have now.

C#

<#+
public class Template2 : Template
{
    public override string TransformText()
    {
        // Text blocks and code generation logic
        // …
        return this.GenerationEnvironment.ToString();
    }
}
#>

Visual Basic

<#+
Public Class Template2
    Inherits Template

    Public Overrides Function TransformText() As String
        ‘ Text blocks and code generation logic
        ‘ …
        Return Me.GenerationEnvironment.ToString()
    End Function

End Class
#>

In order to be compatible with the new version of T4 Toolbox, all existing Template implementations need to be modified to a) replace each RenderCore method override with a TransformText method override and b) return GenerationEnvironment.ToString() at the end of the method.

Template class now provides a new method called Transform, which returns a string and serves the same purpose as TransformText implementation served previously. Just like TransformText, this method is not intended to be called directly by code in your Generator or Script code, where you would normally call Render or RenderToFile. You would only call TransformText (and now Transform) method in unit testing code to validate code generation results without creating the output files. If you have any code that calls the now abstract TransformText method, change it to call the new Transform instead.

Introducing this breaking change was not an easy decision. It was necessary to “unseal” implementation of the TransformText in order to make T4 Toolbox compatible with TextTemplatingFilePreprocessor in Visual Studio 2010 which assumes that this method is abstract. Although it would have been possible to keep backward compatibility, it would result in confusing design and implementations working differently in traditional and preprocessed templates. Taking into consideration the effort required to maintain this going forward, we made a decision to go ahead and convert existing templates now and leave this change behind.

The changes required to upgrade existing code are straightforward. You can use the “Find in Files” feature in Visual Studio to find all instances of RenderCore methods that need to be converted to TransformText. This is a manual change as it requires you to add a return statement at the end of each method. Replacing calls to TransformText method with Transform can be safely accomplished using the “Find and Replace” across multiple files. Including code changes, running unit tests and resolving errors, this change took less than an hour with the existing code generators in T4 toolbox itself.

Under the Hood

In order to make T4 Toolbox compatible with both Visual Studio 2008 and Visual Studio 2010, we now have to separate assemblies: T4Toolbox.dll, a .NET 3.5 assembly which is used in Visual Studio 2008, and T4Toolbox.10.0.dll, a .NET 4.0 assembly which is used in Visual Studio 2010. Introducing a separate new binary was required because Microsoft.VisualStudio.TextTemplating assembly in Visual Studio 2010 has been renamed to Microsoft.VisualStudio.TextTemplating.10.0, which means that TextTransformation base class in Visual Studio 2010 is now physically different than the same class in Visual Studio 2008, forcing you to have separate binaries for descendants.

Producing a .NET 4.0 version of the T4Toolbox assembly required a complete overhaul of the build system. The project files had to be upgraded to Visual Studio 2010. To keep complexity under control and have a single set of project files, they had to be hacked to generate an assembly with different target .NET framework versions depending on build configuration. This had to be done directly in the project files because target framework is exposed as a project-level setting by Visual Studio 2010.

Microsoft SDC tasks don’t appear to support MSBuild 4.0 at this time and generation of Visual Studio project item templates had to be changed to use MSBuild extensions in the DSL SDK.

Because of these changes, the previously published instructions on how to get and compile T4 Toolbox source code are now incorrect. I will not go back and update these instructions until we have a release version of Visual Studio 2010 and things settle down a bit.

©2009 Oleg Sych. All Rights Reserved.

During the first step, the engine preprocesses the template: it parses the processing instructions, text and code blocks, generates a concrete TextTransformation class, and compiles it into a .NET assembly. During the second step, T4 engine creates an instance of the GeneratedTextTransformation class, calls its TransformText method and saves the string it returns to the output file.

Visual Studio 2010 also allows you to preprocess the template at design time, when author of the code generator creates the template itself. At run time, when a developer is using template to generate output code, the hosting application simply creates an instance of the precompiled GeneratedTextTransformation and uses it to generate output as usual. Because preprocessed templates have no hard-coded dependency on Visual Studio, any application can host preprocessed templates, as long as it provides appropriate mechanism for saving or presenting the generated output to the user.

At the time of this writing, no official information is available about preprocessed text templates on MSDN. Gareth Jones and Pablo Galiano published useful information on their blogs (see the References section below). This article is meant to fill the gaps and serve as a comprehensive overview and reference of the functionality supported by preprocessed templates.

Creating a Preprocessed Template

• Create a new C# or Visual Basic project in Visual Studio 2010.
• Select Project -> Add New Item from the main menu

• In the Add New Item dialog, select Preprocessed Text Template item, enter appropriate name and click Add button.
• In Solution Explorer, you should now see the new .tt file and the preprocessed template file it generates. To see the preprocessed template file in a Visual Basic project, make sure to click the Show All Files button in the toolbar of Solution Explorer.

Note that Custom Tool this .tt file was associated with is TextTemplatingFilePreprocessor  instead of the traditional TextTemplatingFileGenerator. Instead of the output file, which would be generated by the traditional template, for the preprocessed template, we the actual source code of the template itself added to the project. This preprocessed template class is compiled as part of the project that contains the .tt file.

• Modify the .tt file to look like so

C#

<#@ template language="C#" #>
Hello World

Visual Basic

<#@ template language="VB" #>
Hello World

• Check the preprocessed template (PreprocessedTextTemplate.vb or .cs) which should look similar to this:

C#

Visual Basic

As you can see, the text, statement, expression and class feature blocks defined in the .tt file become a part of the preprocessed template class just like in a traditional template. However, there are several interesting points to note.

First of all, notice that name of the generated template class is based on the name of the .tt file. In our example, PreprocessedTextTemplate.tt produced class name PreprocessedTextTemplate. In a traditional template, this name would be hard-coded GeneratedTextTransformation. Having file name determine the class name allows us to give template classes meaningful names and organize them in class libraries.

The namespace in which the class is generated is based on the Custom Tool Namespace property of the .tt file. In Visual Basic, this property is My.Templates by default which is what you see generated in the code above. In C# this property is empty by default, causing TextTemplatingFilePreprocessor to construct the namespace based on the default project namespace and location of the .tt file in it. In Visual Basic, clearing the Custom Tool Namespace results in template class being generated in the root namespace of the project. This is also different from traditional templates, which use a hardcoded namespace with a randomly generated suffix, and also allows us to better organize the precompiled templates as part of class libraries.

And last, the preprocessed template class is generated as partial. You can further extend it by adding another partial .cs or .vb file to the project. This allows you to place methods and properties that don’t require templating in a regular source file instead of class feature blocks you would otherwise have to use in a traditional template. Out of the box, Visual Studio offers better debugging experience, IntelliSense and color syntax highlighting for regular source files. However, these differences are reduced only to debugging if you are using the T4 Editor to create your templates.

Using a Preprocessed Template

Executing a preprocessed template is a simple matter of creating an instance of the preprocessed template class and calling its TransformText method. As Pablo Galiano describes here, this can be done in any application. Here we will review how preprocessed templates can be used for code generation in a traditional T4 template, hosted and executed by Visual Studio.

• Add a traditional text template to the project

• Modify it to look like this, using full path to the assembly that contains the precompiled template and its namespace.

C#

<#@ template language="C#" #>
<#@ output extension="txt" #>
<#@ assembly name="C:\...\bin\Debug\CsTemplate.dll" #>
<#@ import namespace="CsTemplate" #>
<#
    PreprocessedTextTemplate t = new PreprocessedTextTemplate();
    this.Write(t.TransformText());
#>

Visual Basic

<#@ template language="VB" #>
<#@ output extension="txt" #>
<#@ assembly name="C:\...\bin\Debug\VbTemplate.dll" #>
<#@ import namespace="VbTemplate.My.Templates" #>
<#
    Dim t As PreprocessedTextTemplate = New PreprocessedTextTemplate()
    Me.Write(t.TransformText())
#>

As you can see, in this traditional template, we are creating a new instance of the precompiled template and writing the code it generates to the standard output file, TextTemplate.txt in this case.

• Check the output file, which should look like this.

Hello World

For simplicity of this example, we have both the preprocessed template and the traditional template using it in the same project. In a real-world scenario, you will typically have a preprocessed template in a separate assembly, which you compile and distribute to your development team for use in other projects.

Differences Between Traditional and Preprocessed Templates

There are several important differences between traditional and preprocessed templates.

Dependency on Microsoft.VisualStudio.TextTemplating assembly

Traditional T4 templates have a dependency on Microsoft.VisualStudio.TextTemplating assembly, which is installed as part of Visual Studio and cannot be redistributed. Preprocessed templates don’t hard-code this dependency and can be used to generate code from a custom hosting application. In the example above, you can see that all code required to execute the preprocessed template is generated as part of its definition and doesn’t rely on any base classes. However, when preprocessed templates are intended for use only within Visual Studio, we can avoid having multiple redundant implementations of the same properties and methods in each template class by using Inherits parameter of the Template directive to specify the standard TextTransformation as the base class.

Template Directive

Language parameter is partially supported. The precompiled template is generated in the specified language, however if template language doesn’t match the language of the project it belongs to, the generated template file will not be compiled. If this happens, the Build Action for the output file is automatically changed to Content.

Debug parameter appears to have no effect on preprocessed templates. Template author is responsible for setting debugging options for the project which contains the preprocessed .tt file.

Inherits parameter is supported and significantly changes the way preprocessed templates are generated. When this parameter is not present, the preprocessed template class will be generated without base class and will have its own implementations of all standard methods inherited by traditional templates from the TextTransformation class. When this parameter is present, the preprocessed template class will be generated with the specified base class and without the standard methods. Template author is responsible for specifying a base class that either inherits from TextTransformation, or provides compatible properties and methods.

Hostspecific parameter is partially supported. The preprocessed template class is generated with the appropriate Host property, however, this introduces compile-time dependency on Microsoft.VisualStudio.TextTemplating assembly. Template author is responsible for adding a reference to this assembly to the project that contains the preprocessed template class. The hosting application is responsible for providing a value for this property at run time.

Culture parameter appears to be fully supported.

Output Directive

The <#@ output #> directive appears to have no effect on preprocessed templates. No error is generated when a preprocessed template contains this directive. The hosting application is responsible for changing the extension of the output file.

Assembly Directive

The <#@ assembly #> directive appears to have no effect on preprocessed templates. No errors is generated when a preprocessed template contains this directive. Template author is responsible for adding appropriate assembly references to the project that contains the .tt file.

Import Directive

The <#@ import #> directive appears to be fully supported by the preprocessed templates. Appropriate using (C#) and imports (Visual Basic) statements are added to the generated template class.

Include Directive

The <#@ include #> directive appears to be fully supported by the preprocessed templates. Template blocks of the included files are merged appropriately in the generated template class.

Custom Directives

Custom directives, such as <#@ xsd #> and <#@ property #> appear to be fully supported by the preprocessed templates. The custom directive processors are invoked appropriately and any additional code they produce is merged with the template code in the generated template class.

References

• DSL 2010 Feature Dives- T4 Preprocessing - Part One - Rationale by Gareth Jones
• DSL 2010 Feature Dives- T4 Preprocessing - Part Two - Basic Design by Gareth Jones
• VS10 Beta 1 / T4 Preprocessing part 1 by Pablo Galiano
• VS10 Beta 1 / T4 Preprocessing part 2 by Pablo Galiano
• Why is a Preprocessed Template the Coolest Thing since Sliced Bread by Kathleen Dollard

Download

• Source code, C#
• Source code, Visual Basic 

©2009 Oleg Sych. All Rights Reserved.

Simplicity and ease of use makes T4 a better code generation tool for application developers, who typically don’t care about generating code in different languages and don’t have time to do it. On the other hand, ability to support multiple output languages makes CodeDOM a better code generation platform for tool developers, who have to support multiple output languages and don’t want to maintain several different code generators that generate the same thing in different languages.

During the past year, we have seen T4 getting adopted by an increasing number of tool developers inside Microsoft, most importantly by the ASP.NET MVC and ADO.NET EF teams. This is great news for application developers, who can now generate their code their way. Or is it? What about tool developers who now have to supply two code generators (T4/C# and T4/VB) instead of one? Actually, make that three code generators - they still have to supply and maintain the original CodeDOM-based generators to support other .NET languages. It is only a matter of time before this added complexity will show its ugly face in the form of quality problems, missing features, missing output languages, etc. In the end, the application developers might just get what they wished for - completely custom code generators that are their to implement and maintain.

Consider the following template from the previous article - Generating Code from DSL Models. This template (ClassTemplate) generates a class declaration based on a definition in a Class Diagram DSL (ModelClass). If you are using T4 Toolbox, you have already recognized the base class, Template, which inherits from TextTranformation. We could have also used pure T4 and have ClassTemplate inherit from TextTransformation directly.

C#

<#+
public class ClassTemplate : Template
{
    public ModelClass ModelClass { get; set; }

    public override string TransformText()
    {
#>
namespace <#= TransformationContext.DefaultNamespace #>
{
    using System;
    using System.Collections.Generic;

    public class <#= this.ModelClass.Name #>
    {
<#+
        foreach (ModelAttribute attribute in this.ModelClass.Attributes)
        {
#>
        public <#= attribute.Type #> <#= attribute.Name #> { get; set; }
<#+
        }
#>
    }
}
<#+
        return this.GenerationEnvironment.ToString();
    }
}
#>

Visual Basic

<#+
Public Class ClassTemplate
    Inherits Template

    Public ModelClass As ModelClass

    Public Overrides Function TransformText()
#>
Imports System
Imports System.Collections.Generic

Namespace <#= TransformationContext.DefaultNamespace #>
    Public Class <#= Me.ModelClass.Name #>
<#+
        For Each attribute As ModelAttribute in Me.ModelClass.Attributes
#>
        Public <#= attribute.Name #> As <#= attribute.Type #>
<#+
        Next
#>
    End Class
End Namespace
<#+
        Return Me.GenerationEnvironment.ToString()
    End Function
End Class
#>

Keep in mind that this template is simple only because it’s a sample. A real-world code generator that provides actual value, such as the LINQ to SQL entity class generator, can easily be thousands lines long. As a tool developer on the T4 Toolbox team, I don’t want to translate the LINQ to SQL template from C# to Visual Basic only so that I have to then maintain both. I would prefer to have a single code base to develop and maintain, but as a tool developer, I feel that I should not limit the audience of my tool just to C# developers.

Do we have to choose between the needs of application developers and the needs of tool developers? Is it possible for tool developers to use CodeDOM to develop code generators and for application developers to use T4 to customize and extend them? Well… YES! Thanks for asking!

How Would That Work?

A Tool developer would implement the ClassTemplate using CodeDOM and encapsulate code that generates individual type members, such as fields or properties, in virtual methods, such as RenderProperty.

C#

public class ClassTemplate : Template
{
  // …
  protected virtual void RenderProperty(ModelAttribute attribute)
  {
    CodeMemberProperty property = new CodeMemberProperty();
    property.Attributes = MemberAttributes.Public | MemberAttributes.Final;
    property.Type = new CodeTypeReference(attribute.Type);
    property.Name = this.LanguageProvider.CreateEscapedIdentifier(attribute.Name);

    property.GetStatements.Add(
      new CodeMethodReturnStatement(
        new CodeFieldReferenceExpression(
          new CodeThisReferenceExpression(),
          this.FieldName(attribute.Name))));

    property.SetStatements.Add(
      new CodeAssignStatement(
        new CodeFieldReferenceExpression(
          new CodeThisReferenceExpression(),
          this.FieldName(attribute.Name)),
        new CodePropertySetValueReferenceExpression()));

    this.LanguageProvider.GenerateCodeFromMember(
      property, this.generationWriter, null);
  }
  // …
}

There is no Visual Basic example of this code, because in this case, the tool developer chose to use C# as the programming language.

An Application developer would override the virtual method in a T4 template and use text templating instead of CodeDOM. The code below illustrates how the code generating properties could be extended to generate DataMemberAttribute declarations and make the properties serializable by WCF.

C#

<#+
    class T4ClassTemplate : ClassTemplate
    {
        protected override void RenderProperty(ModelAttribute attribute)
        {
#>
[DataMember]
<#+
            base.RenderProperty(attribute);
        }
    }
#>

Visual Basic

<#+
    Class T4ClassTemplate
        Inherits ClassTemplate

        Protected Overrides Sub RenderProperty(attribute As ModelAttribute)
#>
<DataMember> _
<#+
            MyBase.RenderProperty(attribute)
        End Sub
    End Class
#>

Note that the application developer is using T4 text templating features to extend a code generator written with CodeDOM. She did not have to reimplement the entire code generator in T4, only modify the logic she was interested in changing. In this example, she has extended the logic that generates property code using her language of choice - either Visual Basic or C#.

What’s the Trick?

Notice that our RenderProperty method uses GenerateCodeFromMember method to convert property declaration from a CodeMemberProperty object to text in the GenerationEnvironment, a StringBuilder object that accumulates output generated by our Template. This is why we can override the RenderProperty method in T4 and use text templating to change the generated code.

The rest of the magic is happening in the TransformText method, which builds the CodeDOM graph of the class we are generating using CodeSnippetTypeMember objects instead of the CodeMemberProperty, CodeMemberField and CodeMemberMethod objects we would use normally. The text for the CodeSnippetTypeMember objects is extracted from the GenerationEnvironment.

public class ClassTemplate : Template
{
  public CodeDomProvider LanguageProvider = CodeDomProvider.CreateProvider("CSharp");
  private StringWriter generationWriter;
  private CodeGeneratorOptions options;

  public ModelClass ModelClass { get; set; }

  public ClassTemplate()
  {
    this.generationWriter = new StringWriter(this.GenerationEnvironment);
  }

  public override string TransformText()
  {
    CodeNamespace @namespace = new CodeNamespace(TransformationContext.DefaultNamespace);
    @namespace.Imports.Add(new CodeNamespaceImport("System"));
    @namespace.Imports.Add(new CodeNamespaceImport("System.Collections.Generic"));

    CodeTypeDeclaration @class = new CodeTypeDeclaration(this.ModelClass.Name);
    @class.TypeAttributes = TypeAttributes.Public;
    @namespace.Types.Add(@class);

    foreach (ModelAttribute attribute in this.ModelClass.Attributes)
    {
       this.RenderField(attribute);
       @class.Members.Add(new CodeSnippetTypeMember(this.GenerationEnvironment.ToString()));
       this.GenerationEnvironment.Length = 0;
    }

    foreach (ModelAttribute attribute in this.ModelClass.Attributes)
    {
      this.RenderProperty(attribute);
      @class.Members.Add(new CodeSnippetTypeMember(this.GenerationEnvironment.ToString()));
      this.GenerationEnvironment.Length = 0;
    }

    this.LanguageProvider.GenerateCodeFromNamespace(@namespace, this.generationWriter, null);
    return this.GenerationEnvironment.ToString();
  }
}

In other words, to generate a particular type member, such as property, we a) build it as a CodeDOM graph; b) convert it to text stored in GenerationEnvironment of the Template; c) extract text from GenerationEnvironment and wrap it in a CodeSnippetTypeMember object; and d) finish building the CodeDOM graph of the type using code snippet objects. Because steps a) and b) occur in a virtual method, we can override it using pure text templating functionality of T4.

What’s the Catch?

Unfortunately, CodeDOM does not provide code snippet descendants for all types. In particular, there is no CodeSnippetTypeDeclaration descendant for CodeTypeDeclaration. Therefore, it is not possible to extract generation of the class declaration from the TransformText method into a virtual method similar to RenderProperty that it could be overridden using T4. Specifically, it would not be possible to extend the code generator in this example and use T4 text templating to generate DataContractAttribute declaration for the class. We can provide an event or a virtual method in ClassTemplate that would allow T4ClassTemplate to access and modify the CodeTypeDeclaration before the output code is generated from it, however T4ClassTemplate would need to manipulate it in the CodeDOM form. It is very much possible to do in T4 code, but would not be as easy as using the text templating.

So What Are You Saying?

As long as a CodeDOM-based generator provides adequate extensibility points, such as virtual methods and/or events, the application developers can extend or replace parts of its logic in T4 without having to reimplement or maintain the entire generator. Although this approach is significantly more complex than creating a T4-based code generator for a single output language, it is comparable in size and complexity to having multiple T4-based generators, producing the same code in different output languages. As the size and complexity of the generated code grows, maintaining several T4-based generators in different .NET languages becomes more complex than maintaining a single CodeDOM-based generator. This approach is a compromise between the needs of application developers, who want to customize generated code using T4 and their language of choice, and tool developers, who want to maintain a single code base.

Download

Source code, C# and Visual Basic. Follow instructions in the previous article to compile and run it.

©2009 Oleg Sych. All Rights Reserved.

Domain-Specific Language Tools

As the name implies, a Domain-Specific Language (DSL) is a programming language dedicated to a particular problem domain. Although we usually associate the term programming language with a textual language, such as C# and Visual Basic, DSLs also take a form of graphical modeling languages, such as UML diagrams, or Excel spreadsheets. Visual Studio SDK includes Domain-Specific Language Tools, which allow you to build DSLs of the graphical variety. Shown below, is a Class Designer built using DSL tools and included as a sample in the Visual Studio SDK.

This sample DSL allows you to model a set of classes in a graphical design environment, similar to a UML diagrams in Visio. The model it creates is stored in XML format.

The Class Designer example also includes T4 templates that you can use as a starting point to generate code from this model. These T4 templates rely on a custom directive processor called ClassDiagrams to make the model data available for code generation. The directive processor ensures that metadata assembly is referenced when the template is compiled and generates a special property called ModelRoot which provides access to the metadata loaded from the model file - Sample.classes.

C#

<#
/***************************************************************************
    Copyright (c) Microsoft Corporation. All rights reserved.
    This code is licensed under the Visual Studio SDK license terms.

    THIS CODE IS PROVIDED *AS IS* WITHOUT WARRANTY OF
    ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY
    IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR
    PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.

***************************************************************************/
#>
<#@ template inherits="ModelingTextTransformation" #>
<#@ output extension=".txt" #>
<#@ ClassDiagrams processor="ClassDiagramsDirectiveProcessor"
  requires="fileName=’Sample.classes’"  #>

Report template

<#= this.ModelRoot.Name #>

<#
  // When you change the DSL Definition, some of the code below may not work.

  foreach (ModelType type in this.ModelRoot.Types)
  {
#>
    <#= type.GetType().Name #> <#= type.Name #>
<#
  }
#>

Visual Basic

<#
REM    Copyright (c) Microsoft Corporation. All rights reserved.
REM    This code is licensed under the Visual Studio SDK license terms.
REM
REM    THIS CODE IS PROVIDED *AS IS* WITHOUT WARRANTY OF
REM    ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY
REM    IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR
REM    PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.
#>
<#@ template inherits="ModelingTextTransformation"language="VB" #>
<#@ output extension=".txt" #>
<#@ ClassDiagrams processor="ClassDiagramsDirectiveProcessor"
  requires="fileName=’Sample.classes’"  #>

Report template

<#= Me.ModelRoot.Name #>

<#
  REM When you change the DSL Definition, some of the code below may not work.

  For Each type As ModelType In Me.ModelRoot.Types
#>
    <#= type.GetType().Name #> <#= type.Name #>
<#
  Next
#>

The output produced by these templates is shown below. In a real-world scenario, you would probably generate something more useful, such as C# or Visual Basic class declarations or, perhaps, SQL table scripts.

Report template

T4Toolbox.Examples.OrderEntry

    ModelClass Customer
    ModelClass Address
    ModelClass Product
    ModelClass Supplier
    ModelClass Order

Because the DSL samples in Visual Studio SDK use standard T4 templates, they can only produce a single output file per template. As a result, you are either limited to generating all artifacts in a single large file, or having multiple templates to generate smaller files. Luckily, T4 Toolbox can be used to generate multiple files from DSL models as we will see later in this article.

Domain-Specific Language Implementation

Before we take a closer look at generating code, we will review some of the basics you will need to understand in order to follow discussion of code generation. For complete details, please refer to the Visual Studio SDK and Domain-Specific Development with Visual Studio DSL Tools written Gareth Jones and other people who lead creation of this technology at Microsoft.

The best way to follow this discussion is by opening the ClassDiagrams example from Visual Studio SDK.

The Class Designer DSL itself is also defined by a DSL model. The screenshot below shows a fragment of its definition located in the DslDefinition.dsl file inside of the Dsl project.

The diagramming notation is similar enough to UML class diagrams for you to be able to interpret it without reading a book on DSLs. As you can see, the sample DSL has a ModelRoot that contains a collection called Types of ModelType objects. ModelClass inherits from ModelType and stores information about a single class, including its collections of Attributes and Operations. In addition to the domain types, like ModelRoot and ModelClass, the DSL definition also includes diagram elements, such as ClassShape and AssociationConnector, which are not shown here.

The Dsl project contains most of the code required to implement the DSL itself and its design environment in Visual Studio. The GeneratedCode folder contains a set of T4 templates that generate most of the required code. In particular, it generates strongly-typed .NET code for accessing the resulting DSL models programmatically. Here is the template called DomainClasses.tt that generates metadata classes for the sample Class DSL.

<#
/***************************************************************************
    Copyright (c) Microsoft Corporation. All rights reserved.
    This code is licensed under the Visual Studio SDK license terms.

    THIS CODE IS PROVIDED *AS IS* WITHOUT WARRANTY OF
    ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY
    IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR
    PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.

***************************************************************************/
#>
<#@ include file="Dsl\DomainClassCodeGenerator.tt" #>
<#@ Dsl processor="DslDirectiveProcessor"
  requires="fileName=’..\DslDefinition.dsl’" #>

The screenshot below shows an extract from the code this template produces. The ModelClass shown in it is generated from the ModelClass element defined in the sample Class DSL diagram above. An instance of the ModelClass represents a particular class defined in the class model, such as Customer or Product shown in the beginning of this article.

ModelRoot, ModelType, ModelClass, ModelAttribute and other classes in this file are what our code generator will use to access model definition created by the Class Designer.

Generating Code From DSL Models

For our DSL code generator, we will use the standard T4 Toolbox design pattern based on Template and Generator classes.

C#

<#@ template language="C#v3.5" hostspecific="True" debug="True" 

    inherits="ModelingTextTransformation" #>
<#@ output extension="txt" #>
<#@ include file="T4Toolbox.tt" #>
<#@ include file="ClassGeneratorCS.tt" #>
<#@ include file="ClassTemplateCS.tt" #>
<#@ ClassDiagrams processor="ClassDiagramsDirectiveProcessor" 

    requires="fileName=’Sample.classes’"  #>
<#
    ClassGenerator generator = new ClassGenerator();
    generator.ModelRoot = this.ModelRoot;
    generator.Run();
#>

Visual Basic

<#@ template language="VBv3.5" hostspecific="True" debug="True" 

    inherits="ModelingTextTransformation" #>
<#@ output extension="txt" #>
<#@ include file="T4Toolbox.tt" #>
<#@ include file="ClassGeneratorVb.tt" #>
<#@ include file="ClassTemplateVb.tt" #>
<#@ ClassDiagrams processor="ClassDiagramsDirectiveProcessor" 

    requires="fileName=’Sample.classes’"  #>
<#
    Dim generator As ClassGenerator = New ClassGenerator()
    generator.ModelRoot = Me.ModelRoot
    generator.Run()
#>

The sample generator consists of two separate classes. ClassTemplate, defined in ClassTemplateCs.tt or ClassTemplateVb.tt, takes ModelClass and generates a class declaration. ClassGenerator, defined in ClassGeneratorCs.tt or ClassGeneratorVb.tt, takes ModelRoot as a parameter, iterates through its list of Types and renders ClassTemplate for each ModelClass it finds. There is nothing special about design or implementation of ClassTemplate and ClassGenerator, except for the ModelClass and ModelRoot types they use as parameters for code generation. These types are provided by the Class Diagrams DSL.

This code generator produces class declarations for types defined in Sample.classes in separate source files.

C#

namespace T4ToolboxDslCs
{
    using System;
    using System.Collections.Generic;

    public class Address
    {
        public String Street { get; set; }
        public String City { get; set; }
        public String State { get; set; }
        public String Zip { get; set; }
    }
}

Visual Basic

Imports System
Imports System.Collections.Generic

Namespace T4ToolboxDsl
    Public Class Address
        Public Street As String
        Public City As String
        Public State As String
        Public Zip As String
    End Class
End Namespace

The main code generation file, SampleCs.tt or SampleVb.tt depending on your language preference, does have several important differences from a typical T4 Toolbox template. First important difference is the <#@ ClassDiagrams #> custom directive placed in the beginning of the template. This custom directive has a requires parameter that expects a value in form fileName=’<FilePath>’, where <FilePath> is a relative path to the DSL model file. The ClassDiagrams directive processor is provided by the Class Diagrams DSL and registered in the experimental registry hive of Visual Studio when you run the Dsl project in the Visual Studio SDK sample. Once the DSL is deployed this directive processor would be registered under standard Visual Studio registry hive.

[HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\9.0Exp\Configuration\TextTemplating\DirectiveProcessors\ClassDiagramsDirectiveProcessor]
@="A directive processor that provides access to ClassDiagrams files"
"Class"="Microsoft.Example.ClassDiagrams.ClassDiagramsDirectiveProcessor"
"CodeBase"="file:///C:/Users/you/Desktop/DslPackage/bin/Debug/Microsoft.Example.ClassDiagrams.Dsl.DLL"

Second important difference is the use of ModelingTextTransformation in the inherits parameter of the template directive. This base class provides additional methods, such as AddDomainModel and ConvertModelRelativePathToTemplateRelativePath, which are required for the code generated by the ClassDiagrams directive processor to be compiled correctly during template transformation.

Running Sample Source Code

Complete source code is available for download in the end this article. It is based on the ClassDiagrams sample from Visual Studio SDK. In order to see the code generation in action, you will need to start debugging the Dsl project in Example.ClassDiagrams solution.

This will start a second instance of Visual Studio in the experimental hive and load Debugging solution. This solution contains several projects. Debugging project came in Visual Studio SDK with this sample and contains standard T4 templates shown at the beginning of this article.

T4ToolboxDslCs and T4ToolboxDslVb are the C# and Visual Basic projects that contain code generators based on the T4 Toolbox. SampleCs.tt and SampleVb.tt are the actual code generators that produce class declarations in separate source files. Remember that you need to click the Show All Files button in the toolbar of Solution Explorer in order to see the output files in Visual Basic projects.

Conclusion

Graphical Domain-Specific Languages are a very powerful tool that can be used to build complete solutions for modeling and generating code for complex problem domains. DSLs are typically developed by Microsoft and other companies who specialize in building development tools. Due to inherent complexity of designing a graphical language and implementing a complete modeling environment integrated in Visual Studio IDE,  DSL development is not feasible in scope of a typical application development project. The investment required to implement and support a viable DSL can only be recouped on some of the largest application development projects that involve dozens of people and last for several years.

These larger projects, where DSL development is justified, typically require generating large volumes of code. Due to the T4 limitation of generating a single output file per template, developers are forced to generate multiple classes in each output file and use multiple template files to generate code from a single model to keep the size of output files manageable. A great example of this problem is the implementation of a DSL itself, which has a single model definition file and multiple code generation templates files. Each of the generated files contains numerous class definitions, making them difficult to read and debug.

The traditional T4 templates are also impossible to extend without resorting to modifying their source code directly. On a large application development project, the team developing the DSL quickly becomes a bottleneck for customization requests. Alternatively, the teams using the DSL start customizing the main T4 templates which requires additional effort to maintain them over time as new versions of the DSL are released.

Although the issues around template extensibility and output file management not be difficult for authors of the DSL, the large number and size of the T4 templates required for non-trivial scenarios present a serious challenge for application developers using it. T4 Toolbox can help you address these problems. By implementing your DSL code generator as a set of Template and Generator classes as shown in this article, you can generate all required output files from a single T4 file and allow developers to extend it with the help of inheritance and encapsulation instead of modifying the main code generator directly.

Download

Source Code, C# and Visual Basic

Resources

• Domain-Specific Language Tools
• Domain-Specific Development with Visual Studio DSL Tools
• Creating reusable code generation templates
• Creating complex code generators
• Making code generators extensible
• Reusing code generators on multiple projects

©2009 Oleg Sych. All Rights Reserved.

Out of the box, TFS 2008 comes with two varieties of process guidance: MSF for Agile Software Development and MSF for CMMI Process Improvement. MSF Agile process was designed for smaller teams and teams that use agile development practices, while MSF CMMI process was designed for larger organizations, particularly those using CMMI (Capability Maturity Model Integration) approach, teams that require a higher degree of control and visibility over their projects.

You can learn more about MSF Agile and MSF CMMI by reading the process guidance that comes with TFS. If you don’t have access to a TFS project with the process flavor you are interested in, you can download it from MSDN using the links below. Simply open ProcessGuidance.html located in the Windows SharePoint Services\Process Guidance folder in the location where you extracted the project template files.

MSF Process Guidance is Only a Template

While MSF Agile and MSF CMMI templates can serve as good starting points for many organizations, there is rarely a perfect fit. The Agile template fits well most of the small- to mid-size IT organizations, however it uses terminology that is too specific to agile software development. Its term Scenario represents what is commonly referred as Service Request or Change Request; the term Bug refers to what is commonly known as Trouble Ticket or Defect Report. The CMMI template uses more appropriate terminology, but due to the large number of work-item types and roles, it is simply an overkill for most organizations.

Today, a typical MIS department usually has a small number of roles (such as Business Analyst, Project Manager, Developer, Database Administrator, Help Desk Technician) established and follows an existing process for managing the change requests and trouble tickets. This process may be based on paper or electronic forms, it may not be perfect, but it is already working. When this organization adopts TFS to improve their process, should they go ahead and adopt MSF Agile or MSF CMMI?

Adopting a new process approach (what you do) at the same time as adopting a new process management tool (how you do it) is a risky proposition. People are creatures of habit and require time to change either the process they follow or the tools they use. Imagine if all cars in a country had to be changed to use voice control instead of traditional steering (change what you do) and that people had to speak to their cars in Esperanto (change how you do it). In the resulting chaos, we can be certain that both the congressman who proposed this improvement and the president who didn’t veto the proposal will not be as popular anymore.

When adopting TFS, unless your IT organization is following the MSF Agile  or the MSF CMMI process to the letter, it is much safer to start by customizing the TFS process to match your own. Once your organization is comfortable using TFS as the new process management tool, you can begin adjusting the process itself using TFS. Microsoft provided extensive documentation (see links below) on how to customize the workflows, work items, reports and other features of TFS. However, customization of the process guidance received relatively little attention. Yet in the process of adopting TFS, providing matching guidance for a customized process is very important to ensure the overall success. Outdated guidance will cause confusion, mistakes, delays and frustration for the MIS department and its business partners. So how do we change it?

VSTS Process Guidance Generator

Microsoft used the Guidance Generator to produce MSF Agile and MSF CMMI guidance for TFS 2008. This tool replaced MSFWinBuild which was used to produce the guidance in TFS 2005. Unlike its predecessor, it uses plain XML file to define contents of the guidance and T4 templates to generate the actual collection of HTML files, included in TFS project templates as process guidance. The remainder of this article will show a typical set of steps you would need to perform to customize guidance for your TFS project.

Installing Guidance Generator

• Download latest version of the Guidance Generator (2.0.2.1 at the time of this writing) to your local hard drive.
• Unblock the ZIP archive you downloaded as shown on the right. This will ensure that the application will not run into any unexpected security errors.
• Extract files from the unblocked archive to your hard drive.

If you are running a 64-bit version of windows, you will need to use the CorFlags command line tool to make sure that the generator runs in 32-bit mode. This will help you to avoid errors that occur when the generator attempts to load 32-bit-only assemblies it requires.

• Use Command Prompt window to execute the following command in the bin\bin sub-folder at the location you extracted the files.

C:\GuidanceGenerator\bin\bin>corflags TextTemplatingMultiprocessor.exe /32BIT+
Microsoft (R) .NET Framework CorFlags Conversion Tool.  Version  3.5.21022.8
Copyright (c) Microsoft Corporation.  All rights reserved.

C:\GuidanceGenerator\bin\bin>

Building Process Guidance

Once you have the Guidance Generator installed, your first task is to pick a template (Agile or CMMI) that you will be using as a starting point in your customization. If you are customizing guidance for an existing TFS project, you will want to minimize the amount of editing, so pick the template that matches matches your project.

• Double-click the Run Tool.cmd file in root folder at the location you extracted the files to start the generator.   
• In the TemplateProcessor window, enter the following parameter values.

  Source DSL File:      agile-guidance.guidance

  Template Directory: HTML_Templates

  Target Directory:       agile-Generated\Process Guidance\Supporting Files

The source file provides content of the guidance. In this example, we are generating process guidance from the source file called agile-guidance.guidance to the output directory called agile-Generated. To generate CMMI guidance, we would need to change these values to cmmi-guidance.guidance and cmmi-Generated respectively.

• Click the Go button.

The generator runs for a while and you should see the a log displayed in its output window with information about the number of files that were generated. Once you see “Done”, you can open the generated guidance by double-clicking the ProcessGuidance.html located in the agile-Generated\Process Guidance folder. The actual output files are generated in the agile-Generated\Process Guidance\Supporting Files folder.

Customizing Guidance Content

The guidance content is located in the .guidance files - agile-guidance.guidance and cmmi-guidance.guidance. These are rather large XML files that define topics describing team member roles, work streams, activities, work products, work items, reports, glossary entries and more.

Limited information about the structure of this file can be found in the UserGuide - Guidance Generator.pdf located in the root folder where you extracted the Guidance Generator files downloaded from CodePlex. This information is by no means complete nor, due to its volume, would not be practical to describe in a single article. In one sentence, this structure can be summarized as this. There is a one-to-one relationship between the major xml elements and the generated HTML files, for example, BusinessAnalyst.htm file is generated from the XML element <Role Key=”BusinessAnalyst”>. If you want to change content for particular HTML file, find the XML element with matching Key and change its contents.

Source XML

Generated Output

Editing such a large and complex XML file is challenging. It was not meant to be edited by hand and the internal tool Microsoft created for editing it was not released to public. You may be tempted to use the XML schema file, GuidanceModelSchema.xsd, to enable IntelliSense and enhance your editing experience in Visual Studio, but unfortunately, it appears to be outdated and doesn’t match the supplied .guidance files.

Instead, you will need to frequently re-run the generator to verify that content you are modifying produces the HTML you expect. You can reduce the amount of time it takes to regenerate output by setting Template Files parameter to regenerate only a the particular type of file you are working on. For example, Business Analyst.htm is generated by the T4 template called Role.htm and you can regenerate only role files like so.

Customizing Guidance Output

Guidance Generator uses T4 templates located in the HTML_Templates directory to generate output HTML files (in the Supporting Files directory). There is one template file for each major type of XML element in the .guidance file (<Role>/Role.htm, <Activity>/Activity.htm). As you can see, the T4 template files are saved with .htm extension which might be confusing at first.

The final HTML files rely on additional web resources - JavaScript, Cascading Style Sheet and image files located, respectively, in the Code, CSS and Images sub-folders of the Supporting Files folder where the generated output files are saved. Each guidance (Agile and CMMI) has its own set of supporting files.

To change look and feel of the generated HTML, you can start by changing the CSS\msf.css style-sheet or changing the image files to display your company’s logo. However, if you need more substantial changes, you may have to modify the T4 templates that generate the HTML. For detailed information about T4, please see my previous article, Text Template Transformation Toolkit.

Publishing New Guidance

Once you finish changing the guidance, you can package it in a TFS process template, or simply upload it to the Process Guidance folder on your team portal.

Conclusion

TFS is a powerful collaboration platform for IT organizations. Although TFS comes with two pre-packaged process templates (MSF Agile and MSF CMMI), it is often beneficial to customize them to better fit the process already used by your organization. It is also important to keep documentation in sync with the process you are using and minimize confusion among the members of the immediate IT team as well as the business users. Guidance Generator is a powerful tool for customizing the built-in process guidance that comes with TFS. Although the Guidance Generator is not very easy to use, the interactive documentation system it produces is well designed, easy to use and would be challenging to replicate in hand-crafted HTML or Word files. In most cases you will want to start with an existing process guidance that most closely matches your current process and customizing it, hopefully by simply removing the MSF concepts you currently don’t use.

References

• MSF for Agile Software Development (TFS Project Template)
• MSF CMMI Process Improvement (TFS Project Template)
• How To: Customize a Process Template in Visual Studio Team Foundation Server
• Team Development with Visual Studio Team Foundation Server
• VSTS Process Guidance Generator
• Text Template Transformation Toolkit
• Team Foundation Administrators: Customizing Process Templates

©2009 Oleg Sych. All Rights Reserved.

T4 Toolbox now comes with a set of project item templates for Visual Basic. Here is what you will see when adding a new item to a Visual Basic project.

If you are familiar with these project item templates in C#, you will recognize the code in Visual Basic.

File

<#@ template language="VBv3.5" hostspecific="True" debug="True" #>
<#@ output extension="txt" #>
<#@ include file="T4Toolbox.tt" #>
<#
‘ <copyright file="File1.tt" company="Your Company">
‘  Copyright © Your Company. All Rights Reserved.
‘ </copyright>

#>

Template

<#+
‘ <copyright file="Template1.tt" company="Your Company">
‘  Copyright © Your Company. All Rights Reserved.
‘ </copyright>

Public Class Template1
    Inherits Template

    Public Overrides Function TransformText()    

        Return Me.GenerationEnvironment.ToString()
    End Function

End Class
#>

Generator

<#+
‘ <copyright file="Generator1.tt" company="Your Company">
‘  Copyright © Your Company. All Rights Reserved.
‘ </copyright>

Public Class Generator1
    Inherits Generator

    Protected Overrides Sub RunCore()    

    End Sub

End Class
#>

Entire functionality of the core framework of T4 Toolbox is now available to Visual Basic developers. I have updated the T4 tutorial series to provide examples in both C# and Visual Basic.

• Creating your first code generator
• Troubleshooting code generation errors
• Debugging code generation files
• Creating reusable code generation templates
• Creating complex code generators
• Reusing code generators on multiple projects
• Handling errors in code generators
• Unit testing code generators
• Making code generators extensible

Under the hood

In order to make Visual Basic support possible, the entire core framework, including TransformationContext, Template, Generator and other classes, was moved from .tt (Text Template) files to .cs (C#) files and is now compiled in the T4Toolbox.dll assembly. Some of the original extensions, such as initialization of the TransformationContext and TestRunner classes had to be reimplemented using custom directive processors.

Note that the set of templates available in Visual Basic is smaller than for C#. That is because ready-to-use code generators, such as AzMan wrapper, Enum SQL view and LINQ to SQL model are still implemented in C#. There are no plans to convert them to Visual Basic at this time.

Having the entire core framework code compiled as part of a normal C# class library project, we will finally be able to use Sandcastle and generate documentation for T4 Toolbox.

Known issues

Developers working primarily with C#, like myself, may be surprised to find that Visual Basic doesn’t display generated output files by default. You have to select the Show All Files button in toolbar of Solution Explorer in Visual Studio in order to see them.

A bigger issue is using Visual Basic in partial templates that contain code meant to be included and never used separately, such as Template and Generator classes. Without the template directive, current version of the T4 Editor assumes the language to be C# and displays a large number of errors for Visual Basic code. As a workaround, you can temporarily add a template directive to such file. To avoid warnings, it has to be removed once the file is included and compiled as part of another .tt file because T4 does not support multiple template directives.

This workaround is quite painful to use. To make this work right, the T4 editor needs to somehow infer the language from files without the template directive. One of the possible ways to do this is with file extensions. For example, partial C# text templates could use extension .ttcs and partial Visual Basic templates could use .ttvb. What do you think? Is there a better way to solve this?

©2009 Oleg Sych. All Rights Reserved.

Of course the company can only be as great as the people working there make it. I would like to thank these great people working at Microsoft for the inspiration and support: Joe Healy, Gareth Jones, Jim Blizzard, Jean-Mark Prieur, Jeff Barnes, Scott Hanselman, Rob Caron, Tim Fischer. Regardless of whether "community work" is in your job description or not, you helped me in one way or another and I am grateful for that.

This award also would not be possible without the prominent members of the Microsoft community: Scott Doorman, Stan Schultes, Dave Noderer, John Dunagan, Steve Lane, Shawn Weisfeld, Roy Lawson, Ken Tucker and many many more. So much goes on behind the scenes to keep the user groups going and prepare the code camps. Without your work the development community in Florida wouldn’t be as great as it is today.

And last, but not least, I want to thank my friends and colleagues at Catapult: Jeff Odell, Randy Patterson, Terri Burmeister, James McAuliffe. Whether it was buying pizza for a user group meeting, discussing technical issues after hours or having a beer after work, your encouragement and support is very much appreciated.

Sincerely,
Oleg

©2009 Oleg Sych. All Rights Reserved.

Over the past year, several people have asked me to provide instructions on how to compile T4 Toolbox. The process is not difficult, but since I don’t do it every day, I have to dig through my email archive to find all the steps every time I need to provide the answer. This seems to be as good time as any to write everything down, so here it goes…

You will need latest versions of the following tools installed in order to compile T4 Toolbox.

• Visual Studio 2008 Professional or Team edition
• StyleCop
• WiX 3.0
• Visual J# 2.0 Redistributable Package
• SDC Tasks

Installing Tools

You can install most of the tools with default options.

• When installing StyleCop, make sure that MSBuild integration files feature is installed.

• SDC Tasks don’t have an installer; binary files come in a ZIP archive. For T4 Toolbox, you will need to extract these files to C:\Windows\Microsoft.NET\Framework\v3.5 directory.

• Open the extracted Microsoft.Sdc.Common.tasks XML file and comment out the <PropertyGroup/> element in the beginning of the file (enclose it with <!– and –>). This element generates warnings during compilation.
• Unless you have BizTalk installed on your workstation, you will also need to comment out the entire section at the bottom of this file that declares BizTalk tasks implemented in Microsoft.Sdc.Tasks.BizTalk.dll assembly. Otherwise you will see a large number of errors referring to BizTalk assemblies during compilation.

Accessing Source Code

You have two different ways of getting T4 Toolbox source code. If you have not been added to the list of developers,

• On the Source Code tab of the project page, click a Download link for the latest check-in.

• After accepting the license agreement, you will be prompted to download a ZIP archive that contains the latest version of the T4 Toolbox source code. Save the file to your local hard drive first and unblock it before extracting the files. Unless you unblock the ZIP archive first, the Visual Studio will not “trust” it and you will get obscure security errors and warnings later on.

• Extract all files from this archive to a folder on your hard drive, such as C:\T4Toolbox, and open T4Toolbox.sln located in the Source subfolder.

• When Visual Studio opens T4 Toolbox solution for the first time, select the Load project normally option. T4Toolbox.csproj uses MSBuild extensions from SDC tasks and StyleCop, you will need to select this option in order to compile it.

At this point, you should be able to compile the T4 toolbox. Make changes until it works the way you need it and then upload the files you modified as a patch on CodePlex. Myself or someone else will take a look at the changes you’ve made and commit them to the main source code of the project.

Using Team Explorer

When you are regularly working on T4 Toolbox source code, having to download a source code snapshot becomes a hassle. When your CodePlex account has been added to the list of T4 Toolbox developers, you can start using Team Explorer from Visual Studio.

• Download and install Team Explorer from here.
• Once you have it installed, go to T4 Toolbox page on CodePlex, make sure you are logged (1), open the Source Code tab (2) and then click the Visual Studio Team Explorer link (3). On the right-hand side, you will see the information you will need to use in Team Explorer (4).

• In Visual Studio, select View->Team Explorer from the main menu.
• In Team Explorer, click Add Existing Team Project button
• In the Connect to Team Foundation Server dialog, click Servers button
• In the Add/Remove Team Foundation Server dialog, click Add button
• In the Add Team Foundation Server dialog, enter server name, port number and protocol from the CodePlex page and click OK.

• Back in the Add/Remove Team Foundation Server dialog, click Close button
• Back in the Connect to Team Foundation Server dialog, select T4Toolbox project and click OK button
• Back in Team Explorer, you should now see the T4Toolbox project and its information.

• Double-click the Source Control node to open Source Control Explorer

• In Source Control Explorer, select T4Toolbox folder on the left side and click the Not mapped link on the right.
• In the Map dialog, enter name of the local folder you want to use for T4 Toolbox source code and click Map button.

• Back in Source Control Explorer, click Get Latest Version  button to download the source code to your local hard drive.
• Once downloaded, double-click T4Toolbox.sln in the Source sub-folder to open the solution.

At this point, you can check files out, change them and check them in. The source control functionality is similar to Visual SourceSafe and Subversion, so it should be familiar to most developers.

©2009 Oleg Sych. All Rights Reserved.

This limitation of the built-in LINQ to SQL functionality is one of the reasons developers start using a second tool, such as SQL Server Database Diagrams, Visio for Enterprise Architects or Enterprise Architect by Sparx Systems, and designing the database schema separately from the application model. While this approach works, it significantly increases complexity of the development process. Every change now needs to be made in two places - the database model and the application model. In all but trivial cases, making these changes by hand is too tedious and error-prone to be practical. You are better off choosing one model and somehow synchronizing it with the other. Today, developers typically choose database model as the source of truth and update the entity model based on it, or in other words, use database-driven or database-first approach.

LINQ to SQL includes two tools that allow you to generate .dbml from an existing database - O/R Designer and SqlMetal.exe . Unfortunately, both of these tools overwrite existing dbml and don’t allow you to preserve customizations made in the raw relational model imported from the database. SqlMetal also allows you to generate application code from a database directly. By choosing these tools, you are limiting your LINQ to SQL model to a direct representation of your relational database schema and eliminate the possibility of taking advantage of the object-oriented features the framework supports, such as inheritance of entity classes, member access and virtual member modifiers. Alternatively, you can use these tools to create the initial version of the entity model you will customize later, however that brings you back to the issue of maintaining two models separately.

There are third-party tools that make ongoing database-driven development of LINQ to SQL applications more practical. In particular, PLINQO, a set of code generation templates developed by CodeSmith, includes a sophisticated model synchronization logic, which updates .dbml based on database schema while attempting to preserve the customizations. However, these tools don’t eliminate the fundamental requirement of maintaining the two models separately. By itself, this requirement is not a problem. On a large-scale project with a complex database schema, this may well be a necessity. However, on a small- to medium-scale project, having to maintain the two models separately is often an additional, unnecessary burden.

Build 9.5 of T4 Toolbox extends the LINQ to SQL generator introduced in version 9.1 with support for database schema generation. This gives you the ability to generate both application code, such as entity and data context classes, and database code, such as table, primary and foreign key scripts, from a single LINQ to SQL dbml file in your solution. Generated database scripts can be automatically added to a VSTS Database project, which will automatically handle majority of database deployment issues by automatically generating the ALTER scripts. In other words, combined with VSTS Database edition, T4 Toolbox  allows you to use model-driven or model-first approach for ongoing development of LINQ to SQL applications.

Usage

• Download and install the latest version of T4 Toolbox from CodePlex.
• In Visual Studio, create a new or open an existing C# project and select Project->Add New Item in the main menu.
• In the Add New Item dialog, select Code Generation->LINQ to SQL model

• Enter the name you want to use for the new LINQ to SQL model (.dbml) file. Note that this name also determines name of the DataContext class that will be generated (OrderEntryDataContext will be generated for OrderEntry.dbml). Click the Add button when finished.

Just like with the previous version of this template, you will see that two files, a .dbml and a .tt, were added to your project. This time, instead of populating the entity model by dragging and dropping tables from the Server Explorer to the O/R Designer, take a couple of minutes to create the model from scratch and take advantage of some of the LINQ to SQL features like inheritance and associations. There is no need to enter mapping information, such as table or column names and database types. The generator will automatically deduce them from the type names, property names and CLR types respectively.

Alternatively, you can download the sample source code accompanying this article. The sample includes an entity model of a simplistic Order Entry application shown above. This model showcases some of the features supported by the code generator, including auto-generated properties (Contact.Id), version properties (Contact.RowVersion), derived types (Employee, Customer, Vendor) , associations (Contact_Employee) and associations where a derived type serves as a parent (Customer_Order).

• Add a new VSTS database project to the solution.

Make sure to select an appropriate project type in the Database Projects  folder of the Add New Project dialog. Legacy database project template, located under Other Project Types -> Database will not work. If you don’t have Database edition of VSTS installed, you can use any C# project type instead.

• In the previously created .tt file (OrderEntry.tt in this example), modify the code to assign DatabaseProject property of the LinqToSqlGenerator instance.

<#@ template language="C#v3.5" hostspecific="True" debug="True" #>
<#@ output extension="log" #>
<#@ include file="T4Toolbox.tt" #>
<#@ include file="T4Toolbox\LinqToSql.tt" #>
<#
    // Generate entity classes from a LINQ to SQL class model (.dbml file)
    LinqToSqlGenerator generator = new LinqToSqlGenerator();
    generator.DbmlFile = "OrderEntry.dbml";
    generator.DatabaseProject = @"..\Database\Database.dbproj";
    generator.Run();
#>

DatabaseProject value is a path to the database project file, relative to the location of the .tt file. When you save the .tt file, it will generate table, primary and foreign key scripts in this project. At this point, you can build the VSTS Database project and deploy the OrderEntry database to a SQL server. Detailed discussion of the VSTS Database project functionality is outside of scope this article, suffice it to say that this should require only right-clicking the database project in Solution Explorer and selecting Deploy from the context menu.

As you continue using O/R designer to add new properties and types to the LINQ to SQL model, simply right-click the OrderEntry.tt file in the Solution Explorer and select Run Custom Tool to regenerate the database code, build the database project, deploy and let VSTS automatically apply ALTER scripts to the existing database.

Feature Highlights

Let’s review some the features supported by this code generator.

CREATE TABLE dbo.Contact
(
    Id INT NOT NULL IDENTITY,
    Type tinyint,
    RowVersion ROWVERSION,
    FirstName NVARCHAR(4000),
    LastName NVARCHAR(4000),
    StreetAddress NVARCHAR(4000),
    City NVARCHAR(4000),
    State NVARCHAR(4000),
    Zip NVARCHAR(4000),
    Phone NVARCHAR(4000),
    Email NVARCHAR(4000),
    SocialSecurityNumber NVARCHAR(4000),
    EmergencyContactId INT,
    AccountNumber NVARCHAR(4000),
    CreditCardNumber NVARCHAR(4000),
    CreditCardExpiration NVARCHAR(4000)
);

Derived types mapped to base table

The generator automatically generates columns for derived types, such as Employee, in the table of their base type, such as Contact. In particular, SocialSecurityNumber is a property of the Employee type, stored in the Contact table.

IDENTITY columns

The generator automatically maps integer properties with Auto Generated Value attribute set to true, such as Id,  to IDENTITY columns.

ROWVERSION data type

The generator automatically maps byte[] and Binary properties with Time Stamp attribute set to true, such as RowVersion, to the ROWVERSION data type.

Enum columns

The generator automatically maps enum properties, such as Type property of the Contact class, to SQL data type INT. You can also specify the server data type explicitly. In the application, the property will be generated with the original enum type.

private ContactType type;

[Column(Name = "Type", /* … */ DbType = "tinyint")]
public ContactType Type { /* … */ }

This allows you to write strongly-typed queries like so:

var dc = new OrderEntryDataContext(Settings.Default.Database);
var employees = from c in dc.Contacts
                where c.Type == ContactType.Employee
                select c;

Primary keys

ALTER TABLE dbo.Contact
    ADD CONSTRAINT PK_Contact
    PRIMARY KEY CLUSTERED (Id);

The generator automatically produces a primary key script for properties with Primary Key attribute set to true.

Delimited Identifiers

CREATE TABLE dbo.[Order]
(
    Id INT NOT NULL IDENTITY,
    OrderDate DATETIME NOT NULL,
    EmployeeId INT NOT NULL,
    ProductId INT NOT NULL,
    RowVersion ROWVERSION,
    CustomerId INT NOT NULL,
    Price DECIMAL(29,4) NOT NULL,
    PONumber NVARCHAR(4000)
);

The generator automatically uses delimited identifiers to avoid errors when original identifier is a reserved SQL keyword, such as Order, or contains spaces, such as Order  Details. Otherwise, the generator uses the original, un-delimited identifiers to improve readability.

Foreign Keys

ALTER TABLE dbo.[Order]
    ADD CONSTRAINT FK_Customer_Order
    FOREIGN KEY (CustomerId)
    REFERENCES dbo.Contact (Id);

The generator automatically produces foreign key scripts for associations. Note that in case of Customer_Order association, the parent type Customer is actually derived from Contact. The foreign key correctly references the base table, however the generated association property (shown below), is of the derived type, Customer.

[Association(Name = "Customer_Order", /* … */ IsForeignKey = true)]
public Customer Customer { /* … */ }

Sample Source Code

Sample source code, available for download below, includes a solution with three projects - Model, Database and Application. Model is a C# class library project that contains OrderEntry.dbml (LINQ to SQL model) and OrderEntry.tt (code generator). Entity and Data Context classes are generated in the Model project, while SQL scripts are generated in the Database project. Note that this project requires VSTS Database edition. Application is a C# console application that creates a sample database on the local SQL Server instance, populates it with sample data and executes several queries.

Closing Thoughts

The new version of LINQ to SQL generator in T4 Toolbox allows you to use model-first approach for ongoing developing database applications.

So what’s the big deal? Simplicity.

• There is no need to use a separate modeling tool when developing a database application. You can generate both database and application code without leaving Visual Studio IDE.
• There is only one diagram to maintain, which reflects both entity and database models. If you are editing both at the same time, there is no way you can forget update one of them after updating the other.
• There is only one step in the code generation process - Run Custom Tool from Solution Explorer. You don’t need to remember to make the changes in the modeling tool, export the schema, build the schema, synchronize the .dbml file with the schema and finally generate the application code from .dbml.

What’s the catch then? Limited scalability and flexibility.

• O/R designer is limited to a single diagram, which puts practical limitations on the number of tables you can manage with this approach.
• What you can generate is limited by what you can model in the O/R designer - tables, primary keys and foreign keys. (INSERT, UPDATE and DELETE procedures can also be generated, this is under development now).
• Entity and database models are closely related and cannot be developed separately. If you have an existing database, you may not be able to use this approach.

Will it work for you? It depends.

This approach will work best on greenfield projects with a small- to medium-size database and teams who prefer implementing business logic in application code, not stored procedures and triggers. You can certainly optimize time-critical logic by selectively implementing it stored procedures and exposing them in the LINQ to SQL model. However, you will be swimming against the current if you want to rely on the stored procedures extensively. On brownfield projects, this approach will work as well if you have a database of appropriate size and can limit database modifications to the O/R Designer. This approach will not work well if you have a database with hundreds of tables and/or multiple database developers actively creating tables and stored procedures.

Download

• T4 Toolbox
• Sample source code

©2009 Oleg Sych. All Rights Reserved.

As a consultant, I feel obligated to advise my clients on using integrated Windows authentication as the most secure authentication mechanism for their networks and custom solutions. Unfortunately, as a consultant, I also need to access my clients’ networks from my Catapult laptop, across domain boundaries. This is where integrated Windows authentication makes my job more difficult. Typically, I cannot use my Catapult domain credentials to access resources on the client’s domain and have to supply an alternative set of Windows credentials explicitly. Some applications, like Internet Explorer and Visual Studio are smart enough to prompt you to enter Windows credentials when necessary. While these applications may prompt you to enter credentials more often than you might like, other applications don’t prompt you at all and simply fail. In particular, database applications connecting to SQL Server will not prompt for credentials and display the infamous “Cannot establish SSPI context” error when you try use them across domain boundaries.

Thankfully, there is a relatively small number of tricks you need to use to get most applications using integrated Windows authentication work successfully across domain boundaries. These tricks are described below, but before you rush to try them, make sure you have configured name resolution and can successfully connect to a network resource on the target domain using it’s name. Windows authentication will not work without successful name resolution.

Stored user names and passwords

Windows allows you to store login credentials for various authentication mechanisms, including Windows or Active Directory credentials. You can store a single Windows login (CLIENT\account) and password for an entire domain (*.client.lan) and Windows will use these credentials automatically whenever you access any computer on it.

Not all types of applications support stored user names and passwords. HTTP applications, such as Internet Explorer, Team Foundation Explorer in Visual Studio, and other applications using WebClient, are one of them. Whenever you type http://target.client.lan in Internet Explorer, the WebClient will send the matching stored credentials to the target computer automatically, without prompting you. Windows Explorer, Performance Monitor, Event Viewer and other applications relying on network shares also support this mechanism. As long as you have credentials stored, these applications will just work, as if your computer was a member of the target domain. Here is how you set it up.

• Open User Accounts from Control Panel and click the Manage your network passwords link on the left.

• If you are using an older version of Windows, such as XP or 2003 Server, you will need to click the Manage Passwords button on the Advanced tab of the User Accounts dialog.

• In the Stored User Names and Passwords dialog you need to click the Add button and create a credential entry for the target domain you need to access. Assuming that one of the computers you need to access has a fully-qualified domain name target.client.lan, you want to store credentials for a wildcard *.client.lan, which will used when you access any computers on that domain.

You can find more information about storing user names and passwords in the Microsoft knowledge base article 281660.

Internet security zone

As an added security measure, Internet Explorer will send your current or stored credentials to the target host only when it believes to be a member of the Intranet zone. It will assume that any host addressed with a fully-qualified domain name such as target.client.lan is on the public internet and will not send the credentials automatically. This protects your identity from being stolen by a rouge web site getting your Windows credentials without your permission.

As you can see from the snapshot of Internet Explorer security settings below, even adding the target host to the list of trusted web sites will not work.

In order for Windows authentication to work automatically across domain boundaries, without you having to type in your credentials for every page, you will need to add a particular host or its entire domain to the Local Intranet zone.

• Select Internet Options in Control Panel
• On the Security tab of the Internet Properties dialog, select Local intranet icon and click Sites button
• In the Local intranet dialog, click Advanced button

• In the second Local intranet dialog, enter URL of the web site and click Add button.

Just like with stored user names and passwords, you can use wildcards that apply to all computers on the target domain. For example, if you enter http://*.client.lan, Internet Explorer will consider target.client.lan and all other computers on the client.lan domain to be in the Local intranet zone and send stored user names and passwords to them automatically.

WebClient configuration

Unfortunately, Microsoft Office applications will continue prompting you to enter credentials when accessing files stored on a SharePoint site even after you add the target domain to the Local intranet zone. You will need to modify WebClient configuration in registry to prevent this from happening.

• Open HKLM\System\CurrentControlSet\Services\WebClient\Parameters key in Registry Editor.

• Double-click the AuthForwardServerList value
• In the Edit Multi-String dialog, enter the target URL on a separate line and click OK.

This setting also supports wildcards. If you enter http://*.client.lan, Microsoft Office applications will automatically send stored Windows credentials to any SharePoint you access on the client.lan domain. You can learn more about this in Microsoft knowledge base article 941050.

Runas /netonly

Some applications don’t support stored user names and passwords. For example SQL Server Management Studio will not be able to connect to a database on a different domain using Windows authentication. In these cases you can use the runas command with netonly option and make the application access network resources under a different set of credentials.

c:\>runas /user:CLIENT\account /netonly ssms.exe
Enter the password for CLIENT\account:
Attempting to start ssms.exe as user "CLIENT\account" ...

When started with the command line above, SQL Server Management Studio will use CLIENT\account to connect to all SQL servers instead of the actual account you used to log into your computer (CATAPULT\osych in my case). In other words, if you need to connect to a database on sqlserver.client.lan, it will use CLIENT\account credentials you supply.

This trick works for all applications that need to connect to SQL Server using integrated windows authentication, not just SQL Server Management Studio. You can use it to run Visual Studio (devenv.exe) and your custom applications.

You can also use the ShellRunas utility to provide netonly credentials interactively. Once you downloaded and saved the utility in a permanent location on your hard drive, you will need to register it like so.

c:\Program Files (x86)\SysInternals>ShellRunas /reg /quiet  

c:\Program Files (x86)\SysInternals>ShellRunas /regnetonly /quiet

The registration adds Run as different user and Run as different user (netonly) menu items to the shell (i.e. Explorer) context menus for executables.

You can start Visual Studio with netonly context by right-clicking it in the Start menu and selecting the option. Simply provide the credentials required by the target computer you want to access on the other domain and click OK.

While you can also run an application completely under another account, not just netonly, this will only work if the domain you are trying to access has a trust relationship with your own domain. Even then, running an application completely under another account is less convenient than netonly because it runs in a different user profile, loosing your settings, shortcuts and favorites. Remember to select netonly option as it is easy to accidentally omit it.

Local accounts

If none of the options described above works, your last resort is local accounts and workgroup authentication. In workgroup mode, when there is no Active Directory domain controller, i.e. no centralized authentication authority, Windows relies on having matching local accounts on all computers in a workgroup. For example, an application running under local Administrator on your computer, let’s say it’s called CATLAPTOP, can access computer called TARGET on another domain under its local Administrator account as long as CATLAPTOP\Administrator and TARGET\Administrator accounts have the same passwords on both computers. This works for any account, not just Administrator, subject to permissions assigned to the local account on the target computer.

Using local accounts and workgroup authentication appears to be the recommended approach for some advanced cross-domain networking scenarios, such as remote debugging and SQL Server integrated security. However, these particular scenarios work with runas/netonly approach as well. Unlike the workgroup authentication approach, runas/netonly doesn’t require you to get access to a local account on each target computer. Maintaining these accounts is a hassle for IS administrators and they are typically reluctant to do it. You will probably want to use this option only as a last resort.

If you have to use this option, make sure that your local security policy allows workgroup authentication.

• Select Local Security Policy from Control Panel under Administrative Tools
• In the Local Security Policy console, find Security Options under Local Policies

• Make sure that the Network access: Sharing and security model for local accounts option is set to Classic - local users authenticate as themselves.

Conclusion

Relying on integrated Windows authentication and Active Directory for centralized authentication is a great way to make information systems in your organization more secure. While making life of users and administrators easier by allowing them to use and manage a single set of login credentials for various network resources and applications, integrated Windows authentication presents an additional barrier when you need to access network resources across domain boundaries. Luckily a relatively small set of configuration changes can help you solve most of these difficulties.

This article describes configuration changes and utilities in the order from easiest and most commonly used to most difficult and infrequently used. You will probably want to store credentials for the new domain and add it to the Local intranet security zone first. This enables most modern applications to access network resources on the target domain without errors and without bothering you with constant prompts for credentials. If you need to access Microsoft Office documents on a SharePoint site, you may also want to configure WebClient to automatically use your stored credentials instead of prompting you every time. For advanced scenarios, such as database applications and remote debugging, use the runas command with netonly option or its interactive equivalent, ShellRunas. If all other options fail, you may have to resort to using local accounts and workgroup authentication.

Name resolution and authentication are not the only problems you have to solve to access network resources. Advanced scenarios may require you to make changes in firewall, group policy, security policy, registry and file configurations a particular application may be affected by. Discussion of these scenarios is outside of scope of this article. You can usually find specific instructions in Microsoft knowledge base.

©2009 Oleg Sych. All Rights Reserved.