From The Software Development Trenches

Elementary, my dear Watson

2011-06-24T08:31:00.001+02:00

<rant> I don’t know about any other .NET developer out there, but I’m finding the current speculation with regards to what’s going to happen with vNext of the .NET developer tool stack to be totally frustrating to say the least. This is causing so much angst and damage in the .NET developer community. The uncertainty about what’s going to happen with technology stacks like SL, WPF etc. and the unwillingness of MS to be open about their future is killing a lot of momentum and goodwill and just adding more fuel onto the “oh-what-is-MS-up-to-now-again” discussions. Yes, I know, all will become crystal clear at the BUILD conference somewhere in September, but why didn’t MS keep quiet until then? Figure out what you want to do with the different technology stacks, restructure and get enough prototyping done BEFORE starting to communicate to the developer community. Why even mention something like “HTML 5 and JavaScript is the way to go” with the first demo preview of Windows 8. Surely MS should have expected the reaction by developers wanting to know what the future holds for the other technology stacks? Anyway, I hope MS gets this right and that v1 of the next, seemingly consolidated UX platform, does not set us back another 2/3 years in waiting for it to become a really usable technology stack. </rant>

So until BUILD in September we are left to act as Sherlock Holmes – looking at information surfacing via leaked e-mails and other blog posts. Alternatively we can just let it be and hope that everything will indeed fall together in a big “ah-hah moment” come September. I just hope that an already fatigued .NET developer community will have enough energy left to buy into whatever MS is going to preach next and that it will be “Elementary, my dear Watson. Elementary indeed.” Time to get some rest before September then

Managing .NET RESX Duplication – Part 2: Identifying Resource Duplication

2011-06-17T08:03:00.001+02:00

Update 20/06/2011: Fixed bug in C# snippet that shows how to process list of resources files

This is the second post in a three part series of posts where I look at a possible solution for managing duplication of resources between your .NET RESX files. In part 1 I covered the overall solution requirements and design to fit our environment. In this second part I’m going to delve straight into the custom MSBuild task that identifies the resource duplication across the resource files being used in our code base. As mentioned, the solution consists out of two distinct phases. Phase 1 identifies the resources duplicates across the .RESX files. Phase 2 attempts to minimize the duplication in these files. We decided to implement both these phases as custom MSBuild tasks that run as part of every solution compilation to facilitate our Continuously Execute requirement.

DATASTRUCTURES

Before looking at the custom task, let’s consider the algorithm and data structures that we will use to track the resource duplication. The MSBuild task will receive as input a list of .RESX files that it will scan for duplication. Whilst scanning through the files it will build up a dictionary containing all the resource entries for a single key. This can be visualized as illustrated below:

For every entry found, we need to track the File as well as the ResXDataNode found and for this we use a ResXDuplicateEntry class:

 1   public class ResXDuplicateEntry
 2   {
 3       #region Properties
 4 
 5       public string File { get; set; }
 6       public ResXDataNode Node { get; set; }
 7       
 8       #endregion
 9 
10       #region Methods
11 
12       public static string GetLanguageNeutralFile(string file, string culture)
13       {
14           return culture == null ? file : file.Replace("." + culture + ".resx", ".resx");
15       }
16 
17       public static string GetLanguageSpecificFile(string file, string culture)
18       {
19           return culture == null ? file : file.Replace(".resx", "." + culture + ".resx");
20       }
21       
22       #endregion
23   }
24

A single resource key can contain multiple resource entries. We use a ResXDuplicate container to track all the ResXDuplicateEntry instances for a single resource key across the different files:

 1   public class ResXDuplicate
 2   {
 3       private static readonly ResXDuplicateComparer comparer = new ResXDuplicateComparer();
 4 
 5       public ResXDuplicate(TaskLoggingHelper log, string culture)
 6       {
 7           Log = log;
 8           Culture = culture;
 9           Entries = new List<ResXDuplicateEntry>();
10       }
11 
12       #region Properties
13 
14       public int Count
15       {
16           get { return Entries.Count; }
17       }
18 
19       public string Culture { get; private set; }
20 
21       public string Key
22       {
23           get { return Node.Name; }
24       }
25 
26       public ResXDataNode Node
27       {
28           get { return Entries[0].Node; }
29       }
30 
31       private List<ResXDuplicateEntry> Entries { get; set; }
32 
33       #endregion
34 
35       #region Methods
36 
37       public void AddEntry(ResXDataNode node, string file)
38       {
39           Entries.Add(new ResXDuplicateEntry {Node = node, File = file});
40       }
41 
42       #endregion
43   }

Once we have scanned through all the resource files and build up our dictionary of resource keys with resource entries, we identify the list of duplicates by filtering the dictionary to all entries containing more than one ResxDuplicateEntry class. For these duplicates, we then proceed to compare the individual entries for the same key across different files to each other. For this we created a few extensions methods on the ResXDataNode class to make reading and setting the values a bit easier.

 1   public static class ResXExtensions
 2   {
 3       public static string GetValue(this ResXDataNode node)
 4       {
 5           return node.GetValue((ITypeResolutionService)null).ToString().Trim();
 6       }
 7 
 8       public static ResXDataNode Tag(this ResXDataNode node, string tagToken)
 9       {
10           ResXDataNode taggedNode = new ResXDataNode(node.Name, tagToken);
11           return taggedNode;
12       }
13 
14       public static bool IsTagged(this ResXDataNode node, string tagToken)
15       {
16           return GetValue(node).StartsWith(tagToken);
17       }
18   }
19

When comparing the different values for the same key, we can one of two outcomes:

All the entries match – If all the entries match we can safely report all of them as duplicates
Some entries differ – If only some entries match up, we first find the value amongst the entries that are duplicated the most, and use that to report the number of duplicates identified

We use a ResXStatsSummary class to report the statistics of the resources scan.

 1   public struct ResXStatsSummary
 2   {
 3       #region Properties
 4 
 5       public int Duplicates { get; set; }
 6 
 7       public int NotDuplicates
 8       {
 9           get { return Scanned - Duplicates; }
10       }
11 
12       public int Scanned { get; set; }
13       public int DuplicatesTagged { get; set; }
14 
15       public int TotalResourcesToTranslate
16       {
17           get { return (Duplicates - DuplicatesTagged) + NotDuplicates; }
18       }
19 
20       #endregion
21 
22       #region Methods
23 
24       public override string ToString()
25       {
26           return String.Format(CultureInfo.InvariantCulture,
27               @"Scanned: {1} resource(s){0}Unique: {3} resource(s){0}Duplicates: {4} resource(s){0}Duplicates tagged: {5} resource(s){0}Total to translate: {2} resource(s)",
28               Environment.NewLine, Scanned, TotalResourcesToTranslate, NotDuplicates, Duplicates, DuplicatesTagged);
29       }
30 
31       #endregion
32   }

Scanning Resource Files

Now that we’ve covered the basis of the scanning algorithm and the data structures used for storing the results, let’s look at the actual code that scans the resources files to build the dictionary of ResXDuplicate entries. All the MSBuild tasks that we create inherit from a ResXTaskBase class that contains the shared scanning logic. Here is the definition of the class along with its member variables. Notice the list of files is received as an input parameter for the MSBuild task on line 19. Also notice the dictionary of ResxDuplicate entries that will contain the result of scanning all the resource files on line 10.

 1     public abstract class ResXTaskBase : Task
 2     {
 3         public const string TagToken = "[## SHARED ##]";
 4 
 5         protected IEnumerable<ResXDuplicate> _distinct;        
 6         protected IEnumerable<ResXDuplicate> _duplicates;
 7         protected IEnumerable<ResXDuplicate> _notDistinct;
 8 
 9         // Dictionary keyed on Resource Key that contains all the files containing an entry for the same key
10         protected Dictionary<string, ResXDuplicate> _processed;
11         protected Dictionary<string, List<ResXDataNode>> _distinctFileDuplicates = new Dictionary<string, List<ResXDataNode>>();
12         protected Dictionary<string, List<ResXDataNode>> _mostUsedFileDuplicates = new Dictionary<string, List<ResXDataNode>>();
13 
14         protected ResXStatsSummary _stats;
15 
16         #region Properties
17 
18         [Required]
19         public ITaskItem[] SourceFiles { get; set; }
20 
21         #endregion

Next up is the scanning algorithm. As we want to use the same scanning algorithm to scan all the files specific to a culture, we make sure that we can specify what culture to take into account as a parameter. By default the culture will be null which implies that all the culture neutral resource files will be scanned:

1     protected Dictionary<string, ResXDuplicate> ParseResxFiles(string culture = null)

We start the scanning algorithm by first building up the list of resource files to scan by taking the culture parameter into account. If we are for example scanning the Brazilian Portuguese resources, we make sure that we add the pt-Br culture onto the file name being processed:

1     Dictionary<string, ResXDuplicate> processed = new Dictionary<string, ResXDuplicate>();
2 
3     // Build a list of files to process
4     List<string> resxFiles = new List<string>();
5     foreach (var sourceFile in SourceFiles)
6     {
7         string fileName = ResXDuplicateEntry.GetLanguageSpecificFile(sourceFile.ItemSpec, culture);
8         resxFiles.Add(fileName);
9     }

Once we have the actual list of file names to process, we can now simply run through the list of files, reading all the entries to populate our dictionary with the respective ResXDuplicate entries:

 1     foreach (var resxFile in resxFiles)
 2     {
 3         if (!File.Exists(resxFile))
 4             continue;
 5 
 6         using (ResXResourceReader reader = new ResXResourceReader(resxFile))
 7         {
 8             reader.UseResXDataNodes = true;
 9             Log.LogMessage(MessageImportance.Low, @"Scanning {0}...", Path.GetFileName(resxFile));
10             foreach (DictionaryEntry entry in reader)
11             {
12                 string key = entry.Key.ToString();
13 
14                 ResXDataNode node = (ResXDataNode) entry.Value;
15                 ResXDuplicate duplicate;
16                 if (!processed.TryGetValue(key, out duplicate))
17                 {
18                     duplicate = new ResXDuplicate(Log, culture);
19                     processed.Add(key, duplicate);
20                 }
21                 duplicate.AddEntry(node, resxFile);
22             }
23         }
24     }
25     return processed;

With the dictionary now fully populated with all resource entries, we can proceed to filtering out the duplicate entries. As mentioned, when doing the filtering we identify for every resource key whether all the entries contain the same value - i.e. is distinct or not. For this we use the IsDistinct method created on the ResXDuplicate class:

1     public bool IsDistinct()
2     {
3         List<ResXDuplicateEntry> notAlreadyShared = Entries.Where(x => !x.Node.IsTagged(ResXTaskBase.TagToken)).ToList();
4         return notAlreadyShared.Count == 0 || notAlreadyShared.Distinct(comparer).Count() == 1;
5     }

We first ignored all nodes that have been tagged as shared (we’ll cover this in the next post) and then use our ResXDuplicateComparer and a LINQ query to check whether all entries in the list match. We stored the results of our filtering into two separate lists of ResXDuplicate entries:

 1   public override bool Execute()
 2   {
 3       // Parse the language neutral resource files
 4       _processed = ParseResxFiles(null);                    
 5 
 6       // Identify which entries contain duplicates (i.e. > 1 entry for the same key)
 7       _duplicates = _processed.Values.Where(x => x.Count > 1);    
 8       
 9       // Identify which duplicates contain the same value across all files
10       _distinct = _duplicates.Where(x => x.IsDistinct());         
11       
12       // Identify which duplicates contain different values across all files
13       _notDistinct = _duplicates.Where(x => !x.IsDistinct());

Once we have the lists, we can setup the stats for the scan as follow:

1     // Setup the stats
2     _stats = new ResXStatsSummary
3      {
4          Scanned = _processed.Values.Sum(x => x.Count),
5          Duplicates = _duplicates.Sum(x => x.Count),
6          DuplicatesTagged = _duplicates.Sum(x => x.EntriesTaggedCount()) 
7      };

Lastly, we finish off the scan by creating two separate dictionaries of all resource nodes keyed per file from the existing _distinct and _notDistinct filtered. This is useful for reporting which files contained duplicates and will be used extensively in the next blog post where we cover how to actually eliminate the resource duplication.

1       // Setup Dictionaries keyed per file for the respective duplicate resources
2       _distinctFileDuplicates = BuildResxFileList(_distinct, x => x.ToDictionary());
3       _mostUsedFileDuplicates = BuildResxFileList(_notDistinct, x => x.FindMostUsedEntries().ToDictionary(y => y.File, y => y.Node));

Here is the BuildResxFileList routine:

 1     protected Dictionary<string, List<ResXDataNode>> BuildResxFileList(IEnumerable<ResXDuplicate> duplicates, Func<ResXDuplicate, Dictionary<string, ResXDataNode>> findEntriesFilter)
 2     {
 3         Dictionary<string, List<ResXDataNode>> resourceFiles = new Dictionary<string, List<ResXDataNode>>();
 4         foreach (var duplicate in duplicates)
 5         {
 6             Dictionary<string, ResXDataNode> entries = findEntriesFilter(duplicate);
 7             foreach (var entry in entries)
 8             {
 9                 if (!resourceFiles.ContainsKey(entry.Key))
10                     resourceFiles.Add(entry.Key, new List<ResXDataNode>());
11 
12                 resourceFiles[entry.Key].Add(entry.Value);
13             }
14         }
15         return resourceFiles;
16     }

With all of this infrastructure in place in the ResxTaskBase, creating the ResXFindDuplicate task is straightforward as illustrated below.

 1     public class ResXFindDuplicates : ResXTaskBase
 2     {
 3         public enum LogOutput
 4         {
 5             Text = 0,
 6             Xml = 1,
 7             Csv = 2
 8         }
 9 
10         #region Properties
11 
12         [Output]
13         public ITaskItem[] FilesWithDuplicates { get; set; }
14 
15         [Output]
16         public ITaskItem[] FilesWithNoDuplicates { get; set; }
17 
18         [Required]
19         public string LogFile { get; set; }
20 
21         [Required]
22         public int LogType { get; set; }
23 
24         #endregion
25 
26         #region Methods
27 
28         public override bool Execute()
29         {
30             Log.LogMessage(MessageImportance.Normal, @"Finding duplicate resources within {0} resx files...", SourceFiles.Length);
31 
32             // Parse the language neutral resources
33             bool result = base.Execute();
34 
35             Log.LogMessage(MessageImportance.High, _stats.ToString());
36 
37             // Create the output parameters containing the files affected/not affected
38             List<ITaskItem> duplicates = new List<ITaskItem>();
39             List<ITaskItem> noDuplicates = new List<ITaskItem>();
40             foreach (ITaskItem sourceResxFile in SourceFiles)
41             {
42                 if (_distinctFileDuplicates.ContainsKey(sourceResxFile.ItemSpec))
43                     duplicates.Add(sourceResxFile);
44                 else
45                     noDuplicates.Add(sourceResxFile);
46             }
47             FilesWithDuplicates = duplicates.ToArray();
48             FilesWithNoDuplicates = noDuplicates.ToArray();
49 
50             // Create the Log file
51             if (Enum.IsDefined(typeof (LogOutput), LogType))
52             {
53                 Log.LogMessage(MessageImportance.Normal, @"Writing {0} log to {1}...", Enum.GetName(typeof (LogOutput), LogType), LogFile);
54 
55                 if ((LogOutput) LogType == LogOutput.Text)
56                     WriteTextLog(_stats, _duplicates);
57                 else if ((LogOutput) LogType == LogOutput.Xml)
58                     WriteXmlLog(_stats, _duplicates);
59                 else if ((LogOutput) LogType == LogOutput.Csv)
60                     WriteCsvLog(_duplicates);
61             }
62             else
63             {
64                 Log.LogError("Invalid LogType specified");
65                 result = false;
66             }
67 
68             return result;
69         }
70 
71         private void WriteCsvLog(IEnumerable<ResXDuplicate> duplicates)
72         {
73             StringBuilder sb = new StringBuilder();
74             sb.AppendLine("Shared,File,Name,Value");
75             foreach (var entry in duplicates.OrderByDescending(x => x.Count))
76             {
77                 sb.AppendFormat(entry.ToCsvText());
78             }
79 
80             WriteLogFile(sb, LogFile);
81         }
82         
83         ...

We receive the LogFile and LogType as input parameters to the MSBuild task and we have separate FilesWithDuplicates and FilesWithNoDuplicates output parameters that will contain the names of the files with duplicates and without duplicates. After calling into the base class (line 33) to scan the files and setup all the data structures, we simply setup the list of output files (lines 40-48) and also create the desired log output (lines 51-66).

Invoking the MSBuild Task

Now that we’ve covered the MSBuild task, let’s see how we would invoke this when compiling our projects within Visual Studio. As every .csproj in Visual Studio is a MSBuild file, invoking the task is actually quite easy. We will hook into the BeforeBuild target of the project containing our resource files to execute our ResXFindDuplicate task. We therefore edit the specific .csproj and start by adding the following to identify the set of resource files to scan:

 1   <PropertyGroup>
 2     <ResxDuplicatesFile>Resources\SharedResources.resx</ResxDuplicatesFile>
 3     <ResxFindDuplicatesLog>Resources\SharedResources.csv</ResxFindDuplicatesLog>
 4     <NEW_LINE>%0D%0A</NEW_LINE>
 5     <TAB>%09</TAB>
 6   </PropertyGroup>
 7   <ItemGroup>
 8     <ResxInputs Include="Resources\*.resx" Exclude="Resources\*.*.resx">
 9       <Visible>false</Visible>
10     </ResxInputs>
11     <ResxOutputs Include="$(ResxFindDuplicatesLog)">
12       <Visible>false</Visible>
13     </ResxOutputs>
14   </ItemGroup>
15

Notice that the ResxInputs ItemGroup (lines 8-10) is set to match only the resource neutral resource files by excluding any culture specific files. Also notice that we set the visibility of these item groups to false to prevent them from showing up in the Visual Studio solution explorer. We can now invoke our custom task by hooking it into the BeforeBuild target. Every .csproj has a BeforeBuild and AfterBuild target that Visual Studio will invoke before and after every time it compiles the project.

 1   <Import Project="..\..\Tools\MSBuild\Pragma.MSBuild.Tasks\Pragma.MSBuild.Tasks.Targets" />
 2   <Target Name="BeforeBuild" Inputs="@(ResxInputs)" Outputs="@(ResxOutputs)">
 3     <!-- Find the resx files containing duplicates -->
 4     <ResXFindDuplicates SourceFiles="@(ResxInputs)" LogType="2" LogFile="$(ResxFindDuplicatesLog)">
 5       <Output TaskParameter="FilesWithDuplicates" ItemName="ResxFilesWithDuplicates" />
 6       <Output TaskParameter="FilesWithNoDuplicates" ItemName="ResxFilesWithNoDuplicates" />
 7     </ResXFindDuplicates>
 8     <Message Text="Resx files with duplicates:$(NEW_LINE)$(TAB)@(ResxFilesWithDuplicates->'%(RecursiveDir)%(FileName)%(Extension)', '$(NEW_LINE)$(TAB)')" Importance="normal" />
 9     <Message Text="Resx files with no duplicates:$(NEW_LINE)$(TAB)@(ResxFilesWithNoDuplicates->'%(RecursiveDir)%(FileName)%(Extension)', '$(NEW_LINE)$(TAB)')" Importance="normal" />
10   </Target>

After importing our compiled MSBuild tasks, we simply invoke it as showed on lines 4-7 by specifying what kind of log file we want (2 = CSV) and also where the log should be created. We can investigate and mine the log output to further assist us with out translation efforts. Finally, here is the output that we will now receive in Visual Studio every time we compile the project containing our resource files:

Conclusion

Now that we know the amount of resource duplication we have across our resource files, we need a mechanism to try and eliminate as much of the duplication as possible. This we’ll cover in the next post of the series. Till then…

Managing .NET RESX Duplication - Part 1: Solution Requirements and Design

2011-04-17T16:33:00.001+02:00

Last time around I blogged a bit about the work that we are doing to add multi-language support to On Key v5.4, Pragma’s Enterprise Asset Management System (EAMS). As mentioned, we have 250+ resources files containing around 8000+ resource entries that need to be translated. After some further investigation, it turns out that we have close to 49% duplication across the resources files being used by our screens. This high level of duplication is mostly due to these files being code generated. So during the last week we’ve spend some time looking at various options for minimizing and managing the duplication and thus the amount of strings that we need to pass onto Translators to get translated.

I’m going to cover the solution we are thinking of using in a series of 3 blog posts. Here is a high level overview of the posts:

Solution Requirements and Design
Code: Identifying resource duplication
Code: Managing resource duplication

Let’s start by looking at our high level requirements for the solution.

Requirements

On a high level we identified the following requirements to guide us in creating a solution:

Continuously Execute - The solution must be continuously executed and not only once off as we will be adding more resources over time.
Minimize/Eliminate Rework - Minimize/eliminate rework on all the existing screens
Minimize/Eliminate Duplication – Minimize/eliminate the duplication in the resource files
Streamline Translation Effort – Prevent Translators from wasting time on translation duplicates more than once

I think it is important to note that we were looking for a solution that in our opinion strikes the best balance between all of these requirements. Take for example Requirements 2 and 3. Solving Requirement 3 in the best possible way may impact negatively on Requirement 2.

SOLUTION

We broke our solution up into two distinct phases. Both phases have been implemented as custom MSBuild tasks to allow it to run as part of the build process and thus satisfy the Continuously Execute requirement. In Phase One we identify the resources that are duplicated across all of the .RESX resource files. This is done continuously on every compilation of the source code in Visual Studio. The overhead is small - 3 seconds on my 3 year old laptop. The output of this phase is a report in either CSV/Xml format and a list of .RESX files containing duplicates. The data in the report can be mined by both the Development Team and Translators. The Development Team can identify areas of duplication that we need to address, inconsistencies in translations being applied across screens and much more. The Translators can use it to make sure they don’t translate values more than once and also for other general query purposes (e.g. what screens are using a value etc.)

The list of .RESX file containing the duplicates can then be fed into Phase Two that attempts to minimize the duplication. It works by copying the duplicates into a single, separate shared resource file. The idea being that instead of translating every duplicate, we only translate the entries in the shared resource file and re-use them across all the screens were they apply. This cuts down on about half of the duplicated strings for our screens. To make use of the shared resources, we created a custom Resource Manager to always first query the shared resource file when it looks for a resource. If the resource is found in the shared file, it is used from there and not from the screen specific file. We can therefore ignore the duplicate entries in the screen specific files and not translate them as they are automatically overridden by their counterparts from the shared resource file. This satisfies the Minimize/Eliminate Duplication requirement. We investigated the possibility of actually deleting these entries from the screen specific resource files, but this caused issues from a binding perspective in our XAML screens. So to satisfy the Minimize/Eliminate Rework requirement we decided to leave them intact. We do however update the comment in the resource files to indicate that an entry is being shared (using [## SHARED ##] tag).

The down side is that we are obviously still carry these strings around in all our language neutral resource files and thus in a translation solution like Amanuens. The Translators should however never have to translate the duplicate shared resource entries if they use the CSV report or look at the comment tags. This then satisfies our Streamline Translation Effort requirement.

As mentioned, the first phase is continuously executed as part of compiling our source code and requires no manual intervention. Phase 2 can also be executed as part of compiling the solution, but it does require you to first check out the shared .RESX file from TFS and setting a MSBuild variable so that the MSBuild task will execute. More on this later.

Stats

Here is a rundown of the stats generated after execution Phase 1 and 2 on all our existing screen resource files. We specifically only used the screen resources files as they account for the majority of duplication in our system.

Phase 1

Files Scanned: 270
Files With Duplicates: 263
File With No Duplicates: 7
Resources Read: 6458
Duplicate Resources Identified: 5025 resources
Duplicate Resources that can shared: 2514 resources identified by 428 keys
Duplicate Resources that cannot be shared: 2511 resources identified by 197 keys

Phase 2

Files Scanned: 263
Resources extracted to shared file: 2514 resources identified by 428 keys

An area of current concern for us is that we will have to go through a cleanup exercise on our own resource files before getting the Translators cracking. This is illustrated by the 2511 duplicate resources that we cannot currently extract as shared resources. To illustrate our problem let’s take the ‘Id’ key as an example. After mining the key using the CSV report, we discovered that we have 247 duplicate entries for the ‘Id’ key. Of these 247 duplicates, 242 use the same value ‘Id’ . The remaining 5 entries use the value ‘Parent Id’. Because of these differences we cannot safely extract ‘Id’ as a shared resource. The differences for these 5 specific screens may be valid based on the screen context. The differences may however also be invalid due to spelling mistakes, incorrect specs or even a wrong implementation from the developer. The Translator will however now be wasting time translating this over and over again

We therefore think it is prudent for us to first spend a bit of time going through our resources files to see what we can fix. The spelling mistakes and other errors should be easy to fix. The contextual differences will be more complex and will require looking at the specific screen to determine its validity. For scenarios like the ‘Id’ key above I think we should rather create a separate ‘ParentId’ key and update the 5 screens to use the ‘ParentId’ instead of ‘Id’ as a resource. This violates the Minimize/Eliminate Rework requirement but it does improve the Minimize/Eliminate Duplication and Streamline Translation Effort requirements.

Summary

In the next post I will look at some of the code for the MSBuild task that identifies the duplication across the resources files. Stay tuned!

Pragma On Key Silverlight Localization

2011-04-11T09:07:00.001+02:00

One of the big features that we are adding to Version 5.4 of On Key, Pragma’s Enterprise Asset Management System, is the ability to use the system in different languages. It seems like the first language we will support in addition to English will be Brazilian Portuguese to support customers for our Pragma Brazil service company. So I’ve been spending some time during the past week or two looking at the various aspects of On Key that we need to localize to determine the best solutions for getting everything localized. In this post I will cover some useful information I discovered in my research.

Static Text

Fortunately we’ve been following the recommended best practice for localizing Silverlight applications by storing all the text that needs to be localized in external resource files. We currently have around 290 resources files with around 8200 strings that need to be translated from English into other languages. The resources stored within these files are all static type resources like menu text, captions, tooltips, error messages etc. I was looking for a solution that would make the process of managing and translating these strings as easy as possible.

I’ve been keeping an eye on Amanuens since I first read about their localization solution back in 2010, so I headed off to their web-site to see how it has matured since the early beta phases. Amanuens gives you the ability to upload your resource files into the cloud to get translators to translate them into your language of choice. You can assign your own people as translators or even get a quote from an external organization to translate your resources into your language of choice. This is all done via an easy-to-use web-based interface. The output is the actual translated resource files in a variety of formats of which .NET resource files is one. Amanuens also has some other really nifty features like a shared Translation Memory, the ability to attach screenshots to provide context for translations and much more.

After creating a public facing Subversion repository containing our resource files, I signed up for a trial to experience the work flow involved in getting some of our resource files translated. I was quite happy with the usability of the system and it seems to be a great solution for translating our resources into multiple languages over a period of time. The change tracking capabilities of automatically picking up new strings or changes to existing strings to notify translators of work to be done is a really powerful feature and one that will really simplify the management of the resource files over time.

Dynamic Text

In addition to the static text, On Key also has a nifty feature called Phrase Translations. A phrase is defined as a piece of text delimited by the “{“ and “}” brackets. Any customer can create translations for their own phrases via the On Key UI to allow the engineers, artisans and other people using On Key to view certain terminology (like technical terms) in their language of preference. Take for example the phrase {Check} the {Bearing} as the description of a maintenance task to be executed. When an artisan views the description, the text will automatically be translated into their language preference specified at logon, i.e. Inspekteer the Draer if they specified Afrikaans. The technical implementation for these Phrase Translations is actually quite interesting as we utilize the SQLCLR to get the best performance possible, but that is a post for another day.

Dynamically loading Resources

One of the problems with Silverlight localization is that all the resources for the different languages are packaged into the single XAP file that is downloaded to the clients. Ideally we only want to download the resources for each client’s neutral and language specific cultures. One solution to the problem is to create culture specific XAP files for every language supported. When the initial request for the application hits your server, you inspect the client’s culture settings passed via the ASP.NET request and redirect the client to load the relevant culture specific XAP file. I wasn’t keen on creating separate build configurations for all the future languages that we will support and did some further research into the issue. I then came across this excellent video presentation by Guy Smith-Ferrier on Internationalizing Silverlight applications. In the presentation he covers a technique whereby he creates separate XAP files using a MSBuild task that contains only the localized resources and not the complete application. All of this happens without having to create separate build configurations in Visual Studio. He then further illustrates how to use MEF to dynamically download the correct culture specific XAP file at run-time when the Silverlight application starts.

This was more to my liking and after spending some time looking at the source code, I implemented something similar in our environment. A problem that Guy does not address in his solution is the loading of the culture neutral resources, i.e. it only loads the culture specific resources. As every CultureInfo instance has a reference to its Parent culture, I set about to create a solution whereby I traverse the culture tree upwards, trying to download a XAP file for every culture neutral Parent resource until I arrived at the Invariant Culture that is compiled into the original application XAP file. Here is some sample C# code that illustrates how to accomplish this:

 1   public App()
 2   {
 3     Startup += Application_Startup;
 4     Exit += Application_Exit;
 5     UnhandledException += Application_UnhandledException;
 6 
 7     InitializeComponent();
 8   }
 9 
10   private void Application_Startup(object sender, StartupEventArgs e)
11   {
12     var options = new NinjectSettings {InjectAttribute = typeof (InjectAttribute)};
13     IocContainer.Kernel = new StandardKernel(options, new IocInfrastructureModule(), new IocNavigationModule(), new IocServiceModule());
14 
15     DownloadLanguageResources(Thread.CurrentThread.CurrentUICulture);
16   }
17 
18   private void DownloadLanguageResources(CultureInfo culture)
19   {
20     string xapResourceFilename = String.Format(CultureInfo.InvariantCulture, "{0}.{1}.xap", "Pragma.OnKey5.Client", culture.Name);
21 
22     XapLoader xapLoader = new XapLoader(xapResourceFilename);
23     xapLoader.Completed += (s, args) =>
24     {
25       if (culture.Parent.IsNeutralCulture)
26       {
27         // Search for user's region neutral resources
28         DownloadLanguageResources(culture.Parent);
29       }
30       else
31       {
32         RootVisual = new Page();
33 
34         // Set the resource to use for all Telerik controls
35         LocalizationManager.DefaultResourceManager = Strings.ResourceManager;                            
36       }
37     };
38     xapLoader.DownloadAsync();
39   }

You will notice that I’m not using MEF to dynamically download the XAP files, but instead I’m using a custom XapLoader class. I had two reasons for not using MEF:

I quite new to MEF and I couldn’t figure out how to get MEF to dynamically download multiple DeploymentCatalog instances as I traverse the culture tree. If a reader knows how to do this, please respond in the comments.
I wasn’t interested in the whole MEF discovery mechanism to wire up modules etc. I only wanted to download the XAP file. This might change at a later stage when we start refactoring On Key into a more modular approach, but for now this was unnecessary overhead that I wanted to avoid.

Here is the source code for the XapLoader class. I cannot seem to find the original link to give the original author some credit, but the code is really self explanatory so I leave it to speak for itself:

 1   /// <summary>
 2   /// XAP file loader to dynamically download and load additional XAP resources at run-time
 3   /// </summary>
 4   [SuppressMessage("Microsoft.Naming", "CA1704:IdentifiersShouldBeSpelledCorrectly", MessageId = "Xap")]
 5   public class XapLoader
 6   {
 7     private readonly string _xapName;
 8 
 9     [SuppressMessage("Microsoft.Naming", "CA1704:IdentifiersShouldBeSpelledCorrectly", MessageId = "xap")]
10     public XapLoader(string xapName)
11     {
12         Check.ArgumentNotNullOrEmptyString(xapName, "xapName");
13         _xapName = xapName;
14     }
15 
16     public event EventHandler<XapLoadedEventArgs> Completed;
17 
18     #region Methods
19 
20     public void DownloadAsync()
21     {
22         Uri uri = new Uri(_xapName, UriKind.Relative);
23         WebClient wc = new WebClient();
24         wc.OpenReadCompleted += OnXapLoadingResponse;
25         wc.OpenReadAsync(uri);
26     }
27 
28     private void InitXap(Stream stream)
29     {
30         StreamResourceInfo xapStreamInfo = new StreamResourceInfo(stream, null);
31         string appManifest = new StreamReader(System.Windows.Application.GetResourceStream(
32             xapStreamInfo, 
33             new Uri("AppManifest.xaml", UriKind.Relative)).Stream).ReadToEnd();
34 
35         XElement deploy = XDocument.Parse(appManifest).Root;
36 
37         List<XElement> parts = (from assemblyParts in deploy.Elements().Elements()
38                                 select assemblyParts).ToList();
39 
40         foreach (XElement xe in parts)
41         {
42             string source = xe.Attribute("Source").Value;
43             AssemblyPart asmPart = new AssemblyPart();
44             StreamResourceInfo streamInfo = System.Windows.Application.GetResourceStream(
45                 xapStreamInfo,
46                 new Uri(source, UriKind.Relative));
47             asmPart.Load(streamInfo.Stream);
48         }
49     }
50 
51     #endregion
52 
53     #region Event Handlers
54 
55     private void OnXapLoadingResponse(object sender, OpenReadCompletedEventArgs e)
56     {
57         if ((e.Error == null) && (e.Cancelled == false))
58             InitXap(e.Result);
59 
60         if (Completed != null)
61         {
62             XapLoadedEventArgs args = new XapLoadedEventArgs();
63             args.Error = e.Error;
64             args.Cancelled = e.Cancelled;
65             Completed(this, args);
66         }
67     }
68 
69     #endregion
70   }
71 
72   [SuppressMessage("Microsoft.Naming", "CA1704:IdentifiersShouldBeSpelledCorrectly", MessageId = "Xap")]
73   public class XapLoadedEventArgs : EventArgs
74   {
75     #region Properties
76 
77     public bool Cancelled { get; set; }
78     public Exception Error { get; set; }
79 
80     #endregion
81   }

Here is an example of the code in action when I start up my browser with Brazilian Portuguese as my preferred browser language:

Notice how the original XAP file is downloaded and also the two subsequent request for the language specific and language neutral XAP files. The language specific XAP file (pt-BR.xap) was not found, but the language neutral XAP file for Portuguese was found and the UI is therefore displayed in Portuguese using these neutral resources.

Conclusion

There are obviously a lot of other dimensions to consider when localizing your Silverlight application like right-to-left support, different time zones and much more. Guy Smith-Ferrier covers a lot of this in his presentation so I highly recommend watching the video and/or downloading the slides. I might follow up with another post on some of these topics at a later point in time, but for now this is about all I have to say. Lekker lees

Pragma Software Development: Check-in Procedure

2011-04-04T09:44:00.001+02:00

When working together in a software development team it is important to have a common definition for when something is DONE. An important aspect of getting things DONE is a well-defined check-in procedure that all developers follow to commit something into the source control repository. Here is the procedure that we use at Pragma for committing changes into our Team Foundation Source Control repository:

Code has been integrated with a recent version of the code in the TFS repository – at least a version of the TFS repository on the same day.
Code has been formatted according to our ReSharper Pragma Full Cleanup profile. This automatically formats all code to follow the same formatting and layout standards.
Code compiles without any compiler warnings.
Code adheres to our coding standards - I’ll write more about this in a future post. We try to automate the application of our coding standards (where possible) using tools like ReSharper. Alternatively, use a code review checklist to quickly identify areas of concern.
No ReSharper code analysis Error or Warning violations (i.e. green square in scrollbar). Any violation needs to be identified and agreed upon. Suggestions and Hints can be ignored if deemed unnecessary.
No FxCop violations. Any violation needs to be identified, agreed upon and justified using the in-code SuppressMessage attribute.
New functionality has been covered by tests and the code coverage has been verified using NCover.
Bugs have been covered by tests and verified using NCover.
Code adherers to the Logging strategy and the developer has inspected the usefulness of the log statements using Log4View.
Code adheres to the Exception handling strategy for managing exceptions.
Error Messages and translations have been added to the resource files.
The associated TFS work item (development ticket/task) has been updated to reflect the effort involved (actual hours, comments etc.)

Once the code satisfies the above mentioned criteria, the developer runs a gated check-in using the TeamCity Visual Studio Plug-in. This ensures that all the unit and integration tests are executed by the TeamCity build server after integrating the local changes with the latest TFS repository changeset. Only if all the tests pass will TeamCity commit the changes onto the Mainline. The observant reader may have noticed that we don’t do an upfront code review. As we work from home for 2-3 days a week, we conduct our code reviews on the checked-in artefacts. I’ll write more about the code review procedure in a future blog posting.

Lastly, for service pack development, the developer needs to consider whether the change needs to be merged across to the Mainline for the next major version and merge the changes across if required.

Pragma Software Development Tools

2011-02-05T08:24:00.001+02:00

Last time around I mentioned I would blog about the software development tools we are using. They say a picture is worth a thousand words, so with time to blog being a bit short in the week running up to the final release of On Key, here’s a mind map showing the different software development tools we are using in the different software development disciplines! Btw, if anybody knows about a decent Live Writer plug-in to insert a thumbnail, be sure to leave me a comment.

Pragma Software Development Team

2011-01-20T09:43:00.001+02:00

Following on my intention to blog a bit about our internal development environment here at PRAGMA, let’s start with a look at the software development team, how we communicate, where we work and what Software Development Lifecycle we follow.

Team

The software development team has grown quite a bit during the past 2 years. Currently we have a Software Development Manager who takes the responsibility of managing the project, prioritizing the feature set, interacting with clients, ensuring the product vision etc. Reporting directly to the Software Development Manager is a Development, Test and Analyst lead. They are responsible for managing the 13 developers, 3 testers and 1 analyst that compromise the rest of the team. The management function forms only a small part of their responsibility as they are all hands-on, doing full-time development, testing and analysis together with the rest of the team. We believe in a flat structure and in the team taking collective ownership for the solution. As Development Lead, I share the overall responsibility for the architecture with another senior developer in the team. We also have a Project Manager/Assistant who helps with administrating the project plan and also currently plays the role of SCRUM Master.

Communication

One of the benefits of working at PRAGMA is that we get to work from home for 2 days a week. Initially I found it very hard to adapt to not having the whole team co-located. Especially in the early stages of the project, where I had to do a lot of mentoring, I missed the simple “luxury” of pairing with a developer to illustrate some concepts or to fledge out a design on a whiteboard or through some code. As the project progressed and things settled in terms of architecture and skills the issue became less of a problem. I still feel a co-located team is the most productive setup, but there is also a lot of merit in allowing people to work remotely for a day or two in the week by giving them the flexibility of not having to commute into office every day.

When working remotely keeping in touch is obviously very important. We use Skype for IM and VOIP calls to interact between individuals and small groups of people. I specifically mention small groups as we struggled to use Skype for calls that involved the whole team. The call quality was not good and people would frequently drop and had to reconnect to the conference call. This might be due to bad connections from the different ISP’s that the remote workers are using so your own mileage might differ. Fortunately for us PRAGMA makes use of Interwise Participant for web conferencing so our daily feedback session currently involves a combination of using Interwise for visuals and PRAGMA’s normal teleconference facilities for audio. This obviously incurs a bit of cost for the remote workers but it doesn’t nearly weigh up against the other benefits like not having to commute. As the broadband scenario in South Africa is getting better and better we re-evaluate this setup every now and then to see whether we can’t get away with only using Skype. For simple remote pairing/debugging sessions we find Microsoft SharedView to be an excellent solution.

Office Space

Our offices are located in the Bellville Business Park in the Northern Suburbs of Cape Town so fortunately we miss out on all city traffic. We moved into our current premises end of November 2009 and the new, modern offices are equipped with a very nice canteen and other facilities that make working at the office a pleasure. Of course there is some good, free coffee as well The seating inside the building is based on an open plan setup. The software development team sit together in what is refer to as the Lab with each person having his/her own small cubicle. This is a picture of my cubicle in the Lab.

The cubicle barriers are about 1.6 m high and allow you to work without being distracted when somebody walks by your desk. This setup works quite nicely although there are some days that I feel the cubicle barriers should rather be taken down to create a feeling of more openness and to “increase communication” in the team.

We have whiteboards against all the walls for design discussions and two small adjacent meeting rooms to the Lab for ad-hoc design discussions and meetings. We also have our build monitors setup against a wall in the Lab that shows our TeamCity builds and other dashboard related statistics relevant to our ongoing development efforts.

We will hopefully be replacing the two smallish monitors with one big monitor during this year to make it easier to read as we also use these monitors as an electronic SCRUM task board during our daily at-the-office SCRUM stand-up sessions.

SDLC

Talking about SCRUM, one of my first responsibilities when I joined the team way back in July 2008 was to look at adopting a more formal software development lifecycle. I’m a firm believer in an agile development methodology and all the disciplines involved with doing agile development so we adopted SCRUM. We went through quite a few Sprints trying out the different aspects of SCRUM to find out what works best in our environment. After starting with 2 week Sprints we eventually settled on 3 weeks as our ideal Sprint length. We tried Planning Poker for a few Sprints but eventually dropped this in favour of just estimating the tasks for a Sprint in Ideal hours. We had some big debates about using Story Points versus Ideal Hours and how we should track our Velocity. We took quite a long time to get a Prioritized Product Backlog in place and also struggled to map the backlog back into a project plan that was required due to immense pressure from a big international client. We eventually got into some kind of SCRUM rhythm, but we still have quite a few areas where we want to improve on going forward.

For a start I think we need to break the team into smaller groups that focus on specific functional areas of On Key. We can then use a Scrum of Scrums meeting to discuss the areas of overlap and integration. We also need to get better at breaking down the functionality in coarser tasks (or work items). I feel we are breaking down our tasks into too fine grained work items and this causes unnecessary management overhead and also totally clutters the task board with too much detail. I think a lot of this is as a result of us rewriting and expanding on an existing system, i.e. we have a very good idea of what is required, but I think we must still try and manage the work at a higher level. This might ease the pain of integrating back into the project plan as well. Having said all of this, we have mastered a lot of the technical disciplines (like continuous integration, TDD, automated acceptance testing etc.) required for agile development in place and we do succeed at delivering small increments of working functionality in our 3 week Sprint cycle.

Sprint

So how does a typical 3-week Sprint look taking into account that we work remotely for about half of the time? The following 3 week roster gives a quick summary of the high-level activities involved:

Mon	Tue	Wed	Thu	Fri
At the office Individual Planning and Design Discussions	At the office Daily Scrum @ 10:00 Daily Build @ 22:00	Working remotely Daily Scrum @ 10:00 Daily Build @ 22:00	At the office Daily Scrum @ 10:00 Daily Build @ 22:00	Working remotely Daily Scrum @ 10:00 Daily Build @ 22:00
Working Remotely Daily Scrum @ 10:00 Daily Build @ 22:00	At the office Daily Scrum @ 10:00 Daily Build @ 22:00	Working remotely Daily Scrum @ 10:00 Daily Build @ 22:00	At the office Daily Scrum @ 10:00 Daily Build @ 22:00	Working remotely Daily Scrum @ 10:00 Daily Build @ 22:00
Working Remotely Daily Scrum @ 10:00 Daily Build @ 22:00	At the office Daily Scrum @ 10:00 Daily Build @ 22:00	Working remotely Daily Scrum @ 10:00 Daily Build @ 22:00	At the office Daily Scrum @ 10:00 Sprint Build @ 22:00	At the office Retrospective @ 09:00 High level Planning @ 10:00 Demo @ 13:00 Design, Training @ 14:00 Drinks @ 16:00

Well, I think that’s enough for now about the overall work environment. In my next post I’ll look at the Development Tools that we are using.

Pragma On Key Software Development Environment

2011-01-14T22:33:00.001+02:00

As I mentioned towards the end of last year, we delivered an important Release 3 milestone of Pragma On Key V5. We are currently working hard to further improve the application performance and to fix the remaining bugs with the aim of going live with a big international customer 1 March 2011. For those unfamiliar with PRAGMA and On Key [Source]:

PRAGMA is a global engineering company providing physical asset management improvement services and products to our clients. We believe that the most important value derived from our service is peace of mind. A client's CEO and his production or service team can focus on delivering their own client promise, whilst we, along with the in-house maintenance team, take care of optimising the performance and longevity of their assets over the life cycle of those assets.

Pragma On Key is our homegrown Enterprise Asset Management System (EAMS). Apart from the traditional functionalities generally expected from an EAMS, it has a variety of sophisticated modules that allow the client virtual and real-time access to asset information.

We are based in the beautiful Cape Town, South Africa and we have customers across a wide spectrum of industries in South Africa and also overseas.

We currently support two major versions of On Key:

Version 4 – A Windows forms client/server application that runs on top of SQL Server as DBMS. The application was developed using Delphi and has been in production for quite a few years already.
Version 5 – A new web-based solution architected as a RIA application. It uses Silverlight as presentation technology and a layered server architecture using .NET 4.0 on top of SQL Server as DBMS.

Both versions support interfacing with other ERP systems like SYSPRO and SAP via custom interfaces that we build. In addition to rewriting a lot of the V4 features onto V5, we also added some really nifty new features to provide our customers with great flexibility in managing their physical assets. In V5 we also started creating a SDK that allows third parties to build additional services on top of the core On Key application platform.

Besides these cool features in On Key itself, we have also IMO come along way in creating a solid software development environment within PRAGMA. The team closed to doubled in size during the last 3 years (we are now 22 people) and we simply had to mature our SDLC practices. There are of course still a lot of areas where we can and will improve on, but I think we have a good foundation to build on for the future.

I want to start a series of blog posts where I write a bit about our software development environment - reflecting on the work environment, what tools and development practices we use and also areas that we want to improve on in future. I also want to spend some time writing about the design of some of the core features within On Key. I don’t quite know how all of this is going to pan out and how many posts I am going to create, but off the cuff, here is a short list of areas that I want to cover:

Work environment – Team Composition, Office Space, SDLC (Scrum), Typical Sprint…
Development Tools – Source Control, Build Server, Wiki, Work Item Management…
Development Practices – Check-in procedure, Code Reviews, Coding Standards, Code Analysis (Static/Dynamic), Code Generation, TDD…
Build Automation – Versioning, Continuous Integration, Continuous Performance Testing, Build Pipelines…
Testing – Automated Functional Regression Testing…
On Key Architecture – Client Architecture, Server Architecture, Localization, Exception Handling, Logging, Security…
On Key Features – Rule Engine, Validation Framework, Background Tasks, Rollout, Synchronization…

This list is quite long! Time permitting, I’ll try and keep posting on some topic hopefully once a week. Till later…

My 2011 Software Development Reading List

2011-01-03T16:23:00.001+02:00

A wonderful 2011 for all! I’ve not made many New Year resolutions, but one resolution that I do want to try and keep is to work through my library of ever growing IT books that have been gathering some dust on my desk and hard-drive. In stead of spending time reading through blogs, I want to rather concentrate on broadening/sharpening my skills through working through some of my books. I’ve got quite a few good books where I’ve only read some part of the book and I also want to complete these as well. So without further ado, here is my list of IT books for 2011 that I want to read:

[Partially Read] Release It!: Michael T. Nygard - Excellent advise on getting and keeping your software running in production.
[Partially Read] Architecting Applications for the Enterprise: Dino Esposito, Andrea Saltarello – A great resource for all budding architects and a book that I want to re-read to evaluate our current architecture against some of the best practices mentioned in here.
[Partially Read] Continuous Delivery: Jez Humble, David Farley – Excellent advise on using build, test and deployment automation to manage your software releases. We’ve got quite a good deployment environment at work as I’m a big fan of continuous integration, but it is always nice to read some further insights from experts to see what areas we can still improve on.
The Art of Application Performance Monitoring: Ian Molyneaux – Performance monitoring is high on the agenda for our Enterprise Asset Management System (EAMS).
Algorithms In A Nutshell: George Heineman, Gary Pollice & Stanley Selkow – Time to learn and brush up on the Graph, Search, Path finding and Network Flow algorithms. I’ve got a sneaky suspicion that we are missing out on some opportunities to use better algorithms within our EAMS.
Pragmatic Thinking and Learning: Andy Hunt – Time to figure out the best way to think about solving problems.
Growing Software – Proven Strategies for Managing Software Engineers: Louis Testa
Programming Ruby 1.9 (3rd Edition): Dave Thomas – Lots of people are raving about Ruby and I want to start learning why.
The Ruby Programming Language: David Flanagan & Yukihiro Masumoto

Happy reading :-)

Pragma On Key – Release 3 Milestone Reached

2010-11-30T16:25:00.001+02:00

I’m watching the final TeamCity daily build as I type this and I’m super excited about the next major milestone that we’ve accomplished for Pragma On Key, our company’s Enterprise Asset Management System (EAMS). It’s been 30 months of hard work to re-write and extend the system using Silverlight and .NET. We started off as early adopters with the Silverlight Beta’s and .NET 3.5 and with the latest build we are using Silverlight 3 and .NET 4.0 as the technology platform. We evaluated Silverlight 4 but due to some unresolved memory issues in the framework we decided to stick with Silverlight 3 for now. We hope to be upgrading to Silverlight 4 soon here after.

I’m very proud of the way the team pulled through all the difficult times to end up with what I feel is a solid platform to build on for the future. As always there is still a lot of things we can and will improve on going forward, but I think we have come along way since the start of the project. From adopting a new SDLC (SCRUM) to learning a complete new technology skill set (C#; Silverlight, TDD, CI etc.) and in the process close to doubling in team size. Currently we have 2 Analysts, 5 Testers, 13 Developers (me included) and a Software Development Manager. I can honestly say that within the 2.5 years we only had one or two isolated incidents with the team dynamics. I think this is a great testimony to the quality of the people working here at Pragma.

We are now entering the final phases of testing the new release of On Key at a big international client who is planning to go live with the system end of Feb 2011. So after more than 20 000 CI builds, 230 tested builds, 26 820 commits and with just more than 9000 unit and integration tests it is great to finally see the new version of On Key coming of age! We’ve got some real nifty features included in this release and I think customers will enjoy the new web based interface as well. I hope to be blogging a bit more about this and our internal development environment in the future. Well done to all involved!

TransactionScope Fluent Interface

2010-11-21T18:29:00.001+02:00

During the past 2 weeks I’ve been spending a lot of time investigating the performance of Pragma On Key, our company’s Enterprise Asset Management System (EAMS). Part of the investigation was to resolve some issues we were encountering with deadlocks occurring for some of the longer running transactions. We are using the TransactionScope class added in .NET 2.0 to mark blocks of code in our Application Controllers as participating in a transaction. Out of the box, the default IsolationLevel used by new a TransactionScope instance is Serializable. This ensures the best data integrity by preventing dirty reads, nonrepeatable reads and phantom rows. However, all of this comes at the cost of concurrency as the range lock acquired by the DBMS prevents other transactions of updating or deleting rows in the range. So I set off to carefully consider the locking requirements for the different transactions to make sure we are acquiring the right level of locking for the different transactions.

I’ve always found the different settings of the TransactionScope class to be rather confusing. I always need to remind myself what the different items of the IsolationLevel and TransactionScopeOption enumerations actually mean. There is also the tricky scenario where, when passing in a new TransactionOptions instance to set a different IsolationLevel, you need to remember to set the transaction Timeout value if you are using a custom Timeout setting in your web.config file. So I decided to create a TransactionScopeBuilder class with a fluent interface to construct new TransactionScope instances with the idea to try and hide the complexity and also provide a more intent revealing interface of what the actual TransactionScope will do. I’ll start by showing some examples usages of the builder and then conclude with the source code for the builder itself.

The first call on the builder is a method to indicate the transactional behaviour of the scope through using the TransactionScopeOption enumeration. To start a new transaction:

1 // RequiresNew
2 TransactionScope scope = TransactionScopeBuilder.StartNew();            
3 // Required
4 TransactionScope scope = TransactionScopeBuilder.StartOrJoinAmbient();  
5 // Suppress
6 TransactionScope scope = TransactionScopeBuilder.IgnoreAmbient();

The next step is to then set the TransactionOptions. To set the Isolation Level, call any of the following methods on the builder:

1 // IsolationLevel.ReadUncommitted
2 TransactionScope scope = TransactionScopeBuilder.StartNew().AllowDirtyReads();          
3 // IsolationLevel.ReadCommitted
4 TransactionScope scope = TransactionScopeBuilder.StartNew().PreventDirtyReads();          
5 // IsolationLevel.RepeatableRead
6 TransactionScope scope = TransactionScopeBuilder.StartNew().PreventNonRepeatableReads();  
7 // IsolationLevel.Serializable
8 TransactionScope scope = TransactionScopeBuilder.StartNew().PreventPhantomReads();

Having the method indicate what scenarios it will allow/prevent makes for easier reading IMO. Lastly we can set the timeout associated with the transaction by calling any of the following methods on the builder:

1 TransactionScope scope = TransactionScopeBuilder.StartNew().RunsFor(TimeSpan.FromMinutes(3));
2 TransactionScope scope = TransactionScopeBuilder.StartNew().RunsForMinutes(3);
3 TransactionScope scope = TransactionScopeBuilder.StartNew().RunsForSeconds(180);

By defining an implicit cast operator from my TransactionScopeBuilder to a TransactionScope, I can complete the construction process without having to do any additional method calls as illustrated above. However, for scenarios where you need don’t want to assign the constructed instance to a variable (like your integration tests where you simply want to always rollback), you can complete the construction process by calling the Instance method as shown below:

1 using (TransactionScopeBuilder.StartNew().PreventPhantomReads().Instance)

Putting all of this together, let’s look at the syntax for creating a nested TransactionScope instance:

 1 using (TransactionScopeBuilder.StartNew().PreventPhantomReads().Instance)
 2 {
 3   using (TransactionScope scope = TransactionScopeBuilder.StartOrJoinAmbient()
 4                                                          .PreventDirtyReads()
 5                                                          .RunsFor(TimeSpan.FromMinutes(3)))
 6   {
 7       Assert.That(scope, Is.Not.Null);
 8       Assert.That(Transaction.Current.IsolationLevel, Is.EqualTo(IsolationLevel.Serializable));
 9   }
10 }

Some other benefits I get from encapsulating the construction logic in the builder are:

Ability to add logging to show when new instances are created and what the settings for the new transaction are.
Ability to enforce my own default settings for constructing new instances through reading values from configuration etc.
Ability to auto-escalate the nested transaction to the IsolationLevel of the ambient transaction. As all transactions participating in the same ambient transaction needs to use the same IsolationLevel, this is especially helpful in some scenarios where the scope might differ based on the context in the which the nested transaction is being created.

The code is pretty self-explanatory, so instead of going through it line-by-line, it just include it for you to have a look at:

  1 /// <summary>
  2 ///   Builder Pattern to create configured <see cref = "TransactionScope" /> instances.
  3 /// </summary>
  4 public sealed class TransactionScopeBuilder
  5 {
  6   private static ILogger _logger;
  7   private static IConfigRegistry _config = ConfigRegistry.Instance;
  8 
  9   private readonly bool _autoEscalateToAmbientIsolationLevel;
 10   private IsolationLevel _isolationLevel;
 11   private readonly bool _joinExisting;
 12   private readonly TransactionScopeOption _scopeOptions;
 13   private TimeSpan _timeout;
 14 
 15   private TransactionScopeBuilder(TransactionScopeOption scopeOptions, bool autoEscalateToAmbientIsolationLevel)
 16   {
 17       Check.ArgumentNotNull(scopeOptions, "scopeOptions");
 18 
 19       _scopeOptions = scopeOptions;
 20       _joinExisting = scopeOptions == TransactionScopeOption.Required;
 21       _autoEscalateToAmbientIsolationLevel = autoEscalateToAmbientIsolationLevel;
 22       _isolationLevel = IsolationLevel.ReadCommitted;
 23       _timeout = Config.DatabaseTransactionTimeout;
 24   }
 25 
 26   #region Properties
 27 
 28   /// <summary>
 29   ///   Constructs the <see cref = "TransactionScope" /> instance.
 30   /// </summary>
 31   /// <returns></returns>
 32   public TransactionScope Instance
 33   {
 34       get { return Build(this); }
 35   }
 36 
 37   /// <summary>
 38   ///   Gets or sets the logger.
 39   /// </summary>
 40   public static ILogger Logger
 41   {
 42       get
 43       {
 44           if (_logger == null)
 45           {
 46               _logger = LoggerFactory.GetLogger(typeof(TransactionScopeBuilder));
 47           }
 48 
 49           return _logger;
 50       }
 51       set { _logger = value; }
 52   }
 53 
 54   /// <summary>
 55   ///   Gets or sets the configuration registry.
 56   /// </summary>
 57   /// <summary>
 58   ///   Gets or sets the configuration registry.
 59   /// </summary>
 60   public static IConfigRegistry Config
 61   {
 62       get { return _config; }
 63       set { _config = value; }
 64   }
 65 
 66   private IsolationLevel IsolationLevel
 67   {
 68       get { return _isolationLevel; }
 69   }
 70 
 71   private TransactionScopeOption ScopeOptions
 72   {
 73       get { return _scopeOptions; }
 74   }
 75 
 76   private TimeSpan Timeout
 77   {
 78       get { return _timeout; }
 79   }
 80 
 81   #endregion
 82 
 83   #region Methods
 84 
 85   /// <summary>
 86   ///   Creates a new <see cref = "TransactionScope" /> that ignores the ambient
 87   ///   transaction through using <see cref = "TransactionScopeOption.Suppress" /> option.
 88   /// </summary>
 89   public static TransactionScopeBuilder IgnoreAmbient()
 90   {
 91       return new TransactionScopeBuilder(TransactionScopeOption.Suppress, false);
 92   }
 93 
 94   /// <summary>
 95   ///   Creates a new <see cref = "TransactionScope" /> that starts a new ambient
 96   ///   transaction through using <see cref = "TransactionScopeOption.RequiresNew" /> option.
 97   /// </summary>
 98   public static TransactionScopeBuilder StartNew()
 99   {
100       return new TransactionScopeBuilder(TransactionScopeOption.RequiresNew, false);
101   }
102 
103   /// <summary>
104   ///   Creates a new <see cref = "TransactionScope" /> that joins the existing ambient
105   ///   transaction if it exists or alternatively starts a new one through using 
106   ///   <see cref = "TransactionScopeOption.Required" /> option.
107   /// </summary>
108   /// <param name = "autoEscalateToAmbientIsolationLevel">Flag to indicate whether to automatically use
109   ///   the same <see cref = "IsolationLevel" /> as the existing ambient transaction.</param>
110   /// <exception cref="InvalidOperationException">
111   /// Thrown for auto escalation when the isolation level requested by nested transaction is 
112   /// finer grained than the existing ambient transaction's isolation level.
113   /// </exception>
114   public static TransactionScopeBuilder StartOrJoinAmbient(bool autoEscalateToAmbientIsolationLevel = true)
115   {
116       return new TransactionScopeBuilder(TransactionScopeOption.Required, autoEscalateToAmbientIsolationLevel);
117   }
118 
119   /// <summary>
120   ///   Allow dirty reads using <see cref = "System.Transactions.IsolationLevel.ReadUncommitted" />
121   /// </summary>
122   public TransactionScopeBuilder AllowDirtyReads()
123   {
124       return SetIsolationLevel(IsolationLevel.ReadUncommitted);
125   }
126 
127   /// <summary>
128   ///   Prevent dirty reads using <see cref = "System.Transactions.IsolationLevel.ReadCommitted" />
129   /// </summary>
130   public TransactionScopeBuilder PreventDirtyReads()
131   {
132       return SetIsolationLevel(IsolationLevel.ReadCommitted);
133   }
134 
135   /// <summary>
136   ///   Prevent non-repeatable reads using <see cref = "System.Transactions.IsolationLevel.RepeatableRead" />
137   /// </summary>
138   public TransactionScopeBuilder PreventNonRepeatableReads()
139   {
140       return SetIsolationLevel(IsolationLevel.RepeatableRead);
141   }
142 
143   /// <summary>
144   ///   Prevent phantom reads using <see cref = "System.Transactions.IsolationLevel.Serializable" />
145   /// </summary>
146   public TransactionScopeBuilder PreventPhantomReads()
147   {
148       return SetIsolationLevel(IsolationLevel.Serializable);
149   }
150 
151   /// <summary>
152   ///   Sets the timeout for the transaction in minutes.
153   /// </summary>
154   /// <param name = "minutes">The minutes to run before timeout.</param>
155   public TransactionScopeBuilder RunsForMinutes(int minutes)
156   {
157       return RunsFor(TimeSpan.FromMinutes(minutes));
158   }
159 
160   /// <summary>
161   ///   Sets the timeout for the transaction in seconds.
162   /// </summary>
163   /// <param name = "seconds">The seconds to run before timeout.</param>
164   public TransactionScopeBuilder RunsForSeconds(int seconds)
165   {
166       return RunsFor(TimeSpan.FromSeconds(seconds));
167   }
168 
169   /// <summary>
170   ///   Sets the timeout for the transaction.
171   /// </summary>
172   /// <param name = "timeout">The timeout.</param>
173   public TransactionScopeBuilder RunsFor(TimeSpan timeout)
174   {
175       _timeout = timeout;
176       return this;
177   }
178 
179   private static TransactionScope Build(TransactionScopeBuilder scopeBuilder)
180   {
181       if (Logger.IsDebugEnabled)
182       {
183           if (scopeBuilder.ScopeOptions == TransactionScopeOption.Required && Transaction.Current != null)
184           {
185               Logger.Debug("Creating nested transaction (Scope={0};Isolation Level={1};Timeout={2}).", scopeBuilder.ScopeOptions, scopeBuilder.IsolationLevel, scopeBuilder.Timeout);                    
186           }
187           else
188           {
189               Logger.Debug("Creating new transaction (Scope={0};Isolation Level={1};Timeout={2}).", scopeBuilder.ScopeOptions, scopeBuilder.IsolationLevel, scopeBuilder.Timeout);
190           }
191       }                
192 
193       return new TransactionScope(scopeBuilder.ScopeOptions, new TransactionOptions
194                                                               {
195                                                                   IsolationLevel = scopeBuilder.IsolationLevel,
196                                                                   Timeout = scopeBuilder.Timeout
197                                                               });
198   }
199 
200   private TransactionScopeBuilder SetIsolationLevel(IsolationLevel requiredLevel)
201   {
202       IsolationLevel acceptedLevel = requiredLevel;
203       if (_joinExisting && _autoEscalateToAmbientIsolationLevel && Transaction.Current != null)
204       {
205           // Can only escalate to ambient level if the required level is a less-fine grained lock
206           // IsolationLevel Enum values: Serializable = 0; RepeatableRead = 1; ReadCommitted = 2; ReadUncommitted = 3
207           if (requiredLevel >= Transaction.Current.IsolationLevel)
208           {
209               acceptedLevel = Transaction.Current.IsolationLevel;
210               if (Logger.IsDebugEnabled && requiredLevel > Transaction.Current.IsolationLevel)
211               {
212                   Logger.Debug("Building a nested transaction (Isolation Level={0}) that will auto escalate to the Ambient transaction (Isolation Level={1})", requiredLevel, acceptedLevel);
213               }
214           }
215           else
216           {
217               throw new ArgumentException(string.Format(CultureInfo.InvariantCulture, "TransactionScope could not be auto escalated. Requested level = {0}, Ambient Level = {1}", requiredLevel, Transaction.Current.IsolationLevel));
218           }
219       }
220       _isolationLevel = acceptedLevel;
221       return this;
222   }
223 
224   #endregion
225 
226   /// <summary>
227   ///   Performs an implicit conversion from 
228   ///   <see cref = "Pragma.OnKey.Core.TransactionScopeBuilder" /> to 
229   ///   <see cref = "System.Transactions.TransactionScope" /> by building the actual instance.
230   /// </summary>
231   /// <param name = "scopeBuilder">The scope builder.</param>
232   /// <returns>The result of the conversion.</returns>
233   public static implicit operator TransactionScope(TransactionScopeBuilder scopeBuilder)
234   {
235       return Build(scopeBuilder);
236   }
237 }
238

Hope you find this useful for helping you manage your own TransactionScope instances!

Understanding MSIL Memory Leaks

2010-09-30T21:23:00.001+02:00

We are using a custom RuntimeDataBuilder class that dynamically constructs a .NET type in-memory using Reflection.Emit to create simple property getters/setters for the information that we receive from a generic WCF Data Service. The service exposes a "DataSet like" structure consisting out of 1..m DataTableDefinition classes each containing 1..m DataTableColumnDefinition classes. When the information is received client side, we generate the dynamic type with its property setters/getters to improve the performance and facilitate binding on our Silverlight client. All of this works fine.

I’m currently trying to figure out how the memory management for these custom dynamic types work. I’m thinking that we might be causing a possible memory leak if we regenerate the type. Reason why I am contemplating this is that the user can change the query parameters which may then result in more/less information coming across the wire. It therefore invalidates the previous type that I created and I want to make sure that I’m are able to free up the memory used by this previous type definition as this can happen quite frequently. From this article on MSDN we gather that if you are using Light Weight Code Generation (LCG) the code is allocated on the managed heap which will be reclaimed by the GC when there is nothing holding a reference to it. But LCG only seems to apply to dynamic methods. My concern is for the Type with all its property getters/setters that is now not required anymore. If this is allocated on the unmanaged heap our only hope for reclaiming the memory seems to be to make sure that the type is loaded into a temporary AppDomain that we can unload when it is not required anymore. This opens up a whole new can of worms for inter AppDomain communication which I would rather avoid.

So my question is any of my readers can shed some more light on this topic. Is my assumption correct or is there another way of reclaiming the memory? I also posted the question to StackOverflow if you are interested in checking out the responses over there. Thx :-)

Ultimate .NET Development Tools 2010 Edition

2010-09-01T07:48:00.001+02:00

Here is my updated 2010 list of development tools that I prefer to use when doing .NET development. I specifically do not include any third party control/report libraries.

Tools

* = free/open source

[IDE] Visual Studio 2010 Premium Edition
[IDE] ReSharper for refactoring, unit test runner and so much more
[IDE] CodeSmith for generating code. Also consider T4 with Clarius’s Visual T4 Editor.
[IDE]* GhostDoc for inserting xml code comments
[IDE] Altova Xml Suite for any xml related work. XmlPad is the best, free alternative I know of.
[DBMS] SqlServer 2008 for DBMS
[SCM]* Subversion for source control
[SCM]* TortoiseSVN as windows shell extension for Subversion
[SCM] VisualSVN for integration of TortoiseSVN into VS. AnkhSVN is the best, free alternative I know of.
[SCM]* KDiff3 for merging
[TDD]* NUnit as preferred xUnit testing framework
[TDD]* moq as mock framework.
[TDD] NCover for code coverage stats
[CI]* TeamCity as build server
[CI]* MSBuild Extension Pack for additional MSBuild tasks.
[FR]* log4net as logging framework. Also see Log4View for an excellent UI for the log files.
[FR]* ANTLR and ANTLRWorks for creating custom DSL’s.
[FR] PostSharp as Aspect Oriented Programming framework
[FR]* Ninject as IoC container
[FR] MindScape LightSpeed as my Object-Relational-Mapper. NHibernate is the best free alternative I’m aware of.
[UT]* Reflector to drill down to the guts of any code library (also check-out the nice plug-ins)
[UT] Silverlight Spy to dissect any Silverlight application.
[UT] RegexBuddy for managing those difficult regular expressions. Regulator is the best, free alternative I know of.
[UT]* LINQPad as a easy way to query SQL databases using LINQ and as a general scratchpad application to test C#/VB.NET code snippets.
[UT]* Fiddler to debug all your HTTP traffic in IE. Also see the neXpert plugin for monitoring performance problems.
[UT]* Firebug to assist with testing web applications running in Firefox. Also see YSlow add-on for performance testing and Web Developer add-on for additional Firefox web development tools.
[CA]* FxCop to enforce .NET coding guidelines
[CA] NDepend to get all the static code metrics I'd ever want
[CA] ANTS Profiler for performance and memory profiling
[MD] Enterprise Architect to do UML Modelling and Model Driven Design if required. Alternatively use Visio with these simple templates.
[MD]* FreeMind as mind mapping tool
[TC]* ScrewTurn Wiki for team collaboration
[QA]* Eviware soapUI for functional and load testing of SOA web services
[QA]* Telerik WebAii Testing Framework for automated regression testing of Web 2.0 apps
[DP]* Windows Installer XML (WiX) for creating Windows Installers

Threading in C# – Free E-book Updated

2010-08-31T15:19:00.001+02:00

This is just a quick post to highlight the fact that Joseph Albahari, author of the excellent LINQPad and C# 4.0 In A Nutshell book, has recently updated hits free Threading in C# e-book. The e-book contains some really great, concise content on all the latest threading related topics in .NET. The latest version includes coverage of new .NET 4.0 constructs like ThreadLocal<T> and Lazy<T> as well as a whole chapter dedicated to the new Parallel Programming extensions to the .NET framework. Highly recommended!

Log4View – Getting the most out of your log4net/log4j log files

2010-08-29T14:55:00.001+02:00

If you are using log4net or log4j for writing log files, do yourself a favour and get the whole team and production support a copy of Log4View as soon as possible. It really is a wonderful tool that sits on top of your log4net or log4j files to give you a birds-eye view on what’s happening in your application. Need to monitor your servers remotely as they are running? No problem, just add a reference to their TCP log appender that allows you to configure log4net to log all statements to a TCP port. This gives you the flexibility of remotely monitoring the server as the application is running in production (providing you open the port on the firewall of course).

One of the most powerful features IMO is the grouping feature in the message view. Look at the following screenshot that shows how easy it is to group messages according to session id/thread id or any other custom log information added to your log files. Simply drag the columns you want to group by into the Group By area above the grid:

This specific grouping allows us to easily inspect the separate requests associated with a certain user in the system. Highly recommended!

RunSharp – IL Generation for Dummies

2010-08-29T00:27:00.001+02:00

We’ve been doing some interesting work the past 2-3 weeks on creating a Rule Engine for our company’s flagship product, On Key. The Rule Engine needs to evaluate rules entered by the end user at run-time according to our pre-defined grammar to determine a true/false answer. This allows us tremendous flexibility in that end-users are able to specify under which conditions certain actions should occur within the system. The main requirements for the technical solution was that it should be as quick as possible but it also has to cater for rules being changed at run-time by the end-users.

Before heading off to create our own custom solution, we did some research to consider possible solutions to solving the same kind of problem. Some of the solutions we came across were:

After looking at all of these solutions, we decided to rather create our own to give us the ultimate control and flexibility over extending the Rule Engine going forward. Setting up the grammar using the excellent ANTLR and ANTLRWorks was quite easy to do. ANTLR takes care of generating the C# code that will do the lexing, parsing and type checking of the rules entered by the user. Thereafter we moved onto the run-time evaluation/compilation of the rules represented in the Abstract Syntax Tree created by ANTLR. For this purpose, we created 3 tree walkers/visitors on our AST to compare against each other:

Interpreter – Evaluate the rules dynamically as we walk the AST
C# Code Generator – Walk the tree and generate C# code that is compiled using the C# compiler into an assembly
IL Code Generator – Walk the tree and generate dynamic IL code in memory

Interpreter

The main benefit of writing an interpreter is that you get rid of the complexities associated with managing rules that change at run-time as you don’t need to compile the rules. You can still use caching to store a format of the constructed AST for the parsed rules to prevent the overhead of parsing/lexing the rule every time, but you don’t need to worry about unloading/loading DLL’s of code that was previously compiled. The obvious down side of this solution is the performance overhead compared to compiled C# code.

C# Code Generator

The main benefit of writing a C# code generator is that we can use the C# compiler and our existing C# skill set to create rules that perform as quickly as possible. The downside however is that you cannot unload an assembly out of a .NET AppDomain. To support the requirement of the rules changing at run-time, we therefore would have to create a temporary AppDomain and load the assembly in there. This is a technique used by the excellent CS-Script to evaluate C# scripts at run-time. You also need FullTrust permissions to launch the C# compiler at run-time, which is problematic for our Silverlight IIS hosted application.

IL Code Generator

The main benefit of writing a IL generator is that it is gives you the performance equivalent to that of the compiled C# code but also the dynamics of the interpreter in the sense that you can generate the IL in memory at run-time and basically drop the DynamicMethod from memory when rule changes are made by the end users and the rules therefore need to be recompiled. However, generating IL directly is notoriously difficult and error-prone. Our immediate concerns were for the technical complexities of managing this solution going forward. That was until, we discovered RunSharp. We came across RunSharp in this CodeProject article. It essentially provides a run-time wrapper on top of Reflection.Emit IL generation to allow you to easily construct dynamic code at run-time. To illustrate, let me take an example of C# rule generated by the C# Code Generator we created and illustrate the equivalent RunSharp code.

C# Code

public static bool Rule2(IOptionValueProvider options)
{
  // "CA_FIL_STE_LIFTING_UNIT = 'WITH FIL STE LIFTING UNIT'"

  return String.Compare(options.GetOptionValueAsString(1782), "WITH FIL STE LIFTING UNIT", StringComparison.OrdinalIgnoreCase) == 0;
}

IL Code

public static void Rule2CIL(TypeGen typeGen)
{
   // "CA_FIL_STE_LIFTING_UNIT = 'WITH FIL STE LIFTING UNIT'"

   CodeGen g = typeGen.Public.Static.Method(typeof (bool), "Rule2").Parameter(typeof (IOptionValueProvider), "options");
   {
       Operand result = Static.Invoke(typeof (String), "Compare", g.Arg("options").Invoke("GetOptionValueAsString", 1782), "WITH FIL STE LIFTING UNIT", StringComparison.OrdinalIgnoreCase) == 0;
       g.Return(result);
   }
}

You’ll see that by using the RunSharp API we are able to construct code that looks quite similar to the C# code, except that in the background RunSharp will take care of generating the IL for us. RunSharp is able to provide us with such an API through the judicious use of implicit type casting and operator overloading. Let’s consider a more complex rule involving multiple operands:

C# Code

public static bool Rule10022(IOptionValueProvider options)
{
   // "CA_PACKSHAPE eq 'B' and NOT ('2755900-0100' IN CA_REBUILDING_KIT)"

   return (String.Compare(options.GetOptionValueAsString(1009), "B", StringComparison.OrdinalIgnoreCase) == 0) && (!(options.CheckValueInOption("2755900-0100", 1495, OptionDataType.String)));
}

IL Code

public static void Rule10022CIL(TypeGen typeGen)
{
   CodeGen g = typeGen.Public.Static.Method(typeof (bool), "Rule10022").Parameter(typeof (IOptionValueProvider), "options");
   {
       Operand op1 = Static.Invoke(typeof (String), "Compare", g.Arg("options").Invoke("GetOptionValueAsString", 1009), "B", StringComparison.OrdinalIgnoreCase) == 0;
       Operand op2 = g.Arg("options").Invoke("CheckValueInOption", "2755900-0100", 1495, OptionDataType.String);
       g.Return((op1) && (!op2));
   }
}

Creating a Tree walker/visitor to traverse the AST and emit the appropriate RunSharp API calls was a snap to do and really gives us the best of both worlds - the speed of compiled code and the flexibility of managing rules that change at run-time without having to worry about the complexities of IL generation. The one or two issues that we discovered were quickly corrected by the author of the library Stefan Simek. So if you ever want to create IL code at run-time again, you should really seriously consider RunSharp. As the post title says, it really is IL generation for Dummies.

C# Coding Standards Using ReSharper

2010-06-01T12:35:00.001+02:00

I found the following great blog post by Tim Lloyd on how to setup coding standards for your development environment using ReSharper 5.0. We’ve done pretty much the same thing in our environment although I wasn’t aware of the ReSharper Settings Manager which I will test drive soon as I have found sharing settings between team members to be a real pain. Seems like JetBrains will look at this for v6 though. Things we are doing in addition to the content mentioned by Tim are:

Using Agent Smith to ensure the correctness of the spelling for our resource files etc.
Created a more advanced layout template to order all our code using the same structure when doing Code Cleanup

It works great if everybody in the team follows the agreed upon procedure. I find that this makes code reviews focus on the right things – the actual business logic!

Code Metrics

2010-04-19T10:04:00.001+02:00

NOTE: This is a repost of on old post as I am moving onto the Blogger platform

I've been wanting to do a post about code metrics for quite a while now - mostly to organize my thoughts on the topic as it is something that I want to introduce at work, but also to get some feedback from other people as to if and how they are using metrics to assist them in crafting quality code. After reading Jeremy Miller's post on the topic, I thought I might as well take the plunge. I'll start by musing over what metrics I found to be useful, then continue with looking at tool support for generating these metrics and finish off with considering when to use these metrics.

Metrics

I am not going to cover all the different metrics in detail but instead highlight what seems to me to be the most useful metrics and refer to articles/links where other people who have done an excellent job on covering these metrics in detail. Here are the reference articles that I used:

Robert C. Martin's article on OO design quality metrics [1]
Wikipedia's summary of these package metrics [2]
Kirk Knoernschild's excellent introductory article on metrics with sample refactorings included [3]
Patrick Smacchia's (developer of NDepend software) excellent coverage on all the types of metrics supported by NDepend [4]
Write up on the software metrics supported by the Software Design Metrics software [5]

Size metrics

Size metrics are consistently good indicators of fault-proneness: large methods/classes/packages contain more faults [5].

Source Lines Of Code (SLOC) measures the amount of lines of code. To be really useful comment lines and lines that have been broken into multiple lines need to be factored out. Some people refer to this as the logical LOC vs. the physical LOC.

"2 significant advantages of logical LOC over physical LOC are:

Coding style doesn’t interfere with logical LOC. For example the LOC won’t change because a method call is spawn on several lines because of a high number of argument.

logical LOC is independent from the language. Values obtained from assemblies written with different languages are comparable and can be summed." [4]

Complexity metrics

There is a direct correlation between complexity and the defect rate of software, so keeping code simple is a solid first step toward lowering the defect rate of software [3].

Cyclomatic Complexity (CC) measures code complexity by counting the number of linearly independent paths through code. Complex conditionals and boolean operators increase the number of linear paths, resulting in a higher CCN. Methods with a CCN of five or higher are good refactoring candidates to help ensure code remains easy to understand [3]

Package metrics

Coupling/Dependency metrics

Excessive dependencies between packages compromise architecture and design. Complex dependencies inhibit the testability of your system and presents numerous other challenges as presented in [1] and [3].

Afferent Coupling (Ca) measures the number of types outside a package that depend on types within the package (incoming dependencies). High afferent coupling indicates that the concerned packages have many responsibilities. [1]
Efferent Coupling (Ce) measures the number of types inside a package that depends on types outside of the package (outgoing dependencies). High efferent coupling indicates that the concerned package is dependant. [1]

It goes to reason that:

"...afferent and efferent coupling allows you to more effectively evaluate the cost of change and the likelihood of reuse. For instance, maintaining a module with many incoming dependencies is more costly and risky since there is greater risk of impacting other modules, requiring more thorough integration testing. Conversely, a module with many outgoing dependencies is more difficult to test and reuse since all dependent modules are required ... Concrete modules with high afferent coupling will be difficult to change because of the high number of incoming dependencies. Modules with many abstractions are typically more extensible, so long as the dependencies are on the abstract portion of a module." [3]

Instability (I) measures the ratio of efferent coupling (Ce) to total coupling. I = Ce / (Ce + Ca). This metric is an indicator of the package's resilience to change. The range for this metric is 0 to 1, with I=0 indicating a completely stable package and I=1 indicating a completely instable package. [1]
Abstractness (A) measures the ratio of the number of internal abstract types (i.e abstract classes and interfaces) to the number of internal types. The range for this metric is 0 to 1, with A=0 indicating a completely concrete package and A=1 indicating a completely abstract package. [1]
Distance from main sequence (D) measures the perpendicular normalized distance of a package from the idealized line A + I = 1 (called main sequence). This metric is an indicator of the package's balance between abstractness and stability. A package squarely on the main sequence is optimally balanced with respect to its abstractness and stability. Ideal packages are either completely abstract and stable (I=0, A=1) or completely concrete and instable (I=1, A=0). The range for this metric is 0 to 1. [1][4]

"A value approaching zero indicates a module is abstract is relation to its incoming dependencies. As distance approaches one, a module is either concrete with many incoming dependencies or abstract with many outgoing dependencies. The first case represents a lack of design integrity, while the second is useless design." [3]

Cohesion metrics

"A low cohesive design element has been assigned many unrelated responsibilities. Consequently, the design element is more difficult to understand and therefore also harder to maintain and reuse. Design elements with low cohesion should be considered for refactoring, for instance, by extracting parts of the functionality to separate classes with clearly defined responsibilities." [5]

Relational Cohesion (H) measures the average number of internal relationships per type. Let R be the number of type relationships that are internal to this package (i.e that do not connect to types outside the package). Let N be the number of types within the package. H = (R + 1)/ N. The extra 1 in the formula prevents H=0 when N=1. The relational cohesion represents the relationship that this package has to all its types. As classes inside an package should be strongly related, the cohesion should be high. On the other hand, too high values may indicate over-coupling. A good range for RelationalCohesion is 1.5 to 4.0. Packages where RelationalCohesion < 1.5 or RelationalCohesion > 4.0 might be problematic. [4]
Lack of Cohesion of Methods (LCOM) The single responsibility principle states that a class should not have more than one reason to change. Such a class is said to be cohesive. A high LCOM value generally pinpoints a poorly cohesive class [4]

Inheritance metrics

"Deep inheritance structures are hypothesized to be more fault-prone. The information needed to fully understand a class situated deep in the inheritance tree is spread over several ancestor classes, thus more difficult to overview. Similar to high export coupling, a modification to a design element with a large number of descendents can have a large effect on the system." [5]

Depth of Inheritance Tree (DIT) measures the number of base classes for a class or structure. Types where DIT is higher than 6 might be hard to maintain. However it is not a rule since sometime your classes might inherit from tier classes which have a high value for depth of inheritance. [4]

Tools

There is a visual GUI (VisualNDepend) that allows you to browse your code structure and evaluate the metrics as well as a console application that you can generate the metrics with. Patrick has also created a CQL (Code Query Language) which allows NDepend to consider your code as a database with CQL being the query language with which you can check some assertions on this database. As a consequence, CQL is similar to SQL and supports the typical SELECT TOP FROM WHERE ORDER BY patterns. Here is an example of a CQL query:

WARN IF Count > 0 IN SELECT METHODS 
WHERE NbILInstructions > 200 ORDER BY NbILInstructions DESC
// METHODS WHERE NbILInstructions > 200 are extremely complex and            
// should be split in smaller methods.

How cool is this! To quote:

"CQL constraints are customisable and typically tied with a particular application. For example, they can allow the specification of customized encapsulation constraints, such as, I want to ensure that this layer will never use this other layer or I want to ensure that this class will never be instantiated outside this particular namespace."

Other tools that that you can have a look at include SourceMonitor, DevMetrics, Software Design Metrics and vil to name a few. vil does not support .NET 2.0 and does not seem to be under active development. DevMetrics, after being open-sourced, seemed to have stagnated with no visible activity on SourceForge. SourceMonitor is actively under development and supports a variety of programming languages. However, it supports only a small subset of the metrics mentioned which does not include support for important metrics like efferent and afferent coupling etc. Software Design Metrics takes a novel approach in that it measures the complexity based on the UML models for the software. This has the advantage of being language independent, but you obviously need to have UML models to run
the analysis.

When to use

I think I will stick with an approach of identifying a subset of useful metrics and using these as part of an overall process of static code analysis on a regular basis. When I say regular basis I feel it should be part of your continuous build process to prevent people from committing code into your repository that does not satisfy your constraints. With a tool like NDepend you can create your own custom level of acceptance criteria by which the build will fail/succeed and exclude metrics that you feel should not apply to your code base.

As mentioned, the code metrics should form part of a bigger quality process that includes:

Visual inspection/QA via peer code reviews (as mentioned having the metrics via a tool like VisualNDepend can greatly assist in the QA)
Automated code standards check (I prefer FxCop)
Automated code metric check (NDepend seems like the tool to use here)
Automated code coverage statistics (I prefer NCover and NCoverExplorer)

Well, that basically covers my thoughts on the topic for now. I'm interested to know how many people are actively using metrics and what metrics they are using. I'd also like to know what processes or tools people are using to generate these metrics on their code. Let me know what's working for you in your environment.

Vacancy: Intermediate C# Software Engineer; Cape Town - South Africa

2010-01-13T11:03:00.001+02:00

We’ve been struggling to get some good CV’s from recruitment agencies for an intermediate-senior level C# Software Engineer. I’m hoping that I might find an interested reader that fits our requirements. The position is for a year contract starting as soon as possible.

So if you are in the Cape Town area and looking for a great opportunity to work in an agile environment (SCRUM) using the latest technologies (Silverlight 3.0, .NET 3.5, SQL Server 2008) with a strong focus on quality (TDD) send me your CV. You can find more details on the position as well as our company and contact details in the job specification here. Please include an indication of your availability as well as your current monthly remuneration.

Please note:

If you hear nothing from me, assume that we are not interested, i.e. we’ll contact you.
Please no recruitment agencies.

Code Metrics

2009-11-01T20:59:00.001+02:00

NOTE: This is a repost of on old post as I am moving onto the Blogger platform

I've been wanting to do a post about code metrics for quite a while now - mostly to organize my thoughts on the topic as it is something that I want to introduce at work, but also to get some feedback from other people as to if and how they are using metrics to assist them in crafting quality code. After reading Jeremy Miller's post on the topic, I thought I might as well take the plunge. I'll start by musing over what metrics I found to be useful, then continue with looking at tool support for generating these metrics and finish off with considering when to use these metrics.

Metrics

Robert C. Martin's article on OO design quality metrics [1]
Wikipedia's summary of these package metrics [2]
Kirk Knoernschild's excellent introductory article on metrics with sample refactorings included [3]
Patrick Smacchia's (developer of NDepend software) excellent coverage on all the types of metrics supported by NDepend [4]
Write up on the software metrics supported by the Software Design Metrics software [5]

Size metrics

Size metrics are consistently good indicators of fault-proneness: large methods/classes/packages contain more faults [5].

Source Lines Of Code (SLOC) measures the amount of lines of code. To be really useful comment lines and lines that have been broken into multiple lines need to be factored out. Some people refer to this as the logical LOC vs. the physical LOC.

"2 significant advantages of logical LOC over physical LOC are:

Coding style doesn’t interfere with logical LOC. For example the LOC won’t change because a method call is spawn on several lines because of a high number of argument.

logical LOC is independent from the language. Values obtained from assemblies written with different languages are comparable and can be summed." [4]

Complexity metrics

There is a direct correlation between complexity and the defect rate of software, so keeping code simple is a solid first step toward lowering the defect rate of software [3].

Cyclomatic Complexity (CC) measures code complexity by counting the number of linearly independent paths through code. Complex conditionals and boolean operators increase the number of linear paths, resulting in a higher CCN. Methods with a CCN of five or higher are good refactoring candidates to help ensure code remains easy to understand [3]

Package metrics

Coupling/Dependency metrics

Afferent Coupling (Ca) measures the number of types outside a package that depend on types within the package (incoming dependencies). High afferent coupling indicates that the concerned packages have many responsibilities. [1]
Efferent Coupling (Ce) measures the number of types inside a package that depends on types outside of the package (outgoing dependencies). High efferent coupling indicates that the concerned package is dependant. [1]

It goes to reason that:

"...afferent and efferent coupling allows you to more effectively evaluate the cost of change and the likelihood of reuse. For instance, maintaining a module with many incoming dependencies is more costly and risky since there is greater risk of impacting other modules, requiring more thorough integration testing. Conversely, a module with many outgoing dependencies is more difficult to test and reuse since all dependent modules are required ... Concrete modules with high afferent coupling will be difficult to change because of the high number of incoming dependencies. Modules with many abstractions are typically more extensible, so long as the dependencies are on the abstract portion of a module." [3]

Instability (I) measures the ratio of efferent coupling (Ce) to total coupling. I = Ce / (Ce + Ca). This metric is an indicator of the package's resilience to change. The range for this metric is 0 to 1, with I=0 indicating a completely stable package and I=1 indicating a completely instable package. [1]
Abstractness (A) measures the ratio of the number of internal abstract types (i.e abstract classes and interfaces) to the number of internal types. The range for this metric is 0 to 1, with A=0 indicating a completely concrete package and A=1 indicating a completely abstract package. [1]
Distance from main sequence (D) measures the perpendicular normalized distance of a package from the idealized line A + I = 1 (called main sequence). This metric is an indicator of the package's balance between abstractness and stability. A package squarely on the main sequence is optimally balanced with respect to its abstractness and stability. Ideal packages are either completely abstract and stable (I=0, A=1) or completely concrete and instable (I=1, A=0). The range for this metric is 0 to 1. [1][4]

"A value approaching zero indicates a module is abstract is relation to its incoming dependencies. As distance approaches one, a module is either concrete with many incoming dependencies or abstract with many outgoing dependencies. The first case represents a lack of design integrity, while the second is useless design." [3]

Cohesion metrics

Relational Cohesion (H) measures the average number of internal relationships per type. Let R be the number of type relationships that are internal to this package (i.e that do not connect to types outside the package). Let N be the number of types within the package. H = (R + 1)/ N. The extra 1 in the formula prevents H=0 when N=1. The relational cohesion represents the relationship that this package has to all its types. As classes inside an package should be strongly related, the cohesion should be high. On the other hand, too high values may indicate over-coupling. A good range for RelationalCohesion is 1.5 to 4.0. Packages where RelationalCohesion < 1.5 or RelationalCohesion > 4.0 might be problematic. [4]
Lack of Cohesion of Methods (LCOM) The single responsibility principle states that a class should not have more than one reason to change. Such a class is said to be cohesive. A high LCOM value generally pinpoints a poorly cohesive class [4]

Inheritance metrics

Depth of Inheritance Tree (DIT) measures the number of base classes for a class or structure. Types where DIT is higher than 6 might be hard to maintain. However it is not a rule since sometime your classes might inherit from tier classes which have a high value for depth of inheritance. [4]

Tools

When it comes to tools, the Mercedes Benz of .NET code metrics tools from my point of view has to be NDepend 2.0. NDepend 2 provides more than 60 metrics (including all of the metrics listed above) and includes integration into your automated build via support for MSBuild, NAnt and CruiseControl.NET. Browse to here for a sample report and here for a demo on how to integrate it into your build.
There is a visual GUI (VisualNDepend) that allows you to browse your code structure and evaluate the metrics as well as a console application that you can generate the metrics with. Patrick has also created a CQL (Code Query Language) which allows NDepend to consider your code as a database with CQL being the query language with which you can check some assertions on this database. As a consequence, CQL is similar to SQL and supports the typical SELECT TOP FROM WHERE ORDER BY patterns. Here is an example of a CQL query:



WARN IF Count > 0 IN SELECT METHODS 
WHERE NbILInstructions > 200 ORDER BY NbILInstructions DESC
// METHODS WHERE NbILInstructions > 200 are extremely complex and            
// should be split in smaller methods.

How cool is this! To quote:

"CQL constraints are customisable and typically tied with a particular application. For example, they can allow the specification of customized encapsulation constraints, such as, I want to ensure that this layer will never use this other layer or I want to ensure that this class will never be instantiated outside this particular namespace."

VisualNDepend also provides a CQL editor which supports intellisense and verbose compile error descriptions to make writing CQL queries a lot easier. Enough said! Browse to here for a complete overview of the NDepend 2 features.
Other tools that that you can have a look at include SourceMonitor, DevMetrics, Software Design Metrics and vil to name a few. vil does not support .NET 2.0 and does not seem to be under active development. DevMetrics, after being open-sourced, seemed to have stagnated with no visible activity on SourceForge. SourceMonitor is actively under development and supports a variety of programming languages. However, it supports only a small subset of the metrics mentioned which does not include support for important metrics like efferent and afferent coupling etc. Software Design Metrics takes a novel approach in that it measures the complexity based on the UML models for the software. This has the advantage of being language independent, but you obviously need to have UML models to run
the analysis.

When to use

When should one use these metrics? I agree with Jeremy Miller in his post that the metrics should not replace the visual inspection/QA process and be performed in addition to it. It would be nice to have these metrics at hand to assist in the QA process though. I also agree with Frank Kelly in his post that a working system with no severity 1/2 errors and happy end users are more important than getting the right balance of Ca/Ce or whatever metric you are interested in.
I think I will stick with an approach of identifying a subset of useful metrics and using these as part of an overall process of static code analysis on a regular basis. When I say regular basis I feel it should be part of your continuous build process to prevent people from committing code into your repository that does not satisfy your constraints. With a tool like NDepend you can create your own custom level of acceptance criteria by which the build will fail/succeed and exclude metrics that you feel should not apply to your code base.
As mentioned, the code metrics should form part of a bigger quality process that includes:

Visual inspection/QA via peer code reviews (as mentioned having the metrics via a tool like VisualNDepend can greatly assist in the QA)
Automated code standards check (I prefer FxCop)
Automated code metric check (NDepend seems like the tool to use here)
Automated code coverage statistics (I prefer NCover and NCoverExplorer)

Code Reviews

2009-11-01T20:58:00.001+02:00

NOTE: This is a repost of on old post as I am moving onto the Blogger platform

Code reviews are a proven, effective way to minimize defects, improve code quality and keep code more maintainable. It encourages team collaboration and assists with mentoring developers. Yet, not many projects employ code reviews as a regular part of their development process. Why is this? Programmer egos and the hassles of packaging source code for review are often sited as some reasons for not doing code reviews.

I felt that code reviews should form part of a good code quality control process that includes:

Visual inspection/QA via peer code reviews
Automated code standards check (I prefer FxCop)
Automated code metrics check (I prefer NDepend)
Automated code coverage statistics (I prefer NCover and NCoverExplorer)

In this post I will spend some time looking at code reviews. I'll start by considering different styles of code review and some code review metrics. I'll then move on to some thoughts on review frequencies and best practices for peer code review. I'll finish off by considering some tools that can assist with the code review process itself.

References

I used the following books and articles to organize my thoughts on the topic:

Smart Bear Software's free, excellent book Best Kept Secrets of Peer Code Review. [Cohen01]
Smart Bear Software's whitepaper on Best Practices for Peer Code Review. [Cohen02]
Robert Bogue's article on Effective Code Reviews Without the Pain. [Bogue]
Wikipedia article on Code Review. [Wikipedia01]
Wikipedia article on Hawtorne Effect. [Wikipedia02]

Review Types

Code review practices often fall into two main categories: formal code review and lightweight code review.

Formal Code Review

Formal code review, such as a Fagan inspection, involves a careful and detailed process with multiple participants and multiple phases. Software developers attend a series of meetings and review code line by line, usually using printed copies of the material. An Introductory meeting is held where the goals and rules for the review are explained and the review material is handed out. Reviewers inspect the code privately and feedback is given in a subsequent Inspection meeting. The author fixes any defects identified and the reviewers verify that the defects are fixed in a further Verification meeting.

Formal inspections are extremely thorough and effective at finding defects in code but take a long time to complete.

Lightweight Code Review

Lightweight code review typically requires less overhead than formal code inspections. However, if done properly, it can be equally effective. Lightweight reviews are often conducted as part of the normal software development process and attempt to improve the cost-benefit factor – providing improved code quality without incurring the overhead of traditional meetings-based inspections [Wikipedia01] [Cohen01]:

Over-the-shoulder review – One developer standing over the author’s workstation while the author walks the reviewer through a set of code changes. Simple to execute, these kinds of reviews lend themselves to learning and sharing between developers and gets people interacting with each other instead of hiding behind impersonal e-mail and instant messages. The informality and simplicity unfortunately also leads to some shortcomings such as the process not being enforceable; lack of code review process metrics; does not work for distributed teams; the reviewer being led too hastily through the code and the reviewer not verifying that defects were fixed properly.
Email pass-around – The author or source code management system packages the code changes and sends an e-mail to the reviewers. Relatively simple to execute, these kinds of reviews have the added benefit of working equally well with distributed teams. Other people, like domain experts, can also be brought in to review certain areas or a reviewer may even defer to another reviewer. Some shortcomings include the difficulty to track the various threads of conversation or code changes; lack of code review process metrics and the reviewers not being able to verify that defects were fixed properly.
Tool-assisted reviews – Specialized tools are used in all aspects of the review such as collecting files, transmitting and displaying files, commentary, collecting metrics and controlling the code review process workflow. The major drawback of any tool-assisted review is the cost of buying a commercial offering or the development cost associated with developing an in-house tool.
Pair Programming – Two developers write code together at a single workstation using continuous free-form discussion and review such as is common in Extreme Programming. Some people argue in favour of the deep insight the reviewer has into the problem at hand, whilst other argue that the closeness is exactly what you do not want from a reviewer as you want a fresh, unbiased opinion. Some people suggest doing both pair programming and a follow-up standard review using a fresh pair of eyes.

In a current project that I am working on, we are using the over-the-shoulder kind of review process. Rather than doing no review, we opted for using this approach as the best way to balance the cost-benefit factor of our reviews. Every submission into the repository needs to be reviewed by a fellow developer. The details of the reviewer is added to the check-in note to provide the necessary traceability and visibility. Authors rotate the developers doing the reviews based on the area of the system changed and also to do some cross-skilling.

From our experience, we also find the informality of the process as its biggest downfall. As pressure starts to build, less time is spend on doing reviews and the process sometimes gets perilously close to becoming a mere formality.

Review Metrics

The following metrics are typically used to measure the effectiveness of a code review process:

Inspection Rate: How fast are we able to review code? Normally measured in kLOC (thousand Lines Of Code) per man-hour.
Defect Rate: How fast are we able to find defects? Normally measured in number of defects found per man-hour.
Defect Density: How many defects do we find in a given amount of code? Normally measured in number of defects found per kLOC. The higher the defect density, the more defects are being identified which usually implies a more effective review process.

This begs the question, what are good values for these metrics? [Cohen01] uses two examples to illustrate the difficulties with evaluating these metrics:

In Example A a few lines of code to a mission critical module are evaluated by 3 reviewers to absolutely ensure that no defects are introduced. The review results in a high defect density (say 4 defects in a few lines), slow inspection rate and a defect rate of 1 defect per hour. In Example B changes made to a GUI dialog box results in 120 lines of changed code, of which some of the code was generated by the GUI designer. One reviewer is assigned to verify the changes and the reviewer chooses to ignore the designer generated code and only evaluates the code behind the GUI elements added. This results in a low defect density (say 1 bug in 120 lines), fast inspection rate (say 30 minutes) and a defect rate of 2 defects per hour.

As evident, the metrics are quite different. Should the high defect density rate of Example A reflect badly on the developer? [Cohen01] argues probably not as the high defect density is the result of the multiple reviewers scrutinizing every line of code to make absolutely sure no defects are introduced. Should the low defect density rate of Example B reflect badly on the reviewer? [Cohen01] argues probably not as it is difficult to review designer generated code. As the code is also not mission critical like in Example A, the reviewer was perhaps intentionally tasked to spend less time reviewing the changes.

Review Frequency

There are no hard and fast rules for determining the frequency of code reviews. The frequency of code reviews can be influenced by quite a few factors and has to be determined contextually. Some of the factors to consider include:

Review type: Formal Code Review requires more time and effort to complete than Lightweight Code Review.
Review purpose: Code reviews used to mentor and improve team collaboration might need to be executed more frequently.
Frequency and scope of code changes: Frequent, small check-ins versus irregular, big check-ins.
Nature of the development effort: Open Source; Onshore; Offshore.
Business need for quality: Mission critical applications need to ensure a high level of quality.

Having said of all this, doing infrequent code reviews or waiting to do reviews till code complete seriously undermines the benefit and integrity of the code review process. Reviewing code frequently motivates the developers to ensure the quality of the code being delivered – better known as the Hawthorne Effect [Wikipedia02].

Best Practices for Peer Code Review

[Cohen02] and [Bogue] present several techniques to ensure that your code reviews improve your code without wasting your developer’s time:

Verify that defects are actually fixed!
Remember the purpose: Code reviews often start off on the wrong foot because they are perceived as an unnecessary step that has been forced upon the developers or, in some cases evidence that management doesn’t trust the developers. Code reviews are a proven, effective way to minimize defects and they are, at their core, an industry best practice.
A matter of approach: Prevent code reviews from becoming mental jousting matches where people take shots at each other. Consider the review process as a forum to collaborate and learn from one another. Reviewers should ask questions rather than making statements; remember to praise and also be mindful of the fact that there is often more than one way to approach a solution. Authors should try to hear past attacking comments and try to focus on the learning that they can get out of the process.
Review fewer than 200-400 lines of code at a time and aim for an inspection rate of less than 300-500 LOC/hour: Take your time when doing a review - faster is not better.
Never review for more than 60-90 minutes at a time.
Authors should consider how to best explain their changes before the review begins: [Cohen02] refers to this process as “annotating the source code”. As the author has to re-think and explain the changes during the annotation process, the author will uncover many defects before the review even begins.
Checklists substantially improve results for both authors and reviewers: Omissions are the hardest defects to find. A checklist is a great way to avoid this problem as it reminds the reviewer or author to take the time to look for something that might be missing. Publish the checklists on a wiki. As each person typically makes the same 15-20 mistakes, also consider creating personal checklists.
Management must foster a good code review culture in which finding defects is viewed positively: A negative attitude towards defects found can sour a whole team and sabotage the bug-finding process.
Beware of the “Big Brother” effect: Code review metrics should never be used to single out developers, particularly not in front of their peers. Metrics should be used to measure the efficiency or the effect of the process.
The Ego Effect: Do at least some code review, even if you don’t have time to review it all. The Ego effect drives developers to review their own work and write better code because they know others will be looking at their code.

Tools

I found the following tools to support a lightweight, tool-assisted peer code review process:

Code Collaborator - Commercial offering from Smart Bear Software that supports online reviews with inline source commenting, threaded contextual chat, asynchronous reviews for when participants are separated by many timezones, review metrics and reports, version control integration, customisable workflow and much more.
Crucible - Commercial offering from Atlassian that supports online reviews with inline source commenting, threaded comments, review metrics, version control integration, workflow and much more.
Codestriker - Open source project that supports online reviews with version control integration and review metrics.

Conclusion

Well that about covers my thoughts on the topic. I'm interested to know how many people are actually doing some form of peer code review. What type of reviews are you doing? How often do you do reviews? Do you use any tools to assist you with your reviews? What code review practices work best in your environment?

FxCop

2009-11-01T20:55:00.001+02:00

NOTE: This is a repost of on old post as I am moving onto the Blogger platform

In previous posts about Code Metrics and Code Reviews, I explored some metrics and techniques that I felt should form part of any good software quality control process. One of the tools that I mentioned is FxCop. In this post I take a closer look at FxCop. I start by looking at how FxCop works and how you can fit it into your development process. I then consider the different rule sets to use and look at how you can utilise FxCop to guide your VS 2003/2005/2008 development efforts. I finish off the article by linking to articles that show you how to develop your own custom FxCop rules.

Introduction

FxCop analyses .NET assemblies for potential code compliance problems and forms part of static code analysis. With static code analysis the compiled code is checked for compliance to identify possible defects before executing the code. FxCop employs assembly analysis of generated assemblies using an introspection engine. Since the analysis is performed on the generated intermediate language (MSIL) code, FxCop is not dependent on any particular .NET implementation language. However, it is important to remember that assembly analysis cannot analyse certain aspects of the original source code, such as code comments, since these are not carried over in the compilation process. You may also run into a scenarios where there are differences between the FxCop violations in Debug and Release configurations due to the compiler optimisations being applied in Release mode.

FxCop ships with a large rule set analysing library design, globalization, interoperability, maintainability, mobility, naming, performance, portability, reliability, security and usage aspects of the assembly. The rule set can be extended to include new custom rules and existing rules can be switched off if necessary.

Process

FxCop will typically be used by the following people:

Developers use FxCop continuously during development, because it helps with familiarising themselves with the .NET coding best practices. As they progress they should find that they break fewer rules and can consequently rely on the tool less often. However, they should still continue to evaluate their code at predefined intervals in the SDLC. By actively using this tool, developers raise the standard of code going into code review sessions. I advocate for using the tool before every commit.
Code reviewers use the tool to verify that developers are indeed complying with the best practices as defined in the rule set. They should also check that they agree with any "excluded" defects. Use of FxCop should rapidly improve both the speed and breadth of code review sessions by instantly highlighting the most obvious problems.

I strongly recommend using FxCop together with a refactoring tool like ReSharper which makes fixing the rule violations a lot easier and less error prone.

FxCop Backlogs

The following articles present some ideas on how to introduce FxCop into your SDLC:

Rule sets

The rule sets provided with FxCop are quite extensive. Some rule sets (i.e Globalization/Mobility rule set) also includes rules that only apply to applications that target certain platforms/cultures/languages etc. Microsoft itself uses a sub-set of the complete FxCop rules to guide their own internal development efforts. From experience, I have created two rule sets to guide my own development efforts.

Base Rule Set

The Base Rule Set is the rule set that I use for all new .NET development efforts. This rule set excludes the following rules:

Globalization Rules: CA1300: Specify MessageBoxOptions
Globalization Rules: CA1301: Avoid duplicate accelerators
Globalization Rules: CA1302: Do not hardcode locale specific strings
Globalization Rules⁺: CA1303: Do not pass literals as localized parameters
Naming Rules: CA1701: Resource string compound words should be cased correctly
Naming Rules: CA1702: Compound words should be cased correctly
Naming Rules: CA1703: Resource strings should be spelled correctly
Naming Rules: CA1704: Identifiers should be spelled correctly
Naming Rules: CA1726: Use preferred terms
Usage Rules: CA2204⁺: Literals should be spelled correctly
Usage Rules: CA2243: Attribute string literals should parse correctly

_{+ Not available in VS 2008}

Minimum Rule Set

The Minimum Rule Set is the rule set that I use for all existing .NET development efforts. The idea is that these projects will most likely hit more rule violations on their existing code base and I therefore want to exclude some rules that add a lot of noise without providing IMO a lot of benefit. This rule set adds the following additional exclusions to those already excluded within the Base Rule Set:

Design Rules: CA1002: Do not expose generic lists
Design Rules: CA1003: Use generic event handler instances
Design Rules: CA1004: Generic method should provide type parameter
Design Rules: CA1005: Avoid excessive parameters on generic types
Design Rules: CA1007: Use generics where appropriate
Design Rules: CA1010: Collections should implement generic interface
Design Rules: CA1020: Use properties where appropriate
Design Rules: CA1024: Avoid namespaces with few types
Design Rules: CA1031: Do not catch general exception types
Design Rules: CA1064: Exceptions should be public
Globalization Rules: CA1304: Specify CultureInfo
Globalization Rules: CA1305: Specify IFormatProvider
Globalization Rules: CA1306: Set locale for data types
Naming Rules: CA1705⁺: Long acronyms should be pascal-cased
Naming Rules: CA1706⁺: Short acronyms should be uppercase
Naming Rules: CA1707: Identifiers should not contain underscores
Naming Rules: CA1713: Events should not have before or after prefix
Naming Rules: CA1714: Flags enums should have plural names
Naming Rules: CA1717: Only FlagsAttribute enums should have plural names
Mobility Rules: CA1600: Do not use idle process priority
Mobility Rules: CA1601: Do not use timers that prevent power state changes
Performance Rules: CA1800: Do not cast unnecessary
Performance Rules: CA1802: Use literals where appropriate
Performance Rules: CA1805: Do not initialize unnecessary
Performance Rules: CA1822: Mark members as static
Usage Rules: CA2201: Do not raise reserved exception types
Usage Rules: CA2205: Use managed equivalent of win32 api
Usage Rules: CA2225: Operator overloads have named alternates
Usage Rules: CA2242: Test for NaN correctly

_{+ Not available in VS 2008}

Implementation

There are differences between using FxCop for Visual Studio 2003 and Visual Studio 2005/2008 development. For projects that have been upgraded from VS 2003 to VS 2005, read the following article for tips on how to migrate your existing VS 2003 FxCop exclusions file over to VS 2005 FxCop code exclusions.

VS 2003

Obtain FxCop from here. Analyses can be performed from either using the FxCop GUI directly or from the FxCop command-line utility which allows you to integrate into a continuous integration process.

VS 2005

VS 2005 adds static code analysis through integrating FxCop directly into the IDE. A developer can select to include static code analysis as part of the compilation process which will fire off FxCop and produce a list of errors/warnings in the Error list pane that prevents the code from compiling successfully. Code Analysis Policy for VS Team System can also be created that makes this code analysis process a compulsory step within the compilation process. This will prevent developers from skipping the static code analysis process.

VS 2008

VS 2008 expands and improves on the code analysis provided in the VS 2005 IDE. Some of the improvements made are:

Also be sure to check out the custom FxCop rule to support the multi-targeting feature of VS 2008.

Continuous Integration

FxCop can easily be integrated into a continuous integration process by using the standalone FxCop command-line utility for VS 2003 or by using VS 2005/2008 Code Analysis. As already mentioned, development teams using Visual Studio Team System can include Code Analysis Policy to enforce the continuous use of FxCop in their development environments.

Custom rule development

It is possible to write your own rules to integrate with FxCop running standalone or integrated into the VS IDE. Here are some articles on the topic:

Continuous Integration in .NET: From Theory to Practice

2009-11-01T20:53:00.001+02:00

NOTE: This is a repost of on old post as I am moving onto the Blogger platform

Continuous Integration (CI) is a popular incremental integration process whereby each change made to the system is integrated into a latest build. These integration points can occur continuously on every commit to the repository or on regular intervals like every 30 minutes. They should however not take longer than a day i.e. you need to integrate at least once daily. In this article I take a closer look at CI. The article is divided into two main sections: theory and practice. In the theory section, I consider some CI best practices; look at the benefits of integrating frequently and reflect on some recommendations for introducing CI into your environment. The practice section provides an in-depth example of how to implement a CI process using .NET development tooling. I conclude the article by providing some additional, recommended reading.

References

I used the following references in creating the article:

[McConnell] Code Complete, 2nd Edition by Steve McConnell.
[Fowler] Continuous Integration by Martin Fowler.
[Miller] Using Continuous Integration? Better do the "Check In Dance" by Jeremy Miller.
[Elssamadisy] Patterns of Agile Practice Adoption: The Technical Cluster by Amr Elssamadisy.
[Duvall] Continuous Integration Anti-Patterns by Paul Duvall.

Practices

Martin Fowler presents the following set of practices for CI [Fowler]:

Maintain a single source repository - Use a decent source code management system and make sure its location is well known as the place where everyone can go to get source code. Also ensure that everything is put into the repository (test scripts, database schema, third party libraries etc.)
Automate the build - Make sure you can build and launch your system using MSBuild/NAnt scripts in a single command. Include everything into your build. As a simple rule of thumb: anyone should be able to bring in a clean machine, check the sources out of the repository and issue a single command to have a running system on their machine.
Make your build self-testing - Include automated tests in your build process.
Everyone commits every day - "A commit a day keeps the integration woes away" [Duvall]. Frequent commits encourage developers to break down their work into small chunks of a few hours each. Before committing their code, they need to update their working copy with the mainline, resolve any conflicts and ensure that everything still works fine. Jeremy Miller refers to this process as the "Check In Dance" [Miller].
Every commit should build the mainline on an integration machine - Use a CI server like or CruiseControl.NET or Team Foundation Build. If the mainline build fails, it needs to be fixed right away.
Keep the build fast - If the build takes too long to execute, try and create a staged build/build pipeline where multiple builds are done in a sequence. The first build (aka ''commit build'') executes quickly and gives other people confidence that they can work with the code in the repository. Further builds can run additional, slower running unit tests, create code metrics, check the code against coding standards, create documentation etc.
Test in a clone of the production environment - Try to set up your test environment to be as exact a mimic of your production environment as possible. Use the same versions of third party software, the same operating system version etc. Consider using virtualization to make it easy to put together test environments.
Make it easy for anyone to get the latest executable - Make sure that there is a well known place where people can find the latest executable.
Everyone can see what's happening - Make the state of the mainline build as visible as possible. Use various feedback mechanisms to relate build status information [Duvall]. Some fun examples include using lava lamps, a big screen LCD and an ambient orb.
Automate deployment - Have scripts that allow you to automatically deploy into different environments, including production. For web applications, consider deploying a trial build to a subset of users before deploying to the full user base.

Jeremy Miller adds the following advice [Miller]:

Check in as often as you can - Try breaking down your workload into meaningful chunks of code and integrate these pieces of code when you have a collection of code in a consistent state. Checking in once a week seriously compromises the effectiveness of a CI process.
Don't leave the build broken overnight - Developers need to be immediately notified upon a build breakage and make it a top priority to fix a broken build.
Don't ever check into a busted build.
If you are working on fixing the build, let the rest of the team know.
Every developer needs to know how to execute a build locally and troubleshoot a broken build.

Paul Duvall also warns against some additional CI anti-patterns that tend to produce adverse effects [Duvall]. The ones that have not been covered above are:

The cold shoulder of spam feedback - Team members sometimes quickly become inundated with build status e-mails (success and failure and everything in between) to the point where they start to ignore messages. Try to make the feedback succinctly targeted so that people don't receive irrelevant information.
Don't delay feedback with a slow machine - Get a build machine that has optimal disk speed, processor, and RAM resources for speedy builds.

Benefits

Numerous benefits result from integrating continuously [McConnell]:

Errors are easier to locate - New problems can be narrowed down to the small part that was recently integrated.
Improved team morale - Programmers see early results from their work.
Better customer relations - Customers like signs of progress and incremental builds provide signs of progress frequently.
More reliable schedule estimates & more accurate status reporting - Management gets a better sense of progress than the typical "coding is 99% percent complete" message.
Units of the system are tested more fully - As integration starts early in the project the code is exercised as part of the overall system more frequently.
Work that sometimes surfaces unexpectedly at the end of a project is exposed early on

Where do I start?

Here are some steps to consider for introducing a CI process [Fowler]:

Get the build automated - Get everything into source control and make sure you can build the whole system with a single command.
Introduce automated testing in the build - Identify major areas where things go wrong and start adding automated tests to expose these failures.
Speed up the build - Try aiming at creating a build that runs to completion within ten minutes. Constantly monitor your build and take action as soon as your start going slower than the ten minute rule.
Start all new projects with CI from the beginning

.NET Implementation & Tools

I have written a set of articles that describe a complete setup of a .NET CI solution. The CI process uses a wide variety of tools including CruiseControl.NET, Subversion, MSBuild, Visual Studio 2008, NUnit, FxCop, NCover, WiX and SandCastle. You can download the series as a fully searchable PDF with PDF bookmarks and PDF links.
Team Foundation Server users can check out the new and improved out-of-the-box support for continuous integration provided by Team Foundation Server 2008. This, coupled with the recent announcement on the TFS Licensing Change for TFS 2008, makes using Team Foundation Server less expensive and a more viable option to consider.

Continuous Integration in .NET: From Theory to Practice 2nd Edition

2009-11-01T19:27:00.001+02:00

NOTE: This is a repost of on old post as I am moving onto the Blogger platform

During last year I created a guide on implementing Continuous Integration (CI) for a .NET 2.0 development environment. The guide illustrates how to create a complete CI setup using VS 2005 and MSBuild (no NAnt) together with tools like FxCop, NCover, TypeMock, NUnit, Subversion, InstallShield, QTP, NDepend, Sandcastle and CruiseControl.NET. The good news is that I spend some time during the last 2 weeks greatly improving the setup for use on a new VS 2008 project and I have decided to release a 2nd Edition of the guide covering the much improved setup. Instead of creating another series of blog posts to cover the content, I'm releasing the 2nd edition only as a downloadable PDF guide together with all the associated code and build artefacts. This will allow new teams to get up and running with CI a lot quicker.

For readers of the first edition of the guide, the most notable differences between the second edition and the first edition of the guide are:

Updated to use VS 2008, .NET 3.5 and MSBuild 3.5 (including new MSBuild features like parallel builds and multi-targeting).
All tools (NUnit, NDepend, NCover etc.) are now stored in a separate Tools folder and kept under source control. The only development tools a developer needs to install are VS 2008, SQL Server 2005 and Subversion. The rest of the tools are retrieved form the mainline along with the latest version of the source code.
Added the CruiseControl.NET configuration (custom style sheets, server setup etc.) to source control and created a single step setup process for the build server. This greatly simplifies the process of setting up a new build server.
Changed from using InstallShield to Windows Installer XML (WiX) for creating a Windows installer (msi).
Added support for running MbUnit tests in addition to the NUnit tests.
Added support for running standalone FxCop in addition to running VS 2008 Managed Code Analysis.
Added targets to test the install and uninstall of the Windows installer created.
Consolidated the CodeDocumentationBuild to become part of the DeploymentBuild.
Removed the QTP integration as this was not a requirement for the new project. If you want to integrate QTP, please refer to the QtpBuild of the first edition of the guide.
Used the latest version of all the tools available. The tools used in the guide are VS 2008, Subversion, CruiseControl.NET, MSBuild, MSBuild.Community.Tasks, NUnit/MbUnit, FxCop, TypeMock/Rhino.Mocks, WiX, Sandcastle, NCover, NCoverExplorer and NDepend.

I hope you find it to be a useful resource for assisting you with creating your own CI process by harnessing the power of MSBuild! If you have any questions, additional remarks or any suggestions, feel free to drop me a comment.

Download

Here are the links:

My Ultimate .NET Development Tools 2010 Edition

2009-10-31T22:50:00.001+02:00

Here is my 2010 updated list of development tools that I prefer to use when doing .NET development. I specifically decided to not include any third party control/report libraries. I focus instead on the tools that assist me in crafting high-quality code quickly and effectively.

Tools

* = free/open source

[IDE] Visual Studio 2010 Premium Edition
[IDE] ReSharper for refactoring, unit test runner and so much more
[IDE] CodeSmith for generating code. Also consider T4 with Clarius’s Visual T4 Editor.
[IDE]* GhostDoc for inserting xml code comments
[IDE] Altova Xml Suite for any xml related work. XmlPad is the best, free alternative I know of.
[DBMS] SqlServer 2008 for DBMS
[SCM]* Subversion for source control
[SCM]* TortoiseSVN as windows shell extension for Subversion
[SCM] VisualSVN for integration of TortoiseSVN into VS. AnkhSVN is the best, free alternative I know of.
[SCM]* KDiff3 for merging
[TDD]* NUnit as preferred xUnit testing framework
[TDD]* moq as mock framework.
[TDD] NCover for code coverage stats
[CI]* TeamCity as build server
[CI]* MSBuild Extension Pack for additional MSBuild tasks.
[FR]* log4net as logging framework. Also see Log4View for an excellent UI for the log files.
[FR]* ANTLR and ANTLRWorks for creating custom DSL’s.
[FR] PostSharp as Aspect Oriented Programming framework
[FR]* Ninject as IoC container
[FR]* RunSharp for generating IL at run-time
[FR] MindScape LightSpeed as my Object-Relational-Mapper. NHibernate is the best free alternative I’m aware of.
[UT]* Reflector to drill down to the guts of any code library (also check-out the nice plug-ins)
[UT] Silverlight Spy to dissect any Silverlight application.
[UT] RegexBuddy for managing those difficult regular expressions. Regulator is the best, free alternative I know of.
[UT]* LINQPad as a easy way to query SQL databases using LINQ and as a general scratchpad application to test C#/VB.NET code snippets.
[UT]* Fiddler to debug all your HTTP traffic in IE. Also see the neXpert plugin for monitoring performance problems.
[UT]* Firebug to assist with testing web applications running in Firefox. Also see YSlow add-on for performance testing and Web Developer add-on for additional Firefox web development tools.
[CA]* FxCop to enforce .NET coding guidelines
[CA] NDepend to get all the static code metrics I'd ever want
[CA] ANTS Profiler for performance and memory profiling
[MD] Enterprise Architect to do UML Modelling and Model Driven Design if required. Alternatively use Visio with these simple templates.
[MD]* FreeMind as mind mapping tool
[TC]* ScrewTurn Wiki for team collaboration
[QA]* Eviware soapUI for functional and load testing of SOA web services
[QA]* Telerik WebAii Testing Framework for automated regression testing of Web 2.0 apps
[DP]* Windows Installer XML (WiX) for creating Windows Installers

From The Software Development Trenches

Elementary, my dear Watson

Managing .NET RESX Duplication – Part 2: Identifying Resource Duplication

DATASTRUCTURES

Scanning Resource Files

Invoking the MSBuild Task

Conclusion

Managing .NET RESX Duplication - Part 1: Solution Requirements and Design

Requirements

SOLUTION

Stats

Phase 1

Phase 2

Summary

Pragma On Key Silverlight Localization

Static Text

Dynamic Text

Dynamically loading Resources

Conclusion

Pragma Software Development: Check-in Procedure

Pragma Software Development Tools

Pragma Software Development Team

Team

Communication

Office Space

SDLC

Sprint

Pragma On Key Software Development Environment

My 2011 Software Development Reading List

Pragma On Key – Release 3 Milestone Reached

TransactionScope Fluent Interface

Understanding MSIL Memory Leaks

Ultimate .NET Development Tools 2010 Edition

Categories

Tools

Threading in C# – Free E-book Updated

Log4View – Getting the most out of your log4net/log4j log files

RunSharp – IL Generation for Dummies

Interpreter

C# Code Generator

IL Code Generator

C# Coding Standards Using ReSharper

Code Metrics

Metrics

Size metrics

Complexity metrics

Package metrics

Coupling/Dependency metrics

Cohesion metrics

Inheritance metrics

Tools

When to use

Vacancy: Intermediate C# Software Engineer; Cape Town - South Africa

Code Metrics

Metrics

Size metrics

Complexity metrics

Package metrics

Coupling/Dependency metrics

Cohesion metrics

Inheritance metrics

Tools

When to use

Code Reviews

References

Review Types

Formal Code Review

Lightweight Code Review

Review Metrics

Review Frequency

Best Practices for Peer Code Review

Tools

Conclusion

FxCop

Introduction

Process

FxCop Backlogs

Rule sets

Base Rule Set

Minimum Rule Set