My team is getting along fine with a physical task board for tracking during sprints, but I was looking forward to drop-and-drop Product Backlog prioritisation after using the awkward Rank field in TFS 2010. Alas, JIRA stubbornly refused to behave! It just didn’t react when I attempted to drag an item to a different position. After trying and failing to get it working with the my preferred “find out why it doesn’t work” approach, I found success down the traditional “trial and error config changes” route.

I can’t yet tell you exactly why this fixed it (presumably something to do with the centrally-administered permissions configuration) but it worked for me so there is probably someone else out there with the same problem. Here’s what I did:

1. Go into the project’s GreenHopper configuration
2. Click the link to repair the ranking field (see screenshot)
3. Under General Configuration, tick the box labelled Scheduling Permission (see screenshot)
4. That’s it!

Repair the ranking field:

Scheduling permission:

.

By the way, this is JIRA 4. Hope that helps!

It looks great and I’m liking the message after creating a new Team Project:

However, there are three factors that (for some companies) reduces the price of TFS to zero while allowing them to roll it out to their entire organisation. Yes, it sounds a little too good to be true. Here are the factors:

MSDN Subscription

Visual Studio MSDN subscriptions come with one TFS server license for production use and one CAL. So team members like developers and testers, who are often provided with MSDN subs, don’t need anything more to use TFS. If you don’t get MSDN subscriptions for developers I can heartily recommend it as pretty much the whole Microsoft stack of OS, server tools, and development tools is included – upgrades and all – for development and testing purposes. Bear in mind that subscriptions can be cheaper (sometimes much cheaper) from a reseller. I can recommend Parkway Gordon.

Note: MSDN subscriptions are per-user not per-machine so a single subscription covers use at any location (office, home, customer sites) on any number of machines.

Work Item Only View

TFS users who only enter and edit their own work items, using TFS web access, don’t need a CAL. To make it easy to ensure users don’t overstep this provision, there is even a Work Item Only View security group which includes the relevant permissions and no more. This means anyone in your organisation can submit requirements, changes and bugs without impacting TFS licensing.

Reports

A recent development is that users do not need a CAL for accessing reports. So managers and other stakeholders can help themselves to project summary data and graphs without a license either.

Other Free TFS Stuff

• Team Explorer Everywhere (the Eclipse IDE plug-in) is free in itself, although a CAL is needed (Brian Harry’s blog again)
• Two system administrators can access a TFS instance for free (TFS licensing whitepaper)
• Access via Microsoft System Center Operations Manager is free (ditto)
• Static data from TFS can be distributed manually for free (ditto)
• Five users don’t need a CAL when TFS server is bought at retail (ditto)
• TFS 11 Express is completely free for up to 5 users (Brian Harry’s blog)
• Team Explorer (Visual Studio Shell) is free, although you would need a CAL (or retail/express edition exemption) if you access more than reports

Summary

A good understanding of Visual Studio and TFS licensing, canny use of security groups, and potentially free client tools (web access, Team Explorer, Visual Studio Express) can quite realistically reduce the price of TFS to zero.

If your development team have MSDN subscriptions (a good idea in any event) there might be no further cost to adopting TFS in the team, and even rolling it out to a much wider audience.

Final note: Don’t dive in based on this blog post alone – the licensing is complex so read the TFS licensing whitepaper first and clear up any questions you have.

His descriptions of organisations that were ad-hoc (reactive, unplanned) or analytic (bureaucratic, rigid) were very familiar to me and provided a good backdrop to thinking about what are known as organisational impediments in Scrum.

Organisational impediments block or slow progress – just like other impediments – but are very difficult to resolve because they are baked into the wider organisation’s structure and processes. There are good lists on the Scrum Alliance website and elsewhere but ones that chime with me include:

• Focusing on utilisation and ignoring productivity (an uncharitable but evocative term for this is busy fools)
• Unsuitable office accommodation discouraging collaboration and/or causing distraction
• Lack of organisational drive and (perhaps as a result) individual enthusiasm for getting great at developing software

Understanding what kind of organisation you are dealing with in terms of the Marshall Model categories (ad-hoc, analytic, synergistic, chaordic – or in transition from one to the other) is certainly not a silver bullet but it does help flesh out the business context. For example, an analytic organisation may have invested heavily in separate functions for business analysis, development, and testing and find the idea of cross-functional teams mystifying. Their whole mindset is reductionist – they believe that breaking things down into parts is the right way to make sense of them – so mixing it all up again can sound like heresy. Even suggesting that they try a different approach can run counter to the analytical idea that it is possible to solve an issue completely upfront by researching the problem, breaking it down, and designing the “perfect” solution.

There’s no easy way to break these organisational impediments down. I think the best approach is to go a little native – get inside the organisation, understand where it is coming from, and try to win hearts and minds. For example, in an analytical organisation, explain that the Agile approach is empirical rather than chaotic, its effectiveness is backed up by research and – above all – demonstrate that it works, even if at first that needs to be on a very small project. It is hard to argue with good results, and if you can carefully direct an organisation down the Agile path, remembering to inspect and adapt continuously, the payback in productivity, quality and morale should be plain to see.

This came about when I was rolling out TFS work items to a team that had subject matter experts regularly changing specifications for complex business rules. Another team produced configuration files based on the specifications, and both files were kept in source control. The experts often needed to review the change history and could identify which source files were relevant. I knew that changeset comments, diff tools, and the annotate feature (aka “blame allocator”) do a great job of showing what has changed, but these users wanted to know why a change was made – a business-focused description and justification. This is the job of work items, but you can’t easily generate a list of work items from a specific source file or folder.

To be fair, Visual Studio does make this information available. But to see all work items that have been linked to a source file you need to open the history, then changeset details, then the work items channel… now make a note of any linked work items and repeat for every changeset. For a source file with thirty changes that could take ten minutes or more, so is impractical for regular use. The work4source extension replaces the process with one click, and the query is performed on a separate thread so you can continue working.

If you too have felt the pain of searching for work items from source items (or are just curious) please download the tool, try it out, and let me know what you think. I hope the community find it useful and welcome contributions to its development.

Credit: This extension was coded by a friend and colleague of mine Dijon Smit, so thanks to him for agreeing to add it to Codeplex.

I discovered the IT Kanban website from the Meetup and am impressed by the quality of the materials on the site. Together with books such as David Anderson’s and the recent ALM Rangers guidance (including TFS 2010 and 11 process templates) the community support for Kanban is getting pretty strong.

If you are thinking that the TFS Power Tools Backup and Restore wizard does all this anyway, you would be right – but only if you are restoring back to the same machine that the backups came from. If you are restoring to a fresh TFS installation on a different operating system and AD domain it won’t be so simple.

A few caveats:

• This assumes you are have the source TFS instance database backups (e.g. created using the TFS Power Tools backup and restore wizard).
• I have tried to get the process as quick as possible, but I can’t guarantee that it will produce an absolutely perfect working replica. It should however be good enough to play with source control, work items and build.
• Your source and target machines will probably be configured differently from mine, so you may need to take some additional steps. I have included a few troubleshooting ideas at the end.
• I have assumed you are cloning onto an existing standalone TFS installation – with the database and application tiers together on one machine and pre-existing databases. This process will replace the databases, so make sure you don’t want to keep the existing instance!
• Some of the steps are quite hacky, so don’t be tempted to administer your production TFS instance this way!

Here goes:

1. Copy the database backups to local machine
2. Open a command prompt and issue:

   cd C:\Program Files\Microsoft Team Foundation Server 2010\Tools
   TFSServiceControl.exe quiesce

3. Start SQL Server Management Studio
   1. Note the physical location of the TFS databases e.g. C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA
   2. Detach all TFS databases:
      1. TFS_*
      2. ReportServer
      3. ReportServer_TempDB
      4. SharePoint_Config
      5. WSS_Content
   3. Delete, rename or move the physical files for the TFS databases
4. Start the TFS Admin Console
   1. Click Team Foundation Backups on the left
   2. Click Restore Databases on the right
   3. Follow the Restore Wizard
      1. Choose the Network Path to restore from
      2. Modify the SQL server instance to localhost
   4. Use SQL Server Management Studio to manually restore any failed databases
5. Start SQL Server Management Studio
   1. Locate the Tfs_Configuration database, right-click on it and select Properties
   2. Click Extended Properties on the left
   3. On the right locate the row for TFS_BACKUP_PLAN_CONTROLLER
   4. Change the value to the new server name e.g. localhost
   5. Locate the row for TFS_INSTANCE
   6. Make a note of the GUID value
6. Repoint TFS web services to the new instance
   1. Navigate to folder C:\Program Files\Microsoft Team Foundation Server 2010\Application Tier\Web Services
   2. Open web.config in a plain text editor
   3. Locate the applicationId key
   4. Change the value to the TFS_INSTANCE value noted in the step above
7. Open a command prompt and issue:

   cd C:\Program Files\Microsoft Team Foundation Server 2010\Tools
   TFSServiceControl.exe unquiesce
   TFSConfig remapdbs /databaseName:localhost;Tfs_Configuration /sqlInstances:localhost /continue
   TFSConfig identities /change /fromdomain:{SourceDomain} /todomain:{TargetDomain} /account:{SourceAdminAccount} /toaccount:{TargetAdminAccount}

   The last command (TFSConfig identities) is intended to repoint an account with TFS administrative privileges on the source instance with an account that is accessible on the target machine (your own account is the obvious choice). Repeat it for any additional accounts which you want to use against the cloned instance. If the target machine is not in a domain, set {TargetDomain} to the machine name.

8. Start the TFS Admin Console
   1. Click Application Tier on the left
      1. Add a user account accessible on the target machine to Administration Console Users (e.g. your own account)
      2. Click Change URLs on the right
      3. Change the notification URL to http://localhost:8080/tfs, test, then click the OK button
      4. Click Reapply Account on the right, enter the password, then click OK
   2. Click Build Configuration on the left
      1. Reconfigure the build service as needed
9. Clear the TFS application tier cache by deleting the contents of C:\Program Files\Microsoft Team Foundation Server 2010\Application Tier\Web Services\_tfs_data
10. Clear the Visual Studio TFS cache by deleting the contents of C:\Users\Andy\AppData\Local\Microsoft\Team Foundation\3.0\Cache
11. Start Visual Studio and connect to the cloned instance

That should do it!

Troubleshooting

• The clone will have the same instance ID as the original instance, and if you are able to connect to both from the same machine you may notice some strange behaviour due to the cache not being able to differentiate between them. If this is a problem look into TFSConfig changeserverid.
• If the restore reports that it failed to restore Reporting Services encryption key it may be because the Reporting Services database is detached at that point. Use rskeymgmt -a -f [KEYFILE] -p [PASSWORD] or Reporting Services Configuration Manager to restore the key instead (if you want to use Reporting Services).
• If you get a message saying that the service account does not have access to SQL Server, start SQL Server Management Studio and issue the SQL grant connect sql to “domain\account”
• If you get a message Failed to add ‘AccountName’ account to the TFSEXECROLE role on the Tfs_Warehouse relational database follow the instructions at http://support.microsoft.com/kb/2025896
• If you get TF30040: The database is not correctly configured when connecting from Visual Studio it may indicate that the service account does not have permission to access the database(s). Give the account permission or change the service account.
• If you get TF14061 Workspace does not exist it might indicate that the user account which owns the workspace has been remapped. Use tf workspace /newowner or the workspaces GUI to remap the workspace.

The Goal

The aim here is to:

1. Create a sample application
2. Create a build definition for the application
3. Customise the build process to stamp the application’s output assembly with the full build number, so you only need the assembly to trace right back to source code and work items

The customisation also includes a common versioning approach, which is well documented in the ALM Rangers Build Customisation Guide. The ALM guide is targeted to intermediate to advanced users, whereas this is more of a tutorial for developers with little exposure to build.

I will assume you have already installed the build service and created a build controller and build agent.

Sample Application

This is going to be a console application that simply outputs the assembly properties we are interested in.

1. Start Visual Studio and create a new C# Console Application called SampleApplication or similar (VB.NET would be fine, but the sample code below is C#). Tick the box to add it to source control.
2. Open the AssemblyInfo.cs file and add some text within the AssemblyDescription that you will recognise – anything will do. Delete the existing comment which has an AssemblyVersion example as it will confuse the versioning workflow that comes later.
3. Open the Program.cs file, paste in the code below, then run the application to see the output.
4. Check in to source control.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Reflection;
using System.Diagnostics;

namespace AssemblyVersioning
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine("AssemblyVersion: {0}",
Assembly.GetExecutingAssembly().GetName().Version.ToString());
Console.WriteLine("AssemblyFileVersion: {0}",
FileVersionInfo.GetVersionInfo(Assembly.GetExecutingAssembly().Location).FileVersion.ToString());
Console.WriteLine("AssemblyDescription: {0}",
FileVersionInfo.GetVersionInfo(Assembly.GetExecutingAssembly().Location).Comments.ToString());

Console.ReadLine();
}
}
}

Build Definition

We will set up a build using the default template to make sure it works before we start changing things. Note that there are many other configuration options available, but those below are the minimum required for now.

1. In Visual Studio open the Team Explorer panel.
2. Expand your Team Project, then right-click on Builds and select New Build Definition…
3. Select General on the left. Give the definition a name.
4. Select Trigger on the left. Leave this as Manual.
5. Select Workspace on the left. Under the Source Control Folder heading modify the location to the folder containing the sample application solution file. The build system will get the latest version of everything under this folder (and it’s also the trigger folder for continuous integration builds).
6. Select Build Defaults on the left. Ensure the build controller is selected, then enter a folder in the drop location field. Under live use this would normally be a file share accessible to the team, but for this purpose could be local to the build machine. Note that even if it’s a local folder it needs to be in the \\machinename\fileshare format.
7. Select Process on the left. Expand Items to Build and set the Projects to Build to the sample application’s solution file. If the sample application is still open, this will have been populated for you.
8. Save the build definition.
9. In Team Explorer, right-click the new build definition and select Queue New Build then click the Queue button.
10. The Build Explorer will open and you can double-click the running build to see its progress.
11. Go to the drop location configured above and verify that the compiled application is present and runs OK.

Custom Build Activity

The template changes depend on a custom build activity, so we will need to get that in place first.

1. Create a new source control folder called CustomActivities (or similar). (A logical location would be under the BuildProcessTemplates, or you could make both “templates” and “activities” child folders of a new BuildProcess folder.)
2. Download the Community TFS Build Extensions. Open the zip file, copy the contents of the Visual Studio 2010 folder into the new CustomActivities folder, then add the files to source control and check-in.
3. In Visual Studio open the Team Explorer panel.
4. Right-click on Builds and select Manage Build Controllers…
5. Select your build controller then click the Properties button
6. Under Version control path to custom assemblies… enter the path to the CustomActivities folder
7. Click OK
8. You will now be back at the Manage Build Controllers dialog and will see that the controller and agent are both offline. This happens while new build activities are being processed. After a short while the controller then the agent should come back online.

Process Template

Now we will create a customised process template and add a custom versioning activity to it.

1. Locate the build template DefaultTemplate.xaml in source control, by default this is under the BuildProcessTemplates folder.
2. Make a copy of the file and rename it to CustomTemplate.xaml (or similar). Add to source control but don’t check-in yet.
3. Create a new C# class library called BuildTemplates and put it under the BuildProcessTemplates folder. Tick the box to add it to source control.
4. Delete the Class1.cs file.
5. In Solution Explorer right-click the project and select Add -> Existing Item then select the CustomTemplate.xaml file. Click the small downwards pointing triangle on the Add button  and select Add As Link.
6. Add references to all the DLLs in the CustomActivities folder.
7. Add references to the following assemblies on the .NET tab of the Add Reference dialog: System.Drawing, System.Xaml, Microsoft.TeamFoundation.Build.Client, Microsoft.TeamFoundation.VersionControl.Client, Microsoft.TeamFoundation.WorkItemTracking.Client
8. Add references to the following assemblies located in folder C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\: Microsoft.TeamFoundation.Build.Workflow, Microsoft.TeamFoundation.TestImpact.BuildIntegration
9. Add a reference to the following assembly located in folder C:\Windows\assembly\GAC_MSIL\Microsoft.TeamFoundation.TestImpact.Client\10.0.0.0__b03f5f7f11d50a3a\: Microsoft.TeamFoundation.TestImpact.Client
10. Double-click CustomTemplate.xaml to open the workflow editor.
11. Open the Toolbox panel. You should see many workflow activities organised into different tabs.
1. Right-click within the Toolbox and select Add Tab… then name the tab Community TFS Build Extensions
   
2. Right-click the new tab then select Choose Items…
   
3. Click the Browse button and navigate to the BuildActivities folder.
4. Select TfsBuildExtensions.Activities.dll then click the Open button.
5. Click the OK button
6. Right-click the toolbox tab again and select Sort Items Alphabetically (recommended)
12. You should now be back at CustomTemplate.xaml in the editor. Click the Arguments tab at the bottom.
13. Add three new arguments VersioningFilespec, DisableVersioning and VersioningStartDate with types and defaults as shown under the Arguments heading below.
14. Still in the list of arguments locate the Metadata entry and click the ellipsis button (three little dots) in the Default value column. Add three entries, one for each new argument, as shown under the Metadata heading below. In addition, set the View this parameter when: drop-down for all three to Always show the parameter.
15. Now you need to edit the workflow.
1. Locate the activity Run On Agent then expand it
2. Within Run On Agent locate Initialize Workspace
3. From the Control Flow tab of the toolbox drag an If activity right after Initialize Workspace (before If CreateLabel)
   
4. Open the properties for the If activity and set the DisplayName to If Versioning and the Condition to DisableVersioning = False
5. Expand the If Versioning activity so you can see the Then and Else boxes
6. From the Control Flow tab of the toolbox drag a Sequence activity into the Then box then set its DisplayName to Versioning
7. This should have given you some understanding of how to modify the workflow, but it would be tortuous to continue this way so we will cut-and-paste to finish this work
8. Save and close CustomTemplate.xaml then right-click the file in Solution Explorer and select View Code.
9. Locate the root Activity node and add a new namespace attribute as follows: xmlns:tat=”clr-namespace:TfsBuildExtensions.Activities.TeamFoundationServer;assembly=TfsBuildExtensions.Activities”
10. Find the text If Versioning then replace the whole If element with the XML shown under the heading Workflow XML below.
11. New close the file and re-open in the Workflow editor again. The versioning workflow should look like the example under the Workflow heading below.
12. Check the template and project into source control.

Note: The forthcoming Windows Workflow Foundation 4.5 has a search function which will make finding your way around the workflow much easier

Arguments

.

Metadata

.

Workflow XML

<If Condition="[DisableVersioning = False]" DisplayName="If Versioning" sap:VirtualizedContainerService.HintSize="552,995" mtbwt:BuildTrackingParticipant.Importance="Low">
<sap:WorkflowViewStateService.ViewState>
<scg:Dictionary x:TypeArguments="x:String, x:Object">
<x:Boolean x:Key="IsPinned">True</x:Boolean>
<x:Boolean x:Key="IsExpanded">True</x:Boolean>
</scg:Dictionary>
</sap:WorkflowViewStateService.ViewState>
<If.Then>
<Sequence DisplayName="Versioning" sap:VirtualizedContainerService.HintSize="427,889">
<Sequence.Variables>
<Variable x:TypeArguments="scg:IEnumerable(x:String)" Name="VersioningFiles" />
<Variable x:TypeArguments="x:String" Name="major" />
<Variable x:TypeArguments="x:String" Name="minor" />
<Variable x:TypeArguments="x:String" Name="version" />
<Variable x:TypeArguments="x:String" Name="filecontents" />
</Sequence.Variables>
<sap:WorkflowViewStateService.ViewState>
<scg:Dictionary x:TypeArguments="x:String, x:Object">
<x:Boolean x:Key="IsExpanded">True</x:Boolean>
<x:Boolean x:Key="IsPinned">True</x:Boolean>
</scg:Dictionary>
</sap:WorkflowViewStateService.ViewState>
<mtbwa:FindMatchingFiles DisplayName="Find Versioning Files" sap:VirtualizedContainerService.HintSize="405,22" mtbwt:BuildTrackingParticipant.Importance="Low" MatchPattern="[SourcesDirectory &amp; String.Join(&quot;;&quot; &amp; SourcesDirectory, VersioningFilespec)]" Result="[VersioningFiles]" />
<ForEach x:TypeArguments="x:String" DisplayName="For Each File" sap:VirtualizedContainerService.HintSize="405,703" mtbwt:BuildTrackingParticipant.Importance="Low" Values="[VersioningFiles]">
<sap:WorkflowViewStateService.ViewState>
<scg:Dictionary x:TypeArguments="x:String, x:Object">
<x:Boolean x:Key="IsPinned">True</x:Boolean>
<x:Boolean x:Key="IsExpanded">True</x:Boolean>
</scg:Dictionary>
</sap:WorkflowViewStateService.ViewState>
<ActivityAction x:TypeArguments="x:String">
<ActivityAction.Argument>
<DelegateInArgument x:TypeArguments="x:String" Name="file" />
</ActivityAction.Argument>
<Sequence DisplayName="Version File" sap:VirtualizedContainerService.HintSize="375,597" mtbwt:BuildTrackingParticipant.Importance="Low">
<sap:WorkflowViewStateService.ViewState>
<scg:Dictionary x:TypeArguments="x:String, x:Object">
<x:Boolean x:Key="IsExpanded">True</x:Boolean>
<x:Boolean x:Key="IsPinned">True</x:Boolean>
</scg:Dictionary>
</sap:WorkflowViewStateService.ViewState>
<mtbwa:WriteBuildMessage sap:VirtualizedContainerService.HintSize="353,22" Importance="[Microsoft.TeamFoundation.Build.Client.BuildMessageImportance.High]" Message="[String.Format(&quot;Versioning file {0}&quot;, file)]" mva:VisualBasic.Settings="Assembly references and imported namespaces serialized as XML namespaces" />
<Assign DisplayName="Read file contents" sap:VirtualizedContainerService.HintSize="353,58" mtbwt:BuildTrackingParticipant.Importance="Low">
<Assign.To>
<OutArgument x:TypeArguments="x:String">[filecontents]</OutArgument>
</Assign.To>
<Assign.Value>
<InArgument x:TypeArguments="x:String">[System.IO.File.ReadAllText(file)]</InArgument>
</Assign.Value>
</Assign>
<Assign DisplayName="Get major version" sap:VirtualizedContainerService.HintSize="353,58" mtbwt:BuildTrackingParticipant.Importance="Low">
<Assign.To>
<OutArgument x:TypeArguments="x:String">[major]</OutArgument>
</Assign.To>
<Assign.Value>
<InArgument x:TypeArguments="x:String">[New System.Text.RegularExpressions.Regex(".*?assembly: AssemblyVersion\(""(?&lt;major&gt;[\d]*)\.(?&lt;minor&gt;[\d]*)").Match(filecontents).Groups("major").Value]</InArgument>
</Assign.Value>
</Assign>
<Assign DisplayName="Get minor version" sap:VirtualizedContainerService.HintSize="353,58" mtbwt:BuildTrackingParticipant.Importance="Low">
<Assign.To>
<OutArgument x:TypeArguments="x:String">[minor]</OutArgument>
</Assign.To>
<Assign.Value>
<InArgument x:TypeArguments="x:String">[New System.Text.RegularExpressions.Regex(".*?assembly: AssemblyVersion\(""(?&lt;major&gt;[\d]*)\.(?&lt;minor&gt;[\d]*)").Match(filecontents).Groups("minor").Value]</InArgument>
</Assign.Value>
</Assign>
<tat:TfsVersion AssemblyVersion="{x:Null}" Build="{x:Null}" DateFormat="{x:Null}" FailBuildOnError="{x:Null}" IgnoreExceptions="{x:Null}" Revision="{x:Null}" TextEncoding="{x:Null}" TreatWarningsAsErrors="{x:Null}" Action="GetAndSetVersion" AssemblyDescription="[BuildDetail.BuildNumber]" CombineBuildAndRevision="False" Delimiter="." DisplayName="Set Version" Files="[{file}]" ForceSetVersion="False" sap:VirtualizedContainerService.HintSize="242,22" mtbwt:BuildTrackingParticipant.Importance="Low" LogExceptionStack="True" Major="[major]" Minor="[minor]" PaddingCount="0" PaddingDigit="" SetAssemblyDescription="True" SetAssemblyFileVersion="True" SetAssemblyVersion="False" StartDate="[VersioningStartDate]" UseUtcDate="False" Version="[version]" VersionFormat="Elapsed" VersionTemplateFormat="0.0.0.0" />
<mtbwa:WriteBuildMessage sap:VirtualizedContainerService.HintSize="353,22" Importance="[Microsoft.TeamFoundation.Build.Client.BuildMessageImportance.High]" Message="[String.Format(&quot;Version has been set to {0}&quot;, version)]" mva:VisualBasic.Settings="Assembly references and imported namespaces serialized as XML namespaces" />
</Sequence>
</ActivityAction>
</ForEach>
</Sequence>
</If.Then>
</If>

Workflow

Update Build Definition

Now we need to modify the build definition created earlier to use the customised template.

1. Right-click the build definition created earlier and select Edit Build Definition…
2. Select Process on the left. On the right under Build process template click Show details.
3. Click the New button then choose the option to Select an existing XAML file
4. Choose CustomTemplate.xaml then click the OK button
5. You will notice the new Versioning parameters but don’t need to edit them – the defaults will work fine.
6. Save the build definition.
7. Right-click the build definition and select Queue New Build the click the Queue button.
8. Once the build has finished:
1. In the build output click View Log and find the message saying that the version has been set
2. Go to the drop location and check that the file version matches the one generated by the build.
3. Also check that the file comments contains the full build number. Prior to Windows 7 you could simply bring up the file properties to see this. In Windows 7 it is frustratingly not shown in Windows Explorer but can be displayed using Powershell – change the directory to the drop folder containing the executable then run (dir “SampleApplication.exe”).VersionInfo.Comments

How it Works

After the build agent retrieves the latest files from source control, the process locates files matching the pattern specified and uses community build extension TfsVersion to do the rest – remove the read-only flag on the file, set the version and description, then put the read-only flag back on to avoid upsetting future gets from source control. Compilation then proceeds as normal, oblivious to the fact that something has been changed behind the scenes.

• When you cut corners in software development you accumulate a “debt”
• The additional effort required to do the job right is the amount of the debt
• “Interest” must be paid, in the form of additional resources needed to support and maintain the software

Anyone who has coded for a while won’t need persuading that quick and dirty hacking is a false economy, but it is tough to explain a programmer’s gut feeling for good and bad code to those who shy away from Excel formulas. However, bring in the idea of technical debt and they should start to get it pretty quickly. I introduced the concept in a recent role and found it very useful – I circulated articles, banged on about it at the drinks machine, and invited the team to an online seminar by a leading expert. They were getting it.

But the online seminar concerned me. The presenter was talking about quantifying technical debt, making trade-offs, and strategic technical debt. The idea of technical debt doesn’t stand up to being stretched this far, and it is dangerous when it happens because the resulting decisions will be based on a broken model. Only luck will prevent those decisions from turning out bad. Here’s where I think the analogy breaks down:

.
Looking at it soberly, technical debt is not much like financial debt at all. It’s an enlightening metaphor, no more.

What we are really trying to get people to understand through technical debt is this: quality matters, ignore it and your software will rot. It will become unreliable, risky and expensive to support and maintain, difficult to extend, and demoralising to work with. You might be able to hide the symptoms from customers by pouring money into the problem, but if that becomes uneconomical the cracks will open up and your reputation may suffer. What business wants that?

* * * * *

While writing this I came across a recent article on the topic from Better Software magazine – which is excellent further reading.

But even these monstrosities can benefit from modern ALM practices such as automated build and deployment. So in this post I’m going to talk about automated deployment of a real-life legacy web application using scripting, Psexec and other command-line tools – all coordinated by TFS 2010.

The Scenario

The application had been developed chaotically by many people over a decade or more using every Microsoft technology you could name: C++, VB, VBScript, VBA, C#, .NET 1.1, .NET 2.0, ASP, ASP.NET… you name it, it was in there somewhere. There had never been a rewrite, re-architecting or even any hints of refactoring. Yet development on this key application continued, and while there was an appreciation that things needed to improve there was certainly no appetite for any major rework.

The Challenge

The goal was to improve quality and productivity, and reduce risk. I know it’s a cliché, but the way to eat an elephant is one bite at a time, and that is how I approached this problem. The steps below were taken to tame this beast, tentatively, over several months while development continued relatively uninterrupted:

1. Migrated source control from Visual SourceSafe to TFS 2010 (there is plenty of documentation on this, just make sure you practice before doing the real migration).
2. Productised the application – defined the elements that it is (and isn’t) made of, and got them all organised in source control to establish a “single source of truth”.
3. Migrated issue tracking from a proprietary system to TFS 2010 Work Items so changes going forward are transparently tracked.
4. Got the application to compile with a minimal (but fully documented) set of components required to be installed. Later work cleaned up referenced dependencies, which were scattered and sometimes in conflict.
5. Introduced TFS automated builds, starting with continuous integration.

So far, so good. By this point, we knew what we were dealing with and were building it on every check-in. There were even unit tests – which were patchy in coverage and quality, but much better than nothing.

The next step was to get this caged monster deploying to a test environment, fully automated, starting from a standard base configuration. The existing  process was documented as manual steps over dozens of pages, and it still wouldn’t work first time – days would be lost “setting it up” after following the install procedure. If you can’t easily deploy your application onto fresh hardware, then something is seriously wrong.

The benefits of a clean and endlessly repeatable deployment process are many, but getting there wouldn’t be easy.

WebDeploy

An early decision was made to restrict the deployment to a collection of ASP.NET web applications, as this was the more modern part of the app, and under constant development. The latest and greatest technology for this is the unimaginatively named Web Deployment Tool, also known as WebDeploy and MSDeploy, which is built on top of MSBuild – Microsoft’s core compilation technology.

We spent a good deal of time working with WebDeploy, but it turned into a battle. The constraints of old application and OS technology, and multiple web applications were a big headache and the pretty Visual Studio tooling soon fell short, leaving us working directly with MSBuild files.

Personally, I like the change of direction in TFS 2010 to restrict MSBuild to compilation, and use Windows Workflow for build “orchestration”. Once you go beyond compilation and directly related activities, changing XML-based MSBuild scripts feels awkward compared to the intuitive free-flow of WF (the abbreviation is not WWF – I think even Microsoft is scared of the World Wide Fund for Nature). So we ditched WebDeploy and decided to roll our own TFS Build and scripting solution.

The Solution

I came across many issues as I put our deployment solution together, so I will set the whole process out here and pick out the problems.

When to Deploy

The default TFS build process template is necessarily complex, and I intuitively put an Invoke Process activity to start the deployment after the steps which compile and test the solution. However, our solution had functional tests (calling high-level methods which use web services and the database) so you would get test failures because the tests relied on the deployment itself. So I simply moved the deployment step before the tests, and ensured any parts which the tests depended on would run synchronously so they finished before the tests started (more on this later).

There is a potential downside to this, which is that the compile and test steps are within a for-each loop for each solution and configuration combination. If you compile more than one solution and/or configuration in your build definition (which I personally don’t like to do) it will get messy. The deployment would run multiple times, but if you added a condition of some sort to ensure it runs only after the last compilation, it might be too late for any tests which depend on it. I didn’t need to deal with this issue, but controlling the order of compilation and breaking up the deployment seems like a solution. Another option would be to have multiple builds, each with a single solution and configuration, and have one build trigger the next.

Passing Parameters

It is very useful to pass information from the build definition and build system through to the deployment scripts. To achieve this I created a handful of new arguments in a new “Deployment” category, including:

• An on/off toggle
• Email addresses to notify of build completion
• Processes to invoke to do the deployment
• Text about the deployment to display in the build log

The processes argument could include placeholders in curly braces, which the build template was customised to recognise and replace appropriately. For example, placeholders of {BUILDNUMBER} and {BUILDID} would be substituted by the actual build number and numeric build ID respectively. The scripts could use this information to stamp the website with a build number and send an email with a link to the TFS Web Access version of the build log. (Incidentally, it’s an annoyance for a pedant like me that the build number is not a number, that’s the build ID – am I alone in being bothered by that?)

.

Strategy

I settled on a fairly simple process that worked for the first application and all those it was extended to. It used a file share on the target server as a working folder, and went like this:

1. Clear the file share on the target server
2. Copy the scripts and any other files to the file share on the target server
3. Repeat steps 1 and 2 if there are multiple servers in the deployment
4. Stop application websites and start the maintenance website on the remote server(s). I used Psexec for this initially but found it unreliable (more on that later) so switched to iisweb with much better results
5. Package the application into one or more zip files.
1. I used the 7-Zip command line tool 7za.exe, and a combination of copy and xcopy. Temporarily I used the incredibly flexible robocopy to deal with the horrible consequences of referencing different versions of the same assembly from different projects in the same solution – the application was fixed properly later.
2. Any web.config or app.config transforms will need to take place before this step. I posted previously about how to trigger this without a full publish action.
6. Copy the zip package(s) to the target server(s)
7. Run the deployment actions in an inside-out sequence. So in a three-tier architecture: database, then the application layer, then the UI layer.
• I used Psexec to asynchronously hand over the deployment to each target server. This takes the load off the build server and means the deployment runs as a process local to the server, which can be helpful for security and performance reasons.
• If you want the build server to wait for the some or all of the deployment to finish, either run the deployment remotely from the build server or use a signal such as a “semaphore file” to tell the build server when the process is complete (see the ALM Rangers build guidance for more on that).
• If using 7-Zip to extract zip files use the -y switch to avoid prompts hanging the process.
8. When the deployment actions were complete, the final server in the deployment sequence was scripted to:
1. Stop the maintenance website and start the application’s website
2. Copy all the generated logs into the build drop folder
3. Send an email notification of the deployment

Preparatory steps and signposting of deployment steps were output into the build log itself, but process calls with extensive output were set to use log files instead. The build log doesn’t lend itself well to very verbose logging.

Psexec

This is a command line tool from Sysinternals which performs the magic of triggering a process on a remote machine. It is very temperamental about how it is used but not unreliable once you find a stable usage pattern. Here are the lessons I learned:

• Admin shares (\\server\C$ style file shares) need to be enabled on the remote machine.
• This requires some one-time config changes if the server has been hardened against it (which should only be done on an internal network with appropriate approval). I needed to set Server and Computer Browser services to run automatically, set registry key HKLM\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters to 1, and kill/restart the explorer.exe process.
• The full path to Psexec is needed for it to be found by the build agent (even if it is under the PATH environment variable)
• cmd /c is necessary to run Psexec from an InvokeProcess build activity as it is a console command, not a standalone executable
• Psexec returns its own output as StdErr, causing the build log to contain an error for every line of output – and failing the whole build. You can use output handler 2>&1 to push StdErr into StdOut.
• Psexec likes to output empty lines but you can strip them out within (or before) a WriteBuildMessage activity
• For reliability I found I needed switches -d and -accepteula to run non-interactively and accept the embedded license agreement (the latter is common to Sysinternals tools).
• I used Psexec only once per server, which seemed necessary to avoid overlapping processes (I don’t know why that should have been a problem, but it caused poor reliability). Having this constraint actually helped ensure the deployment process was well structured.
• If you don’t specify a user account, Psexec uses impersonation which may result in the remote machine lacking access to network resources. This can be avoided by explicitly passing the username and password.
• To pass the output of a remotely executed command to a log requires a middle-man script on the remote server. If you try to log on the calling server, you will get the output of Psexec itself, not the command being called.

Shared Deployment Scripts

This deployment strategy was later reused for other applications. To avoid duplication of componentised scripts and other shared items, I moved them to their own source folder and included them in the workspace of the build definition. You can’t map two source folders into the same working folder, so you have to figure out the correct relative paths, but it’s a small price to pay for the elimination of duplication (it’s evil!)

Summary

I will readily admit that this deployment approach is a little archaic. It’s also littered with gotchas, but new technology is not immune from that either.

Going forward, WebDeploy is still my first choice whenever possible. But if faced again with a trade-off between extensive MSBuild scripting and relatively transparent TFS Build + Windows scripting, I would still go for the latter.