Damir's Corner

EventLogTraceListener Can Cause an Application to Crash

Damir Arh — Wed, 01 Sep 2010 14:56:09 GMT

Trace listeners are a great mechanism for troubleshooting and monitoring applications in production environment. After you have decorated your code with the necessary Trace and Debug calls, you only have to add the desired listeners either in code or in the configuration file and voila: the application starts emitting information to the configured destinations. Usually there is no need to have trace listeners attached all the time, you can only add them to the configuration file when you need to troubleshoot a problem.

What I didn’t know until recently, is that by adding a trace listener to your application you can cause it to crash. That’s definitely not something I wanted or expected to happen! After investigating the issue further the offender turned out to be EventLogTraceListener which (obviously) logs information to the event log. To do that the event log source is required and therefore it needs to be specified when adding (initializing) the EventLogTraceListener. And here lies the root of the problem.

To register a new event log source, administrative privileges are required. If the event log source is not yet registered and the user running the application is not an administrator, the call to Trace will raise a security exception. If this exception is not handled properly, the application will crash. To be on the safe side all Trace and Debug calls in your application should be in a try/catch block. And I have often seen this not being the case, in particular when the calls are already placed in a catch block. The other option is of course to ensure that the account has administrative privileges or that the event log source is already registered. But is it really possible to be 100% sure of that in production environment?

What I find surprising is that there doesn’t seem to be any consistency between different trace listeners provided in the framework. TextWriterTraceListener has a similar security related problem when the user doesn’t have write permissions for the specified log file location. But this situation is handled in the trace listener itself which simply doesn’t write the information to the log file if it can’t. It doesn’t raise any exceptions which (at least for me) is the expected behavior. Why is this not the case with EventLogTraceListener?

Using FTP Task in CruiseControl.NET for Publishing Websites

Damir Arh — Sat, 07 Aug 2010 19:19:23 GMT

Since version 1.5 CruiseControl.NET includes an FTP Task which can be used for uploading build results to a remote FTP server. Its most typical use case is for publishing websites to a remote server. Configuring this correctly requires some thought and at the moment there are very few helpful resources available. Below are some suggestions for the most simple case when the files to be uploaded only have to be retrieved from the source control and there is no additional build process involved.

The core of the configuration is of course the FTP task. Its configuration is pretty straightforward. You only need to supply it with login details and the path to the local and the remote folder. This also means that there are almost no configuration options available – you can only choose between uploading the files recursively or not.

The problem with the lack of options is that most source control systems put additional files in the local folder which you don’t want to upload. In the case of Subversion this is a .svn folder in every local folder. Because the FTP task can’t be configured to skip or ignore certain files, these files have to be deleted before the FTP task is executed. The best way to do that is by executing a batch file:

FOR /F "tokens=*" %%G IN ('DIR /B /AD /S *.svn*') DO RMDIR /S /Q "%%G"

If you do this directly in the working directory where the files are retrieved from Subversion, then the next retrieval from source control will fail unless you set the cleanCopy configuration element to true and retrieve all of the files from Subversion every time. Doing this has two downsides:

If the project is large, this will significantly decrease performance. Downloading lots of files from Subversion takes time.
The file timestamps will be reset every time. This information could potentially be useful for the FTP task (but isn’t yet as I explain at the end of the post).

To avoid this you should first copy the retrieved files to a different folder. The best way to do this is by using the build publisher. Just don’t forget to set the alwaysPublish configuration element to true. Otherwise it won’t copy anything because you are using it in the tasks section where the build is not yet successful.

To sum it all up, your tasks section should something look like this (I’ve used preprocessor constants for most of the values to increase readability):

<tasks>
    <buildpublisher>
        <sourceDir>$(WorkingDir)</sourceDir>
        <publishDir>$(PublishDir)</publishDir>
        <cleanPublishDirPriorToCopy>true</cleanPublishDirPriorToCopy>
        <useLabelSubDirectory>false</useLabelSubDirectory>
        <alwaysPublish>true</alwaysPublish>
    </buildpublisher>
    <exec>
        <executable>$(ClearSvnFilesBatch)</executable>
        <baseDirectory>$(PublishDir)</baseDirectory>
    </exec>
    <ftp>
        <serverName>$(FtpServerName)</serverName>
        <userName>$(FtpUserName)</userName>
        <password>$(FtpPassword)</password>
        <action>UploadFolder</action>
        <ftpFolderName>$(FtpRemoteFolder)</ftpFolderName>
        <localFolderName>$(PublishDir)</localFolderName>
        <recursiveCopy>true</recursiveCopy>
    </ftp>
</tasks>

I’ll conclude this post with a bad news. Unfortunately there is no support in the FTP task to only upload the files that have changed since the previous build. For a small website this might not be an issue but for larger ones this is quite a problem. You really don’t want to upload tens or even hundreds of megabytes every time. Let’s just hope that the issue will be fixed soon.

Avoiding Queue Starvation in CruiseControl.NET

Damir Arh — Wed, 28 Jul 2010 20:05:47 GMT

Recently I’ve been solving some issues with queue starvation in CruiseControl.NET. Since I haven’t found much information about this problem I’ve decided to round up my findings in this blog post.

So, what is queue starvation? It’s a problem that can occur when scheduling tasks in priority queues if there are too many high priority tasks so that lower priority tasks never get scheduled and are just waiting indefinitely. For example, let’s say we have three queues: high, medium and low priority. Tasks in the high priority one always get processed first. Once this queue is empty the tasks from the medium priority queue get processed. Of course the tasks in the low priority queue get processed only if the other two queues are empty. If there are too many high and medium priority tasks incoming all the time so that the queues never get empty, then the low priority tasks are not processed at all.

And how is this related to CruiseControl.NET? As you might be aware, projects can be assigned to different queues. Only projects in the same queue (as the name implies) are built sequentially while projects in different queues are built in parallel to one another. Queues themselves therefore cannot cause starvation because different queues don’t depend on one another and projects are always put at the end of its queue. This means that sooner or later each project will be built.

Now, queue priorities enter the stage. Each project can be given a priority defining where the project will be put into the queue: before all of the lower priority projects already in the queue instead of at the end. Effectively this partitions a single sequential queue into multiple queues with different priorities as described at the beginning of the post. We have all the necessary prerequisites for queue starvation. If too many projects with higher priorities have to be built (typically caused by checking in code or triggering a forced build), the projects with lower priorities will never get processed.

What I have observed is that this can happen even if higher priority projects don’t have to be built at all. The reason for that is in the way CruiseControl.NET is processing the projects to determine whether there were any source code changes. This is taken care of with an interval trigger which puts a project in the queue, but it only gets built if a modification exists. Nevertheless, even checking if such a modification does exists, takes some time. In extreme cases this can be enough to cause queue starvation and this is also what happened in my case. Lower priority projects never got built, not even during the night and no matter how often a build was forced for them.

Fortunately there is a solution for this problem: the seconds attribute which specifies how often the project is put into the queue for checking if a modification exists. By default it is set to 60 seconds. Increasing the value can solve the queue starvation issue. Instead of simply defining the trigger:

<intervalTrigger />

A longer period can be specified:

<intervalTrigger seconds="300" />

Of course this has its downside: the latency before the project gets built after a modification increases. Therefore you should use this solution with caution. Only increase the value as much as you have to. And don’t do it at all if you can apply any of the other solutions:

Increase the performance of the build server to speed up the processing and building of the projects.
Parallelize the build process by using more queues. Put the projects in the same queue only if you have to.
Don’t use queue priorities if it’s not necessary. In most cases you can get by without them.

Programmatic Manipulation of Binary Excel (.xls) Files

Damir Arh — Sun, 13 Jun 2010 11:47:10 GMT

A few months ago I worked on a small spare time project which included some manipulation of binary Excel (.xls) files. This seemingly simple task soon turned out to be quite a challenge if you want to handle it right. The post you are reading is a short summary of my experiences. They should make your choices easier if you are about to tackle a similar problem.

The most obvious choice for handling .xls files is Excel automation using the Excel object model. As long as your application is always going to be used interactively, you should be fine. It’s probably the best method to use in spite of a few downsides:

The object model is COM based which means you’ll have to do interop if you are developing a .NET application. Fortunately there are some nice improvements in .NET Framework which make coping with COM in C# much easier.
Your application will have to be compiled in 32-bit in order for it to work on 64-bit Windows. This usually isn’t a real limitation just don’t forget to switch the target platform from Any CPU to x86 if you’re doing development on a 32-bit OS to avoid the problem. (You’re going to notice it yourself when developing on a 64-bit OS.)
Don’t try unit testing the Excel automation code outside your IDE (e.g. on your continuous integration server). As described below this scenario is not supported.

As soon as you want your code to be run non-interactively, you’re out of luck with Excel automation. Since all Office applications assume to be running on the interactive desktop, there are several reasons why such usage is not supported and is even strongly discouraged by Microsoft. If you want to run your code on a web server, as a Windows service, a scheduled task or just automatically test it on your build server, you’ll have to find a different approach. And in this case there are no obvious choices.

Your best bet is to use Open XML file format (.xlsx, .xlsm) instead of the binary one (.xls) if this is an option for you. Since this is a well documented XML based format you can manipulate it directly without running Excel at all. You can even use Open XML SDK for Microsoft Office which includes strongly typed classes that simplify many common tasks.

This is not the case with the binary format. Microsoft doesn’t provide or support any SDK or API for manipulating the files directly. You are only provided with detailed documentation of the format. Based on it some third party solutions have been developed. Aspose and SoftArtisans have their own commercial offerings which I haven’t evaluated because as such they weren’t suitable for me.

On the other hand the only free library that I have found is MyXLS and this one leaves much to be desired. It also seems to be pretty much abandoned with the latest release almost a year ago and only a single commit to the repository in this year so far. That being said, it still might prove useful if you only want to create the files, mostly focusing on the appearance with only minimal requirements regarding formulas. According to the samples this seems to work fine. Reading existing files is another story. You are more or less on your own as soon as you need to read cells with formulas. This made the library useless for me therefore I used another approach based on the fact that the OLE DB Provider for Jet can be used to read and write data in Excel worksheets.

At first I haven’t even considered this possibility because I was convinced that this method can only be used on worksheets designed as database tables (having columns of data with or without header columns). This article proved me wrong and in spite of many issues in the demo project it showed off techniques which turned out really useful for me. The most important one was the possibility to address regions, not only complete worksheets – this can be used to retrieve and set individual cell values as well as for accessing database table-like blocks of data in a part of the worksheet. Let’s take a look at a sample:

SELECT F1 FROM [Sheet1$C2:C2];

This query retrieves a value of a single cell. The same syntax can be used to access any region: after the worksheet name with a trailing $ character the region is defined just like in Excel formulas. If the region doesn’t include column headers (specified by including HDR=No in the Extended Properties of the connection string) the columns are named F# where # is the sequential number of the column in the defined region. Similarly the following query can be used to set a value of an individual cell:

UPDATE [Sheet1$C2:C2] SET F1 = 'Value';

A few more things worth mentioning:

Be aware of the IMEX option in the connection string. It doesn’t seem to be documented very well, probably its best description is here.
Make sure you keep the connection to the file open if you plan to do more data access later. Opening the connection takes quite some time therefore the performance will be terrible if you keep closing and reopening it.
The technique doesn’t seem to work if the worksheet name begins with a space. At least I couldn’t make it work, no matter how I set the quotes.
I encountered a file which couldn’t be opened in this way but the problem was resolved after I opened the file in Excel and saved it again.

Setting Up SVN and CC.NET for .NET Development

Damir Arh — Sun, 30 May 2010 10:45:16 GMT

Subversion and CruiseControl.NET can be invaluable tools in your .NET development process. There are many resources available to help you get started which I’ll try to gather in this post along with some of my personal experiences.

Let me start with the list of recommended software:

VisualSVN Server is the ultimate Windows version of Subversion including a simple setup and powerful management tools. If you are planning to install a Subversion server on Windows it should be your first choice.
AnkhSVN is a Subversion Source Control Provider (SCC) for Visual Studio. As long as you’re not using Express editions of Visual Studio, this is the suggested way of working with SVN directly from Visual Studio IDE.
TortoiseSVN is a Windows shell extension for working with Subversion from within Windows Explorer. When you're not working with Visual Studio solutions this is the best choice for using SVN.
CruiseControl.NET is a continuous integration server including a web dashboard and CCTray - a system tray client application for monitoring and controlling builds.

If you’re not already familiar with the above mentioned products, you should consult their documentation or search for tutorials. I will rather focus on setting up your development and release process. If you haven’t done so already I suggest you first read the following articles by Ariejan de Vroom:

I mostly based my configuration on the ideas in these articles. I have projects configured in CC.NET to build all copies of the project: trunk (ProjectName-Trunk), all branches (ProjectName-REL-#.#) and all tags (ProjectName-v#.#.#). To identify individual builds I am using CC.NET’s Assembly Version Labeller together with AssemblyInfo MsBuild Community Task.

Assembly Version Labeller is really simple to configure. You only need to add a short snippet to each project:

<labeller type="assemblyVersionLabeller">
    <major>1</major>
    <minor>0</minor>
    <build>0</build>
</labeller>

I’m using the following versioning policy:

I start each project with version 1.0.0.
Once it’s ready for release I make a copy of the trunk in the branches directory, named REL-#.# containing the major and the minor version number. Immediately afterwards I bump the version of the trunk (only minor or major and minor, depending on the nature of the new features planned).
In the release branch I make the necessary changes before release (e.g. I change the AssemblyProduct name to distinguish between development and release quality builds) and make another copy in the tags directory, named v#.#.# containing the major, minor and build version numbers. Immediately afterwards I increase the build version number in the release branch.
I make no changes to the copies in the tags directory. All bug fixes go to the release branch. Once I’m ready for a new release I repeat the previous step.

Since I don’t specify the revision number directly, the SVN Revision number gets used automatically. This makes it possible to match each build to the revision of the code in SVN.

To put the generated assembly version in the build I am using the AssemblyInfo MsBuild task. There are two steps involved in doing this.

First you need to move the AssemblyProduct, AssemblyInfo and AssemblyFileVersion attributes from the auto generated AssemblyInfo.cs file into a new file. In my case the AssemblyVersion.cs has the following contents:

using System.Reflection;

[assembly: AssemblyProduct("ProjectName DEV")]
[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyFileVersion("1.0.0.0")]

Next you have to modify your project file (*.csproj) by importing the community tasks and adding a call to the AssemblyInfo MsBuild task:

<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets" />
<Target Name="BeforeBuild">
    <AssemblyInfo Condition="'$(CCNetLabel)' != ''"
        CodeLanguage="CS" 
        OutputFile="Properties\AssemblyVersion.cs" 
        AssemblyProduct="ProjectName TRUNK" 
        AssemblyVersion="$(CCNetLabel)" 
        AssemblyFileVersion="$(CCNetLabel)" />
</Target>

If you have never edited a project file before, you might want to read these first:

One more thing to note which might not be all that obvious. The Condition in the AssemblyInfo task is met only when building from CC.NET. For builds in Visual Studio the task doesn’t regenerate the AssemblyVersion.cs file therefore the revision number is always 0 and the AssemblyProduct has a DEV suffix as defined in the original file. Also I remove the TRUNK suffix from the AssemblyProduct attribute of the AssemblyInfo task when moving code from trunk to release branches to separate between the two.

Excel 2007 Query Parameters

Damir Arh — Wed, 28 Apr 2010 18:01:23 GMT

Query parameters are a very useful Excel feature allowing parameterization of database queries used to import data in Excel. They are really simple to use as well. On the Definition tab of the Connection Properties dialog there is a Parameters… button at the bottom. It gets enabled as soon as there is a parameter defined in the Command text – you define it by typing in a question mark (?) instead of a value in the WHERE clause of the query, as seen in the image below.

Unfortunately there is a limitation for using this functionality which turns out pretty unintuitive in Excel 2007. It is only supported for Microsoft Query based queries. For all other types of queries available in Excel you get the error “No value given for one or more parameters.” when you add a parameter to the Command text. Since the error doesn’t even hint at the real cause of the problem it took me some time before I figured it out. The thing to remember is: If you want to use query parameters, you must select Microsoft Query as the external data source when importing the data for the first time. This can’t be changed at a later time.

Send E-Mail As and On Behalf Of

Damir Arh — Sun, 09 Aug 2009 17:39:38 GMT

Microsoft Exchange supports Send As and Send On Behalf Of permissions to be granted to users for individual e-mail addresses. Sending e-mail from Outlook for these users is very simple – they just enter the desired address in the From field of a new message (toggled with the Show From command on the Options ribbon) and if they have the required permission it will be sent accordingly – either as if it was actually sent from that address or as sent by the user on behalf of the address in the From field.

If you want to achieve this from code there is a little more work involved. First of all the user must be authenticated on the server using one of the methods below:

SmtpClient smtp = new SmtpClient("smtp.domain.com");

// use user’s existing credentials
smtp.UseDefaultCredentials = true;

// pass username and password
smtp.Credentials = new NetworkCredentials("username", "password");

The next step is to set up the correct headers in the message otherwise the server will return error code 5.7.1 describing the permission the user does not have.

To send the e-mail as only the From property has to contain the desired address:

MailMessage mail = new MailMessage();
mail.From = new MailAddress("send.as@domain.com");

To send the e-mail on behalf of another user the Sender property must additionally contain the user’s e-mail address:

MailMessage mail = new MailMessage();
mail.From = new MailAddress("send.as@domain.com");
mail.Sender = new MailAddress("user.address@domain.com");

On a related note, the required permissions can be granted using PowerShell.

To grant the Send As permission:

Add-ADPermission –Identity "user1" –User "user2" –ExtendedRights Send-As

To grant the Send On Behalf Of permission:

Set-Mailbox "user1" -GrantSendOnBehalfTo "user2"

In both cases the user1 specifies the mailbox to grant the permission for and the user2 specifies the user to grant the permission to.

Identifying unrecognized devices in Device Manager

Damir Arh — Sat, 01 Aug 2009 17:09:49 GMT

Today I decided to get to the bottom of the missing driver issue on my computer running Windows 7 RC. It shows up as PCI Simple Communications controller and it really bugged me since I don't have a modem or a similar device on the motherboard.

It turned out that there is a away to identify such a device from the information available in Device Manager. The first step is to open the Properties window for this device and move to the Details tab. After selecting the Hardware Ids in the Property dropdown the device identifiers are displayed.

The important ones are the numbers written after the VEN and DEV keywords. The first one is the Vendor ID and the second one is the Device ID. So in my case the Vendor ID is 8086 (from VEN_8086) and the Device ID is 29A4 (from DEV_29A4).

All that's left to do know is to go to PCIDatabase.com and enter the ids into the corresponding search boxes. In my case it turned out that it was a device from Intel - Intel Management Engine Interface (HECI). Unfortunately it doesn't have a driver for Windows 7 yet and the Vista one doesn't install. But hey, at least I know which driver is missing.

Gama System eArchive accredited

Damir Arh — Sat, 05 Jul 2008 20:37:14 GMT

Gama System eArchive, one of the two products in our document product line, received accreditation from the Archives of the Republic of Slovenia last week. This acknowledgement by our national body means that any document stored in Gama System eArchive is automatically legally valid.

This is important for both our company and other companies looking for a long term electronic document storage solution. Our product is the first service oriented solution to receive the accreditation.

Congratulations to everyone involved in the product. Well done!

Microsoft Pro Photo Tools decimal separator problem

Damir Arh — Sun, 15 Jun 2008 20:03:01 GMT

Today I've taken my new handeld GPS device for a test run. It did its job pretty well but the real challenge started afterwards when I tried geotagging the photos I've taken. I decided to use Microsoft Pro Photo Tools which have just been released with geotagging as its main feature. Downloading the track data from the GPS device with Garmin MapSource software was quick and simple. But the problems started soon afterwards. MapSource can only export track data in its proprietary format GDB which can't be used in Microsoft Pro Photo Tools.

GPSBabel came to the rescue. This free tool can probably convert files between any two existing GPS formats, at least judging from its list of supported formats. I used it to convert my data to the GPX XML format only to find out that Microsoft Pro Photo Tools have problems with it. Converting to NMEA or KML instead didn't help either. Fortunately the latter returned a strange error (Degrees must be between 0 and 90, found degree 46298501) which put me on the right track. Of course there was no such value in the KML file so I correctly deduced that the decimal separator was to blame.

The value in the file was 46.298501 but the Slovenian regional settings have comma as the decimal separator therefore the value was misinterpreted. Temporarily changing the decimal separator to dot solved the problem - the track was successfully imported immediately afterwards. This issue won't keep me from using this otherwise very useful tool with a really nice feature set. It could even fix the mismatching time settings between my GPS unit and the camera with a single setting. I just hope they address this bug soon so that I won't have to change my regional settings every time I use the program.

The only thing I still have to figure out is why the geotags somehow lost resolution when I uploaded the photos from Picasa to Picasa Web Albums. I just fixed them manually and decided to address the issue next time. Any tips are welcome.