NSilverBullet

100% CPU usage in BizTalk development environment

Joshua Anthony — Thu, 16 Apr 2009 07:53:37 GMT

These last few days I have been having enormous problems with my BizTalk development environment running in a windows Server 2003 Virtual PC environment. Whenever the BizTalkServerApplication host was started CPU usage would stay at 100% consistently. Stopping all my applications and even deleting them changed nothing. No running or suspended instances were visible. For some reason the sqlservr.exe process was consuming all available CPU resources. I found a couple of links that helped me solve the problem:

http://www.tech-archive.net/Archive/BizTalk/microsoft.public.biztalk.general/2006-11/msg00243.html
http://www.digitaldeposit.net/saravana/post/2008/06/04/Read-this-before-using-bts_CleanupMsgBox-stored-procedure.aspx
http://technet.microsoft.com/en-us/library/bb727781.aspx

Basically running the following SQL:

exec bts_CleanupMsgbox

After installing the script according to the instructions provided by Microsoft solved my issues! I suspect that this could have to do with a problem with zombie messages that I have had.

There is also a hotfix available in case you have the wrong version installed. http://support.microsoft.com/kb/924715

Reflections on three days at Tech Ed

Joshua Anthony — Wed, 12 Nov 2008 13:26:08 GMT

This is my third day at Tech Ed 2008 in Barcelona, the breadth of the technologies and languages that we as .Net developers now need to at least understand and preferably master to some degree is enormous. I believe that developers will need to specialize more explicitly on certain technology areas to remain productive in the future. It has been impossible to master all of Microsoft's products for many years now but up until a year or so ago it was at least conceivable for a developer to master all of the developer technologies that Microsoft had on offer. With the introduction of cloud based services and all the news in .Net, both libraries and new languages I cannot see how any one person will be able to master it all... I want to sum of some of my thoughts on the sessions that I have attended here at Tech Ed on Visual Studio 2010.

Fixing Continuous Integration

CI is a great principle and has moved into the mainstream along with many other agile principles. While Continuous Integration as such has never been broken there has always been a problem with developer discipline. If you check in something that is broken then it will be discovered quickly with CI but if you don't fix the broken code other developers will be less productive if they cannot get the latest code or have their own checkins fail in CI due to previously broken builds. With Visual Studio Team System 2010 Microsoft is introducing a new concept for CI, the gated check. It is not a big change to the CI flow but a very clever use of a shelving, a feature in TFS which I find generally is underutilized. Instead of doing a standard checking in and then running a build on that code, gated checkins shelve the developers changes and then performs a build on the shelveset. If this build is successful then the shelveset is promoted to a proper check in, if the build fails then the check in is not committed to the mainline. The visibility that CI offers is maintained but using gated checkins other developers will never get failing code from other developers which was discoverable! Obviously undiscovered errors can still sneak into the code due to bad or missing tests or incorrect configuration of the build environment but it I believe that this will lower or eliminate the fear of checking in that some developers have. It will also allow you to check in before you go home without the need to wait and see that the CI build completes successfully. If I check in my code 5 minutes before I go home I should ideally wait to see that the build is successful, if it does not then I need to fix that immediately since other developers will be getting that code tomorrow morning when they come in. Using a gated check in I can go home, let the CI build complete on my shelved changes and if the build is successful then everyone can get my changes in the morning, if it fails I can safely wait until the morning before fixing the code and no one else is impacted by my errors.

Reproduction of Errors and Architecture Compilation Failures

Visual Studio 2010 looks like it contains some pretty good testing tools like the ability to record a testers GUI while performing manual tests and also including debugging and tracing info from remote machines. Pretty powerful stuff! It seems Microsoft have their mind set on eliminating the problem of irreproducible errors.

Another interesting feature is the architecture layer diagram which allows you to specify how the different layers in your application may and may not call each other. This is useful for validating that your code follows the intended architecture at compile time instead of detecting it when problems arise. All of these features (and there are more) are great but unfortunately Microsoft still has silly distinctions between different editions of Visual Studio Team System, so many of the cool demo features will probably be marginalized in most organisations due to them not actually having the appropriate edition of Visual Studio.

BizTalk 2006 MQSC Adapter Encoding Solution

Joshua Anthony — Mon, 03 Nov 2008 15:45:34 GMT

Working with Websphere MQ and the MQSC adapter is quite new for me and I have been struggling with a couple of problems which have been very difficult to find solutions to. One of our main issues has been getting the correct character encoding of outgoing messages to MQ. BizTalk would output the message with the correct encoding (using a file send port to verify) and it looked like the messages on the WebSphere MQ queue were correctly encoded when we checked them using rhutilc. In our case we encoded our unicode messages as UTF-8 but when the Java system which is the destination read the messages any non-standard US characters came out as rubbish. They were correctly encoded as two bytes in the message but the destination system was reading them using codepage 850. Also all unicode messages that the Java system sent to us encoded with UTF-8 over MQ worked correctly in BizTalk.

What I have found out through much trial and error and the sparse clues I have been able to find on the Internet is that the WebSphere client that BizTalk uses under the covers will set the character encoding of any messages that you write to the systems default codepage, which for windows is 850 according to IBM. This setting does not take into account the actual encoding that BizTalk 2006 has done to the message. If you force your message to be encoded as 850 in BizTalk then it will most likely work as long as you don't use characters that are outside the codepage. There is a context property in MQSeries.dll called MQMD_CodedCharSetId which will allow you to set the codepage value (CCSID in the MQ world) for your outgoing message.

I have found three ways to remedy the encoding problem:

Set the context property in an orchestration to whatever value you require. The drawback with this is that if you need to change your encoding you have to recompile and redeploy everything. Also changing your context property does not actually change the encoding just how MQ should interpret the message.
Set an MQ environment variable named MQCCSID to the codepage that you send your messages as. This will override the system default value and force messages to be tagged with your required codepage. Obviously you can only have one system wide default so if you need to send messages out with different encodings this isn't going to help. Again the actual encoding of your messages is not going to be changed.
Create an encoding pipeline component that allows you to set the MQ codepage for an outgoing message. This is the option that we have settled on for now. The encoding of your message is still not actually changed, unless you implement it, but now you can have different encodings for different destinations and you can change the encoding value using MMC and binding files.

One important point to remember is that UTF-8 is codepage 65001 in Windows but 1208 in IBM, so wherever you set the codepage you need to have this in mind - as far as I know all other codepages have the same ids.

Using rfhutilc you can verify the codepage of a message in an MQ queue by browsing to the message and then checking what codepage has been set under the MQMD tab.

I have not verified this but the solutions that I have outlined above should work for EBCDIC and any other odd encodings that you may require as long as you can actually create a message with that encoding.

Add intellisense for any file extension in Visual Studio 2005 or 2008

Joshua Anthony — Thu, 30 Oct 2008 21:56:57 GMT

I have been messing about with some template experiments in dasBlog using Notepad++ and Visual Studio 2008. That is changing the site layout by editing homeTemplate.blogtemplate, dayTemplate.blogtemplate and itemTemplate.blogtemplate for a dasBlog Theme. These files are basically html files with dasBlog specific macros for inserting the blog functionality.

Notepad++

First off I love Notepad++ as a replacement for Notepad. I have had a hard time doing more complex things in Notepad++ due to my addiction to Microsoft's intellisense... Editing the .blogtemplate files is dead simple in Notepad++ since it is very easy to get syntax coloring - you simply choose a language (html) from the language menu and viola everything is color coded. The file that you are editing doesn't need to have the correct extension or even be written in the language you choose. Notepad++ does its best and highlights whatever it can recognize.

There is also an auto completion function that you access by pressing CTRL+space, but this function is really only a text completion function and does not take any notice of the context that you are typing in. It just pops up an alphabetical list of potential text candidates based on the letters that you have already typed. For simple programming tasks this is more than adequate and Notepad++ does have a significantly smaller footprint than Visual Studio, so you can install it on pretty much any computer or server without anyone complaining for quick debugging tasks.

What I really have been missing is proper html intellisense for the .blogtemplate files.

Visual Studio 2008

Obviously Visual Studio does not have support for the .blogtemplate extension. For awhile I hacked my way around it by renaming the files to have a html extension while editing, which sort of works. The main problem is that I keep forgetting to change the extension back before I start testing my changes - so this was causing unnecessary pain... I had a vague memory of configuring something like this before and found these instructions. For the sake of completeness I include them here also:

1. Go to the Tools->Options menu.
2. Pick Text Editor -> File Extension from the tree in the left part of the Options dialog.
3. Type your file extension, .blogtemplate in my case, in the Extension text box.
4. Select the appropriate editor from the Editor dropdown.
5. Click Add and then Ok to close the dialog and re-open your files.

If you cannot see the Text Editor node in the tree view of the Options dialog then you probably need to click on Show all settings in the bottom left hand corner of the dialog...

Using this feature you can set up pretty much any color coding and intellisense support that you like in Visual Studio for file types that Visual Studio lacks support for. All I am missing now is auto completion for dasBlog macros in .blogtemplate files, this seems like a much tougher problem.

Remove Byte Order Mark from a BizTalk 2006 outgoing messages

Joshua Anthony — Fri, 10 Oct 2008 08:53:02 GMT

The BOM is a two to four byte sequence at the start of an encoded file which tells the recipient if the characters are encoded using big-endian or little-endian byte order. In BizTalk generally the byte order mark is added when a file is UTF-8 or UTF-16 encoded using an XML assembler in a send pipeline. Sometimes applications cannot handle the initial bytes and you don't want them to be added to your message. I recently read two articles about removing the Byte Order Mark (BOM) from outgoing messages in BizTalk 2006 one by Saravana Kumar and the other an official Microsoft KB article. Both of them recommend creating a custom pipeline with an XML assemble stage where the PreserveBom property is set to false. I find it a little strange that they don't point out that you can accomplish this using the default XMLTransmit pipeline also. Just open the pipeline configuration in the Send Port properties and set PreserveBom to false... No custom Pipeline, no coding required.

URL Rewrite Does Not Work on Vista

Joshua Anthony — Wed, 08 Oct 2008 05:04:56 GMT

While trying out a complex URL rewriting scenario in ASP.Net I ran into a pretty strange problem that resulted in a 404 error:

Description: HTTP 404. The resource you are looking for (or one of its dependencies) could have been removed, had its name changed, or is temporarily unavailable.  Please review the following URL and make sure that it is spelled correctly. 

Requested Url: /6.aspx/t.aspx

My code is very similar to a solution that I have created previously in ASP.Net 2.0 on Windows XP so I was a little surprised that I was getting an error. The strange thing is that /6.aspx is the original URL and /default.aspx is the rewritten url, in the error message it looks like the original URL has partially overwritten the new URL!

The exception that is caught has a stack trace that ends with: System.Web.UI.Util.CheckVirtualFileExists(VirtualPath virtualPath)

Looking around the Internet for people who have already encountered this problem (as I always do) didn't lead to any immediate solutions. On the IIS forum there is one discussion on Module doesn't work on IIS7 under vista business which lead me to a solution. It seems to me that URL rewriting is broken on Vista prior to SP1. Part of the SP1 upgrade incorporates changes to IIS 7 that were made in Windows Server 2008 as detailed on Microsoft TechNet. So after installing SP1 my solution works as expected.

SFTP Adapter authentication for BizTalk

Joshua Anthony — Fri, 04 Jul 2008 10:38:14 GMT

There is a nice little SFTP adapter for BizTalk on CodePlex http://www.codeplex.com/SftpAdapter/. I have been trying this out for a couple of days and it works quite nicely. I have been using freeSSHd as SFTP server and WinScp as a client to see that the SFTP functionality works. WinScp comes with PuttyGen for generating key pairs for authentication and there are a couple of guides on the web which offer clues as to how this should all be configured but I still had to resort to trial and error to get the SFTP authentication working. So this is my super concise guide to getting it up and running for local testing (presuming you have installed all the software first).

Generate a keypair with PuTTYGen

Under WinScp in the start menu you have "Key Tools/PuTTYGen"

Click "Generate" and move the mouse about. When PuTTYGen has created your keypair enter a comment so that you key is easier to identify, you can use a passphrase for your master key if you like. Save your private key as a .ppk file somewhere safe this is your master file and you can open it in PuTTYGen later if you need to export it again. Export the key (under the Conversions menu) as an OpenSSH key, but remove the passphrase first since the SFTP adapter doesn't currently support passphrases for private keys, save this key somewhere where the SFTP adapter can access it. This key should also be kept safe since it is the equivalent of a password. Select the entire textbox with the public key and copy to the clipboard.

Configure freeSSHd for public key authentication

Open up the freeSSHd settings dialogue and go to Authentication. Enter a directory where the public keys for your users will be stored. Make sure that "Public Key Authentication" is not set to disabled and that "Password Authentication" isn't required.

You also need to set a directory under the SFTP tab for storing the files. Add users in the Users tab selecting "Public key (SSH only)" as the authorization method. Also make sure that the SFTP checkbox is selected.

Go to the directory that has been configured for freeSSHd keys. Create a text file named "username" ( the users login name and no extension) and paste the public key string from PuTTYGen into it making sure it is all on one line.

Test it with WinScp before trying it from BizTalk. Now you should be up and running with your SFTP adapter and an SFTP server.

Is EDA the way to go?

Joshua Anthony — Tue, 01 Jul 2008 10:37:26 GMT

I have been reading a lot about Event Driven Architecture (EDA) in relation to SOA recently and while the ideas are very interesting and the benefits are alluring it adds an additional complexity and uncertainty to the overall behaviour. Two good articles on EDA are http://martinfowler.com/eaaDev/EventCollaboration.html and http://eaipatterns.com/docs/EDA.pdf

From my view the main problem is that when implementing a business process that spans several services (for example entity services) we do not generally want the entity services to react to events and then store or retrieve data based on the events because this means that the services need to have some kind of knowledge of the expected behaviour relating to the process - generally seen as a bad thing in SOA, as the northbound - southbound dependencies get reversed. We also want to have some stability from a callers perspective that the overall flow will be moved forward, traditionally done by issuing a command. If events get turned into commands then I believe we will have problems coordinating the different actors (presuming that the command event was only meant to be acted on once). Jack Hof has an interesting take on the relationship between SOA and EDA especially his extended pdf version has some interesting ideas about how to realise SOA and EDA together which contains a potential solution to the problem, basically by just combining EDA and SOA and knowing when to implement which principles.

A related problem is what should our messages contain - are command messages even required? Two slightly different takes on the problem are presented here and here. Thinking about events and messages has lead me to attempt to create (a very crude) classification system for events that are related to a solution that I am developing for a client.

State-change events: no messages are published as long as the underlying values are not updated, the message is related to a specific instance of an entity or service. There is no expected behaviour for the recipient.
Scheduled events: messages are published at a certain frequency whether the values are updated or not, all available data should be published simultaneously within a single message or as multiple messages. There is no expected behaviour for the recipient.
Request or reservation events: these messages are requests for behaviour. A receiver may safely ignore the message, send a negative response if it cannot fulfil the request or a positive response if it can fulfil the request. The event needs to have a predefined response message which the event source can understand and the response needs to be able to be routed back to the source without the listener needing to have prior knowledge of all sources. This is basically an event which initiates a dialogue between two parties based on the reservation pattern. It is the requestors responsibility to coordinate the possible responses and outcomes. There is an expected behaviour for recipients who return a positive response but no prescribed behaviour for those who ignore the event or actively return a negative response.
Command events: This must be an anti-pattern since a command denotes that the sender expects the recipient to perform some specific task and thus exhibit an expected behaviour, which is impossible in a loosely coupled, event-driven, service oriented world since the sender does not know who is listening.

The first two event types are just statements, the third (reservation) is really just a variation of a state-change event with additional semantics attached to it. Using EDA the expected behaviour that is needed for a specific business process must be realised with the help of some kind of process controller (notification server in Jack Hof's world) which follows the traditional Command and Control (call stack) approach for coordination but uses events to track what stage the overall process or task is at.

Remember to always have a static customErrors page

Joshua Anthony — Thu, 26 Jun 2008 11:10:37 GMT

I was just surfing around on http://forums.microsoft.com and got a yellow screen of death!

I must say I was slightly surprised. In web projects that I have been involved in we have always added two customErrors pages one to handle generic errors which redirects to an errorpage.aspx page and then a static html page errorpage.html which is only used when an error occurs on the dynamic error page... A sample configuration using location follows:

<configuration>

<system.web>
<customErrors mode="on" defaultRedirect="errorpage.aspx"/>
</system.web>

<location path="errorpage.aspx">
<system.web>
<customErrors mode="on" defaultRedirect="errorpage.html"/>
</system.web>
</location>

</configuration>

That way if we have a problem with our application that affects the entire site, for instance menu and navigational rendering, we can still show a nice error page with a message like "The site is temporarily unavailable, please try again in a few minutes."

Duplicate User Accounts in Active Directory

Joshua Anthony — Thu, 26 Jun 2008 08:55:10 GMT

We have just had a major Active Directory issue that has affected the Team Foundation Server instance that all the developers at the client that I am working at use. The problem occured when some users where erased in the Active Directory and were subsequently recreated with the same user names. These new user accounts cannot access the Team Foundation Server.

A large part of the problem was solved by just removing users from TFS and then putting them back in again. This recreates all the wiring under the surface in TFS so that the users can access the server but all their existing workspace info, pending checkins and so on are lost locally yet exist in Team Foundation Server with a different Owner under the same user name... The only way to tell them apart is by their SID and that the original account may get suffixed with a number in workspace lists: AD\username:61.

My first thought was to remove all the workspaces and pending checkins and let people manually resynch their projects. When I clicked undo pending change for a bunch of changes in Attrice Sidekicks an unhandled exception popped up: "Object reference not set to an instance of an object". I just can't undo these changes, there is a clue to the problem when I try to unlock a changed file, I get an Unlock error:

Failed to unlock $/Project/Path/file (TF14061: The workspace COMPUTERNAME; AD\username does not exist.)

I can't search for workspaces by username either, even doing a simple TFSSecurtiy /i "ad\username" /server:TFSName for one of the affected accounts returns "Error: The identity cannot be resolved." Since the original accounts have been removed from Active Directory I cannot easily get hold of their SIDs for querying with TFSSecurity. Attempting to delete the workspaces from the command line return the following error:

TF14061: The workspace WORKSPACENAME;AD\username does not exist.

Things seemed pretty messed up... I found some clues as to what was required to get things working again:
http://support.microsoft.com/kb/948679
http://support.microsoft.com/kb/823278
http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=454060&SiteID=1

According to the KnowledgeBase articles trying to create a new project while TFS is in this state may result in the following error:

TF30170: the plugin Microsoft.ProjectCreationWizard.WorkItemTracking Failed during task WITS from group WorkItemTracking

Basically Team Foundation Server has a bug that causes deleted accounts to not be resynched correctly and SharePoint doesn't handle recreated user accounts at all - you have to resynch them manually. Unfortunately in addition to this there is a bug in TFS SP1 which prevents TfsAdminUtilI from correcting the SIDS! You get the following error:

ERROR: Could not access database.

In the end I managed to correct all the problems, here are the steps required:

Contact Microsoft Support and request the hotfix for http://support.microsoft.com/kb/934216
Install the hotfix...
Follow the steps in http://support.microsoft.com/kb/948679 to resynch the users SIDs in TFS
Follow the steps in http://support.microsoft.com/kb/823278 to resynch the users in SharePoint
Recycle the TFS App Pool to force an update of all users and groups in TFS.

Look out with users who have their workspaces mapped to "c:\documents and settings\" the new user accounts cannot necessarily access their old files there since they now log on with a new account (which just happens to have the same name).

Jim Webber: "Business people are spaghettiheads!"

Herbj�rn Wilhelmsen — Thu, 29 May 2008 06:29:20 GMT

This is a guest post by Herbjörn Wilhelmsen @ Objectware

After working for years with business people and SOA Jim Webber has come around to thinking that fighting against real world complexity is neither successful nor valuable.

Dr Jim "World Wide" Webber - an author as well as global architecture lead for Thoughtworks - gave an interview last month after finishing his "Guerilla SOA" presentation at the Developer Summit 2008 conference in Stockholm.

Watch the interview to see Jim Webber

Say that business people are spaghettiheads, tell us why business people scare him and that business people should be the architects of our projects!
Come down hard on message oriented middleware (ESB’s) and WSDL
Give his definition of a what a service should be
Talk about why M E S T and SSDL makes for better SOA
Describe how MEST and SSDL helps you handle versioning problems in an SOA
Talk about his next big thing: Middleware! (despite his grudge for ESB’s)

Jim does a great job - he explains things that seem contradictory in a very smooth and straightforward way and in the end you just can't help yourself: You have to like him!

Thanks for the interview Jim!

I also want to thank my colleague Joshua Anthony (the owner of this blog) for standing behind the camera and doing a lot of post production work!

video://jim%20webber%201.flv,jimwebber_part1.jpg

video://jim%20webber%202.flv,jimwebber_part2.jpg

Update:
During the interview Jim talked about the SOYA project and a case study. Here are some links:

SOYA @ sourceforge
A description of the LIXI case study can be found inside Patric Fornasier’s master thesis