BizTalk Server 2004 : B2B, EAI and .NET

I'm now blogging at..

2005-09-16T16:08:00.000-04:00

I'm now blogging at Geeks-with-Blogs. C'ya there.

About Zombies and Convoys...

2005-05-27T18:51:00.000-04:00

This week was Load Testing week for our latest set of Interfaces. Incidentally, this was our first set of Interfaces which implemented Uniform Sequential Convoys in their Orchestrations. The implementation was more or less like my previous post. Meaning the convoys had a timeout mechanism, which lead the Orchestration to completion, if no new messages would arrive for a period of time.

So far - so good. But under a realistic amount of load, we observed that almost 1 out of every 5-6 messages would fail to go through to the destination. The HAT would show the following:

Completed with discarded messages

Of course the 1/5 is a very subjective number, depending upon the actual convoy implementation in question. After some snooping around (mostly at Lee Graber’s blog), I figured out that the source of the problem. But no amount of the aforementioned snooping yielded any easy and/or elegant solution to this little pickle.

What happens is that the control might pass to the delay branch of the Listen shape where we implement some logic to escape the loop. During this time interval, if a message arrives at the MessageBox, Biztalk sees our Orchestration as “Running” and since our Orchestration is a match for the Message’s subscription, Biztalk assigns this orchestration to take care of this message. Now our orchestration is blissfully unaware that it has a message or messages to take care of, and runs to completion. Hence, these message(s) are left in the message box with no one to process them or in other words in a Zombie state.

In my opinion, the first thing to do in this situation is (and I am not being facetious here) to verify that your correlation set is actually exact enough for your business scenario. For e.g. if we are transmitting patient medical records, we need to correlate only the same patients and not ALL patients. In other words don’t correlate JUST on MessageType if you can correlate on MessageType AND PatientID.

Secondly set your delay interval to an appropriate value. Setting it too low creates more such race timespans, per unit time. This will reduce your Zombie chances, but won’t give you a 100% success guarantee.

Thirdly have a proper plan to deal with these Zombies according to your business scenario. Retransmit? Discard? What?

Finally ponder about the “Receive Draining” pattern mentioned in Lee Graber’s blog, and if you have a nice elegant way of doing it, tell me about it FIRST!

'Non-Deterministic' Sequential Convoys

2005-04-04T12:36:00.000-04:00

A long, long break between posts. Actually was working on some custom adapters, which kinda left me with little to post. Anyways, today morning a colleague of mine was facing a problem with Sequential Convoys. He didn’t know how many messages he was going to receive in a particular convoy. Consequently, he didn’t know how many times to loop the Receives that follow the co-relation set (Hence, the "Non-Deterministic" in the title).

A simple solution exists to solve this problem. Use a Listen shape! A Listen shape with the Following Receive(s) in one branch and a Delay shape in the other, provides us with a graceful 'Timeout' kind of mechanism. Here’s a screen shot of how to plug in the Listen logic in your sequential convoys.

Most information in my previous post about Uniform Sequential Convoys holds true in this case too.

Post a comment if you need any clarifications.

The South-East Asia Earthquake and Tsunami

2004-12-29T13:41:00.000-05:00

Totally off topic but very important nevertheless. For those of you who wish to find the right places to contribute towards the Relief Operations for Tsunami victims in South-Asia, please visit the following link.

The South-East Asia Earthquake and Tsunami

Getting the BizTalk SQL Server DB Name

2004-11-18T09:15:00.000-05:00

A quick one. How to get the name of the SQL Server that BizTalk is running? A relatively easy way is to get it from the registry. Heres a code sippet that does just that.

string strBtsSvrSubKey =
"Software\\Microsoft\\BizTalk Server\\3.0\\Administration";
string strBtsSqlSvrKey = "MgmtDBServer";
Microsoft.Win32.RegistryKey key = Microsoft.Win32.Registry.LocalMachine.OpenSubKey(strBtsSvrSubKey);
string strBtsSqlSvrName = key.GetValue(strBtsSqlSvrKey).ToString();

Any queries, post a comment.

HIPAA Accelerator 3.0 Error deciphering

2004-11-16T16:38:00.000-05:00

The BizTalk Accelerator for HIPAA 3.0 throws up a mighty detailed error in the event log when it fails during Xml to EDI translation and/or validation. These errors tend to look a little scary (they scared me at the very least). An example would be :

Status report details: SRH DevBT03 200411122204111 DEFAULT 2 DEFAULT
SRM 00000000 00010168 101 0 837 004010HC09translate failed 0000010000010000000000010 10168
SRE 001030ProcessSegment 00053 000TS837Q1_2420A_NM103__RenderingProviderLastOrOrganizationName source format: [101 0 ,XML 1.0]\r\nsource document: [837 004010DEFAULT X X098A1,Health Care Claim: Professional]\r\nsource segment: [data#53,def#232,name=TS837Q1_2420A_NM1_RenderingProviderName]\r\nsource element: [def#4,elm#0,comp#0,name=TS837Q1_2420A_NM103__
RenderingProviderLastOrOrganizationName], (msgnr:1 segnr:53)(line:2 pos:11671 filepos:11953)

Look for the last numeric code following the SRE element. In this case its 001030. Ignore the 0010 part and you are left with 30. Open [document's home]\HIPAA_EDI\Subsystem\EIF\live.eif with notepad. Please be careful not to edit this file!! Preferrably create and open a copy. The first section of the file contains code to message mapping entries like :
ET1000001The path does not exist. Check the folder location.

Look for the entry matching our code i.e. 30. We have :
ET1000030The document does not contain a required element. Contact the sender.

So, the actual interpretation of the event log message is :
"The required element/attribute TS837Q1_2420A_NM103__RenderingProviderLastOrOrganizationName in the segment TS837Q1_2420A_NM1_RenderingProviderName is missing."

Now that's a LOT less frightening I think. :)

HIPAA Accelerator 3.0 Gotchas

2004-11-11T11:07:00.000-05:00

I stumbled across a couple of things w.r.t the BTS 2K4 and the HIPAA Accelerator 3.0, which I feel are worthy of a mention.

SQL Server 2000 client components are needed on the BizTalk boxes when SQL Server 2000 is installed on a separate box.
DTC (Distributed Transaction Coordinator) must be enabled for Network Access (via Windows Component Setup) on all computers participating in the configuration, administration and message processing.
The user under which the HIPAA EDI windows service is configured to run, must be a member of the BizTalk Application Users group.
All HIPAA 3.0 schemas should be validated using the XSD2EDI utility located at ......Program Files\Microsoft BizTalk Accelerator for HIPAA 3.0\HIPAA_EDI\Subsystem at least once (on all BizTalk boxes where HIPAA is installed) before the HIPAA Accelerator can start processing messages.
For multi-box BizTalk Installations you might want to consider making the HIPAA Documents Home directory a network share.
For ANSI 835p there exists a context property named CountOfTranslatedDocs, which can be used for correlating 835's from a single file into a sequential convoy.
Ensure that the login credentials for the HIPAA EDI Windows service and the BizTalk Host under which the HIPAA Adapter runs, match.

More additions later. Any queries, post a comment.

Non-Uniform Sequential Convoys in BTS 2004 using Correlation

2004-09-29T13:49:00.000-04:00

This is another primer on Sequential Convoys, but non-uniform ones this time. With a uniform Sequential Convoy, correlated messages have to be of the same type. But there might be situations where we would want to implement Sequential Convoys with different messages. The sample below illusatrates this concept by taking 2 messages of different types, and merging them to produce 1 output message. The input messages are correlated into a single orchestration instance by a promoted property called "Primary".

Non-Uniform Sequential Convoy

Most information in my previous post about Uniform Sequential Convoys holds true in this case too.

You can get the source files here

Post a comment if you need any clarifications.

Using XmlSerializer in the BizTalk Web Service Proxy.

2004-09-27T13:28:00.000-04:00

This is something that most people will be aware of but might be helpful for newbies. The BizTalk Web Services Publishing Wizard, creates new datatypes for input and output messages during creation of the Web Service that exposes Orch/Schemas. Adding a web reference to this web service will generate the same datatypes, whose definitions can be found in the Reference.cs file. If the external system already generates XML, creating the input message object can be a drag.
A quick way of creating the input message objects from XML and converting the output message objects to XML, is by using the XML Serializer class. Here's the sample code.

//creating a new RequestMsg
objectRequestMsg objRequestMsg = new RequestMsg();

//Create an XML Serializer for RequestMsg document
XmlSerializer serRequestMsg = new XmlSerializer(objRequestMsg.GetType(), "http://schemas.devdutt.com/XMLSchema/v1/InputMsg.xsd");

//Deserializing XML into RequestMsg class
//strRequestMsgXml is the XML String
objRequestMsg = (RequestMsg)(serRequestMsg.Deserialize(new XmlTextReader(new StringReader(strRequestMsgXml))));

//Call RequestMsg Web Service and send the Request
//orchInvoker is the Web Service Proxy Object
//Submit is the Web Method
object obj = (orchInvoker.Submit(objRequestMsg));

//Create an XML Serializer for OutputMsg document
XmlSerializer serResponseMsg = new XmlSerializer(obj.GetType(), "http://schemas.devdutt.com/XMLSchema/v1/OutputMsg.xsd");

//Create a temporary memory object for storing xml stream
MemoryStream ms = new MemoryStream();

//Serialize the output document
serResponseMsg.Serialize(ms, obj);

//Convert the memory stream into the string buffer
string strResponseMsgXml = System.Text.Encoding.ASCII.GetString(ms.GetBuffer());

Any questions, post a comment.

Using recursion in the In-line XSLT Template type Scripting Functoid

2004-09-18T21:25:00.000-04:00

People using In-Line XSLT Templates in their scripting functoids often run into the need for looping, WITHOUT using an <xsl:for-each>. An example in the BizTalk context would be, if you had a comma separated string in the source schema and wanted to tokenize it into multiple destination schema elements. The following XSLT template does that for you.

<xsl:template name="TokenizeCommaSepString">
<xsl:param name="stringToTokenize" />
<xsl:param name="destinationElementName" />

<xsl:if test="$stringToTokenize != ''">
<xsl:choose>
<xsl:when test="contains($stringToTokenize, ',')=0">
<xsl:element name="{$destinationElementName}">
<xsl:value-of select="$stringToTokenize" />
</xsl:element>
</xsl:when>
<xsl:otherwise>
<xsl:element name="{$destinationElementName}">
<xsl:value-of select="substring-before($stringToTokenize, ',')" />
</xsl:element>
</xsl:otherwise>
</xsl:choose>

<xsl:call-template name="TokenizeCommaSepString">
<xsl:with-param name="stringToTokenize" select="substring-after($stringToTokenize,',')" />
<xsl:with-param name="destinationElementName" select="$destinationElementName" />
</xsl:call-template>
</xsl:if>
</xsl:template>

Drag a Scripting functoid onto the BizTalk Mapper. Go to properties and choose Configure functoid script. Choose Inline XSLT Call Template as the Script Type. Paste the above code in the Script Buffer text area. This functoid will take the comma separated string in as the first parameter and the destination element name as the second parameter. It will split the string into parts and create a sequence of elements in the destination message.

Observe the use of recursion (marked in bold), to solve similar problems. Have any questions, post a comment.

Sending Messages as FORM Variables with the HTTP Adapter

2004-08-31T17:56:00.000-04:00

Here's a quickie. If you want to send a message, as a FORM Variable in a HTTP request, this is what you do:

1) Set the Content Type property of the HTTP send port, to application/x-www-form-urlencoded.

2) Create a custom pipeline and plug-in a Custom Pipeline Component, that pre-pends Variable-Name= in front of the message, and url encode it. e.g: ClaimPacket=

3) Use this pipeline with your HTTP send port, and you are done!

Post a comment for a sample.

Update : Screen shot of the Content Type property:

Event Log Message Interception with WMI

2004-08-26T14:26:00.000-04:00

Out of the box pipeline components have decent error logging, but often don't give us the flexibility to do our own thing. In addition to a suspended queue listener (see Martinjn's blog), an Event Log Watcher windows service might be a good failover mechanism. Below is some skeleton code for going about it.

using System;
using System.Collections;
using System.ComponentModel;
using System.Data;
using System.Diagnostics;
using System.ServiceProcess;
using System.Management;
using System.IO;

namespace MyEventLogWatcher
{
public class BizTalkEventLogWatcher : System.ServiceProcess.ServiceBase
{
private System.Management.ManagementEventWatcher eventLogWatcher;
///

/// Required designer variable.
///

private System.ComponentModel.Container components = null;

public BizTalkEventLogWatcher()
{
// This call is required by the Windows.Forms Component Designer.
InitializeComponent();
}

static void Main()
{
System.ServiceProcess.ServiceBase ServiceToRun;

ServiceToRun = new BizTalkEventLogWatcher();

System.ServiceProcess.ServiceBase.Run(ServiceToRun);
}
#region Component Designer generated code
///

/// Required method for Designer support - do not modify
/// the contents of this method with the code editor.
///

private void InitializeComponent()
{
this.eventLogWatcher = new System.Management.ManagementEventWatcher();
//
// eventLogWatcher
//

this.eventLogWatcher.Query = new System.Management.EventQuery("SELECT * FROM __InstanceCreationEvent WHERE TargetInstance ISA \"Win32_NTLogEvent\"" +
" AND TargetInstance.Logfile = \"Application\"");

//
// BizTalkEventLogWatcher
//
this.ServiceName = "BizTalkEventLogWatcher";

}
#endregion

///

/// Clean up any resources being used.
///

protected override void Dispose( bool disposing )
{
if( disposing )
{
if (components != null)
{
components.Dispose();
}

}
base.Dispose( disposing );
}

///

/// Set things in motion so your service can do its work.
///

protected override void OnStart(string[] args)
{
eventLogWatcher.Scope = new System.Management.ManagementScope("\\\\"+Environment.MachineName+"\\root\\CIMV2");
eventLogWatcher.EventArrived += new System.Management.EventArrivedEventHandler(eventLogWatcher_EventArrived);
eventLogWatcher.Start();
}

///

/// Stop this service.
///

protected override void OnStop()
{
eventLogWatcher.Stop();
}
protected void eventLogWatcher_EventArrived(object sender, EventArrivedEventArgs e)
{
string strEventSourcesToIntercept = "";
try
{
PropertyData propData = e.NewEvent.Properties["TargetInstance"];
StringBuilder strUserFriendlyMsg = new StringBuilder();
if(propData != null)
{
ManagementBaseObject mgmtObj = propData.Value as ManagementBaseObject;
if((mgmtObj.Properties["Message"].Value != null)&&
(mgmtObj.Properties["Source"].Value.ToString().Equals("strEventSourcesToIntercept")))
{

}
}
}
catch(Exception Ex)
{
//error handling
}
}

}
}

Uniform Sequential Convoys in BTS 2004 using Correlation

2004-08-25T11:59:00.000-04:00

This orchestration is a small primer if you want to implement a "Sequential Convoy " with BizTalk 2004. You would want to do this if:

a) You have to force messages coming in at a particular port to correlate together and get processed by the same orchestration instance.
(Correlation with multiple ports is a topic of a subsequent post)
b) You want your output messages to be delivered in the same order that the orchestration got them.
c) Some pipeline components you use, take one message but give out multiple messages, but you want to process these multiple messages together. For eg: the HIPAA and HL7 disassembler components.

...and probably some more scenarios that I can't think of right now.

The attached orchestration correlates messages w.r.t ReceivedFileName (think about the HIPAA 835 scenario mentioned above). You can use whatever correlation parameters you desire.

TIP: Whatever properties you choose to correlate the incoming messages with, make sure that these properties are promoted when the message(s) reach the MessageBox. Since the RecievedFileName property is not promoted by default, I have also attached a pipeline component which promotes it before it reached the MessageBox.

You can get the source files here .

Any questions, lemme know. Cheers!