Real BizTalk

WWF and Cutting down BTS 2004 shapes

2005-10-19T00:32:00.000+01:00

Have you ever wondered if there is a way to cut down the number of BizTalk shapes you need? Well there is, however this technique should only be used if you understand exactly what the effect is... Many BizTalk developers are aware of the ability to review the intermediate code that BizTalk generates in its 'code behind'. This code is used to compile your orchestration into CLR. To simply review this code all that is needed is to right click on the orchestration and open it in notepad, at the end of the orchestration XML designer code you will find the XLANG/S code. This language is largely undocumented and really should be C# but isn’t for some unknown reason (Answers on a postcard). After reviewing the code and the documentation its possible to construct your own shapes. For example to avoid using a constructing message you would write in an expression shape:

construct msgTest { msgTest = orchHelper.CreateMessage(); }

You can combine the above with more than one shape in an expression shape: i.e:

scope
{
message SomeSchema.NACK msgNACK;

body
{

construct msgNACK { msgNACK = orchHelper.CreateTestMessage();
}

exec SomeOrch.HandleNACK(msgNACK, MQOutLoc);

}
exceptions
{
catch (System.Exception ex1)
{
Debugger.WriteLine(ex1); }
}
}

The above expression contains a scope, construct and a start shape. I have used this method in some BTS projects to prevent rewriting NACK code every time. Unfortunately, in BTS 2004 it is not possible to have reusable shapes so the above code has to be a cut and paste job every time. This will be the same in BTS2006 but in windows workflow foundation you can create your own shapes and even use .NET in code behind, just like ASP.NET and Winforms read here for a primer ... http://msdn.microsoft.com/windowsvista/building/workflow/default.aspx?pull=/library/en-us/dnlong/html/WWFIntro.asp John

Assembly probing when adding pipeline components to the Toolbox

2005-08-07T17:49:00.000+01:00

We had an issue at a customer about a month ago when a Pipeline Component was refusing to be added to the Toolbox and the error said something like 'You have selected an invalid Pipeline Component'.

The error message doesn't really give you much to go on, but one of our consultants eventually managed to work it out. It turns out that the pipeline component referenced another assembly that wasn't in the GAC and the design environment must navigate the relationships when the component is added to the Toolbox for some reason.

The approach uses the standard .NET assembly probing rules, so it will look for the assembly in the same directory as the pipeline component (e.g. C:\Program Files\Microsoft BizTalk Server 2004\Pipeline Components\) and then will look in the GAC if it doesn't find it there.

So if you get this spurious message when adding the pipeline component to the Toolbox, check that all other assemblies are where they should be...

Messages are immutable.... or are they??

2005-08-07T08:43:00.000+01:00

David & I wrote a BizTalk exam plus InfoPath exam simulator for TechEd 2005 in Orlando and travelled over in June to have some fun and get some sun . The exam consisted of 25 or so pretty hard questions on real world BizTalk stuff (150 took the exam and I think approx. 10 passed!). One of these questions was based on whether or not messages could be changed. ie: whether they are immutable.

As far as orchestrations are concerned, we know that if you want to change a message, you have to clone it first and then make your ammendment. But how does it work with the rules engine? If you pass in a message to the rules engine and use some Set operations to update the message, when the message is returned to the orcehstration, the message has miracoulously been updated. Therefore, messages are immutable except for when you use the Rules Engine right?

Lee Graber sat the exam (and passed by the way, although he didn't come first!). After the exam we had a chat about the message behaviour with the rules engine and he was adamant that messages are always immutable, so we had a play around and he showed us that he was in fact right, the Rules engine automatically performs the same operation by cloning the message and passing back a copy of the message with the values updated. This was easy to check by looking at the MessageId of the before and after message to confirm that it had in fact changed.

This raises some interesting questions:

What happens with .NET short-term facts? Are these cloned or just referenced as you would expect?
Does the cloning only occur when you use the 'set' operations or are messages always cloned when you use the call rules shape?
What happens if you use .NET messages rather than XML messages?
What is the performance penalty associated with a set operation on an XML message?
Is it more performant to pass in a .NET fact to catch the set operations
Is the same behaviour experienced with calling the Rules Engine from the API?

If I ever get some time off from my day job, I'll have a look into these and post part 2...

Common problem when laptop not plugged in to network

2005-03-20T10:32:00.000+00:00

There's a common problem with BizTalk that's been around since beta days related to IP addresses. If you attempt to install BizTalk when not connected to a network, you will receive SSO errors during the configuration step. The same problem occurs if you install whilst connected and then go away to work on your laptop in a disconnected mode. BizTalk will stop working.

The solution to this is simple. Put an entry in the hosts file that refers to your machine:

127.0.0.1 MACHINENAME

I'm sure you all know this, but just in case. I've seen it bite someone badly during an important customer demo recently ;-)

Performance tip: Parallel vs. Atomic sends

2005-02-28T12:54:00.000+00:00

So, you want to send two messages at the same time and want to do it the most efficient way possible. You use a parallel shape and have two threads kicking off mutiple sends? You could do, but is there a better way?...

See the diagram below with one approach on the left and one on the right...

Parallel vs Atomic multiple sends

There is a cost with spinning up the separate threads, plus an additional persistence point after each send, which means that using an atomic scope is more efficient. This is because the atomic scope batches up the sends until the end of the scope is reached. The messages are sent within the context of the atomic scope's persistence point so there's less database round trips...

Try it and see....

MQ Series - no need to cluster MQSAgent

2005-02-23T12:12:00.000+00:00

The MQ Series Adapter documentation suggests that the MQSAgent should be clustered when using the MQ Series Adapter on an MQ Series cluster built on top of MSCS.

This is actually not required and will not work if it's attempted. There is no need to cluster the MQSAgent component. You just need to install the MQ Series Adapter on both nodes of the cluster (as if they weren't clustered) and at runtime, it will use the local component on the active node.

Hopefully they'll get around to updating the docs soon!

Developing Rules

2005-02-15T12:19:00.000+00:00

BizTalk is integrated with the Business Rules Engine (BRE). The BRE allows you to separate business policy from the flow of your orchestrations. For example, you might have a flow that routes orders for more than a certain monetary value to an expediated process. The value that this decision is made on should not be hardcoded into the orchestration. Instead the orchestration should invoke a business policy which will make the decision.

The way you develop policies is as follows:-

Develop a vocabulary of facts. Facts expose data to rules and implement actions that rules may take. Facts can be bound to .NET properties or methods, XML or databases.
Develop rules that reference your vocabularies. A policy consists of a set of rules and each rule is a set of predicates (i.e. logical tests) and the set of contigent actions.

All good so far...except the way versions are managed makes it impractical to develop rules at all!

The problem is this. Before a rule can reference a fact in a vocabulary the vocabulary must be published. Once a vocabulary is published it cannot be changed and it is not possible to unpublish it!

Consequently, if you use the business rules composer as intended whenever you want to change a fact or add a new fact to your vocabulary you have to publish a new version. Moreover, your rules are bound to particular versions of your vocabularies. So, once you've created and published your new vocabulary version you need to go through each rule and update its fact references to the new version if you intend to delete the old versions.

Like all software development it usually requires many versions of a fact vocabulary before you get it right. It would not be an overstatement to say you easily end up with 20-30 versions of your vocabulary before you've got your rules working as desired.

So, is there an alternative. Yes - but is ain't pretty: the answer is to unpublish the vocabulary by bypassing the rule composer. But the only way to do this is to directly update the rules database.

The process is:

Publish your vocabulary
Test your rules that refer to the vocabulary
Open the re_vocabulary table in the BizTalkRuleEngineDb and change the nStatus field from 1 to 0 (1 = published, 0 = not published). You can identify your vocabulary by its name held in the strName field.
Reload the vocabulary into the rules composer and add/modify your facts.
Save the vocabulary and then set the nStatus field back to 1 - don't re-publish the vocabulary from the rules composer else you will get a primary key violation.
Reload the policies/vocabularies once more in the rules composer and retest your policy.

You can also do the same trick with the policy. Although you don't need to publish the rules to test them using the test facilities of the rules composer, you do if you intend to test them from your orchestration. Clearly, you can find bugs in this process just as much as during your unit tests. Rather than have to create a new version of the policies just change the nStatus field in the re_ruleset table to temporarily unpublish the policy so that you can edit it.

One note of caution, the rules bizarrely cache the fact definitions inside the rule definition. So changing a vocabulary fact won't effect the rule that references it unless you re-reference the vocabulary item from the rule. So, although this process is fairly painless for adding new items to a vocabulary you have to be more careful with updates to facts.

Clearly, having to jump through hoops like this is regrettable and it can only be hoped that Microsoft do something about this in the next release of BizTalk.

Business Policy & Subscriptions

2005-02-15T10:44:00.000+00:00

Here's a handy pattern which we developed to cope with a difficult requirement.

I was working on a project to implement a message broker for SWIFT messages.

Initially the concept was quite simple: all of the 40+ applications in a bank would route their SWIFT messages through the message broker which would decided if money laundering compliance checks needed to be carried out.

Just before development started a new requirement was introduced - some of the messages would require special processing. The driver for this was that the bank was centralising one of its back office functions and wanted certain sets of messages transformed or processed so that they could be integrated with the new functionality.

At first the new requirement didn't seem to complex but after some analysis we realised that not only did different subsets of message types required special processing but that different instances (i.e. messages for certain accounts or destinations) also required special processing.

Now, SWIFT has over 350+ different sorts of messages so it wouldn't be practical to manage 350+ orchestrations instead the design for the original requirement was to have a common orchestration that processes untyped XML messages. The idea was that business rules would be used to extract the key data required for the routing and compliance process from the XML blob.

With the introduction of the new requirement we suddenly had a more complex situation. Sometimes we would want an orchestration specific to the messages type (so that transformations and distinguished properties could be used) and sometimes we'd want common processing for whole sets of messages.

How could we make this all work with the BizTalk subscription model?

Before I outline our solution, a quick recap on how subscription in BizTalk works.

When you add an activating receive shape to an orchestration and then bind and deploy your solution you are adding a new subscription for the orchestration.

Normally, your subscription consists of:

The message type (specified using the xsd namespace for your message with a # followed by the top-level element name - e.g. http://mysolution#MyElement)
The receive port identifier
Any filter predicates, i.e. tests of message context properties

When a message leaves the adapter framework after processing by the receive pipeline it is dumped into the message box. The BizTalk engine then looks through the subscription table and checks the promoted properties against the subscriptions. Key properties are, of course, the message type and the receive port ID.

Now imagine we had an architecture whereby 300+ of our SWIFT messages were to be processed by a common type agnostic orchestration whereas a small subset of messages were to be processes by type specific orchestrations. When one of the messages in the type specific subset is received it would match the subscriptions of the common orchestration and the type specific orchestration and suddenly the bank has transferred £2,000,000 when it should have been £1,000,000!

Moreover, in some circumstances we would want a common type agnostic orchestration to process whole sets of messages. For example, we might have the requirement that all payment and cash messages go through special processing.

Somehow, we need a way to control the subscription mechanism in a fine grained way.

The answer we came up with was to create a custom pipeline component for managing the subscription. The pipeline component's job was to invoke a subscription policy using the Business Rules Engine. The subscription policy was used to decide on an appropriate business process for the message. This decision could take into account the message type, any data in the message (e.g. priority, etc.) and could use database lookups (e.g. lookup certain accounts or destinations that required centralised processing).

Once the subscription policy had reached its decision it returned the name of the appropriate business process to the pipeline component. The pipeline component then simply promoted this value as a subcription property in the message context.

In this way orchestrations would not only subscribe to message type (if appropriate) and port then would also subscribe to the business process.

The common type-agnostic orchestration that handles the bulk of the messages would subscribe to the "Generic" business process whereas other orchestrations might subscribe to "CentralisedPaymentsAndCash" or somesuch. In this way different orchestrations could subscribe to the same message type (or no message type) from the same port but with different business processes.

This is a very generic pattern that we're sure is going to crop up time and again. Hope you find it useful!

Little known MQSeries feature

2005-01-25T12:06:00.000+00:00

One of the less publicised features of the MQ Series adapter is that you can make it rollback the transaction and disable the receive location if an exception occurs in the pipeline.

The reason for this is that you may not want to suspend the inbound message if processing occurs in the pipeline that is important for the handling of that message. For example, if you execute business rules in the pipleine to determine a subscription policy (see Dr. Regan's recent post), or because you're using the A4Swift adapter (lots of rules), you won't want the message suspended if the Rules Engine causes an exception due to an infrastructure problem (e.g. could not connect to DB / Rules Engine Update Service not running). You want to be alerted to the problem so it can be fixed, and then re-enable the receive location so that processing continues as if nothing ever happened.

To enable this feature, set the Ordering property of the Receive Location adapter configuration to "No Order With Stop".

If the condition occurs the MQ Series Adapter will raise an appropriate event to the event log that can be picked up by MOM to alert an operator.

It works very well and is crucial for processing financial messages that must succeed...

Useful retry pattern

2004-12-15T20:46:00.000+00:00

Reading a ‘Suspended (Not Resumable)’ status for a critical $20M financial transaction is not really the best thing you can wake up to. I’ve woken up to a few in my time, and trust me: this one is particularly unpleasant…

There are a number of reasons it can happen, but where orchestrations are concerned, it generally means that an exception has occurred that hasn’t been gracefully handled.

Although in theory, this can happen anywhere in your orchestration, there are key points that it will happen more than most, and if you protect yourself in these scenarios, you’ll be able to sleep a little easier. Common scenarios that could generate an error are:

Calling into to the Rules Engine to execute a Policy – if the Rule Engine Update Service is not running (why does this never start on my laptop??), or the rule engine runs code that has an exception
Sending a message to a destination that may sometimes be unavailable (that’s pretty much all destinations!)
Executing some .NET code that could go wrong for an infrastructure reason (e.g. database server is down)

All these scenarios are related to infrastructure problems. They’re problems that if you fix the infrastructure and resume the orchestration, then everything should continue as if nothing ever happened.

In these situations, a good way of dealing with it is to execute some retry logic by gracefully putting the orchestration into a Suspended mode and allowing an administrator to resume the orchestration once the infrastructure problem has been resolved.

It’s pretty simple to achieve via nested scopes. Wrap the potentially risky operation in a set of shapes that together form a reusable retry pattern:

Essentially, the pattern requires that the risky operation is encapsulated within a loop shape, which in turn has a non-transactional scope with an exception handler that traps the possible error conditions. If an exception occurs, a String variable and Boolean variable is set that is then used outside the non-transactional scope to determine whether to gracefully Suspend, and to write out a meaningful reason for the suspension. The administrator will get an alert that warns that an instance has been suspended. He can then resolve the problem (e.g. start the Rules Engine Update Service) and then use HAT to resume any orchestration instances. Simple…

Once I’ve worked out how to upload pictures of orchestrations, I’ll post a picture which will make it easier to understand!

One extra issue is that if you’re using this pattern when sending messages and the message fails to be transported, the message to be sent will also go into a Suspended (Resumable) state. If the administrator resumes both the Orchestration service instance and the Send service instance, you will send the same message twice. To work around this problem, you can use delivery notifications and a Nack Handler to automatically clean-up the send service instances, but that’s a subject for another day…

Failover issues with MQ clusters

2004-12-03T12:37:00.000+00:00

One issue we've had with MQ Series in a clustered environment is that if you initiate a failover to the passive node, and then failback, MQ Series will moan about Semaphore locking issues. This is a known issue with MQ that IBM is working on with Microsoft.

The solution (work around) is to always reboot the original node before failing back the resources. This seems to cure the issue!

BAM for operational reports

2004-11-16T21:19:00.000+00:00

When you start developing with BizTalk you envitably start with orchestrations, pipelines, schemas etc. and ignore some of the other cool features. One such feature is Business Activity Monitoring (BAM).

Ian & I were working on a SWIFT messaging system for an investment bank and were designing the reporting system for tracking the flow of messsages. We considered the built in message body tracking facilities and considered a custom solution based on a custom application database - and then we considered BAM.

We'd both thought of BAM as being a management information tool, i.e. a tool for gathering aggregate data for building OLAP cubes. We hadn't previously thought of it as a tool for operational reports but it turns out that it is a scalable and flexible mechanism for capturing data.

Before I discuss our approach, a quick recap of BAM. BAM is an infrastructure for capturing data into a database. It has been designed (using active and partitioned completed tables) to efficiently capture data and to allow fast queries over the data. BAM has tools for designing the data you want to capture - you define busines activities in terms of milestones and data items - and will create the requisite data tables, views, stored procedures and DTS jobs for partioning and archiving data. Moreover, the event bus service (aka the tracking host) provides a robust, asychronous system for data capture.

In our SWIFT messaging system each message received resulted in 5-6 message interactions with other systems. We wanted a way to capture the flow of the messages through our system and a way of capturing the message bodies (in their native format). We decided that BAM provided the ideal infrastructure.

I created two activities:

A flow activity which defined the milestones for the message flow (e.g. received message, sent message to compliance engine, received first response, etc.) and the data items associated with the milestones.
a message body activity which defined the single milestone of a message body processed along with the message body as the associated data item.

I then created two .NET classes (FlowTracker and BodyTracker) for invoking the BAM API to create, update and complete the activities. Each class had a set of methods - one for each milestone - with parameters for the data items appropriate for the milestone.

The next job was to create a simple pipeline component for tracking the message bodies. This component was built as a decoder and encoder that could be used as either the first component in a receive pipeline or the last component in a send pipeline. The component simply used the BodyTracker to create and complete a message body activity.

The orchestrations that process our messages use the FlowTracker to create a flow activity on receipt of the activating message and use the milestone methods to update the activity at various key points in the orchestration.

One of the key problems to solve was how to related the flow activity to the multiple message body activities created in the pipeline. This was done by having the tracking pipeline component work in two modes.

In a receive pipeline, the tracking component simply created the body activity using the message interchange ID. Recall, that the interchange ID is constant for all of the messages in the pipeline and is a property in the message context. When the flow activity is created by the FlowTracker constructor it uses the BAM AddRelatedActivity method to make the body activity a child activity of the flow activity. Moreover, the message interchange ID is recorded as a data item of the message received milestone. Similarly, at the other milestones in the flow where messages are received the body activities are related as children of the flow activity and the interchange IDs are recorded as data items in the flow.

When the orchestration came to send a message it would generate a GUID and assign this as the interchange ID of the sent message. The orchestration would also record the flow activity ID as a custom property in the message context. When the tracking component in the send pipeline processed the message it would create the body activity and use the AddRelatedActivity method to make this a child of the parent flow activity.

Finally, if the message was sent successfully, the flow activity would record this as a milestone along with the sent message interchange ID as a milestone.

With this infrastructure in place we had an efficient, robust and very performant mechanism for capturing both our message bodies and message flows.

Our next design decision was how to report this data. We considered a custom ASP.NET solution but decided on using SQL Reporting Services instead. SQL Reporting Services provides RAD facilites for creating reports that can be delivered in multiple formats (e.g. HTML, EXCEL, PDF etc.) and has facilities for caching, paging, resizing etc. that would all need custom development in ASP.NET. It even has facilities to produce scheduled reports that can be emailed to users.

Using SQL Reporting Services we created a top-level report that with multiple search parameters that produced a table of messsages. The operator can use this report to find messages for a particular date/time period, for different message types, from different sources, to different destinations, etc. A drill down link brings up a detailed report that shows the milestones of the flow and the associated data items and further drill down links display the original message bodies.

Using BAM we've created an efficient operational reporting tool. In the furture we'll use the same data to create management information reports and render these with SQL Reporting Services.

Welcome

2004-11-15T15:22:00.000+00:00

Hi and welcome to the real BizTalk blog.

This is the blog of Cassium Technologies' BizTalk consultants. Cassium is a consultancy working closely with Microsoft predominately in the City of London. We are engaged in cutting edge projects in several of the major investment banks and many of our projects rely on BizTalk Server 2004.

The idea of this blog is to report on how BizTalk is used out there, in the field and give feedback on real-life experiences on mission critical projects.

Hope you like it and find it useful.